Method and device for monitoring data anomaly detection, storage medium and electronic device

Abstract

The embodiment of the application provides a kind of monitoring data exception detection method and device, storage medium, electronic equipment, wherein the method comprises: obtaining the monitoring data set of target object;Data processing operation is executed to monitoring data set, and target data set is obtained;From target data set, extract key features, and obtain target feature set;Using pre-trained target network model, target data set and the exception detection operation of target feature set is executed, and target detection result is obtained.By the application, the problem that the accuracy of the object exception detection is poor in the related art is solved, and the accuracy of the object exception detection is improved.

Description

Technical Field

[0001] This application relates to the field of data processing, and more specifically, to a method and apparatus for detecting anomalies in monitoring data, a storage medium, and an electronic device. Background Technology

[0002] With the development of technology, the use of intelligent methods for anomaly detection is becoming increasingly common. However, due to the complexity of the environment in which the object is located—for example, aircraft flight involves altitude changes in the sea and air, and ships may be affected by ocean currents and wind—existing methods for anomaly detection still suffer from poor accuracy. Summary of the Invention

[0003] This application provides a method, apparatus, storage medium, and electronic device for detecting anomalies in monitoring data, in order to at least solve the problem of poor accuracy in detecting anomalies in related technologies.

[0004] According to one embodiment of this application, an anomaly detection method for monitoring data is provided, comprising: acquiring a monitoring data set of a target object; performing data processing operations on the monitoring data set to obtain a target data set; extracting key features from the target data set to obtain a target feature set; and performing anomaly detection operations on the target data set and the target feature set using a pre-trained target network model to obtain a target detection result. The target network model is a model obtained by training an initial network model using a set of sample data and a corresponding set of sample feature sets. The initial network model includes a first detection model and a second detection model. For each set of sample data and sample feature set, the first detection model outputs a first detection result based on the input sample data set and sample feature set, and the second detection model outputs a sample detection result based on the input sample data set, the sample feature set, and the first detection result.

[0005] In an exemplary embodiment, before performing an anomaly detection operation on the target dataset and the target feature set using a pre-trained target network model to obtain a target detection result, the method further includes: acquiring multiple sets of sample monitoring data sets of the sample object; performing the data processing operation on the multiple sets of sample monitoring data sets to obtain multiple sets of sample data sets; extracting key features from the sample data included in the multiple sets of sample data sets to obtain multiple sets of sample feature sets; and training the initial network model based on the multiple sets of sample data sets and the multiple sets of sample feature sets until the loss function corresponding to the initial network model satisfies a preset loss condition.

[0006] In one exemplary embodiment, obtaining multiple sets of sample monitoring data of a sample object includes: constructing a data transmission network between N determined data sources, wherein data acquisition devices in the N data sources are all connected to the sample object, and N is a natural number greater than or equal to 1; and using the data transmission network to obtain object information and running trajectory information of the sample object from the N data sources to obtain multiple sets of sample monitoring data.

[0007] In one exemplary embodiment, the data processing operation is performed on multiple sets of sample monitoring data to obtain multiple sets of sample data, including: performing data unification operations on the multiple sets of sample monitoring data to obtain multiple sets of standard data, wherein the data unification operation includes unifying the timestamps of data from N data sources included in the sample monitoring data set, and unifying the coordinate positions of data from the N data sources, where N is a natural number greater than or equal to 1; and performing data cleaning operations on the data in the multiple sets of standard data to obtain multiple sets of sample data.

[0008] In an exemplary embodiment, after performing the data processing operation on multiple sets of sample monitoring data to obtain multiple sets of sample data, the method further includes: classifying the multiple sample data according to the data type of the multiple sample data included in each set of sample data to obtain M types of data, wherein M is a natural number greater than or equal to 1; storing the M types of data into K databases respectively, wherein K is a natural number greater than or equal to 1; integrating the K databases into a preset storage architecture to obtain a distributed database; and performing a setting operation on the distributed database using preset data attributes to obtain a target distributed database, wherein the setting operation includes at least one of the following: data partitioning operation, data sharding operation, and data indexing operation, wherein the data in the target distributed database is stored hierarchically according to the data source.

[0009] In one exemplary embodiment, extracting key features from sample data included in multiple sets of sample data to obtain multiple sets of sample feature sets includes: extracting key features from multiple sample data according to the state features of multiple sample data included in multiple sets of sample data to obtain multiple sets of key feature sets, wherein the state features include static features and dynamic features; and using a preset machine learning algorithm to select features that meet a preset weight threshold from the multiple key features included in the multiple sets of key feature sets to obtain multiple sets of sample feature sets.

[0010] In an exemplary embodiment, after extracting key features from sample data included in multiple sets of sample data to obtain multiple sets of sample features, the method further includes: performing version control operations on multiple key features included in multiple sets of target features using a version control tool to determine the version feature of each key feature, wherein the version feature is used to identify feature changes of the key feature.

[0011] In an exemplary embodiment, training the initial network model based on multiple sets of sample data and multiple sets of sample feature sets until the loss function corresponding to the initial network model satisfies a preset loss condition includes: training the initial network model for the kth time through the following steps, where k is a positive integer: obtaining the kth set of training data used for the kth training of the initial network model from the multiple sets of sample data and multiple sets of sample feature sets of training data; inputting the sample data set and the sample feature set from the kth set of training data into the first detection model trained for the (k-1)th time to obtain the kth first detection result, wherein the first detection model is used to compare the sample data set and the sample feature set with preset anomaly rules and preset anomaly events, respectively, to obtain a first detection feature and a second detection feature, and to determine the kth first detection result by combining the first detection feature and the second detection feature; inputting the sample data set, the sample feature set, and the kth first detection result from the kth set of training data into the second detection model trained for the (k-1)th time to obtain the kth sample detection result.

[0012] In an exemplary embodiment, the second detection model determines the k-th sample detection result by: calculating the feature value of each sample feature in the sample feature set using a probability distribution model; using the feature value of each sample feature to determine abnormal sample features and normal sample features in the sample feature set that deviate from a preset numerical range; using a clustering algorithm to perform a clustering operation on the normal sample features to determine the normal behavior cluster of the normal sample features; using the normal behavior cluster and the historical behavior data of the sample object to establish a time series model of the normal behavior data of the sample object; and determining the k-th sample detection result based on the time series model, the abnormal sample features, the sample data set, and the k-th first detection result.

[0013] According to another embodiment of this application, an anomaly detection device for monitoring data is also provided, including a first memory, a first processor, and a first computer program stored in the first memory and executable on the first processor. When the first processor executes the first computer program, it performs the following operations: acquiring a monitoring data set of a target object; performing data processing operations on the monitoring data set to obtain a target data set.

[0014] Key features are extracted from the target dataset to obtain a target feature set; anomaly detection is performed on the target dataset and the target feature set using a pre-trained target network model to obtain a target detection result. The target network model is a model trained on an initial network model using a set of sample datasets and a corresponding set of sample feature sets. The initial network model includes a first detection model and a second detection model. For each set of sample datasets and sample feature sets, the first detection model outputs a first detection result based on the input sample datasets and sample feature sets, and the second detection model outputs a sample detection result based on the input sample datasets, sample feature sets, and the first detection result.

[0015] According to yet another embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0016] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0017] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0018] This application achieves a significant improvement in anomaly detection accuracy by acquiring and preprocessing monitoring data of the target object, extracting key and discriminative features, and employing two collaborative detection models for in-depth data analysis. The first detection model performs initial screening based on the raw monitoring data and key features, quickly filtering out most normal behaviors and reducing the burden on subsequent in-depth detection. The second detection model, building upon the first model, further integrates the preliminary detection results for refined data analysis. Therefore, this approach addresses the problem of poor accuracy in anomaly detection in related technologies, thereby improving the overall accuracy of anomaly detection. Attached Figure Description

[0019] Figure 1 This is a hardware structure block diagram of a server device for an anomaly detection method for monitoring data according to an embodiment of this application;

[0020] Figure 2 This is a flowchart of an anomaly detection method for monitoring data according to an embodiment of this application;

[0021] Figure 3 This is a flowchart illustrating the training process of the target detection model according to an embodiment of this application;

[0022] Figure 4 This is a flowchart illustrating the determination of the set of key features according to embodiments of this application;

[0023] Figure 5 This is a flowchart illustrating the determination of a target abnormal mode according to an embodiment of this application;

[0024] Figure 6 This is a structural block diagram of an anomaly detection device for monitoring data according to an embodiment of this application. Detailed Implementation

[0025] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0027] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of a server device for an anomaly detection method for monitoring data according to an embodiment of this application. For example... Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0028] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the anomaly detection method for monitoring data in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the aforementioned method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0029] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0030] This embodiment provides a method for detecting anomalies in monitoring data. Figure 2 This is a flowchart of an anomaly detection method for monitoring data according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0031] Step S202: Obtain the monitoring data set of the target object;

[0032] Optionally, this embodiment can be applied to various scenarios, including but not limited to maritime and airborne monitoring and safety, traffic management, environmental protection, and defense. For example, it can be used in air and maritime traffic control centers to detect anomalies in the trajectories of aircraft and ships to optimize flight paths, prevent collisions, and reduce traffic congestion. Environmental monitoring agencies can monitor activities that pollute the marine or aerial environment, such as illegal fishing, oil spills, or airborne pollution sources.

[0033] Optionally, the target objects in this embodiment include, but are not limited to, entities moving in the sea and air environment. These can be natural or man-made, and specifically can include aircraft, ships, mobile surface or underwater devices, artificial satellites, or space stations.

[0034] Optionally, the monitoring data set in this embodiment can be a data set formed by collecting and summarizing real-time or historical data about a specific target (such as an aircraft, ship, drone, etc.) from various sensors, devices, or systems. This data can include multi-dimensional information such as the target object's identification information, geographic coordinates, timestamps, speed, heading, and altitude, comprehensively reflecting the target's behavioral characteristics in different times and spaces. For example, monitoring data sets of aircraft can be obtained through air traffic control radar, Automatic Dependent Surveillance-Broadcast (ADS-B), meteorological satellites, airborne sensors, airport surveillance cameras, etc. The monitoring data set includes monitoring data such as the aircraft's flight number, aircraft type, flight altitude, flight speed, heading, takeoff and landing times, real-time location (latitude and longitude), and meteorological conditions (wind speed, cloud cover).

[0035] Step S204: Perform data processing operations on the monitoring data set to obtain the target data set;

[0036] Optionally, the data processing operations in this embodiment include, but are not limited to, the following operations:

[0037] 1) Data cleaning: Removing or correcting noisy and erroneous data in a dataset, including handling missing values, identifying and deleting outliers, and deduplication, to ensure data accuracy and consistency. For example, in aircraft monitoring data, signal interference may cause an abnormally high flight speed for a particular flight, and such values ​​need to be identified and addressed.

[0038] 2) Data format standardization: Convert monitoring data from different data sources into a unified format to facilitate subsequent processing and analysis. For example, unify the timestamps in data from Automatic Identification System (AIS), radar data, and satellite imagery data to Coordinated Universal Time (UTC), and convert geographic coordinates to a standard latitude and longitude format.

[0039] 3) Data transformation: Transforming and deriving from raw monitoring data to generate features with greater analytical value. For example, calculating the rate of change of heading from continuous position records of ships, or extracting climb and descent rates from flight altitude records of aircraft.

[0040] 4) Data Fusion: Integrating multi-source monitoring data, eliminating redundancy, filling gaps, and constructing a comprehensive dataset reflecting target behavior. For example, combining GPS data from drones with image data from onboard cameras can more accurately determine the drone's position and flight status, especially when GPS signals are weak, as image data can serve as supplementary information.

[0041] After the above data processing operations, the resulting target dataset should possess the following characteristics: Completeness: The dataset should contain all relevant target behavior information, with no missing important data. Consistency: The data should be in a uniform format, and information such as timestamps and geographic coordinates should be accurate. Validity: The data in the dataset should be cleaned, free from noise and errors that could affect the analysis results. Real-time Performance: For real-time monitoring data, the processed dataset should be able to be updated promptly, reflecting the latest behavior of the target. Structure: The target dataset should be stored in a structured manner to facilitate subsequent feature extraction and model training.

[0042] Step S206: Extract key features from the target data set to obtain a target feature set;

[0043] Optionally, key features may include features directly extracted from the raw data, such as position, speed, altitude, and heading. They may also include derived features, i.e., features calculated based on the basic features, such as the rate of change of heading, the trend of speed change, and the Closest Point of Approach (CPA) to the nearest target.

[0044] Alternatively, feature selection algorithms (such as recursive feature elimination, model-based feature selection, and correlation analysis) can be used to identify which features are most influential for anomaly detection. Key features should reflect specific behavioral patterns of the target in the sea and air environment, such as high-speed flight, slow navigation, and frequent turning.

[0045] Step S208: Perform anomaly detection on the target data set and the target feature set using a pre-trained target network model to obtain a target detection result. The target network model is a model trained on an initial network model using a set of sample data and a corresponding set of sample features. The initial network model includes a first detection model and a second detection model. For each set of sample data and the sample feature set, the first detection model outputs a first detection result based on the input sample data set and the sample feature set, and the second detection model outputs a sample detection result based on the input sample data set, the sample feature set, and the first detection result.

[0046] Optionally, the first detection model in this embodiment can be a relatively simple model that can quickly output preliminary first detection results based on the input sample data set and sample feature set. This step can be regarded as a preliminary filtering or classification of the data, helping to quickly exclude most normal behaviors and reduce the computational load of subsequent processing.

[0047] The second detection model builds upon the first model, analyzing the data in greater depth. Specifically, by using the first detection result as additional input, it can be a more complex neural network model used to generate more detailed sample detection results based on the input sample data set, sample feature set, and the first detection result. This model focuses on identifying potential abnormal patterns, and through deep learning techniques, it can identify abnormal tracks, pauses, deviations from the normal path, and other behaviors based on historical behavioral patterns.

[0048] Optionally, the sample dataset can be a collection of historical air and sea target trajectory data, which has been preprocessed and feature extracted to ensure quality and applicability. The sample feature set is a collection of key features extracted from the sample dataset, including basic features (such as position and velocity), advanced features (such as rate of change of heading and dwell time), and environmental features (such as weather and sea state). These features combine to form a comprehensive perspective describing the target's behavior.

[0049] In this embodiment, the entity executing the above steps can be a specific processor or controller in a ground-based device, or a processor or processing device or controller that is set up relatively independently from the ground control device, but is not limited to these.

[0050] Through the above steps, by acquiring and preprocessing the monitoring data of the target object, extracting key and discriminative features, and employing two collaborative detection models to perform in-depth analysis of the data, the first detection model performs preliminary screening based on the raw monitoring data and key features, quickly filtering out most normal behaviors and reducing the pressure on subsequent in-depth detection. The second detection model, building upon the first model, further integrates the preliminary detection results to perform refined analysis of the data, thereby significantly improving the accuracy of anomaly detection. Therefore, this approach can solve the problem of poor accuracy in anomaly detection of objects in related technologies, achieving the effect of improving the accuracy of anomaly detection.

[0051] In an exemplary embodiment, before performing an anomaly detection operation on the aforementioned target data set and target feature set using a pre-trained target network model to obtain the target detection result, the method further includes:

[0052] Step S302: Obtain a set of multiple sets of sample monitoring data for the sample object;

[0053] Step S304: Perform the above data processing operation on multiple sets of the above sample monitoring data to obtain multiple sets of the above sample data.

[0054] Step S306: Extract key features from the sample data included in the multiple sets of the above-mentioned sample data sets to obtain multiple sets of the above-mentioned sample feature sets.

[0055] Step S308: Train the initial network model based on multiple sets of sample data and multiple sets of sample features until the loss function corresponding to the initial network model satisfies the preset loss condition.

[0056] Optionally, the sample object can be the same as or a different object from the target object in the above embodiments. It can refer to air and sea targets used to train the model, such as aircraft or ships of specific types and operational states. The diversity of sample objects helps the model learn normal and abnormal behavior patterns in different scenarios.

[0057] Optionally, the multi-sample monitoring dataset is a spatiotemporal data collection gathered from multiple sample objects. This data includes instances of normal and abnormal behavior under different environments and time periods. Each sample dataset corresponds to a specific sample object, which can be its motion trajectory data within a specific time period.

[0058] Alternatively, in machine learning, a loss function is a function used to quantify the difference between the model's predictions and the actual results. During training, the model adjusts its parameters by minimizing the loss function to improve prediction accuracy.

[0059] Optionally, the preset loss condition is a stopping condition during training. The training process stops when the loss function reaches a set threshold or meets a specific convergence condition. The setting of the preset loss condition needs to take into account the complexity of the model, the characteristics of the data, and the expected detection accuracy.

[0060] For example, in one specific embodiment, before using a pre-trained target network model for anomaly detection, the model first needs to be trained with multiple sets of sample data to ensure that it can effectively distinguish between normal and abnormal sea and air target behavior. The specific implementation steps are as follows:

[0061] S1, Data Collection and Preprocessing:

[0062] Data collection: Multiple sets of sample monitoring data are collected from different types of air and sea targets (such as civil aircraft, cargo ships, and drones). This data includes the target's position coordinates, timestamps, speed, heading, and related environmental information, such as weather and ocean currents.

[0063] Data preprocessing: Data cleaning, standardization, and feature extraction are performed on multiple sets of sample monitoring data to obtain multiple sets of sample data. This step ensures data quality and provides a reliable foundation for subsequent feature extraction and model training.

[0064] S2, Feature Extraction: Extract key features from multiple sets of sample data to form multiple sets of sample features. These key features include, but are not limited to, spatiotemporal features (such as changes in position coordinates), kinematic features (such as velocity and acceleration), and behavioral pattern features (such as stop point detection and repeated path recognition).

[0065] S3, Feature Set Version Control: Version control is implemented for each set of sample features, recording the parameters, methods, and processes of feature extraction to ensure the traceability of the training process and the verifiability of the model. Version control is crucial for the continuous optimization and upgrading of the model.

[0066] S4, Model Training: An initial network model is trained using multiple sets of sample data and feature sets. During training, the first and second detection models work together. The first model handles initial anomaly identification, while the second model performs deeper analysis and judgment based on the first model's output. During training, the model adjusts its internal parameters based on feedback from the loss function to minimize the difference between the predicted results and the actual sample data. This process may require multiple iterations until the loss function meets preset conditions, such as reaching a minimum point or stabilizing within a certain threshold.

[0067] Through the above steps, this embodiment can construct a model capable of accurately detecting anomalies in sea and air targets.

[0068] In an exemplary embodiment, in step S302, obtaining multiple sets of sample monitoring data of the sample object includes: constructing a data transmission network between N determined data sources, wherein the data acquisition devices in the N data sources are all connected to the sample object, and N is a natural number greater than or equal to 1; using the data transmission network to obtain object information and running trajectory information of the sample object from the N data sources to obtain multiple sets of sample monitoring data.

[0069] Optionally, N data sources refer to multiple different data collection points. These points can be various sensors, monitoring devices, data recording systems, etc., which are directly or indirectly connected to the sample object, providing diverse information for anomaly detection. N represents the number of data sources, a natural number greater than or equal to 1, meaning there can be one or more data sources. For example, data can be collected from multiple sources such as radar, AIS systems, satellite imaging, ground sensors, and drones. This data includes the target's unique identifier, timestamp, geographic coordinates (such as latitude and longitude), speed, altitude, heading, etc.

[0070] Optionally, the data transmission network refers to the network infrastructure connecting all data sources and the central data processing system. It ensures that data is transmitted quickly, securely, and accurately from each data source to the central processing node. The design of the data transmission network must consider factors such as data volume, transmission rate, data format, and security. For example, establishing a coordinated sensor network to monitor and collect data from target areas; combining remote sensing technology to acquire large-scale and real-time data from maritime and air platforms, etc.

[0071] Optionally, object information refers to basic information about the sample object, including but not limited to the object's type (such as cargo ship, military aircraft), size, speed, heading, load information, etc., which helps to understand the object's behavior patterns in a specific environment.

[0072] Optionally, the trajectory information is used to record the movement trajectory of the sample object within a certain period of time, including position coordinates, timestamp, speed, altitude, heading, etc., which is the core data for anomaly detection.

[0073] For example, in a specific implementation of monitoring a cargo ship:

[0074] Data sources include AIS systems, shipborne GPS, weather stations, ocean current monitoring stations, and maritime traffic monitoring centers.

[0075] Data transmission network: It adopts a hybrid network architecture, which includes real-time wireless data stream transmission (such as AIS and GPS data) as well as timed data packet transmission (such as weather and ocean current data).

[0076] Data Acquisition: The AIS system provides basic information such as the cargo ship's position, course, and name; GPS provides more accurate position coordinates and speed information; weather stations and ocean current monitoring stations provide environmental data; and the maritime traffic monitoring center provides dynamic information on surrounding targets.

[0077] Data fusion and preprocessing: In the central data processing system, data fusion technology and preprocessing algorithms are used to integrate information from all data sources to form a time-synchronized and formatted sample monitoring data set, which is then used for feature extraction and model training.

[0078] This embodiment constructs an efficient and secure data transmission network and acquires and processes detailed information of sample objects from multiple data sources, providing rich and high-quality training data for the anomaly detection model. This improves the model's detection accuracy and generalization ability, ultimately achieving real-time and accurate anomaly detection of sea and air target trajectories.

[0079] In an exemplary embodiment, in step S304, the data processing operation is performed on multiple sets of sample monitoring data to obtain multiple sets of sample data, including: performing data unification operations on the multiple sets of sample monitoring data to obtain multiple sets of standard data, wherein the data unification operation includes unifying the timestamps of the data from N data sources included in the sample monitoring data set, and unifying the coordinate positions of the data from the N data sources, where N is a natural number greater than or equal to 1; and performing data cleaning operations on the data in the multiple sets of standard data to obtain multiple sets of sample data.

[0080] Optionally, data unification is a key step in processing multi-source spatiotemporal data, aiming to solve the problem of temporal and spatial data inconsistencies between different data sources. Specifically, it may include:

[0081] Timestamp unification: Since different data sources may use different time bases, such as local time or UTC time, it is first necessary to unify the timestamps of all data to the same time base, usually Coordinated Universal Time (UTC). This process can be achieved by adding global time synchronization logic during the data preprocessing stage, such as using time synchronization protocols like Network Time Protocol (NTP) to calibrate the timestamps of each data source.

[0082] Coordinate unification: Similarly, geographical location information from different data sources may use different coordinate systems, such as WGS-84, Beijing 54, or other local coordinate systems. To ensure all data can be correctly displayed and analyzed on a unified map, the coordinates need to be unified to a standard coordinate system, with WGS-84 being a common choice. This can be achieved using GIS software or corresponding coordinate transformation APIs.

[0083] Data cleaning operations aim to remove noise, outliers, and missing values ​​from the data, ensuring that subsequent feature extraction and model training can be performed on a clean and accurate data basis. Specifically, this includes: Outlier detection and handling: Using statistical methods to identify and handle outliers in the data, such as speed or height values ​​exceeding the normal range, which may be caused by sensor reading errors or signal interference. Missing value imputation: For missing data points, methods such as interpolation, average substitution, or time series prediction can be used to imput them, maintaining the continuity and integrity of the data. Data deduplication: Removing duplicate records to avoid repeatedly processing the same data point, which affects the accuracy of feature extraction and the efficiency of model training.

[0084] For example, in a specific embodiment, taking a civil flight as an example, flight data may be collected from multiple data sources such as radar, ADS-B, and satellite communications. First, by unifying timestamps and coordinate positions, it is ensured that all data are on the same temporal and spatial scale. Then, data cleaning operations are performed to remove occasional erroneous readings from sensors (such as abnormally high instantaneous speeds), ensuring data reliability. Finally, the constructed sample dataset will be used to train an anomaly detection model, helping airlines and air traffic management departments to promptly detect potential flight anomalies, such as deviations from planned routes, sudden acceleration or deceleration, thereby improving flight safety.

[0085] This embodiment, through the above steps, not only ensures data quality but also provides a solid foundation for subsequent feature extraction and model training, thereby improving the accuracy and real-time performance of anomaly detection for sea and air targets.

[0086] In an exemplary embodiment, after performing the above-described data processing operations on multiple sets of the above-described sample monitoring data to obtain multiple sets of the above-described sample data, the method further includes: for each set of the above-described sample data, classifying the multiple sets of sample data according to the data types of the multiple sets of sample data included in the sample data to obtain M types of data, wherein M is a natural number greater than or equal to 1; storing the M types of the above-described data into K databases respectively, wherein K is a natural number greater than or equal to 1; integrating the K databases into a preset storage architecture to obtain a distributed database; and performing setting operations on the distributed database using preset data attributes to obtain a target distributed database, wherein the setting operations include at least one of the following: data partitioning operation, data sharding operation, and data indexing operation, wherein the data in the target distributed database is data stored hierarchically according to the data source.

[0087] Optionally, sample data classification refers to categorizing data into multiple classes (M classes) based on the data type within the sample dataset, facilitating subsequent storage, retrieval, and analysis. Data types may include, but are not limited to: relational data, key metadata, unstructured data, and semi-structured data.

[0088] Optionally, the categorized data can be stored in multiple databases (K databases), each potentially specializing in storing a particular type of data to improve data storage efficiency and security. Examples include structured databases for storing relational data and key metadata, and NoSQL databases for storing unstructured and semi-structured data.

[0089] Alternatively, a distributed database refers to a database system that distributes data across multiple databases or servers. It can improve data processing speed through parallel processing, ensure data integrity and availability through data redundancy, and reduce the pressure on a single database through load balancing technology.

[0090] Optionally, a pre-designed storage architecture refers to the database structure and data management mechanisms pre-designed before building a distributed database, including data storage formats, access control policies, and data backup and recovery strategies. For example, integrating a structured database and a NoSQL database into a single storage architecture results in a distributed database.

[0091] Optionally, data attribute settings include operations such as data partitioning, data sharding, and data indexing. These settings help optimize data storage and query efficiency, improving the performance of the distributed database. For example, partitioning and sharding the distributed database based on natural keys (such as geographic location, time, etc.) enables parallel querying and data processing, and creating multi-level indexes (such as geospatial indexes and time indexes) in the distributed database enables fast query operations.

[0092] In the distributed database, multi-source datasets are stored in layers according to the lifecycle and usage frequency of each data source (e.g., data is divided into cold, hot, and warm layers), storage resources are allocated reasonably, and long-term inactive and useless data are archived and cleaned up to free up storage space.

[0093] In distributed databases, multi-source datasets are stored in hierarchical layers based on their lifecycle and usage frequency. Common conditions for layering are as follows:

[0094] Hot data: data that is frequently accessed and currently in active use (such as near real-time trajectories and data used for online analysis);

[0095] Warm data: Data accessed at a moderate frequency and used periodically for analysis;

[0096] Cold data: Historical archives, data that is rarely accessed but needs to be checked (such as long-term historical trajectories);

[0097] The stratification criteria include: the time when the data was generated, the current activity level, and the relevance of tasks, which helps with resource optimization and data management.

[0098] It also allows for the implementation of access control policies and encrypted transmission protocols (such as TLS / SSL) in distributed databases to protect data transmission and storage security, and the use of big data processing frameworks for real-time data stream processing and analysis.

[0099] This embodiment constructs a distributed database system capable of efficiently processing and analyzing multi-source spatiotemporal data, providing a solid data foundation for detecting anomalies in the trajectories of sea and air targets. The system can monitor the dynamics of sea and air targets in real time and perform in-depth analysis using historical data, improving the accuracy and coverage of monitoring.

[0100] In one exemplary embodiment, extracting key features from sample data included in multiple sets of sample data to obtain multiple sets of sample feature sets includes: extracting key features from multiple sample data according to the state features of multiple sample data included in multiple sets of sample data to obtain multiple sets of key feature sets, wherein the state features include static features and dynamic features; and using a preset machine learning algorithm to select features that meet a preset weight threshold from multiple key features included in multiple sets of key feature sets to obtain multiple sets of sample feature sets.

[0101] Optionally, in the fields of anomaly detection or data analysis, state features refer to attributes that can describe a data point or entity at a certain time or state, and they are divided into static features and dynamic features.

[0102] Static features refer to features that do not change over time, such as the type, size, and load of sea and air targets. These features provide basic information about the target in anomaly detection.

[0103] Dynamic characteristics refer to features that change over time, such as position coordinates, speed, and heading. These characteristics can reflect the target's motion and behavior in space and time.

[0104] Key features: Among a large number of features, key features refer to those that have a significant impact on the prediction model or analysis results. Extracting key features can reduce data dimensionality, simplify anomaly detection models, and maintain or improve detection accuracy.

[0105] For example, static features can be the basic features in a sample dataset, including spatiotemporal features and kinematic features.

[0106] The spatiotemporal features include: time features, such as timestamps, dates, time intervals (e.g., the interval between two records), and daily time periods; and spatial features, such as geographic coordinates (latitude and longitude) and displacement vectors (spatial displacement between two points).

[0107] Kinematic features include: velocity, calculating the instantaneous velocity at each trajectory point; acceleration, calculating instantaneous acceleration using the rate of change of velocity; heading change, calculating the heading angle and its rate of change based on continuous position changes; and path curvature, calculating the trajectory curvature change based on the trajectory shape, used to identify behaviors such as sharp turns or serpentine maneuvers.

[0108] Optionally, dynamic features can be higher-order features in the sample dataset, including behavioral pattern features, statistical features, and geographical and environmental features.

[0109] Behavioral pattern features include: stop point detection, which identifies points in the trajectory where the target stays for a long time (such as parking areas); and loop path detection, which identifies paths that the target repeatedly passes through (such as patrol paths).

[0110] Statistical features include: mean and variance, which calculate the mean and variance of various motion features to describe the stability of the trajectory; extreme value analysis, which extracts the maximum and minimum values ​​of features such as velocity, acceleration, and heading to identify abnormal behavior.

[0111] Geographic and environmental features include: geographical environmental features, which utilize Geographic Information System (GIS) to extract topographic and marine features along the target's path during its movement; and environmental data integration, which integrates external environmental features such as weather conditions (wind speed, wind direction, and sea state) as additional features.

[0112] Optionally, machine learning algorithms are algorithms used to learn patterns and regularities from data, including but not limited to decision trees, random forests, support vector machines (SVMs), and neural networks. In anomaly detection, machine learning algorithms are often used for feature selection, pattern recognition, and predictive modeling.

[0113] During feature selection, each feature is assigned a weight value, which reflects the importance of the feature to the model. The weight threshold is a pre-set value used to select key features with weight values ​​higher than the threshold.

[0114] For example, algorithms (such as decision trees and Lasso regression) can be used to evaluate the importance of features and select key features. Principal component analysis or t-SNE methods can be used to reduce the dimensionality of the key features to reduce feature redundancy, improve computational efficiency, and obtain a set of key features.

[0115] This embodiment extracts static, dynamic, and environmental features from a sample dataset and then uses machine learning algorithms for feature selection to obtain a set of key features that can effectively describe the state of sea and air targets. This set of key features not only provides real-time motion information of the targets but also considers target type, size, and environmental factors, thus providing a more comprehensive and refined data foundation for subsequent anomaly detection and improving the model's detection accuracy and efficiency.

[0116] In an exemplary embodiment, after extracting key features from sample data included in multiple sets of sample data to obtain multiple sets of sample features, the method further includes: performing version control operations on multiple key features included in multiple sets of target features using a version control tool to determine the version feature of each key feature, wherein the version feature is used to identify feature changes of the key feature.

[0117] Optionally, version control is primarily used to track and manage the change history of code or datasets, allowing team members to collaborate on development while also being able to revert to previous versions at any time. In feature management, version control is used to record every change to key features, ensuring that data scientists and engineers can track the evolution of features.

[0118] Optionally, in the context of detecting anomalies in the trajectories of sea and air targets, feature variation refers to the differences in key features between different versions. These differences may be due to changes in data sources, algorithm optimization, or the addition of new information.

[0119] Optionally, version features are a set of identifiers and descriptive information associated with the version of each set of key features, used to record when the key features of that version were created, by whom, what changes were made, and why such changes were made.

[0120] For example, the process of version control for a set of key features specifically includes:

[0121] S1. Choose a tool and use a version control tool such as Git, Mercurial, or Subversion. These tools can help manage and track historical changes to a feature set.

[0122] S2, assign a version number and description information (such as V1.0, V1.1) to each key feature to identify the differences between different versions, and define the extraction code for each key feature;

[0123] S3 initializes a repository for storing version numbers, description information, and extracted code, and establishes a structured directory and rollback mechanism in the repository based on the corresponding key features. The structured directory organizes feature files by project, target, and version, making them easy to manage and find. The rollback mechanism provides simple commands or scripts to roll back to the previous feature set version through the version control system, ensuring that if analysis errors occur, a quick return to a stable version can be achieved.

[0124] S4. When the set of key features changes, the repository records the changes through logs and uses the branching feature of the repository to develop a new branch. The change records on the new branch and the main branch are merged, and the rationality of the changes in the set of key features is verified by code review (such as peer review to ensure the rationality and correctness of feature changes).

[0125] When the set of key features changes, newly added data is accompanied by metadata for tracing and understanding the data. Metadata includes information such as author, time, change description, and environmental dependencies to support future tracing and understanding.

[0126] This embodiment, through the steps described above, not only maintains multiple versions of the key feature set but also clearly tracks the evolution of each feature. This method is applicable not only to ship monitoring but also to anomaly detection scenarios for air and sea targets such as aircraft and drones, ensuring transparency and traceability in feature management and improving the efficiency and quality of model development.

[0127] In an exemplary embodiment, training the initial network model based on multiple sets of sample data and multiple sets of sample feature sets until the loss function corresponding to the initial network model satisfies a preset loss condition includes: training the initial network model for the kth time through the following steps, where k is a positive integer: obtaining the kth set of training data used for the kth training of the initial network model from the training data of multiple sets of sample data and multiple sets of sample feature sets; inputting the sample data set and sample feature set from the kth set of training data into the first detection model trained for the (k-1)th time to obtain the kth first detection result, wherein the first detection model is used to compare the sample data set and the sample feature set with preset anomaly rules and preset anomaly events, respectively, to obtain first detection features and second detection features, and to determine the kth first detection result by combining the first detection features and the second detection features; inputting the sample data set, the sample feature set, and the kth first detection result from the kth set of training data into the second detection model trained for the (k-1)th time to obtain the kth sample detection result.

[0128] Optionally, the initial network model refers to a machine learning or deep learning model that has not been fully trained, whose parameters have not yet reached their optimal level, and whose performance needs to be improved. In this scenario, the initial network model may be a deep learning model for detecting anomalies in the trajectories of sea and air targets, such as a convolutional neural network (CNN) or a long short-term memory network (LSTM).

[0129] A loss function measures the difference between a model's predictions and the true values; it is the optimization objective during model training. Common loss functions include mean squared error (MSE), cross-entropy loss, and mean absolute error (MAE).

[0130] The preset loss condition is a set threshold for the loss function. When the model reaches or falls below this threshold during training, it can be considered that the model has converged or achieved satisfactory performance, and training can be stopped at this point.

[0131] For example, in one specific embodiment, the scheme for initial anomaly detection using preset anomaly rules specifically includes:

[0132] S1, set speed limit rules, heading rate of change rules, and area restriction rules according to the actual environment;

[0133] 1) Speed ​​limit rules include: speed range, which sets the normal speed range for different air and sea targets (such as passenger aircraft, cargo ships, etc.) based on their type and purpose; and speed limit, which sets speed limits to adapt to different conditions, taking into account the impact of environmental factors (such as weather and wind speed) on speed.

[0134] 2) The heading change rate rules include: the range of course changes, analyzing the rate and frequency of change of the target heading under normal conditions, and setting an acceptable turning range; and the change-of-course monitoring and detection, identifying rapid turning behaviors that may indicate danger or abnormality, and setting strict monitoring rules.

[0135] 3) Area restriction rules include: area restriction range, using Geographic Information System (GIS) to mark no-navigation zones, protected areas and other important areas, and setting area restriction rules that prohibit entry; trajectory restriction range, for targets with clear flight paths, setting how far they deviate from the flight path before being judged as abnormal;

[0136] S2, Threshold setting: Based on the multi-source dataset and the set rules, use statistical methods (such as mean, standard deviation, percentile, etc.) to determine normal behavior patterns and identify abnormal behaviors, and set an abnormality detection threshold according to the identified abnormal behaviors.

[0137] For specific target types and operational areas, discussions can be held with shipping and aviation safety experts to obtain information on key risk points in actual operations;

[0138] S3, execute the set rules and the anomaly detection threshold to perform initial anomaly detection. Initial anomaly detection can flexibly and efficiently perform rule-based target trajectory anomaly detection, ensuring real-time discovery and response to potential anomalies.

[0139] For example, in one specific embodiment, the scheme for initial anomaly detection using preset anomaly events specifically includes:

[0140] S1, Dwelling Event Detection: Define the dwelling criteria for the target (e.g., the time the speed is below a certain threshold within a small range exceeds a preset value) and identify the dwelling areas of the target (e.g., determine which areas are unexpected dwelling areas, such as the middle of the channel, restricted areas, sensitive areas, etc.); use a clustering algorithm to identify densely packed points of the target location in the target trajectory and determine whether dwelling exists. If the determination result is yes, calculate the target's speed and dwell time in the dwelling area to obtain the dwelling result, and then determine whether the dwelling result meets the dwelling criteria. If the determination result is no, a dwelling event is obtained, the dwelling event is recorded and alarmed, and in-depth analysis is performed to determine whether there is any abnormal behavior.

[0141] S2, Repeated Path Event Detection: Define the criteria for repeated paths of a target (e.g., passing through the same geographical area or route multiple times within a short period) and identify repeated behaviors of the target within a short period (detecting repeated paths within a time window to identify repeated behaviors within a short period); calculate the path similarity of the target within different time periods using a similarity measurement method and compare the current path with historical paths to obtain repeated path detection results; determine whether the repeated path detection results contain repeated behaviors; if the determination result is yes, a repeated path event is obtained; record and alarm the repeated path event; and conduct in-depth analysis to determine whether there are any abnormal behaviors.

[0142] S3. Combine the dwell events and the repeated path events for comprehensive analysis, set priorities according to the severity of the events, and then allocate monitoring resources according to the priorities of different events.

[0143] This embodiment, through the steps described above, can train a deep learning model capable of automatically learning and identifying anomalies in the trajectories of sea and air targets, greatly improving monitoring efficiency and accuracy. The model can not only identify known abnormal behavior patterns but also discover new and unknown abnormal features through deep learning, providing a powerful intelligent analysis tool for sea and air safety monitoring.

[0144] In an exemplary embodiment, the second detection model determines the kth sample detection result by: calculating the feature value of each sample feature in the sample feature set using a probability distribution model; using the feature value of each sample feature to determine abnormal sample features and normal sample features that deviate from a preset numerical range in the sample feature set; using a clustering algorithm to perform clustering operations on the normal sample features to determine the normal behavior cluster of the normal sample features; using the normal behavior cluster and the historical behavior data of the sample object to establish a time series model of the normal behavior data of the sample object; and determining the kth sample detection result based on the time series model, the abnormal sample features, the sample data set, and the kth first detection result.

[0145] Optionally, in statistics, probability distribution models describe the probabilistic patterns of the values ​​taken by random variables. In machine learning, probability distribution models are used to describe the characteristic distribution of data, such as Gaussian, Poisson, and exponential distributions.

[0146] Optionally, a feature value refers to the specific numerical value of each sample feature, which reflects the characteristics of the sample. In anomaly detection of sea and air target trajectories, feature values ​​may include the target's speed, heading, dwell time, frequency of repeated paths, etc.

[0147] Optionally, deviation from a preset numerical range: When identifying anomalous features, a numerical range is set. If the feature value of a sample exceeds this range, the feature is considered anomalous. The preset numerical range can be determined based on statistical principles (such as the three-standard-deviation principle) or domain knowledge.

[0148] Alternatively, clustering algorithms are unsupervised learning methods that automatically group samples with similar characteristics into clusters. In anomaly detection, clustering algorithms are used to identify normal behavioral patterns, thereby distinguishing abnormal behavior from normal behavior.

[0149] Optionally, a time series model is a statistical model used to predict or analyze data that changes over time. In the detection of anomalies in the trajectories of sea and air targets, time series models can capture the periodicity or trend of target movement, helping to identify anomalous behavior.

[0150] For example, in one specific embodiment, the process of performing deep anomaly detection includes:

[0151] S1. Establish a probability distribution model (such as Gaussian distribution) for each key feature to obtain the normal mean and variance of each key feature. Use statistical methods to set the threshold range of normal behavior and compare the real-time data of the target with the probability distribution model to obtain the trajectory points that deviate from the normal range.

[0152] S2. The target trajectory data is initially clustered using a clustering algorithm to obtain the center point and outlier points. The center point of each cluster is analyzed (to understand typical behavior patterns) to obtain normal behavior patterns. Outlier points of the clusters are identified and new clustering patterns are introduced for monitoring and detection to obtain abnormal behavior.

[0153] S3. Combine historical data to build a time series model of normal behavior for periodic analysis. Compare the real-time data with the output of the time series model to obtain the time series deviation. Based on the time series deviation, detect the trend in the time series and find the point of sudden change or trend inflection point.

[0154] S4 combines trajectory points that deviate from the normal range, abnormal behavior, and trend inflection points to perform deep anomaly detection.

[0155] Through the steps described above, the second detection model in this embodiment can more accurately identify anomalies in each iteration, which not only improves the accuracy of detection, but also further enhances its detection performance as data accumulates and the model is continuously trained.

[0156] The present application will be described in detail below with reference to specific embodiments:

[0157] This specific embodiment uses an anomaly detection system for spatiotemporal target trajectories in a scenario used for maritime and air monitoring as an example to illustrate an anomaly detection system, including:

[0158] The data access layer receives spatiotemporal data from sensors, satellites, and radar in real time.

[0159] The data processing layer implements data preprocessing, model prediction, and anomaly detection;

[0160] The results output layer is responsible for outputting the anomaly detection results and triggering the anomaly response mechanism.

[0161] The data access layer, data processing layer, and result output layer can be encapsulated into an anomaly detection model. This anomaly detection model is trained using machine learning based on anomalous data. The specific training process includes:

[0162] Supervised and unsupervised learning models are selected to perform deep learning on the normal and abnormal behavior patterns of objects. Then, the supervised and unsupervised learning models after deep learning are combined to obtain the object detection model.

[0163] Supervised learning models can perform: classification algorithms, using classification models such as Support Vector Machine (SVM), Random Forest, and Gradient Boosting Tree (GBDT) to build anomaly detectors; and deep learning, using deep neural networks (such as LSTM or GRU) to process complex time series features.

[0164] Unsupervised learning models can: perform cluster analysis, using clustering algorithms such as K-means and DBSCAN to define normal patterns and thus identify anomalies; and use anomaly detection algorithms, using algorithms such as isolated forest and anomaly detection autoencoders to detect data points that do not conform to normal patterns.

[0165] The training and optimization process also includes: using K-fold cross-validation to test the stability and performance of the model, using grid search or random search to adjust the model's hyperparameters to optimize performance, using feature importance ranking to identify the features that contribute the most to the model, and considering the removal of irrelevant features, etc.

[0166] like Figure 3 As shown, the object detection model is trained through the following steps:

[0167] S301, collect monitoring data (corresponding to the sample monitoring data set mentioned above) and preprocess the monitoring data to obtain a multi-source dataset (corresponding to the sample data set mentioned above), and design a distributed database for storing and managing the multi-source dataset;

[0168] S302, extract features from the multi-source dataset to obtain a feature set, select key features from the feature set to obtain a key feature set (corresponding to the sample data set mentioned above), and then perform version control on the key feature set to manage and record changes to the feature set;

[0169] like Figure 4 As shown, determining the set of key features includes the following steps:

[0170] S401, based on multi-source datasets, performs data cleaning (handling missing values, outliers, and duplicate records) and data normalization (converting to a uniform scale) to obtain a multi-source standard dataset;

[0171] S402, based on a multi-source standard dataset, performs basic feature extraction and high-order feature extraction and integration to obtain the P feature set; basic features include spatiotemporal features and kinematic features, and high-order features include behavioral pattern features, statistical features, and geographical and environmental features; the basic features and high-order features have been described in the above embodiments and will not be repeated here;

[0172] S403: Based on the feature set, use algorithms (such as decision trees, Lasso regression) to evaluate the importance of features and select key features. Use principal component analysis or t-SNE to reduce the dimensionality of the key features to reduce feature redundancy and improve computational efficiency, thus obtaining the key feature set.

[0173] S303, based on multi-source datasets and feature sets, uses rule detection and event detection methods for initial anomaly detection, and then uses statistical models, cluster analysis and time series analysis methods for deep anomaly detection to obtain the target anomaly pattern;

[0174] like Figure 5 As shown, determining the target anomaly pattern includes the following steps:

[0175] S502, based on multi-source datasets and feature sets, uses rule-based detection and event-based detection methods for initial anomaly detection;

[0176] S504, then uses statistical models, cluster analysis and time series analysis to perform deep anomaly detection to obtain the target anomaly pattern.

[0177] S304. Machine learning is performed based on the target anomaly pattern to obtain an anomaly detection model (corresponding to the target network model mentioned above). Then, a system architecture for detecting target trajectories is designed based on the anomaly detection model to obtain an anomaly detection system. The anomaly detection system is used to perform real-time anomaly detection of sea and air target trajectories to obtain anomaly detection results.

[0178] The working process of the trained anomaly detection model includes:

[0179] S1, Data Access and Preprocessing:

[0180] Real-time data stream reception: First, receive real-time data streams from various sensors (such as radar, AIS, ground sensors, drones, etc.). These data include the target's unique identifier, timestamp, geographic coordinates (such as latitude and longitude), speed, altitude, heading, and other information.

[0181] Preprocessing: The received data is preprocessed, including data cleaning (removing missing values, outliers, and duplicate records), data normalization (unifying timestamps and coordinate systems), and the same data integration process as the training phase, to ensure that the format and quality of the input data are consistent with the training data.

[0182] S2, Feature Extraction:

[0183] Basic feature extraction: Extract basic features from real-time data, such as spatiotemporal features (position, time) and kinematic features (velocity, acceleration, heading changes).

[0184] Higher-order feature calculation: Higher-order features are calculated based on basic features, including behavioral pattern features (such as stop point detection and loop path recognition), statistical features (such as the mean and standard deviation of speed and heading), and geographical and environmental features (such as topography, sea area type, and weather conditions).

[0185] Key feature set selection: Utilize version control information from the training phase to select the latest version of the key feature set, transform real-time data into values ​​of these key features, and prepare them for input into the anomaly detection model.

[0186] S3, Anomaly Detection:

[0187] Initial anomaly detection: Real-time data and key feature sets are used to perform preliminary anomaly detection according to the rule detection and event detection methods used in the training phase, such as speed threshold detection, heading change rate analysis, dwell event identification, and repeated path detection.

[0188] Deep anomaly detection: This step involves using a trained statistical model, cluster analysis, and time series analysis to perform in-depth analysis on the initially screened data. In this step, the model identifies more complex and potential anomalous behaviors based on deviations between real-time data and historical normal behavior patterns.

[0189] S4, Model Predictions and Results Output:

[0190] Input features into the model: The processed real-time data (including the values ​​of basic and key features) is input into the anomaly detection model, which makes predictions based on the learned normal and abnormal behavior patterns.

[0191] Model prediction: The model outputs an anomaly score for each target trajectory. The higher the anomaly score, the greater the deviation of the trajectory or behavior pattern from the normal pattern, and the more likely it is to be marked as an anomaly.

[0192] Result judgment: Set an anomaly detection threshold. If the anomaly score of a target exceeds the threshold, it is marked as an anomaly target.

[0193] S5, Exception Response and Follow-up Handling:

[0194] Triggering anomaly response: Once an abnormal behavior or trajectory is detected, the system will immediately trigger an anomaly response mechanism, which may include automatic alarm, notification of monitoring personnel, initiation of more detailed trajectory analysis, or initiation of real-time interception procedures.

[0195] Anomaly Analysis and Recording: Detailed analysis of detected abnormal behaviors is performed, and detailed information about abnormal events is recorded, including time, location, anomaly type, and anomaly score. This information can be used for subsequent investigations and model improvement.

[0196] S6, continuous model optimization:

[0197] Model retraining: The system needs to retrain the model periodically or after detecting a sufficient number of new anomalous behaviors to adapt to changes in the environment and the evolution of the target behavior patterns.

[0198] Feature set update: Based on newly collected data and analysis results, it may be necessary to update the key feature set, including adding, deleting or adjusting features, to ensure that the model can accurately capture the latest anomalous patterns.

[0199] Through the above process, the anomaly detection model can play a key role in the monitoring of maritime and air targets, effectively identify potential security threats, and provide real-time intelligent support for maritime safety, airspace management, environmental protection and defense.

[0200] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0201] This embodiment also provides an anomaly detection device for monitoring data, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0202] Figure 6 This is a structural block diagram of anomaly detection of monitoring data according to an embodiment of this application, such as... Figure 6 As shown, the device includes:

[0203] A first memory 62, a first processor 64, and a first computer program 6402 stored in the first memory 64 and executable on the first processor 64, wherein the first processor executes the first computer program to perform the following operations:

[0204] Obtain the monitoring data set of the target object;

[0205] Perform data processing operations on the monitoring data set to obtain the target data set;

[0206] Extract key features from the target dataset to obtain a target feature set;

[0207] Anomaly detection is performed on the target dataset and the target feature set using a pre-trained target network model to obtain a target detection result. The target network model is a model trained on an initial network model using a set of sample datasets and a corresponding set of sample features. The initial network model includes a first detection model and a second detection model. For each set of sample datasets and the sample feature set, the first detection model outputs a first detection result based on the input sample datasets and the sample feature set, and the second detection model outputs a sample detection result based on the input sample datasets, the sample feature set, and the first detection result.

[0208] In an exemplary embodiment, before performing an anomaly detection operation on the target dataset and the target feature set using a pre-trained target network model to obtain a target detection result, the first processor is further configured to: acquire multiple sets of sample monitoring data sets of the sample object; perform the data processing operation on the multiple sets of sample monitoring data sets to obtain multiple sets of sample data sets; extract key features from the sample data included in the multiple sets of sample data sets to obtain multiple sets of sample feature sets; and train the initial network model based on the multiple sets of sample data sets and the multiple sets of sample feature sets until the loss function corresponding to the initial network model satisfies a preset loss condition.

[0209] In an exemplary embodiment, the first processor is configured to acquire multiple sets of sample monitoring data of a sample object by: constructing a data transmission network between N determined data sources, wherein the data acquisition devices in the N data sources are all connected to the sample object, and N is a natural number greater than or equal to 1; and using the data transmission network to acquire object information and running trajectory information of the sample object from the N data sources to obtain multiple sets of sample monitoring data.

[0210] In an exemplary embodiment, the first processor is configured to perform the data processing operation on multiple sets of sample monitoring data sets to obtain multiple sets of sample data sets by: performing data unification operations on the multiple sets of sample monitoring data sets respectively to obtain multiple sets of standard data sets, wherein the data unification operation includes unifying the timestamps of data from N data sources included in the sample monitoring data sets, and unifying the coordinate positions of data from the N data sources, where N is a natural number greater than or equal to 1; and performing data cleaning operations on the data in the multiple sets of standard data sets to obtain multiple sets of sample data sets.

[0211] In an exemplary embodiment, after performing the data processing operation on multiple sets of sample monitoring data to obtain multiple sets of sample data, the processor is further configured to: classify the multiple sample data according to the data type of the multiple sample data included in each set of sample data to obtain M types of data, wherein M is a natural number greater than or equal to 1; store the M types of data into K databases respectively, wherein K is a natural number greater than or equal to 1; integrate the K databases into a preset storage architecture to obtain a distributed database; and perform a setting operation on the distributed database using preset data attributes to obtain a target distributed database, wherein the setting operation includes at least one of the following: data partitioning operation, data sharding operation, and data indexing operation, wherein the data in the target distributed database is stored hierarchically according to the data source.

[0212] In an exemplary embodiment, the first processor is configured to extract key features from sample data included in multiple sets of sample data sets in the following manner to obtain multiple sets of sample feature sets: extracting key features from multiple sample data according to the state features of multiple sample data included in multiple sets of sample data sets to obtain multiple sets of key feature sets, wherein the state features include static features and dynamic features; and using a preset machine learning algorithm to select features that meet a preset weight threshold from multiple key features included in multiple sets of key feature sets to obtain multiple sets of sample feature sets.

[0213] In an exemplary embodiment, after extracting key features from sample data included in multiple sets of sample data to obtain multiple sets of sample features, the processor is further configured to: perform version control operations on multiple key features included in multiple sets of target feature sets using a version control tool to determine the version feature of each key feature, wherein the version feature is used to identify feature changes of the key feature.

[0214] In an exemplary embodiment, the first processor is configured to train the initial network model based on multiple sets of sample data and multiple sets of sample feature sets in the following manner until the loss function corresponding to the initial network model satisfies a preset loss condition: The initial network model is trained for the kth time through the following steps, where k is a positive integer: The kth set of training data used for the kth training of the initial network model is obtained from the training data of the multiple sets of sample data and multiple sets of sample feature sets; The sample data set and sample feature set in the kth set of training data are input into the first detection model trained for the (k-1)th time to obtain the kth first detection result, wherein the first detection model is configured to compare the sample data set and the sample feature set with preset anomaly rules and preset anomaly events, respectively, to obtain first detection features and second detection features, and to determine the kth first detection result by combining the first detection features and the second detection features; The sample data set, sample feature set, and the kth first detection result in the kth set of training data are input into the second detection model trained for the (k-1)th time to obtain the kth sample detection result.

[0215] In an exemplary embodiment, the second detection model determines the k-th sample detection result by: calculating the feature value of each sample feature in the sample feature set using a probability distribution model; using the feature value of each sample feature to determine abnormal sample features and normal sample features that deviate from a preset numerical range in the sample feature set; using a clustering algorithm to perform a clustering operation on the normal sample features to determine the normal behavior cluster of the normal sample features; using the normal behavior cluster and the historical behavior data of the sample object to establish a time series model of the normal behavior data of the sample object; and determining the k-th sample detection result based on the time series model, the abnormal sample features, the sample data set, and the k-th first detection result.

[0216] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0217] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0218] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0219] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0220] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0221] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0222] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0223] The embodiments described herein also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0224] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0225] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0226] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for detecting anomalies in monitoring data, characterized in that, include: Obtain the monitoring data set of the target object; Perform data processing operations on the monitoring data set to obtain the target data set; Extract key features from the target dataset to obtain a target feature set; Anomaly detection is performed on the target dataset and the target feature set using a pre-trained target network model to obtain a target detection result. The target network model is a model trained on an initial network model using a set of sample datasets and a corresponding set of sample features. The initial network model includes a first detection model and a second detection model. For each set of sample datasets and the sample feature set, the first detection model outputs a first detection result based on the input sample datasets and the sample feature set, and the second detection model outputs a sample detection result based on the input sample datasets, the sample feature set, and the first detection result.

2. The method according to claim 1, characterized in that, Before performing anomaly detection on the target dataset and the target feature set using a pre-trained target network model to obtain the target detection result, the method further includes: Obtain a collection of multiple sets of sample monitoring data for the sample object; The data processing operation is performed on multiple sets of the sample monitoring data to obtain multiple sets of the sample data. Key features are extracted from the sample data included in the multiple sets of sample data to obtain multiple sets of sample feature sets; The initial network model is trained based on multiple sets of sample data and multiple sets of sample feature sets until the loss function corresponding to the initial network model satisfies a preset loss condition.

3. The method according to claim 2, characterized in that, Obtain multiple sets of sample monitoring data for the sample object, including: Construct a data transmission network among N determined data sources, wherein the data acquisition devices in the N data sources are all connected to the sample object, and N is a natural number greater than or equal to 1; The object information and trajectory information of the sample object are obtained from N data sources using the data transmission network to obtain multiple sets of sample monitoring data.

4. The method according to claim 2, characterized in that, The data processing operation is performed on multiple sets of the sample monitoring data to obtain multiple sets of the sample data, including: Data unification operations are performed on multiple sets of sample monitoring data to obtain multiple sets of standard data sets. The data unification operations include unifying the timestamps of the data from N data sources included in the sample monitoring data sets and unifying the coordinate positions of the data from the N data sources, where N is a natural number greater than or equal to 1. Data cleaning operations are performed on the data in multiple sets of standard datasets to obtain multiple sets of sample datasets.

5. The method according to claim 2, characterized in that, After performing the data processing operation on multiple sets of the sample monitoring data to obtain multiple sets of the sample data, the method further includes: For each set of sample data, the sample data is classified according to the data type of the multiple sample data included in the set to obtain M types of data, where M is a natural number greater than or equal to 1. The data of type M are stored in K databases respectively, where K is a natural number greater than or equal to 1; By integrating K of the aforementioned databases into a preset storage architecture, a distributed database is obtained; By performing setting operations on the distributed database using preset data attributes, a target distributed database is obtained. The setting operations include at least one of the following: data partitioning operation, data sharding operation, and data indexing operation. The data in the target distributed database is stored in layers according to the data source.

6. The method according to claim 2, characterized in that, Key features are extracted from the sample data included in the multiple sets of sample data to obtain multiple sets of sample feature sets, including: Based on the state characteristics of multiple sample data included in the multiple sets of sample data, key features are extracted from the multiple sample data to obtain multiple sets of key features, wherein the state characteristics include static features and dynamic features; A preset machine learning algorithm is used to select features that meet a preset weight threshold from multiple key features included in multiple sets of key features, thereby obtaining multiple sets of sample features.

7. The method according to claim 2, characterized in that, After extracting key features from the sample data included in the multiple sets of sample data to obtain multiple sets of sample feature sets, the method further includes: Version control operations are performed on multiple key features included in multiple sets of target features using a version control tool to determine the version feature of each key feature, wherein the version feature is used to identify feature changes of the key feature.

8. The method according to claim 2, characterized in that, The initial network model is trained based on multiple sets of sample data and multiple sets of sample feature sets until the loss function corresponding to the initial network model satisfies a preset loss condition, including: The initial network model is trained for the kth time using the following steps, where k is a positive integer: Obtain the k-th training data used for the k-th training of the initial network model from the multiple sets of sample data sets and the multiple sets of sample feature sets of training data; The sample data set and the sample feature set in the kth training data are input into the first detection model trained for the (k-1)th time to obtain the kth first detection result. The first detection model is used to compare the sample data set and the sample feature set with preset anomaly rules and preset anomaly events, respectively, to obtain the first detection feature and the second detection feature, and to determine the kth first detection result by combining the first detection feature and the second detection feature. The sample data set, the sample feature set, and the kth first detection result from the kth training data are input into the second detection model trained in the (k-1)th iteration to obtain the kth sample detection result.

9. The method according to claim 8, characterized in that, The second detection model determines the detection result of the kth sample in the following way: The feature value of each sample feature in the sample feature set is calculated using a probability distribution model; The abnormal and normal sample features that deviate from the preset numerical range in the sample feature set are determined by using the feature value of each sample feature; Clustering algorithms are used to cluster the features of the normal samples to determine the normal behavior clusters of the features of the normal samples. A time series model of the normal behavior data of the sample objects is established using the normal behavior clustering and the historical behavior data of the sample objects. Based on the time series model, the abnormal sample features, the sample data set, and the kth first detection result, the kth sample detection result is determined.

10. An anomaly detection device for monitoring data, characterized in that, It includes a first memory, a first processor, and a first computer program stored in the first memory and executable on the first processor. When the first processor executes the first computer program, it performs the following operations: Obtain the monitoring data set of the target object; Perform data processing operations on the monitoring data set to obtain the target data set; Extract key features from the target dataset to obtain a target feature set; Anomaly detection is performed on the target dataset and the target feature set using a pre-trained target network model to obtain a target detection result. The target network model is a model trained on an initial network model using a set of sample datasets and a corresponding set of sample features. The initial network model includes a first detection model and a second detection model. For each set of sample datasets and the sample feature set, the first detection model outputs a first detection result based on the input sample datasets and the sample feature set, and the second detection model outputs a sample detection result based on the input sample datasets, the sample feature set, and the first detection result.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 9.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 9.

Images