A multi-model anomaly detection method based on multi-modal time series data

Abstract

The embodiment of the application discloses a multi-model anomaly detection method based on multi-modal time series data. The method comprises the following steps: obtaining multi-modal time series data in the operation of a device, and preprocessing the multi-modal time series data to obtain target multi-modal time series data; fusing the target multi-modal time series data to obtain multi-modal fusion features; obtaining a generative adversarial network generated by training positive sample time series data in advance, and a classification network generated by training positive sample time series data and negative sample time series data in advance; inputting the multi-modal fusion features into the generative adversarial network to obtain a first anomaly detection result of the operation of the device; inputting the multi-modal fusion features into the classification network to obtain a second anomaly detection result of the operation of the device; and determining a target anomaly detection result according to the first anomaly detection result and the second anomaly detection result. The anomaly detection based on multi-modal time series data and combined with multi-model can improve the detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multi-model anomaly detection method based on multimodal time-series data. Background Technology

[0002] In modern industrial production, such as high-precision industries like automobile manufacturing, aerospace, and energy equipment, the generation and processing of time series data has become a crucial step.

[0003] Time series data can be used to monitor equipment operating status in real time, predict potential failures, optimize processes, and improve product quality. For example, in the automotive industry, analyzing time series data can promptly detect abnormalities in the processing and equipment, thereby preventing production line shutdowns or product quality degradation caused by equipment failures.

[0004] However, with the increasing complexity of industrial scenarios, traditional detection methods are no longer sufficient to meet the demands for efficiency and accuracy. Relying solely on data from a single modality cannot fully describe complex industrial scenarios. Using fixed thresholds to determine anomalies also lacks dynamic adaptability and is prone to false alarms or missed alarms due to environmental changes. Summary of the Invention

[0005] This invention provides a multi-model anomaly detection method based on multimodal time series data to improve the accuracy of equipment operation anomaly detection.

[0006] According to one aspect of the present invention, a multi-model anomaly detection method based on multimodal time series data is provided, the method comprising:

[0007] Acquire multimodal time-series data during device operation, and preprocess the multimodal time-series data to obtain target multimodal time-series data;

[0008] The target multimodal time-series data are fused to obtain multimodal fusion features;

[0009] Obtain a generative adversarial network pre-trained with positive sample time series data, and a classification network pre-trained with both positive and negative sample time series data;

[0010] The multimodal fusion features are input into the generative adversarial network to obtain the first anomaly detection result of device operation;

[0011] The multimodal fusion features are input into the classification network to obtain the second anomaly detection result of device operation;

[0012] Based on the first anomaly detection result and the second anomaly detection result, the target anomaly detection result is determined.

[0013] According to another aspect of the present invention, a multi-model anomaly detection device based on multimodal time series data is provided, the device comprising:

[0014] The data preprocessing module is used to acquire multimodal time-series data during device operation and preprocess the multimodal time-series data to obtain target multimodal time-series data.

[0015] The data fusion module is used to fuse the target multimodal time-series data to obtain multimodal fusion features;

[0016] The network acquisition module is used to acquire the generative adversarial network pre-trained with positive sample time series data, and the classification network pre-trained with positive sample time series data and negative sample time series data;

[0017] The first anomaly detection result determination module is used to input the multimodal fusion features into the generative adversarial network to obtain the first anomaly detection result of device operation;

[0018] The second anomaly detection result determination module is used to input the multimodal fusion features into the classification network to obtain the second anomaly detection result of the device operation;

[0019] The target anomaly detection result determination module is used to determine the target anomaly detection result based on the first anomaly detection result and the second anomaly detection result.

[0020] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0021] At least one processor; and

[0022] A memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the multi-model anomaly detection method based on multimodal time-series data according to any embodiment of the present invention.

[0024] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the multi-model anomaly detection method based on multimodal time-series data as described in any embodiment of the present invention.

[0025] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the multi-model anomaly detection method based on multimodal time-series data as described in any embodiment of the present invention.

[0026] The technical solution of this invention acquires multimodal time-series data during device operation and preprocesses the multimodal time-series data to obtain target multimodal time-series data; fuses the target multimodal time-series data to obtain multimodal fusion features; acquires a generative adversarial network (GAN) pre-trained with positive sample time-series data and a classification network pre-trained with both positive and negative sample time-series data; inputs the multimodal fusion features into the GAN to obtain a first anomaly detection result for device operation; inputs the multimodal fusion features into the classification network to obtain a second anomaly detection result for device operation; and determines the target anomaly detection result based on the first and second anomaly detection results. This solves the problems of missed and false detections in device operation anomaly detection, and improves detection accuracy by combining multimodal time-series data with multiple models for anomaly detection.

[0027] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart of a multi-model anomaly detection method based on multimodal time series data according to Embodiment 1 of the present invention;

[0030] Figure 2 This is a schematic diagram of timing alignment of multimodal timing data according to Embodiment 1 of the present invention;

[0031] Figure 3 This is a flowchart of an abnormal timing location determination method according to Embodiment 1 of the present invention;

[0032] Figure 4 This is a flowchart of a multi-model anomaly detection method based on multimodal time series data provided in Embodiment 2 of the present invention.

[0033] Figure 5This is a schematic diagram of the generation of a generative adversarial network according to Embodiment 2 of the present invention.

[0034] Figure 6 This is a schematic diagram of a generative adversarial network for anomaly detection according to Embodiment 2 of the present invention.

[0035] Figure 7 This is a flowchart of another multi-model anomaly detection method based on multimodal time series data provided in Embodiment 2 of the present invention.

[0036] Figure 8 This is a schematic diagram of a multi-model anomaly detection device based on multimodal time series data according to Embodiment 3 of the present invention.

[0037] Figure 9 This is a schematic diagram of the structure of an electronic device that implements the multi-model anomaly detection method based on multimodal time-series data according to an embodiment of the present invention. Detailed Implementation

[0038] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0039] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0040] Example 1

[0041] Figure 1This is a flowchart of a multi-model anomaly detection method based on multimodal time-series data according to Embodiment 1 of the present invention. This embodiment is applicable to the detection of anomalies in equipment operation during automotive industrial production. The method can be executed by a multi-model anomaly detection device based on multimodal time-series data. This device can be implemented in hardware and / or software and can be configured in electronic devices such as computers, servers, visualization devices, or controllers. Figure 1 As shown, the method includes:

[0042] Step 110: Acquire multimodal time series data during equipment operation and preprocess the multimodal time series data to obtain target multimodal time series data.

[0043] Multimodal time-series data can be time-related data such as production process indicators and equipment operating parameters during equipment operation. For example, multimodal time-series data may include: vibration, temperature, and pressure indicators collected by sensors such as vibration sensors, temperature sensors, and pressure sensors; images acquired by image acquisition devices such as infrared thermal imagers; equipment operation logs; and equipment operating parameters such as current and voltage. Comprehensive analysis of multimodal time-series data facilitates the timely detection of equipment operation anomalies.

[0044] Preprocessing of multimodal time series data can include one or more of the following: dimensionless processing, temporal alignment, noise removal, and outlier removal. Dimensionless processing can involve normalizing the time series data for each modality. For example, normalization using the maximum and minimum values ​​of the current modality's time series data can eliminate the influence of dimensions and improve data consistency.

[0045] During production, equipment operation may be affected by various factors, and even if the time series data are collected at the same frequency, the final sequence lengths may not be completely consistent. Therefore, time series alignment can be performed on multimodal time series data to facilitate data analysis.

[0046] Optionally, the multimodal time series data is preprocessed to obtain the target multimodal time series data, including: dividing the multimodal time series data into multiple windows according to a preset window length; extracting the window features of the multimodal time series data in each window; and determining the target multimodal time series data based on the window features.

[0047] For example, the time series data for each modality can be divided into fixed-length windows, and window features of the multimodal time series data can be extracted within each window to complete the time series alignment operation. Window features can be the mean, variance, standard deviation, or extreme values ​​of the time series data. The window features of each modality's time series data within each window collectively constitute the target multimodal time series data.

[0048] When performing time-series alignment of multimodal time-series data, the preset window length needs to be appropriate. The preset window length can be determined empirically or based on the length of the time-series data. Too few windows, determined by the preset window length, may lead to information loss or overly coarse features, thus affecting the accuracy and robustness of subsequent analysis. Too many windows may result in large data processing volumes and difficulties in time-series alignment. The selection of the preset window length can be optimized based on the actual application scenario and data characteristics to balance computational efficiency and feature richness, ensuring that the time-series aligned data can effectively support multimodal fusion and anomaly detection tasks.

[0049] In this embodiment of the invention, a dynamic window length can be introduced, and dynamic time series alignment can be performed based on the dynamic window length to achieve a balance between data analysis accuracy and the amount of data analyzed. Optionally, before dividing the multimodal time series data into multiple windows according to the preset window length, the method further includes: calculating the autocorrelation coefficient of the time series data for each modality; and determining the preset window length for time series alignment of the multimodal time series data based on the autocorrelation coefficient of the time series data for each modality and the preset weight of the time series data for each modality.

[0050] The autocorrelation coefficient of time-series data can be determined by calculating the similarity between the time-series data. For example, the autocorrelation coefficient can be calculated during the initial window partitioning. If the autocorrelation coefficient is greater than a first preset similarity threshold, the window length can be increased; if the autocorrelation coefficient is less than a second preset similarity threshold, the window length can be decreased. The first preset similarity threshold can be greater than or equal to the second preset similarity threshold. Considering the combined influence of time-series data from multiple modalities within the initial window, preset weights for each modality can be considered when determining the preset window length. By weighted summing the preset weights of each modality's time-series data with the autocorrelation coefficient, the adjustment method for the preset window length is determined, thereby determining the preset window length for time-series alignment of multimodal time-series data. By dynamically adjusting the preset window length, appropriate time-series alignment of multimodal time-series data can be performed, ensuring the integrity of the data information.

[0051] Figure 2 This is a schematic diagram of timing alignment for multimodal timing data provided in Embodiment 1 of the present invention. Figure 2As shown, for time series data such as pressure, load, and voltage, multiple sliding windows can be set, and window characteristics can be determined in each window to obtain the target multimodal time series data.

[0052] Step 120: Fuse the target multimodal time series data to obtain multimodal fusion features.

[0053] like Figure 2 As shown, target multimodal time-series data can be fused to obtain multimodal fusion features. Multimodal time-series data fusion integrates information from different data sources or modalities into a unified representation. Fusion fully leverages the complementarity between modalities to enhance the model's understanding of complex scenarios. In industrial time-series data processing, multimodal fusion can combine time-series data collected by sensors and equipment operation data to form a more comprehensive data feature representation. There are various fusion methods. For example, early fusion (directly splicing target multimodal time-series data), late fusion (weighted combination of target multimodal time-series data), or hybrid fusion (layered, stepwise fusion) can be used to achieve multimodal time-series data fusion. For instance, a cross-modal attention mechanism can be used to weightedly fuse multimodal features, dynamically adjusting the importance of each modality to more accurately detect anomalies in equipment operation.

[0054] Step 130: Obtain the generative adversarial network pre-trained with positive sample time series data, and the classification network pre-trained with positive sample time series data and negative sample time series data.

[0055] The positive and negative time-series data can be pre-labeled to ensure high-quality training data. After labeling, the multimodal fusion dataset is divided into training and testing datasets, typically in a 7:3 or 8:2 ratio, to ensure the model learns effective features on sufficient training data. A suitable deep learning model, such as a Long Short-Term Memory (LSTM) network or a Transformer, is used as the classifier. LSTM excels at capturing long-term dependencies in time series data, while the Transformer better models global contextual information through its self-attention mechanism. The classifier is trained using the training data. The optimization objective can be to maximize classification accuracy while minimizing the loss function (such as cross-entropy loss). Furthermore, to prevent overfitting, regularization techniques (such as Dropout) or data augmentation methods can be used to improve the model's generalization ability, ultimately resulting in a classification network.

[0056] When generating a classification network, the classifier's performance can be comprehensively evaluated using a test dataset. Evaluation metrics include, but are not limited to, accuracy, recall, F1 score, and area under the curve (AUC). These metrics reflect the model's performance from different perspectives. For example, the F1 score considers both precision and recall, making it suitable for handling imbalanced datasets; while AUC measures the model's classification ability at different thresholds, particularly suitable for anomaly detection tasks. During evaluation, the focus is on the model's generalization ability, i.e., its performance on unseen data. If the model performs poorly on the test dataset, it can be optimized by adjusting hyperparameters (such as learning rate and batch size), increasing the amount of training data, or improving the model structure. Furthermore, cross-validation techniques can be introduced to further improve the model's stability and reliability. Ultimately, this ensures that the classification network achieves high accuracy while possessing good robustness, enabling it to adapt to detection needs in different scenarios.

[0057] When training a Generative Adversarial Network (GAN), the generator learns from positive sample time-series data, while the discriminator determines the authenticity of the data generated by the generator. The goal is for the generator to produce more realistic fake data to deceive the discriminator, and for the discriminator to improve its ability to distinguish between real and fake data. This process leads to the GAN. Therefore, both GANs and classification networks can be used to detect anomalies in device operation.

[0058] Step 140: Input the multimodal fusion features into the generative adversarial network to obtain the first anomaly detection result of the device operation.

[0059] Step 150: Input the multimodal fusion features into the classification network to obtain the second anomaly detection result of the device operation.

[0060] Step 160: Determine the target anomaly detection result based on the first anomaly detection result and the second anomaly detection result.

[0061] By combining the detection results of generative adversarial networks (GANs) and classification networks, the accuracy of anomaly detection can be improved. For example, if both the classification network and the generator network determine a segment of time-series data as abnormal, it is marked as an anomaly; otherwise, it is considered normal equipment operation. This multi-model approach not only improves the accuracy of detection results but also enhances the system's robustness, effectively addressing anomaly detection needs in complex industrial environments.

[0062] When equipment malfunctions are identified, further analysis can be performed to pinpoint the anomaly's temporal location. For example, an isolated forest method can be used to construct a randomly segmented tree structure for rapid anomaly identification, as anomalies are more easily isolated into shallower nodes of the tree, thus achieving efficient detection. Alternatively, anomalies can be determined by the local density deviation between temporal data and its neighborhood. Or, deep learning-based local anomaly detection methods can be used to identify anomalies. Determining the anomaly's temporal location facilitates equipment maintenance based on data from that location.

[0063] Optionally, after determining the target anomaly detection result based on the first anomaly detection result and the second anomaly detection result, the method further includes: when the target anomaly detection result is a device malfunction, extracting local features from the anomaly time series data corresponding to the device malfunction through a sliding window; determining the anomaly score of each sliding window based on the local features and the density of the area where the local features are located; and determining the anomaly time series location of the device malfunction based on the anomaly score and a preset neighborhood value.

[0064] Figure 3 This is a flowchart illustrating the determination of an abnormal timing location according to Embodiment 1 of the present invention. Figure 3 As shown, for multimodal fusion features, the local features of each feature point in each feature sequence can be calculated to generate a local feature distribution diagram. In the distribution diagram, the density at normal data points is higher than the density at abnormal data points.

[0065] For example, local features can be the mean, standard value, variance, or extreme values ​​of anomalous time-series data within a sliding window. The distribution of multimodal fusion features within each sliding window is determined. Based on this distribution, the density of the region containing the local features of the anomalous time-series data can be determined. Based on the local features and the density of their regions, anomaly scores for each sliding window can be obtained. For example, density and anomaly scores may be positively or negatively correlated. For instance, lower density generally indicates a higher probability of anomaly. Preset neighborhood values ​​can be set based on the anomaly scores of normal data. For example, preset neighborhood values ​​can be determined based on each anomaly score and quantile. For instance, preset neighborhood values ​​can be used to divide normal and anomalous data, with normal data accounting for 95%-99%. Sliding windows with anomaly scores below this preset neighborhood value can then be identified as anomalous time-series locations. Dynamically setting preset neighborhood values ​​can effectively address distributional anomalies in different scenarios, improving the flexibility and accuracy of anomalous time-series location detection, and thus facilitating equipment fault repair based on time-series data at specific anomalous time-series locations.

[0066] Optionally, the anomaly score of each sliding window is determined based on the local features and the density of the region where the local features are located, including: determining the k-neighborhood parameter of the local feature based on the density of the region where the local feature is located; determining the reachability distance between local features in the k-neighborhood based on the timestamp of each local feature in the anomaly time series data, the distance between the local feature and the k-neighborhood, and the distance between local features in the k-neighborhood; determining the local reachability density of the region based on the reachability distance between local features in the k-neighborhood and the k-neighborhood parameter, and using the local reachability density as the anomaly score corresponding to the local feature.

[0067] The following section elaborates on determining the abnormal timing location of equipment operation through local features and the density of the region where the local features are located. First, define the neighborhood: For each point in the region where the local feature is located... Find its k-nearest neighbor (k-NN). k is an important hyperparameter that determines the range of "local". Since data density varies in different regions, this invention uses k that adaptively varies with local density. Specifically, In the formula, This is a threshold (e.g., 0.1), representing the point of sudden change in distance growth; Point In the k-neighborhood, the farthest feature and point The distance; Point In the (k+1)-neighborhood, the farthest feature and point The distance.

[0068] Secondly, reachability distance is calculated: To smooth out the effects of noise, reachability distance is used in this invention instead of simple Euclidean distance. For time-series data, a time decay factor is introduced to give higher weight to recent neighbors. Time The reachable distance is defined as: ,in, It is the Euclidean distance between two points. It is a point The distance to its Kth nearest neighbor. Controlling the rate of time decay, Point timestamp, Point Timestamp. (via point) Time The reachability distance ensures that for In densely populated areas, the distance between points will not be excessively magnified.

[0069] Next, the Local Reachability Density (LRD) is calculated. (Point) The density within the neighborhood is In the formula, yes The set of k-nearest neighbors. yes The number of samples contained in the k-nearest neighbor set, i.e., k. The denominator is... The average reachable distance to all its neighbors. The smaller this distance, the closer the neighbors are, the higher the density of the area, and therefore the larger the LRD value.

[0070] Then, the Local Outlier Factor (LOF), or anomaly score, is calculated. This is done by comparing points. Anomaly scores are obtained by comparing the density of a given element with the average density of its neighbors. .

[0071] if Explanation points The local density of a point is roughly the same as the density of its neighbors. A point that is in the same dense area as its surrounding points is a normal point. If... (Theoretically possible but rare) Explanation The density is higher than that of the neighbors, point It could be the core point of a dense cluster, which is perfectly normal. If Explanation points The local density is significantly lower than that of its neighbors. It is located in a relatively sparse region, surrounded by a denser region, therefore the point It is an anomaly and also the basis for this invention to determine the location of the anomaly sequence.

[0072] Finally, the location of the abnormal timing sequence is determined based on the anomaly score and the preset neighborhood value. Setting the preset neighborhood value: A threshold is set for the LOF score. Windows with scores exceeding this preset neighborhood value are considered abnormal windows. The preset neighborhood value can be determined through business experience, performance on a validation set (if labeled), or by using extreme value statistics (such as the 3-sigma principle, quantiles). Mapping back to the original timing sequence to determine the location of the abnormal timing sequence in device operation: Each sliding window Wi corresponds to a segment in the original timing sequence. In this invention, the center point of this window or the entire window is marked as abnormal. If a point is covered by multiple consecutive abnormal windows, its anomalousness is higher.

[0073] The technical solution of this embodiment acquires multimodal time-series data during device operation and preprocesses the multimodal time-series data to obtain target multimodal time-series data; fuses the target multimodal time-series data to obtain multimodal fusion features; acquires a generative adversarial network pre-trained with positive sample time-series data and a classification network pre-trained with both positive and negative sample time-series data; inputs the multimodal fusion features into the generative adversarial network to obtain a first anomaly detection result for device operation; inputs the multimodal fusion features into the classification network to obtain a second anomaly detection result for device operation; and determines the target anomaly detection result based on the first and second anomaly detection results. This solves the problems of missed and false detections in device operation anomaly detection, and improves detection accuracy by combining multimodal time-series data with multiple models for anomaly detection.

[0074] Example 2

[0075] Figure 4 This is a flowchart of a multi-model anomaly detection method based on multimodal time series data according to Embodiment 2 of the present invention. This embodiment is a further refinement of the above technical solution, and the technical solution in this embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 4 As shown, the method includes:

[0076] Step 410: Obtain multimodal time series data during device operation, and divide the multimodal time series data into multiple windows according to the preset window length.

[0077] Optionally, before dividing the multimodal time series data into multiple windows according to the preset window length, the method further includes: calculating the autocorrelation coefficient of the time series data of each modality; and determining the preset window length for time series alignment of the multimodal time series data based on the autocorrelation coefficient of the time series data of each modality and the preset weight of the time series data of each modality.

[0078] Step 420: Extract window features of multimodal time series data from each window, and determine the target multimodal time series data based on the window features.

[0079] Step 430: Fuse the target multimodal time series data to obtain multimodal fusion features.

[0080] Optionally, the target multimodal time series data are fused to obtain multimodal fusion features, including: learning the correlation between different modal time series data through a cross-modal attention mechanism, and adjusting the fusion weight of each modal time series data according to the correlation; when the fusion weight evaluation is passed, the target multimodal time series data are fused according to the fusion weight to obtain multimodal fusion features.

[0081] In multimodal time-series anomaly detection, sensor parameters such as slide pressure, drive load, drive torque, speed, vibration data, and temperature do not change in isolation, but rather represent closely coupled and mutually influential physical processes. For example, an abnormal increase in drive load or torque is often accompanied by a decrease in speed and an increase in temperature; a sudden change in slide pressure may trigger increased mechanical vibration; and sustained high temperatures may reduce system efficiency, further altering the balance between load and torque. These modes exhibit dynamic and nonlinear physical correlations and causal dependencies, collectively characterizing the health status of the equipment. Therefore, it is necessary to jointly model their collaborative evolution patterns to effectively identify deep-seated abnormal behaviors. This invention assigns appropriate weights to each modality's data through a cross-modal attention mechanism.

[0082] For example, in industrial production, vibration signals may reflect abnormal equipment conditions better than temperature signals, and cross-modal attention mechanisms can automatically assign higher weights to vibration signals. Specifically, this can be achieved by calculating similarity scores between modes or by using self-attention mechanisms to capture global contextual information, thereby generating fused features. Using cross-modal attention mechanisms to fuse target multimodal time-series data not only improves the model's sensitivity to key modes but also enhances its adaptability to complex scenarios, making it particularly suitable for situations where multimodal data distribution is uneven or where there are differences in signal strength.

[0083] The feature fusion using a multimodal attention mechanism can be represented as: .in, This represents the normalized fusion weights corresponding to the time series data of mode k. This represents the feature representation corresponding to the time-series data of mode k. A is the multimodal fusion feature after mode fusion.

[0084] When fusing features using a multimodal attention mechanism, it can be jointly trained with classification networks and generative adversarial networks. The test sample set can determine the detection results of device malfunctions. Based on these results, the fusion weights can be evaluated. If the evaluation is successful, the target multimodal temporal data can be fused according to the fusion weights. If the evaluation fails, the parameters of the multimodal attention mechanism can be adjusted, specifically the fusion weights of each modality's temporal data, to improve the accuracy of device malfunction detection.

[0085] Step 440: Obtain the generative adversarial network pre-trained with positive sample time series data, and the classification network pre-trained with positive sample time series data and negative sample time series data.

[0086] Optionally, obtaining a generative adversarial network pre-trained with positive sample time-series data includes: inputting a random noise vector into a generator and mapping the random noise vector to a high-dimensional data space through a multi-layer neural network to obtain pseudo-time-series data; inputting the positive sample time-series data and pseudo-time-series data into a discriminator to obtain the discrimination result output by the discriminator; evaluating the discrimination result and pseudo-time-series data through a loss function, and iteratively training based on the evaluation result to obtain the generative adversarial network.

[0087] Figure 5 This is a schematic diagram illustrating the generation of a generative adversarial network according to Embodiment 2 of the present invention. Figure 5 As shown, the generator is one of the core components of a generative adversarial network (GAN). The generator learns the distribution of positive sample time-series data and generates pseudo-time-series data that resembles the positive sample time-series data. The generator's input is a random noise vector (represented in the latent space), which is mapped to a high-dimensional data space through a multi-layer neural network (such as fully connected layers or convolutional layers), outputting the generated pseudo-time-series data. Meanwhile, the discriminator, another core component of the GAN, distinguishes whether the input data comes from real positive sample time-series data or pseudo-time-series data generated by the generator. The discriminator's input is either positive sample time-series data or pseudo-time-series data generated by the generator, and its output is a probability value (e.g., a value between 0 and 1) indicating whether the input data is real positive sample time-series data or pseudo-time-series data.

[0088] The generator and discriminator are jointly optimized through adversarial training. The generator attempts to generate more realistic pseudo-data to deceive the discriminator, while the discriminator strives to improve its ability to distinguish between real and pseudo-data. During training, the generator and discriminator optimize each other through adversarial training. Specifically, the discriminator aims to maximize its ability to distinguish between real and pseudo-data, that is, to identify as accurately as possible which data is real and which data is generated by the generator; while the generator aims to minimize the discriminator's ability to identify its generated data, that is, to generate pseudo-data that more closely approximates the distribution of real data to deceive the discriminator. To measure the performance of both, loss functions (such as binary cross-entropy loss) can be used to evaluate the performance of the discriminator and the generator respectively. In the early stages of training, the data generated by the generator may differ significantly from real data, and the discriminator can easily distinguish between the two; however, as training progresses, the generator gradually learns the distribution characteristics of normal data, and the generated pseudo-data becomes increasingly realistic, eventually reaching a point where the discriminator can no longer effectively distinguish between real and pseudo-data. At this point, the generator has successfully learned the distribution of normal data, laying the foundation for subsequent anomaly detection.

[0089] After training the generative adversarial network (GAN), the generator is used to detect anomalies in test samples. For each test sample, the closest generated sample (pseudo-time series data) is found using a latent space search technique. The latent space search process can be understood as finding an optimal noise vector in the generator's latent space that minimizes the difference between the generated pseudo-data and the test sample. This difference can be quantified by calculating the reconstruction error between the test sample and the generated sample. Reconstruction error metrics can include mean squared error (MSE), L1 distance, or dynamic time warping (DTW). A smaller reconstruction error indicates that the test sample is closer to the normal data distribution; conversely, a larger reconstruction error indicates that the test sample may contain anomalies. An anomaly detection threshold is set based on the reconstruction error distribution of normal data. Specifically, during the training phase, the corresponding reconstruction error distribution is calculated using a normal dataset, and an appropriate quantile (such as the 95th or 99th quantile) is selected. The reconstruction error is then divided according to the quantile to determine the reconstruction error threshold. If the reconstruction error of a test sample is lower than this threshold, it is considered normal data; if the reconstruction error is higher than this threshold, it is considered anomalous data. The choice of reconstruction error threshold can be adjusted based on the specific application scenario. For example, in scenarios sensitive to false positive rates, the first quantile can be selected to reduce false positives; while in scenarios sensitive to false negative rates, the second quantile can be selected to improve detection sensitivity. The first quantile is greater than the second quantile. Finally, based on the anomaly score of the test samples and the set threshold, the test samples are divided into normal or abnormal categories. Based on the test results of the test samples, parameter adjustments or retraining of the generative adversarial network (GAN) can be performed to improve the anomaly detection accuracy of the GAN.

[0090] Optionally, obtaining a classification network pre-trained using positive and negative sample time-series data includes: pre-labeling the sample data to generate a sample dataset including positive and negative sample time-series data; dividing the sample dataset into a training dataset and a test dataset according to a preset ratio, and training the classifier using the training dataset to obtain an initial classification network; evaluating the performance of the initial classification network using the test dataset, and obtaining the final classification network when the evaluation passes.

[0091] Optionally, obtaining a classification network pre-trained using positive and negative sample time-series data includes: pre-labeling the sample data to generate a sample dataset including positive and negative sample time-series data; inputting samples from the sample dataset into an initial convolutional network to obtain spatial feature maps for each sample; generating feature vectors based on the spatial feature maps; inputting the feature vectors into the encoder of a self-attention mechanism deep learning model to obtain spatial augmentation features corresponding to the spatial feature maps; inputting the spatial feature maps into a long short-term memory network in chronological order to obtain corresponding temporal augmentation features; fusing the spatial and temporal augmentation features to obtain target spatial and temporal augmentation features corresponding to the samples; and training a classifier based on the target spatial and temporal augmentation features to obtain a classification network.

[0092] The initial convolutional network can be a lightweight CNN (such as a ResNet backbone). The encoder of a Transformer deep learning model utilizes self-attention to allow feature vectors to interact fully, learning global spatial relationships within the entire image. For example, the model can learn that the "head" and "tail" of a scratch are related, even if they are far apart.

[0093] LSTM can capture the continuous state of a device when it malfunctions, such as how the malfunction occurs, expands, and evolves.

[0094] The output features of the Transformer and the output features of the LSTM can be fused. The fusion methods can be varied. For example, feature fusion can be achieved through concatenation or weighted summation. The final target space and temporal augmentation features can be classified through fully connected layers to generate a classification network.

[0095] By combining a self-attention deep learning model with a Long Short-Term Memory (LSTM) network, the strengths of both can be leveraged simultaneously. In practical device anomaly detection, data may contain complex spatial and temporal dependencies. LSTM alone has weak spatial context modeling capabilities and may fail to effectively understand the complex relationship between device anomalies and their surrounding normal regions. While Transformer alone has strong global context modeling capabilities, its positional encoding for ordered sequences (such as continuous data during device anomalies) may not be as natural as the sequence processing mechanism built into LSTM; furthermore, Transformer typically requires a large amount of data and is prone to overfitting. Combining LSTM and Transformer allows both to be fully utilized, improving both spatial and temporal performance to enhance the reliability of the classification network.

[0096] The feature processing of Transformer and LSTM is parallel. They are different "perspectives" on the same input, and they are not mutually exclusive paths, but rather two working modules of the model. They analyze and process the same input data from two dimensions: spatial and temporal. Transformer analyzes global cues within a single image, while LSTM analyzes dynamic patterns in a continuous production process.

[0097] Since LSTM needs to learn the temporal variations associated with device anomalies from the raw features, but temporal variations of static information such as background can cause strong interference, preprocessing of features can be performed before inputting the spatial feature map into the LSTM to highlight the significant changes in data related to device anomalies across frames and suppress irrelevant changes.

[0098] Optionally, the spatial feature maps are input into the Long Short-Term Memory (LSTM) network in chronological order to obtain the corresponding temporal enhancement features. This includes: obtaining the spatial feature maps of adjacent frames corresponding to the current spatial feature map, and determining the multi-scale Gaussian difference between the current spatial feature map and the spatial feature maps of adjacent frames; determining the feature weight values ​​of the current spatial feature map based on the current spatial feature map and the multi-scale Gaussian difference; determining the temporal difference features corresponding to the current spatial feature map based on the feature weight values ​​and the multi-scale Gaussian difference; and inputting the temporal difference features into the LSM network in chronological order to obtain the corresponding temporal enhancement features.

[0099] The current spatial feature map is denoted as The spatial feature map of an adjacent frame can include the spatial feature map of the previous frame. and the spatial feature map of the next frame For each spatial feature map, multi-scale Gaussian differences can be introduced, for example, by using a standard deviation of... A Gaussian filter is used to obtain a set of multi-scale feature maps. .

[0100] Calculate the multi-scale Gaussian difference between the current spatial feature map and the spatial feature maps of adjacent frames. .in, It is an activation function, such as This is used to emphasize "increased" changes in features (more sensitive to device anomaly detection). "||" indicates splicing along the channel dimension. Ultimately, ,in, represents the number of elements in the Gaussian filter that represent the standard deviation. For the number of channels, The height of the feature map, This represents the width of the feature map.

[0101] Current spatial feature map After compressing the channel through a 1×1 convolution, a global average pooling is performed to obtain a global context vector. ,Will Through another 1×1 convolution Map its channel number to This yields the key value K corresponding to the query vector. The global context vector is then used. Through a linear layer Generate a query vector The feature weights of the current spatial feature map are... . The values ​​are between (0,1), with larger values ​​indicating more significant and anomalous temporal changes at that location. Applying the feature weights to multi-scale Gaussian differences yields temporal difference features. .in, It is element-wise multiplication, while GAP is global average pooling, which pools away the spatial dimension and finally obtains a vector. This means that the time-series difference features input to the LSTM are significantly weighted, which achieves the effect of clearly highlighting significant changes related to device anomalies across frames and suppressing irrelevant changes.

[0102] Step 450: Input the multimodal fusion features into the generative adversarial network to obtain the first anomaly detection result of the device operation.

[0103] Optionally, the multimodal fusion features are input into a generative adversarial network to obtain the first anomaly detection result of device operation, including: finding similar pseudo-time series data of the multimodal fusion features in the generator of the generative adversarial network through spatial search; calculating the reconstruction error between the multimodal fusion features and the similar pseudo-time series data; and determining the first anomaly detection result of device operation based on the reconstruction error.

[0104] Figure 6 This is a schematic diagram of a generative adversarial network for anomaly detection according to Embodiment 2 of the present invention. Figure 6 As shown, the application process of generative adversarial networks (GANs) is similar to the process of testing networks using test samples, and will not be elaborated further here. After training, the model is deployed to the production environment to detect new time-series data. Since the model is trained using normal time-series data, the reconstructed scores for normal data are higher, while the reconstructed scores for anomalous data are lower due to larger reconstruction errors caused by deviations from the normal distribution. In this way, the output results of the classification network can be further validated, and the accuracy of anomaly detection can be improved. Using GANs to augment anomalous data can expand the training dataset and improve the model's generalization ability.

[0105] Step 460: Input the multimodal fusion features into the classification network to obtain the second anomaly detection result of the device operation.

[0106] Optionally, the multimodal fusion features are input into the classification network to obtain the second anomaly detection result of the device operation, including: inputting the multimodal fusion features into the classification network to obtain the category label output by the classification network; and determining the second anomaly detection result of the device operation based on the category label.

[0107] Step 470: Determine the target anomaly detection result based on the first anomaly detection result and the second anomaly detection result.

[0108] This system utilizes a trained classification network and a generative adversarial network (GAN) to perform overall anomaly detection on input time-series data. Specifically, the classification network initially determines whether a sequence belongs to an anomaly category, while the GAN further verifies the existence of anomalies by calculating reconstruction errors. The outputs of both networks are combined to filter out sequences deemed anomalous. For example, if both the classification and GANs determine a segment of time-series data to be anomalous, it is marked as such; otherwise, it is considered normal. This multi-model approach not only improves the accuracy of the detection results but also enhances the system's robustness, effectively addressing anomaly detection needs in complex industrial environments.

[0109] Step 480: Based on the target anomaly detection results and multimodal timing data, determine the location of the anomaly in the equipment operation timing.

[0110] Anomaly-prone multimodal time-series data can serve as input for subsequent local anomaly detection, enabling more precise local location of the anomaly. For time-series data identified as anomalous, further analysis of the specific causes of the anomaly can be conducted. This could involve visualizing the differences from normal data or combining it with other detection methods (such as local anomaly detection algorithms) to precisely pinpoint the anomaly point. Furthermore, the detection results can be used to trigger alarm systems or guide subsequent fault diagnosis and maintenance, thereby improving the efficiency and reliability of equipment monitoring in industrial production.

[0111] Optionally, based on the target anomaly detection results and multimodal time series data, the abnormal time series location of the equipment operation is determined, including: when the target anomaly detection result indicates that the equipment operation is abnormal, extracting local features from the abnormal time series data corresponding to the abnormal equipment operation through a sliding window; determining the anomaly score of each sliding window based on the local features and the density of the region where the local features are located; and determining the abnormal time series location of the equipment operation based on the anomaly score and a preset neighborhood value.

[0112] The technical solution of this invention involves acquiring multimodal time-series data during device operation, dividing the multimodal time-series data into multiple windows according to a preset window length, extracting window features of the multimodal time-series data in each window, and determining target multimodal time-series data based on the window features, fusing the target multimodal time-series data to obtain multimodal fusion features, acquiring a generative adversarial network pre-trained with positive sample time-series data, and a classification network pre-trained with both positive and negative sample time-series data, inputting the multimodal fusion features into the generative adversarial network to obtain a first anomaly detection result for device operation, and inputting the multimodal fusion features into the classification network to obtain... The system detects the second anomaly in equipment operation; based on the first and second anomaly detection results, it determines the target anomaly detection result; based on the target anomaly detection result and multimodal time-series data, it determines the anomaly time-series location in equipment operation, thus solving the problems of missed and false detections in equipment operation anomaly detection. Anomaly detection accuracy is improved by combining multimodal time-series data with multiple models; through multimodal data fusion and multi-model combination, the false alarm rate and missed alarm rate are significantly reduced; it has strong scalability, supports the addition of new modal data and detection algorithms, and adapts to the detection needs of various product models; for example, it can be easily extended to other industrial scenarios (such as chemical, energy, and other fields), demonstrating broad applicability.

[0113] In the deployment of generative adversarial networks (GANs) and classification networks, the trained networks can be exported into a standardized, deployable format to ensure seamless migration and use across different operating environments. Docker containerization technology can be used to encapsulate the network, building a lightweight service-oriented deployment solution, thereby improving system scalability and cross-platform compatibility, while simplifying the deployment process and reducing maintenance costs. After deployment, the system can receive new time-series data as input, and through preprocessing, multimodal feature fusion, and model inference, quickly generate prediction results and filter out potential anomalous sequence data. This process not only achieves an end-to-end closed loop from model development to the production environment but also provides efficient and stable technical support for real-time anomaly detection in industrial scenarios, significantly improving detection efficiency and response speed, and ensuring real-time inference capabilities in large-scale production environments. General-purpose hardware (such as industrial cameras and sensors) and open-source software tools can be used, reducing hardware and software development costs. Model service-oriented deployment (such as Docker containerization technology) simplifies the later maintenance process.

[0114] Figure 7 This is a flowchart of another multi-model anomaly detection method based on multimodal time series data provided in Embodiment 2 of the present invention. Figure 7As shown, multi-dimensional indicator data during equipment operation, such as vibration signals, temperature changes, pressure fluctuations, and equipment production and operation parameters, can be collected in real time using vibration sensors, temperature sensors, pressure sensors, and PLC data acquisition devices, forming multi-modal time-series data of equipment operation. The collected multi-modal time-series data facilitates comprehensive monitoring of equipment status and provides a rich data foundation for subsequent analysis. Preprocessing of multi-modal time-series data can be performed, such as max-min normalization, data time-series alignment, missing value imputation, outlier removal, and noise removal. Cross-modal attention mechanisms can be used for feature fusion to form multi-modal fused features.

[0115] It can be combined with distributed time-series databases to achieve efficient storage and management of large-scale data. During the data processing and model development phases, the Python programming language and its powerful ecosystem, including Pandas and NumPy, can be used for efficient data cleaning and preprocessing. TensorFlow and PyTorch can be used to build deep learning models to achieve complex anomaly detection tasks. Model training relies on high-performance GPU servers (such as NVIDIA Tesla V100) to accelerate the computation process, resulting in generative adversarial networks and classification networks. After training, the model is exported to a deployable format via TensorFlow Serving or ONNX Runtime, and Docker containerization technology is used to achieve service-oriented deployment of the model, ensuring high availability, scalability, and cross-platform compatibility, thereby supporting real-time inference and automated detection needs in industrial scenarios.

[0116] By combining anomaly detection results from generative adversarial networks (GANs) and classification networks, target anomaly detection results can be obtained, and anomalous segments can be further analyzed for anomalous time-series data. By fusing multiple modalities of data, production status and equipment operating status can be comprehensively captured, significantly improving the accuracy of anomaly detection. A cross-modal attention mechanism dynamically weights features of each modality, ensuring that key information is prioritized and avoiding the limitations of single-modal data. Combining classification networks, GANs, and local anomaly detection methods forms a multi-layered anomaly detection system capable of identifying both global and local anomalies. Multi-model joint training allows models to complement each other, improving the overall robustness and generalization ability of the system. For complex multimodal time-series data in industrial production, sliding window alignment and statistical feature extraction techniques effectively solve the problem of inconsistent data lengths. Large-scale data is managed through a distributed storage system and a distributed time-series database, supporting high-frequency sampling and real-time analysis.

[0117] In anomaly detection tasks, by combining generative adversarial networks (GANs) and classification networks, accuracy improved from 85% to 95%, recall from 80% to 92%, and F1 score from 82% to 93%. When processing large-scale datasets, the distributed storage and computing architecture of this invention significantly accelerates data loading and model inference. It can cover the detection needs of various product models, demonstrating greater adaptability. For example, by adjusting the sliding window size and cross-modal attention weights, it can be successfully applied to engine assembly lines and body welding workshops of different specifications. With the same detection accuracy, hardware costs are reduced by approximately 30%, and maintenance costs by approximately 20%. Local anomaly detection algorithms (such as isolated forests) accurately locate the time range or specific time point of anomalies. In noisy test datasets, the detection performance of this invention remains stable, with anomaly detection accuracy decreasing by no more than 5%, while the performance of existing technologies decreases by more than 20%.

[0118] Example 3

[0119] Figure 8 This is a schematic diagram of a multi-model anomaly detection device based on multimodal time-series data according to Embodiment 3 of the present invention. Figure 8 As shown, the device includes: a data preprocessing module 810, a data fusion module 820, a network acquisition module 830, a first anomaly detection result determination module 840, a second anomaly detection result determination module 850, and a target anomaly detection result determination module 860. Wherein:

[0120] The data preprocessing module 810 is used to acquire multimodal time series data during equipment operation and preprocess the multimodal time series data to obtain target multimodal time series data;

[0121] The data fusion module 820 is used to fuse the target multimodal time series data to obtain multimodal fusion features;

[0122] The network acquisition module 830 is used to acquire a generative adversarial network pre-trained with positive sample time series data, and a classification network pre-trained with positive sample time series data and negative sample time series data;

[0123] The first anomaly detection result determination module 840 is used to input the multimodal fusion features into the generative adversarial network to obtain the first anomaly detection result of the device operation;

[0124] The second anomaly detection result determination module 850 is used to input the multimodal fusion features into the classification network to obtain the second anomaly detection result of the device operation;

[0125] The target anomaly detection result determination module 860 is used to determine the target anomaly detection result based on the first anomaly detection result and the second anomaly detection result.

[0126] Optionally, the data preprocessing module 810 includes:

[0127] The window partitioning unit is used to divide multimodal time series data into multiple windows according to a preset window length;

[0128] The target multimodal time series data determination unit is used to extract window features of multimodal time series data in each window and determine the target multimodal time series data based on the window features.

[0129] Optionally, the device may also include:

[0130] The data autocorrelation coefficient determination unit is used to calculate the autocorrelation coefficient of the time series data of each mode before dividing the multimodal time series data into multiple windows according to the preset window length.

[0131] The preset window length determination unit is used to determine the preset window length for time alignment of multimodal time series data based on the autocorrelation coefficient of the time series data of each modality and the preset weight of the time series data of each modality.

[0132] Optional, the data fusion module 820 includes:

[0133] The fusion weight determination unit is used to learn the correlation between temporal data of different modalities through a cross-modal attention mechanism, and adjust the fusion weight of each temporal data according to the correlation.

[0134] The data fusion unit is used to fuse the target multimodal time series data according to the fusion weight when the fusion weight evaluation is passed, so as to obtain multimodal fusion features.

[0135] Optionally, the network acquisition module 830 includes:

[0136] The pseudo-time series data determination unit is used to input random noise vectors into the generator and map the random noise vectors to a high-dimensional data space through a multi-layer neural network to obtain pseudo-time series data.

[0137] The discrimination result determination unit is used to input positive sample time series data and pseudo time series data into the discriminator to obtain the discrimination result output by the discriminator;

[0138] Generative Adversarial Network (GAN) training units are used to evaluate the discrimination results and pseudo-time series data using a loss function, and to iteratively train the GAN based on the evaluation results to obtain the GAN.

[0139] Optionally, the network acquisition module 830 includes:

[0140] The sample dataset generation unit is used to pre-annotate the sample data and generate a sample dataset including positive sample time series data and negative sample time series data.

[0141] The classification network training unit is used to divide the sample dataset into training and test datasets according to a preset ratio, and to train the classifier using the training dataset to obtain the initial classification network.

[0142] The classification network performance evaluation unit is used to evaluate the performance of the initial classification network using a test dataset. If the evaluation is successful, the final classification network is obtained.

[0143] Optionally, the network acquisition module 830 includes:

[0144] The sample dataset generation unit is used to pre-annotate the sample data and generate a sample dataset including positive sample time series data and negative sample time series data.

[0145] The spatial feature map determination unit is used to input samples from the sample dataset into the initial convolutional network to obtain the spatial feature map of each sample; and to generate feature vectors based on the spatial feature map.

[0146] The spatial augmentation feature determination unit is used to input the feature vector into the encoder of the self-attention mechanism deep learning model to obtain the spatial augmentation features corresponding to the spatial feature map.

[0147] The temporal enhancement feature determination unit is used to input the spatial feature map into the long short-term memory network in chronological order to obtain the corresponding temporal enhancement features;

[0148] The target spatial and temporal augmentation feature determination unit is used to fuse spatial augmentation features and temporal augmentation features to obtain target spatial and temporal augmentation features corresponding to the sample.

[0149] The classification network determination unit is used to train the classifier based on the target space and time-series augmentation features to obtain the classification network.

[0150] Optional, the temporal enhancement feature determination unit is specifically used for:

[0151] Obtain the spatial feature maps of adjacent frames corresponding to the current spatial feature map, and determine the multi-scale Gaussian difference between the current spatial feature map and the spatial feature maps of adjacent frames;

[0152] Based on the current spatial feature map and multi-scale Gaussian difference, determine the feature weight values ​​of the current spatial feature map;

[0153] Based on the feature weight values ​​and multi-scale Gaussian differences, determine the temporal difference features corresponding to the current spatial feature map;

[0154] The temporal difference features are input into the Long Short-Term Memory network in chronological order to obtain the corresponding temporal enhancement features.

[0155] Optionally, the first anomaly detection result determination module 840 includes:

[0156] The similar pseudo-time series data identification unit is used to find similar pseudo-time series data of multimodal fusion features in the generator of the generative adversarial network through spatial search.

[0157] The first anomaly detection result determination unit is used to calculate the reconstruction error between the multimodal fusion features and similar pseudo-time series data, and to determine the first anomaly detection result of the equipment operation based on the reconstruction error.

[0158] Optionally, the second anomaly detection result determination module 850 includes:

[0159] The category label determination unit is used to input multimodal fusion features into the classification network to obtain the category labels output by the classification network;

[0160] The second anomaly detection result determination unit is used to determine the second anomaly detection result of the equipment operation based on the category label.

[0161] Optionally, the device may also include:

[0162] The local feature extraction module is used to extract local features from the abnormal time series data corresponding to the abnormal equipment operation abnormality by means of a sliding window after determining the target abnormality detection result based on the first abnormality detection result and the second abnormality detection result.

[0163] The anomaly score determination module is used to determine the anomaly score of each sliding window based on local features and the density of the region where the local features are located.

[0164] The abnormal timing location determination module is used to determine the abnormal timing location of the device operation based on the abnormal score and the preset neighborhood value.

[0165] The multi-model anomaly detection device based on multimodal time series data provided in this embodiment of the invention can execute the multi-model anomaly detection method based on multimodal time series data provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0166] Example 4

[0167] Figure 9 A schematic diagram of the structure of an electronic device 10 that can be used to implement embodiments of the present invention is shown. For example... Figure 9As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) or random access memory (RAM), communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from the storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. Input / output (I / O) interfaces are also connected to the bus 14.

[0168] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0169] Processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. In some embodiments, the multi-model anomaly detection method based on multimodal time-series data can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the multi-model anomaly detection method based on multimodal time-series data described above can be performed.

[0170] In the context of this invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0171] In order to provide interaction with the user, the electronic device has: a display device for displaying information to the user, and a keyboard and pointing device, through which the user can provide input to the electronic device.

[0172] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0173] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A multi-model anomaly detection method based on multi-modal time series data, characterized in that, The method comprises the following steps: acquiring multi-modal time series data in the operation of a device, and preprocessing the multi-modal time series data to obtain target multi-modal time series data; wherein the multi-modal time series data comprises at least two of the following: vibration indicators collected by a vibration sensor, temperature indicators collected by a temperature sensor, pressure indicators collected by a pressure sensor, images collected by an infrared thermal imager, device operation logs, current in the operation of the device, and voltage in the operation of the device; fusing the target multi-modal time series data to obtain multi-modal fusion features; acquiring a generative adversarial network generated in advance through positive sample time series data, and a classification network generated in advance through positive sample time series data and negative sample time series data; inputting the multi-modal fusion features into the generative adversarial network to obtain a first anomaly detection result of the operation of the device; inputting the multi-modal fusion features into the classification network to obtain a second anomaly detection result of the operation of the device; determining a target anomaly detection result according to the first anomaly detection result and the second anomaly detection result; acquiring a classification network generated in advance through positive sample time series data and negative sample time series data, comprising: pre-labeling sample data to generate a sample data set comprising positive sample time series data and negative sample time series data; inputting samples in the sample data set into an initial convolutional network to obtain spatial feature maps of the samples; and generating feature vectors according to the spatial feature maps; inputting the feature vectors into an encoder of a self-attention mechanism deep learning model to obtain spatial enhanced features corresponding to the spatial feature maps; inputting the spatial feature maps into a long short-term memory network in time sequence to obtain corresponding time sequence enhanced features; performing feature fusion on the spatial enhanced features and the time sequence enhanced features to obtain target spatial and time sequence enhanced features corresponding to the samples; and performing classifier training according to the target spatial and time sequence enhanced features to obtain a classification network; wherein inputting the spatial feature maps into the long short-term memory network in time sequence to obtain corresponding time sequence enhanced features comprises: acquiring adjacent frame spatial feature maps corresponding to a current spatial feature map, and determining a multi-scale Gaussian difference between the current spatial feature map and the adjacent frame spatial feature maps; determining a feature weight value of the current spatial feature map according to the current spatial feature map and the multi-scale Gaussian difference; determining a time sequence difference feature corresponding to the current spatial feature map according to the feature weight value and the multi-scale Gaussian difference; and inputting the time sequence difference feature into the long short-term memory network in time sequence to obtain corresponding time sequence enhanced features.

2. The method of claim 1, wherein, The preprocessing of the multi-modal time series data to obtain target multi-modal time series data comprises: dividing the multi-modal time series data into a plurality of windows according to a preset window length; extracting window features of the multi-modal time series data in each window, and determining target multi-modal time series data according to the window features.

3. The method of claim 2, wherein, Before dividing the multi-modal time series data into a plurality of windows according to a preset window length, the method further comprises: calculating autocorrelation coefficients of time series data of each modality; Determine a preset window length in time sequence alignment of the multi-modal time sequence data according to the autocorrelation coefficients of the time sequence data of each modality and preset weights of the time sequence data of each modality.

4. The method of claim 1, wherein, Fuse the target multi-modal time sequence data to obtain multi-modal fusion features, including: Learn the correlation between different modal time sequence data through a cross-modal attention mechanism, and adjust the fusion weights of the modal time sequence data according to the correlation; When the fusion weight evaluation passes, fuse the target multi-modal time sequence data according to the fusion weights to obtain multi-modal fusion features.

5. The method of claim 1, wherein, Obtain a generative adversarial network generated by training with positive sample time sequence data in advance, including: Input a random noise vector into the generator, and map the random noise vector to a high-dimensional data space through a multi-layer neural network to obtain pseudo time series data; Input the positive sample time sequence data and the pseudo time series data into the discriminator to obtain a discrimination result output by the discriminator; Evaluate the discrimination result and the pseudo time series data through a loss function, and iteratively train according to the evaluation result to obtain the generative adversarial network.

6. The method of claim 1, wherein, Input the multi-modal fusion features into the generative adversarial network to obtain a first anomaly detection result of device operation, including: Find similar pseudo time series data of the multi-modal fusion features in the generator of the generative adversarial network through spatial search; Calculate the reconstruction error between the multi-modal fusion features and the similar pseudo time series data, and determine the first anomaly detection result of device operation according to the reconstruction error.

7. The method of claim 1, wherein, After determining the target anomaly detection result according to the first anomaly detection result and the second anomaly detection result, further including: When the target anomaly detection result is device operation anomaly, extract local features in abnormal time sequence data corresponding to device operation anomaly through a sliding window; Determine the anomaly score of each sliding window according to the local features and the density of the region where the local features are located; Determine the abnormal time sequence position of device operation according to the anomaly score and a preset neighborhood value.

8. The method of claim 7, wherein, Determine the anomaly score of each sliding window according to the local features and the density of the region where the local features are located, including: Determine the k-neighborhood parameter of the local features according to the density of the region where the local features are located; Determine the reachable distance between each local feature in the k-neighborhood according to the timestamps of each local feature in the abnormal time sequence data, the distance between the local feature and the k-neighborhood, and the distance between each local feature in the k-neighborhood; Determine the local reachable density of the region according to the reachable distance between each local feature in the k-neighborhood and the k-neighborhood parameter, and take the local reachable density as the anomaly score corresponding to the local feature.

Images