Data anomaly detection method, system and equipment based on deep learning and medium

Abstract

The invention discloses a data anomaly detection method, system and device based on deep learning, and a medium, mainly relates to the technical field of data anomaly detection, and is used for solving the problems that a traditional anomaly detection method has limitation when facing high-dimensional data, and the anomaly detection method is mostly designed for a single anomaly type of a specific scene. Comprising the following steps: processing data into preset low-latitude to-be-identified data; updating auto-encoder parameters in the trained deep neural network model by using the to-be-identified data, and performing feature extraction on the to-be-identified data by using the trained deep neural network model to obtain feature representation of the data; and based on the extracted feature representation, performing anomaly detection on the feature representation by using a trained anomaly detection algorithm to obtain abnormal data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data anomaly detection, and particularly relates to a data anomaly detection method, system, device and medium based on deep learning. BACKGROUND

[0002] With the development of information technology, the amount of data generated by various systems is growing explosively. How to efficiently and accurately find anomalies from massive data has become an important research topic. Traditional anomaly detection methods such as statistical-based methods and clustering algorithms have limitations when facing high-dimensional data.

[0003] Although traditional anomaly detection methods have made significant progress, there are still key technical bottlenecks, especially in terms of universality. Existing methods are mostly designed for a single anomaly type in a specific scenario, for example, models optimized for time series data anomalies are difficult to adapt to image data anomaly detection, and network structures developed for numerical data anomalies perform poorly on text data anomaly recognition. This scenario dependence results in high model migration costs and makes it difficult to meet the needs of anomaly detection for multi-domain and multi-type data, limiting the large-scale application of traditional anomaly detection methods in this field. SUMMARY

[0004] The present application provides a data anomaly detection method, system, device and medium based on deep learning to solve the problem that traditional anomaly detection methods have limitations when facing high-dimensional data and anomaly detection methods are mostly designed for a single anomaly type in a specific scenario.

[0005] In a first aspect, the present application provides a data anomaly detection method based on deep learning, the method comprising: reading the data type of the original data, determining the corresponding preprocessing algorithm based on the data type, and processing the original data into pre-set low-dimensional data to be recognized; updating the autoencoder parameters in the trained deep neural network model using the data to be recognized, and then using the trained deep neural network model to extract features from the data to be recognized to obtain feature representations of the data; based on the extracted feature representations, using the trained anomaly detection algorithm to detect anomalies in the feature representations, and obtaining anomaly data.

[0006] In an implementation manner of the present application, before using the trained deep neural network model to extract features from the data to be recognized to obtain feature representations of the data, the method further comprises: training the deep neural network model and the anomaly detection algorithm using pre-set normal sample data to obtain the trained deep neural network model and the trained anomaly detection algorithm; The trained deep neural network model and the trained anomaly detection algorithm are evaluated by using test data to obtain performance indicators of the trained deep neural network model and the trained anomaly detection algorithm. According to a preset time period, the latest test data is obtained, and the performance indicator detection is performed again, and when the performance indicator is lower than a threshold, the training is performed again.

[0007] In an implementation manner of the present application, the autoencoder parameters in the trained deep neural network model are updated by using the to-be-recognized data, and specifically includes: The autoencoder parameters are adjusted by using a loss function: . Wherein, N represents the total number of the to-be-recognized data, represents the to-be-recognized data of the i-th input, i represents the encoder, represents the decoder, represents the autoencoder parameters.

[0008] In an implementation manner of the present application, the trained anomaly detection algorithm includes a trained isolation forest algorithm. Based on the extracted feature representation, the trained anomaly detection algorithm is used to perform anomaly detection on the feature representation to obtain abnormal data, and specifically includes: For each feature representation y, the number of edges passed from the root node to the leaf node in each decision tree of the isolation forest algorithm is counted as the path length h(y). The abnormal score is calculated by using the formula: . Wherein, represents the average path length of the tree, represents the average path length of the feature representation y in all isolation trees. When the abnormal score is greater than a preset threshold, it is determined as abnormal data, otherwise it is not abnormal data.

[0009] In an implementation manner of the present application, the trained deep neural network model is used to extract features of the to-be-recognized data to obtain feature representations of the data, and specifically includes: The trained deep neural network model and the updated autoencoder parameters are loaded; The top classification layer of the trained deep neural network model is removed; The output of the preset specified output layer is obtained by forward propagation to obtain the feature representation of the data.

[0010] ​In an implementation form of the present application, the data type of the original data is read, a corresponding preprocessing algorithm is determined based on the data type, and the original data is processed into the preset low-dimensional to-be-identified data, specifically comprising: The data type of the original data is read, a corresponding text conversion program is determined based on the data type, and text data is obtained; Based on the source address of the text data, a semantic mapping rule corresponding to the source address is determined; The text data is processed into preset standard data using the semantic mapping rule; The preset standard data is processed into the preset dimension to-be-identified data through dimension compression technology.

[0011] In a second aspect, the present application provides a data anomaly detection system based on deep learning, which comprises: The preprocessing module is used for preprocessing the unit, reading the data type of the original data, determining the corresponding preprocessing algorithm based on the data type, and processing the original data into the preset low-dimensional to-be-identified data; The extraction module is used for updating the auto-encoder parameters in the trained deep neural network model using the to-be-identified data, and then performing feature extraction on the to-be-identified data using the trained deep neural network model to obtain the feature representation of the data; The detection module is used for performing anomaly detection on the data based on the extracted feature representation using the trained anomaly detection algorithm to obtain the abnormal data.

[0012] In an implementation form of the present application, the system further comprises a training module, for training the deep neural network model and the anomaly detection algorithm using the preset normal sample data to obtain the trained deep neural network model and the trained anomaly detection algorithm; The trained deep neural network model and the trained anomaly detection algorithm are evaluated using test data to obtain the performance indicators of the trained deep neural network model and the trained anomaly detection algorithm; According to the preset time period, the latest test data is obtained, and the performance indicator detection is performed again, and when the performance indicator is lower than the threshold, the training is performed again.

[0013] In a third aspect, the present application provides a data anomaly detection device based on deep learning, which comprises: A processor; And a memory having executable code stored thereon, when the executable code is executed, the processor executes any one of the above data anomaly detection methods based on deep learning.

[0014] In a fourth aspect, the present application provides a non-volatile computer storage medium having computer instructions stored thereon, the computer instructions, when executed, implementing a deep learning-based data anomaly detection method according to any one of the above.

[0015] From the above technical solutions, the present application has the following advantages: First, by determining the corresponding preprocessing algorithm based on the data type, the unified to-be-recognized data is obtained, and the effective recognition of multiple types of abnormal data can be realized without being limited to a single abnormal type. In addition, the data is processed into a preset low dimension, breaking the limitation of traditional anomaly detection methods in high-dimensional data processing. It can read the data type of the original data first, and determine the corresponding preprocessing algorithm based on the data type to process the original data into the to-be-recognized data of the preset low dimension. The preprocessing step can enable the subsequent anomaly detection process to be carried out on the basis of lower-dimensional data, reduce the consumption of computing resources, and improve the efficiency of data processing. Second, with the synergistic effect of the deep neural network model and the anomaly detection algorithm, the limitation of traditional anomaly detection methods that are designed for a specific scene and a single abnormal type is overcome, and the anomaly detection capability is wider. On the one hand, the self-encoder parameters in the trained deep neural network model are updated by using the to-be-recognized data, and then the model is used to extract features of the to-be-recognized data to obtain the feature representation of the data. This way of updating the model parameters based on the data can make the model better adapt to the features of the current to-be-detected data, and the extracted feature representation is more targeted and accurate, which can accurately capture the complex patterns and potential correlations in the data. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0017] Figure 1 is a flow chart of a deep learning-based data anomaly detection method provided by an embodiment of the present application.

[0018] Figure 2 is a schematic diagram of the internal structure of a deep learning-based data anomaly detection system provided by an embodiment of the present application.

[0019] Figure 3 is a schematic diagram of the internal structure of a deep learning-based data anomaly detection device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0020] Clearly, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present application.

[0021] Those skilled in the art should understand that the embodiments described below are only preferred embodiments of the present disclosure, and do not represent the only way to implement the present disclosure. The preferred embodiments are only used to explain the technical principles of the present disclosure, and are not used to limit the protection scope of the present disclosure. Based on the preferred embodiments provided by the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts still fall within the protection scope of the present disclosure.

[0022] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed, or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

[0023] The technical solutions of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0024] The embodiments provide a data anomaly detection method based on deep learning, as shown in Figure 1 The method provided by the embodiments of the present application mainly includes the following steps: Step 110, reading the data type of the original data, determining the corresponding preprocessing algorithm based on the data type, and processing the original data into preset low-dimensional to-be-recognized data.

[0025] In some embodiments, reading the data type of the original data, determining the corresponding preprocessing algorithm based on the data type, and processing the original data into preset low-dimensional to-be-recognized data, specifically includes: Reading the data type of the original data, determining the corresponding text conversion program based on the data type, and then obtaining text data; Based on the source address of the text data, determining the semantic mapping rule corresponding to the source address; Using the semantic mapping rule, processing the text data into preset standard data; By dimension compression technology, the preset standard data is processed into preset dimension to-be-recognized data.

[0026] Based on the above description, the present step realizes the standardization and efficiency of data processing through the data type driven preprocessing process. The effects are reflected in three aspects: first, the dynamic algorithm selection based on data type enables the system to automatically adapt to heterogeneous data sources such as text and image, without manual intervention to complete the initial format conversion; second, the introduction of semantic mapping rules solves the semantic difference problem of different source texts, such as converting social media abbreviations, professional terms, etc. into standard expressions, improving data consistency; finally, the dimension compression technology reduces the data volume to the preset dimension through feature extraction and redundancy elimination, significantly reducing the computational load of subsequent recognition algorithms. The overall scheme makes the original data have the dual characteristics of structure and low dimension, providing a standardized data format that can be directly input into the machine learning model.

[0027] It should be noted that the semantic mapping rules can be obtained by the person skilled in the art through multiple experiments.

[0028] Step 120, updating the autoencoder parameters in the trained deep neural network model with the to-be-identified data, and then using the trained deep neural network model to extract features of the to-be-identified data to obtain the feature representation of the data.

[0029] It should be noted that the present application also includes an algorithm / model training phase, which can be specifically: Training the deep neural network model and the anomaly detection algorithm using the preset normal sample data to obtain a trained deep neural network model and a trained anomaly detection algorithm; Evaluating the trained deep neural network model and the trained anomaly detection algorithm using test data to obtain the performance indicators of the trained deep neural network model and the trained anomaly detection algorithm; According to the preset time period, obtaining the latest test data, and detecting the performance indicators again, when the performance indicators are lower than the threshold, training again.

[0030] In the step, the autoencoder parameters in the trained deep neural network model are updated using the to-be-identified data, which specifically includes: Adjusting the autoencoder parameters through the loss function: Wherein, N represents the total number of to-be-identified data, represents the i th input to-be-identified data, represents the encoder, represents the decoder, represents the autoencoder parameters.

[0031] ​In the step, the trained deep neural network model is used for feature extraction of the to-be-identified data to obtain a feature representation of the data, and the feature extraction specifically includes: loading the trained deep neural network model and the updated autoencoder parameters; removing the top classification layer of the trained deep neural network model; forward propagation to obtain the output of the preset specified output layer to obtain the feature table of the data.

[0032] Based on the above description, the step realizes continuous optimization of data processing through dynamic updating and feature extraction mechanisms. The effect is reflected in three aspects: first, the iterative training of the preset normal samples enables the model to have stable baseline feature learning ability, and the performance evaluation mechanism driven by test data ensures the reliability of the model in actual application; second, the autoencoder parameters are updated by minimizing the reconstruction error (loss function), so that the model can adapt to the distribution changes of new data, such as signal drift caused by equipment aging in industrial detection; finally, the feature extraction stage removes the classification layer and fixes the specified output layer, which converts the original data into a high-dimensional feature vector. This processing method preserves the essential features of the data while avoiding the interference of the classification task on the features. The overall scheme enables the model to maintain the recognition ability of known patterns and gradually adapt to the evolution of data distribution, providing stable feature input for subsequent anomaly detection.

[0033] In step 130, based on the extracted feature representation, the trained anomaly detection algorithm is used to perform anomaly detection on the feature representation to obtain abnormal data.

[0034] It should be noted that the trained anomaly detection algorithm includes a trained isolation forest algorithm.

[0035] This step can be specifically: For each feature representation y, the number of edges passed from the root node to the leaf node in each decision tree of the isolation forest algorithm is counted as the path length h(y); The anomaly score is calculated by the formula: wherein, represents the average path length of the tree, represents the average path length of the feature representation y in all isolated trees; When the anomaly score is greater than a preset threshold, it is determined to be abnormal data, otherwise it is not abnormal data.

[0036] ​Based on the above description, this step realizes accurate identification of data anomalies through an isolation forest-based anomaly detection mechanism. The direct effects are reflected in three aspects: first, the isolation forest algorithm calculates the path length of feature representation by constructing random decision trees, and automatically discovers data deviation without pre-setting an anomaly model by using the sparse characteristics (short path length) of abnormal data distribution; second, the abnormal score calculation combines the path length of a single tree with the overall average path length, and through standardization processing, the detection results of different feature dimensions are comparable, such as processing multiple source signals such as temperature and vibration in industrial sensor data; finally, the preset threshold determination mechanism converts the anomaly detection into a clear binary classification problem, and directly outputs the abnormal data when the abnormal score exceeds the threshold. This processing method ensures detection efficiency while avoiding complex probability calculation processes. The overall scheme enables the system to quickly locate anomalies in high-dimensional feature representation, providing a direct basis for subsequent warning or processing.

[0037] Based on the foregoing description, the embodiment determines the corresponding preprocessing algorithm based on the data type to obtain uniform to-be-identified data, and realizes effective identification of multiple types of abnormal data without being limited to a single abnormal type. In addition, the data is processed into a preset low dimension, breaking through the limitations of traditional anomaly detection methods in high-dimensional data processing. It can read the data type of the original data first, determine the corresponding preprocessing algorithm based on the data type, and process the original data into to-be-identified data of a preset low dimension. This preprocessing step enables the subsequent anomaly detection process to be carried out on the basis of lower-dimensional data, reduces the consumption of computing resources, and improves the efficiency of data processing. With the synergistic effect of the deep neural network model and the anomaly detection algorithm, it breaks through the limitations of traditional anomaly detection methods that are designed for a single abnormal type in a specific scene, and has more extensive anomaly detection capabilities. On the one hand, the to-be-identified data is used to update the parameters of the trained autoencoder in the deep neural network model, and then the model is used to extract features of the to-be-identified data to obtain feature representation of the data. This method of updating model parameters based on data can better adapt the model to the features of the current to-be-detected data, and the extracted feature representation is more targeted and accurate, which can accurately capture the complex patterns and potential associations within the data.

[0038] In addition, the present application Figure 2 A data anomaly detection system based on deep learning is provided for the embodiments of the present application. As shown in Figure 2 The system provided by the embodiments of the present application mainly includes: The preprocessing module 210 is used for preprocessing unit, which is used for reading the data type of the original data, determining the corresponding preprocessing algorithm based on the data type, and processing the original data into to-be-identified data of a preset low dimension.

[0039] Based on the above description, the preprocessing module 210 realizes efficient standardized processing of data through a data type driven preprocessing process. The effect is reflected in three aspects: first, the system can automatically identify the type of original data (such as text, image, numerical value, etc.), and call the corresponding preprocessing algorithm to complete the initial conversion, for example, unstructured text is converted into standardized word vectors, without manual configuration, it can adapt to multi-source data; second, the preprocessing process unifies heterogeneous data into low-dimensional feature representation through semantic mapping and dimension compression technology, for example, different formats of sensor data are converted into fixed-dimensional numerical matrix, eliminating the influence of data source difference on subsequent analysis; finally, the output to be recognized data has the characteristics of structure and low redundancy, which not only retains the key information of the original data, but also reduces the storage and calculation cost, providing a standardized format that can be directly input to the machine learning model. The overall scheme makes the data processing process have the ability of automatic adaptation and efficient conversion, improving the reliability and efficiency of subsequent analysis tasks.

[0040] The extraction module 220 is configured to update the autoencoder parameters in the trained deep neural network model using the to-be-recognized data, and then use the trained deep neural network model to extract features of the to-be-recognized data to obtain feature representation of the data.

[0041] Based on the above description, the extraction module 220 realizes continuous optimization of data processing through dynamic parameter updating and feature extraction. The effect is reflected in three aspects: first, updating the autoencoder parameters using the to-be-recognized data enables the model to adapt to changes in data distribution, for example, it can gradually adapt to signal drift caused by equipment aging in industrial scenarios; second, through the parameter adjustment mechanism driven by the loss function, the model can retain key features while suppressing noise interference, for example, in image processing, it can effectively remove interference caused by changes in lighting; finally, the feature extraction stage obtains the essential representation of the data through a fixed output layer, which avoids the interference of classification tasks on features and provides a structured input for subsequent analysis, for example, in anomaly detection, it can directly output a feature vector strongly related to the task. The overall scheme enables the model to maintain the ability to recognize known patterns while gradually adapting to new data characteristics.

[0042] The detection module 230 is configured to use a trained anomaly detection algorithm to detect anomalies in the data based on the extracted feature representation, and obtain abnormal data.

[0043] Based on the above description, the detection module 230 realizes accurate identification of data anomalies through the combination of feature representation and anomaly detection. The effects are reflected in three aspects: first, the trained anomaly detection algorithm can directly process the feature representation, avoiding the interference of redundant information in the original data on the detection result, for example, effectively filtering irrelevant environmental noise in industrial equipment monitoring; second, based on the preset judgment rule (such as the anomaly score threshold of the isolation forest), the anomaly detection is converted into a clear yes / no judgment, enabling the system to quickly output abnormal data, for example, real-time marking of suspicious behavior in financial transactions; finally, the low-dimensional nature of the feature representation enables the detection algorithm to run efficiently, ensuring processing timeliness even in high-frequency data stream scenarios. The overall scheme enables the system to have automated anomaly positioning capability for structured data, providing a direct basis for subsequent warning or processing.

[0044] The system also includes a training module, for training a deep neural network model and an anomaly detection algorithm using preset normal sample data, obtaining a trained deep neural network model and a trained anomaly detection algorithm; using test data to evaluate the trained deep neural network model and the trained anomaly detection algorithm, obtaining performance indicators of the trained deep neural network model and the trained anomaly detection algorithm; According to the preset time period, the latest test data is obtained, and the performance indicator detection is performed again. When the performance indicator is lower than the threshold, the training is performed again.

[0045] Based on the foregoing description, the embodiment determines the corresponding preprocessing algorithm based on the data type, obtains uniform to-be-identified data, and realizes effective identification of multiple types of abnormal data without being limited to a single abnormal type. In addition, the data is processed to a preset low dimension, breaking through the limitations of traditional anomaly detection methods in high-dimensional data processing. It can read the data type of the original data first, and determine the corresponding preprocessing algorithm based on the data type to process the original data into to-be-identified data of a preset low dimension. This preprocessing step enables the subsequent anomaly detection process to be carried out on the basis of lower-dimensional data, reducing the consumption of computing resources and improving the efficiency of data processing. With the synergistic effect of the deep neural network model and the anomaly detection algorithm, it breaks free from the limitations of traditional anomaly detection methods that are designed for a single abnormal type in a specific scene, and has more extensive anomaly detection capability. On the one hand, the to-be-identified data is used to update the autoencoder parameters in the trained deep neural network model, and then the model is used to extract features from the to-be-identified data to obtain the feature representation of the data. This method of updating model parameters based on data can better adapt the model to the features of the current to-be-detected data, and the extracted feature representation is more targeted and accurate, which can accurately capture the complex patterns and potential associations within the data.

[0046] The above is a method embodiment in the present application. Based on the same inventive concept, the present application embodiment also provides a deep learning-based data anomaly detection device. As shown in the figure, the device comprises a processor and a memory having executable code stored thereon, which, when executed, causes the processor to perform a deep learning-based data anomaly detection method as described above. Figure 3 The above is a method embodiment in the present application. Based on the same inventive concept, the present application embodiment also provides a deep learning-based data anomaly detection device. As shown in the figure, the device comprises a processor and a memory having executable code stored thereon, which, when executed, causes the processor to perform a deep learning-based data anomaly detection method as described above.

[0047] Specifically, the server reads the data type of the original data, determines the corresponding preprocessing algorithm based on the data type, processes the original data into preset low-dimensional to-be-recognized data, updates the auto-encoder parameters in the trained deep neural network model using the to-be-recognized data, and then uses the trained deep neural network model to extract features of the to-be-recognized data to obtain the feature representation of the data. Based on the extracted feature representation, the trained anomaly detection algorithm is used to detect the feature representation for anomaly, and the abnormal data is obtained.

[0048] In addition, the present application embodiment also provides a non-volatile computer storage medium having executable instructions stored thereon, which, when executed, implement a deep learning-based data anomaly detection method as described above.

[0049] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data anomaly detection method based on deep learning, characterized in that, The method includes: Read the data type of the raw data, determine the corresponding preprocessing algorithm based on the data type, and process the raw data into preset low-dimensional data to be identified; The autoencoder parameters in the trained deep neural network model are updated using the data to be identified, and then the trained deep neural network model is used to extract features from the data to be identified to obtain the feature representation of the data. Based on the extracted feature representation, anomalies are detected using a trained anomaly detection algorithm to obtain abnormal data.

2. The data anomaly detection method based on deep learning according to claim 1, characterized in that, Before using a trained deep neural network model to extract features from the data to be identified and obtain the feature representation of the data, the method further includes: The deep neural network model and the anomaly detection algorithm are trained using preset normal sample data to obtain the trained deep neural network model and the trained anomaly detection algorithm. The trained deep neural network model and the trained anomaly detection algorithm were evaluated using test data to obtain their performance metrics. According to the preset time period, the latest test data is obtained, and the performance indicators are tested again. When the performance indicators are lower than the threshold, training is performed again.

3. The data anomaly detection method based on deep learning according to claim 1, characterized in that, Updating the autoencoder parameters in the trained deep neural network model using the data to be identified specifically includes: Through the loss function: Adjust the autoencoder parameters; Where N represents the total number of data to be identified. Indicates the first i The input data to be identified, Indicates encoder, Indicates decoder, This represents the parameters of the autoencoder.

4. The data anomaly detection method based on deep learning according to claim 1, characterized in that, Well-trained anomaly detection algorithms include well-trained Isolation Forest algorithms; Based on the extracted feature representation, an anomaly detection algorithm is used to detect anomalies in the feature representation to obtain anomalous data, specifically including: For each feature representation y, the number of edges traversed from the root node to the leaf node in each decision tree of the isolated forest algorithm is counted as the path length h(y). Through the formula: Calculate the outlier score; in, This represents the average path length of the tree. The feature represents the average path length of y across all isolated trees; Data is identified as abnormal if the abnormal score is greater than a preset threshold; otherwise, it is not considered abnormal data.

5. The data anomaly detection method based on deep learning according to claim 1, characterized in that, The trained deep neural network model is used to extract features from the data to be identified, resulting in a feature representation of the data. This specifically includes: Load the trained deep neural network model and the updated autoencoder parameters; Remove the top classification layer of the trained deep neural network model; Forward propagation acquires the output of a predefined output layer, resulting in a feature table of the data.

6. The data anomaly detection method based on deep learning according to claim 1, characterized in that, The data type of the raw data is read, and the corresponding preprocessing algorithm is determined based on the data type. The raw data is then processed into pre-defined low-dimensional data to be identified, specifically including: Read the data type of the raw data, determine the corresponding text conversion program based on the data type, and then obtain the text data; Based on the source address of the text data, determine the semantic mapping rules corresponding to the source address; Using semantic mapping rules, text data is processed into preset standard data; By using dimensionality compression technology, preset standard data is processed into data to be identified with preset dimensions.

7. A data anomaly detection system based on deep learning, characterized in that, The system includes: The preprocessing module is used by the preprocessing unit to read the data type of the raw data, determine the corresponding preprocessing algorithm based on the data type, and process the raw data into preset low-dimensional data to be identified. The extraction module is used to update the autoencoder parameters in the trained deep neural network model using the data to be identified, and then use the trained deep neural network model to extract features from the data to be identified to obtain the feature representation of the data. The detection module is used to detect anomalies in the data based on the extracted feature representation and a trained anomaly detection algorithm, thereby obtaining abnormal data.

8. The data anomaly detection system based on deep learning according to claim 7, characterized in that, The system also includes a training module. It is used to train a deep neural network model and anomaly detection algorithm using preset normal sample data, so as to obtain a trained deep neural network model and a trained anomaly detection algorithm. The trained deep neural network model and the trained anomaly detection algorithm were evaluated using test data to obtain their performance metrics. According to the preset time period, the latest test data is obtained, and the performance indicators are tested again. When the performance indicators are lower than the threshold, training is performed again.

9. A data anomaly detection device based on deep learning, characterized in that, The device includes: processor; And a memory having executable code stored thereon, which, when executed, causes the processor to perform a deep learning-based data anomaly detection method as described in any one of claims 1-6.

10. A non-volatile computer storage medium, characterized in that, It stores computer instructions, which, when executed, implement a deep learning-based data anomaly detection method as described in any one of claims 1-6.

Images