Abnormal data identification and cleaning method and system

Abstract

The invention provides a method and a system for identifying and cleaning abnormal data, which relate to the technical field of data processing, and are used for identifying and cleaning abnormal data by acquiring real-time data flow, determining a self-adaptive threshold value by using a trained Gaussian mixture model and detecting abnormal data points. A deep belief network and long and short term memory network composite model is adopted to carry out deep feature learning and anomaly verification, and high-level feature representation and anomaly degree quantitative evaluation results are obtained; and evaluating a conditional dependency relationship among the abnormal data points by using the probability graph model, constructing a target probability dependency graph, and generating an abnormal data point recognition result. Based on an identification result, a reward mechanism excitation algorithm is applied to optimize a cleaning strategy; according to the method, the to-be-determined abnormal region is identified, the multi-modal fusion technology is introduced to perform correlation analysis, and the data in the to-be-determined abnormal region is judged and cleaned based on the analysis result to obtain the target cleaning strategy, so that the abnormal data identification and cleaning efficiency and accuracy are improved, and the method has relatively high adaptive capability and expansibility.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a method and system for identifying and cleaning abnormal data. Background Art

[0002] In the era of big data, data has become an important basis for corporate decision-making. Whether in the fields of finance, medicine, e-commerce or social media, high-quality data is the basis for accurate analysis and intelligent decision-making. However, in practical applications, abnormal data is often generated due to errors in the data collection process, equipment failures or human factors. If these abnormal data are not identified and cleaned, they may cause distortion of data analysis results and affect the accuracy of business decisions. Therefore, for various application scenarios, it is particularly important to develop an efficient and accurate method for identifying and cleaning abnormal data.

[0003] At present, statistical methods are mainly used to deal with abnormal data. Existing statistical methods artificially set thresholds by calculating the basic statistics of the data set, and regard data points beyond a certain range as outliers; statistical methods rely on data distribution assumptions and are not effective for non-normal or complex distributed data; the selection of reasonable thresholds requires operators to have rich domain knowledge, and different application scenarios may require different threshold settings, which increases the complexity of operations; some statistical methods are very sensitive to extreme values ​​and may lead to misjudgment. Summary of the invention

[0004] The embodiment of the present invention provides a method and system for identifying and cleaning abnormal data, so as to solve the problems in the prior art of poor effect on non-normal or complex distributed data, difficulty in threshold selection, and easy misjudgment of extreme value sensitivity.

[0005] In a first aspect, an embodiment of the present invention provides a method for identifying and cleaning abnormal data, comprising: Receiving real-time data streams from different sources, the real-time data streams containing structured data and unstructured data; Acquire a real-time data stream of the target to be tested, perform preliminary anomaly detection on the real-time data stream using an adaptive threshold determined by a trained Gaussian mixture model, and detect abnormal data points; Based on the abnormal data points, a composite model of a deep belief network combined with a long short-term memory network is used to perform deep feature learning and abnormality verification, thereby obtaining high-level feature representation and quantitative evaluation results of the degree of abnormality; Based on the high-level feature representation and the quantitative evaluation result of the abnormality degree, a probabilistic graph model is used to evaluate the conditional dependency relationship between the abnormal data points, a target probabilistic dependency graph is constructed based on the conditional dependency relationship, and an abnormal data point identification result is generated based on the target probabilistic dependency graph; Based on the abnormal data point identification result, a defined reward mechanism incentive algorithm is applied to optimize and adjust the existing cleaning strategy to obtain an optimized cleaning strategy; Identify the abnormal area to be determined in the evaluation process of the probability graph model, introduce multimodal fusion technology to perform correlation analysis on the abnormal area to be determined, and obtain the correlation analysis result; based on the correlation analysis result, perform abnormal data judgment and abnormal data cleaning on the data in the abnormal area to be determined, and obtain the abnormal data processing result to be determined; based on the abnormal data processing result to be determined, adjust the cleaning strategy in the optimization to obtain the target cleaning strategy.

[0006] Optionally, the identifying of the abnormal region to be determined in the evaluation process of the probability graph model, introducing a multimodal fusion technology to perform correlation analysis on the abnormal region to be determined, obtaining a correlation analysis result, performing abnormal data judgment and abnormal data cleaning on the data in the abnormal region to be determined based on the correlation analysis result, obtaining a processing result of the abnormal data to be determined, and adjusting the cleaning strategy in the optimization based on the processing result of the abnormal data to be determined to obtain a target cleaning strategy, including: Analyze the nodes and edges in the target probability dependency graph, identify the target nodes and target edges whose conditional dependencies are higher than a preset threshold, and determine the abnormal area to be determined based on the target nodes and the target edges; Collecting multimodal data corresponding to each data point in the abnormal area to be determined, fusing the multimodal data using multimodal fusion technology to generate a comprehensive feature representation, and based on the comprehensive feature representation, using a mutual information method to calculate the correlation between each data point in the abnormal area to be determined, to obtain a correlation analysis result; Based on the correlation analysis result, abnormal data judgment and cleaning operations are performed on the data points in the abnormal area to be determined to obtain the abnormal data processing result to be determined, and based on the abnormal data processing result to be determined, the cleaning strategy in the optimization is evaluated and adjusted to generate a target cleaning strategy.

[0007] Optionally, the multimodal data is fused using a multimodal fusion technology to generate a comprehensive feature representation, and based on the comprehensive feature representation, a mutual information method is used to calculate the correlation between each data point in the abnormal area to be determined to obtain a correlation analysis result, including: Based on the multimodal data, a convolutional neural network is used to extract image features, and a recurrent neural network is used to extract time series features; Constructing a multi-layer perception structure, based on the multi-layer perception machine structure, fusing the image features and the time series features to generate a preliminary comprehensive feature representation, selecting a loss function based on the preliminary fused feature representation, and applying a back propagation algorithm to minimize the value of the loss function to obtain a minimized loss function, using the minimized loss function to train the multi-layer perception structure to perform iterative optimization until a stop condition is met to obtain a multi-layer perception machine model, wherein the stop condition is reaching a preset threshold of the loss function, and based on the multi-layer perception machine model, taking the image features and the time series features as inputs, and outputting a comprehensive feature representation; Based on the comprehensive feature representation, a mutual information method is used to calculate the mutual information value of each pair of data points in the abnormal area to be determined, the mutual information values ​​are integrated to generate a correlation matrix, and a correlation analysis result is generated based on the correlation matrix.

[0008] Optionally, the acquiring of the real-time data stream of the target to be detected, performing preliminary anomaly detection on the real-time data stream using an adaptive threshold determined by a trained Gaussian mixture model, and detecting abnormal data points, comprises: Acquire a real-time data stream of the target to be measured, construct an initial Gaussian mixture model based on the real-time data stream, train the initial Gaussian mixture model using an expectation-maximization algorithm, calculate the posterior probability that each data point belongs to each Gaussian component, and update the initial Gaussian mixture model parameters according to the posterior probability, repeatedly perform calculation operations and update operations until the parameter change of the Gaussian mixture model is less than a preset threshold, and obtain a trained Gaussian mixture model; Based on the trained Gaussian mixture model, a likelihood score of each data point is calculated, and based on the likelihood score, existing historical data, and the real-time data stream, an initial threshold is dynamically adjusted to obtain an adaptive threshold, wherein the initial threshold is formulated according to industry standards; Based on the adaptive threshold, each data point in the real-time data stream is analyzed, and if the likelihood score of the data point to be determined is lower than the adaptive threshold, the data point to be determined is marked as an abnormal data point.

[0009] Optionally, based on the high-level feature representation and the quantitative evaluation result of the abnormality degree, evaluating the conditional dependency relationship between the abnormal data points using a probabilistic graph model, constructing a target probabilistic dependency graph based on the conditional dependency relationship, and generating an abnormal data point identification result based on the target probabilistic dependency graph, includes: Based on the high-level feature representation and the quantitative evaluation results of the abnormality degree, a probabilistic graphical model is constructed using a Bayesian network, wherein the nodes in the probabilistic graphical model represent abnormal data points, and the edges represent conditional dependencies between the abnormal data points; Based on the probabilistic graph model, a variational inference method is used to calculate the conditional probability distribution of each node, and based on the conditional probability distribution, the conditional dependency relationship between nodes is evaluated in combination with mutual information and conditional mutual information to obtain a node conditional dependency result; Based on the node conditional dependency results, an initial probability dependency graph is constructed using graph embedding technology, and the initial probability dependency graph is optimized to obtain a target probability dependency graph, and based on the degree centrality, betweenness centrality, and closeness centrality of the nodes in the target probability dependency graph, key nodes in the target probability dependency graph are identified; An integrated learning method is used to comprehensively evaluate key nodes by combining the evaluation results of multiple models to obtain key node evaluation results. Based on the key node evaluation results, abnormal data point identification results are obtained. The multiple models include decision trees and gradient boosting machines.

[0010] Optionally, based on the node conditional dependency result, using graph embedding technology to construct an initial probability dependency graph, optimizing the initial probability dependency graph to obtain a target probability dependency graph, and identifying key nodes in the target probability dependency graph based on degree centrality, betweenness centrality, and closeness centrality of nodes in the target probability dependency graph, including: Based on the node conditional dependency results, the abnormal data points in the probability graph model are vectorized using graph embedding technology to generate a node vector representation of each node in a low-dimensional space, and an initial probability dependency graph is constructed based on the node vector representation; Based on the initial probability dependency graph, the node vector representation is iteratively optimized until a preset maximum number of iterations is reached to obtain an optimized node vector representation, and the initial probability dependency graph is optimized based on the optimized node vector representation to obtain a target probability dependency graph; Calculate the degree centrality, betweenness centrality, and closeness centrality of all nodes in the target probability dependency graph to obtain a comprehensive centrality score corresponding to each node; Nodes whose comprehensive centrality scores are higher than a preset threshold are selected from the target probability dependency graph, and nodes whose comprehensive centrality scores are higher than the preset threshold are taken as key nodes.

[0011] Optionally, based on the abnormal data points, a composite model of a deep belief network combined with a long short-term memory network is used to perform deep feature learning and abnormality verification to obtain high-level feature representation and quantitative evaluation results of the abnormality degree, including: Using the abnormal data points to perform unsupervised learning on multiple restricted Boltzmann machines to obtain a deep belief network constructed by the learned multiple restricted Boltzmann machines, and using the deep belief network to extract deep-level feature representations of the abnormal data points; Using the deep-level feature representation as an input of a preset long short-term memory network to obtain a composite model, and training the composite model using a supervised learning method to adjust weights and bias items of the composite model to obtain a trained composite model; Based on the trained composite model, forward propagation processing is performed on the abnormal data points to generate a high-level feature representation of each abnormal data point, and a quantitative evaluation result of the abnormal degree of each abnormal data point is calculated.

[0012] In a second aspect, an embodiment of the present invention provides a system for identifying and cleaning abnormal data, including: An acquisition module is used to acquire a real-time data stream of a target to be tested, and to perform preliminary anomaly detection on the real-time data stream using an adaptive threshold determined by a trained Gaussian mixture model to detect abnormal data points; A learning module is used to perform deep feature learning and anomaly verification based on the abnormal data points by using a composite model of a deep belief network combined with a long short-term memory network to obtain high-level feature representation and quantitative evaluation results of the degree of abnormality; A construction module is used to evaluate the conditional dependency relationship between the abnormal data points using a probabilistic graph model based on the high-level feature representation and the quantitative evaluation result of the abnormality degree, construct a target probabilistic dependency graph based on the conditional dependency relationship, and generate an abnormal data point identification result based on the target probabilistic dependency graph; An adjustment module is used to optimize and adjust the existing cleaning strategy based on the abnormal data point identification result by applying a defined reward mechanism incentive algorithm to obtain an optimized cleaning strategy; The identification module is used to identify the abnormal area to be determined in the evaluation process of the probability graph model, introduce multimodal fusion technology to perform correlation analysis on the abnormal area to be determined, and obtain the correlation analysis result; based on the correlation analysis result, perform abnormal data judgment and abnormal data cleaning on the data in the abnormal area to be determined, and obtain the abnormal data processing result to be determined; based on the abnormal data processing result to be determined, adjust the cleaning strategy in the optimization to obtain the target cleaning strategy.

[0013] In a third aspect, an embodiment of the present invention provides a computing device, comprising a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute any method for identifying and cleaning abnormal data as described in the first aspect.

[0014] In a fourth aspect, an embodiment of the present invention provides a computer storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement a method for identifying and cleaning abnormal data as described in any one of the first aspects.

[0015] In an embodiment of the present invention, a real-time data stream of a target to be detected is obtained, and a preliminary anomaly detection is performed on the real-time data stream using an adaptive threshold determined by a trained Gaussian mixture model to detect abnormal data points; based on the abnormal data points, a composite model of a deep belief network combined with a long short-term memory network is used to perform deep-level feature learning and anomaly verification, and a high-level feature representation and anomaly degree quantitative evaluation result are obtained; based on the high-level feature representation and the anomaly degree quantitative evaluation result, a probability graph model is used to evaluate the conditional dependency relationship between the abnormal data points, a target probability dependency graph is constructed based on the conditional dependency relationship, and an abnormal data point identification result is generated based on the target probability dependency graph; based on the abnormal data point identification result, an existing cleaning strategy is optimized and adjusted by applying a defined reward mechanism incentive algorithm to obtain an optimized cleaning strategy; an abnormal area to be determined in the evaluation process of the probability graph model is identified, a multimodal fusion technology is introduced to perform correlation analysis on the abnormal area to be determined, and a correlation analysis result is obtained; based on the correlation analysis result, abnormal data judgment and abnormal data cleaning are performed on the data in the abnormal area to be determined, and an abnormal data processing result to be determined is obtained; based on the abnormal data processing result to be determined, the optimized cleaning strategy is adjusted to obtain a target cleaning strategy. The technical solution provided by the present invention improves the efficiency and accuracy of abnormal data identification and cleaning, and at the same time has strong adaptability and scalability, and is suitable for various complex data environments, which is of great significance for improving data quality and ensuring the accuracy of business decisions. Among them, through the composite model of the Gaussian mixture model with adaptive thresholds and the deep belief network combined with the long short-term memory network, abnormal data points can be identified more accurately; the conditional dependency relationship between abnormal data points is evaluated by the probability graph model, and potential associations can be found to improve the robustness of the system; through multimodal fusion technology and mutual information methods, the correlation between data points can be more comprehensively analyzed, potential abnormal patterns can be found, and the robustness of the system can be improved; by evaluating and adjusting the cleaning strategy in the optimization, the cleaning effect can be gradually improved, the false alarm rate and the missed alarm rate can be reduced, and the efficiency of abnormal data processing can be improved.

[0016] These and other aspects of the present invention will become more apparent from the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1A flowchart of a method for identifying and cleaning abnormal data provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of an abnormal data identification and cleaning system provided by an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a computing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0020] In some of the processes described in the specification and claims of the present invention and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0022] Figure 1 A flowchart of a method for identifying and cleaning abnormal data is provided for an embodiment of the present invention, such as Figure 1 As shown, the method includes: In today's big data era, the quality of data directly affects the accuracy of decision-making and the effectiveness of business. However, due to various reasons, such as sensor failure, human error or malicious attack, data sets are often mixed with abnormal data points. If these abnormal data are not identified and cleaned, they may cause distortion of analysis results, thereby affecting key decisions based on the data. In order to solve the problems existing in traditional anomaly detection methods and realize accurate identification and effective cleaning of abnormal data in real-time data streams, based on this, the present invention provides a method for identifying and cleaning abnormal data, such as Figure 1 ,include: Step 101: obtaining a real-time data stream of a target to be tested, and performing preliminary anomaly detection on the real-time data stream using an adaptive threshold determined by a trained Gaussian mixture model to detect abnormal data points; In this step, the Gaussian mixture model (GMM) of the adaptive threshold refers to a probabilistic model used to detect anomalies in the data, and the adaptive threshold is dynamically adjusted according to historical data to improve the detection accuracy.

[0023] This step obtains real-time data streams from the target to be tested. For example, data is collected from e-commerce platforms to obtain sales data of business opportunities, including sales volume, comments and other information; the obtained real-time data streams are preprocessed, including data type conversion, missing value filling, outlier detection, duplicate data identification and removal, and data error correction operations; the data type conversion operation converts different types of data into a unified format, such as converting a date string into a date type; the missing value filling operation uses mathematical modeling to fill in missing values, and calculates the filling values ​​calculated based on the mean, median and neighborhood methods respectively. The three filling values ​​are weighted averaged according to a pre-set proportional coefficient to obtain the final filling value, and the final filling value is added to the real-time data stream to ensure the integrity of the data; the outlier detection operation uses the Z score method to detect outliers in the data; a hash function is used to hash each piece of data to generate a unique hash value; the hash values ​​are compared to identify duplicates. Data records; choose to keep or delete duplicate records according to business logic; duplicate data identification and removal operations use hash functions to hash each piece of data to generate a unique hash value; compare hash values ​​to identify duplicate data records; choose to keep or delete duplicate records according to business logic; data error correction operations use regular expressions and pattern matching techniques to identify format errors and spelling errors in data; use machine learning algorithms to automatically identify inconsistencies in data; based on the identification results, apply preset correction rules to correct the data; initialize the Gaussian mixture model and use the expectation maximization (EM) algorithm to train the model. Based on the trained Gaussian mixture model, calculate the likelihood score of each data point, set the initial threshold based on industry standards, and regularly update the threshold based on the likelihood score, historical data, and real-time data streams. Compare the likelihood score of each data point with the dynamically adjusted threshold, and mark the data point below the threshold as an abnormal data point. For example, if a user's click frequency suddenly increases and the click pattern does not match normal user behavior, its likelihood score is lower than the dynamically adjusted threshold, and it is marked as an abnormal data point.

[0024] More specifically, the embodiment of the present invention provides a calculation formula for the final fill value in the preprocessing operation, and the specific calculation formula is as follows: ; in, represents the final missing value; represents the weighted average coefficient of the mean; represents the first filling value calculated based on the mean method; represents the weighted average coefficient of the median; represents the second filling value calculated based on the median method; represents the weighted average coefficient of the initial filling value; Represents the third fill value calculated based on the neighborhood method.

[0025] The final filling value comprehensively considers the first filling value calculated based on the mean method, the second filling value calculated based on the median method, and the third filling value calculated based on the neighborhood method. It can effectively fill the real-time data stream and improve data integrity and accuracy.

[0026] The calculation formula of the first filling value is as follows: ; in, Represents non-missing values ​​in the original data series that contain missing values; Indicates non-missing values; is the number of non-missing values; Indicates the first fill value calculated based on the mean method.

[0027] The calculation formula of the second filling value is as follows: ; in, represents the second filling value calculated based on the median method; A data series representing non-missing values; Represents the median of the series.

[0028] The calculation formula of the third filling value is as follows: ; in, represents the third filling value calculated based on the neighborhood method; Represents data points in the neighborhood of non-missing value data series; Indicates data points; Represents the weight of the corresponding data point calculated based on similarity; Represents the number of non-missing values.

[0029] The embodiment of the present invention also provides a Z score formula, and the specific calculation formula is as follows: ; in, express The score represents the number of standard deviations from the mean. is a data point; is the mean of the data; is the standard deviation of the data.

[0030] For example, when the data The absolute value of the calculated Z quantile is greater than the preset threshold, such as , then it is believed that is an outlier.

[0031] Step 102: Based on the abnormal data points, a composite model of a deep belief network combined with a long short-term memory network is used to perform deep feature learning and abnormality verification to obtain high-level feature representation and quantitative evaluation results of the abnormality degree; In this step, the deep belief network (DBN) refers to an unsupervised learning neural network model used to extract deep feature representations of data. The long short-term memory network (LSTM) refers to a variant of the recurrent neural network (RNN) that is suitable for processing time series data and can capture long-term dependencies.

[0032] This step combines the deep belief network and the long short-term memory network to build a composite model, perform deep feature learning and anomaly verification on abnormal data points, and obtain high-level feature representation and quantitative evaluation results of the degree of anomaly.

[0033] Step 103: Based on the high-level feature representation and the quantitative evaluation result of the abnormality degree, a probabilistic graph model is used to evaluate the conditional dependency relationship between the abnormal data points, a target probabilistic dependency graph is constructed based on the conditional dependency relationship, and an abnormal data point recognition result is generated based on the target probabilistic dependency graph; In this step, the probabilistic graphical model refers to a probabilistic model that uses a graph structure to represent the conditional dependency between random variables. The conditional dependency refers to the dependency between nodes in the probabilistic graphical model, indicating how the state of one node affects the state of another node.

[0034] This step uses a Bayesian network to build a probabilistic graph model, where nodes represent abnormal data points and edges represent conditional dependencies between abnormal data points. Based on this probabilistic graph model, the variational inference method is used to calculate the conditional probability distribution of each node, and the conditional dependencies of the nodes are evaluated by combining mutual information and conditional mutual information. Based on the conditional dependencies, the initial probabilistic dependency graph is constructed using graph embedding technology, and optimized to obtain the target probabilistic dependency graph. The degree centrality, betweenness centrality, and closeness centrality of the nodes in the target probabilistic dependency graph are calculated, and key nodes are identified based on the calculation results. For example, assuming that there is a certain correlation between the abnormal click behaviors of multiple users, the probabilistic graph model can be used to discover the conditional dependencies between these users and identify the key nodes therein, such as a user who may be the source of malicious clicks.

[0035] Step 104: Based on the abnormal data point identification result, a defined reward mechanism incentive algorithm is applied to optimize and adjust the existing cleaning strategy to obtain an optimized cleaning strategy; In this step, the reward mechanism incentive algorithm is defined, which refers to a reinforcement learning algorithm that optimizes and adjusts the cleaning strategy by defining a reward mechanism.

[0036] This step defines a reward function based on the abnormal data point identification results to motivate the algorithm to optimize and adjust the existing cleaning strategy to obtain the cleaning strategy being optimized. Existing cleaning strategies include reducing false alarm rates and improving anomaly detection accuracy. For example, assuming that the current cleaning strategy has a high false alarm rate, by defining a reward mechanism and applying a reinforcement learning algorithm, the cleaning strategy can be gradually optimized to reduce the false alarm rate and improve the accuracy of anomaly detection.

[0037] Step 105: Identify the abnormal area to be determined in the evaluation process of the probability graph model, introduce multimodal fusion technology to perform correlation analysis on the abnormal area to be determined, obtain correlation analysis results, and based on the correlation analysis results, perform abnormal data judgment and abnormal data cleaning on the data in the abnormal area to be determined to obtain abnormal data processing results to be determined, and adjust the cleaning strategy in the optimization based on the abnormal data processing results to obtain the target cleaning strategy; In this step, multimodal fusion technology refers to the technology of combining multiple different types of data or information sources for comprehensive analysis. Correlation analysis refers to analyzing the degree of association between different data points or features.

[0038] For example, suppose a user's click behavior also shows abnormalities in image and text data. By analyzing the correlation of these data through multimodal fusion technology, we can more comprehensively judge whether the user's behavior is abnormal and perform corresponding data cleaning.

[0039] The embodiments of the present invention can achieve the following beneficial effects through the above steps: The composite model of Gaussian mixture model with adaptive threshold and deep belief network combined with long short-term memory network can more accurately identify abnormal data points; Using probabilistic graphical models to evaluate the conditional dependencies between abnormal data points can discover potential associations and improve the robustness of the system; By defining a reward mechanism to motivate the algorithm to optimize and adjust the existing cleaning strategy, the false alarm rate can be effectively reduced and the accuracy of anomaly detection can be improved; The introduction of multimodal fusion technology can analyze data more comprehensively and improve the effectiveness of abnormal data judgment and cleaning; By timely discovering and processing abnormal data, the accuracy of the e-commerce data obtained can be guaranteed and the analysis effect can be improved.

[0040] In modern data processing processes, accurate identification and effective cleaning of abnormal data are key steps to ensure the reliability of data analysis results. Although existing methods can detect and process abnormal data to a certain extent, they often have limitations when facing complex data environments, especially when dealing with abnormal areas that are difficult to directly judge. In order to overcome these challenges, based on this, the present invention provides a specific embodiment, in which step 105 identifies the abnormal area to be determined in the evaluation process of the probability graph model, introduces multimodal fusion technology to perform correlation analysis on the abnormal area to be determined, and obtains the correlation analysis result. Based on the correlation analysis result, the data in the abnormal area to be determined is judged and the abnormal data is cleaned to obtain the abnormal data processing result to be determined. Based on the abnormal data processing result to be determined, the cleaning strategy in the optimization is adjusted to obtain the target cleaning strategy, which specifically includes the following steps: Step 501: Analyze the nodes and edges in the target probability dependency graph, identify the target nodes and target edges whose conditional dependencies are higher than a preset threshold, and determine the abnormal area to be determined based on the target nodes and the target edges; This step analyzes the nodes and edges in the target probability dependency graph, that is, the abnormal data points and conditional dependencies; based on business needs and historical data analysis, a preset threshold is set to determine the importance of conditional dependencies; target nodes and target edges whose conditional dependencies are higher than the preset threshold are identified. Based on the target nodes and target edges, the abnormal areas to be determined are determined, which may contain multiple interrelated abnormal data points.

[0041] Step 502: Collect multimodal data corresponding to each data point in the abnormal region to be determined, fuse the multimodal data using multimodal fusion technology to generate a comprehensive feature representation, and calculate the correlation between each data point in the abnormal region to be determined using a mutual information method based on the comprehensive feature representation to obtain a correlation analysis result; In this step, the mutual information method refers to a statistical method that measures the correlation between two random variables and is used to quantify the amount of information they share.

[0042] This step collects the multimodal data corresponding to each data point in the abnormal area to be determined, fuses the multimodal data using multimodal fusion technology, generates a comprehensive feature representation, and uses the mutual information method to calculate the correlation between each data point in the abnormal area to be determined based on the comprehensive feature representation; through the mutual information method, the correlation analysis results between each data point are obtained, which helps to further understand the correlation between the data points.

[0043] Step 503: Based on the correlation analysis result, abnormal data judgment and cleaning operations are performed on the data points in the abnormal area to be determined to obtain abnormal data processing results to be determined, and based on the abnormal data processing results to be determined, the cleaning strategy in the optimization is evaluated and adjusted to generate a target cleaning strategy; This step is based on the results of correlation analysis to determine the abnormal data points in the identified abnormal area. If the correlation analysis results of a data point deviate significantly from the normal range, it will be marked as an abnormal data point; the identified abnormal data points are cleaned, and the cleaned data is standardized to ensure the consistency of data format, unit and coding. Cleaning operations include deleting abnormal data, correcting abnormal values ​​or marking abnormal data. Deleting abnormal data includes determining the data points that need to be deleted based on the results of correlation analysis, ensuring that the original data is backed up before deleting the data to prevent accidental deletion, deleting abnormal data points from the database or data set, and recording the time, reason and executor of the deletion operation for subsequent auditing and tracking. Correcting outliers includes determining the outliers that need to be corrected based on the results of correlation analysis, using the most recent historical data for correction or using the mean, median or other statistics for correction, and can also be corrected according to business logic and rules; marking abnormal data includes determining the data points that need to be marked based on the results of correlation analysis, adding a marking field such as is-suspicious in the database or data set, marking the abnormal data points as suspicious or abnormal, notifying the relevant team or personnel for further review through system notifications or emails, and recording the time, reason and executor of the marking operation. By performing cleaning operations on these abnormal data points, the stability of the system is improved. According to the abnormal data judgment and data cleaning results, the cleaning strategy in the optimization is evaluated and adjusted to further improve the effect of anomaly detection and cleaning, and finally generate a target cleaning strategy to ensure that the system can handle abnormal data more effectively. For example, after the cleaning operation, the false alarm rate of the system has been reduced, but some abnormal data has not been completely removed. By evaluating and adjusting the cleaning strategy, more rules and algorithm optimization are introduced, and finally a more effective target cleaning strategy is generated.

[0044] The embodiments of the present invention can achieve the following beneficial effects through the above steps: By analyzing the nodes and edges in the target probability dependency graph, key abnormal data points and their dependencies can be more accurately identified, thereby improving the accuracy of anomaly detection; Through multimodal fusion technology and mutual information methods, the correlation between data points can be analyzed more comprehensively, potential abnormal patterns can be discovered, and the robustness of the system can be improved; By evaluating and adjusting the cleaning strategy in optimization, the cleaning effect can be gradually improved, the false positive rate and the false negative rate can be reduced, and the efficiency of abnormal data processing can be improved; By timely discovering and processing abnormal data, the accuracy of data acquisition on the e-commerce platform can be guaranteed and the effect of data application can be improved; Through the detailed abnormal data identification and cleaning process, it can provide strong data support for e-commerce data statistical analysis, helping them make more informed decisions.

[0045] In the modern data-driven decision-making process, the accuracy and reliability of real-time data streams are crucial. The existence of abnormal data will not only interfere with normal business processes, but may also mislead the data analysis results and affect the final decision quality. In order to ensure the high quality of data, timely detection and processing of abnormal data has become a key task. Traditional abnormal data detection methods often rely on fixed thresholds and assumed data distributions, which are inadequate when faced with dynamically changing data streams. Based on this, the present invention provides a specific embodiment, wherein step 101 obtains the real-time data stream of the target to be tested, and uses the adaptive threshold determined by the trained Gaussian mixture model to perform preliminary anomaly detection on the real-time data stream to detect abnormal data points, specifically including the following steps: Step 111: obtaining a real-time data stream of the target to be measured, constructing an initial Gaussian mixture model based on the real-time data stream, training the initial Gaussian mixture model using an expectation-maximization algorithm, calculating the posterior probability that each data point belongs to each Gaussian component, and updating the initial Gaussian mixture model parameters according to the posterior probability, repeatedly performing calculation operations and update operations until the parameter change of the Gaussian mixture model is less than a preset threshold, thereby obtaining a trained Gaussian mixture model; In this step, real-time data stream refers to the data continuously obtained from the data source collected by the e-commerce platform. The expectation maximization algorithm (EM algorithm) refers to an iterative algorithm for estimating the parameters of a probability model with hidden variables. The posterior probability refers to the conditional probability of an event occurring given the observed data.

[0046] This step obtains real-time data streams from data sources such as transaction records of e-commerce platforms; constructs an initial Gaussian mixture model based on the real-time data stream, and obtains a trained Gaussian mixture model by repeatedly executing the expectation step and the maximization step until the change in the parameters of the initial Gaussian mixture model is less than a preset threshold, such as 0.001. The expectation step includes calculating the posterior probability that each data point belongs to each Gaussian component, and the maximization step includes updating the parameters of the initial Gaussian mixture model according to the posterior probability. For example, 1,000 data points were extracted from user click stream data, and a Gaussian mixture model containing 3 Gaussian components was initialized. The model parameters are continuously updated through the expectation maximization algorithm until the parameter change is less than 0.001, and finally a trained Gaussian mixture model is obtained.

[0047] Step 112: Calculate the likelihood score of each data point based on the trained Gaussian mixture model, and dynamically adjust the initial threshold based on the likelihood score, existing historical data, and the real-time data stream to obtain an adaptive threshold, wherein the initial threshold is formulated according to industry standards; In this step, the likelihood score refers to the probability density value of the data point under a given model. The adaptive threshold refers to the threshold that is dynamically adjusted based on historical data and real-time data to distinguish normal data points from abnormal data points.

[0048] This step calculates the likelihood score for each data point based on the trained Gaussian mixture model; sets the initial threshold based on industry standards or experience. For example, the initial threshold is set based on the statistical characteristics of historical data, such as mean and standard deviation; the initial threshold is dynamically adjusted based on the likelihood score, existing historical data, and real-time data stream to obtain an adaptive threshold. For example, the likelihood scores of 1,000 data points are calculated, and the initial threshold is set to 0.1 based on the statistical characteristics of historical data. Through dynamic adjustment, the adaptive threshold is 0.08 Step 113: Analyze each data point in the real-time data stream based on the adaptive threshold, and if the likelihood score of the data point to be determined is lower than the adaptive threshold, mark the data point to be determined as an abnormal data point; This step analyzes each data point in the real-time data stream based on the adaptive threshold. If the likelihood score of the data point to be determined is lower than the adaptive threshold, it is marked as an abnormal data point. For example, suppose a user's click frequency suddenly increases, and the click pattern does not match normal user behavior. Its likelihood score is 0.05, which is lower than the adaptive threshold of 0.08, so it is marked as an abnormal data point.

[0049] The embodiments of the present invention can achieve the following beneficial effects through the above steps: Through the Gaussian mixture model and expectation maximization algorithm, the data distribution can be fitted more accurately, thereby improving the accuracy of anomaly detection; The method of dynamically adjusting the threshold can adapt to data changes, reduce false positives and false negatives, and improve the robustness of the system; Timely discovery and processing of abnormal data can reduce unnecessary resource consumption and improve system operation efficiency; By timely discovering and processing abnormal data, the stable operation of the e-commerce platform can be guaranteed, the user experience can be improved, and system failures and user complaints caused by abnormal data can be reduced; In modern data analysis, accurately identifying and processing abnormal data is an important part of ensuring data quality and decision-making accuracy. Through preliminary anomaly detection, we have been able to identify potential abnormal data points from real-time data streams. However, in order to have a deeper understanding of the relationship between these abnormal data points and further improve the accuracy of anomaly detection, more advanced methods are needed to evaluate the dependencies between them. Based on this, the present invention provides a specific embodiment, in which step 103, based on the high-level feature representation and the quantitative evaluation results of the degree of abnormality, a probabilistic graphical model is used to evaluate the conditional dependencies between the abnormal data points, a target probabilistic dependency map is constructed based on the conditional dependencies, and an abnormal data point identification result is generated based on the target probabilistic dependency map, specifically including the following steps: Step 301: Based on the high-level feature representation and the quantitative evaluation result of the abnormality degree, a probabilistic graphical model is constructed using a Bayesian network, wherein the nodes in the probabilistic graphical model represent abnormal data points, and the edges represent the conditional dependency relationship between the abnormal data points; In this step, high-level feature representation refers to the high-level features of the data points extracted by the composite model.

[0050] This step builds a Bayesian network based on the high-level feature representation and the quantitative evaluation results of the abnormality degree. Each node represents an abnormal data point, and each edge represents the conditional dependency between two abnormal data points. For example, high-level feature representation is extracted from user click stream data, and the abnormality degree score of each user is calculated. A Bayesian network is constructed, where nodes represent users and edges represent the click behavior dependency between users.

[0051] Step 302: Based on the probabilistic graphical model, a variational inference method is used to calculate the conditional probability distribution of each node, and based on the conditional probability distribution, the conditional dependency relationship between nodes is evaluated in combination with mutual information and conditional mutual information to obtain a node conditional dependency result; In this step, variational inference refers to a method for approximating a complex probability distribution by optimizing a simple distribution to approximate the target distribution. Mutual information refers to a measure of the correlation between two random variables. Conditional mutual information refers to a measure of the correlation between two random variables given a third variable.

[0052] This step is based on the constructed Bayesian network and uses the variational inference method to calculate the conditional probability distribution of each node; the conditional dependency between nodes is evaluated by combining mutual information and conditional mutual information to obtain the node conditional dependency result.

[0053] Step 303: Based on the node conditional dependency results, an initial probability dependency graph is constructed using graph embedding technology, and the initial probability dependency graph is optimized to obtain a target probability dependency graph, and key nodes in the target probability dependency graph are identified based on the degree centrality, betweenness centrality, and closeness centrality of the nodes in the target probability dependency graph; In this step, graph embedding technology refers to the technology of mapping nodes in the graph structure to a low-dimensional vector space. Degree centrality refers to the number of direct connections between nodes. Betweenness centrality refers to the frequency of a node as an intermediate point on the shortest path. Closeness centrality refers to the average distance between a node and all other nodes.

[0054] This step uses graph embedding techniques such as Node2Vec and DeepWalk to build an initial probability dependency graph based on the results of node conditional dependencies; optimizes the initial probability dependency graph, such as by minimizing the loss function and gradient descent, to obtain the target probability dependency graph; identifies key nodes based on the degree centrality, betweenness centrality, and closeness centrality of the nodes in the target probability dependency graph. For example, based on the optimized graph, it is analyzed that user A has higher degree centrality, betweenness centrality, and closeness centrality, and is marked as a key node.

[0055] Step 304: using an ensemble learning method to comprehensively evaluate the key nodes by combining the evaluation results of multiple models to obtain key node evaluation results, and obtaining abnormal data point identification results based on the key node evaluation results, wherein the multiple models include a decision tree and a gradient boosting machine; In this step, ensemble learning methods refer to methods that improve overall performance by combining the predictions of multiple models. Decision trees refer to a classification or regression model based on feature segmentation. Gradient boosting machines (GBMs) refer to models that optimize predictions by iteratively adding weak learners.

[0056] This step uses models such as decision trees and gradient boosting machines to train key nodes. Each model generates its own prediction results. Through ensemble learning methods such as voting and stacking, the prediction results of multiple models are combined to obtain key node evaluation results. Based on the key node evaluation results, the abnormal data point identification results are finally obtained. For example, using decision trees and gradient boosting machines to train key nodes, each model generates its own abnormal detection results. Through ensemble learning methods, these results are combined to finally determine that user A is an abnormal data point.

[0057] The embodiments of the present invention can achieve the following beneficial effects through the above steps: Through Bayesian networks and variational inference methods, the conditional dependencies between abnormal data points can be captured more accurately, thereby improving the accuracy of anomaly detection; Through graph embedding technology and optimization algorithms, the relationship between data points can be analyzed more comprehensively, potential abnormal patterns can be discovered, and the robustness of the system can be improved; Timely discovery and processing of abnormal data can reduce unnecessary resource consumption and improve system operation efficiency; In the process of identifying abnormal data, accurately understanding the conditional dependency between data points is crucial to improving detection accuracy. The present invention provides a specific embodiment, in which step 303 uses graph embedding technology to construct an initial probability dependency graph based on the node conditional dependency results, and optimizes the initial probability dependency graph to obtain a target probability dependency graph, and identifies key nodes in the target probability dependency graph based on the degree centrality, betweenness centrality, and closeness centrality of the nodes in the target probability dependency graph. The specific steps include: Step 311: Based on the node conditional dependency result, the abnormal data points in the probability graph model are vectorized using graph embedding technology to generate a node vector representation of each node in a low-dimensional space, and an initial probability dependency graph is constructed based on the node vector representation; In this step, select a suitable graph embedding technology, such as Node2Vec, DeepWalk, or GraphSAGE; based on the node conditional dependency results, use graph embedding technology to vectorize the abnormal data points in the probabilistic graph model and generate a node vector representation of each node in a low-dimensional space. For example, use the Node2Vec algorithm to vectorize the abnormal data points, and user A, user B, and user C are represented as vectors vA, vB, and vC, respectively. Based on the generated node vector representation, construct the initial probabilistic dependency graph.

[0058] Step 312: Based on the initial probability dependency graph, the node vector representation is iteratively optimized until a preset maximum number of iterations is reached to obtain an optimized node vector representation, and the initial probability dependency graph is optimized based on the optimized node vector representation to obtain a target probability dependency graph; This step iteratively optimizes the node vector representation until the preset maximum number of iterations is reached to obtain the optimized node vector representation; during each iteration, the node vector representation is updated by minimizing the loss function, such as the reconstruction error; based on the optimized node vector representation, the initial probability dependency graph is optimized to obtain the target probability dependency graph. For example, the node vector representation is continuously updated through 100 iterations, and finally the optimized node vector representations vA′, vB′, and vC′ are obtained, and the target probability dependency graph is constructed based on these vector representations.

[0059] Step 313: Calculate the degree centrality, betweenness centrality, and closeness centrality of all nodes in the target probability dependency graph to obtain a comprehensive centrality score corresponding to each node; This step calculates the degree centrality, betweenness centrality, and closeness centrality of each node in the target probability dependency graph, and combines the degree centrality, betweenness centrality, and closeness centrality to calculate the comprehensive centrality score corresponding to each node.

[0060] More specifically, the present invention provides a calculation formula for a centrality comprehensive score to identify key nodes. The specific calculation formula is as follows: ; in, Representation Node The centrality composite score of Representation Node The degree centrality of Representation Node Betweenness centrality of ; Representation Node The closeness centrality; Represents the weight coefficient, which can be adjusted according to actual needs.

[0061] The centrality comprehensive score formula comprehensively considers degree centrality, betweenness centrality and closeness centrality, so as to improve the accuracy and efficiency of anomaly detection by comparing the centrality comprehensive scores, selecting key nodes and identifying abnormal data.

[0062] Among them, the node The calculation formula of degree centrality is as follows: ; in, Representation Node The degree centrality of Representation Node The degree of Represents the total number of nodes in the network.

[0063] The degree centrality formula is used to measure the number of other nodes directly connected to a node in the graph. In abnormal data identification, nodes with high degree centrality may indicate users or behaviors with frequent abnormal activities. For example, in the field of e-commerce, users who frequently click on multiple products but do not buy them may have high degree centrality. In general, the degree centrality formula is suitable for identifying active nodes or key players.

[0064] Among them, the node The calculation formula of betweenness centrality is as follows: ; in, Representation Node Betweenness centrality of ; Represents a slave node To Node Passing Node The number of shortest paths; Represents a slave node To Node The total number of shortest paths.

[0065] The betweenness centrality formula is used to measure the frequency of a node as an intermediate point on the shortest path. In abnormal data identification, nodes with high betweenness centrality may represent nodes that play an intermediary role in multiple abnormal behavior paths. In general, the betweenness centrality formula is suitable for identifying key nodes or bridge nodes for information transmission.

[0066] Among them, the node The calculation formula of closeness centrality is as follows: ; The closeness centrality formula is used to measure the average distance from a node to all other nodes. In the identification of abnormal data, nodes with high closeness centrality may represent those abnormal nodes with broad influence. For example, in the field of e-commerce, some users may manipulate product reviews through extensive social networks or comments, and these users tend to have high closeness centrality. In general, the closeness centrality formula is suitable for identifying nodes with broad influence.

[0067] Step 314: Selecting nodes whose centrality comprehensive scores are higher than a preset threshold from the target probability dependency graph, and taking the nodes whose centrality comprehensive scores are higher than the preset threshold as key nodes; In this step, a preset threshold is set based on business needs and historical data analysis to determine the importance of the node; nodes with a centrality comprehensive score higher than the preset threshold are selected from the target probability dependency graph as key nodes. For example, the preset threshold is set to 0.6, and by comparing each centrality comprehensive score, it is found that the score of 0.8 is higher than the preset threshold, so it is selected as a key node.

[0068] The embodiments of the present invention can achieve the following beneficial effects through the above steps: Through graph embedding technology and optimization algorithms, the conditional dependencies between abnormal data points can be captured more accurately, thereby improving the accuracy of anomaly detection; Through graph embedding technology and optimization algorithms, data can be analyzed more comprehensively to timely discover and process abnormal data, which can reduce unnecessary resource consumption and improve system operation efficiency; By timely discovering and processing abnormal data, the stable operation of the e-commerce platform can be guaranteed, the user experience can be improved, and system failures and user complaints caused by abnormal data can be reduced; Through a detailed abnormal data identification process, it can provide strong data support for the operation team of the e-commerce platform and help them make more informed decisions.

[0069] The present invention provides a specific embodiment, in which step 102, based on the abnormal data points, a composite model of a deep belief network combined with a long short-term memory network is used to perform deep feature learning and abnormality verification to obtain a high-level feature representation and a quantitative evaluation result of the abnormality degree, and the specific steps include: Step 211: using the abnormal data points to perform unsupervised learning on multiple restricted Boltzmann machines to obtain a deep belief network constructed by the learned multiple restricted Boltzmann machines, and using the deep belief network to extract deep-level feature representations of the abnormal data points; In this step, Restricted Boltzmann Machine (RBM) refers to a generative unsupervised learning model used to learn feature representations from input data.

[0070] This step initializes multiple restricted Boltzmann machines, each of which contains a visible layer and a hidden layer; uses abnormal data points to perform unsupervised learning on each restricted Boltzmann machine, and updates the weights and bias terms of the restricted Boltzmann machine through the Contrastive Divergence (CD) algorithm so that it can learn the feature representation of the data. Multiple learned restricted Boltzmann machines are stacked to form a deep belief network (DBN), and the output of each layer of restricted Boltzmann machines in the deep belief network is used as the input of the next layer of restricted Boltzmann machines. For example, abnormal click stream data is extracted from the log data of an e-commerce platform. Use this data to perform unsupervised learning on three restricted Boltzmann machines, and each restricted Boltzmann machine learns different feature representations. Then stack these three restricted Boltzmann machines to form a three-layer deep belief network. The abnormal data points are input into the deep belief network, and the deep feature representation of the data is extracted layer by layer through the forward propagation process. The top-level output of the deep belief network is the deep feature representation of the data point. For example, the abnormal click stream data is input into the deep belief network, and after forward propagation, a deep feature representation of each data point is obtained. For example, the click behavior of user A is converted into a high-level feature vector.

[0071] Step 212: using the deep feature representation as an input of a preset long short-term memory network to obtain a composite model, and training the composite model using a supervised learning method to adjust the weights and bias items of the composite model to obtain a trained composite model; This step initializes a preset long short-term memory network (LSTM), takes the output of the deep belief network as the input of the long short-term memory network, and forms a composite model; uses labeled abnormal data points to perform supervised learning on the composite model, adjusts the weights and bias items of the composite model through the back propagation algorithm, optimizes the model performance, and obtains the trained composite model.

[0072] Step 213: Based on the trained composite model, forward propagation processing is performed on the abnormal data points to generate a high-level feature representation of each abnormal data point, and a quantitative evaluation result of the abnormal degree of each abnormal data point is calculated; In this step, the abnormal data points are input into the trained composite model. Through the forward propagation process, a high-level feature representation of each data point is generated. Based on the high-level feature representation, the quantitative evaluation result of the abnormal degree of each data point is calculated. A classifier, such as the Sigmoid function, can be used to output the abnormal probability, or a regression method can be used to output the abnormal score.

[0073] The embodiments of the present invention can achieve the following beneficial effects through the above steps: By combining the deep belief network and the long short-term memory network, the deep feature representation of the data can be extracted more accurately, thereby improving the accuracy of anomaly detection; Deep belief networks can capture the complex structure of data, and long short-term memory networks can process time series data. The combination of the two can better cope with different types of abnormal patterns and improve the robustness of the system.

[0074] Figure 2 A schematic diagram of a big data processing system for implementing hybrid data analysis is provided for an embodiment of the present invention. Figure 2 As shown, the system includes: An acquisition module 21 is used to acquire a real-time data stream of a target to be detected, and perform preliminary anomaly detection on the real-time data stream using an adaptive threshold determined by a trained Gaussian mixture model to detect abnormal data points; A learning module 22 is used to perform deep feature learning and anomaly verification based on the abnormal data points using a composite model of a deep belief network combined with a long short-term memory network to obtain high-level feature representation and quantitative evaluation results of the degree of abnormality; A construction module 23 is used to evaluate the conditional dependency relationship between the abnormal data points using a probabilistic graph model based on the high-level feature representation and the quantitative evaluation result of the abnormality degree, construct a target probabilistic dependency graph based on the conditional dependency relationship, and generate an abnormal data point identification result based on the target probabilistic dependency graph; An adjustment module 24 is used to optimize and adjust the existing cleaning strategy based on the abnormal data point identification result by applying a defined reward mechanism incentive algorithm to obtain an optimized cleaning strategy; The identification module 25 is used to identify the abnormal area to be determined in the evaluation process of the probability graph model, introduce multimodal fusion technology to perform correlation analysis on the abnormal area to be determined, and obtain the correlation analysis result. Based on the correlation analysis result, the data in the abnormal area to be determined is judged as abnormal data and the abnormal data is cleaned to obtain the processing result of the abnormal data to be determined. Based on the processing result of the abnormal data to be determined, the cleaning strategy in the optimization is adjusted to obtain the target cleaning strategy.

[0075] Figure 2 The abnormal data identification and cleaning system can perform Figure 1 The implementation principle and technical effect of the method for identifying and cleaning abnormal data described in the embodiment shown are not repeated here. The specific way in which each module and unit performs operations in the system for identifying and cleaning abnormal data in the above embodiment has been described in detail in the embodiment of the method, and will not be described in detail here.

[0076] In one possible design, Figure 2An abnormal data identification and cleaning system of the illustrated embodiment can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32; The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32 .

[0077] The processing component 32 is used to: obtain the real-time data stream of the target to be tested, perform preliminary anomaly detection on the real-time data stream using the adaptive threshold determined by the trained Gaussian mixture model, and detect abnormal data points; based on the abnormal data points, use a composite model of a deep belief network combined with a long short-term memory network to perform deep-level feature learning and anomaly verification, and obtain high-level feature representation and quantitative evaluation results of the degree of abnormality; based on the high-level feature representation and the quantitative evaluation results of the degree of abnormality, use a probability graph model to evaluate the conditional dependency relationship between the abnormal data points, build a target probability dependency map based on the conditional dependency relationship, and generate abnormal data point identification results based on the target probability dependency map; based on the abnormal data point identification results, apply a defined reward mechanism incentive algorithm to optimize and adjust the existing cleaning strategy to obtain an optimized cleaning strategy; identify the abnormal area to be determined in the evaluation process of the probability graph model, introduce a multimodal fusion technology to perform correlation analysis on the abnormal area to be determined, and obtain a correlation analysis result; based on the correlation analysis result, perform abnormal data judgment and abnormal data cleaning on the data in the abnormal area to be determined, and obtain a processing result of the abnormal data to be determined; based on the processing result of the abnormal data to be determined, adjust the optimized cleaning strategy to obtain a target cleaning strategy.

[0078] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0079] The storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0080] Of course, the computing device may also include other components, such as input / output interfaces, display components, communication components, etc.

[0081] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc.

[0082] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0083] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0084] The embodiment of the present invention further provides a computer storage medium storing a computer program, which can achieve the above-mentioned Figure 1 A method for identifying and cleaning abnormal data in the illustrated embodiment.

[0085] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0086] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0087] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying and cleaning abnormal data, characterized in that: include: Acquire a real-time data stream of the target to be tested, perform preliminary anomaly detection on the real-time data stream using an adaptive threshold determined by a trained Gaussian mixture model, and detect abnormal data points; Based on the abnormal data points, a composite model of a deep belief network combined with a long short-term memory network is used to perform deep feature learning and abnormality verification, thereby obtaining high-level feature representation and quantitative evaluation results of the degree of abnormality; Based on the high-level feature representation and the quantitative evaluation result of the abnormality degree, a probabilistic graph model is used to evaluate the conditional dependency relationship between the abnormal data points, a target probabilistic dependency graph is constructed based on the conditional dependency relationship, and an abnormal data point identification result is generated based on the target probabilistic dependency graph; Based on the abnormal data point identification result, a defined reward mechanism incentive algorithm is applied to optimize and adjust the existing cleaning strategy to obtain an optimized cleaning strategy; Identify the abnormal area to be determined in the evaluation process of the probability graph model, introduce multimodal fusion technology to perform correlation analysis on the abnormal area to be determined, and obtain the correlation analysis result; based on the correlation analysis result, perform abnormal data judgment and abnormal data cleaning on the data in the abnormal area to be determined, and obtain the abnormal data processing result to be determined; based on the abnormal data processing result to be determined, adjust the cleaning strategy in the optimization to obtain the target cleaning strategy.

2. The method according to claim 1, characterized in that The method of identifying the abnormal region to be determined in the evaluation process of the probability graph model, introducing the multimodal fusion technology to perform correlation analysis on the abnormal region to be determined, obtaining the correlation analysis result, and based on the correlation analysis result, performing abnormal data judgment and abnormal data cleaning on the data in the abnormal region to be determined to obtain the abnormal data processing result to be determined, and adjusting the cleaning strategy in the optimization based on the abnormal data processing result to obtain the target cleaning strategy, including: Analyze the nodes and edges in the target probability dependency graph, identify the target nodes and target edges whose conditional dependencies are higher than a preset threshold, and determine the abnormal area to be determined based on the target nodes and the target edges; Collecting multimodal data corresponding to each data point in the abnormal area to be determined, fusing the multimodal data using multimodal fusion technology to generate a comprehensive feature representation, and based on the comprehensive feature representation, using a mutual information method to calculate the correlation between each data point in the abnormal area to be determined, to obtain a correlation analysis result; Based on the correlation analysis result, abnormal data judgment and cleaning operations are performed on the data points in the abnormal area to be determined to obtain the abnormal data processing result to be determined, and based on the abnormal data processing result to be determined, the cleaning strategy in the optimization is evaluated and adjusted to generate a target cleaning strategy.

3. The method according to claim 2, characterized in that The multimodal data are fused by using multimodal fusion technology to generate a comprehensive feature representation, and based on the comprehensive feature representation, the correlation between each data point in the abnormal area to be determined is calculated using a mutual information method to obtain a correlation analysis result, including: Based on the multimodal data, a convolutional neural network is used to extract image features, and a recurrent neural network is used to extract time series features; Constructing a multi-layer perception structure, based on the multi-layer perception structure, fusing the image features and the time series features to generate a preliminary comprehensive feature representation, selecting a loss function based on the preliminary comprehensive feature representation, and applying a back propagation algorithm to minimize the value of the loss function to obtain a minimized loss function, using the minimized loss function to train the multi-layer perception structure to perform iterative optimization until a stop condition is met to obtain a multi-layer perception machine model, wherein the stop condition is reaching a preset threshold of the loss function, and based on the multi-layer perception machine model, taking the image features and the time series features as inputs, and outputting a comprehensive feature representation; Based on the comprehensive feature representation, a mutual information method is used to calculate the mutual information value of each pair of data points in the abnormal area to be determined, the mutual information values ​​are integrated to generate a correlation matrix, and a correlation analysis result is generated based on the correlation matrix.

4. The method according to claim 1, characterized in that: The step of acquiring a real-time data stream of the target to be detected, and performing preliminary anomaly detection on the real-time data stream using an adaptive threshold determined by a trained Gaussian mixture model to detect abnormal data points includes: Acquire a real-time data stream of the target to be measured, construct an initial Gaussian mixture model based on the real-time data stream, train the initial Gaussian mixture model using an expectation-maximization algorithm, calculate the posterior probability that each data point belongs to each Gaussian component, and update the initial Gaussian mixture model parameters according to the posterior probability, repeatedly perform calculation operations and update operations until the parameter change of the Gaussian mixture model is less than a preset threshold, and obtain a trained Gaussian mixture model; Based on the trained Gaussian mixture model, a likelihood score of each data point is calculated, and based on the likelihood score, existing historical data, and the real-time data stream, an initial threshold is dynamically adjusted to obtain an adaptive threshold, wherein the initial threshold is formulated according to industry standards; Based on the adaptive threshold, each data point in the real-time data stream is analyzed, and if the likelihood score of the data point to be determined is lower than the adaptive threshold, the data point to be determined is marked as an abnormal data point.

5. The method according to claim 1, characterized in that The method of evaluating the conditional dependency relationship between the abnormal data points using a probabilistic graph model based on the high-level feature representation and the quantitative evaluation result of the abnormality degree, constructing a target probabilistic dependency graph based on the conditional dependency relationship, and generating an abnormal data point recognition result based on the target probabilistic dependency graph includes: Based on the high-level feature representation and the quantitative evaluation results of the abnormality degree, a probabilistic graphical model is constructed using a Bayesian network, wherein the nodes in the probabilistic graphical model represent abnormal data points, and the edges represent the conditional dependency relationships between the abnormal data points; Based on the probabilistic graph model, a variational inference method is used to calculate the conditional probability distribution of each node, and based on the conditional probability distribution, the conditional dependency relationship between nodes is evaluated in combination with mutual information and conditional mutual information to obtain a node conditional dependency result; Based on the node conditional dependency results, an initial probability dependency graph is constructed using graph embedding technology, and the initial probability dependency graph is optimized to obtain a target probability dependency graph, and based on the degree centrality, betweenness centrality, and closeness centrality of the nodes in the target probability dependency graph, key nodes in the target probability dependency graph are identified; An integrated learning method is used to comprehensively evaluate key nodes by combining the evaluation results of multiple models to obtain key node evaluation results. Based on the key node evaluation results, abnormal data point identification results are obtained. The multiple models include decision trees and gradient boosting machines.

6. The method according to claim 5, characterized in that Based on the node conditional dependency results, an initial probability dependency graph is constructed using graph embedding technology, and the initial probability dependency graph is optimized to obtain a target probability dependency graph, and based on the degree centrality, betweenness centrality, and closeness centrality of the nodes in the target probability dependency graph, key nodes in the target probability dependency graph are identified, including: Based on the node conditional dependency results, the abnormal data points in the probability graph model are vectorized using graph embedding technology to generate a node vector representation of each node in a low-dimensional space, and an initial probability dependency graph is constructed based on the node vector representation; Based on the initial probability dependency graph, the node vector representation is iteratively optimized until a preset maximum number of iterations is reached to obtain an optimized node vector representation, and the initial probability dependency graph is optimized based on the optimized node vector representation to obtain a target probability dependency graph; Calculate the degree centrality, betweenness centrality, and closeness centrality of all nodes in the target probability dependency graph to obtain a comprehensive centrality score corresponding to each node; Nodes whose comprehensive centrality scores are higher than a preset threshold are selected from the target probability dependency graph, and nodes whose comprehensive centrality scores are higher than the preset threshold are taken as key nodes.

7. The method according to claim 1, characterized in that Based on the abnormal data points, a composite model of a deep belief network combined with a long short-term memory network is used to perform deep feature learning and abnormality verification, and obtain high-level feature representation and quantitative evaluation results of the abnormality degree, including: Using the abnormal data points to perform unsupervised learning on multiple restricted Boltzmann machines to obtain a deep belief network constructed by the learned multiple restricted Boltzmann machines, and using the deep belief network to extract deep-level feature representations of the abnormal data points; Using the deep-level feature representation as an input of a preset long short-term memory network to obtain a composite model, and training the composite model using a supervised learning method to adjust weights and bias items of the composite model to obtain a trained composite model; Based on the trained composite model, forward propagation processing is performed on the abnormal data points to generate a high-level feature representation of each abnormal data point, and a quantitative evaluation result of the abnormal degree of each abnormal data point is calculated.

8. A system for identifying and cleaning abnormal data, characterized in that: include: An acquisition module is used to acquire a real-time data stream of a target to be tested, and to perform preliminary anomaly detection on the real-time data stream using an adaptive threshold determined by a trained Gaussian mixture model to detect abnormal data points; A learning module is used to perform deep feature learning and anomaly verification based on the abnormal data points by using a composite model of a deep belief network combined with a long short-term memory network to obtain high-level feature representation and quantitative evaluation results of the degree of abnormality; A construction module is used to evaluate the conditional dependency relationship between the abnormal data points using a probabilistic graph model based on the high-level feature representation and the quantitative evaluation result of the abnormality degree, construct a target probabilistic dependency graph based on the conditional dependency relationship, and generate an abnormal data point identification result based on the target probabilistic dependency graph; An adjustment module is used to optimize and adjust the existing cleaning strategy based on the abnormal data point identification result by applying a defined reward mechanism incentive algorithm to obtain an optimized cleaning strategy; The identification module is used to identify the abnormal area to be determined in the evaluation process of the probability graph model, introduce multimodal fusion technology to perform correlation analysis on the abnormal area to be determined, and obtain the correlation analysis result; based on the correlation analysis result, perform abnormal data judgment and abnormal data cleaning on the data in the abnormal area to be determined, and obtain the abnormal data processing result to be determined; based on the abnormal data processing result to be determined, adjust the cleaning strategy in the optimization to obtain the target cleaning strategy.

9. A computing device, characterized in that It comprises a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement an abnormal data identification and cleaning method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a computer, the method for identifying and cleaning abnormal data as described in any one of claims 1 to 7 is implemented.

Images