An artificial intelligence-based data security cleaning method
By constructing a behavioral baseline model for each edge cleaning node and analyzing its operational metrics in real time, a comprehensive anomaly risk score is generated, and a tiered security response strategy is implemented. This solves the problem of security risk spread caused by malicious program attacks in the distributed cleaning system, thereby improving the system's security and processing efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING JUNDE SPACETIME TECH CO LTD
- Filing Date
- 2026-02-05
- Publication Date
- 2026-06-02
AI Technical Summary
Existing distributed data cleaning systems face challenges when facing malicious attacks, including the wide spread of security risks and the difficulty in effectively controlling them. This is especially true in open network environments where the high access permissions of nodes pose a serious threat of data leakage.
By building a personalized behavioral baseline model for each edge cleaning node, the temporal characteristics and state transition patterns of its operating indicators are analyzed in real time to generate a comprehensive anomaly risk score. Based on the score, a graded safety response strategy is implemented, including normal processing, enhanced monitoring, task restriction, and mandatory isolation.
It enables proactive security monitoring of the distributed cleaning system, accurately identifies latent malicious activities, controls the spread of potential threats, improves the system's inherent security and long-term operational adaptability, and ensures data processing efficiency.
Smart Images
Figure CN122132682A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data cleaning technology, and in particular to a data security cleaning method based on artificial intelligence. Background Technology
[0002] Data cleaning refers to the process of inspecting, correcting, transforming, and integrating data collected from different sources to improve data quality and usability. With the rapid growth of data volume, the processing power of a single node can no longer meet the demands, leading to the development of distributed data cleaning technology. This technology distributes data cleaning tasks across multiple computing nodes for parallel execution, thereby significantly improving processing efficiency.
[0003] Currently, there are two main architectures for distributed data cleaning. The first is a centralized processing architecture, where all data to be cleaned is aggregated to a central server node for processing. However, this architecture suffers from single-point performance bottlenecks and single-point failure risks, and its scalability is limited. The second is a distributed architecture based on a master database, where multiple cleaning nodes connect to a central master database. Each node reads data from the master database according to task requirements and writes the cleaning results back. While this approach improves parallelism, nodes typically have high data access permissions.
[0004] In open network environments, nodes performing data scrubbing tasks may obtain raw data from diverse sources with unknown security from the internet, which may contain malicious programs. In existing distributed data scrubbing systems, once a node is controlled by malicious programs, its extensive access to system data or frequent interaction with other nodes may lead to a wider data leak and even threaten the security of the entire system. Therefore, while improving the efficiency of distributed data scrubbing, how to effectively control security risks and prevent the spread of localized threats has become an urgent problem to be solved. Summary of the Invention
[0005] This application provides an artificial intelligence-based data security cleaning method, aiming to improve the security of the data cleaning process in distributed systems.
[0006] This application provides a data security cleaning method based on artificial intelligence, including: S101: Collect multi-dimensional operational indicators of each edge cleaning node in the distributed system according to the preset sampling period, and generate a time-series feature vector sequence within the preset monitoring time window; S102, based on the temporal feature vector sequence of each edge cleaning node and its pre-built behavioral baseline model, a comprehensive abnormal risk score is generated within the current monitoring time window using a preset comprehensive risk analysis mechanism, which is used to characterize the degree of behavioral abnormality of the node. S103 compares the comprehensive abnormal risk score with the preset risk threshold range and triggers the corresponding level of security response strategy according to the range in which it is located.
[0007] Preferably, generating the time-series feature vector sequence within the preset monitoring time window includes: A1. Perform standardized preprocessing on all multi-dimensional operational indicators within the monitoring time window to generate a standardized operational feature vector for each sampling moment; A2. Arrange the standardized operational feature vectors of each sampling moment within the monitoring time window in time sequence to form a time-series feature vector sequence. J represents the total number of sampling points within the monitoring time window. Let j be the standardized running feature vector at the j-th sampling time. .
[0008] Preferably, the behavioral baseline model of the edge cleaning node includes at least: the mean and standard deviation of each operating indicator, the first feature value range baseline of the multi-dimensional operating indicator data, the state transition probability matrix describing the transition law between different behavioral states of the node, and the baseline state transition entropy; the behavioral baseline model of each edge cleaning node is constructed based on the historical operating indicator data collected when it performs standard data cleaning tasks in a controlled environment.
[0009] Preferably, the preset comprehensive risk analysis mechanism specifically includes: S201, calculate the feature deviation degree based on the degree to which each feature value in the time-series feature vector sequence deviates from the normal range of feature values in the behavioral baseline model; S202, calculate the state transition anomaly degree based on the degree of deviation between the behavioral state transition reflected by the temporal feature vector sequence and the state transition pattern in the behavioral baseline model; S203, the comprehensive anomaly risk score is obtained by integrating the feature deviation degree and the state transition anomaly degree.
[0010] Preferably, step S201 specifically includes: averaging the instantaneous deviations of all standardized running feature vectors in the time-series feature vector sequence to obtain the feature deviation:
[0011] in, The characteristic deviation is J, where J is the total number of sampling points within the monitoring time window. Let be the instantaneous deviation of the standardized running feature vector at the j-th sampling time.
[0012] Preferably, S202 specifically includes: C1. Mapping the temporal feature vector sequence to a behavioral state sequence ,in, Let be the behavioral state at the j-th sampling time, and , For a predefined set of finite discrete behavioral states; C2. Based on the behavioral state sequence S, obtain the empirical state transition matrix P', and calculate the actual state transition entropy of the behavioral state sequence. ; C3. Calculate the state transition entropy deviation based on the baseline state transition entropy and the actual state transition entropy of the behavioral state sequence. ; C4. Traverse each state transition in the behavioral state sequence S and detect the number of rare transition events C.
[0013] Preferably, the detection of the rare transfer event is achieved by setting a low probability threshold. Iterate through each state transition in the behavioral state sequence S. If its corresponding probability value in the baseline state transition probability matrix P is less than... If so, it is determined to be a rare transfer event.
[0014] Preferably, the comprehensive anomaly risk score is calculated according to the following formula:
[0015] Where R represents the comprehensive anomaly risk score. For feature deviation, Let C be the state transition entropy deviation, C be the number of rare transition events, and J be used to normalize C to obtain the frequency of rare transition events. , and These are the fusion weight coefficients for feature deviation, state transition entropy deviation, and frequency of rare transition events, respectively, satisfying... And all are greater than 0.
[0016] Preferably, the security response strategy includes at least one of normal processing, enhanced monitoring and task restrictions, and forced interruption and isolation.
[0017] Preferably, the predefined set of finite discrete behavioral states includes "idle", "data reception", "local cleaning", "waiting for integration" and "data upload".
[0018] One or more technical solutions provided in this application have at least the following technical effects or advantages: By constructing a personalized behavioral baseline model for each edge cleaning node and analyzing the temporal characteristics and state transition patterns of its operational indicators in real time, the comprehensive anomaly risk score of the node can be accurately quantified. Then, based on this score, a tiered security response strategy is implemented, ranging from normal access and restricted monitoring to mandatory isolation. This achieves proactive security monitoring at the micro-behavioral level, effectively identifying disguised or lurking malicious activities. While ensuring the overall data processing efficiency of the distributed cleaning system, it controls the spread of potential security threats within a single node, significantly improving the system's inherent security and long-term operational adaptability. Attached Figure Description
[0019] Figure 1 This is a flowchart illustrating the data security cleaning method based on artificial intelligence according to an embodiment of the present invention. Detailed Implementation
[0020] To facilitate understanding of the present invention, a more complete description of this application will be given below with reference to the accompanying drawings, which illustrate preferred embodiments of the invention. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to enable a more thorough and complete understanding of the disclosure of the present invention.
[0021] It should be noted that the terms "vertical," "horizontal," "up," "down," "left," "right," and similar expressions used in this article are for illustrative purposes only and do not represent the only possible implementation.
[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention; the term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.
[0023] Example 1: Figure 1 This is a flowchart illustrating the data security cleaning method based on artificial intelligence according to an embodiment of the present invention.
[0024] like Figure 1 As shown, an artificial intelligence-based data security cleaning method includes the following steps: S101 collects multi-dimensional operational indicators of each edge cleaning node in the distributed system according to a preset sampling period, and generates a time-series feature vector sequence within a preset monitoring time window.
[0025] Specifically, lightweight monitoring agents deployed at each edge cleaning node collect multi-dimensional operational metrics at a preset sampling period T (e.g., T=10 seconds). These multi-dimensional operational metrics include at least one or more of the following: disk I / O throughput, network receive rate, network send rate, CPU utilization, and active working set size in memory. Here, T is a positive real number preset based on the system's processing capacity and monitoring accuracy requirements, in seconds.
[0026] Specifically, generating a sequence of time-series feature vectors within a preset monitoring time window includes: A1. Perform standardized preprocessing (e.g., normalization to unify dimensions) on all multi-dimensional operational indicators within the monitoring time window to generate a standardized operational feature vector for each sampling time.
[0027] For example, the standardized running feature vector at sampling time t is represented as: , These represent the disk I / O throughput, network receive rate, network send rate, CPU utilization, and active working set size after normalization and preprocessing at sampling time t, respectively.
[0028] A2. Arrange the standardized operational feature vectors of each sampling moment within the monitoring time window in time sequence to form a time-series feature vector sequence. J represents the total number of sampling points within the monitoring time window. Let j be the standardized running feature vector at the j-th sampling time. .
[0029] S102, based on the temporal feature vector sequence of each edge cleaning node and its pre-built behavioral baseline model, uses a preset comprehensive risk analysis mechanism to generate a comprehensive abnormal risk score within the current monitoring time window, which is used to characterize the degree of behavioral abnormality of the node.
[0030] In some embodiments, the behavioral baseline model of each edge cleaning node is constructed based on historical operational metric data collected when it performs standard data cleaning tasks in a controlled environment. The construction method includes: B1. Collect the set of historical standardized operational feature vectors of edge cleaning nodes.
[0031] Specifically, for example, it includes: During the initial deployment phase of the system or during the designated security learning period, the edge cleaning nodes are allowed to perform a typical complete data cleaning task process in an environment free of anomalies. A lightweight monitoring agent deployed locally on the node continuously collects historical multi-dimensional performance data (sequence length is T) within a preset historical time period (a sufficiently long time period L, such as 24 hours) at a preset sampling period T. The dataset of operational metrics (where N is the total number of historical data sampling points) is preprocessed to form a set of historical standardized operational feature vectors (composed of the historical standardized operational feature vectors of all historical sampling moments).
[0032] B2. Construct the baseline for the first eigenvalue range: Based on this historical standardized operational feature vector set, the statistical distribution characteristics of each operational indicator are calculated. For the i-th operational indicator (e.g., i=1 represents disk I / O), the mean of all sample values in the set is calculated. and standard deviation As a statistical distribution characteristic, the normal dynamic range of this operational indicator is defined as follows: , where k is a preset baseline range coefficient (used to control the tolerance of the normal range. The larger the value of k, the wider the range of normal fluctuations allowed, and the less sensitive the system is to minor anomalies. Conversely, the smaller the value of k, the more sensitive it is. Usually k∈[2,3], and is taken as 2.5). The normal dynamic range of all operating indicators together constitutes the first characteristic value range baseline, which represents the reasonable range of fluctuation of each operating indicator value under normal conditions.
[0033] B3. Construct the state transition probability matrix and baseline entropy: b1. Define the finite discrete set of behavior states of a node. This includes Q predefined behavioral states in the data security cleaning process (in this invention, Q is set to 5, i.e., 5 behavioral states, denoted as: ={"Idle", "Data Reception", "Local Cleaning", "Waiting for Integration", "Data Upload"}).
[0034] b2. Extract each historical standardized operation feature vector from the historical standardized operation feature vector set. ( Mapped to a behavioral state The historical behavior state sequence is obtained. This represents the behavioral state corresponding to the t-th historical standardized feature vector in the set of historical standardized operational feature vectors.
[0035] For example, the mapping method for behavioral states is implemented using preset rules or a pre-trained classifier. Taking a pre-trained classifier as an example, a classification model is trained using labeled historical data to achieve a nonlinear mapping from a continuous feature space to a discrete state space. The specific process is as follows: 1. Training data preparation: During the system deployment or maintenance phase, a large amount of historical standardized operational feature vector data covering various typical work scenarios is collected. The system administrator or by associating the data with system task logs (e.g., when the system log records "Start receiving data block A", the corresponding time period is marked as "data receiving") can manually or semi-automatically label these historical data with the actual behavioral state labels, thus forming a labeled training dataset. , Let w be the w-th training data point, and W be the total number of training data points. To train the w-th historical standardized running feature vector in the dataset, For the corresponding behavior status label, ; 2. Classification Model Selection and Training: Choose a supervised learning algorithm that is suitable for multi-class classification and has strong interpretability as the classifier. For example, decision trees, random forests, or gradient boosting decision trees can be used. The pre-selected classifier is trained using the training dataset. The training process aims to minimize the difference between the predicted state and the true state label, and generate a trained classifier. 3. State inference: The standardized running feature vector (whether historical or real-time data) that needs to be mapped is input into the pre-trained classifier, which outputs a predicted behavior state label, i.e., the mapping result.
[0036] b3. Based on historical behavioral state sequences, statistically analyze the frequency of transitions between states (i.e., statistically analyze the frequency of transitions from one state to another). Transition to the next state frequency ), to obtain the baseline state transition probability matrix (calculate the state transition probability matrix P of the first-order Markov chain), its elements This indicates that in normal historical behavior, from the state Transition to state Conditional probability: , From state in the historical behavior state sequence Transition to state Number of observations Q=5.
[0037] The baseline state transition probability matrix P depicts the temporal pattern of the transitions between activity states of a node during normal operation. For example, the probability of jumping directly from "local cleaning" to "data reception" may be extremely low, while the probability of transitioning from "local cleaning" to "waiting for integration" is relatively high. Based on a large sample of data collected during the node's historical normal operation, the conditional probability matrix obtained through maximum likelihood estimation represents the node's normal behavior pattern.
[0038] b4. Calculate the baseline state transition entropy based on matrix P. As a benchmark value for measuring the temporal randomness of normal node behavior, the calculation formula is as follows: , It is a state The steady-state probability in the historical behavior state sequence (which can be solved) For details, please refer to the relevant technical descriptions of Markov chains, such as through eigenvalue decomposition (which will not be elaborated upon in this invention). When =0, it is stipulated that The baseline state transition entropy will be stored in the baseline model for subsequent analysis and comparison.
[0039] Therefore, the behavioral baseline model of each edge cleaning node includes at least: the mean and standard deviation of each operating indicator, the baseline range of the first feature value of the multi-dimensional operating indicator data, the state transition probability matrix describing the transition law between different behavioral states of the node, and the baseline state transition entropy.
[0040] In some embodiments, the pre-defined comprehensive risk analysis mechanism includes at least the analysis of feature deviation and state transition anomaly, specifically including: S201, calculate the feature deviation degree based on the degree to which each feature value in the time-series feature vector sequence deviates from the normal range of feature values in the behavioral baseline model.
[0041] Specifically, the feature deviation is obtained by averaging the instantaneous deviations of all standardized running feature vectors in the time-series feature vector sequence:
[0042]
[0043] in, The characteristic deviation is J, where J is the total number of sampling points within the monitoring time window. Let be the instantaneous deviation of the standardized operational feature vector at the j-th sampling time, and n be the number of operational indicators. The preset weight value of the i-th operational indicator in the standardized operational feature vector satisfies... This is used to differentiate the abnormal contribution levels of different operational indicators, and is pre-set based on expert experience and historical data. Let be the value of the i-th operational index in the standardized operational feature vector at the j-th sampling time. and These are the mean and standard deviation of the i-th operational indicator read from the baseline model of the node's behavior, respectively, where k is the preset baseline range coefficient. The function ensures that only portions exceeding the normal range are penalized.
[0044] S202, calculate the state transition anomaly degree based on the degree of deviation between the behavioral state transition reflected by the temporal feature vector sequence and the state transition pattern in the behavioral baseline model.
[0045] Specifically, step S202 includes: C1. Using the same method as when constructing the behavioral baseline model, map the temporal feature vector sequence to a behavioral state sequence. ,in, Let be the behavioral state at the j-th sampling time, and .
[0046] C2. Based on the behavioral state sequence S, the empirical state transition matrix P' is statistically obtained, and its elements are: (Calculation method is the same as b3,) , For the current monitoring time window, the status is as follows Transition to state (Number of observations), representing the number of observations from state Transition to state Calculate the actual state transition entropy of the behavioral state sequence based on the empirical conditional probability. :
[0047] in, It is a state The empirical distribution probability in the behavioral state sequence S.
[0048] The empirical state transition matrix P' is a conditional frequency matrix obtained by directly statistically analyzing real-time data collected within the current monitoring time window, reflecting the actual behavior of the node in the current time period.
[0049] C3. Calculate the state transition entropy deviation. :
[0050] in, The base state transition entropy, The actual state transition entropy of the behavioral state sequence. It is used to reflect the difference between the time-series uncertainty of real-time behavioral states and historical correct patterns.
[0051] C4. Detecting rare transition events: Set a low probability threshold. (For example, =0.02), traversing each state transition in the behavioral state sequence S, if its corresponding probability value in the baseline state transition probability matrix P is 0.02), If the event is rare, it is determined to be a rare transfer event. The total number of rare transfer events occurring within the current monitoring time window is counted and denoted as C.
[0052] It should be noted that the baseline state transition probability matrix P is a parameter estimated based on long-term historical normal data and is used to define the standard behavior pattern of the node; the empirical state transition matrix P' is a statistic calculated based on short-term real-time observation data and is used to describe the current behavior of the node. By comparing the differences between the two, it is possible to detect whether the node's behavior deviates from the normal pattern. The construction of P and P' is based on the determination of the state transition probabilities of the Markov chain, and their descriptions can be found in relevant existing technologies; this invention will not elaborate on them further.
[0053] S203, the comprehensive anomaly risk score is obtained by integrating the feature deviation degree and the state transition anomaly degree.
[0054] Specifically, the comprehensive anomaly risk score is calculated using the following formula:
[0055] Where R represents the comprehensive anomaly risk score. For feature deviation, Let C be the state transition entropy deviation, C be the number of rare transition events, and J be used to normalize C to obtain the frequency of rare transition events. , and These are the fusion weighting coefficients for feature deviation, state transition entropy deviation, and rare transition event frequency, respectively. These are constants pre-configured by experts based on business security strategies, and satisfy... And all are greater than 0. The comprehensive anomaly risk score R is a non-negative real number. The larger the value, the greater the comprehensive deviation of the node's behavior pattern in the current window period from its own normal historical baseline, that is, the higher the anomaly risk.
[0056] S103 compares the comprehensive abnormal risk score with the preset risk threshold range and triggers the corresponding level of security response strategy according to the range in which it is located.
[0057] Specifically, security response strategies include at least one of the following: normal processing, enhanced monitoring and task restrictions, and forced interruption and isolation.
[0058] In some embodiments, step S103 specifically includes: Risk threshold range , The specific value is set based on expert experience and the actual situation.
[0059] D1、
[0060] If the node's behavior is deemed normal, proceed with normal processing: allow the node to continue performing data cleaning tasks normally, its data request and upload processes will not be affected, and the time series data in the current monitoring time window can be marked as reliable normal samples for subsequent model updates.
[0061] D2、
[0062] If a node's behavior is deemed suspicious, enhanced monitoring and task restrictions will be implemented: Enhanced monitoring and restriction measures will be initiated, including but not limited to: shortening the monitoring cycle (e.g., changing it to T / 2), suspending the allocation of new data cleaning sub-tasks to the node, restricting data requests (implementing delayed responses, traffic shaping, or providing only partial non-core data for any data acquisition requests initiated by the node), and generating warning logs.
[0063] D3、
[0064] If a node's behavior is deemed highly abnormal and poses a serious security threat, mandatory interruption and isolation will be implemented: Immediately execute mandatory isolation measures, including but not limited to: suspending the node's processes (sending a security command to the node to suspend all its ongoing processing processes), blocking network connections (at the network level, implementing rule updates at the regional control node or gateway to temporarily block unnecessary network connections to the abnormal node), extracting key logs for forensic analysis (if protocols and permissions permit, attempt to securely extract key runtime logs, intermediate data, and final states from the node's memory and disk, and transmit them back to the control center for forensic analysis), and broadcasting risk alerts to achieve collaborative defense (broadcasting the node's risk alerts and identifiers to the data master node and other regional control nodes to achieve threat intelligence sharing and collaborative defense, preventing potential threats from spreading laterally in the distributed system).
[0065] In summary, by constructing a personalized behavioral baseline model for each edge cleaning node and analyzing the temporal characteristics and state transition patterns of its operational indicators in real time, the comprehensive anomaly risk score of the node can be accurately quantified. Furthermore, based on this score, a tiered security response strategy is implemented, ranging from normal access and restricted monitoring to mandatory isolation. This achieves proactive security monitoring at the micro-behavioral level, effectively identifying disguised or lurking malicious activities. While ensuring the overall data processing efficiency of the distributed cleaning system, it controls the spread of potential security threats within a single node, significantly improving the system's inherent security and long-term operational adaptability.
[0066] Traditional distributed system security focuses primarily on the correctness of task results or network communication anomalies. This embodiment delves into the internal runtime behavior of each edge cleaning node, continuously collecting and analyzing multi-dimensional operational metrics such as CPU, memory, disk I / O, and network traffic, and mapping them to specific behavioral state sequences such as "data reception" and "local cleaning." The system can depict the complete "working lifecycle" of a node; it can detect potential threats by identifying subtle deviations in its runtime behavior (such as abnormal state transition patterns) before malicious operations directly affect data or launch network attacks, transforming passive protection into proactive prevention; and it effectively defends against "legitimate appearance" attacks that do not produce obvious erroneous results or initiate abnormal network connections, but instead steal or destroy data by tampering with internal processing logic.
[0067] Abandoning single-threshold alarms, this approach proposes three quantifiable indicators: "feature deviation," "state transition entropy deviation," and "rare event frequency," which are then fused together into a comprehensive anomaly risk score through configurable weights. A node's temporary exceedance of a single indicator due to temporary load fluctuations will receive a lower risk score because its state transition sequence remains normal, avoiding frequent false alarms and reducing operational burden. While sophisticated, slow malicious behavior may carefully control a single indicator from exceeding limits, it cannot perfectly simulate the normal state transition sequence patterns learned over time. State transition entropy deviation and rare event detection can keenly capture this logical inconsistency, preventing missed detections. The comprehensive score and its constituent components provide administrators with a clear decision dashboard, supporting more accurate manual assessment.
[0068] A tiered response mechanism is introduced, mapping the quantified risk score R to a three-tiered response strategy of "normal processing - enhanced monitoring - mandatory isolation." Each tier has a clear set of automated operations. The business flow of the vast majority of low-risk nodes remains unaffected, ensuring the overall system throughput. Restrictions are imposed only on medium- and high-risk nodes, minimizing security costs. For suspicious but unconfirmed nodes (medium risk), flexible measures such as "restricting new tasks and tightening monitoring" are adopted, which can control the spread of potential risks and provide an observation period for fluctuations that may be legitimate, avoiding overreaction. For confirmed high-risk nodes, the system can automatically execute interruption, isolation, and evidence collection processes, instantly cutting off the attack chain, preventing lateral movement, and preserving key evidence.
[0069] The technical solutions described in the embodiments of this application have at least the following technical effects or advantages: By introducing a temporal behavior analysis mechanism, refined monitoring and quantification of abnormal risks of edge cleaning nodes are achieved. By establishing a personalized behavioral baseline model for each node and calculating its comprehensive abnormal risk score in real time, abnormal behaviors deviating from normal patterns can be accurately identified from two dimensions: node operational index values and internal state transition temporal patterns. This deepens the granularity of security monitoring from the task result level to the node operational temporal behavior level, making the perception of potential threats more acute and proactive.
[0070] By designing a multi-layered cascaded risk quantification model, the accuracy and robustness of anomaly detection are improved. Instead of relying on a single threshold for judgment, it integrates multiple features such as feature value deviation, state transition entropy deviation, and the frequency of rare transition events, fusing them through configurable weights. This multi-factor fusion strategy effectively avoids false alarms caused by fluctuations in a single indicator, more comprehensively characterizing the degree of anomaly at nodes and making risk assessment results more reliable.
[0071] By implementing a tiered security response strategy, precise security control and minimal impact were achieved. Different response levels were matched based on quantified risk scores: low-risk nodes were allowed to proceed normally, medium-risk nodes were monitored and restricted, and only high-risk nodes were subject to mandatory isolation. This differentiated response mechanism avoided the drag on overall system efficiency caused by a "one-size-fits-all" approach, and maximized the continuity of distributed cleaning tasks and the utilization of system resources while ensuring security.
[0072] Therefore, it not only solves the prominent problem of uncontrollable security risks when traditional distributed cleaning architectures face the challenge of internal node control, but also achieves a good balance between the parallel efficiency of distributed data cleaning and system security, enabling the safer implementation of distributed data cleaning technology.
[0073] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A data security cleaning method based on artificial intelligence, characterized in that, include: S101: Collect multi-dimensional operational indicators of each edge cleaning node in the distributed system according to the preset sampling period, and generate a time-series feature vector sequence within the preset monitoring time window; S102, based on the temporal feature vector sequence of each edge cleaning node and its pre-built behavioral baseline model, a comprehensive abnormal risk score is generated within the current monitoring time window using a preset comprehensive risk analysis mechanism, which is used to characterize the degree of behavioral abnormality of the node. S103 compares the comprehensive abnormal risk score with the preset risk threshold range and triggers the corresponding level of security response strategy according to the range in which it is located.
2. The data security cleaning method based on artificial intelligence as described in claim 1, characterized in that, The generation of the time-series feature vector sequence within the preset monitoring time window includes: A1. Perform standardized preprocessing on all multi-dimensional operational indicators within the monitoring time window to generate a standardized operational feature vector for each sampling moment; A2. Arrange the standardized operational feature vectors of each sampling moment within the monitoring time window in time sequence to form a time-series feature vector sequence. J represents the total number of sampling points within the monitoring time window. Let j be the standardized running feature vector at the j-th sampling time. .
3. The data security cleaning method based on artificial intelligence as described in claim 2, characterized in that, The behavioral baseline model of the edge cleaning node includes at least: the mean and standard deviation of each operating indicator, the baseline of the first feature value range of the multi-dimensional operating indicator data, the state transition probability matrix describing the transition law between different behavioral states of the node, and the baseline state transition entropy; the behavioral baseline model of each edge cleaning node is constructed based on the historical operating indicator data collected when it performs standard data cleaning tasks in a controlled environment.
4. The data security cleaning method based on artificial intelligence as described in claim 3, characterized in that, The pre-set comprehensive risk analysis mechanism specifically includes: S201, calculate the feature deviation degree based on the degree to which each feature value in the time-series feature vector sequence deviates from the normal range of feature values in the behavioral baseline model; S202, calculate the state transition anomaly degree based on the degree of deviation between the behavioral state transition reflected by the temporal feature vector sequence and the state transition pattern in the behavioral baseline model; S203, the comprehensive anomaly risk score is obtained by integrating the feature deviation degree and the state transition anomaly degree.
5. The data security cleaning method based on artificial intelligence as described in claim 4, characterized in that, S201 specifically includes: averaging the instantaneous deviations of all standardized running feature vectors in the time-series feature vector sequence to obtain the feature deviation: in, The characteristic deviation is J, where J is the total number of sampling points within the monitoring time window. Let be the instantaneous deviation of the standardized running feature vector at the j-th sampling time.
6. The data security cleaning method based on artificial intelligence as described in claim 4, characterized in that, S202 specifically includes: C1. Mapping the temporal feature vector sequence to a behavioral state sequence ,in, Let be the behavioral state at the j-th sampling time, and , For a predefined set of finite discrete behavioral states; C2. Based on the behavioral state sequence S, obtain the empirical state transition matrix P', and calculate the actual state transition entropy of the behavioral state sequence. ; C3. Calculate the state transition entropy deviation based on the baseline state transition entropy and the actual state transition entropy of the behavioral state sequence. ; C4. Traverse each state transition in the behavioral state sequence S and detect the number of rare transition events C.
7. The data security cleaning method based on artificial intelligence as described in claim 6, characterized in that, The detection of the rare transition event is achieved by setting a low probability threshold. Iterate through each state transition in the behavioral state sequence S. If its corresponding probability value in the baseline state transition probability matrix P is less than... If so, it is determined to be a rare transfer event.
8. The data security cleaning method based on artificial intelligence as described in claim 7, characterized in that, The comprehensive anomaly risk score is calculated using the following formula: Where R represents the comprehensive anomaly risk score. For feature deviation, Let C be the state transition entropy deviation, C be the number of rare transition events, and J be used to normalize C to obtain the frequency of rare transition events. , and These are the fusion weight coefficients for feature deviation, state transition entropy deviation, and frequency of rare transition events, respectively, satisfying... And all are greater than 0.
9. The data security cleaning method based on artificial intelligence as described in claim 8, characterized in that, The security response strategy includes at least one of the following: normal processing, enhanced monitoring and task restrictions, and forced interruption and isolation.
10. The data security cleaning method based on artificial intelligence as described in claim 6, characterized in that, The predefined set of finite discrete behavioral states includes "idle", "data reception", "local cleaning", "waiting for integration" and "data upload".