Training view and batch driven anomaly detection closed loop processing method and apparatus

Abstract

The application relates to an abnormality detection closed-loop processing method and device based on a training view and a batch drive, the method comprising the following steps: constructing a training view object to uniformly carry entity granularity, a training time range, a materialized time range, a feature set, a baseline configuration and a release state, so that multiple stages share the same configuration definition; generating a calculation batch according to the training view, driving feature calculation, baseline calculation, materialization, training, release and prediction execution through the calculation batch; establishing an explicit association between a prediction result and the training view, a deployment object and the calculation batch, and storing an alarm event based on the explicit association, so as to form a closed-loop backtracking link from the alarm prediction to the deployment object, the training view and the calculation batch. The method can reduce cross-stage configuration drift and improve the traceability and consistency control level of the whole abnormality detection process.

Description

Technical Field

[0001] This application relates to the fields of network security and abnormal behavior detection technology, and in particular to an anomaly detection closed-loop processing method and apparatus based on training view and batch-driven processing. Background Technology

[0002] User and entity behavior analysis technology has been widely applied in scenarios such as account anomaly detection, unauthorized access detection, internal threat detection, operation auditing, and data security. Unlike traditional intrusion detection, which mainly identifies external attacks, user and entity behavior analysis focuses more on abnormal usage behavior under legitimate identities. Therefore, its technical challenge lies not only in identifying anomalies themselves, but also in how to organize log processing, feature calculation, model training, prediction execution, and alert generation into a stable, traceable, and reusable engineering chain.

[0003] In real-world production environments, anomaly detection is a continuous, full-lifecycle system engineering process. This process includes at least log standardization, feature engineering, training sample preparation, baseline generation, model training, model deployment, prediction invocation, alert generation, and result backtracking. Currently, common processing methods mainly fall into the following categories.

[0004] The first type is a data processing and model training workflow driven by offline jobs or scripts. This type of solution breaks down log processing, feature computation, model training, and inference deployment into multiple independent tasks, each executed by a different script or job. There is a lack of a unified object to carry the running context between the stages, which can easily lead to problems such as inconsistencies between the feature caliber used in training and those called for inference, inconsistencies between the baseline time range and the training time range, and the inability to associate the model version with the feature version.

[0005] The second category is training and inference consistency solutions centered on feature platforms or feature storage. These solutions emphasize feature definition reuse, point-to-point consistent sample construction, and alignment of training and inference standards, which can partially solve the training service deviation problem. However, they usually do not further address the coordinated management between baseline objects, deployment objects, batch objects, and alarm event objects.

[0006] The third category is model lifecycle solutions centered on machine learning operation and maintenance orchestration and model registration. These solutions support training, experiment tracking, model registration, and deployment, but most only address the management of models and artifacts, failing to integrate upstream feature definitions, downstream prediction calls, and alarm results into a unified view management system.

[0007] The fourth type is a business solution that links prediction and alerting services. This type of solution can generate alerts after the model outputs, but it usually only triggers the call by the model name plus a date parameter, lacking a unified binding for training views, feature versions, baseline ranges, deployment objects, and running batches.

[0008] In summary, existing technologies have the following shortcomings: First, configuration is fragmented. Log processing, feature generation, model training, deployment, and prediction calls all rely on different configurations, leading to the easy reference of different time ranges, feature sets, or model versions at different stages. Second, state is fragmented. It is impossible to uniformly track whether features have been calculated, baselines have been completed, training has been completed, deployment has been completed, and predictions have generated alerts within the same business process. Third, results are difficult to trace back. Prediction results and alert results are usually difficult to accurately trace back to their corresponding training views, feature versions, deployment objects, and running batches. Fourth, the closed loop is insufficient. Even if many solutions support training and deployment, they do not truly establish a unified constraint relationship between model deployment, prediction services, and alert governance and upstream configuration objects. They remain at the level where multiple modules can call each other rather than the entire chain can be consistently controlled. Summary of the Invention

[0009] Therefore, it is necessary to provide a closed-loop processing method and apparatus for anomaly detection based on training views and batch-driven processing to address the aforementioned technical problems.

[0010] A closed-loop anomaly detection processing method based on training views and batch-driven processing, the method comprising: Construct a training view object; the training view object is used to uniformly maintain the shared core configuration in the anomaly detection lifecycle. The training view object includes at least entity granularity, training data time range, materialized data time range, feature set, baseline configuration, and release status. The entity granularity is the classification level of the object being detected, and the materialized data time range is the time interval covered by the solidified storage of feature data. A computation batch object is generated based on the training view object; the computation batch object is the smallest running unit that is executed across stages, and the computation batch object includes a unique batch identifier, an associated training view identifier, a computation date range, and a running status; The execution of feature computation, baseline computation, materialization, training, deployment, and prediction is driven by the computational batch object; After performing the prediction and obtaining the prediction result, an explicit association is established between the prediction result and the training view, deployment object, and computation batch object associated with the prediction result. An alarm event is generated based on the prediction result, and the alarm event is associated with the prediction result, the training view, the deployment object, and the calculation batch object according to the explicit association relationship to form a closed-loop backtracking link for the entire lifecycle of anomaly detection.

[0011] In one embodiment, the training view object further includes: The view name and display name, global scaling configuration, weight configuration, materialized view information, and stage status; wherein the materialized view information is used to record the storage location and version information of the solidified feature data corresponding to the training view; The training view object is used to provide the same business definition and temporal semantics for the upstream log and feature calculation stage, training sample construction stage, model release stage, and prediction invocation stage.

[0012] In one embodiment, the compute batch object further includes: Associated deployment objects, associated feature fixed versions, batch types, execution configuration snapshots, running logs or error messages, one-click process run identifiers and stage identifiers; The execution configuration snapshot is used to save the source table, target table, entity key, date column, feature list, missing value handling method and baseline statistics actually used in this run; The method further includes: The overall state of the one-click process is inferred based on the aggregated state of multiple calculation batch objects corresponding to the same one-click process run identifier. If any batch fails, the overall state of the one-click process is determined to be failed. If there are no failed batches and there are batches to be executed or running, the overall state of the one-click process is determined to be in progress. If all relevant batches are completed, the overall state of the one-click process is determined to be successful.

[0013] In one embodiment, the method further includes: constructing consistency control constraints; the consistency control constraints include: The time range of the materialized data covers the time range of the training data; The feature set used in training and the feature set used in post-production prediction both originate from the same training view. Establish explicit relationships between baseline statistics tables, materialized views, deployment objects, and predicted calls; After training is completed, the deployed object inherits key information from the training view and the baseline configuration; Wherein, if the training view is configured with an independent baseline time range, then the baseline time range falls within the training data time range; if the training view is bound to multiple fixed versions, then the multiple fixed versions belong to the same entity granularity and have all been fixed and released.

[0014] In one embodiment, the feature value of the entity to be filled on a preset date is obtained. When the eigenvalue When a feature is missing, it is filled in according to a preset rule to obtain the processed feature value after filling. ; Statistical eigenvalues Historical mean and historical standard deviation (std) within a historical time range; The current deviation z is calculated based on the historical mean and historical standard deviation (std): ; When the historical standard deviation std is less than the preset minimum value, the preset minimum value is used to replace the historical standard deviation std. Obtaining eigenvalues The historical minimum (min) and historical maximum (max) within the historical time range, after processing, are characteristic values. <min or When the value is greater than the maximum, determine the eigenvalue. Deviation from historical baseline; Calculate the deviation rate D: ; in, This is the preset minimum value; The baseline risk score Rb is determined based on the deviation rate D.

[0015] In one embodiment, the method further includes: calculating a comprehensive risk score based on the model anomaly score and the baseline risk score. R = a × Rb + b × Rm; Where a and b are the weights of the baseline risk score and the model anomaly score, respectively, and Rm is the model anomaly score; When the comprehensive risk score R is greater than or equal to the preset threshold T, it is determined that the corresponding entity has an anomaly on the corresponding date.

[0016] In one embodiment, during the alarm interpretation phase, the key features that cause the anomaly are analyzed, and the feature name, current value, lower bound of historical baseline, upper bound of historical baseline, deviation rate, and contribution of the key feature to the anomaly result are recorded. The contribution is calculated based on the model interpretation value, feature weight, or preset contribution score. When the contribution is greater than or equal to the preset contribution threshold, it is determined to be a high contribution. If the deviation rate of key features exceeds the preset deviation threshold and their contribution is high, the key features are written into the feature deviation record, and the feature deviation record, the original log details and the alarm event are associated and stored together. Each alarm event is first traced back to the corresponding prediction call, and then back to the corresponding deployment object, training view, and computation batch object, thus forming a complete backtracking chain.

[0017] A closed-loop anomaly detection processing device based on training views and batch-driven processing, the device comprising: The training view management module is used to construct training view objects. The training view objects are used to uniformly maintain the shared core configuration in the anomaly detection lifecycle. The training view objects include at least entity granularity, training data time range, materialized data time range, feature set, baseline configuration, and release status. The entity granularity is the classification level of the object being detected, and the materialized data time range is the time interval covered by the solidified storage of feature data. The computation batch management module is used to generate computation batch objects based on the training view objects. The computation batch object is the smallest running unit that is executed across stages. The computation batch object includes a unique batch identifier, an associated training view identifier, a computation date range, and a running status. The computation batch object drives the execution of feature computation, baseline computation, materialization, training, deployment, and prediction. The prediction execution module is used to establish an explicit association between the prediction result and the training view, deployment object and computation batch object associated with the prediction result after the prediction is obtained. The alarm generation and backtracking module is used to generate alarm events based on the prediction results, and associate and store the alarm events with the prediction results, the training view, the deployment object and the calculation batch object according to the explicit association relationship, so as to form a closed-loop backtracking link for the entire life cycle of anomaly detection.

[0018] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps: Construct a training view object; the training view object is used to uniformly maintain the shared core configuration in the anomaly detection lifecycle. The training view object includes at least entity granularity, training data time range, materialized data time range, feature set, baseline configuration, and release status. The entity granularity is the classification level of the object being detected, and the materialized data time range is the time interval covered by the solidified storage of feature data. A computation batch object is generated based on the training view object; the computation batch object is the smallest running unit that is executed across stages, and the computation batch object includes a unique batch identifier, an associated training view identifier, a computation date range, and a running status; The execution of feature computation, baseline computation, materialization, training, deployment, and prediction is driven by the computational batch object; After performing the prediction and obtaining the prediction result, an explicit association is established between the prediction result and the training view, deployment object, and computation batch object associated with the prediction result. An alarm event is generated based on the prediction result, and the alarm event is associated with the prediction result, the training view, the deployment object, and the calculation batch object according to the explicit association relationship to form a closed-loop backtracking link for the entire lifecycle of anomaly detection.

[0019] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Construct a training view object; the training view object is used to uniformly maintain the shared core configuration in the anomaly detection lifecycle. The training view object includes at least entity granularity, training data time range, materialized data time range, feature set, baseline configuration, and release status. The entity granularity is the classification level of the object being detected, and the materialized data time range is the time interval covered by the solidified storage of feature data. A computation batch object is generated based on the training view object; the computation batch object is the smallest running unit that is executed across stages, and the computation batch object includes a unique batch identifier, an associated training view identifier, a computation date range, and a running status; The execution of feature computation, baseline computation, materialization, training, deployment, and prediction is driven by the computational batch object; After performing the prediction and obtaining the prediction result, an explicit association is established between the prediction result and the training view, deployment object, and computation batch object associated with the prediction result. An alarm event is generated based on the prediction result, and the alarm event is associated with the prediction result, the training view, the deployment object, and the calculation batch object according to the explicit association relationship to form a closed-loop backtracking link for the entire lifecycle of anomaly detection.

[0020] The aforementioned closed-loop anomaly detection processing method and apparatus based on training view and batch-driven architecture unifies entity granularity, training data time range, materialized data time range, feature set, baseline configuration, and release status within a single configuration object by constructing a training view object. This allows the upstream log and feature calculation stage, training sample construction stage, model release stage, and prediction invocation stage to share the same business definition and temporal semantics, reducing cross-stage configuration drift. By generating computation batch objects based on the training view object and using these batch objects to drive the execution of feature calculation, baseline calculation, materialization, training, release, and prediction, the multi-stage execution process is streamlined. By unifying the status into a single state system, the operational status of each stage in the same business process is managed under a unified batch object, improving the observability and controllability of the entire anomaly detection process. By establishing explicit associations between prediction results and training views, deployment objects, and computation batch objects, and storing alarm events in association with these elements based on these explicit associations, each alarm can be traced back step-by-step along the chain from prediction call, deployment object, training view to computation batch object, forming a closed-loop traceability path from alarm to source, significantly improving the traceability and auditability of anomaly detection results. Furthermore, this application does not rely on a specific anomaly detection algorithm and is compatible with various detection mechanisms based on baseline, clustering, outlier detection, or combinations thereof, making it suitable as the underlying closed-loop control scheme for production-grade anomaly detection platforms. Attached Figure Description

[0021] Figure 1 This is an application scenario diagram of an anomaly detection closed-loop processing method based on training views and batch-driven processing in one embodiment; Figure 2 This is a flowchart illustrating an anomaly detection closed-loop processing method based on training views and batch-driven processing in one embodiment. Figure 3 This is a structural block diagram of an anomaly detection closed-loop processing device based on training views and batch-driven processing in one embodiment. Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0023] The anomaly detection closed-loop processing method based on training views and batch-driven processing provided in this application can be applied to, for example... Figure 1The application environment shown is illustrated. Terminal 102 and server 104 communicate via a network. Server 104 is equipped with an anomaly detection system, which includes a log standardization and enrichment module, a feature calculation and materialization module, a training view management module, a baseline configuration management module, a computation batch management module, a model training and deployment module, a prediction execution module, and an alarm generation and backtracking module. Server 104 receives anomaly detection task requests from terminal 102, determines the entity granularity, training data time range, materialized data time range, feature set, baseline configuration, and deployment status based on predefined training view objects, generates computation batch objects, and drives the execution of feature calculation, baseline calculation, materialization, training, deployment, and prediction through these computation batch objects. After obtaining the prediction results, it establishes an explicit association between the prediction results and the training view, deployment objects, and computation batch objects, and generates alarm events based on the prediction results for associated storage, forming a closed-loop backtracking link. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices, and server 104 can be implemented using a standalone server or a server cluster composed of multiple servers.

[0024] In one embodiment, such as Figure 2 As shown, a closed-loop anomaly detection processing method based on training views and batch-driven processing is provided, which is then applied to... Figure 1 This will be explained using a server as an example, including the following content.

[0025] Step 202: Construct the training view object.

[0026] The training view object is used to uniformly maintain the shared core configuration throughout the anomaly detection lifecycle. The training view object includes at least entity granularity, training data time range, materialized data time range, feature set, baseline configuration, and release status. Entity granularity refers to the classification level of the object being detected, such as job position granularity, personnel granularity, equipment granularity, account granularity, or a combination of these granularities. The materialized data time range refers to the time interval covered by the solidified storage of feature data. Through the training view object, the upstream log and feature calculation stage, training sample construction stage, model release stage, and prediction invocation stage no longer maintain independent configurations, but instead share the same business definition and temporal semantics, reducing configuration drift.

[0027] Step 204: Generate a calculation batch object based on the training view object.

[0028] A computation batch object is the smallest unit of execution across stages, including a unique batch identifier, an associated training view identifier, a computation date range, and a running status. After the training view is created, the system generates one or more computation batches. Each computation batch records a batch identifier, a training view identifier, a fixed version identifier, a computation time range, a batch type, an execution configuration snapshot, and a running status.

[0029] Step 206 involves the execution of batch object-driven feature computation, baseline computation, materialization, training, deployment, and prediction.

[0030] The computation process includes the chaining of features for batch objects, baseline calculation, materialization, training, deployment, and prediction, ensuring that the same business process has a unified state source.

[0031] Step 208: After performing the prediction and obtaining the prediction result, establish an explicit association between the prediction result and the training view, deployment object, and computation batch object associated with the prediction result.

[0032] The prediction results should at least record the prediction task identifier, entity identifier, prediction date or time window, model identifier, anomaly score or label, associated deployment object, and associated training view.

[0033] Step 210: Generate alarm events based on the prediction results, and associate and store the alarm events with the prediction results, training views, deployment objects, and calculation batch objects according to the explicit correlation, so as to form a closed-loop backtracking link for the entire lifecycle of anomaly detection.

[0034] When generating alarms based on prediction results, the prediction results, feature deviation information, original log details, and alarm events are stored together, so that each alarm can be traced back to the prediction call, and then back to the deployment object, training view, and calculation batch, thus forming a complete chain.

[0035] In the aforementioned closed-loop anomaly detection processing method based on training view and batch-driven architecture, a training view object is constructed as a unified configuration carrier. Entity granularity, training data time range, materialized data time range, feature set, baseline configuration, and deployment status are all uniformly contained within the same object. This allows the feature calculation stage, model training stage, model deployment stage, and prediction invocation stage to share the same configuration definition and temporal semantics. By generating computation batch objects from the training view object, the configuration information determined by the training view object is transformed into executable running units. These computation batch objects drive multi-stage execution, ensuring that the running status of each stage in the same business process is managed under a unified batch object. By establishing explicit associations between prediction results and the training view, deployment object, and computation batch objects, and storing alarm events based on these associations, each alarm can be traced back step-by-step along the prediction invocation, deployment object, training view, and computation batch object chain, forming a closed-loop tracing path from alarm to source. This solution does not rely on a specific anomaly detection algorithm and is compatible with various detection mechanisms based on baseline, clustering, outlier detection, or combinations thereof.

[0036] In one embodiment, the training view object also includes a view name and display name, global scaling configuration, weight configuration, materialized view information, and stage status. The materialized view information records the storage location and version information of the solidified feature data corresponding to the training view. The training view object provides a shared business definition and temporal semantics for the upstream log and feature calculation stage, training sample construction stage, model release stage, and prediction invocation stage. The global scaling configuration and weight configuration are used to unify the scale and allocate the weight of different risk sources during subsequent comprehensive risk assessment. By incorporating the view name, stage's global scaling configuration, and weight configuration into the unified management of the training view object, the configuration of the training view object becomes more complete. The downstream prediction and alarm stages can directly utilize these configurations to complete comprehensive risk calculation without repeated configuration.

[0037] In one embodiment, the computation batch object also includes the associated deployment object, the associated feature persistence version, the batch type, the execution configuration snapshot, the running log or error information, the one-click process run identifier, and the stage identifier. The execution configuration snapshot is used to save the source table, target table, entity key, date column, feature list, missing value handling method, and baseline statistics actually used in this run. By saving the execution configuration snapshot, the system can replay and reproduce the complete environment of the batch run when abnormal results occur. Computation batches can be subdivided into feature computation batches, baseline computation batches, training batches, and prediction batches, or they can be managed according to a unified batch type.

[0038] The method also includes inferring the overall state of the one-click process based on the aggregated state of multiple computation batch objects corresponding to the same one-click process run identifier. Specifically, the full-process orchestration stage unifies operations such as feature computation, baseline computation, materialization, deployment, and prediction under the same process run identifier. If a process run identifier corresponds to multiple batches, the process is considered failed when any batch fails; if there are no failed batches but there are still batches to be executed or running, the process is considered in progress; and the process is considered successful when all batches have been completed. In this way, the system can uniformly track the output of feature computation, baseline generation, materialization, deployment, and prediction results around a complete run, and establish the correlation between training views, computation batches, prediction results, feature deviation records, alarm records, and risk cases.

[0039] In one embodiment, the method further includes constructing consistency control constraints, which include at least one of the following.

[0040] The first constraint is that the time range of the materialized data must cover the time range of the training data. Since the training phase relies on the feature data that has been solidified during the materialization phase, if the time range of the materialized data does not cover the time range of the training data, the training phase will not be able to obtain complete feature data. Therefore, imposing this constraint ensures that the feature data required for the training phase has been solidified during the materialization phase.

[0041] The second constraint requires that the feature set used in training and the feature set used for prediction after deployment originate from the same training view. If different feature sets are used in the training and prediction phases, it will lead to inconsistencies in the training inference approach and cause prediction bias. Therefore, imposing this constraint ensures that the same set of feature definitions is used in both the training and prediction phases.

[0042] The third constraint establishes explicit relationships between baseline statistics tables, materialized views, deployment objects, and forecast calls. By establishing these relationships, it is ensured that baseline calculation results can be correctly referenced in the deployment and forecast phases, avoiding a disconnect between baseline calculation results and forecast calls.

[0043] The fourth constraint is that deployed objects inherit key information from the training view and baseline configuration after training. If a deployed object references a changed configuration after deployment, the prediction results will be inconsistent with those during training. Therefore, by having deployed objects inherit key information from the training view and baseline configuration, this can prevent misalignment after deployment.

[0044] Furthermore, when the training view is configured with an independent baseline time range, this baseline time range must fall within the training data time range to ensure consistency between the historical data on which the baseline statistics are based and the training data in terms of time range. When the training view is bound to multiple fixed versions, these multiple fixed versions must belong to the same entity granularity and all have been fixed and released to avoid granularity conflicts between versions. By applying the above consistency control constraints, the system does not simply train a model and output a result, but uses the training view as a unified configuration carrier, the computation batch as a unified execution carrier, and manages the baseline, materialization, prediction, and alarm as a unified closed-loop link to achieve consistency control throughout the entire lifecycle of anomaly detection.

[0045] In one embodiment, feature calculation and baseline calculation include the following.

[0046] First, the feature values ​​of the entity to be filled on a preset date are obtained. When a feature value is missing, it is filled according to a preset rule to obtain the processed feature value after filling. For example, suppose a certain feature value of an entity on a certain date is... The processed feature value obtained after filling is ,in This represents the original feature values ​​before filling. This represents the processed feature value after filling.

[0047] Secondly, the statistical characteristic values ​​are the historical mean and historical standard deviation (std) over the historical time range, where mean represents the historical mean and std represents the historical standard deviation.

[0048] Then, the current deviation z is calculated based on the historical mean and historical standard deviation (std). When the historical standard deviation (std) is less than the preset minimum value, the preset minimum value is used instead of the historical standard deviation (std) for calculation to avoid calculation anomalies caused by division by zero or extremely small values.

[0049] Next, the historical minimum (min) and historical maximum (max) of the feature value within the historical time range are obtained, where min represents the historical minimum and max represents the historical maximum. After processing the feature value... <min or When the value is greater than the maximum, the characteristic value is determined to deviate from the historical baseline.

[0050] Finally, calculate the deviation rate: ; in, This is the preset minimum value.

[0051] When calculating the deviation rate, the denominator is taken as the larger of the absolute value of the corresponding boundary value and the preset minimum value to avoid distortion of the deviation rate due to historical boundary values ​​being too small or zero. After processing the eigenvalues... When the value lies within the interval formed by the historical minimum (min) and the historical maximum (max), the deviation rate is 0. The baseline risk score Rb is determined based on the deviation rate, where Rb represents the baseline risk score.

[0052] Through the aforementioned feature calculation and baseline calculation process, the system not only calculates the degree of statistical deviation based on the mean and standard deviation, but also determines the boundary deviation based on the historical minimum and maximum value ranges. Furthermore, it integrates multiple deviation measurement methods into a unified baseline risk score by combining the deviation rate, thus providing a basis for subsequent comprehensive risk assessment.

[0053] In one embodiment, the generation of the prediction results includes the following.

[0054] The model anomaly score is combined with the baseline risk score for a comprehensive assessment. Let the baseline risk score be Rb, the model anomaly score be Rm, and the preset weights be a and b, where a is the weight of the baseline risk score and b is the weight of the model anomaly score, and the sum of a and b is 1. Comprehensive Risk Score: R = a × Rb + b × Rm.

[0055] The baseline risk score Rb and the model anomaly score Rm are standardized or normalized before fusion to ensure they are within a comparable range, preventing the over-amplification or neglect of a risk source due to differences in units. When the comprehensive risk score R is greater than or equal to a preset threshold T, the corresponding entity is determined to have experienced an anomaly on the corresponding date; otherwise, it is determined to be normal.

[0056] In this way, the system integrates risk assessment based on historical baselines with model-based anomaly identification. After standardization or normalization to unify the dimensions, it performs weighted synthesis, which improves the accuracy and robustness of anomaly identification compared to judgment methods that rely solely on baselines or models.

[0057] In one embodiment, generating an alarm event based on the prediction result includes the following.

[0058] During the alarm interpretation phase, the key features causing the anomaly are analyzed, and the feature name, current value, lower bound of the historical baseline, upper bound of the historical baseline, deviation rate, and contribution of the key feature to the anomaly are recorded. The contribution is calculated based on the model's interpreted value, feature weights, or a preset contribution score. When the contribution is greater than or equal to a preset contribution threshold, it is considered a high contribution.

[0059] If a key feature simultaneously meets both the conditions of having a deviation rate exceeding a preset deviation threshold and a high contribution, then the relevant information of that key feature is written into the feature deviation record, and the feature deviation record, the original log details, and the alarm event are stored together. In this way, during the alarm interpretation phase, not only is the occurrence of the anomaly recorded, but also the key feature that caused the anomaly, its deviation, and its degree of contribution.

[0060] In terms of backtracking, each alarm event is first traced back to the corresponding prediction call, then back to the corresponding deployment object, then back to the corresponding training view, and finally back to the corresponding computation batch object, thus forming a complete backtracking chain. Using this method, when a backtracking request for an alarm event is received, the system can display the key characteristics that led to the alarm event and their deviations, and trace back level by level along the chain from the alarm event to the prediction call to the deployment object to the training view to the computation batch object, providing complete technical evidence for auditing, review, and problem localization.

[0061] It should be understood that, although Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0062] In one embodiment, such as Figure 3 As shown, an anomaly detection closed-loop processing device based on training views and batch-driven processing is provided, including: a training view management module 302, a calculation batch management module 304, a prediction execution module 306, and an alarm generation and backtracking module 308, wherein: The training view management module 302 is used to construct training view objects. The training view objects are used to uniformly maintain the shared core configuration in the anomaly detection lifecycle. The training view objects include at least entity granularity, training data time range, materialized data time range, feature set, baseline configuration, and release status. The entity granularity is the classification level of the object being detected, and the materialized data time range is the time interval covered by the solidified storage of feature data.

[0063] The computation batch management module 304 is used to generate computation batch objects based on training view objects. The computation batch object is the smallest unit of execution across stages. The computation batch object includes a unique batch identifier, an associated training view identifier, a computation date range, and a running status. The computation batch object drives the execution of feature computation, baseline computation, materialization, training, deployment, and prediction.

[0064] The prediction execution module 306 is used to establish an explicit association between the prediction result and the training view, deployment object and computation batch object associated with the prediction result after the prediction is obtained.

[0065] The alarm generation and backtracking module 308 is used to generate alarm events based on the prediction results, and associate and store the alarm events with the prediction results, training views, deployment objects and calculation batch objects according to the explicit association relationship, so as to form a closed-loop backtracking link for the entire life cycle of anomaly detection.

[0066] Specific limitations regarding the anomaly detection closed-loop processing device based on training views and batch driving can be found in the limitations of the anomaly detection closed-loop processing method based on training views and batch driving described above, and will not be repeated here. Each module in the above-mentioned anomaly detection closed-loop processing device based on training views and batch driving can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0067] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores training view data, batch calculation data, feature data, baseline data, prediction result data, and alarm event data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a closed-loop anomaly detection processing method based on training views and batch-driven processing.

[0068] Those skilled in the art will understand that Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0069] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0070] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0071] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0072] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0073] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A closed-loop anomaly detection processing method based on training views and batch-driven processing, characterized in that, The method includes: Construct a training view object; the training view object is used to uniformly maintain the shared core configuration in the anomaly detection lifecycle. The training view object includes at least entity granularity, training data time range, materialized data time range, feature set, baseline configuration, and release status. The entity granularity is the classification level of the object being detected, and the materialized data time range is the time interval covered by the solidified storage of feature data. A computation batch object is generated based on the training view object; the computation batch object is the smallest running unit that is executed across stages, and the computation batch object includes a unique batch identifier, an associated training view identifier, a computation date range, and a running status; The execution of feature computation, baseline computation, materialization, training, deployment, and prediction is driven by the computational batch object; After obtaining the prediction result, an explicit association is established between the prediction result and the training view, deployment object, and computation batch object associated with the prediction result; the prediction result shall at least record the prediction task identifier, entity identifier, prediction date or time window, model identifier, anomaly score or label, associated deployment object, and associated training view; An alarm event is generated based on the prediction result, and the alarm event is associated with the prediction result, the training view, the deployment object, and the calculation batch object according to the explicit association relationship to form a closed-loop backtracking link for the entire lifecycle of anomaly detection.

2. The method according to claim 1, characterized in that, The training view object also includes: The view name and display name, global scaling configuration, weight configuration, materialized view information, and stage status; wherein the materialized view information is used to record the storage location and version information of the solidified feature data corresponding to the training view; The training view object is used to provide the same business definition and temporal semantics for the upstream log and feature calculation stage, training sample construction stage, model release stage, and prediction invocation stage.

3. The method according to claim 1, characterized in that, The batch calculation object also includes: Associated deployment objects, associated feature fixed versions, batch types, execution configuration snapshots, runtime logs or error messages, process runtime identifiers, and stage identifiers; The execution configuration snapshot is used to save the source table, target table, entity key, date column, feature list, missing value handling method and baseline statistics actually used in this run; The method further includes: The overall process status is inferred based on the aggregated status of multiple computation batch objects corresponding to the same process run identifier; wherein, if any batch fails, the overall process status is determined to be failed; if there are no failed batches and there are batches to be executed or running, the overall process status is determined to be in progress; if all relevant batches are completed, the overall process status is determined to be successful.

4. The method according to claim 1, characterized in that, The method further includes: Construct consistency control constraints; the consistency control constraints include: The time range of the materialized data covers the time range of the training data; The feature set used in training and the feature set used in post-production prediction both originate from the same training view. Establish explicit relationships between baseline statistics tables, materialized views, deployment objects, and predicted calls; After training is completed, the deployed object inherits key information from the training view and the baseline configuration; Wherein, if the training view is configured with an independent baseline time range, the baseline time range falls within the training data time range; if the training view is bound to multiple fixed versions, the multiple fixed versions belong to the same entity granularity and have all been fixed and released.

5. The method according to claim 1, characterized in that, The feature calculation and baseline calculation include: Get the feature value of the entity to be filled on the preset date. When the eigenvalue When a feature is missing, it is filled in according to a preset rule to obtain the processed feature value after filling. ; Statistical eigenvalues Historical mean and historical standard deviation (std) within a historical time range; The current deviation z is calculated based on the historical mean and historical standard deviation (std): When the historical standard deviation std is less than the preset minimum value, the preset minimum value is used to replace the historical standard deviation std. Obtaining eigenvalues The historical minimum (min) and historical maximum (max) within the historical time range, after processing, are characteristic values. <min or When the value is greater than the maximum, determine the eigenvalue. Deviation from historical baseline; Calculate the deviation rate D: in, This is the preset minimum value; The baseline risk score Rb is determined based on the deviation rate D.

6. The method according to claim 5, characterized in that, The generation of prediction results includes: Based on the model anomaly score and the baseline risk score, the comprehensive risk score is calculated as follows: R = a × Rb + b × Rm; Where a and b are the weights of the baseline risk score and the model anomaly score, respectively, and Rm is the model anomaly score. When the comprehensive risk score R is greater than or equal to the preset threshold T, it is determined that the corresponding entity has an anomaly on the corresponding date.

7. The method according to claim 6, characterized in that, The alarm events generated based on the prediction results include: During the alarm interpretation phase, the key features that cause the anomaly are analyzed, and the feature name, current value, lower bound of historical baseline, upper bound of historical baseline, deviation rate, and contribution of the key feature to the anomaly result are recorded. The contribution is calculated based on the model interpretation value, feature weight, or preset contribution score. When the contribution is greater than or equal to the preset contribution threshold, it is determined to be a high contribution. If the deviation rate of key features exceeds the preset deviation threshold and their contribution is high, the key features are written into the feature deviation record, and the feature deviation record, the original log details and the alarm event are associated and stored together. Each alarm event is first traced back to the corresponding prediction call, and then back to the corresponding deployment object, training view, and computation batch object, thus forming a complete backtracking chain.

8. An anomaly detection closed-loop processing device based on training views and batch-driven processing, characterized in that, The device includes: The training view management module is used to construct training view objects. The training view objects are used to uniformly maintain the shared core configuration in the anomaly detection lifecycle. The training view objects include at least entity granularity, training data time range, materialized data time range, feature set, baseline configuration, and release status. The entity granularity is the classification level of the object being detected, and the materialized data time range is the time interval covered by the solidified storage of feature data. The computation batch management module is used to generate computation batch objects based on the training view objects. The computation batch object is the smallest running unit that is executed across stages. The computation batch object includes a unique batch identifier, an associated training view identifier, a computation date range, and a running status. The computation batch object drives the execution of feature computation, baseline computation, materialization, training, deployment, and prediction. The prediction execution module is used to establish explicit associations between the prediction results and the training view, deployment object, and computation batch object associated with the prediction results after the prediction results are obtained. The prediction results at least record the prediction task identifier, entity identifier, prediction date or time window, model identifier, anomaly score or label, associated deployment object, and associated training view. The alarm generation and backtracking module is used to generate alarm events based on the prediction results, and associate and store the alarm events with the prediction results, the training view, the deployment object and the calculation batch object according to the explicit association relationship, so as to form a closed-loop backtracking link for the entire life cycle of anomaly detection.

Images