A deep learning-based data asset anomaly detection method and system

Abstract

The application provides a kind of data asset anomaly detection method and system based on deep learning, it is related to data asset security monitoring technical field. The multi-source original record generated in the running process of data asset is associated as a target running event according to event identification or space-time correlation information;Extract multi-dimensional running features for unified coding and establish feature mapping index;Build an event correlation graph containing target event nodes, entity association nodes and historical event nodes, write default zero vector representation and missing mask when node type is missing;Use graph neural network to pass messages and aggregate according to relationship edge type and time weight, generate fixed dimension fusion features;Calculate grouping reconstruction residual and anomaly score through feature reconstruction network, and output anomaly detection results according to detection threshold verified by drift test. It solves the problem that multi-source heterogeneous records are difficult to be uniformly represented, the input dimension of the model changes due to the missing of node type, and the stability of cross-dimensional association anomaly detection is insufficient.

Description

Technical Field

[0001] This application belongs to the field of data asset security monitoring technology, specifically involving a data asset anomaly detection method and system based on deep learning. Background Technology

[0002] As enterprise digital systems continue to operate, databases, business applications, interface gateways, identity authentication systems, and terminal management systems generate business access logs, database audit logs, interface call records, network session records, and terminal status records. These records respectively reflect the operating entity, the accessed data asset, the accessing device, the network access status, the operation time, and the execution result, and are the basic technical data for computer systems to identify abnormal data access processes.

[0003] Existing anomaly detection methods typically employ manual rules, fixed thresholds, or machine learning models based on manually constructed features. For example, they may independently judge based on access time, access frequency, Internet Protocol Address (IP address), permission scope, or the amount of data returned. Different data sources may use inconsistent recording formats and identification methods for the same access process, and transmission retries may generate duplicate records. Therefore, the same process can easily be split into multiple detection objects.

[0004] Even when no individual metric exceeds the limits, the combined relationships between users, devices, network environment, time status, operation type, and data assets may still deviate from historical normal patterns. If multi-source records cannot be associated as a unified event object, the model input dimensions of different node types cannot be kept stable, and cross-dimensional relationships cannot be jointly processed, problems such as unstable identification of association anomalies, threshold mismatch with changes in business distribution, and lack of repeatable judgment rules for anomaly interpretation are likely to occur. Summary of the Invention

[0005] This application provides a data asset anomaly detection method and system based on deep learning to solve the technical problems of difficulty in forming a unified event representation from multi-source heterogeneous operation records, changes in model input dimensions due to missing node types, and insufficient stability in cross-dimensional correlation anomaly detection and interpretation.

[0006] The technical solution adopted in this application is as follows.

[0007] This application provides a deep learning-based method for anomaly detection in data assets, executed by an anomaly detection system in a data asset management platform. The method includes:

[0008] Acquire multi-source raw records generated during the operation of data assets. The multi-source raw records include at least two of the following: business access records, database audit records, interface call records, identity authentication records, network session records, and terminal status records. Each raw record includes at least the source system, source record identifier, and record time. Based on the event identifier information or spatiotemporal correlation information in the multi-source raw records, associate the multi-source raw records to obtain the target operation event.

[0009] Extract operational features from at least two of the eight data dimensions (event level, user, data asset, device, network environment, operation, time status, and historical behavior) from the target operational event. Unify the operational features to obtain a fixed-length global feature vector. Establish a feature mapping index to record the position of each operational feature in the global feature vector and the corresponding node type.

[0010] Based on the association between the target running event and the associated object, an event association graph is generated, including the target event node, entity association node, and historical event nodes that meet the temporal association conditions. The running features are mapped to the node features of the corresponding nodes according to the feature mapping index. When the preset node type does not exist, a default node representation and a missing mask are set for the node type. The default node representation adopts a zero vector.

[0011] Based on the event association graph, the node features are transformed according to the relationship edge type by the graph neural network, and the relationship messages of the temporal association edges are weighted according to the time interval between the historical events and the target running event. The relationship messages of each preset node type are aggregated, and the node type representation or default node representation corresponding to each preset node type and the missing mask are combined in a fixed order to obtain fixed-dimensional fusion features and fusion grouping index.

[0012] The fused features are input into a feature reconstruction network trained based on historical normal operation events to obtain reconstructed features. When training the feature reconstruction network, the missing mask is used to mask the reconstruction error of the feature group corresponding to the missing node type. The group reconstruction residual between the fused features and the reconstructed features is calculated according to the fused group index. An anomaly score is determined according to each group reconstruction residual and the missing mask. The anomaly score is compared with the detection threshold to obtain the anomaly detection result of the target operation event.

[0013] In one implementation, the preset node types include target event types, user types, data asset types, device types, network environment types, operation types, time status types, and historical event types; each preset node type corresponds to a fixed node slot and a fixed representation dimension.

[0014] In one implementation, the historical normal operation events are selected and determined according to the following process: all historical events within a preset observation period that have not been triggered by existing independent detection rules are selected as initial screening samples. The existing independent detection rules refer to pre-deployed detection methods based on fixed thresholds or simple rules, independent of the anomaly detection method of this application; the initial screening samples are stratified and sampled according to time period, user role, and data asset type, with a sampling ratio of not less than 5%, and the sampled samples are reviewed and confirmed by security operation and maintenance personnel; when the sampling review accuracy rate reaches 99% or higher, the reviewed and confirmed sampled samples are used as historical normal operation events; the normal operation event set is reconstructed every quarter, expired data is removed, and recently confirmed normal operation records are included. When the number of reviewed and confirmed sampled samples is insufficient to support model training, data augmentation technology is used to expand the sampled samples. The data augmentation includes adding small-amplitude random noise following a normal distribution (the noise standard deviation is 0.05 times the standard deviation of the feature) to continuous features and performing random replacement on discrete features (i.e., for discrete features, the feature value is randomly replaced with other category values ​​of the feature in the training set with a probability of 0.05). The preset observation period is determined based on the historical alarm coverage of the business system. Specifically, it involves counting the number of historical events that trigger alarms daily over the past 90 days, selecting the time span during which the number of alarms is stable and lower than 120% of the daily average number of alarms as the observation period. The observation period shall be no less than 7 days and no more than 90 days.

[0015] In one implementation, when original records have the same event identifier, request identifier, or link identifier, they are associated according to the same identifier. When a common identifier is missing, a baseline record is determined according to a preset data source priority. The record with the smallest time difference and which satisfies the same user identifier, data asset identifier, and operation type is selected from the candidate records of each data source. Parallel candidates are eliminated by session identifier, device identifier, and record time order. The association duration threshold is calibrated using cross-source matching records carrying a common identifier: first, clock synchronization is completed for each data source, and the absolute record time difference of each matching record pair is calculated; the quantile values ​​of the time difference set at the 0.95, 0.97, 0.99, and 0.995 quantile levels are taken respectively, and added to the upper limit of clock synchronization error to form the candidate association duration threshold; the smallest candidate value that enables the matching recall rate to reach a preset value and the false association rate to not exceed the allowable false association rate is selected on independently labeled samples.

[0016] In one implementation, records with the same source record identifier, or the same source system and content digest, and a time difference not exceeding the retransmission duration threshold, are merged. The number of records before merging is reduced by one to obtain the retransmission count. Records with independent event identifiers or different execution times that remain after retransmission merging are retained, and duplicate operation features are generated. The retransmission duration threshold is calibrated using confirmed retransmission records: the absolute record time difference of retransmission record pairs corresponding to the same source record identifier or the same transmission sequence identifier is calculated; the quantile values ​​of this time difference set at the 0.95, 0.97, 0.99, and 0.995 quantile levels are taken respectively, and added to the upper limit of clock synchronization error to form candidate retransmission duration thresholds; the smallest candidate value that enables the retransmission identification recall rate to reach a preset value and the false merging rate to not exceed the allowable false merging rate is selected from the independently labeled samples, and this candidate value is required to be less than the associated duration threshold. The association duration threshold and retransmission duration threshold are only recalibrated when the system is deployed, the data source interface is changed, or the clock synchronization strategy is changed. After being verified by independent samples, they take effect with the new configuration version. No event-driven adaptive changes are performed during online detection of the same configuration version.

[0017] In one implementation, numerical placeholders and feature missing flags are written for missing operational features; outlier flags are generated using the upper and lower quantile values ​​of historical normal operation events; and different types of features are uniformly encoded using the class mapping, mean, standard deviation, and sequence length saved during the training phase.

[0018] In one implementation, the node types of the event association graph include target event nodes, user nodes, data asset nodes, device nodes, network environment nodes, operation type nodes, time status nodes, and historical event nodes. Each node type corresponds to a fixed node slot. When a node type does not exist, a zero vector is written as the default node representation and a missing mask is set to maintain a fixed dimension for the graph neural network output and the feature reconstruction network input.

[0019] In one implementation, the graph neural network calls different message transformation parameters for different entity relationship edge types and determines the time message weights of temporally related edges according to an exponential decay function; it performs average aggregation on relationship messages of the same node type and combines the node type representations and missing masks in a fixed order. The graph neural network adopts a graph attention network (GAT) structure, specifically including two graph attention layers, each layer with a total node representation dimension of 256 dimensions, 4 attention heads, and each attention head outputting 64 dimensions. The outputs of all attention heads are concatenated to still have 256 dimensions; each entity relationship edge type corresponds to an independent linear transformation matrix W_r and a bias b_r, where W_r has a dimension of 256×256 and b_r has a dimension of 256. The residual update formula after message aggregation at each layer is h_v^(l+1)=h_v^(l)+λ_v^(l)·Aggregation({h_u^(l)|u∈N(v)}). Where v is the node to be updated, u is a neighbor node of v, N(v) is the set of neighbor nodes of v, l is the graph attention layer number, h_v^(l) and h_v^(l+1) are the 256-dimensional node representations of node v at the input and output ends of layer l, respectively, h_u^(l) is the node representation of neighbor node u at layer l, Aggregation(·) represents the 256-dimensional aggregation result obtained by weighted summation of neighbor relationship messages after relation type transformation and attention normalization, and λ_v^(l) is the scalar learnable residual weight corresponding to node v at layer l. λ_v^(l) is initialized to 1.0 and trained together with other model parameters using the Adam optimizer.

[0020] In one implementation, the feature reconstruction network employs an autoencoder structure. Let the fixed dimension of the fused features be D_F. Then, the input layer dimension of the encoder and the output layer dimension of the decoder are both D_F. The encoder sequentially includes fully connected layers with dimensions D_F→1024→256→64, and the decoder sequentially includes fully connected layers with dimensions 64→256→1024→D_F. The intermediate layers use the ReLU activation function, while the bottleneck layer and output layer do not apply any activation function. A dropout layer with a dropout rate of 0.2 is set after each encoding layer. D_F is uniquely determined by the preset number of node types, the fixed representation dimension of each node type, and the number of existence masks, thereby ensuring that the input dimension of the autoencoder is consistent with the dimension of the fused features.

[0021] Preferably, the grouping weights are determined as follows: After model training is completed, the grouping reconstruction residual statistics of each feature group are calculated based on the validation set; let the mean value of the reconstruction residual of the g-th feature group on the validation set be μ_g and the standard deviation be σ_g, then the grouping weight w_g is determined according to w_g=1 / max(μ_g+σ_g,ε), where ε is a positive stable term; the grouping weights are fixed after model training is completed and remain unchanged during the online detection phase; when the model parameters are incrementally updated, the grouping weights are synchronously recalculated based on the current validation set to ensure the adaptability of the weights to the updated model.

[0022] In one implementation, the group reconstruction residuals of each feature group are calculated based on the fusion group index; the residuals of the effective feature groups are weighted and averaged using group weights and a missing mask to obtain an anomaly score; the upper quantile level of the normal score reference set is determined based on the allowable false positive rate of normal events, and the detection threshold is determined using the corresponding upper quantile value. The unsupervised pre-training of the graph neural network and the feature reconstruction network uses the fusion features of historical normal operation events as input, the weighted average of the group reconstruction residuals of the effective feature groups as the pre-training loss, and employs the Adam optimizer with an initial learning rate of 1×10^-3; pre-training stops when the validation set reconstruction loss no longer decreases for five consecutive training epochs. During pre-training, a missing mask is used to mask the reconstruction error of the feature group corresponding to the missing node type, preventing the default zero vector from participating in gradient updates.

[0023] The system cold start adopts the following approach: During the initial deployment of the system, before a set of historical normal operation events suitable for model training has been formed, conservative rules are used for detection, while real operational data is accumulated. These conservative rules consist of multiple single-dimensional threshold rules, including an upper limit on the amount of data returned in a single query, an upper limit on the number of accesses per day, and alarm rules for access during non-working hours. The upper limit on the amount of data returned in a single query and the upper limit on the number of accesses per day are primarily determined based on statistical values ​​from the business logs of the 30 days prior to deployment, after removing confirmed abnormal records; if available business logs are lacking, the absolute value of the security baseline provided by the business department is used. Multiple rules are combined using an "OR" logic, meaning that an alarm is generated when any rule is triggered. Initially, the allowed false alarm ratio ρ is set to 0.001. After accumulating no fewer than 1000 manually verified normal events, the threshold calibration method based on the actual normal score distribution is switched to. All abnormal candidate events during the cold start phase must undergo manual verification, and the verification results serve as the first batch of feedback samples.

[0024] In one implementation, feedback samples are used to recalculate anomaly scores using the current model before being used for threshold calibration or incremental training. Feedback samples are accumulated over a 7-day period: when the total number of confirmed normal events and confirmed false positive events reaches the minimum sample size N_T for threshold calibration, only detection threshold calibration is performed; candidate model training is only performed when the total number of confirmed normal events and confirmed false positive events reaches the minimum sample size N_n for normal training, and the total number of confirmed abnormal events and missed events reaches the minimum sample size N_a for anomaly constraints. For example, N_T = 200, N_n = 1000, and N_a = 50; if the required sample size for any processing is not reached, the corresponding current threshold or current model remains unchanged, and sample accumulation continues.

[0025] In one implementation, the drift in the scoring distribution is determined by the maximum cumulative difference between the recent normal scoring distribution and the baseline normal scoring distribution. The drift determination threshold δ_D is set to a fixed value between 0.12 and 0.20, determined based on business sensitivity during system deployment and remaining unchanged during operation. Candidate detection thresholds are only used in the next detection cycle if the false alarm rate on independent normal verification events does not exceed the allowable false alarm rate. The screening method for independent normal verification events is described in step S500.

[0026] In one implementation, a missed event refers to an event in the historical data that did not trigger an alarm in the current model but was subsequently identified as an anomaly during manual auditing or business review. The discovery of missed events relies on a manual auditing process that runs in parallel with the anomaly detection system; the two operate independently. During incremental training, missed events and confirmed anomaly events together form an anomaly constraint set, used to generate anomaly separation loss, and are not considered as normal reconstruction targets.

[0027] In one implementation, candidate node types are determined according to the cumulative contribution ratio of the grouped reconstruction residuals; the abnormal contribution degree is calculated by node-by-node message masking, and the target contributing node is determined according to the cumulative proportion of the abnormal contribution degree. For each target contributing node, its corresponding feature group is located according to the feature mapping index, the dimensional reconstruction residual of each dimension of the feature group is calculated, and the target attribute is selected from largest to smallest according to the dimensional reconstruction residual; each target attribute generates an abnormal feature item according to a fixed field structure of "node type, relation edge type, attribute name, current attribute value, normal reference value or interval, deviation direction, and abnormal contribution degree", and the abnormal feature items are arranged from largest to smallest to form an abnormal feature combination. Historical event nodes undergo abnormal pre-screening before entering the event association graph: when a historical event itself has been judged as an abnormal candidate event by the current model or has been confirmed as abnormal, the historical event node does not participate in the message transmission of the temporal association edge, and its time message weight is reset to zero.

[0028] In one implementation, all events that have been verified as normal, as well as historical normal events that have been stratified and supplemented by time-based sampling from the historical normal operation event set, are used as normal reconstruction training samples. The sampling ratio of historical normal replay events does not exceed 30% of the total normal reconstruction training set to avoid time distribution bias in the training samples. Confirmed abnormal events and missed events are used as abnormal constraint samples. Abnormal constraint samples are used to form abnormal separation loss and are not used as normal reconstruction targets, thereby avoiding the feature reconstruction network from learning to reconstruct abnormal patterns.

[0029] This application also provides a data asset anomaly detection system based on deep learning, including an event association module, a feature processing module, a graph processing module, a fusion processing module, and an anomaly detection module, each module being used to perform the corresponding processing in the aforementioned method.

[0030] This application also provides an electronic device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the aforementioned data asset anomaly detection method.

[0031] Due to the adoption of the above technical solution, this application has at least the following beneficial effects.

[0032] First, by directly associating records with common identifiers and performing deterministic matching based on time difference, session identifier, and device identifier when common identifiers are missing, technical records from different data sources such as business applications, database auditing, and terminal management can form a unified running event. By distinguishing between transmission retransmission and actual repeated operations, the interference of repeated collection on access frequency and behavioral sequence characteristics can be reduced.

[0033] Second, by using fixed node slots, default zero vector representation, and missing mask, fixed-dimensional fusion features are still generated even when node types such as users, devices, or network environments are missing, thus preventing the feature reconstruction network from failing to execute due to changes in input dimensions. During the training phase, a missing mask is used to mask the reconstruction error of missing nodes, ensuring that the training objective is consistent with the inference objective.

[0034] Third, by using the message transformation parameters corresponding to the relationship edge type and the time message weight that decays with the time interval, machine aggregation is performed on the association information between the current event and users, data assets, device environment and historical events, so that the anomaly score can reflect the situation where the individual feature does not obviously cross the boundary but the association combination deviates from the normal pattern.

[0035] Fourth, the detection threshold is determined by the upper quantile of the normal score corresponding to the allowable false alarm ratio, and verified by score distribution drift test, single change limit and independent normal verification event, which can reduce the risk of threshold mutation or cyclic update caused by changes in business scoring scale.

[0036] Fifth, the abnormal interpretation objects are determined by the cumulative proportion of group residuals and the cumulative proportion of node occlusion contribution, so that the selection of candidate node types and target contributing nodes has repeatable computational boundaries; by using abnormal samples for separation constraints instead of normal reconstruction targets, the ability to reconstruct normal patterns and distinguish abnormalities can be maintained during incremental updates. Attached Figure Description

[0037] The accompanying drawings are provided to further understand this application and constitute a part of this application. The embodiments shown in the drawings are used to explain this application and do not constitute an undue limitation on the scope of protection of this application.

[0038] Figure 1 A schematic diagram of the operating environment and module architecture of the data asset anomaly detection system provided in this application embodiment;

[0039] Figure 2 A flowchart illustrating the data asset anomaly detection method provided in this application embodiment;

[0040] Figure 3 This is a schematic diagram of an event association diagram provided for an embodiment of this application. Detailed Implementation

[0041] The technical solution of this application will be further described below with reference to the accompanying drawings and embodiments. The following embodiments are used to illustrate the implementation of this application and are not intended to limit the scope of protection of this application.

[0042] In this application, "multiple" means two or more. "Get", "transfer", "invoke", and "connect" can be executed directly or through an interface, log broker, cache, message queue, or communication network.

[0043] Runtime events are machine-processable event objects generated when data assets are accessed, queried, modified, deleted, exported, or called via an interface. Target runtime events are the runtime events currently being detected by the anomaly detection system.

[0044] Event identification information includes event identifier, request identifier, link identifier, session identifier, or business serial number. Spatiotemporal correlation information includes user identifier, data asset identifier, operation type, device identifier, source system, and record time, used to determine whether records from different sources belong to the same operational process when a common event identifier is lacking. Specifically, the "time" dimension requires that the absolute record time difference between records from each source does not exceed a calibrated correlation duration threshold, and the "space" dimension requires that records from each source involve the same user identifier, data asset identifier, and operation type. Correlation can only be established based on spatiotemporal correlation information when both conditions are met. The correlation duration threshold can be saved as a threshold configuration item for different combinations of data sources.

[0045] The feature mapping index records the start and end positions of each running feature in the global feature vector, as well as the corresponding node type, and is used to write the running features into the event association graph. The fusion grouping index records the start and end positions of the target event representation and each node type representation in the fused features, and is used for grouping and reconstructing residual calculation. Both are applied in the node initialization phase and the anomaly scoring phase, respectively.

[0046] The preset node type set includes target event type, user type, data asset type, device type, network environment type, operation type, time status type, and historical event type. Each node type corresponds to a fixed node slot and a fixed representation dimension. When a node type exists, the existence mask value is 1; when a node type does not exist, a zero vector is written in the corresponding fixed node slot as the default node representation, and the existence mask value is 0.

[0047] A confirmed normal event is an event that has been verified as normal; a confirmed false alarm event is an event that was initially identified as an anomaly candidate by the current model but has been verified as normal; a confirmed abnormal event is an event that has been verified as abnormal; a missed event refers to an event that did not trigger an alarm in the current model in historical events but was confirmed as abnormal in subsequent manual audits or business reviews. The discovery of missed events depends on the manual audit process that runs in parallel with the anomaly detection system, and the two operate independently.

[0048] Figure 1 The operating environment of the anomaly detection system is illustrated. Data source systems may include business application systems, database auditing systems, identity authentication systems, access control systems, interface gateways, and terminal management systems. Each data source system sends raw records to the anomaly detection system via interfaces, log brokers, or message queues. The anomaly detection system performs the association, encoding, graph construction, model reasoning, and anomaly detection of the raw records.

[0049] The anomaly detection system includes an event association module, a feature processing module, a graph processing module, a fusion processing module, and an anomaly detection module. Optionally, the anomaly detection system also includes a result interpretation module and a feedback update module. The event association module outputs the target running events; the feature processing module outputs global feature vectors and feature mapping indices; the graph processing module outputs an event association graph with completed node initialization; the fusion processing module outputs fixed-dimensional fusion features and fusion grouping indices; and the anomaly detection module outputs anomaly scores and anomaly detection results.

[0050] Before online detection, the anomaly detection system trains a graph neural network and a feature reconstruction network using historical normal operation events. The training and detection phases use the same set of node types, fixed node slots, category mapping, standardized parameters, relation edge types, representation dimensions, and representation combination order.

[0051] The screening of historical normal operation events follows this process: All historical events within a preset observation period that were not triggered by existing independent detection rules are selected as initial screening samples. These existing independent detection rules refer to pre-deployed detection methods based on fixed thresholds or simple rules (e.g., alarm rules based on single-dimensional thresholds), independent of the anomaly detection method of this application, to ensure that the screening of training data does not depend on the technical solution of this application and to avoid circular dependencies. The initial screening samples are stratified and sampled by time period (e.g., every 4 hours as a time period), user role, and data asset type, with a sampling ratio of no less than 5%. Security operations personnel verify and confirm the sampled data. When the sampling verification accuracy reaches 99% or higher, the verified sampled data is used as historical normal operation events. The normal operation event set is reconstructed quarterly, removing expired data and including recently confirmed normal operation records. When the number of samples confirmed through verification is insufficient to support model training, data augmentation techniques are used to expand the sample size. This data augmentation includes adding small-amplitude random noise following a normal distribution (the noise standard deviation is 0.05 times the standard deviation of the feature) to continuous features and performing random replacements on discrete features (i.e., for discrete features, the feature value is randomly replaced with another category value of that feature in the training set with a probability of 0.05). Non-alarm events are not directly used as all normal training samples to reduce the probability of unidentified anomalies being mixed into the normal training set. The preset observation period is determined based on the historical alarm coverage of the business system. Specifically, it involves counting the number of historical events triggering alarms daily over the past 90 days, selecting a time span where the number of alarms is stable (i.e., the coefficient of variation of the daily alarm number does not exceed 20% over 30 consecutive days) and is less than 120% of the daily average alarm number as the observation period. The observation period is no less than 7 days and no more than 90 days.

[0052] During initial training, for each historical normal operation event, event association, feature encoding, graph node mapping, and graph neural network fusion are performed sequentially to obtain fused features. These fused features are then generated by the feature reconstruction network. The reconstruction error of normal events is used as the primary training objective, and the node encoding parameters, message transformation parameters corresponding to each entity relationship edge type, graph neural network parameters, and feature reconstruction network parameters are updated in reverse. During training, a missing node mask is used to mask the reconstruction error of feature groups corresponding to missing node types. This ensures that the feature reconstruction network only learns to reconstruct actual node types and does not generate gradient updates for the default zero vectors corresponding to missing nodes, thus guaranteeing consistency between the training and inference objectives.

[0053] The graph neural network adopts a graph attention network (GAT) structure, specifically consisting of two graph attention layers. Each layer has a total node representation dimension of 256, with four attention heads. Each attention head outputs 64 dimensions, and the concatenation of these outputs results in a 256-dimensional output. Each entity relationship edge type corresponds to an independent linear transformation matrix W_r and a bias b_r, where W_r has a dimension of 256×256 and b_r has a dimension of 256. The residual update formula after message aggregation at each layer is h_v^(l+1)=h_v^(l)+λ_v^(l)·Aggregation({h_u^(l)|u∈N(v)}). Where v is the node to be updated; u is a neighboring node of v; N(v) is the set of neighboring nodes of node v; l is the graph attention layer number; h_v^(l) and h_v^(l+1) are the node representations of node v at the input and output ends of layer l, respectively; h_u^(l) is the node representation of neighboring node u at layer l; Aggregation(·) is the aggregation result obtained by performing message transformation W_r·h_u^(l)+b_r on the neighboring node representations according to the relation edge type, and then normalizing and weighting the sum according to the GAT attention coefficients; λ_v^(l) is the scalar learnable residual weight corresponding to node v at layer l. λ_v^(l) is initialized to 1.0 and trained with W_r, b_r and other network parameters using the Adam optimizer. The final node representation is taken from the output of the last layer GAT.

[0054] The feature reconstruction network employs an autoencoder structure. Let the number of preset node types be K, the fixed representation dimension of each node type be D_h, and each node type correspond to an existence mask. Then, the fixed dimension of the fused feature is D_F = K·D_h + K; the existence mask for the target event type is fixed at 1. The input layer dimension of the encoder and the output layer dimension of the decoder are both D_F. The encoder consists of fully connected layers of D_F→1024→256→64, and the decoder consists of fully connected layers of 64→256→1024→D_F. The intermediate layers use the ReLU activation function, while the bottleneck layer and output layer do not apply any activation function. A dropout layer with a dropout rate of 0.2 is set after the encoding layer. Therefore, the input and output dimensions of the feature reconstruction network are consistent with the actual fused feature dimensions.

[0055] The unsupervised pre-training of the graph neural network and the feature reconstruction network is conducted as follows: The fused features of historical normal operation events are used as input, and the weighted average of the reconstruction residuals of effective feature groups is used as the pre-training loss. The Adam optimizer is employed with an initial learning rate of 1×10^-3. The training samples are divided into training and validation sets in an 8:2 ratio. Pre-training stops when the reconstruction loss of the validation set no longer decreases after five consecutive training epochs. During pre-training, a missing mask is used to mask the reconstruction errors of feature groups corresponding to missing node types.

[0056] The node representation dimension D_h is selected from the candidate set {128, 256, 512}: Candidate models are trained using the same training and validation sets respectively. The validation set reconstruction loss and average inference time per event are calculated for each candidate model. The smallest D_h is selected when the validation set reconstruction loss is no higher than 1.02 times the minimum validation set reconstruction loss and the allowed inference time is met. The number of attention heads should ensure that the output dimension of each attention head is an integer. D_h, the number of attention heads, and the autoencoder layer width are only reselected through the above offline validation process during system deployment or model version updates, taking effect with the model version and remaining fixed during online detection.

[0057] System cold start: When the system is first deployed and there are no historical normal operation events accumulated, the following method is used to start it:

[0058] Conservative rules are used for detection, while real-world operational data is accumulated. The maximum amount of data returned in a single query and the maximum number of accesses per day are determined based on the business logs from the 30 days prior to deployment: after removing records confirmed as abnormal by independent rules or manual auditing, the average daily returned data and average daily accesses per user are calculated, and 3 times and 5 times these are used as the corresponding upper limits; when available historical business logs are unavailable, the absolute security baseline provided by the business department is used. Non-working hours are exemplarily set to 22:00 to 06:00 the following day. Multiple rules are combined using an "OR" logic, meaning that an alarm is generated when any rule is triggered.

[0059] In the initial stage of system launch, a conservative strategy was adopted for detection thresholds, setting the allowable false alarm ratio ρ to 0.001 (i.e., allowing 1 false alarm in every 1000 normal events). After accumulating no less than 1000 normal events that have been manually verified, the threshold calibration method based on the actual normal score distribution was switched.

[0060] All abnormal candidate events during the cold start phase must be manually reviewed, and the review results are used as the first batch of feedback samples for accumulation.

[0061] Figure 2A data asset anomaly detection method is shown. The main detection process includes steps S100 to S500; when interpreting anomaly candidate events, step S600 is further executed; when a feedback label is obtained or a drift in the score distribution is detected, step S700 is executed based on the sample size condition.

[0062] S100: Acquire multi-source raw records generated during the operation of the data asset; associate the multi-source raw records with event identifier information or spatiotemporal correlation information to obtain the target operation event. The multi-source raw records are machine records from at least two different data source systems.

[0063] Multi-source raw records include at least two of the following: business access records, database audit records, interface call records, identity authentication records, network session records, and terminal status records. Each raw record must include at least the source system, source record identifier, and record time, and may include user identifier, data asset identifier, operation type, event identifier, request identifier, link identifier, session identifier, device identifier, network access area, execution result, execution duration, and returned data volume.

[0064] When records from different sources carry the same event identifier, request identifier, or link identifier, the event association module will group the corresponding records into the same target runtime event. For example, when interface gateway records, business access records, and database audit records carry the same request identifier, the three will be directly associated.

[0065] When no records share a common identifier, the event association module first selects a baseline record based on a preset data source priority. The data source priority is determined during deployment based on how closely a record relates to actual data operations; for example, database audit records take precedence over business access records, and business access records take precedence over terminal status records.

[0066] For the baseline record, the event association module filters candidate records from each of the other data sources that match the user identifier, data asset identifier, and operation type, and whose absolute record time difference is no greater than the association duration threshold T_a. For multiple candidate records from the same data source, the one with the smallest absolute record time difference is selected; if the time differences are the same, the sequence is: session identifier match, device identifier match, and the record with the earliest record time. The selected record is marked as assigned and no longer belongs to other target running events.

[0067] Original records that do not meet the association conditions are not forcibly merged into existing events; they can form target running events independently or enter the association cache. The waiting time in the association cache does not exceed the association time threshold T_a; if no matching record is found after this time, the waiting ends.

[0068] Before the target operation event is formed, the event association module identifies retransmission records. If two records have the same source record identifier, or they have the same source system and content digest, and the absolute record time difference is not greater than the retransmission duration threshold T_r, they are determined to be retransmissions of the same collection or transmission record. The content digest is obtained by performing hash calculation after serializing the operation type, target object, main parameters and execution result according to a fixed field order.

[0069] Merge the retransmission records, and the number of retransmissions is equal to the number of records before merging minus one. After retransmission merging, the same operations with different event identifiers or different execution times that are still retained are repeated operations at the service layer, and are not merged. The number of repetitions and the interval between adjacent operation times are counted in the subsequent historical window. Retransmission determination is performed prior to service repeated operation determination: when a record satisfies both the retransmission merging condition (same source system and content digest, time difference not greater than the retransmission duration threshold T_r) and the service repeated condition (different event identifiers or different execution times), it is processed preferentially as retransmission merging to eliminate the interference of repeated records at the collection level on feature statistics. The retransmission frequency feature and the repeated operation feature respectively represent information at two different levels: collection-level retransmission and service-level repeated operation, and do not conflict with each other. For example, an interface gateway retransmits the same call record twice within 500 ms due to network jitter. Even if the two retransmissions carry different event identifiers, the system still preferentially determines them as retransmission records for merging, rather than treating them as two independent service repeated operations.

[0070] T_a and T_r are determined according to the following offline calibration process. First, complete clock synchronization for each data source, and determine the upper limit of clock synchronization error e_clk according to the clock synchronization log. For cross-source matched record pairs with the same event identifier, request identifier or link identifier, calculate the absolute record time difference to form a set C_a; for confirmed retransmission record pairs with the same source record identifier or the same transmission sequence identifier, calculate the absolute record time difference to form a set C_r. Calculate Q_q(C_a)+e_clk and Q_q(C_r)+e_clk on the quantile level set {0.95, 0.97, 0.99, 0.995} respectively to form an association duration candidate set and a retransmission duration candidate set. Each candidate value is verified using independent manually labeled samples. T_a takes the minimum candidate value with a matching recall rate not lower than 99% and a false association rate not higher than 1%, T_r takes the minimum candidate value with a retransmission identification recall rate not lower than 99% and a false merging rate not higher than 1%, and it is required that T_r<T_a. When the requirements cannot be met simultaneously, T_a is set separately for different data source combinations, and retransmission determination preferentially relies on the source record identifier or the transmission sequence identifier. T_a and T_r are recalibrated only when deployment or data source configuration changes and take effect with the configuration version, and remain fixed during online detection.

[0071] S200 extracts operational features from the target operational events, including event level, user, data asset, device, network environment, operation, time status, and historical behavior data dimensions, and performs unified encoding to obtain a fixed-length global feature vector and feature mapping index.

[0072] Event-level characteristics include returned data volume, execution time, execution result, number of retransmissions, and number of repeated operations; user characteristics include user role, permission level, and authentication status; data asset characteristics include asset type, sensitivity level, and data scale; device characteristics include device identifier, terminal trust status, and device type; network environment characteristics include access area, Internet Protocol address class, network session status, and access method; operation characteristics include operation type, operation object, batch flag, and interface method; time status characteristics include hour category, workday flag, and business shift; historical behavior characteristics include the number of accesses within the historical window, the time interval between repeated operations, and the operation sequence. In actual implementation, at least two of the above eight data dimensions should be extracted as operational characteristics.

[0073] For missing continuous features, write the numerical placeholder determined during the training phase at the corresponding position and set the feature missing flag to 1; if the actual feature value exists, set the feature missing flag to 0. The numerical placeholder is only used to maintain the input dimension and does not represent the actual business value.

[0074] For a continuously running feature x, standardization is performed using the mean μ and standard deviation σ obtained from historical normal running events during the training phase. To avoid a zero standard deviation, a positive stabilizing term ε is set, and the standardized value x_norm is calculated according to the following formula:

[0075] x_norm=(x-μ) / max(σ,ε); where x is the current continuous running feature value, μ and σ are the mean and standard deviation of the feature in historical normal running events, respectively, and ε is a stable term greater than 0.

[0076] Let q_L and q_U be the lower and upper quantile levels, respectively, and Q_(q_L) and Q_(q_U) be the quantile values ​​of continuous operating characteristics at q_L and q_U in historical normal operating events, respectively, with the normal value range being [Q_(q_L), Q_(q_U)]. When x is lower than Q_(q_L) or higher than Q_(q_U), the outlier flag is set to 1; otherwise, it is set to 0. For example, quantile levels q_L = 0.01 and q_U = 0.99, with corresponding lower and upper quantile values ​​of Q_(0.01) and Q_(0.99), respectively.

[0077] Discrete features are converted into category numbers or embedding vectors according to the category dictionary saved during the training phase; unregistered categories are written with unknown category numbers. Access times are converted into time period categories according to hours or business shifts. The interval between adjacent events is standardized in the same way as continuous features. Operation sequences are sorted according to the time of event occurrence and truncated or padded with zeros to a fixed length L. When truncating an operation sequence, the nearest L events are taken, and zero padding is used when padding.

[0078] The encoded runtime features, feature missing flags, retransmission count features, repeated operation features, and outlier flags are combined into a global feature vector in a fixed order. The feature mapping index records the start and end positions, feature type, and corresponding node type for each feature. For example, roles and permissions are mapped to user nodes, asset types and sensitivity levels are mapped to data asset nodes, and returned data volume and execution results are mapped to target event nodes.

[0079] The repetitive operation feature is generated as follows: a history window is set to the 100 most recent event records or the operation records within the last 10 minutes, whichever comes first; the number of operation records of the same type as the current operation is counted within the history window (cnt), and the average time interval (seconds) between adjacent repetitive operations is calculated (avg_interval); if there are no repetitive operations within the history window (cnt=1, i.e., only the current operation itself is included), then cnt is set to 0 and avg_interval is set to a placeholder value of -1; the repetitive operation feature is a tuple (cnt, avg_interval), which is encoded and then concatenated with other features.

[0080] S300 constructs an event association graph based on the relationship between the target running event and the associated objects, and completes the initialization of fixed node slots.

[0081] like Figure 3 As shown, the event association diagram is centered on the target event node and includes user nodes, data asset nodes, device nodes, network environment nodes, operation type nodes, time status nodes, and historical event nodes. User nodes represent the operation subject, data asset nodes represent the accessed data objects, and device nodes and network environment nodes represent the current access technology conditions.

[0082] The selection criteria for historical event nodes are as follows: the historical running event and the target running event share at least one of the same user identifier, device identifier, or data asset identifier, and the time interval between them is no greater than the time-series association window T_w. T_w is calibrated offline using historical normal running event pairs: the time intervals of adjacent historical normal running event pairs sharing at least one of the above identifiers are statistically analyzed to form a time interval set C_w, and the 0.95 quantile of C_w is determined as T_w. Historical running events meeting the criteria are sorted in ascending order of time interval, and events not exceeding the maximum number of historical nodes N_h are selected; N_h is taken as the 0.95 quantile of the number of candidate events within the T_w window of historical normal running events, and is constrained by the system's maximum allowed number of nodes. T_w and N_h are fixed with the model version.

[0083] Before entering the event association graph, historical event nodes undergo anomaly pre-screening: if a historical event itself has been identified as an anomalous candidate event by the current model or has been confirmed as an anomalous event, the historical event node does not participate in the message passing of the temporal association edges, and its corresponding time message weight is reset to zero, in order to prevent anomalous events from affecting the representation of the current event through temporal association edges. This pre-screening does not affect the existence of historical event nodes as part of the graph structure, but only affects their degree of participation in message passing.

[0084] An operation subject relationship edge is established between the target event node and the user node; an operation object relationship edge is established between the target event node and the data asset node; an equipment source relationship edge is established between the target event node and the device node; a network source relationship edge is established between the target event node and the network environment node; an operation category relationship edge is established between the target event node and the time status node; and a time sequence relationship edge is established between the target event node and the historical event node.

[0085] The graph processing module writes each running feature into the corresponding node according to the feature mapping index, and converts the node features of different original dimensions into a fixed node representation dimension D_h through the node encoding layer. D_h is determined according to the aforementioned offline verification method for candidate dimensions. Each preset node type reserves a fixed node slot; when a certain node type does not exist, a zero vector of dimension D_h is written into the corresponding slot as the default node representation, and the existence mask of that node type is set to 0.

[0086] S400 performs relational message transformation, time weighting, and node type aggregation based on the event association graph to obtain fixed-dimensional fusion features.

[0087] Let h_u be the node representation of the associated node u, v be the target event node, r be the relation edge type, and W_r and b_r be the corresponding message transformation matrix and bias, respectively. Then the entity relation message m_(u→v)^r is calculated according to the following formula:

[0088] m_(u→v)^r=W_r·h_u+b_r; where W_r·h_u represents the product of matrix W_r and column vector h_u, and b_r is the bias vector with the same dimension as the message.

[0089] For time-series related edges, let the time interval between historical events and the target running event be Δt, the time decay scale be τ, and the time message weight ω_t be determined according to the following formula:

[0090] ω_t=exp(-Δt / τ); where Δt≥0, τ>0. τ is determined according to the time-series correlation window T_w, and is exemplarily set to τ=T_w / ln(10), so that the time message weight when the time interval is equal to T_w is 0.1.

[0091] The time-series relation message is equal to the scalar multiplication result of ω_t and the corresponding relation message, i.e., m_time = ω_t·m_(u→v)^r, where the midpoint "·" indicates multiplication. Since ω_t monotonically decreases as Δt increases, historical events closer to the target running event have a larger message weight.

[0092] For multiple relational messages of the same preset node type g, the node type representation z_g is obtained by average aggregation; while keeping the output dimension fixed, a normalized weighted average with a weight sum of 1 can also be used. When the node type does not exist, z_g is a D_h-dimensional zero vector z_g^0, and the mask a_g is 0 when the node type exists; when the node type exists, a_g is 1.

[0093] Let the number of preset node types be K, and the representation dimension of each node type be D_h. The K node type representations and their K existence masks are concatenated into a fusion feature F according to a fixed node type order. Then, the fixed dimension of F is D_F = K·D_h + K. The existence mask for the target event type is fixed at 1. The fusion group index records the start and end positions of each node type representation group in F; the existence mask is retained as an independent input dimension but is not included in the reconstruction residual of the corresponding node representation group.

[0094] The S500 takes fixed-dimensional fused features as input to the feature reconstruction network, calculates the grouped reconstruction residuals, anomaly scores and detection thresholds, and outputs anomaly detection results.

[0095] The feature reconstruction network employs an autoencoder, comprising an encoder, a latent layer, and a decoder. The encoder transforms the D_F-dimensional fused feature F into latent features, and the decoder generates a reconstructed feature F_prime with the same dimensions as F, where each dimension of F_prime corresponds one-to-one with F.

[0096] The grouping weights are determined as follows: Let the total number of effective feature groups be G. After model training is complete, the grouping reconstruction residuals of each feature group are calculated based on a historical normal validation set that is independent of the training set. The mean and standard deviation of the residuals of the g-th feature group on the validation set are μ_g and σ_g, respectively. The grouping weight w_g = 1 / max(μ_g + σ_g, ε_w), where g ∈ {1, 2, ..., G}, and ε_w is a stable term greater than 0, exemplarily taken as 1 × 10^-6. The grouping weights remain fixed during the online detection phase of the same model version; they are recalculated synchronously based on the new validation set when the model is updated.

[0097] Let the g-th feature group contain d_g representation dimensions that do not include those with masks, and let F_(g,i) and F_prime_(g,i) be the fused value and reconstructed value of the i-th dimension of the feature group, respectively. Then the group reconstruction residual R_g is calculated according to the following formula:

[0098] R_g=(1 / d_g)·Σ_(i=1)^(d_g)[F_(g,i)-F_prime_(g,i)]^2; where d_g is the number of dimensions of the g-th feature group, and i is the dimension index within the feature group.

[0099] Let w_g be the grouping weight of the g-th feature group, and a_g be the presence mask of that feature group. If a_g of the target event feature group is fixed at 1, then the anomaly score A is calculated according to the following formula:

[0100] A = {Σ_(g=1)^G[a_g·w_g·R_g]} / {Σ_(g=1)^G[a_g·w_g]}; where G is the total number of feature groups. Since a_g of the target event feature group is fixed at 1 and w_g>0, the denominator is always greater than 0.

[0101] The same model version uses the same grouping weights during training, threshold calibration, and online detection. The a_g value for missing node types is 0; the default zero vector for this node type does not participate in anomaly score calculation. Anomaly score A is calculated independently of the fused and reconstructed features, without depending on the detection threshold; the detection threshold is only used to label consecutive anomaly scores as anomaly candidates or to indicate that the anomaly candidate condition has not been met.

[0102] A baseline normal score set S_0 is established based on the anomaly scores of historical normal operation events. Let ρ be the allowable false alarm rate for normal events, then the upper quantile level q_T = 1 - ρ, and the initial detection threshold T_0 = Q_(q_T)(S_0). Here, Q_(q_T)(S_0) represents the quantile value of set S_0 at quantile level q_T; for example, when ρ = 0.01, q_T = 0.99, and T_0 takes the 0.99 quantile value of S_0.

[0103] The normal scoring sliding window only accepts abnormal scores recalculated under the current model for confirmed normal events and confirmed false alarm events. The window contains the most recent M scores, where M is between 200 and 1000 and is predetermined based on business throughput. Let the window score set be S_w, the empirical cumulative distribution function of S_w be F_w(s), and the empirical cumulative distribution function of the baseline normal score set S_0 be F_0(s). Then, the maximum cumulative distribution difference D is determined according to the following formula:

[0104] D = max_(s∈S_w∪S_0)|F_w(s)-F_0(s)|; where s is the abnormal rating value, F_w(s) represents the ratio of the number of ratings in S_w that are not greater than s to the total number of samples in S_w, F_0(s) represents the ratio of the number of ratings in S_0 that are not greater than s to the total number of samples in S_0, and max represents finding the maximum value among all different rating values ​​in S_w and S_0.

[0105] D is the two-sample Kolmogorov-Smirnov statistic. A drift in the scoring distribution is determined when the sample size of S_w is not less than the minimum sample size N_D for the drift test and D is greater than the drift threshold δ_D; no drift conclusion is drawn when the sample size is insufficient. For example, N_D is set to 200. δ_D is selected between 0.12 and 0.20 using independent validation data: validation data includes normal scoring sequences that maintain the original business distribution and known distribution change sequences formed by time period switching, changes in user role ratios, or changes in data asset type ratios; the minimum value of δ_D is selected from the candidate δ_D values ​​that minimizes the number of false triggers to maintain the original distribution sequence and can trigger all known distribution change sequences, and remains fixed during the same model version.

[0106] When the threshold update cycle ends or the score distribution drifts, the candidate detection threshold T_c is determined based on the quantile value of S_w at the quantile level q_T. Let the current detection threshold be T_cur, and the maximum single change be ΔT_max. When |T_c-T_cur|>ΔT_max, the threshold to be verified T_val=T_cur+sign(T_c-T_cur)·ΔT_max; otherwise, T_val=T_c. ΔT_max is taken as a preset proportion of the interquartile range of the baseline normal score set, exemplarily set to 0.5 times the interquartile range, and remains fixed during the same model version.

[0107] The threshold to be verified is validated using confirmed normal verification events that are independent of the threshold calculation samples. These independent normal verification events are selected as follows: events from historical normal operation events that do not overlap with the time of the threshold calculation samples and have not been used for any model training or threshold calibration are selected as verification events; the set of verification events is reconstructed each time the threshold is calibrated to ensure independence from the threshold calculation samples. When the proportion of abnormal scores reaching the threshold to be verified in normal verification events is not greater than the allowable false alarm proportion ρ, the threshold to be verified takes effect in the next detection cycle; otherwise, the current detection threshold continues to be used. The current event always uses the threshold that was already in effect before the start of the current detection cycle to avoid the current event participating in both threshold calculation and its own judgment.

[0108] When the anomaly score A reaches or exceeds the current detection threshold, the target running event is identified as an anomaly candidate event; when A is below the current detection threshold, it is identified as an event that does not meet the anomaly candidate condition. The grouped reconstruction residuals of the anomaly candidate events, the event correlation graph, and the anomaly score are sent to the result interpretation module.

[0109] S600 performs group filtering and node message masking for abnormal candidate events, and generates abnormal explanation results.

[0110] First, feature groups with a mask of 0 are removed. The residuals from each group are then sorted from largest to smallest and accumulated sequentially until the ratio of the accumulated residual to the sum of all valid group residuals reaches the residual contribution ratio η_g. The node types corresponding to the feature groups participating in the accumulation are then determined as candidate node types. Feature groups with the same cutoff residual are also included. η_g is selected from the candidate set {0.7, 0.8, 0.9} of the independent interpretation validation set, choosing the candidate value that covers the main anomalies labeled manually and has the smallest average number of candidate nodes, and this value is fixed with each model version.

[0111] For each candidate node type, each node of that type in the local event subgraph is identified as a candidate associated node. When performing message masking, the relationship edges between the candidate associated nodes and the target event node are temporarily removed from the adjacency matrix of the local event subgraph, while the remaining nodes, relationship edges, model parameters, and detection thresholds remain unchanged. The graph neural network forward propagation and feature reconstruction are then re-executed to obtain the masked anomaly score A_j_mask.

[0112] The anomaly contribution C_j of candidate associated node j is calculated according to the following formula:

[0113] C_j=max(A-A_j_mask,0); where A is the anomaly score before occlusion, A_j_mask is the anomaly score recalculated after occluding the relation edge corresponding to the candidate associated node j, and max(·) is used to truncate the negative contribution to 0.

[0114] Nodes are sorted in descending order of C_j and accumulated sequentially until the ratio of the accumulated abnormal contribution to the sum of all positive abnormal contributions reaches the node contribution ratio η_c. Nodes participating in the accumulation are then identified as target contributing nodes. Nodes with the same contribution at the cutoff position are also selected. η_c is determined from the candidate set {0.7, 0.8, 0.9} using the same offline verification method as η_g, and remains fixed with the model version.

[0115] For each target contribution node, the corresponding feature group is determined based on the feature mapping index. For each running feature dimension k within the feature group, the dimension reconstruction residual e_(j,k)=[F_(j,k)-F_prime_(j,k)]^2 is calculated, and target attributes are selected from largest to smallest e_(j,k) until the sum of the selected dimension reconstruction residuals accounts for 0.8 of the total residuals of the feature group; attributes with the same cutoff position residuals are included together. An anomalous feature item is generated for each target attribute, using a fixed field structure: node type, relation edge type, attribute name, current attribute value, normal reference value or interval, deviation direction, and anomalous contribution degree. The normal reference value or interval is taken from the class mode or [Q_(q_L), Q_(q_U)] saved during the training phase, and the deviation direction is determined based on the current attribute value relative to the normal reference value or interval. Each abnormal feature item is arranged from largest to smallest according to its abnormal contribution. When the contribution is the same, they are arranged in the order of user, device, network environment, time status, operation type, data asset, and fixed node type of historical event. The set of abnormal features after the arrangement constitutes a structured abnormal feature combination.

[0116] Structured anomaly feature combinations can be further converted into natural language descriptions using preset templates. For example, when the anomaly feature items for the node type (device), time status, and operation type are "device identifier deviates from the common device set," "access time is during non-working hours," and "operation type is batch export," respectively, the template outputs "uncommon device—non-working hours—batch export." This natural language description is merely a display of the structured anomaly feature items and is not directly equivalent to the confirmed root cause of the anomaly. Starting from the target event node, preceding historical events are retrieved along the relational edges and sorted by time to form an anomaly tracing path. The structured anomaly feature combinations, the target contributing node, and the anomaly tracing path together constitute the anomaly explanation result.

[0117] The S700 performs threshold calibration, candidate model training, validation, and switching based on feedback labels and score distribution drift.

[0118] Feedback samples are accumulated over a 7-day period. Let the minimum number of samples for threshold calibration be N_T, the minimum number of samples for normal training be N_n, and the minimum number of samples for anomaly constraints be N_a. For example, N_T = 200, N_n = 1000, and N_a = 50. Threshold calibration can be performed when the total number of confirmed normal events and confirmed false positive events reaches N_T; candidate model training is only performed when the total number of confirmed normal events and confirmed false positive events reaches N_n, and the total number of confirmed anomaly events and missed events reaches N_a. If the corresponding quantity requirements are not met, the current detection threshold or the current model remains unchanged.

[0119] When only a score distribution drift is detected, the total number of confirmed normal events and confirmed false alarm events is checked first. If the total number reaches N_T, the candidate detection threshold is determined and verified according to step S500; if the total number does not reach N_T, the current detection threshold is kept unchanged and verification samples are collected again. Score distribution drift itself does not directly trigger model parameter updates.

[0120] Before being used for threshold calibration or incremental training, feedback samples (including confirmed normal events, confirmed false positives, confirmed abnormal events, and missed events) need to undergo forward propagation calculations again using the current model to obtain abnormal scores under the current model parameters. Because the parameters change before and after the model update, there may be a systematic scale shift between the abnormal scores under the old model and the score distribution of the new model; directly using the old scores for threshold calibration will introduce bias. The recalculated scores are used to update the normal score sliding window, determine candidate detection thresholds, and calculate the loss for incremental training.

[0121] When the total number of confirmed normal events and confirmed false alarms reaches N_n, and the total number of confirmed abnormal events and missed events reaches N_a, a normal reconstruction training set D_n is formed by combining all events confirmed as normal through review and historical normal events sampled stratified by time from the historical normal operation event set. An abnormal constraint set D_a is formed by combining confirmed abnormal events and missed events. The sampling ratio of historical normal replay events does not exceed 30% of the total D_n. N_n is greater than N_T because model parameter updates involve more learnable parameters and require more normal samples than threshold calibration. Model parameter updates are not initiated if the number of samples in either category is insufficient, and samples in D_a are not used as normal reconstruction targets.

[0122] Let A_n be the anomaly score of the nth normal sample in D_n. The normal reconstruction loss L_rec is calculated according to the following formula:

[0123] L_rec = (1 / |D_n|)·Σ_(n∈D_n)A_n; Let the anomaly score of the a-th anomalous sample in D_a be A_a, and the preset anomaly interval be γ, then the anomaly separation loss L_sep = (1 / |D_a|)·Σ_(a∈D_a)max(0, γ-A_a). Where |D_n| and |D_a| are the sample sizes of the two sample sets, respectively. γ is determined based on the current normal detection threshold T_cur and the normal score standard deviation σ_A, for example, γ = T_cur + 0.5·σ_A.

[0124] The training loss L of the candidate model is determined according to the following formula:

[0125] L = L_rec + η_sep·L_sep + η_reg·L_reg; where η_sep is the anomaly separation loss weight, η_reg is the regularization loss weight, and L_reg is the L2 regularization term of the model parameters.

[0126] η_sep and η_reg are selected offline using an independent validation set. For example, η_sep takes values ​​in {0.1, 0.5, 1.0}, and η_reg takes values ​​in {1×10^-5, 1×10^-4, 1×10^-3}. Candidate models are trained separately, and among candidate combinations where the false positive rate for normal validation events is no greater than the allowable false positive ratio ρ and the average inference time per event does not exceed the allowable inference time, the weight combination with the highest abnormal event recognition rate is selected. The selected weights are fixed with each candidate model version.

[0127] The updated graph neural network and feature reconstruction network are used as candidate models. A new candidate normal score reference set is established based on the anomaly scores of the candidate models on independent normal validation events, and the candidate detection threshold is determined according to the allowable false alarm ratio. It is confirmed that anomaly validation events are only used to evaluate the recognition rate and do not participate in the calculation of the normal threshold quantile.

[0128] The candidate model and the current model process the same validation events in parallel, and the false positive rate (FPR) for normal events and the test positive rate (TPR) for abnormal events are calculated separately. Let κ_FPR be the evaluation weight for the normal event false positive suppression index, and κ_TPR be the evaluation weight for the abnormal event detection rate. Then, the overall performance score S of the candidate model is calculated according to the following formula:

[0129] S = κ_FPR·(1-FPR) + κ_TPR·TPR; where κ_FPR ≥ 0, κ_TPR ≥ 0, and κ_FPR + κ_TPR = 1. When false alarm suppression and anomaly identification are equally important, κ_FPR = κ_TPR = 0.5; when the business security baseline specifies that the false alarm rate takes priority, κ_FPR is increased and fixed in the model validation configuration.

[0130] When the overall performance score of a candidate model is higher than that of the current model, the false positive rate of the candidate model in normal verification events is no greater than the preset upper limit false positive rate, and the inference time of the candidate model does not exceed 1.2 times the allowed inference time, the candidate model and candidate detection threshold will take effect in the next detection cycle; otherwise, the current model and current detection threshold will be retained. This determination simultaneously constrains overall performance, false positive level, and inference latency, avoiding the reliance on weighted scoring alone to mask the degradation of individual indicators.

[0131] After a model switch, the previous version of the current model and its corresponding thresholds are retained. If the false positive rate of a candidate model exceeds the rollback false positive rate (e.g., 1.5 times the allowed false positive rate) or the inference time exceeds the allowed inference time within the observation window (e.g., 7 days) after the candidate model goes live, the previous model version is restored. The model update results do not change the historical event detection results that have already been completed.

[0132] The following is an example of the process. A user typically queries order data during work hours using a fixed office device. This time, in the early morning, they initiate a batch query using another device already connected to the corporate network. The identity authentication system, business application system, database auditing system, and terminal management system each generate records.

[0133] The API call record, business access record, and database audit record contain the same request identifier and are therefore directly associated. The terminal status record does not contain this request identifier, but the user identifier and device identifier match, and the time difference with the baseline record is 0.4 seconds, which is less than T_a=3 seconds, so it is associated with the same target runtime event.

[0134] The graph processing module establishes target event nodes, user nodes, order data asset nodes, device nodes, network environment nodes, batch query operation nodes, early morning time status nodes, and historical event nodes. If the network session status is not obtained this time, the fixed slot of the network environment node type is written with the default representation of the zero vector, the network environment exists with a mask set to 0, and the fusion feature dimension remains unchanged.

[0135] As an illustrative calculation, the grouped reconstruction residuals of the six valid characterization groups are 0.05, 0.04, 0.03, 0.55, 0.60, and 0.08, respectively, with a grouping weight of 1 for each group. Therefore, the anomaly score is the average of the six residuals, which is 0.225. At the current detection threshold of 0.18, this event is identified as a candidate anomaly.

[0136] Accumulating the residuals based on η_g=0.8, the sum of the residuals for the time state group and the device group is 1.15, accounting for 85.2% of the total effective residuals of 1.35. Therefore, the time state type and device type are identified as candidate node types. After occluding the time state node and device node, the anomaly scores decreased by 0.10 and 0.08 respectively. The cumulative contribution of the two exceeds η_c=0.8 of the total positive contribution, therefore, they are identified as target contribution nodes.

[0137] If the recent normal scoring sliding window contains 300 confirmed normal or confirmed false alarm events, and the calculated D=0.16, which is greater than δ_D=0.12 (drift judgment threshold), then the scoring distribution is determined to have drifted. The upper quantile of the sliding window at 0.99 is 0.21, the current threshold is 0.18, and the maximum single change is 0.03; therefore, the threshold to be verified is 0.21. Its false alarm rate on independent normal verification events is 0.8%, which is not greater than the allowable false alarm rate of 1%, so 0.21 will take effect in the next detection cycle. The above values ​​are used to illustrate the calculation chain and do not constitute fixed parameter limitations.

[0138] If the verification confirms that the access was a temporary authorization operation, a false positive label is generated and the account is added to the normal reconstruction training set; if the account is confirmed to have been used by an unauthorized entity, an abnormal label is generated and the account is added to the abnormal constraint set. Feedback labels are not prerequisites for calculating the abnormal score in this instance; they are only used for threshold calibration or model updates in subsequent cycles.

[0139] This application also provides an electronic device, including a processor and a memory. The memory stores a computer program, model parameters, category mappings, normalization parameters, fixed node slot configurations, feature mapping index configurations, fusion grouping index configurations, and detection thresholds. When the processor executes the computer program, it implements the aforementioned data asset anomaly detection method.

[0140] This application embodiment may also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned data asset anomaly detection method.

[0141] The association duration threshold T_a, retransmission duration threshold T_r, and temporal association window T_w are determined according to the time difference sample quantiles and independent labeled sample validation process described in this embodiment; the node representation dimension D_h is selected according to the reconstruction loss and inference time of the candidate dimension validation set; the outlier detection quantile levels q_L, q_U, and detection threshold quantile level q_T are determined according to the preset normal coverage range and the allowable false alarm ratio, respectively; the loss weights η_sep and η_reg are selected according to the independent validation set. The above parameters are only calibrated offline when the system is deployed, the data source configuration changes, or the model version is updated. After verification, they take effect with the new configuration or new model version and remain fixed during the online detection of the corresponding version; the online detection threshold is only updated according to the sliding window, drift test, change limit, and independent validation process described in step S500.

Claims

1. A data asset anomaly detection method based on deep learning, characterized in that, The method, executed by the anomaly detection system in the data asset management platform, includes: Acquire multi-source raw records generated during the operation of data assets. The multi-source raw records include at least two of the following: business access records, database audit records, interface call records, identity authentication records, network session records, and terminal status records. Each raw record includes at least the source system, source record identifier, and record time. Based on the event identifier information or spatiotemporal correlation information in the multi-source raw records, associate the multi-source raw records to obtain the target operation event. Extract operational features from at least two of the eight data dimensions (event level, user, data asset, device, network environment, operation, time status, and historical behavior) from the target operational event. Unify the operational features to obtain a fixed-length global feature vector. Establish a feature mapping index to record the position of each operational feature in the global feature vector and the corresponding node type. Based on the association between the target running event and the associated object, an event association graph is generated, including the target event node, entity association node, and historical event nodes that meet the temporal association conditions. The running features are mapped to the node features of the corresponding nodes according to the feature mapping index. When the preset node type does not exist, a default node representation and a missing mask are set for the node type. The default node representation adopts a zero vector. Based on the event association graph, the node features are transformed according to the relationship edge type by the graph neural network, and the relationship messages of the temporal association edges are weighted according to the time interval between the historical events and the target running event. The relationship messages of each preset node type are aggregated, and the node type representation or default node representation corresponding to each preset node type and the missing mask are combined in a fixed order to obtain fixed-dimensional fusion features and fusion grouping index. The fused features are input into a feature reconstruction network trained based on historical normal operation events to obtain reconstructed features. When training the feature reconstruction network, the missing mask is used to mask the reconstruction error of the feature group corresponding to the missing node type. The group reconstruction residual between the fused features and the reconstructed features is calculated according to the fused group index. An anomaly score is determined according to each group reconstruction residual and the missing mask. The anomaly score is compared with the detection threshold to obtain the anomaly detection result of the target operation event. Based on the event identifier information or spatiotemporal correlation information in the multi-source original records, the multi-source original records are correlated to obtain the target running event, including: When original records from different sources contain the same event identifier, request identifier, or link identifier, the original records containing the same identifier will be associated with the same target running event; When original records from different sources do not contain the same identifier, a baseline record is determined according to a preset data source priority. Original records with the same user identifier, data asset identifier, and operation type as the baseline record and a recording time difference not greater than the association duration threshold are identified as candidate records. From the candidate records corresponding to each data source, the candidate record with the smallest recording time difference is selected. When the recording time differences are the same, the associated records are determined in sequence according to session identifier matching, device identifier matching, and recording time order, and each original record is restricted to belong to only one target running event. The association duration threshold is pre-calibrated based on the absolute recording time difference sample of cross-source matching records carrying common identifiers and remains fixed during online detection of the same model version. Before forming the target running event, original records with the same source record identifier, or original records with the same source system and content digest and a record time difference not greater than the retransmission duration threshold, are identified as retransmission records. The retransmission records are merged, and the number of records before merging is reduced by one to obtain the retransmission count. The merged records, the retransmission count, and the source information are written into the target running event. The retransmission duration threshold is pre-calibrated based on the absolute record time difference sample of the confirmed retransmission records and remains fixed during online detection of the same model version.

2. The method according to claim 1, characterized in that, Extracting operational features from different data dimensions from the target operational event, and uniformly encoding the operational features, including: For running features with missing feature values, write a numerical placeholder at the corresponding feature position and set a feature missing flag to indicate whether the feature value is missing. The retransmission count is encoded as a retransmission count feature. For identical operation records with different event identifiers or different execution times that are still retained after retransmission merging, a repeated operation feature is generated based on the number of repeated operations within the historical window and the time interval between adjacent repeated operations. The repeated operation feature is a tuple (cnt, avg_interval), where cnt is the cumulative number of occurrences of the same operation type within the historical window, and avg_interval is the average time interval between adjacent repeated operations. If there are no repeated operations within the historical window, cnt is set to 0 and avg_interval is set to a placeholder value of -1. The historical window is the most recent 100 event records or operation records within the most recent 10 minutes, whichever comes first. The retransmission count feature and the repeated operation feature respectively represent information from two different levels: retransmission at the acquisition layer and repeated operations at the service layer. The normal value range is determined based on the lower and upper quantile values ​​of continuous operation features in historical normal operation events. When the current continuous operation feature exceeds the normal value range, an outlier flag is set. The continuous operation features are standardized using the mean and standard deviation saved during the training phase. The discrete operation features are categorically encoded. The access time is converted into a time period category, the interval between adjacent events is converted into a standardized interval value, and the operation sequence is truncated or padded to a preset sequence length. When the operation sequence is truncated, the nearest L events are taken. When padded, zero padding is used. The global feature vector is obtained by combining the encoded running features, the feature missing flag, the retransmission count feature, the repeated operation feature, and the outlier flag in a fixed dimension order, and the feature mapping index is established according to the position of each feature and the corresponding node type.

3. The method according to claim 2, characterized in that, Based on the association relationships between the target runtime event and associated objects, an event association graph is generated, including the target event node, entity association nodes, and historical event nodes that meet the temporal association conditions, including: The target event node is set, and entity association nodes are set to represent users, data assets, devices, network environments, operation types, and time states, respectively. Historical running events that have at least one of the same user identifier, device identifier, or data asset identifier as the target running event and whose time interval is not greater than the time-series association window are set as historical event nodes. The time-series association window is pre-calibrated based on the time interval samples of event pairs that share the user identifier, device identifier, or data asset identifier in the historical normal running events, and remains fixed during online detection of the same model version. Establish entity relationship edges based on the relationship between the target running event and the objects represented by each entity associated node, and establish temporal relationship edges between the target event node and the historical event node; Fixed node slots are set for each preset node type. Event-level running features are written to the target event node according to the feature mapping index. Running features corresponding to associated objects are written to the corresponding entity associated nodes. Event-level running features of historical running events are written to the corresponding historical event nodes. When any preset node type does not exist, a zero vector is written to the fixed node slot corresponding to that node type as a default node representation, and the corresponding missing mask is set to the missing state.

4. The method according to claim 3, characterized in that, Based on the event association graph, a graph neural network is used to transform node features according to the type of relational edge, and the relational messages of the temporal association edges are weighted according to the time interval between historical events and the target running event. The relational messages of each preset node type are aggregated, and the node type representation or default node representation corresponding to each preset node type and the missing mask are combined in a fixed order to obtain fixed-dimensional fusion features and fusion grouping indexes, including: Centered on the target event node, nodes and relational edges within a preset association layer are extracted from the event association graph to obtain a local event subgraph, wherein the preset association layer is equal to the number of layers in the graph neural network; For each type of entity relationship edge in the local event subgraph, the corresponding message transformation parameters are called to generate a relationship message; for the temporal association edge, the time message weight is determined by an exponential decay function that monotonically decreases as the time interval increases, based on the time interval between the historical event node connected by the temporal association edge and the target event node, and the corresponding relationship message is weighted by the time message weight; Average aggregation is performed on relational messages corresponding to the same preset node type to obtain the node type representation of that node type; when the preset node type does not exist, the corresponding zero vector default node representation is used as the node type representation, and the corresponding missing mask is kept in the missing state. The event representation of the target event node, the node type representation of each preset node type, and the corresponding missing mask are combined in a fixed order to obtain the fixed-dimensional fusion feature. The start and end positions of each representation in the fusion feature are recorded to obtain the fusion group index.

5. The method according to claim 4, characterized in that, Determining the anomaly score and the detection threshold includes: According to the fusion grouping index, the fusion feature and the reconstructed feature are divided into multiple feature groups respectively; for each feature group, the average of the squared differences of the corresponding dimensions is calculated to obtain the grouped reconstruction residual of the feature group; Based on the preset grouping weights and corresponding missing masks of each feature group, the weighted average of the group reconstruction residuals of feature groups that are not in a missing state is calculated to obtain the anomaly score. The anomaly score is calculated independently by the feature reconstruction network and its value does not depend on any detection threshold. A normal score reference set is established based on the anomaly scores of historical normal operation events, and the detection threshold is the upper quantile value of the normal score reference set. The upper quantile level is determined based on the allowable false alarm rate of normal events, and the quantile value of the normal score reference set at the upper quantile level is determined as the initial detection threshold. When the threshold update cycle ends or the score distribution drifts, a candidate detection threshold is determined based on the quantile value of the normal score sliding window at the same upper quantile level, and the single change of the candidate detection threshold relative to the current detection threshold is limited to a preset maximum change. The size of the normal score sliding window is fixed as the scores of the most recent M events that have been verified as normal, and the value of M ranges from 200 to 1000. Whether the scoring distribution has drifted is determined based on whether the maximum cumulative distribution difference between the recent normal scoring distribution and the baseline normal scoring distribution is greater than a drift determination threshold. The drift determination threshold is set to a fixed value between 0.12 and 0.20, determined during system deployment and remaining unchanged during operation. When the false alarm ratio of the candidate detection threshold on independent normal verification events is not greater than the allowable false alarm ratio, the candidate detection threshold is made effective in the next detection cycle; otherwise, the current detection threshold is maintained. When the abnormal score reaches or exceeds the current detection threshold, the target running event is determined as an abnormal candidate event.

6. The method according to claim 5, characterized in that, Generate anomaly explanation results for the aforementioned candidate anomalies, including: The feature groups that are not in a missing state are sorted from largest to smallest according to the group reconstruction residuals. The group reconstruction residuals of each feature group are accumulated sequentially until the ratio of the accumulated residuals to the sum of the residuals of all valid feature groups reaches the preset residual contribution ratio. The node types corresponding to the feature groups that participate in the accumulated residuals are determined as candidate node types. The associated nodes corresponding to the candidate node types are determined as candidate associated nodes. For each candidate associated node, while keeping other nodes, model parameters and detection thresholds unchanged, the edge between the candidate associated node and the target event node is temporarily removed from the adjacency matrix of the local event subgraph, the forward propagation of the graph neural network is re-executed, and the anomaly score after occlusion is recalculated. The positive value obtained by subtracting the abnormal score after occlusion from the abnormal score before occlusion is determined as the abnormal contribution of the corresponding candidate associated node; the abnormal contribution is sorted from largest to smallest and accumulated sequentially until the ratio of the accumulated abnormal contribution to the sum of all positive abnormal contributions reaches the preset node contribution ratio, and the candidate associated nodes participating in the accumulation are determined as the target contribution nodes. For each target contribution node, the feature group corresponding to the target contribution node is determined according to the feature mapping index. The running feature with the largest dimension reconstruction residual in the feature group is selected as the target attribute. Anomaly feature items are generated in the order of fixed fields: node type, relation edge type, attribute name, current attribute value, normal reference value or interval, deviation direction, and abnormal contribution degree. The abnormal feature items are arranged in descending order of abnormal contribution degree, and the set of arranged abnormal feature items is determined as the abnormal feature combination. The preceding running events associated with the target running event are retrieved along the entity relation edge and the temporal correlation edge. An anomaly tracing path is generated according to the event time order. The abnormal feature combination and the anomaly tracing path are used as the anomaly explanation result.

7. The method according to claim 5, characterized in that, In response to receiving feedback labels or detecting a drift in the rating distribution, threshold calibration and model updates are performed, including: When a drift in the scoring distribution is detected, if the number of confirmed normal events and confirmed false alarm events in the normal scoring sliding window reaches the preset threshold calibration minimum sample size, then the candidate detection threshold is determined and verified; if the preset threshold calibration minimum sample size is not reached, then the current detection threshold and the current model remain unchanged. When the number of confirmed normal events and confirmed false alarm events reaches the preset minimum number of normal training samples, and the number of confirmed abnormal events and missed events reaches the preset minimum number of abnormal constraints, all events that have been verified as normal and historical normal events that have been stratified and supplemented by time sampling from the historical normal operation event set are combined into a normal reconstruction training set, and confirmed abnormal events and missed events are combined into an abnormal constraint set, and the events in the abnormal constraint set are not used as normal reconstruction targets. The normal reconstruction loss is determined based on the reconstruction error of the normal reconstruction training set, the anomaly separation loss is determined based on the difference between the anomaly score and the preset anomaly interval in the anomaly constraint set, and the graph neural network and the feature reconstruction network are updated according to the weighted sum of the normal reconstruction loss and the anomaly separation loss to obtain the candidate model. The candidate detection threshold is re-determined based on the anomaly score of the candidate model on normal verification events; the candidate model and the current model are processed in parallel for the same verification events, and a comprehensive performance score is calculated for each, which is determined based on the false positive rate of normal events and the recognition rate of anomalies; when the comprehensive performance score of the candidate model is higher than that of the current model and the inference time of the candidate model does not exceed 1.2 times the allowed inference time, the candidate model and the candidate detection threshold are used for the next detection cycle; otherwise, the current model and the current detection threshold are retained.

8. A data asset anomaly detection system based on deep learning, characterized in that, include: The event association module is used to acquire multi-source raw records generated during the operation of data assets and associate them as target running events according to event identification information or spatiotemporal association information. The multi-source raw records include at least two of the following: business access records, database audit records, interface call records, identity authentication records, network session records, and terminal status records. Each raw record includes at least the source system, source record identifier, and record time. The feature processing module is used to extract operational features from the target operational event, including at least two of the eight data dimensions: event level, user, data asset, device, network environment, operation, time status, and historical behavior, and to uniformly encode the operational features to obtain a fixed-length global feature vector and feature mapping index. The graph processing module is used to construct an event association graph containing target event nodes, entity association nodes, and historical event nodes, write node features according to the feature mapping index, and set a default node representation and a missing mask for missing node types. The default node representation adopts a zero vector. The fusion processing module is used to perform message transformation and node type aggregation by using graph neural networks according to relation edge type and time message weight, and to combine node type representations or default node representations and missing masks in a fixed order to obtain fixed-dimensional fusion features and fusion grouping indexes. An anomaly detection module is used to reconstruct the fixed-dimensional fusion features. When training the feature reconstruction network, the missing mask is used to mask the reconstruction error of the feature group corresponding to the missing node type. The group reconstruction residual and anomaly score are calculated based on the fusion group index. The anomaly score is compared with the detection threshold to obtain the anomaly detection result. Based on the event identifier information or spatiotemporal correlation information in the multi-source original records, the multi-source original records are correlated to obtain the target running event, including: When original records from different sources contain the same event identifier, request identifier, or link identifier, the original records containing the same identifier will be associated with the same target running event; When original records from different sources do not contain the same identifier, a baseline record is determined according to a preset data source priority. Original records with the same user identifier, data asset identifier, and operation type as the baseline record and a recording time difference not greater than the association duration threshold are identified as candidate records. From the candidate records corresponding to each data source, the candidate record with the smallest recording time difference is selected. When the recording time differences are the same, the associated records are determined in sequence according to session identifier matching, device identifier matching, and recording time order, and each original record is restricted to belong to only one target running event. The association duration threshold is pre-calibrated based on the absolute recording time difference sample of cross-source matching records carrying common identifiers and remains fixed during online detection of the same model version. Before forming the target running event, original records with the same source record identifier, or original records with the same source system and content digest and a record time difference not greater than the retransmission duration threshold, are identified as retransmission records. The retransmission records are merged, and the number of records before merging is reduced by one to obtain the retransmission count. The merged records, the retransmission count, and the source information are written into the target running event. The retransmission duration threshold is pre-calibrated based on the absolute record time difference sample of the confirmed retransmission records and remains fixed during online detection of the same model version.

9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method of any one of claims 1-7.

Images