A big data problem clue mining method based on mutual exclusivity rule

Abstract

A big data problem clue mining method based on mutual exclusivity rules. It relates to the technical field of big data supervision, and particularly relates to the technical field of a big data problem clue mining method based on mutual exclusivity rules. The present application takes the mutual exclusivity contradiction between data records as the core breakthrough point and analysis dimension, accurately captures at the data level, and extracts the key features that violate logical consistency. The method comprises the following steps: obtaining a multi-source heterogeneous data set and pre-processing to obtain an initial feature set and an entity subgraph; extracting mutual exclusion features according to the initial feature set; generating evidence features based on the mutual exclusion features; constructing a double-channel model to calculate the mutual exclusion risk: using a decision tree improvement algorithm as a rule-enhanced feature channel; constructing a graph relationship perception channel: through a dynamic weighting method, the abnormal probability of the feature space and the abnormal probability based on the graph structure are aggregated into a risk score.

Description

Technical Field

[0001] This invention relates to the field of big data supervision technology, specifically to a method for mining big data problem clues based on mutual exclusion rules. Background Technology

[0002] In the field of big data supervision, traditional supervision methods mainly rely on manual screening and simple data analysis tools. Manual screening is not only labor-intensive and inefficient, but also prone to errors and omissions. At the same time, it lacks systematicness and comprehensiveness in the analysis of massive amounts of data.

[0003] In recent years, with the application of big data technology, some big data supervision platforms have emerged. However, these platforms still have the following shortcomings:

[0004] (1) Data analysis tools have strong functional limitations: the analysis dimensions are relatively simple and mainly limited to preset simple query conditions, making it difficult to automatically discover hidden and unexpected complex patterns in the data.

[0005] It has shortcomings in deep correlation mining, making it difficult to efficiently integrate multi-source heterogeneous data for fusion analysis, and unable to automatically construct complex relationship networks between entities (such as people, units, and projects) and identify abnormal information from them.

[0006] Unable to identify deep-seated logical contradictions: Simple tools cannot systematically and automatically detect mutually exclusive constraints hidden in business processes, financial rules, policies and regulations (such as "the same person cannot sign in at two locations at the same time", "the same project budget cannot be reimbursed repeatedly", and "the operation sequence of specific permissions must conform to the process").

[0007] Poor flexibility: Rule adjustments require technical personnel to manually modify queries or scripts, making it difficult to quickly respond to changes in monitoring strategies.

[0008] (2) Existing big data monitoring platforms have shortcomings in key analytical capabilities: Insufficient depth of mutual exclusion analysis: Existing platforms have not systematically modeled and applied "mutual exclusion rules" as a core analytical dimension. They can only issue alerts based on a single rule, but they are unable to build complex, dynamic and combinatorial mutual exclusion rule models, and they also lack the ability to autonomously mine potential mutual exclusion rules from data using machine learning technology.

[0009] The level of intelligence is limited, relying mainly on pre-set hard rules and lacking necessary adaptive learning and model optimization mechanisms. Therefore, it is difficult to effectively cope with constantly changing violation methods. It is also difficult to accurately distinguish between normal business differences and abnormal mutually exclusive situations suspected of violations.

[0010] It is difficult to balance the accuracy and recall of clues: simple rule engines are prone to false alarms and cover up real clues; while being too conservative leads to frequent missed alarms and makes it difficult to achieve accurate supervision.

[0011] Lack of closed-loop optimization: The inadequate mechanism for evaluating the effectiveness of models or rules, feedback, and iterative optimization leads to stagnation in analytical capabilities. Summary of the Invention

[0012] To address the limitations of existing technologies in data analysis and their shortcomings in analyzing mutual exclusion rules, this invention proposes a big data problem clue mining method based on mutual exclusion rules. It uses "mutual exclusion contradictions between data records" as the core entry point and analytical dimension to accurately capture key features that violate logical consistency at the data level.

[0013] The method includes the following steps:

[0014] S1. Obtain multi-source heterogeneous datasets and perform preprocessing to obtain initial feature sets and entity subgraphs;

[0015] S2. Extract mutually exclusive row features based on the initial feature set;

[0016] S3. Based on mutually exclusive row features, generate evidence features: direct conflict features, context enhancement features, and pattern concealment features;

[0017] S4. Construct a dual-channel model to calculate mutual exclusion risk:

[0018] S41. Use decision tree to improve the algorithm as a rule-enhancing feature channel;

[0019] The improved decision tree algorithm specifically adds a node splitting priority mechanism when splitting the decision tree;

[0020] By inputting evidentiary features into the rule-enhanced feature channels, anomaly probabilities are obtained in the feature space. ;

[0021] S42. Constructing a graph relationship perception channel: The graph relationship perception channel passes through the input and output sequentially, including an improved random walk sampling algorithm, an improved graph attention network, and a graph anomaly detection module;

[0022] S421. The improved random walk sampling algorithm is as follows: during dynamic sampling, a new sampling type is added, and the sampling weight is dynamically adjusted.

[0023] S422, the graph attention improvement network specifically adds a rule and risk guidance mechanism when calculating the attention coefficients in the first layer of GAT convolution;

[0024] An interactive gating unit is added between the first and second GAT convolutional layers;

[0025] A residual connection is introduced between the output of the second GAT convolution and the input of the first GAT convolution.

[0026] Input the entity subgraph into the graph relationship-aware channel to obtain the anomaly probability based on the graph structure. ;

[0027] S5. By dynamically weighting the anomaly probabilities in the feature space and the graph-based anomaly probabilities, we obtain: anomaly clues and their corresponding risk scores. .

[0028] Furthermore, the preprocessing includes: data cleaning and denoising, data fusion and entity alignment, data standardization and feature engineering.

[0029] Furthermore, in step S2, extracting mutually exclusive row features includes: defining and identifying mutually exclusive rows and extracting key features;

[0030] The definition and identification of mutual exclusion lines include: based on a predefined rule base, relationship graph identification and clustering anomalies, defining and identifying time mutual exclusion, state mutual exclusion, permission mutual exclusion, numerical logic mutual exclusion, business process mutual exclusion and combined mutual exclusion;

[0031] Key feature extraction includes: basic feature extraction, contextual feature extraction, strong conflict feature extraction, and pattern feature extraction.

[0032] Furthermore, in step S3, the process of generating evidence features is as follows: digitize the rules and dynamically fuse the digitized rules with mutually exclusive row features to generate direct conflict features, context-enhanced features, and pattern concealment features.

[0033] Furthermore, hard rule filtering modules are built at the input and output ends of the dual-channel model to match the input and output data with hard rules. If the match is successful, it is marked as a confirmed violation; otherwise, it is not marked.

[0034] Hard rules are rules that have a unique and definite result;

[0035] Transform soft rules into Boolean features and input them into a dual-channel model;

[0036] The dual-channel model also includes a loss function, which adds a soft rule violation penalty term to the loss function.

[0037] Soft rules are rules that do not have a unique and definite result.

[0038] Furthermore, in step S41, the process of enhancing the feature channels according to rules is as follows:

[0039] Step S411: Extract the rule features from the input data, create rule feature labels, and calculate the confidence level of each rule feature;

[0040] Step S412: Sequentially determine whether the confidence level of each rule feature exceeds the confidence level threshold. If it does, increase the gain of the rule feature; otherwise, do not increase it.

[0041] Step S413: Dynamically generate a decision tree based on the gain adjusted in step S412;

[0042] In step S413, a dynamic weight coefficient is also set to control the degree to which the rule features guide the decision tree splitting.

[0043] Furthermore, in step S421, the new sampling type is a rule weight term;

[0044] The formula for dynamically adjusting the sampling weights is:

[0045] in, This represents the dynamic sampling probability value. This represents a traditional bias term based on the historical risk of the target entity. Indicates the rule weight item. This represents an adjustable hyperparameter used to balance the influence of traditional bias terms based on the historical risk of the target entity and rule weight terms on dynamic sampling.

[0046] Furthermore, the calculation formula for the interactive gating unit is: , This represents the weighted interaction result of each feature header. Indicates a gating signal. in, This represents the activation function. This represents the output of the first GAT convolution layer. Represents the weight matrix. This represents the bias vector.

[0047] Furthermore, in step S5, risk scoring... The formula for calculation is:

[0048] in, and Both represent mutual exclusion coefficients.

[0049] The beneficial effects of the method described in this invention are as follows:

[0050] (1) This invention extracts mutually exclusive row features, efficiently integrates multi-source heterogeneous data for analysis, and constructs entity subgraphs. In the complex relationship network of entities (such as personnel, units, and projects), it identifies abnormal information, generates evidence features, and supports the subsequent dual-channel model (rule-enhanced feature channel) to identify deep-seated logical contradictions and analyze mutual exclusion risks.

[0051] (2) This invention takes “mutual exclusion rule” as the core analysis dimension, constructs a dual-channel model, systematically and accurately identifies and extracts mutually exclusive rows and their diverse features that contain potential violation indications in the data, autonomously mines differences beyond the basic field values, and extracts deep-level contextual features (entity historical behavior, group universality), conflict intensity features (violation level, degree of difference) and pattern features (periodicity, correlation).

[0052] (3) The rule-enhanced feature channel constructed in this invention prioritizes the selection of features highly related to expert rules when splitting nodes. By amplifying the weight of expert rule features in decision tree splitting, the model is forced to learn core rules first, thus solving the problem of missed detection caused by key rules being submerged by noise features in traditional methods. For example, the missed detection rate of mutual exclusion of permissions is reduced from 75% to 7%.

[0053] (4) The graph relationship perception channel constructed in this invention first assigns a 5 times sampling weight to high-risk entities (historical violations > 3 times) in the dynamic subgraph sampling stage, focusing on the hidden association network and solving the problem that high-risk entities are diluted due to uniform sampling in traditional methods. For example, the false negative rate of benefit transfer is reduced from 92% to 18%.

[0054] Furthermore, by improving graph neural networks (GNNs), key abnormal patterns such as densely connected circles, abnormal fund flows, and identity spoofing are captured, solving the problem of difficult identification of complex "team crime" behaviors. For example, the false negative rate of "team crime" has been reduced from 97% to 22%.

[0055] (5) By setting an adaptive fusion engine, the present invention adjusts the weights of the two channels according to the mutual exclusion type, eliminates the blind spot of a single model, and reduces the false detection rate of spatiotemporal mutual exclusion and conflict of interest by more than 80% simultaneously.

[0056] (6) This invention is compatible with the strong constraints of hard rules and the adaptive learning and model optimization mechanism based on soft rules. It learns complex, fuzzy and dynamic violation patterns, handles uncertainty and effectively adapts to constantly changing violation methods. The two complement each other and are indispensable. Attached Figure Description

[0057] Figure 1 This is a flowchart of the method described in this invention;

[0058] Figure 2 This is a flowchart of the mutual exclusion behavior analysis model described in this invention;

[0059] Figure 3 This is a flowchart of the dual-channel model described in this invention;

[0060] Figure 4 This is a flowchart of the rule-enhanced feature channel described in this invention;

[0061] Figure 5 This is a flowchart of the decision tree splitting process described in this invention;

[0062] Figure 6 This is a flowchart of the graph relationship sensing channel described in this invention. Detailed Implementation

[0063] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] Example 1

[0065] This embodiment provides a method for mining big data problem clues based on mutual exclusion rules. The flowchart of the method is as follows: Figure 1 As shown.

[0066] The method described in this invention collects multi-source heterogeneous datasets from supervisory departments and internet organizations through API calls and direct database connections. It then performs data cleaning and denoising on these datasets, followed by data fusion and alignment (fuzzy matching and graph linking techniques) and data standardization and feature engineering (normalization and encoding conversion techniques) to identify records from different data sources pointing to the same real-world entity (person / enterprise / project). This process constructs relationships between entities (e.g., person-unit), resulting in an entity subgraph (a local subgraph extracted from the full knowledge graph) and an initial feature set formed by basic features such as time difference, frequency, and statistics.

[0067] Based on the initial feature set, define and identify mutually exclusive rows: using methods such as a predefined rule base, relationship graph, and clustering anomalies, define and identify contradictory relationships where two or more records describe facts that cannot be simultaneously true under specific business rules or logical constraints. Examples include time mutual exclusion, state mutual exclusion, permission mutual exclusion, numerical logic mutual exclusion, business process mutual exclusion, and combinational mutual exclusion.

[0068] Time-exclusivity phenomenon: This occurs when the same person (entity A) signs in or submits a travel expense claim at different locations (location B and location C) within completely overlapping time periods (T1 to T2).

[0069] State mutual exclusion: refers to the situation where the same asset (entity A) is simultaneously marked as "scrapped" and "in use" at the same time (T).

[0070] The issue of mutually exclusive permissions: In the same business process, if the same person (entity A) plays both the role of "approver" and "applicant", it violates the basic principle of separation of duties.

[0071] Numerical logic mutual exclusion: The total actual expenditure of the project (Record 1) exceeds its total approved budget (Record 2).

[0072] Business processes are mutually exclusive: the "Contract Signing" record (Record 1) is earlier than the "Bidding Result Announcement" record (Record 2).

[0073] Combinatorial mutual exclusion: Person A is recorded as attending a meeting in department B at time T1, and at the same time is recorded as handling personal business in location C at time T1+Tdelta (Tdelta is less than reasonable travel time).

[0074] Identification method:

[0075] Based on a predefined rule base: Domain experts (auditors) predefine common mutually exclusive rule templates based on laws and regulations, policy documents, internal control systems, business process diagrams, etc.

[0076] Based on the relationship graph: On the constructed entity relationship graph, find records where the attribute values ​​carried by the edges connecting the same entity have logical conflicts.

[0077] Based on cluster anomalies: When clustering along a specific dimension (such as "person-time-location"), records that fall near multiple cluster centers or are far from all clusters may constitute mutually exclusive candidates.

[0078] Key feature extraction includes: basic feature extraction, contextual feature extraction, strong conflict feature extraction, and pattern feature extraction.

[0079] Basic feature extraction: the original field values ​​of the mutually exclusive record pair / group itself (such as amount, time, location code); the difference in field values ​​(such as absolute time difference, amount difference, geographical distance, text similarity); the entities associated with the mutually exclusive records, including individuals, units or projects, need to be examined for their basic attributes, such as the individual's job level, department, and project type.

[0080] Contextual feature extraction: the frequency with which the entity exhibits similar mutually exclusive behavior in historical data; the prevalence of similar mutually exclusive behavior within the entity's unit / department; the timing of the mutually exclusive events (e.g., whether they occurred during holidays, before or after an audit period); and characteristics of other related entities (e.g., whether the counterparty is on a blacklist or in a high-risk supplier database).

[0081] Conflict intensity feature extraction: Importance level of violated rules (expert definition or model learning); temporal overlap / difference magnitude (relative to a reasonable threshold); monetary difference (relative to project size or historical average); reasonableness of location differences (whether accessible within a reasonable travel time).

[0082] Pattern feature extraction: Does the pattern of mutually exclusive records appearing in the data stream (such as periodicity, burstiness, specific combinations) involve multiple mutually exclusive rules being triggered simultaneously? Characteristics of masking behavior (such as whether small-amount splitting or related-party masquerading is used).

[0083] Example 2

[0084] This embodiment is a further limitation of Embodiment 1.

[0085] like Figure 2 As shown, the method described in this invention constructs a mutual exclusion behavior analysis model. First, the mutual exclusion principle in business rules is transformed into a quantifiable risk assessment indicator (rule digitization), which can classify the credibility of contradictions: distinguishing between "explainable normal contradictions" (such as system input errors) and "unexplainable abnormal contradictions" (suspected violations). Finally, the core objective of the mutual exclusion behavior analysis model is to dynamically calculate the risk score (0-1) of abnormal contradictions by constructing a dual-channel model. The score depends on the severity of the rule violation, the strength of the contradictory evidence, and the risk profile of the related entities.

[0086] Rule digitization—the process of defining computable mutual exclusion relationships is shown in Table 1:

[0087]

[0088] The digitized rules are dynamically fused with mutually exclusive row features to generate evidence features: direct conflict features, context-enhanced features, and pattern concealment features.

[0089] Direct conflict characteristics: Basic indicators such as time overlap and amount difference are directly generated using rule digitization technology;

[0090] Context-enhanced features: link entity profiles with historical behavior to determine whether contradictions are reasonable: for example: (entity risk) the number of violations recorded by the party in the past three years and the risk level assessment of the related unit; (behavioral consistency) the historical time error range of the entity (such as average check-in delay <5 minutes); and (group comparison) the occurrence rate of similar events in the same department (such as 90% of colleagues have no such conflicts).

[0091] Pattern concealment features: Identify deliberate avoidance behaviors (e.g., splitting large transactions into multiple small mutually exclusive records, and detecting high-frequency small mutually exclusive behaviors through the model) and proxy behaviors (e.g., using related person accounts to create fake mutual exclusions).

[0092] like Figure 3 As shown, a dual-channel model is constructed, which includes a rule-enhanced feature channel and a graph relationship-aware channel;

[0093] This invention uses an improved decision tree algorithm as a rule-enhanced feature channel;

[0094] like Figure 4 As shown, in the rule enhancement feature channel, rule feature labels are created based on the input features (evidence features), i.e., an expert rule feature list is created: a list named rule_features is defined, which explicitly lists specific feature names representing core mutually exclusive rules, such as "time_overlap_ratio", "role_conflict_level", and "over_budget_ratio".

[0095] The confidence level of the regular features in the list is calculated. This confidence level is used to dynamically adjust feature priorities in subsequent feature selection stages. Features in the list are evaluated based on their feature values ​​(confidence values). Feature instances that meet the high confidence criteria (e.g., time_overlap_ratio greater than 0.8) are marked with a flag (e.g., setting the high_confidence attribute to 1). This flag is used in subsequent steps to identify which regular features provide highly credible evidence of conflict on the current data sample.

[0096] The improved decision tree algorithm specifically involves adding a node splitting priority mechanism during decision tree splitting. That is, within the GBDT decision tree training framework (taking XGBoost as an example), a rule-feature priority mechanism is implemented through a custom splitting evaluation function. The specific process is as follows:

[0097] like Figure 5 As shown, when a decision tree node splits, it checks whether the candidate evidence features belong to the predefined expert rule feature list such as ["time_overlap_ratio","role_conflict_level","over_budget_ratio"].

[0098] If the confidence statistic (e.g., maximum histogram value) of this feature at the current node exceeds a preset confidence threshold (e.g., time overlap ratio > 0.8), then the original information gain is amplified: gain × (1 + ... )( (This is a configurable amplification factor).

[0099] If the rule feature is illegal or does not reach the confidence threshold, the original information gain value is retained.

[0100] The process of integrating the decision tree improvement algorithm into model training is as follows: The custom evaluation function mentioned above is injected into the training process through the XGBoost callback interface, which forces the model to prioritize the selection of expert rule features with high confidence when making split decisions, thereby enhancing its ability to learn core business rules (such as spatiotemporal mutual exclusion and permission conflicts).

[0101] The screening process described in this invention prioritizes evaluating features that meet the high confidence criteria. These high confidence features are triggered by the strength of rule features, representing their higher relevance to business logic. Furthermore, information gain (IG) can be combined to comprehensively measure the contribution of features to the target classification, as shown in the formula Gain(Fi) = IG(Fi) + λ•RuleFeature(Fi), where Fi represents the candidate evidence feature, RuleFeature(Fi) represents the rule feature confidence of the candidate evidence feature, λ represents the dynamic weight coefficient used to adjust the influence of the rule feature, IG(Fi) represents the information gain of the candidate evidence feature, and Gain(Fi) represents the rule feature extraction value of the candidate evidence feature. In this way, a close logical connection is established between rule feature extraction and screening through feature confidence and the comprehensive gain formula, making the interaction relationship of the entire invention clearer and more cohesive.

[0102] After the decision tree nodes are split, the output of the rule-enhanced feature channel is obtained: the anomaly probability in the feature space. .

[0103] The method described in this invention solves the problem of rule features being submerged due to noise interference in traditional models by dynamically amplifying the information gain of rule features, and significantly improves the detection rate of key violation clues (such as reducing the missed detection rate of mutual exclusion of permissions by more than 70%).

[0104] like Figure 6 As shown, the graph relationship-aware channel employs an improved random walk sampling algorithm based on risk bias to dynamically sample the input entity subgraph. The specific process is as follows:

[0105] Input and parameter settings:

[0106] Using conflicting entities as the sampling starting point, local subgraphs (entity subgraphs) are extracted from the full knowledge graph.

[0107] Set the walking step size (steps=3) to limit sampling to a 3-hop relationship range around the entity.

[0108] A risk preference matrix (bias=risk_bias_matrix) is introduced, and the sampling weights are dynamically adjusted based on the entity's historical risk labels (such as the number of violations) (for example, high-risk entities are given a weight of 5 times).

[0109] Dynamic sampling process:

[0110] The rule-aware, biased random walk is performed using the `rule_aware_randomwalk_sampling` function (an improved random walk sampling algorithm). The core innovation of this algorithm lies in its transition probabilities, which not only depend on the entity's historical risk labels but also deeply incorporate the association preferences defined by the currently triggered mutual exclusion rules.

[0111] Input matching rules:

[0112] The input consists of a conflicting entity and one or more mutually exclusive rule types (such as rule type: "Conflict of Interest").

[0113] Query the expert rule knowledge base to obtain the most relevant edge types and entity attributes for this type of rule. For example, for the "conflict of interest" rule, the knowledge base definition should prioritize traversing edge types such as [shareholding relationship, kinship relationship, transaction relationship, employment relationship].

[0114] Calculation of transition probability in rule-aware learning:

[0115] Traditional methods: The transition probability P(next_node|current_node) is based solely on edge weights (such as transaction amount, number of associations) or static risk scores.

[0116] The method described in this invention dynamically adjusts the transition probability formula, assigning higher weight coefficients to the relationship types recommended by the rules. Algorithm formula (example):

[0117] in, This represents the dynamic sampling probability value. This represents a traditional bias term based on the historical risk of the target entity. This represents the rule weight, which is predefined by experts or learned by the model, indicating the importance of a certain relationship edge (such as kinship) under the current rule type. This represents an adjustable hyperparameter used to balance the influence of traditional bias terms based on the historical risk of the target entity and rule weight terms on dynamic sampling. Its value range is: ,when When the value approaches 1, the sampling process will strictly follow the guidance of the current mutual exclusion rule.

[0118] The output of dynamic subgraph sampling is a local subgraph that is highly relevant to the current mutual exclusion rule. This subgraph not only includes conflicting entities and their high-risk association networks, but also focuses more precisely on specific relationship paths that are most relevant to the violation pattern (for example, for conflict of interest rules, the sampling results will focus on paths such as family shareholding and frequent fund transfers).

[0119] The method described in this invention improves upon traditional random walk sampling by replacing the general sampling module with a rule-aware dedicated sampling module, enabling deep integration between the sampling process and upstream mutual exclusion rule analysis. This solves the "bias noise" problem that may arise from solely relying on historical risk (for example, a historically high-risk entity may have diverse associations in different scenarios, but the current conflict is only related to one specific association). The sampling is now highly accurate, directly serving the mutual exclusion type currently being analyzed.

[0120] The output of the dynamic subgraph sampling is then input into the graph attention improvement network.

[0121] Graph Attention Improvement Network Layer Design:

[0122] This invention makes key improvements to the standard (Graph Attention Network) GAT architecture:

[0123] By injecting prior knowledge of expert rules and enhancing information interaction within the model, it can more accurately and efficiently capture complex graph patterns related to violation clues.

[0124] Improvement 1: Rule-guided multi-head attention mechanism (first layer GAT convolution)

[0125] The core idea is to make attention calculation not only rely on the similarity of node features, but also be guided by upstream expert rules and entity risk profiles, so that the model can prioritize the associations that are "suspicious in business logic".

[0126] Implementation mechanism: In addition to the standard node feature x and edge connection edge_index, two new inputs are added: 1. rule_embedding: the embedding vector extracted from the rule triggered by the current mutual exclusion event (such as rule type, severity level); 2. risk_embedding: the embedding vector extracted from the entity risk profile (such as historical violation count, risk label).

[0127] The following formula is used to calculate the _th ... Attention on the node Its neighboring nodes Unnormalized attention coefficient :

[0128] ,

[0129] Then, by working with the nodes All neighbors The final attention coefficients are obtained by normalizing using the softmax function. :

[0130]

[0131] in, Indicates the first In each attention head, the node For neighboring nodes The unnormalized attention coefficient. Used to measure the node's attention. and nodes The strength of the correlation between them. Indicates the first Trainable weight vectors for each attention head are used to calculate the similarity of node features. : No. The weight matrix of each attention head is used to perform a linear transformation on the feature vector of the node. Representing nodes respectively and nodes The eigenvectors are denoted by LeakyReLU, which represents the activation function used to introduce non-linearity. It allows a small subset of negative values ​​to pass through, avoiding the vanishing gradient problem. Indicates the first Another trainable weight vector for each attention head is used to combine rule and risk information. Represents a node The rule embedding vector contains semantic information related to the business rules. Represents a node The risk embedding vector contains semantic information related to risk assessment. Indicates the first In each attention head, the node For neighboring nodes The normalized attention coefficient. Used to determine the contribution of each neighbor node when aggregating neighbor information. Represents a node The set of neighboring nodes, This represents an index variable used for traversing nodes. All neighboring nodes .

[0132] Effect: This design makes the model additionally consider "nodes" when calculating the importance of neighbors. "Triggered rules" and "nodes" The degree of matching between "risk attributes". For example, for a "conflict of interest" rule, the model will instinctively assign higher attention weights to high-risk related nodes such as "relatives" and "holding companies", even if the feature similarity between them is not high.

[0133] Improvement 2: Cross-head interaction and aggregation module (before the second layer GAT convolution)

[0134] Core idea: In standard GAT, each attention head works independently, and the data is simply concatenated or averaged at the end. This invention adds a lightweight cross-head interaction module, allowing different patterns captured by different heads (such as "fund flow", "personnel relationships", "spatiotemporal anomalies") to exchange information and collaborate before final aggregation.

[0135] Implementation mechanism: Let the output of the first-layer GAT be H=[head_1,head_2,head_3,head_4] (size is...) Instead of directly inputting it into the second-level GAT, it first passes through a shared interactive gating unit: in the interactive gating unit, it is processed by formula... Calculate the gating signal. Indicates a gating signal. This represents the multi-head output of the first-layer GAT, indicating the node features from multiple perspectives; This represents the weight matrix, used to linearly map features and extract signals suitable for interaction. This represents the bias vector, used to adjust the baseline of the gating signal. This represents the activation function, which transforms the weighted signal into gated weights (ranging from [0,1]). The formula is used to... The gated signal and the multi-head output of the first-layer GAT are weighted and fused for further processing in subsequent layers. This represents the weighted interaction result of each feature header.

[0136] Effects: This module allows the model to learn the dependencies between different attention patterns. For example, the model can learn that "when the 'funds flow' head (head1) and the 'personal relationships' head (head2) are activated simultaneously, the current node is extremely risky," thereby achieving the fusion and cross-validation of different rule clues and significantly improving the ability to identify complex disguised gangs.

[0137] Improvement 3: Rule-enhanced output layer (after final representation)

[0138] The core idea is to integrate the learned graph representation with the original rule features again to ensure that the model's final decision is strongly correlated with the core business rules.

[0139] Implementation mechanism: After the final node representation h_final is output from the second-layer GAT convolution, it is not directly used to calculate the loss or make predictions. Instead, a residual connection is introduced to add the initially extracted rule features (input from the first-layer GAT convolution) rule_features (after dimensionality adjustment by a projection network Φ) to h_final: h_output = h_final + Φ(rule_features), where h_output represents the output of the graph attention enhancement network.

[0140] Effect: This is equivalent to providing a "rule anchor" for the model, preventing it from becoming overly reliant on data-driven approaches and deviating from the most fundamental business logic, thus enhancing the model's interpretability and robustness.

[0141] Graph attention improves the forward propagation process of networks:

[0142] Input node features (x) and edge relationships (edge_index).

[0143] After the first convolutional layer, nonlinear features are extracted using the ReLU activation function.

[0144] The second convolutional layer generates node-level outputs, representing entity risks (such as the strength of abnormal associations between people / projects).

[0145] The output of the graph attention enhancement network is then input into the graph anomaly detection module to calculate the anomaly probability based on the graph structure. ; The calculation formula is: in, This represents the trainable weight matrix, the weight parameters of a single-layer neural network, whose function is to map a high-dimensional concatenated vector to a scalar value. These represent the final vector representations of the two mutually exclusive entities (nodes). The feature vector representing the edge (relationship) corresponding to the current mutually exclusive event. This represents the trainable weight matrix and bias terms, which are parameters of a single-layer neural network. Their function is to map the concatenated high-dimensional feature vector to a scalar value. This represents the vector concatenation operation, that is, joining two vectors together to form a new vector.

[0146] The technical advantages of the graph relationship-aware channel constructed by the method described in this invention are as follows:

[0147] The improved multi-head attention mechanism effectively identifies complex mutually exclusive patterns (such as identity spoofing and dense association circles), reducing the false negative rate by more than 85%.

[0148] We improve upon traditional random walk sampling by employing hierarchical feature aggregation to enhance the model's ability to perceive conflict contexts (such as interest transfer chains across 3-hop relationships and assigning 5 times the weight to high-risk entities).

[0149] The graph relationship perception channel enables end-to-end learning to automatically construct "entity-relationship" risk profiles, supporting the subsequent calculation of dynamic risk scores.

[0150] The method described in this invention aggregates the anomaly probabilities in the feature space and the anomaly probabilities based on the graph structure through dynamic weighting to obtain: anomaly clues and the risk scores corresponding to the anomaly clues. (Risk score is the core quantitative indicator output by the dual-channel model in this method.) The calculation formula is: in, and Both represent mutual exclusion coefficients. and The value is dynamically determined by the mutex type;

[0151] Spacetime / numerical mutual exclusion → increase (Depends on precise features); Permissions / conflicts of interest → Increase (Dependency mining). The range of values ​​is The higher the score, the more significant the risk.

[0152] Anomaly clues are extracted from multi-source heterogeneous datasets using the method described herein, combined with (expert) mutual exclusion rules. and Corresponding feature composition;

[0153] For example, in a scenario where auditors need to screen civil servants for irregularities in their travel expense reimbursements, the multi-source heterogeneous dataset includes: Record A: Zhang attended a "policy training meeting" in Beijing from 14:00 to 18:00 on June 1 (meeting system check-in record); Record B: Zhang reimbursed a catering expense at a restaurant in Shanghai at 15:30 on June 1 (reimbursement system entry record).

[0154] By processing the above multi-source heterogeneous datasets (record A and record B), evidence features (e.g., the system automatically identifies that these two records have a mutually exclusive contradiction in "time" and "location" (the same person cannot be in two places at the same time), and extracts it as a mutually exclusive row feature to be investigated, and generates evidence features based on the mutually exclusive row feature: direct conflict feature) and entity subgraphs (e.g., Zhang's work affiliation and Zhang's family social relationships).

[0155] The aforementioned evidence features and entity subgraphs are input into the dual-channel model. The rule-enhanced feature channel outputs an anomaly probability based on the evidence features: time overlap ratio (overlapping for 150 minutes), geographical distance between the two locations (over 1000 kilometers), and Zhang's historical behavior (no cross-city business trip approval record). The anomaly probability is calculated based on the precise features (Zhang has no cross-city business trip approval record, and within 150 minutes, simultaneously held a meeting and a meal across 1000 kilometers). .

[0156] The graph relationship perception channel analyzes Zhang's relationship network (relatives, colleagues, business partners, etc.) based on the entity subgraph, and examines whether his associates are engaging in fraudulent expense reimbursement or embezzlement. It then outputs an anomaly probability based on the relationship feature (Zhang's wife's accommodation record in Shanghai on June 1st). .

[0157] Since the above case is "spatiotemporally exclusive," the model dynamically adjusts the weights (relying more on the analysis results of the feature channels, i.e., increasing...). , reduce ).

[0158] By using dynamic weighting, abnormal clues were extracted (Zhang had no record of cross-city business trip approval, and within 150 minutes, he simultaneously held a meeting and had a meal across 1,000 kilometers; on the same day, Zhang's wife stayed in Shanghai).

[0159] Risk score corresponding to this abnormal clue

[0160] In this embodiment, , .

[0161] The dual-channel model used in this invention has a complementary effect: it captures both direct evidence and uncovers hidden connections. Practical verification has shown that the overall false negative rate of clues is reduced by more than 80% compared to any single-channel model.

[0162] Example 3

[0163] This embodiment is a further limitation of embodiment 2.

[0164] Hard rule filtering: (Rules with unique and deterministic results) Logical constraints defined by domain experts, possessing 100% determinism. Once triggered, they are immediately deemed violations, eliminating the need for model reasoning. Its integration is reflected in two stages:

[0165] Pre-prediction filtering

[0166] Implementation: After data preprocessing and feature extraction are completed, but before being input into the model, a hard-rule fast matcher is set up. This matcher consists of an efficient conditional rule engine (such as Drools, Aviator) or a compiled and optimized SQL query.

[0167] Technical approach: Pre-compile expert-defined hard rules (e.g., the same person ID appearing at two check-in locations more than 1,000 kilometers apart on the same day) into executable code.

[0168] Workflow: Incoming mutually exclusive record pairs are matched against all hard rules. Once any hard rule is triggered, the record pair is immediately marked as "definitely violated" and is no longer fed into the subsequent machine learning model for prediction; instead, it is directly sent to the high-scoring clue library.

[0169] Functions and advantages: It intercepts the most obvious and certain violations, greatly reducing the amount of data the model needs to process and lowering overall system overhead. It ensures that these serious violations that are unlikely to be misjudged will not be misclassified as normal or low-risk by the model (due to potential probability errors), fundamentally eliminating the underreporting of critical clues.

[0170] Post-PredictionCorrection

[0171] Implementation method: After the model outputs the predicted risk score and label, the hard rule engine is used to review all model results again.

[0172] Workflow: For samples that the model predicts to be low-risk or normal, if they trigger a hard rule, the system will forcibly override the model's prediction, set their risk score to 1.0 (highest risk), and mark them as "confirmed violation".

[0173] Function and Advantages: As a safety barrier, it compensates for potential "fatal blind spots" in the model or serious misjudgments caused by data quality issues, providing the ultimate guarantee for the system's high reliability. When a new violation pattern appears that has never appeared in the training data, the model may fail to recognize it, but experts can quickly define a new hard rule to immediately capture it and forcefully correct the model's error through this mechanism.

[0174] Soft rule constraints: (rules without a unique and definitive outcome) Expert experience, business logic, and common patterns that are not 100% certain but are highly instructive. They do not directly provide judgments but rather "educate" and "guide" the model in two ways:

[0175] Transformed into feature input (Feature Engineering)

[0176] Implementation method: This is the most direct and common approach. Expert rules are quantified into features that the model can understand.

[0177] Technical Approach: Direct Features: For example, transforming the rule "The approver and the applicant are the same person" into a Boolean feature `is_self_approval`. Distance / Similarity Features: For example, transforming the rule "Transaction prices should not deviate significantly from the market average price" into the feature `price_deviation_ratio`. Frequency Features: For example, transforming the rule "Frequent transactions within a short period are suspicious" into the feature `transaction_count_last_7_days`.

[0178] Function: It transforms human prior knowledge into the language of the model, greatly reducing the difficulty for the model to learn patterns and helping the model quickly focus on key signals.

[0179] The dual-channel model also includes a loss function, which adds a soft rule violation penalty term: transforming the soft rule into a regularization term in the model's loss function.

[0180] Implementation method: This is a deeper level of integration. By modifying the model's optimization objective, the model's prediction results are forced to conform to the trend defined by the soft rules.

[0181] Technical approach: During model training, add a rule violation penalty term after the standard loss function (such as cross-entropy loss) for (dual-channel models).

[0182] Here, `Rule_Violation_Loss` represents the rule violation loss, a custom penalty term that quantifies the extent to which the model's predictive behavior violates expert-defined soft rules. It measures the difference between the model's prediction and the expected business logic. Even if the model correctly predicts the label, it will still be penalized if its prediction method does not align with business intuition. This represents a hyperparameter used to balance the weights of standard loss and rule violation loss, helping the model to consider both data fit and adherence to business rules during optimization. Standard_Loss represents the standard loss, which is the standard objective function that the model would normally minimize without considering any business rules. It measures the difference between the model's prediction and the true label. Its sole objective is to make the model's prediction (e.g., risk score) as close as possible to the manually labeled reality. Total_Loss represents the total loss, which combines standard loss and rule violation loss, guiding the model to simultaneously optimize its ability to fit the data and adhere to business rules. For example, a soft rule might be "the larger the amount of a mutually exclusive event, the higher the risk." A penalty term could be designed so that if the model predicts a low risk score for a mutually exclusive event with a large amount, it will incur a large loss.

[0183] Function: By injecting expert knowledge into the model at the optimization process level, the model parameters are guided to be updated in a direction that conforms to business logic, so that the final model not only fits the data, but also matches the expert's intuition, and generates more business-reasonable prediction results.

[0184] The combination of hard and soft rules: Expert knowledge (hard rules) is used to capture highly deterministic violation patterns and provide strong constraints; machine learning models (soft rules) are used to learn complex, fuzzy, and dynamic violation patterns and handle uncertainty. The two are complementary and indispensable.

[0185] Setting risk thresholds is not a one-time solution; it needs to be dynamically adjusted based on business objectives (whether to pursue broad coverage or precise targeting), actual lead processing capabilities, and model evaluation results to ensure the usability and effectiveness of the output leads.

[0186] Based on business needs and combined with the analysis of PR curves and ROC curves on the validation set, the method described in this invention determines the optimal threshold for the model's predicted risk score (for example, when the risk score exceeds 0.85, the abnormal clue is determined to be a high-risk abnormal clue).

[0187] The method described in this invention sets up a dynamic threshold adjustment mechanism, which allows for flexible adjustment of risk thresholds based on lead backlog, business priorities, and seasonal factors in actual applications.

[0188] Rule optimization: Based on the analysis results of the dual-channel model and actual survey feedback, the method described in this invention continuously optimizes and expands the content and priority settings of the predefined rule base.

[0189] Example 4

[0190] This embodiment further defines Embodiment 2. This embodiment demonstrates how to train a dual-channel model and perform closed-loop optimization.

[0191] First, set up training, validation, and test sets for training the dual-channel model. The training set is used to train the model parameters. The validation set is used to adjust hyperparameters (such as tree depth, learning rate, and regularization coefficient), select the model, and implement early stopping to prevent overfitting. The test set is used to finally evaluate the model's generalization ability on unknown data. All sets are strictly isolated and used only once.

[0192] Closed-loop adaptive training framework:

[0193] The training process of this invention is not a simple "data input - model output," but rather constructs a closed-loop adaptive training framework that deeply integrates business feedback and data characteristics. Its core innovation lies in driving training optimization through dynamic feedback, enabling the model to continuously evolve in complex scenarios. The key process is as follows:

[0194] 1) Feedback-driven hyperparameter space construction (alternative grid / random search)

[0195] Problem Insight: Traditional parameter tuning relies on a pre-defined fixed hyperparameter space, which is difficult to adapt to the dynamic nature of the distribution of mutually exclusive row features as business rules and violation methods evolve.

[0196] The implementation process of the method described in this invention is as follows:

[0197] Initial spatial anchoring: Based on historical validation set performance and expert experience, only the initial range of key hyperparameters (such as the maximum depth of the tree and the learning rate) is set.

[0198] Feedback Dynamic Expansion: Utilizing the real clue labels from manual surveys and the underreported samples predicted by the online model (mutually exclusive behaviors predicted as "normal" with high confidence but ultimately discovered as violations by humans), the reasons for misjudgments are analyzed:

[0199] If feature interactions are insufficient (e.g., cross-source association information is missing), adjust and expand the complexity parameters such as the depth of the decision tree and the number of leaf nodes as appropriate.

[0200] If the model is insufficient in recognizing novel avoidance patterns (such as small-amount, high-frequency mutual exclusion behavior), the regularization strength and sampling strategy need to be adjusted to enhance the model's ability to detect sparse anomaly patterns.

[0201] If the entity context modeling is weak (e.g., neglecting group associations), then optimize the hyperparameter weights of the graph channels.

[0202] Result: The hyperparameter space becomes a dynamically growing living entity, always focusing on the most pressing business pain points that need to be addressed.

[0203] 2) Hierarchical sampling strategy for mutual exclusion characteristics

[0204] Problem Insight: The mutual exclusion row data is highly imbalanced (with very few outliers), and different types of mutual exclusion rows (time-space, permission-based, combined, etc.) have different importance. Uniform sampling is prone to missing key samples.

[0205] The implementation process of the method described in this invention is as follows:

[0206] Rule-triggered stratified sampling: When constructing the training set, the samples are divided into different levels according to the predefined rule types triggered by mutually exclusive rows (such as "time mutual exclusion" and "permission mutual exclusion") and the rule risk levels labeled by experts.

[0207] Dynamic weight allocation: during training (e.g., gradient calculation in GBDT or loss function of a neural network):

[0208] For mutually exclusive samples triggered by high-level rules, assign them higher weights and processing priority;

[0209] For mutually exclusive type samples that are frequently missed in recent manual feedback, their sampling probability will be dynamically increased.

[0210] Samples that the model predicted with high confidence but were manually verified as false alarms were included in the negative sample library to enhance the model's self-correction ability.

[0211] Results: Model training resources were accurately deployed to high-risk and error-prone areas, significantly improving the ability to identify key violation patterns.

[0212] Model evaluation metrics:

[0213] Core classification metrics: Precision, Recall, F1 score (harmonic average of Precision and Recall), AUC-ROC (measures ranking ability), and confusion matrix.

[0214] Business metrics: Lead conversion rate (the percentage of sales leads that are converted into actual purchases), average lead value (weighted by the amount of money involved in the lead, its level, etc.), and lead discovery timeliness.

[0215] Model optimization and iteration:

[0216] Incremental learning mechanism: As new data continues to flow in, this mechanism allows the model to be updated incrementally within the existing knowledge framework, effectively avoiding the high cost of retraining on full data.

[0217] Online learning: For scenarios with extremely high real-time requirements, the model can support streaming data input and real-time parameter updates.

[0218] Active learning: For samples with high prediction uncertainty, the model prompts for manual annotation, improving model performance with minimal annotation cost.

[0219] Human feedback: After auditors investigate and verify the clues output by the model, they feed the results (true clues / false clues) back to the system. The system uses this newly generated labeled data to retrain the model periodically (e.g., weekly / monthly) or triggered by events.

[0220] Feature engineering iteration: Based on model performance analysis, new business rules, and human feedback, continuously design, experiment with, and integrate new features, including new mutually exclusive rule features and context features.

[0221] Problem clue discovery and display:

[0222] The discovery and display of clues are accomplished using existing technologies such as online mining processes, clue information integration and correlation, intelligent interactive visualization, and collaboration and analysis tools.

[0223] The feedback results from the problem clue mining can also be used for mutual exclusion row feature extraction, mutual exclusion row analysis models, and closed-loop training of dual-channel models.

Claims

1. A method for clue mining of big data problems based on mutual exclusivity rules, the method comprising: receiving a set of data records; receiving a set of rules; and generating a set of clues based on the set of rules and the set of data records. The method comprises the following steps: S1, obtaining a plurality of source heterogeneous data sets of real entities from the Internet and supervision departments through API calling and database direct connection, and performing preprocessing; Through the preprocessing, records of different data sources in the plurality of source heterogeneous data sets pointing to the same real entity and time difference and frequency are identified, and an entity subgraph and an initial feature set are constructed; The real entity comprises personnel, enterprises and projects; S2, extracting mutually exclusive line features according to the initial feature set; S3, generating evidence features based on the mutually exclusive line features: direct conflict features, context enhancement features and pattern concealment features; S4, constructing a double-channel model to calculate the mutual exclusion risk: S41, using a decision tree improvement algorithm as a rule enhancement feature channel; The decision tree improvement algorithm specifically comprises: adding a node splitting priority mechanism when splitting the decision tree; The evidence features are input into the rule enhanced feature channel to obtain an anomaly probability of the feature space ; S42, constructing a graph relationship perception channel: the graph relationship perception channel sequentially passes through a random walk sampling improvement algorithm, a graph attention improvement network and a graph anomaly detection module from input to output; S421, the random walk sampling improvement algorithm specifically comprises: adding a sampling type and dynamically adjusting the sampling weight when dynamic sampling; S422, the graph attention improvement network specifically comprises: adding a rule and risk guide mechanism when calculating the attention coefficient in the first layer GAT convolution; An interaction gate unit is added between the first layer GAT convolution and the second layer GAT convolution; A residual connection is introduced between the output end of the second layer GAT convolution and the input end of the first layer GAT convolution. inputting the entity sub-graph into a graph relation perception channel to obtain a graph structure-based anomaly probability ; S5, aggregate the feature space anomaly probability and the graph structure based anomaly probability in a dynamic weighting manner to obtain an anomaly clue and a risk score corresponding to the anomaly clue . 2.The big data clue mining method based on mutual exclusivity rule of claim 1, wherein, The preprocessing comprises data cleaning and denoising, data fusion and entity alignment, data standardization and feature engineering. 3.The big data clue mining method based on mutual exclusivity rule of claim 1, wherein, In step S2, the mutually exclusive line features comprise mutually exclusive line definition and identification and key feature extraction; The mutually exclusive line definition and identification comprise defining and identifying time mutual exclusion, state mutual exclusion, permission mutual exclusion, numerical logic mutual exclusion, business process mutual exclusion and combined mutual exclusion based on a pre-defined rule library, an association relationship graph and clustering anomaly; The key feature extraction comprises basic feature extraction, context feature extraction, strong conflict feature extraction and pattern feature extraction.

4. The method of claim 3, wherein, In step S3, the process of generating evidence features is: digitizing rules, dynamically fusing the digitized rules with mutually exclusive line features, and generating direct conflict features, context enhancement features and pattern concealment features. 5.The big data clue mining method based on mutual exclusivity rule of claim 1, wherein, A hard rule filtering module is constructed at the input end and the output end of the double-channel model, which is used to match the input data and the output data with the hard rules, if the matching is successful, it is marked as a confirmed violation, otherwise it is not marked; The hard rule represents a rule with a unique and determined result; The soft rule is converted into a Boolean type feature and input into the double-channel model; The double-channel model also comprises a loss function, and a soft rule violation penalty term is added in the double-channel model loss function; The soft rule represents a rule without a unique and determined result.

6. The method of claim 1, wherein, In step S41, the working process of the rule enhancement feature channel is as follows: Step S411, extracting rule features of the input data, creating rule feature labels, and calculating the confidence of each rule feature; Step S412, judging whether the confidence of each rule feature exceeds the confidence threshold value, if yes, increasing the gain of the rule feature, otherwise, not increasing; Step S413, dynamically generating the decision tree according to the gain adjusted in step S412; In step S413, a dynamic weight coefficient is also set to control the guiding degree of the rule feature to the decision tree column.

7. The method of claim 1, wherein, In step S421, a new sampling type is added as a rule weight item; The relationship formula for dynamically adjusting the sampling weight is: wherein, denotes a dynamic sampling probability value, denotes a traditional bias term based on the target entity historical risk, denotes a rule weight term, denotes an adjustable hyper-parameter for balancing the degree of guidance of the traditional bias term based on the target entity historical risk and the rule weight term on the dynamic sampling. 8.The method of claim 1, wherein, The calculation formula of the interaction gating unit is: , denotes the interaction result after weighting of each feature head, denotes a gating signal, wherein, denotes an activation function, denotes the output of the first layer GAT convolution, denotes a weight matrix, denotes a bias vector. 9.The big data clue mining method based on mutual exclusivity rule of claim 1, wherein, In step S5, the risk score is calculated as follows: wherein and both represent the mutual exclusion coefficient.

Images