Big data compliance risk control method and system

Abstract

The invention provides a big data compliance risk control method and system, and relates to the technical field of big data processing and risk control. By obtaining the original business data and adding the metadata information for packaging, standardization and traceability of the data source are realized. Then, the metadata information is utilized to carry out preliminary matching in the data mode dictionary, and the potential mode of the data can be quickly identified. The data structure and the field type are inferred through the probability model, and the inference confidence is output, so that the problems of analysis errors and label noise caused by nonstandard data format and update lag of the analyzer in the prior art are effectively solved.

Description

Technical Field

[0001] This application relates to the field of big data processing and risk control technology, and more specifically, to a big data compliance risk control method and system. Background Technology

[0002] In the complex data processing environments of large enterprises, business departments often fail to strictly adhere to data specifications in pursuit of efficiency, resulting in diverse and inaccurate data source formats. This necessitates frequent adjustments to the data parser. During updates or when encountering unforeseen format variations, parsing delays or errors are easily triggered.

[0003] To accommodate non-standard data, engineers wrote temporary adaptation scripts. While these scripts worked effectively in standard scenarios, they contained hidden flaws in edge cases, potentially mislabeling anomalous data as normal. This erroneous data could pass quality checks and enter the model training library, creating persistent label noise. This noise caused parameter drift in the risk classification model during retraining, reducing its ability to identify novel and hidden risks. Unidentified risk data was further mislabeled as "normal," reinforcing model misjudgments and creating a negative feedback loop that continuously reduced identification capability.

[0004] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0005] This application discloses a big data compliance risk control method and system, which aims to solve the technical problem that traditional risk identification methods are unable to effectively cope with massive, diverse and inconsistent data streams in complex and ever-changing enterprise data environments, as well as the problem that business departments may not strictly adhere to regulations when pursuing efficiency. This leads to data parsing errors, defects in adaptation script logic, and accumulation of label noise, which in turn reduces the sensitivity of machine learning models to the identification of emerging compliance risk patterns, forming a negative feedback loop.

[0006] The technical solution of this application is as follows: Firstly, this application discloses a big data compliance risk control method, which includes: Obtain raw business data, add metadata information to the raw business data, encapsulate it, and send it; the metadata information includes data source identifier, data type, and data version number; Based on the received metadata information, a preliminary match is performed in the data pattern dictionary to determine the data pattern that matches the metadata information; Based on the probability model, infer the data structure and field types within the metadata information under this data pattern, and output the inference confidence level; If the confidence level of the inference is lower than a preset threshold, it is determined that the original business data is at risk and diverted to the isolation zone. The original business data in the isolation zone is then reviewed, and the data pattern dictionary is updated based on the review results.

[0007] This technical solution enables early identification and dynamic control of big data compliance risks. Through metadata encapsulation, data pattern matching, and probability inference, it effectively identifies potentially risky data and continuously optimizes the data pattern dictionary using an isolation review mechanism, thereby overcoming the shortcomings of traditional methods in handling non-standard data formats and identifying hidden risks.

[0008] Further, based on the received metadata information, a preliminary match is performed in the data pattern dictionary to determine the data pattern that matches the metadata information, including: Using the data source identifier, data type, and data version number from the metadata information as keys, the data pattern dictionary is matched to extract the historical field list, prior types, and nesting relationships that match the metadata information, thus forming the data pattern of the current data.

[0009] Based on the above, the probability model includes a Bayesian inference model or a conditional random field model; Based on the probabilistic model, the data structure and field types within the metadata information under the data pattern are inferred, and the inference confidence is output. This includes: analyzing and obtaining the structural features of the original business data based on the fields and corresponding prior types under the data pattern; the structural features include delimiters, field order, value type distribution, and nesting level; performing probabilistic analysis based on the structural features, calculating the posterior confidence of each field type and semantics, and generating a structural inference result with the posterior confidence.

[0010] In some preferred embodiments, the original business data of the isolated area is reviewed, and the data pattern dictionary is updated based on the review results. This includes: adjusting the allocation priority of review resources according to the frequency and complexity of pattern evolution corresponding to the original business data of the isolated area; performing structured processing on the review results based on manual review of the original business data, and extracting key features and semantic correction rules of the pattern evolution; comparing the structured review results with the data pattern dictionary to determine the lagging parts in the data pattern dictionary; generating update information for the data pattern dictionary based on the lagging parts, and performing data pattern dictionary update.

[0011] Furthermore, the review results are structured to extract key features and semantic correction rules for pattern evolution. This includes: pre-analyzing the new pattern features and semantic correction rules identified in the review results to determine if they have semantic conflicts with the data pattern dictionary; resolving the identified semantic conflicts according to preset priority rules to obtain the resolved new pattern features and semantic correction rules; marking the key features or semantic rules identified in the review results that are missing in the data pattern dictionary, and prioritizing them according to business importance, data frequency, and potential impact on the pattern inference engine to obtain the ranked new knowledge to be integrated; and merging the resolved new pattern features and semantic correction rules and the ranked new knowledge to be integrated with the data pattern dictionary according to the priority, and performing a semantic compatibility check to obtain an updated data pattern dictionary.

[0012] As a technical improvement, the new pattern features and semantic correction rules identified in the review results are pre-analyzed to determine whether they have semantic conflicts with the data pattern dictionary. This includes: identifying the business domain identifier associated with the new pattern features and semantic correction rules; extracting relevant pattern knowledge related to the business domain identifier and cross-domain pattern knowledge associated with the pattern knowledge from the data pattern dictionary based on the business domain identifier; and comparing the semantic representation of the new pattern features and semantic correction rules with the semantic representation of the relevant pattern knowledge and the cross-domain pattern knowledge to determine whether a semantic conflict occurs.

[0013] In one implementation, the semantic representation of the new pattern feature and the semantic correction rule is compared with the semantic representation of the related pattern knowledge and the cross-domain pattern knowledge to determine whether a semantic conflict occurs. This includes determining whether the new pattern feature or the semantic correction rule has any of the following characteristics with the related pattern knowledge or the cross-domain pattern knowledge: data type mismatch, enumeration value range conflict, or semantic ambiguity of key fields. If any of these exist, it is determined that a semantic conflict exists.

[0014] This technical solution enables a clear determination of whether semantic conflicts exist through precise comparison of specific data types, enumeration value ranges, and key field semantics. This provides a clear basis for subsequent conflict resolution and improves the accuracy and operability of conflict identification.

[0015] To improve the solution, the identified semantic conflicts are resolved according to preset priority rules to obtain new pattern features and semantic correction rules after resolution. This includes: determining the type of semantic conflict; calculating the resolution priority of the semantic conflict based on the type of semantic conflict, the business domain associated with the semantic conflict, the amount of data involved in the semantic conflict, and the expected compliance risk impact of the semantic conflict; selecting a resolution strategy based on the resolution priority and resolving the semantic conflict to obtain new pattern features and semantic correction rules after resolution; monitoring the processing effect feedback of the resolution strategy on the business data flow, and optimizing the calculation rules of the resolution priority based on the processing effect feedback.

[0016] As a further improvement, the monitoring and mitigation strategy provides feedback on the processing effect of the business data flow, including: deploying probes at key nodes of the business data flow; the probes collecting monitoring data in a non-blocking manner; the monitoring data including data parsing success rate, data conversion consistency, field value distribution deviation, and abnormal data ratio; preprocessing and aggregating the monitoring data locally to obtain aggregated indicator data, and sending the aggregated indicator data to the monitoring center; the monitoring center performing real-time streaming analysis on the aggregated indicator data to obtain analysis results; comparing the analysis results with a preset baseline, triggering an alarm if the result exceeds the preset baseline, and recording a snapshot of the abnormal data flow to evaluate the actual processing effect of the mitigation strategy.

[0017] Secondly, this application also discloses a big data compliance risk control system, which includes: an acquisition and encapsulation module for acquiring raw business data, adding metadata information to the raw business data, encapsulating and sending it; the metadata information includes a data source identifier, data type, and data version number; a matching module for performing preliminary matching in a data pattern dictionary based on the received metadata information to determine the data pattern that matches the metadata information; a probability inference module for inferring the data structure and field type within the metadata information under the data pattern based on a probability model, and outputting the inference confidence level; and a risk diversion module for determining that the raw business data is at risk and diverting it to an isolation zone in response to the inference confidence level being lower than a preset threshold, and reviewing the raw business data in the isolation zone and updating the data pattern dictionary based on the review results. Beneficial effects

[0018] This application provides a big data compliance risk control method that achieves standardization and traceability of data sources by acquiring raw business data and encapsulating it with added metadata. Subsequently, preliminary matching is performed in a data pattern dictionary using the metadata information to quickly identify potential data patterns. Based on this, a probabilistic model is used to infer the data structure and field types, and the inference confidence score is output, effectively solving parsing errors and label noise problems caused by non-standard data formats and lagging parser updates in existing technologies. When the inference confidence score falls below a preset threshold, the system can promptly divert the risky raw business data to an isolated area for review and dynamically update the data pattern dictionary based on the review results, thus forming a closed-loop risk identification, review, and optimization mechanism. This method overcomes the limitations of traditional risk identification methods when handling massive and diverse data streams, effectively avoids incorrectly labeled data caused by defects in the adaptation script logic, prevents the negative impact of label noise on machine learning model training, significantly improves the model's sensitivity to emerging and hidden compliance risk patterns, breaks the negative feedback loop, and ensures the effectiveness of enterprise data compliance and risk control. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the big data compliance risk control method disclosed in an embodiment of the present invention; Figure 2 This is a flowchart of the steps for updating the data pattern dictionary based on the review results disclosed in an embodiment of the present invention; Figure 3 This is a schematic diagram of the big data compliance risk control system structure disclosed in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0022] Traditional enterprise big data compliance risk control methods suffer from several drawbacks when facing complex and ever-changing data environments. These include the need for frequent customization and adjustments to data format parsers, logical flaws in temporary adaptation scripts leading to incorrect data labeling, and decreased sensitivity of machine learning models due to label noise. These issues collectively present enterprises with significant challenges in compliance risk control, necessitating a more efficient and intelligent solution.

[0023] In response, this application proposes a big data compliance risk control method, such as... Figure 1 As shown, the method includes: S101, Obtain raw business data, add metadata information to the raw business data, encapsulate and send it; the metadata information includes data source identifier, data type, and data version number; S102, perform preliminary matching in the data pattern dictionary based on the received metadata information to determine the data pattern that matches the metadata information; S103, infer the data structure and field types within the metadata information under the data pattern based on the probability model, and output the inference confidence level; S104, in response to the inference confidence level being lower than a preset threshold, it is determined that the original business data is at risk and diverted to the isolation zone, and the original business data in the isolation zone is reviewed, and the data pattern dictionary is updated according to the review results.

[0024] "Raw business data" refers to various types of data generated by an enterprise during its daily operations, which are either unprocessed or only preliminarily collected, such as transaction records, user behavior logs, and sensor data. This data typically exists in multiple formats and may contain sensitive information or compliance risks. "Metadata information" refers to data describing the raw business data. It does not contain the business data itself but provides information about its structure, source, type, and version. In this application, metadata information includes data source identifiers, data types, and data version numbers, which are the foundation for data pattern matching and risk inference. "Data pattern dictionary" is a knowledge base storing known data patterns, containing the structural definitions, field types, nesting relationships, and historical evolution information of various business data. This dictionary is the core basis for data pattern matching, and its accuracy and completeness directly affect the efficiency and accuracy of risk identification. "Probabilistic model" refers to a statistical model used to infer data structure and field types, such as a Bayesian inference model or a conditional random field model. These models can make probabilistic predictions about unknown or uncertain data structures based on existing data patterns and metadata information, and output inference confidence levels. "Inference confidence level" is a measure of the reliability of the inference results from a probabilistic model regarding data structure and field types. A higher confidence level indicates a more reliable inference result; a lower confidence level indicates greater uncertainty in the inference result, potentially posing compliance risks. The "isolation zone" is a dedicated area for storing potentially risky raw business data. Data in this zone is isolated from normal business data flow to prevent the spread of potential risks and to provide a secure environment for subsequent manual review and schema updates.

[0025] The method provided in this application can be implemented in a distributed big data processing platform, such as a cluster environment based on technologies like Hadoop and Spark. Each stage, including data acquisition, processing, storage, and analysis, can be deployed on corresponding computing nodes, and data interaction and function calls can be performed through message queues, API interfaces, and other methods. This application provides a big data compliance risk control method, the core of which lies in achieving risk identification, diversion, and adaptive pattern updates of raw business data through a series of steps.

[0026] First, raw business data needs to be acquired, and metadata information needs to be added to this data before it is encapsulated and sent. Metadata information describes the data and includes the data source identifier, data type, and data version number. For example, raw business data can be collected in real-time or in batches from various internal business systems (such as CRM systems, ERP systems, log systems, etc.). During the data acquisition process, metadata information can be automatically generated or extracted for each data item or each batch of data. The data source identifier can be a unique string indicating the specific source system or module of the data; the data type can be a predefined enumeration value, such as "transaction data," "user behavior data," "system log," etc.; the data version number is used to track the evolution of the data pattern or format. This metadata information is then encapsulated together with the raw business data into a data packet and sent to subsequent processing modules via a message queue or API interface.

[0027] Secondly, based on the received metadata information, a preliminary match is performed in the data pattern dictionary to determine the data pattern that matches the metadata information. The data pattern dictionary is a pre-built knowledge base that stores information such as the structure definitions, field types, and nesting relationships of various known business data. When a data packet with metadata information is received, the system uses the data source identifier, data type, and data version number in the metadata information as query keys to search in the data pattern dictionary. For example, if the metadata information indicates that the data comes from the "order system," the type is "transaction data," and the version number is "1.0," the system will attempt to find a pattern definition in the dictionary that matches these characteristics. After a successful match, the corresponding data pattern is extracted, which describes the expected data structure and field specifications.

[0028] Next, the system infers the data structure and field types within the metadata information under the data pattern based on a probabilistic model, and outputs the inference confidence score. After initially matching and obtaining the data pattern, the system uses a probabilistic model to perform deeper structure and type inference on the original business data. For example, a Bayesian inference model or a conditional random field model can be used. These models combine prior knowledge defined in the data pattern (such as field names and expected type ranges) with the actual content of the original business data to perform probabilistic analysis on the data structure (such as separators, field order, and nesting level) and field type (such as integer, string, date, and boolean value) of each field. The inference process outputs a confidence score, which represents the model's confidence in the inference result. For example, if the model is highly confident that a field is a date type, the confidence score will be high; conversely, if there are multiple possibilities, the confidence score will be relatively low.

[0029] Finally, in response to the inference confidence level being lower than a preset threshold, the original business data is determined to be risky and diverted to an isolation zone. The original business data in the isolation zone is then reviewed, and the data pattern dictionary is updated based on the review results. The preset threshold is a configurable parameter used to define the degree of data risk. When the inference confidence level output by the probability model is lower than this threshold, it indicates a significant deviation between the actual structure or type of the original business data and the definition in the data pattern dictionary, potentially indicating compliance risks or data quality issues. In this case, the system marks the original business data as "risky" and diverts it to an independent isolation zone. Data in the isolation zone does not enter the normal business processing flow but awaits manual or semi-automated review. Reviewers will conduct a detailed analysis of the data in the isolation zone to identify the reasons for the low confidence level, such as new data formats, changes in business rules, data entry errors, or malicious tampering. Based on the review results, the data pattern dictionary can be updated to accommodate new data patterns or correct erroneous pattern definitions, thereby improving the accuracy of subsequent data processing.

[0030] The big data compliance risk control method provided in this application works by constructing an adaptive, closed-loop risk identification and pattern learning mechanism. First, by adding and encapsulating metadata information to the original business data, standardized input is provided for subsequent data pattern matching and risk inference. The data source identifier, data type, and data version number in the metadata information enable the system to quickly locate the range of possible data patterns.

[0031] Subsequently, the system uses this metadata information to perform a preliminary match in the data pattern dictionary, thereby obtaining the known data pattern that is closest to the current data. This step effectively narrows the search space for subsequent probabilistic inference and improves efficiency. After obtaining the preliminary matching data pattern, the probabilistic model is used to perform refined structure and field type inference on the original business data. The probabilistic model can handle the uncertainty and variability in the data and outputs an inference confidence score, quantifying the reliability of the inference result.

[0032] When the inference confidence level falls below a preset threshold, it indicates that the original business data may contain anomalies that deviate from known patterns, thus being identified as having potential risks. In this case, this risky data will not directly enter the normal business process but will be diverted to an isolation zone. The isolation zone is designed to prevent the spread of potential risks and provide a secure environment for manual review. Within the isolation zone, professionals conduct in-depth reviews of the risky data, analyze the causes of its anomalies, and identify new data patterns or the necessity of correcting existing patterns.

[0033] Ultimately, based on the review results, the data pattern dictionary is updated. This update mechanism enables the system to learn from new data patterns, continuously improving its understanding and identification capabilities of business data. In this way, the method of this application can effectively address the challenges brought about by business innovation and data evolution, avoiding the problems of frequent parser adjustments and temporary script defects caused by data format changes in traditional methods. The entire process forms a closed loop of continuous learning and improvement, enabling the big data compliance risk control system to continuously adapt and optimize with business development, thereby significantly improving the accuracy and efficiency of risk identification and reducing potential compliance risks.

[0034] Compared to existing technologies, the big data compliance risk control method provided in this application demonstrates significant innovation and technological advancement. Traditional methods often rely on predefined, static data format specifications. When business data formats undergo minor changes or new data types emerge, manual intervention is required to modify the parser or write temporary adaptation scripts. This approach is not only inefficient, but temporary scripts are also prone to introducing logical flaws, leading to potential compliance risks being incorrectly labeled as normal data. This, in turn, contaminates the training data of machine learning models, creating a negative feedback loop that continuously diminishes the ability to identify risks.

[0035] The core innovation of this application lies in the introduction of metadata information and a probabilistic model, and the construction of an adaptive data pattern dictionary update mechanism. By adding metadata information such as data source identifiers, data types, and data version numbers to the original business data, the system can perform preliminary pattern matching more intelligently, rather than simply relying on hard-coded format definitions. The application of the probabilistic model enables the system to infer data structures and field types probabilistically and output confidence levels, thereby identifying data that deviates from known patterns but is not entirely wrong, avoiding the "black and white" judgment patterns of traditional methods. More importantly, this application forms a continuous learning and evolution mechanism by diverting low-confidence data to an isolated area for review and dynamically updating the data pattern dictionary based on the review results. This mechanism allows the system to proactively adapt to new data patterns and business changes, promptly correct and improve its risk identification capabilities, and effectively avoid the accumulation of label noise and model parameter drift. For example, when a new business data format emerges, traditional methods may require weeks or even months to update and deploy the parser; however, the method in this application can incorporate the new pattern into the recognition scope in a shorter time through isolation zone review and pattern dictionary updates, significantly improving response speed and accuracy. This adaptive capability is not available in existing technologies, enabling enterprises to stay ahead in big data compliance risk control and effectively cope with the rapidly changing business environment and increasingly stringent compliance requirements.

[0036] Specifically, the steps described above, which involve performing a preliminary match in the data pattern dictionary based on the received metadata information to determine the data pattern matching the metadata information, include: using the data source identifier, data type, and data version number from the metadata information as keys, matching in the data pattern dictionary, extracting a list of historical fields, prior types, and nesting relationships that match the metadata information, and forming the data pattern of the current data. Here, the data source identifier, data type, and data version number from the metadata information are used as a composite key. The data source identifier is used to uniquely identify the original source of the data, such as a specific business system, database, or external interface. The data type indicates the overall format or content category of the data, such as log data, transaction data, or user behavior data. The data version number is used to distinguish the evolution of the same data source and type at different points in time or under different specifications. By combining these three as a key, an accurate and unique lookup in the data pattern dictionary can be ensured.

[0037] The data pattern dictionary is a pre-built knowledge base storing a large number of known and verified data patterns. Each data pattern is associated with a specific data source identifier, data type, and data version number, and includes a list of historical fields, the prior types of each field, and the nesting relationships between fields. When new metadata information is received, the system uses the aforementioned composite key to quickly match it in the data pattern dictionary. Upon successful matching, the system extracts from the data pattern dictionary a list of historical fields, the prior types of each field, and the nesting relationships between fields that match the current metadata information. The list of historical fields lists all field names expected to appear under that specific data pattern. The prior types provide the expected value types for these fields, such as strings, integers, floating-point numbers, and dates. The nesting relationships describe the hierarchical structure between fields; for example, in JSON or XML formats, a field may contain multiple subfields. By extracting this information, a preliminary data pattern for the current data can be formed, providing a foundation for subsequent data structure and field type inference.

[0038] This application's solution uses the data source identifier, data type, and data version number from metadata information as unique identification keys to efficiently match patterns in a data pattern dictionary, enabling rapid and accurate retrieval of historical data patterns corresponding to the current data stream. This composite key-based matching mechanism effectively utilizes knowledge accumulated from historical data, avoiding the complexity of performing pattern recognition on each new data stream from scratch. Therefore, it provides a structured, prior-knowledge-based benchmark for subsequent probabilistic model inference, significantly improving the efficiency and accuracy of pattern inference.

[0039] In some embodiments described above in this application, a preliminary match is performed in the data pattern dictionary based on received metadata information to determine the matched data pattern, and then a probabilistic model is used to infer the data structure and field types. However, in practical applications, if the choice of the probabilistic model is not specific enough or the inference process is not refined enough, it may lead to insufficient accuracy in inferring the structure and field types of complex or evolving data patterns, thereby affecting the accuracy of compliance risk identification. Therefore, this application further proposes to concretize the probabilistic model and elaborates on the specific steps for inferring the data structure and field types to improve the accuracy and reliability of the inference.

[0040] The probabilistic model includes a Bayesian inference model or a conditional random field model. It infers the data structure and field types within the metadata information under the data pattern based on the probabilistic model and outputs the inference confidence score. This includes: analyzing and obtaining the structural features of the original business data based on the fields in the data pattern and their corresponding prior types; the structural features include delimiters, field order, value type distribution, and nesting level; performing probabilistic analysis based on the structural features, calculating the posterior confidence score of each field type and semantic, and generating a structural inference result with the posterior confidence score. The Bayesian inference model is a probabilistic inference method based on Bayes' theorem, which calculates the posterior probability by combining prior probabilities and observed data, and is suitable for handling uncertain information and performing pattern recognition. The conditional random field model is a discriminative probabilistic model, commonly used in sequence labeling and structured prediction tasks. It can effectively capture contextual dependencies in data sequences and has advantages in inferring data patterns with complex structures. In practical applications, one or two models can be selected and combined according to the characteristics of the specific business data and inference requirements.

[0041] Specifically, the structural features refer to the description of the inherent organizational form and content attributes of the original business data. The separators are symbols used to distinguish data fields, such as commas, tabs, or pipes; the field order refers to the order in which fields appear in a data record; the value type distribution refers to the frequency and proportion of different data types (such as strings, integers, floating-point numbers, dates, etc.) in a given field; and the nesting level refers to the depth and complexity of substructures or subfields within the data structure. Analyzing these structural features helps to comprehensively understand the inherent patterns of the original business data.

[0042] Furthermore, the probabilistic analysis refers to using a selected probabilistic model (such as a Bayesian inference model or a conditional random field model) combined with the analyzed structural features to quantitatively evaluate the type and semantics of each field. By calculating the posterior confidence of each field's type and semantics, the probability that a certain field belongs to a specific type or has a specific semantic under a given data pattern and structural features can be obtained. Thus, a structural inference result with the aforementioned posterior confidence is generated. This result not only indicates the inferred data structure and field type but also provides a measure of the inference's reliability.

[0043] This application's solution, by explicitly specifying the probabilistic model as a Bayesian inference model or a conditional random field model and refining the inference process, can more accurately capture the inherent structure and field types of the original business data. Specifically, by analyzing structural features such as delimiters, field order, value type distribution, and nesting levels, rich input information is provided to the probabilistic model. Based on this, the probabilistic model can perform more refined probabilistic analysis, calculating the posterior confidence of each field type and semantic, thereby overcoming the problem of insufficient inference accuracy that may exist in traditional generalized probabilistic models. Especially when facing data pattern evolution or complex data structures, it can provide more reliable inference results.

[0044] As a specific implementation method, a concrete example is given below. Assume the system receives a batch of raw business data. Its metadata indicates that the data originates from a certain log system, the data type is text log, and the version number is 1.0. The matching module performs a preliminary match in the data pattern dictionary based on this metadata information, determining a historical log data pattern. This pattern includes a list of fields, prior types, and nesting relationships.

[0045] In the probabilistic inference module, a Conditional Random Field (CRF) model can be used for inference. First, based on the fields and their prior types within the log data pattern, structural feature analysis is performed on the original business data. For example, the delimiter in log entries is identified as "|", the field order as "timestamp|user ID|operation type|result code", the user ID field's value type distribution is mainly a combination of numbers and letters, and there may be nested JSON-formatted "result code" fields. Next, the CRF model uses these structural features to perform probabilistic analysis, calculating the posterior confidence of each field's type (e.g., date / time, string, enumeration value, JSON object) and semantics (e.g., event occurrence time, operation executor, specific operation, operation feedback). For example, the model might infer that the "timestamp" field is a date / time type with a confidence of 0.98; and the "result code" field is a JSON object type with a confidence of 0.95. Finally, a structural inference result with the aforementioned posterior confidence is generated, which clearly describes the precise structure of the log data and the type and semantics of each field, providing a high-confidence input for subsequent risk assessment.

[0046] This application further proposes to review the original business data of the quarantine area, such as Figure 2 As shown, the steps for updating the data schema dictionary based on the review results include: S201, adjust the allocation priority of review resources according to the frequency and complexity of the pattern evolution corresponding to the original business data of the isolation zone; S202, Based on manual review of the original business data, the review results are structured and the key features and semantic correction rules of the pattern evolution are extracted; S203, compare the structured review results with the data pattern dictionary to determine the lagging part in the data pattern dictionary; S204, Generate update information for the data pattern dictionary based on the lagging portion, and perform data pattern dictionary update.

[0047] Specifically, when reviewing the original business data in the quarantine zone and updating the data schema dictionary, the priority of review resource allocation is first adjusted based on the frequency and complexity of schema evolution corresponding to the original business data in the quarantine zone. The frequency of schema evolution refers to the number of times a specific data schema changes within a certain period, while complexity refers to indicators such as the number of fields involved in the schema changes, data type changes, and the depth of nested structures. By analyzing these indicators, limited review resources, such as human reviewers or automated review tools, can be dynamically allocated to original business data with frequent or complex schema evolution, ensuring that high-risk or high-change-rate data receives timely and thorough review.

[0048] Furthermore, the original business data is reviewed manually. Human review leverages expertise and experience to identify subtle pattern changes and potential compliance risks that automated tools struggle to detect. The review results are then structured to transform unstructured or semi-structured review comments and findings into a standardized, machine-readable format. This allows for the extraction of key features of pattern evolution and semantic correction rules. Key features refer to the core elements of data pattern changes, such as new fields, changes in field types, and adjustments to enumeration value ranges. Semantic correction rules refer to how to adjust the data pattern dictionary to accurately reflect the new data semantics and structure in response to these pattern changes. Subsequently, the structured review results are compared with the data pattern dictionary to identify lagging parts. Lagging parts refer to portions of the data pattern dictionary that fail to reflect the actual changes in the current original business data patterns, such as missing newly introduced fields, field types that do not match reality, or outdated semantic definitions. Through comparison, the specific content in the dictionary that needs updating can be precisely identified. Finally, update information for the data pattern dictionary is generated based on the identified lagging parts, and the data pattern dictionary is updated accordingly. Updated information may include adding new schema entries, modifying existing schema definitions, and deleting obsolete schemas. Performing a data schema dictionary update aims to integrate this updated information into the data schema dictionary, ensuring that the dictionary accurately and promptly reflects the latest data schemas and compliance requirements.

[0049] This application's solution, by incorporating considerations of the frequency and complexity of pattern evolution, achieves intelligent allocation of review resources, avoiding waste and improving review efficiency. It is precisely because of the structured processing of review results and the extraction of key features and semantic correction rules that the experience and knowledge of manual review can be systematically captured and utilized, providing a precise basis for subsequent dictionary updates. By comparing the structured review results with the data pattern dictionary, lagging parts in the dictionary can be accurately identified, ensuring the relevance and effectiveness of updates. Finally, updates are generated and executed based on the lagging parts, enabling the data pattern dictionary to continuously keep pace with the evolution of actual business data, thereby effectively addressing the ever-changing compliance risks in the big data environment.

[0050] In some preferred embodiments, a specific example is given below. Suppose that a financial institution's transaction data stream introduces a new transaction type due to new business needs, and its data structure includes a new field, "Risk Rating Code." Initially, this field is not included in the data pattern dictionary. When the original business data containing this new field is acquired and metadata information is added, during the initial matching and probability inference stages, the presence of the "Risk Rating Code" field may cause the inference confidence level to be lower than a preset threshold, thus diverting this original business data to an isolated area.

[0051] Within the quarantine zone, the system prioritizes the allocation of review resources based on the frequency of the transaction data pattern's evolution (e.g., the frequency of the new transaction type's appearance) and complexity (e.g., the number and type of newly added fields), prioritizing human reviewers for this batch of data. Human reviewers identify the "risk rating code" as a new key feature, determining its data type to be a string with a specific range of enumerated values. The review results are then structured, extracting the "risk rating code" key feature and its semantic correction rules (e.g., this field represents the transaction's risk level, with values ​​ranging from A to D).

[0052] Subsequently, the system compares the structured review results with the current data pattern dictionary. The comparison results show that the data pattern dictionary lacks a definition for the "risk rating code" field, which is identified as a lagging part of the data pattern dictionary. Based on this lagging part, the system generates update information, including adding the definition of the "risk rating code" field, its data type, and its enumeration value range to the existing transaction data pattern. Finally, the data pattern dictionary is updated, integrating the new pattern features into the data pattern dictionary. As a result, subsequent transaction data containing the "risk rating code" field will be able to be correctly identified and processed, thereby ensuring data compliance and avoiding misjudgments of risk due to pattern mismatch.

[0053] In some of the embodiments described above in this application, a scheme is proposed to review the original business data of the isolated area and update the data pattern dictionary based on the review results. However, when structuring the review results and extracting key features of pattern evolution and semantic correction rules, failure to effectively identify and resolve potential conflicts between old and new pattern knowledge, or failure to reasonably prioritize and integrate new knowledge, may lead to a decline in the quality of the data pattern dictionary update, or even introduce new compliance risks. Therefore, this application further proposes an optimized scheme for structuring the review results and extracting key features of pattern evolution and semantic correction rules to ensure a more robust and accurate data pattern dictionary update process.

[0054] Specifically, the above-mentioned structuring of the review results, extracting key features of pattern evolution and semantic correction rules, includes the following steps: The new pattern features and semantic correction rules identified in the review results are pre-analyzed to determine whether they semantically conflict with the data pattern dictionary. The purpose of this pre-analysis is to identify potential inconsistencies or contradictions before integrating new knowledge into the data pattern dictionary, thereby avoiding the introduction of errors or confusion in existing pattern inference logic.

[0055] Based on preset priority rules, identified semantic conflicts are resolved to obtain new pattern features and semantic correction rules. Semantic conflict resolution aims to resolve contradictions between old and new knowledge, ensuring the internal consistency of the data pattern dictionary. Preset priority rules can be set based on factors such as business importance, data volume, and risk impact to guide the selection of conflict resolution strategies.

[0056] Key features or semantic rules identified in the review results but missing from the data pattern dictionary are marked and prioritized according to business importance, data frequency, and potential impact on the pattern inference engine, resulting in a ranked list of new knowledge to be integrated. This step aims to identify and prioritize new knowledge that is crucial for data compliance risk control but is not currently covered in the data pattern dictionary, ensuring the timely integration of critical information.

[0057] Based on the aforementioned priority, the resolved new pattern features, semantic correction rules, and sorted new knowledge to be integrated are fused with the data pattern dictionary, and a semantic compatibility check is performed to obtain an updated data pattern dictionary. The fusion process integrates the processed new knowledge into the existing data pattern dictionary, while the semantic compatibility check serves as the final quality control step, ensuring that the updated data pattern dictionary maintains overall logical consistency and validity.

[0058] This application's solution systematically addresses semantic conflicts and knowledge integration issues that may arise when updating a data pattern dictionary by introducing steps such as pre-analysis, conflict resolution, prioritization, and fusion and compatibility checks. Specifically, the pre-analysis step proactively identifies potential contradictions between new pattern features and semantic correction rules and the existing data pattern dictionary, thus preventing the direct introduction of inconsistent knowledge into the system. Subsequently, pre-defined priority rules resolve identified semantic conflicts, ensuring that reasonable decisions are made based on established strategies when faced with contradictions, resulting in consistent new pattern features and semantic correction rules. Furthermore, for key features or semantic rules missing from the data pattern dictionary, this solution marks them and prioritizes them based on business importance, data frequency, and the potential impact on the pattern inference engine, ensuring the priority integration of core and high-value new knowledge and avoiding the omission of important information. Finally, by integrating the resolved new pattern features and semantic correction rules with the sorted new knowledge to be integrated into the data pattern dictionary and conducting a rigorous semantic compatibility check, the updated data pattern dictionary is ensured to not only contain the latest pattern knowledge, but also maintain a high degree of semantic consistency and integrity, thereby significantly improving the accuracy and reliability of the data pattern dictionary.

[0059] In some preferred embodiments, a specific example is given below. Suppose that in a financial institution's data pattern dictionary, the "Customer Age" field is defined as an integer with a value range of 18-100. Now, by reviewing the original business data in the segregated area, it is discovered that in a new batch of business data, the "Customer Age" field is defined as a string type and contains non-numeric values ​​such as "Unknown" or "N / A". Additionally, a completely new field, "Customer Credit Rating," is also discovered, which does not exist in the existing data pattern dictionary.

[0060] First, during the pre-analysis of new pattern features (e.g., "Customer Age" is a string type containing an "Unknown" value) and semantic correction rules (e.g., mapping "Unknown" to a specific default value) identified in the review results, the system will identify data type conflicts (integer vs. string) and value range conflicts (18-100 vs. including "Unknown"). Simultaneously, the system will identify "Customer Credit Rating" as a completely new key feature missing from the data pattern dictionary. Second, according to preset priority rules, for example, if the compliance risk impact of the "Customer Age" field is high, the priority for resolving its semantic conflicts will be increased. The system may choose a resolution strategy, such as correcting the type of the "Customer Age" field to support a mixed type of string and integer, and defining compliance handling rules for "Unknown," thereby obtaining the resolved new pattern features and semantic correction rules. Secondly, for the missing key feature "customer credit rating," the system prioritizes it based on its business importance (e.g., crucial for risk assessment), data frequency (e.g., prevalent in new business data), and potential impact on the pattern inference engine. Features with higher priority are marked as high-priority new knowledge to be integrated. Finally, based on these priorities, the system merges the new pattern features and semantic correction rules of the resolved "customer age" field, along with the prioritized "customer credit rating" new knowledge to be integrated, with the data pattern dictionary. During the fusion process, the system performs semantic compatibility checks, ensuring, for example, that the new "customer age" definition does not create new conflicts with other patterns that rely on it as a pure integer, and that the introduction of the "customer credit rating" field does not disrupt the integrity of existing data patterns. Through these steps, an updated data pattern dictionary is ultimately obtained, which can accurately identify and process new business data containing "unknown" age values ​​and can identify and utilize the new key business information, "customer credit rating," thereby improving the comprehensiveness and accuracy of big data compliance risk control.

[0061] In some of the embodiments described above in this application, when structuring the examination results and extracting key features and semantic correction rules for pattern evolution, it is necessary to pre-analyze these new pattern features and semantic correction rules to determine whether they have semantic conflicts with the data pattern dictionary. Specifically, the pre-analysis of the new pattern features and semantic correction rules identified in the examination results to determine whether they have semantic conflicts with the data pattern dictionary includes: Determine the business domain identifier associated with the new pattern features and semantic correction rules, and extract relevant pattern knowledge related to the business domain identifier and cross-domain pattern knowledge associated with the relevant pattern knowledge from the data pattern dictionary based on the business domain identifier; The semantic representations of the new pattern features and semantic correction rules are compared with the semantic representations of the related pattern knowledge and the cross-domain pattern knowledge to determine whether a semantic conflict occurs.

[0062] The new pattern features and semantic correction rules refer to new structures, fields, value ranges, or correction rules for existing patterns identified from the review results of the original business data in the isolated area, reflecting the evolution of data patterns. These new pattern features and semantic correction rules need to be integrated into the data pattern dictionary to maintain its real-time performance and accuracy. The business domain identifier can be understood as a specific business category or classification label related to the original business data or its pattern features, such as "financial transactions," "user behavior," or "logistics information." By determining the business domain identifiers associated with the new pattern features and semantic correction rules, the scope for matching and comparison in the data pattern dictionary can be effectively narrowed, improving the efficiency and accuracy of semantic conflict detection.

[0063] The data pattern dictionary is a knowledge base storing known data patterns, field definitions, semantic rules, and their interrelationships. It provides a benchmark and reference for big data compliance risk control. The relevant pattern knowledge refers to information in the data pattern dictionary that is directly related to the business domain identifier associated with the current new pattern features and semantic correction rules, such as pattern definitions, field types, and semantic constraints. The cross-domain pattern knowledge refers to pattern knowledge related to the business domain identifier but shared or associated across different business domains; for example, certain common data types or coding standards may be common across multiple business domains. The purpose of extracting this knowledge is to comprehensively assess the potential semantic impact of the new pattern features and semantic correction rules. The semantic representation refers to transforming the new pattern features, semantic correction rules, and existing knowledge in the data pattern dictionary into a machine-understandable and comparable semantic form using technologies such as natural language processing, knowledge graphs, and vector embedding. For example, field names, descriptions, and enumeration values ​​can be encoded as vectors for similarity calculation. The semantic representations of the new pattern features and semantic correction rules are compared with the semantic representations of the related pattern knowledge and the cross-domain pattern knowledge. The aim is to determine whether the introduction of new knowledge will lead to semantic inconsistencies or ambiguities within the data pattern dictionary by calculating their semantic similarity, consistency or conflict.

[0064] This application's solution first identifies the business domain identifiers associated with new pattern features and semantic correction rules, enabling precise extraction of the most relevant pattern knowledge and cross-domain pattern knowledge from a vast data pattern dictionary. This focused extraction method avoids blind comparison of the entire dictionary, significantly improving detection efficiency. Subsequently, by comparing the semantic representations of the new pattern features and semantic correction rules with the semantic representations of these extracted relevant pattern knowledge and cross-domain pattern knowledge, a deep semantic analysis can be conducted to determine whether inconsistencies, contradictions, or ambiguities exist between the new and old knowledge. For example, if a new pattern feature defines a field with the same name but completely different meaning as an existing pattern, or if a semantic correction rule conflicts with existing rules, potential semantic conflicts can be identified promptly. This ensures that newly introduced pattern knowledge undergoes thorough semantic verification before updating the data pattern dictionary, thereby maintaining the internal consistency and accuracy of the data pattern dictionary.

[0065] In some of the embodiments described above in this application, when pre-analyzing the new pattern features and semantic correction rules identified in the examination results, it is necessary to determine whether they semantically conflict with the data pattern dictionary. Specifically, in order to accurately determine the occurrence of semantic conflicts, this application proposes a more refined method for determining semantic conflicts.

[0066] Specifically, the semantic representation of the new pattern feature and the semantic correction rule is compared with the semantic representation of the related pattern knowledge and the cross-domain pattern knowledge to determine whether a semantic conflict occurs. This includes determining whether the new pattern feature or the semantic correction rule and the related pattern knowledge or the cross-domain pattern knowledge have any of the following characteristics: data type mismatch, enumeration value range conflict, or semantic ambiguity of key fields. If any of these exist, it is determined that a semantic conflict exists.

[0067] Specifically, the data type mismatch refers to an incompatibility between the expected data type (e.g., integer, string, date, etc.) of a field in the new pattern feature or semantic correction rule and the data type already defined for the corresponding field in the data pattern dictionary. For example, the new rule might define a "quantity" field as a string type, while the existing dictionary defines this field as an integer type. The enumeration value range conflict refers to a situation where, when a field is defined to accept only a preset set of enumeration values, the new pattern feature or semantic correction rule introduces a value that exceeds or contradicts the existing enumeration value range. For example, the existing "status" field allows "active," "inactive," and "pending," while the new rule introduces "completed," but it is not defined as an allowed value. The semantic ambiguity of key fields refers to a situation where, even if the data types of key fields are compatible, their meaning or interpretation in the new pattern feature or semantic correction rule differs from the meaning established in the data pattern dictionary, leading to ambiguity. For example, a "code" field might refer to a product code in one context and a region code in another.

[0068] This application's solution makes the semantic conflict detection process more precise and automated by explicitly defining specific types of semantic conflicts. When the semantic representation of a new pattern feature or semantic correction rule is compared with relevant pattern knowledge and cross-domain pattern knowledge in the data pattern dictionary, it no longer simply judges "whether a conflict exists," but specifically checks for three core types of conflicts: data type mismatch, enumeration value range conflict, or semantic ambiguity of key fields. This meticulous checking mechanism can effectively identify potential data parsing errors, data transformation inconsistencies, and business logic misunderstandings, thereby providing accurate conflict location information for subsequent conflict resolution and data pattern dictionary updates.

[0069] This application further proposes a method for resolving semantic conflicts identified above based on preset priority rules, resulting in new pattern features and semantic correction rules after resolution, including: Determine the type of semantic conflict, and calculate the resolution priority of the semantic conflict based on the type of semantic conflict, the business domain associated with the semantic conflict, the amount of data involved in the semantic conflict, and the expected degree of compliance risk impact of the semantic conflict; Based on the resolution priority, a resolution strategy is selected to resolve semantic conflicts in order to obtain new pattern features and semantic correction rules after resolution. Monitor the feedback on the processing effect of the resolution strategy on the business data stream, and optimize the calculation rules for resolution priority based on the processing effect feedback.

[0070] Specifically, semantic conflicts can be categorized into, but are not limited to, any of the following: data type mismatch, enumeration value range conflict, or semantic ambiguity of key fields. These types have been identified in the aforementioned implementation. When calculating the resolution priority of semantic conflicts, multiple dimensions are considered. For example, the business domain associated with the semantic conflict can indicate its importance to a specific business process or data asset; the amount of data involved in the semantic conflict can reflect the breadth of its impact; and the expected compliance risk impact of the semantic conflict directly quantifies the potential risks it may bring, such as data leakage, data misuse, or regulatory violations. By weighting these factors, a quantified resolution priority can be obtained.

[0071] The selection of resolution strategies is based on calculated resolution priorities. For example, for high-priority conflicts, automatic correction or immediate manual intervention may be prioritized; for medium-priority conflicts, a strategy of issuing warnings and awaiting manual confirmation may be adopted; and for low-priority conflicts, they may simply be recorded or processed in batches under specific conditions. The goal of the resolution process is to generate new pattern features and semantic correction rules that are compatible with the existing data pattern dictionary and meet business requirements.

[0072] In practical applications, to ensure the effectiveness and adaptability of the resolution strategy, it is necessary to continuously monitor its processing effect on the business data flow. Monitoring data can include data parsing success rate, data transformation consistency, field value distribution deviation, and the proportion of abnormal data. Based on this feedback, the calculation rules for resolution priority can be optimized, such as adjusting the weights of various influencing factors or introducing new considerations, thereby making priority calculation more accurate and resolution processing more efficient.

[0073] This application's solution introduces a semantic conflict resolution priority calculation mechanism. This mechanism enables refined evaluation of identified semantic conflicts based on multiple dimensions, including conflict type, business impact, data volume, and compliance risk. This ensures that limited review resources are prioritized for conflicts with the greatest business impact or highest compliance risk. Furthermore, by selecting appropriate resolution strategies based on resolution priority, a "one-size-fits-all" approach is avoided, allowing conflicts of different natures to be resolved most appropriately. Moreover, by continuously monitoring the processing effect of resolution strategies on business data flows and optimizing the resolution priority calculation rules accordingly, this application's solution achieves self-learning and adaptation. This allows the big data compliance risk control system to continuously improve its conflict resolution capabilities as business data patterns evolve, significantly enhancing the system's robustness and accuracy in handling complex data environments.

[0074] In some preferred embodiments, a specific example is given below. Suppose that in a big data compliance risk control scenario of a financial institution, the system continuously receives raw business data from different business systems (such as transaction systems and customer management systems).

[0075] When the system detects semantic conflicts in the data schema dictionary, such as: 1. Conflict A: The data type of the "Transaction Amount" field in the transaction system has changed from integer to floating-point, while some historical data remains in integer format. 2. Conflict B: The encoding rules for the "Customer ID" field in the customer management system have changed, leading to semantic ambiguity with another field, "Account ID," in certain situations. 3. Conflict C: Newly introduced regulatory requirements stipulate that all transaction data must include an "Anti-Money Laundering Risk Level" field, but this field is missing from the existing data schema.

[0076] At this point, the proposed solution will calculate the resolution priority for these semantic conflicts: For conflict A, its type is data type mismatch, the related business domain is transactions, and the amount of data involved may be large, but the expected compliance risk impact is relatively low (usually only affecting accuracy and not directly causing major violations). Therefore, its resolution priority may be calculated as medium to low. The system may choose a resolution strategy of automatic conversion or issuing an early warning when conversion fails.

[0077] For conflict B, its type is semantic ambiguity in a key field, the related business area is customer management and transactions, the amount of data involved may be moderate, but the expected compliance risk is high (potentially leading to customer identification errors, fraud, or compliance vulnerabilities). Therefore, its resolution priority may be calculated as high. The system may choose to immediately trigger a manual review and require the business department to provide a resolution strategy with clear semantic correction rules.

[0078] For conflict C, its type is a missing critical field, the related business area is compliance and regulation, the amount of data involved may be all transaction data, and the expected compliance risk is extremely high (direct violation of regulatory requirements). Therefore, its resolution priority will be calculated as the highest. The system may choose to immediately block the relevant data flow and force the business system to supplement the field or provide an alternative resolution strategy.

[0079] After the resolution strategy is executed, the system continuously monitors the processing effect feedback of the business data flow. For example, for the automatic conversion strategy of conflict A, the system monitors the data parsing success rate and the deviation of field value distribution. If a large number of data parsing failures or abnormal distributions are found after conversion, it indicates that the strategy is not effective. The system will optimize the calculation rules for resolution priority based on the feedback, such as increasing the priority of such data type mismatch conflicts or adjusting the threshold for automatic conversion. For the manual review strategy of conflict B, the system monitors the structured processing of review results and the integration effect of new pattern features. If the manual review is found to be inefficient or the correction rules are unclear, the priority calculation rules will be optimized, such as adding consideration to the response speed of business departments. Through this continuous monitoring and optimization, the big data compliance risk control system can continuously improve its ability to identify, prioritize, and resolve complex semantic conflicts, ensuring the accuracy and compliance of the data pattern dictionary.

[0080] In some embodiments described above in this application, a method is proposed to calculate the resolution priority of semantic conflicts based on the type of semantic conflict, the associated business domain, the amount of data involved, and the expected impact of compliance risks, and to select a resolution strategy for processing to obtain new pattern features and semantic correction rules after resolution. To ensure the effectiveness of the selected resolution strategy and to continuously optimize it, it is necessary to monitor the processing effect of the resolution strategy in the actual business data flow. Specifically, monitoring the feedback of the resolution strategy's processing effect on the business data flow may include the following steps: Deploy probes at key nodes in the business data flow; wherein, the probes collect monitoring data in a non-blocking manner; the monitoring data includes data parsing success rate, data conversion consistency, field value distribution deviation, and abnormal data ratio; The monitoring data is preprocessed and aggregated locally to obtain aggregated indicator data, and then the aggregated indicator data is sent to the monitoring center. The monitoring center performs real-time streaming analysis on the aggregated index data to obtain analysis results; it compares the analysis results with a preset baseline, and if the results exceed the preset baseline, it triggers an alarm and records a snapshot of the abnormal data stream to evaluate the actual processing effect of the mitigation strategy.

[0081] Specifically, the key nodes refer to important processing links in the entire data flow chain, from the source of business data generation, through cleaning, transformation, storage, to final application, such as the data access layer, data processing layer, data storage layer, or data consumption layer. Deploying probes at these key nodes aims to comprehensively cover all stages of the data flow, ensuring the completeness of monitoring. The probes collect monitoring data in a non-blocking manner, meaning that their data collection process will not cause delays or interruptions to normal business data flow processing, thereby ensuring the stable operation of the business system. The monitoring data, such as data parsing success rate, measures whether the data can be correctly identified and structured; data conversion consistency assesses whether the semantic and content accuracy is maintained when data is converted between different formats or structures; field value distribution deviation detects whether the value of a specific field deviates from the expected range or pattern, potentially indicating data anomalies; and the proportion of abnormal data directly reflects the amount of data in the data flow that does not conform to preset rules or patterns.

[0082] Furthermore, the collected monitoring data undergoes local preprocessing and aggregation. Local preprocessing may include data cleaning, format standardization, and deduplication to improve data quality. Aggregation involves summarizing the scattered monitoring data according to dimensions such as time windows, business types, or key nodes, forming aggregated indicator data with more macro-level insights. For example, the success rate of data parsing and the average field value distribution deviation of a key node over a period of time can be calculated. This aggregated indicator data is then sent to the monitoring center for centralized analysis and management.

[0083] At the monitoring center, the aggregated indicator data undergoes real-time streaming analysis. Real-time streaming analysis refers to the immediate processing and analysis of continuously flowing data, enabling rapid detection of anomalies or trend changes in the data stream. This analysis yields results regarding the actual effectiveness of the mitigation strategy. These results are compared to a preset baseline, which is a range of normal operating indicators set based on historical data, business needs, or compliance standards. If the analysis results exceed the preset baseline—for example, a significant decrease in data parsing success rate or a sharp increase in the proportion of abnormal data—an alarm is triggered, promptly notifying relevant personnel to intervene. Simultaneously, the system records snapshots of the abnormal data stream; that is, when an anomaly occurs, detailed information about the relevant data stream is captured and saved. These snapshots are invaluable for subsequent problem investigation, root cause analysis, and evaluation of the actual effectiveness of the mitigation strategy. They help operations personnel deeply understand the causes and impacts of problems and provide data support for optimizing mitigation strategies.

[0084] This application's solution deploys non-blocking probes at key nodes of the business data flow, enabling real-time, non-intrusive collection of key indicators such as data parsing success rate, data transformation consistency, field value distribution deviation, and the proportion of abnormal data. This allows for direct feedback on the data processing effectiveness of the mitigation strategy in actual business scenarios. Local preprocessing and aggregation of this monitoring data effectively reduces the load on transmission and analysis, while ensuring data timeliness and accuracy. Subsequently, the monitoring center performs real-time streaming analysis on the aggregated indicator data and compares it with a preset baseline, enabling any anomalies deviating from the normal range to be quickly identified and trigger alarms. Simultaneously, snapshots of the abnormal data flow are recorded, providing detailed contextual information for subsequent in-depth analysis and problem localization. It is precisely this end-to-end, real-time feedback mechanism that allows the actual processing effect of the mitigation strategy to be objectively and accurately evaluated, providing a solid data foundation for subsequent strategy optimization.

[0085] Furthermore, this application proposes a big data compliance risk control system, such as... Figure 3 As shown, the system includes: The acquisition and encapsulation module 301 is used to acquire raw business data, add metadata information to the raw business data, encapsulate it, and send it; the metadata information includes data source identifier, data type, and data version number. The matching module 302 is used to perform a preliminary matching in the data pattern dictionary based on the received metadata information to determine the data pattern that matches the metadata information; The probability inference module 303 is used to infer the data structure and field type within the metadata information under the data pattern based on the probability model, and output the inference confidence level; The risk diversion module 304 is used to determine that the original business data is at risk and divert it to the isolation zone in response to the inference confidence level being lower than a preset threshold, and to review the original business data in the isolation zone and update the data pattern dictionary based on the review results.

[0086] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A big data compliance risk control method, characterized in that, The method comprises: Obtaining original business data, adding metadata information to the original business data, encapsulating and sending; the metadata information includes data source identification, data type, data version number; According to the received metadata information, preliminary matching is performed in the data mode dictionary to determine the data mode matched with the metadata information; According to the probability model, the data structure and field type in the metadata information under the data mode are inferred, and the inference confidence is output; In response to the inference confidence being lower than the preset threshold, it is determined that the original business data is at risk and is diverted to the isolation area, and the original business data in the isolation area is reviewed, and the data mode dictionary is updated according to the review result.

2. The big data compliance risk control method of claim 1, wherein, According to the received metadata information, preliminary matching is performed in the data mode dictionary to determine the data mode matched with the metadata information, comprising: Taking the data source identification, data type, and data version number in the metadata information as the key, matching is performed in the data mode dictionary, the historical field list, prior type, and nesting relationship matched with the metadata information are extracted, and the data mode of the current data is formed.

3. The big data compliance risk control method of claim 2, wherein, The probability model includes a Bayesian inference model or a conditional random field model; According to the probability model, the data structure and field type in the metadata information under the data mode are inferred, and the inference confidence is output, comprising: According to the fields under the data mode and the corresponding prior type, the structural characteristics of the original business data are analyzed and obtained; the structural characteristics include separators, field order, value type distribution, and nesting level; According to the structural characteristics, probability analysis is performed to calculate the posterior confidence of each field type and semantic, and a structural inference result with the posterior confidence is generated.

4. The big data compliance risk control method of claim 1, wherein, The original business data in the isolation area is reviewed, and the data mode dictionary is updated according to the review result, comprising: According to the frequency and complexity of the mode evolution corresponding to the original business data in the isolation area, the allocation priority of the review resources is adjusted; Based on manual review of the original business data, the review result is structured, the key features of the mode evolution and the semantic correction rules are extracted; The structured review result is compared with the data mode dictionary to determine the lag part in the data mode dictionary; According to the lag part, update information of the data mode dictionary is generated to update the data mode dictionary.

5. The big data compliance risk control method of claim 4, wherein, The structured review result is compared with the data mode dictionary to determine the lag part in the data mode dictionary; According to the lag part, update information of the data mode dictionary is generated to update the data mode dictionary. The structured review result is compared with the data mode dictionary to determine the lag part in the data mode dictionary; According to the lag part, update information of the data mode dictionary is generated to update the data mode dictionary. label the key features or semantic rules determined in the review result and missing in the data schema dictionary, and prioritize according to business importance, data frequency, and potential impact on the schema inference engine to obtain sorted new knowledge to be integrated; According to the priority, the new mode feature and semantic correction rule after resolution and the sorted new knowledge to be integrated are fused with the data schema dictionary, and a semantic compatibility check is performed to obtain an updated data schema dictionary.

6. The big data compliance risk control method of claim 5, wherein, The pre-analysis of the new mode feature and semantic correction rule determined in the review result includes: determining the business domain identifier associated with the new mode feature and semantic correction rule, and extracting relevant mode knowledge related to the business domain identifier and cross-domain mode knowledge associated with the relevant mode knowledge from the data schema dictionary; compare the semantic representation of the new mode feature and semantic correction rule with the semantic representation of the relevant mode knowledge and the cross-domain mode knowledge to determine whether a semantic conflict occurs.

7. The big data compliance risk control method of claim 6, wherein, The comparison of the semantic representation of the new mode feature and semantic correction rule with the semantic representation of the relevant mode knowledge and the cross-domain mode knowledge to determine whether a semantic conflict occurs includes: determine whether the new mode feature or the semantic correction rule has any of the following: data type mismatch, enumeration value range conflict, and key field semantic ambiguity with the relevant mode knowledge and the cross-domain mode knowledge, if any, then determine that there is a semantic conflict.

8. The big data compliance risk control method of claim 5, wherein, The resolution processing of the determined semantic conflict according to the preset priority rule to obtain the new mode feature and semantic correction rule after resolution includes: determine the type of the semantic conflict, and calculate the resolution priority of the semantic conflict according to the type of the semantic conflict, the business domain associated with the semantic conflict, the data volume involved in the semantic conflict, and the expected compliance risk impact degree of the semantic conflict; According to the resolution priority, select a resolution strategy to resolve the semantic conflict to obtain the new mode feature and semantic correction rule after resolution; monitor the processing effect feedback of the resolution strategy on the business data stream, and optimize the calculation rule of the resolution priority according to the processing effect feedback.

9. The big data compliance risk control method of claim 8, wherein, The monitoring of the processing effect feedback of the resolution strategy on the business data stream includes: deploy a probe at a key node of the business data stream; wherein the probe collects monitoring data in a non-blocking manner; the monitoring data includes data parsing success rate, data conversion consistency, field value distribution deviation, and abnormal data proportion; perform local preprocessing and aggregation on the monitoring data to obtain aggregated index data, and send the aggregated index data to a monitoring center; The monitoring center performs real-time stream analysis on the aggregated index data to obtain an analysis result; according to the analysis result and a preset baseline, if the analysis result exceeds the preset baseline, an alarm is triggered, and a snapshot of the abnormal data stream is recorded to evaluate the actual processing effect of the resolution strategy.

10. A big data compliance risk control system, characterized in that, The system comprises: An acquisition and packaging module, configured to acquire original business data, add metadata information to the original business data, package and send the original business data; the metadata information comprises data source identification, data type, data version number; A matching module, configured to perform preliminary matching in a data mode dictionary according to the received metadata information, and determine a data mode matched with the metadata information; A probability inference module, configured to infer data structure and field type in the metadata information under the data mode according to a probability model, and output an inference confidence; A risk shunting module, configured to determine that the original business data has a risk and shunt the original business data to an isolation area in response to the inference confidence being lower than a preset threshold, review the original business data in the isolation area, and update the data mode dictionary according to a review result.

Images