A big data compliance risk control method and system

Abstract

The application provides a big data compliance risk control method and system, and relates to the technical fields of big data processing and risk control. By obtaining original business data and adding metadata information for encapsulation, the standardization and traceability of the data source are realized. Subsequently, the metadata information is preliminarily matched in the data mode dictionary, and the potential mode of the data can be quickly identified. The data structure and field type are inferred through a probability model, and the inference confidence is output, effectively solving the parsing error and label noise problems caused by the non-standard data format and the lagging parser update in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data processing and risk control, in particular to a big data compliance risk control method and system. BACKGROUND

[0002] In the complex data processing environment of large enterprises, business departments often do not strictly comply with data specifications in pursuit of efficiency, resulting in diverse and biased data source formats, which makes the data parser need to be frequently adjusted. During updates or in the face of unforeseen format variants, parsing delays or errors are easily caused.

[0003] To be compatible with non-standard data, engineers write temporary adaptation scripts. These scripts are effective in regular scenarios, but have hidden defects in edge cases that may incorrectly label abnormal data as normal. Such erroneous data can pass quality checks and enter the model training library, forming persistent label noise. Such noise causes the risk classification model to drift in parameters during retraining, reducing its ability to identify new and hidden risks. Unidentified risk data is further mislabeled as "normal", reinforcing model misjudgment and forming a negative feedback loop of continuously decreasing identification ability.

[0004] The prior art needs to be improved in view of the above problems. SUMMARY

[0005] The present application discloses a big data compliance risk control method and system, aiming to solve the technical problem that in a complex and variable enterprise data environment, traditional risk identification methods are difficult to effectively deal with massive, diverse and format-variant data streams, and business departments may not strictly comply with specifications in pursuit of efficiency, resulting in data parsing errors, adaptation script logic defects, label noise accumulation, and further reducing the identification sensitivity of machine learning models to emerging compliance risk patterns, forming a negative feedback loop.

[0006] The technical solution of the present application is as follows:

[0007] In a first aspect, the present application discloses a big data compliance risk control method, which comprises:

[0008] Obtaining original business data, adding metadata information to the original business data, encapsulating and sending the original business data; the metadata information includes data source identification, data type, and data version number;

[0009] Performing preliminary matching in a data mode dictionary according to the received metadata information, and determining a data mode matched with the metadata information;

[0010] Infer the data structure and field type in the metadata information under the data mode according to a probability model, and output an inference confidence;

[0011] In response to the inference confidence being lower than the preset threshold, it is determined that the original business data is at risk and is shunted to an isolated area, and the original business data in the isolated area is reviewed, and the data mode dictionary is updated according to the review result.

[0012] Through the technical solution, early identification and dynamic control of big data compliance risks can be achieved, potential risk data can be effectively identified through metadata encapsulation, data mode matching and probability inference, and the data mode dictionary is continuously optimized using the isolation review mechanism, thereby overcoming the shortcomings of traditional methods in handling non-standard data formats and identifying hidden risks.

[0013] Further, the preliminary matching in the data mode dictionary according to the received metadata information to determine the data mode matching the metadata information includes:

[0014] The data source identifier, data type, and data version number in the metadata information are used as keys to match in the data mode dictionary, and the historical field list, prior type, and nested relationship matching the metadata information are extracted to form the data mode of the current data.

[0015] On the basis of the above, the probability model includes a Bayesian inference model or a conditional random field model.

[0016] According to the probability model, the data structure and field type in the metadata information under the data mode are inferred, and an inference confidence is output, including: according to the fields and corresponding prior types under the data mode, the structural characteristics of the original business data are analyzed and obtained; the structural characteristics include separators, field order, value type distribution, and nested level; according to the structural characteristics, probability analysis is performed to calculate the posterior confidence of each field type and semantic, and a structural inference result with the posterior confidence is generated.

[0017] In some preferred embodiments, the original business data in the isolated area is reviewed, and the data mode dictionary is updated according to the review result, including: according to the frequency and complexity of the mode evolution corresponding to the original business data in the isolated area, the allocation priority of the review resources is adjusted; based on manual review of the original business data, the review result is structured, and the key features of the mode evolution and semantic correction rules are extracted; the structured review result is compared with the data mode dictionary to determine the lagging part in the data mode dictionary; the update information of the data mode dictionary is generated according to the lagging part, and the data mode dictionary update is performed.

[0018] Further, the review result is structured, key features of the schema evolution and semantic correction rules are extracted, including: pre-analyzing the new schema features and semantic correction rules determined in the review result to determine whether there is a semantic conflict with the data schema dictionary; according to the preset priority rules, the determined semantic conflicts are resolved to obtain the resolved new schema features and semantic correction rules; the key features or semantic rules determined in the review result and missing in the data schema dictionary are marked, and according to the business importance, data frequency and potential degree of influence on the schema inference engine, the priority is sorted to obtain the sorted new knowledge to be integrated; according to the priority, the resolved new schema features and semantic correction rules and the sorted new knowledge to be integrated are merged with the data schema dictionary, and a semantic compatibility check is performed to obtain an updated data schema dictionary.

[0019] As a technical improvement, the new schema features and semantic correction rules determined in the review result are pre-analyzed to determine whether there is a semantic conflict with the data schema dictionary, including: determining the business domain identifier associated with the new schema features and semantic correction rules, extracting relevant schema knowledge related to the business domain identifier and cross-domain schema knowledge associated with the schema knowledge from the data schema dictionary; compare the semantic representation of the new schema features and semantic correction rules with the semantic representation of the relevant schema knowledge and the cross-domain schema knowledge to determine whether a semantic conflict occurs.

[0020] In an embodiment, comparing the semantic representation of the new schema features and semantic correction rules with the semantic representation of the relevant schema knowledge and the cross-domain schema knowledge to determine whether a semantic conflict occurs includes: determining whether the new schema features or the semantic correction rules have any of the following: data type mismatch, enumeration value range conflict, and key field semantic ambiguity with the relevant schema knowledge and the cross-domain schema knowledge, if any, then it is determined that there is a semantic conflict.

[0021] Through this technical solution, the existence of semantic conflict can be determined through accurate comparison of specific data types, enumeration value ranges and key field semantics, providing clear judgment basis for subsequent conflict resolution, and improving the accuracy and operability of conflict identification.

[0022] To perfect the scheme, the determined semantic conflict is resolved according to a preset priority rule to obtain a new mode feature after resolution and a semantic correction rule, including: determining the type of the semantic conflict, calculating the resolution priority of the semantic conflict according to the type of the semantic conflict, the business domain associated with the semantic conflict, the data volume involved in the semantic conflict, and the expected compliance risk impact degree of the semantic conflict; according to the resolution priority, a resolution strategy is selected to resolve the semantic conflict to obtain a new mode feature after resolution and a semantic correction rule; the processing effect feedback of the resolution strategy on the business data stream is monitored, and the calculation rule of the resolution priority is optimized according to the processing effect feedback.

[0023] As a further improvement, the monitoring of the processing effect feedback of the resolution strategy on the business data stream includes: deploying a probe at a key node of the business data stream; wherein the probe collects monitoring data in a non-blocking manner; the monitoring data includes data parsing success rate, data conversion consistency, field value distribution deviation, and abnormal data proportion; the monitoring data is preprocessed and aggregated locally to obtain aggregated index data, and the aggregated index data is sent to a monitoring center; the monitoring center performs real-time stream analysis on the aggregated index data to obtain an analysis result; according to the analysis result and a preset baseline, if the analysis result exceeds the preset baseline, an alarm is triggered, and a snapshot of the abnormal data stream is recorded to evaluate the actual processing effect of the resolution strategy.

[0024] In a second aspect, the application also discloses a big data compliance risk control system, which comprises: an acquisition and encapsulation module, which is used for acquiring original business data, adding metadata information to the original business data, encapsulating the original business data, and sending the original business data; the metadata information comprises data source identification, data type, and data version number; a matching module, which is used for performing preliminary matching in a data mode dictionary according to the received metadata information, and determining a data mode matched with the metadata information; a probability inference module, which is used for inferring data structure and field type in the metadata information under the data mode according to a probability model, and outputting an inference confidence; and a risk shunting module, which is used for determining that the original business data has a risk and shunting the original business data to an isolation area in response to the inference confidence being lower than a preset threshold, reviewing the original business data in the isolation area, and updating the data mode dictionary according to a review result. Advantages

[0025] The application provides a big data compliance risk control method, which realizes standardization and traceability of data sources by obtaining original business data and adding metadata information for encapsulation. Subsequently, the metadata information is used for preliminary matching in the data mode dictionary, which can quickly identify the potential mode of the data. On this basis, the data structure and field type are inferred by a probability model, and the inference confidence is output, effectively solving the parsing error and label noise problems caused by non-standard data format and lagging parser update in the prior art. When the inference confidence is lower than the preset threshold, the system can timely divert the original business data with risks to the quarantine area for review, and dynamically update the data mode dictionary according to the review result, thereby forming a closed-loop risk identification, review and optimization mechanism. The method overcomes the limitations of traditional risk identification methods in processing massive and diversified data streams, effectively avoids the error marking data caused by the logical defects of the adaptive script, and prevents the negative influence of label noise on the training of the machine learning model, significantly improves the identification sensitivity of the model to emerging and hidden compliance risk patterns, breaks the negative feedback cycle, and ensures the effectiveness of enterprise data compliance and risk control. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0027] Figure 1 is a flowchart of the big data compliance risk control method disclosed by the embodiments of the application;

[0028] Figure 2 is a step flowchart of updating the data mode dictionary according to the review result disclosed by the embodiments of the application;

[0029] Figure 3 is a structural schematic diagram of the big data compliance risk control system disclosed by the embodiments of the application. DETAILED DESCRIPTION

[0030] The technical solutions in the application will be described in detail below with reference to the drawings in the application. Obviously, the described embodiments are only some of the embodiments of the application, not all the embodiments.

[0031] The conventional existing enterprise big data compliance risk control method faces complex and changeable data environment, and has problems such as frequent customization adjustment of data format parser, logical defects of temporary adaptation script leading to error marking data, and recognition sensitivity of machine learning model due to label noise. These problems together lead to serious challenges in compliance risk control for enterprises, and an efficient and intelligent solution is urgently needed.

[0032] To this end, the present application provides a big data compliance risk control method, as shown in Figure 1 The method comprises the following steps:

[0033] S101, acquiring original business data, adding metadata information to the original business data, encapsulating and sending; the metadata information includes data source identification, data type, data version number;

[0034] S102, performing preliminary matching in the data mode dictionary according to the received metadata information, and determining the data mode matched with the metadata information;

[0035] S103, inferring the data structure and field type in the metadata information under the data mode according to the probability model, and outputting the inference confidence;

[0036] S104, in response to the inference confidence being lower than a preset threshold, determining that the original business data has risks and being shunted to an isolation area, and reviewing the original business data in the isolation area, and updating the data mode dictionary according to the review result.

[0037] The "original business data" refers to various data generated by enterprises in the process of daily operation, which has not been processed or only undergone preliminary collection, such as transaction records, user behavior logs, sensor data, etc. These data usually exist in multiple formats and may contain sensitive information or compliance risk points. The "metadata information" refers to data describing the original business data, which does not contain business data itself, but provides information about the structure, source, type, version, etc. of the business data. In this application, the metadata information includes data source identification, data type and data version number, which are the basis for data pattern matching and risk inference. The "data pattern dictionary" is a knowledge base that stores known data patterns, which contains the structure definition, field type, nested relationship and historical evolution information of various business data. The dictionary is the core basis for data pattern matching, and its accuracy and completeness directly affect the efficiency and accuracy of risk identification. The "probability model" refers to a statistical model used to infer data structure and field type, such as Bayesian inference model or conditional random field model. These models can make probabilistic predictions on unknown or uncertain data structures based on existing data patterns and metadata information, and output inference confidence. The "inference confidence" is a measure of the reliability of the inference results of data structure and field type by the probability model. The higher the confidence, the more reliable the inference result; the lower the confidence, the greater the uncertainty of the inference result, which may have compliance risks. The "isolation area" is an area specially used to store original business data with potential risks. The data in this area is isolated from the normal business data stream to prevent the spread of potential risks and provide a safe environment for subsequent manual review and pattern update.

[0038] The method provided in the present application can be implemented in a distributed big data processing platform, such as a cluster environment based on Hadoop, Spark, etc. Each link of data collection, processing, storage and analysis can be deployed on the corresponding computing node, and data interaction and function calling can be realized through message queue, API interface, etc. The present application provides a big data compliance risk control method, the core of which is to realize risk identification, diversion and pattern adaptive update of original business data through a series of steps.

[0039] First, raw business data needs to be acquired and metadata information added to the data, which is then packaged and sent. Metadata information is data that describes data, including data source identification, data type, and data version number. For example, raw business data can be collected from various business systems (such as CRM systems, ERP systems, log systems, etc.) within an enterprise in real time or in batches. During data collection, metadata information can be automatically generated or extracted for each piece of data or each batch of data. The data source identification can be a unique string indicating the specific source system or module of the data; the data type can be a predefined enumeration value, such as "transaction data", "user behavior data", "system log", etc.; and the data version number can be used to track the evolution of data schema or format. These metadata information are then packaged with the raw business data into a data package and sent to subsequent processing modules through message queues or API interfaces.

[0040] Second, according to the received metadata information, a preliminary matching is performed in the data schema dictionary to determine the data schema that matches the metadata information. The data schema dictionary is a pre-constructed knowledge base that stores the structure definition, field type, and nested relationship information of various known business data. When a data package with metadata information is received, the system uses the data source identification, data type, and data version number in the metadata information as query keys to search in the data schema dictionary. For example, if the metadata information indicates that the data comes from the "order system", the type is "transaction data", and the version number is "1.0", the system will try to find a schema definition in the dictionary that matches these characteristics. After a successful match, the corresponding data schema is extracted, which describes the expected data structure and field specifications.

[0041] Third, according to the probability model, the data structure and field type within the metadata information under the data schema are inferred, and the inference confidence is output. After the preliminary matching of the data schema, the system uses a probability model to perform a deeper level of structure and type inference on the raw business data. For example, a Bayesian inference model or a conditional random field model can be used. These models combine the prior knowledge defined in the data schema (such as field name, expected type range) and the actual content of the raw business data to probabilistically analyze the data structure (such as separator, field order, nesting level) and field type (such as integer, string, date, boolean) of each field. The inference process outputs a confidence level, which represents the degree of confidence of the model in the inference result. For example, if the model is highly confident that a field is of date type, the confidence level will be high; conversely, if there are multiple possibilities, the confidence level will be relatively low.

[0042] Finally, in response to the inference confidence being lower than a preset threshold, it is determined that the original business data is at risk and is diverted to an isolation area, and the original business data in the isolation area is reviewed, and the data pattern dictionary is updated according to the review result. The preset threshold is a configurable parameter used to define the degree of data risk. When the inference confidence output by the probability model is lower than the threshold, it indicates that the actual structure or type of the original business data deviates greatly from the definition in the data pattern dictionary, and there may be compliance risks or data quality problems. At this time, the system will mark the original business data as "at risk" and divert it to a separate isolation area. The data in the isolation area will not enter the normal business processing flow, but will wait for manual or semi-automatic review. Reviewers will analyze the data in the isolation area in detail to find out the reasons for the low confidence, such as new data format, business rule changes, data entry errors, or malicious tampering. According to the review result, the data pattern dictionary can be updated to adapt to new data patterns or correct incorrect pattern definitions, thereby improving the accuracy of subsequent data processing.

[0043] The big data compliance risk control method provided in the present application has the overall working principle of constructing a self-adaptive and closed-loop risk identification and pattern learning mechanism. First, by adding metadata information to the original business data and encapsulating it, a standardized input is provided for subsequent data pattern matching and risk inference. The data source identifier, data type and data version number in the metadata information enable the system to quickly locate the possible data pattern range.

[0044] Subsequently, the system uses this metadata information to perform preliminary matching in the data pattern dictionary, thereby obtaining the closest known data pattern to the current data. This step effectively reduces the search space for subsequent probability inference, improving efficiency. After obtaining the preliminary matching data pattern, a probability model is used to perform fine-grained structure and field type inference on the original business data. The probability model can handle uncertainty and variability in the data and output an inference confidence, quantifying the reliability of the inference result.

[0045] When the inference confidence is lower than the preset threshold, it indicates that the original business data may have abnormal conditions that do not conform to the known patterns, thus being determined to have potential risks. At this time, these risk data will not directly enter the normal business flow, but will be diverted to an isolation area. The isolation area is designed to prevent the spread of potential risks and provide a safe environment for manual review. In the isolation area, professionals conduct in-depth review of the risk data, analyze the reasons for the anomaly, and identify the need for new data patterns or correction of existing patterns.

[0046] Finally, the data pattern dictionary is updated according to the review results. This updating mechanism enables the system to learn from new data patterns and continuously improve its understanding and recognition of business data. In this way, the method of the present application can effectively cope with the challenges brought about by business innovation and data evolution, avoiding the frequent adjustment of parsers and temporary script defects caused by changes in data format in traditional methods. The entire process forms a closed loop of continuous learning and improvement, enabling the big data compliance risk control system to continuously adapt and optimize as the business develops, thereby significantly improving the accuracy and efficiency of risk identification and reducing potential compliance risks.

[0047] Compared with the prior art, the big data compliance risk control method provided by the present application embodies significant innovation and technical progress. Traditional methods often rely on predefined, static data format specifications. When business data formats change slightly or new data types appear, manual intervention is required to modify the parser or write temporary adaptation scripts. This approach is not only inefficient, but temporary scripts are prone to logical defects, leading to potential compliance risks being incorrectly labeled as normal data, thereby polluting the training data of the machine learning model, forming a negative feedback loop and causing the risk identification ability to continue to decline.

[0048] The core innovation of the present application lies in the introduction of metadata information and a probability model, and the construction of a self-adaptive data pattern dictionary updating mechanism. By adding metadata information such as data source identifier, data type and data version number to the original business data, the system can intelligently perform preliminary pattern matching, rather than simply relying on hard-coded format definitions. The application of the probability model enables the system to infer data structures and field types in a probabilistic manner and output confidence levels, thereby identifying data that deviates from known patterns but is not completely incorrect, avoiding the "black or white" judgment mode in traditional methods. More importantly, the present application separates low-confidence data to a quarantine area for review and dynamically updates the data pattern dictionary based on the review results, forming a continuous learning and evolution mechanism. This mechanism enables the system to actively adapt to new data patterns and business changes, and timely correct and improve its risk identification ability, effectively avoiding the accumulation of label noise and model parameter drift. For example, when a new business data format appears, traditional methods may take weeks or even months to complete parser updates and deployment; while the method of the present application can incorporate new patterns into the identification range in a shorter period of time through quarantine review and pattern dictionary updating, significantly improving response speed and accuracy. This adaptive ability is not possessed by the prior art, and it enables enterprises to maintain a leading position in big data compliance risk control, effectively coping with the rapidly changing business environment and increasingly stringent compliance requirements.

[0049] Specifically, the step of performing preliminary matching in the data schema dictionary according to the received metadata information to determine the data schema matching the metadata information comprises: taking the data source identifier, data type, and data version number in the metadata information as keys to match in the data schema dictionary, and extracting the historical field list, prior type, and nested relationship matching the metadata information to form the data schema of the current data. The data source identifier, data type, and data version number in the metadata information are used as a composite key. The data source identifier is used to uniquely identify the original source of the data, such as a specific business system, database, or external interface. The data type indicates the overall format or content category of the data, such as log data, transaction data, or user behavior data. The data version number is used to distinguish the evolution of the same data source and type at different time points or under different specifications. By combining the three as a key, accurate and unique lookup in the data schema dictionary can be ensured.

[0050] The data schema dictionary is a pre-constructed knowledge base that stores a large number of known and verified data schemas. Each data schema is associated with a specific data source identifier, data type, and data version number, and contains the historical field list, prior type of each field, and nested relationship between fields under this schema. When new metadata information is received, the system will use the composite key described above to quickly match in the data schema dictionary. After a successful match, the system will extract the historical field list, prior type of each field, and nested relationship between fields from the data schema dictionary that matches the current metadata information. The historical field list lists all the field names expected to appear under this specific data schema. The prior type provides the expected value type for these fields, such as string, integer, float, date, etc. The nested relationship describes the hierarchical structure between fields, such as in JSON or XML format, a field may contain multiple subfields. By extracting this information, a preliminary data schema for the current data can be formed, providing a basis for subsequent data structure and field type inference.

[0051] The scheme of the present application uses the data source identifier, data type, and data version number in the metadata information as unique identification keys to efficiently match in the data schema dictionary, thereby quickly and accurately retrieving the historical data schema corresponding to the current data stream. This composite key-based matching mechanism effectively utilizes the accumulated knowledge of historical data, avoiding the complexity of starting from scratch to identify the pattern of each new data stream. As a result, a structured and prior-knowledge-based benchmark can be provided for subsequent probabilistic model inference, significantly improving the efficiency and accuracy of pattern inference.

[0052] In some embodiments of the present application, the data structure and field type are inferred by performing a preliminary match in the data schema dictionary according to the received metadata information and determining the matched data schema, and then using a probability model. However, in practical applications, if the selection of the probability model is not specific enough or the inference process is not fine enough, it may lead to insufficient accuracy in inferring the structure and field type of complex or evolving data schema, thereby affecting the accuracy of compliance risk identification. To this end, the present application further proposes to specify the probability model and elaborates specific steps for inferring the data structure and field type, so as to improve the accuracy and reliability of the inference.

[0053] The probability model includes a Bayesian inference model or a conditional random field model; the data structure and field type in the metadata information under the data schema are inferred according to the probability model, and an inference confidence is output, including: analyzing and obtaining the structural characteristics of the original business data according to the fields under the data schema and the corresponding prior type; the structural characteristics include separators, field order, value type distribution, and nesting level; performing probability analysis according to the structural characteristics, calculating the posterior confidence of each field type and semantic, and generating a structure inference result with the posterior confidence. The Bayesian inference model is a probability reasoning method based on Bayes' theorem, which calculates the posterior probability by combining prior probability and observed data, and is suitable for processing uncertain information and performing pattern recognition. The conditional random field model is a discriminative probability model, which is commonly used for sequence labeling and structured prediction tasks, and can effectively capture the context dependency in data sequences, and has an advantage in inferring data schema with complex structure. In practical applications, one or a combination of the two models can be selected according to the characteristics of specific business data and inference requirements.

[0054] Specifically, the structural characteristics refer to the description of the internal organization form and content attribute of the original business data. The separator refers to the symbol used to distinguish between data fields, such as comma, tab, or pipe symbol, etc.; the field order refers to the order of appearance of each field in the data record; the value type distribution refers to the frequency and proportion of different data types (such as string, integer, float, date, etc.) in a field; and the nesting level refers to the depth and complexity of the data structure containing sub-structure or sub-field. The analysis of these structural characteristics helps to fully understand the internal mode of the original business data.

[0055] Further, the probability analysis refers to using a selected probability model (such as a Bayesian inference model or a conditional random field model) to quantitatively evaluate the type and semantics of each field in combination with the analyzed structural features. By calculating the posterior confidence of each field type and semantics, the probability that a certain field belongs to a specific type or has a specific semantics under a given data mode and structural features can be obtained. Thus, a structural inference result with the posterior confidence is generated, which not only indicates the inferred data structure and field type, but also provides a reliability measure of the inference.

[0056] The scheme of the present application can more accurately capture the inherent structure and field type of the original business data by explicitly specifying the probability model as a Bayesian inference model or a conditional random field model and refining the inference process. Specifically, by analyzing the separator, field order, value type distribution, and nested level, etc., structural features provide rich input information for the probability model. On this basis, the probability model can perform more detailed probability analysis to calculate the posterior confidence of each field type and semantics, thereby overcoming the problem of insufficient inference accuracy that may exist in traditional generalized probability models, especially when facing data mode evolution or complex data structure, more reliable inference results can be provided.

[0057] As a specific implementation, the following is described through a specific example. Assume that the system receives a batch of original business data, whose metadata information indicates that the data is from a certain log system, the data type is text log, and the version number is 1.0. The matching module preliminarily matches in the data mode dictionary according to these metadata information and determines a historical log data mode, which contains a field list, prior type, and nested relationship.

[0058] In the probability inference module, a conditional random field model can be used for inference. First, according to the fields and their prior types in the log data mode, the structural characteristics of the original business data are analyzed. For example, it is identified that the separator in the log entry is "|", the field order is "timestamp | user ID | operation type | result code", the value type distribution of the user ID field is mainly numbers and letter combinations, and there may be a nested JSON format "result code" field. Then, the conditional random field model uses these structural characteristics for probability analysis to calculate the posterior confidence of the type (such as date and time, string, enumeration value, JSON object) and semantics (such as event occurrence time, operation performer, specific operation, operation feedback) of each field (such as "timestamp", "user ID", "operation type", "result code"). For example, the model may infer that the "timestamp" field is of date and time type with a confidence of 0.98, and the "result code" field is of JSON object type with a confidence of 0.95. Finally, a structural inference result with the posterior confidence is generated, which clearly describes the accurate structure of the log data and the types and semantics of each field, providing high-confidence input for subsequent risk assessment.

[0059] The application further proposes to review the original business data in the isolated area, as shown in Figure 2 The steps of updating the data mode dictionary according to the review result include:

[0060] S201, according to the frequency and complexity of the mode evolution corresponding to the original business data in the isolated area, adjusting the allocation priority of the review resources;

[0061] S202, based on manual review of the original business data, structuring the review result, extracting key features and semantic correction rules of the mode evolution;

[0062] S203, comparing the structured review result with the data mode dictionary to determine the lagging part in the data mode dictionary;

[0063] S204, generating update information of the data mode dictionary according to the lagging part, and performing data mode dictionary update.

[0064] Specifically, in the process of reviewing the original business data of the isolated area and updating the data schema dictionary, first, according to the frequency and complexity of the schema evolution corresponding to the original business data of the isolated area, the allocation priority of the review resources is adjusted. Among them, the frequency of schema evolution can refer to the number of times a specific data schema changes within a period of time, while the complexity can refer to the number of fields involved in the schema change, data type change, nesting structure depth, etc. Through the analysis of these indicators, the limited review resources, such as manual review personnel or automated review tools, can be dynamically allocated to the original business data with frequent or complex schema evolution, to ensure that high-risk or high-change-rate data is reviewed in a timely and sufficient manner.

[0065] Further, manual review is performed on the original business data. Manual review can utilize human expertise and experience to identify subtle schema changes and potential compliance risks that automated tools may miss. The review results are structured to convert unstructured or semi-structured review opinions and findings into standardized, machine-readable formats. In this way, key features of schema evolution and semantic correction rules can be extracted. Key features refer to the core elements of data schema changes, such as new fields, field type changes, enumeration value range adjustments, etc.; semantic correction rules refer to how to adjust the data schema dictionary to accurately reflect the new data semantics and structure in response to these schema changes. Then, the structured review results are compared with the data schema dictionary to determine the lagging part in the data schema dictionary. The lagging part refers to the part of the data schema dictionary that fails to timely reflect the actual schema changes of the current original business data, such as missing newly appeared fields, inconsistent field types, or outdated semantic definitions, etc. Through comparison, the specific content that needs to be updated in the dictionary can be accurately identified. Finally, the update information of the data schema dictionary is generated according to the determined lagging part, and the data schema dictionary update is performed. The update information can include adding new schema entries, modifying existing schema definitions, deleting obsolete schemas, etc. Performing data schema dictionary update aims to integrate these update information into the data schema dictionary, so that the dictionary can accurately and timely reflect the latest data schema and compliance requirements.

[0066] The scheme of the present application realizes intelligent allocation of review resources by introducing consideration of mode evolution frequency and complexity, avoids waste of review resources, and improves review efficiency. It is due to the structured processing of review results and the extraction of key features and semantic correction rules that the experience and knowledge of manual review can be systematically captured and utilized, providing accurate basis for subsequent dictionary updating. By comparing the structured processed review results with the data mode dictionary, the lagging part in the dictionary can be accurately identified, ensuring the pertinence and effectiveness of the update. Finally, the update is generated and executed according to the lagging part, so that the data mode dictionary can keep pace with the evolution of actual business data, thereby effectively coping with the changing compliance risks in the big data environment.

[0067] In some preferred embodiments, the following is described by a specific example. Assuming that a new transaction type is added to the transaction data stream of a certain financial institution due to new business needs, and a new field "risk rating code" is included in the data structure thereof. Initially, the data mode dictionary does not contain this field. After the original business data containing this new field is acquired and added with metadata information, in the preliminary matching and probability inference stage, the inference confidence may be lower than the preset threshold due to the presence of the "risk rating code" field, so that these original business data are diverted to the isolation area.

[0068] In the isolation area, the system adjusts the allocation priority of review resources according to the frequency (for example, the frequency of occurrence of the new transaction type) and complexity (for example, the number and type of newly added fields) of the evolution of the transaction data mode, and preferentially arranges manual review personnel to review these data. The manual review personnel identify that "risk rating code" is a new key feature, and determine that its data type is string and has a specific enumeration value range. The review results are structured processed, and the key feature "risk rating code" and its semantic correction rules (for example, the field represents the risk level of the transaction, and the value range is A, B, C, D) are extracted.

[0069] Subsequently, the system compares the structured processed review results with the current data mode dictionary. The comparison result shows that the definition of the "risk rating code" field is missing in the data mode dictionary, which is determined as the lagging part of the data mode dictionary. Based on this lagging part, the system generates update information, including adding the definition of the "risk rating code" field, its data type and enumeration value range to the existing transaction data mode. Finally, the data mode dictionary update is executed, and the new mode feature is integrated into the data mode dictionary. Thus, subsequent transaction data containing the "risk rating code" field can be correctly identified and processed, thereby ensuring data compliance and avoiding risk misjudgment due to mode mismatch.

[0070] In some embodiments of the application described above, a scheme is proposed to review the original business data in the isolation area and update the data schema dictionary according to the review results. However, when structuring the review results and extracting the key features of schema evolution and semantic correction rules, if the potential conflicts between new and old schema knowledge cannot be effectively identified and resolved, or if the new knowledge cannot be reasonably prioritized and integrated, it may lead to a decline in the quality of data schema dictionary updates, or even introduce new compliance risks. To this end, the application further proposes an optimization scheme for structuring the review results, extracting the key features of schema evolution and semantic correction rules, to ensure that the update process of the data schema dictionary is more robust and accurate.

[0071] Specifically, the above-mentioned structuring of the review results, extracting the key features of schema evolution and semantic correction rules, includes the following steps:

[0072] Pre-analyzing the new schema features and semantic correction rules determined in the review results to determine whether there is a semantic conflict with the data schema dictionary. The purpose of pre-analysis is to identify potential inconsistencies or contradictions before integrating new knowledge into the data schema dictionary, so as to avoid introducing errors or causing confusion in the existing schema inference logic.

[0073] According to the preset priority rules, the semantic conflicts determined are resolved to obtain the resolved new schema features and semantic correction rules. The resolution of semantic conflicts aims to solve the contradictions between new and old knowledge and ensure the internal consistency of the data schema dictionary. The preset priority rules can be set based on business importance, data volume, risk impact, etc. to guide the selection of conflict resolution strategies.

[0074] Marking the key features or semantic rules determined in the review results and missing in the data schema dictionary, and prioritizing according to business importance, data frequency and potential impact on schema inference engine to obtain the sorted new knowledge to be integrated. This step aims to identify and prioritize the new knowledge that is crucial to data compliance risk control but is not yet covered in the data schema dictionary, ensuring the timely integration of key information.

[0075] According to the priority, the resolved new schema features and semantic correction rules and the sorted new knowledge to be integrated are integrated with the data schema dictionary, and a semantic compatibility check is performed to obtain the updated data schema dictionary. The integration process integrates the processed new knowledge into the existing data schema dictionary, and the semantic compatibility check serves as the final quality control link to ensure that the updated data schema dictionary maintains logical consistency and effectiveness as a whole.

[0076] The scheme of the present application systematically solves the semantic conflicts and knowledge integration problems that may occur when updating the data schema dictionary by introducing steps such as pre-analysis, conflict resolution, priority sorting, and fusion and compatibility checking. Specifically, the pre-analysis step can actively identify potential conflicts between new schema features and semantic correction rules and the existing data schema dictionary, thereby avoiding the direct introduction of inconsistent knowledge into the system. Subsequently, the identified semantic conflicts are resolved by pre-set priority rules, ensuring that when faced with conflicts, a reasonable decision can be made according to the established strategy, thereby obtaining consistent new schema features and semantic correction rules. At the same time, for the missing key features or semantic rules in the data schema dictionary, the scheme marks them and sorts them based on business importance, data frequency, and the potential degree of influence on the schema inference engine, ensuring the priority integration of core and high-value new knowledge and avoiding the omission of important information. Finally, by fusing the resolved new schema features and semantic correction rules and the sorted new knowledge to be integrated with the data schema dictionary, and performing strict semantic compatibility checking, it is ensured that the updated data schema dictionary not only contains the latest schema knowledge, but also maintains a high degree of consistency and integrity in semantics, thereby significantly improving the accuracy and reliability of the data schema dictionary.

[0077] In some preferred embodiments, the following is described by a specific example. Assume that in the data schema dictionary of a certain financial institution, the "customer age" field is defined as an integer type with a value range of 18-100. Now, through the review of the original business data in the quarantine area, it is found that in a batch of new business data, the "customer age" field is defined as a string type and contains non-numeric values such as "unknown" or "N / A", and a completely new field "customer credit rating" is found, which does not exist in the existing data schema dictionary.

[0078] First, when pre-analyzing the new schema features (e.g., "customer age" is a string type containing "unknown" value) and semantic correction rules (e.g., mapping "unknown" to a specific default value) identified in the review results, the system identifies the data type conflict (integer vs. string) and value range conflict (18-100 vs. containing "unknown") of the "customer age" field. At the same time, the system identifies that "customer credit rating" is a completely new key feature missing in the data schema dictionary. Second, according to the preset priority rules, for example, if the compliance risk impact of the "customer age" field is high, the priority of resolving its semantic conflict will be raised. The system may choose a resolution strategy, such as modifying the type of the "customer age" field to a mixed type supporting both string and integer, and defining the compliance processing rules for "unknown", thereby obtaining the new schema features and semantic correction rules after resolution. Third, for the missing key feature "customer credit rating", the system prioritizes it according to its business importance (e.g., crucial for risk assessment), data frequency (e.g., prevalent in new business data), and potential impact on the schema inference engine. Assuming it has a high priority, it is marked as high-priority new knowledge to be integrated. Finally, the system integrates the new schema features and semantic correction rules of the resolved "customer age" field, and the sorted "customer credit rating" new knowledge to be integrated, with the data schema dictionary according to the above priority. During the integration process, the system performs semantic compatibility checks, such as ensuring that the new "customer age" definition does not create new conflicts with other modes that rely on it being a pure integer, and ensuring that the introduction of the "customer credit rating" field does not disrupt the integrity of the existing data schema. Through these steps, an updated data schema dictionary is finally obtained, which can accurately identify and process new business data containing "unknown" age values, and can identify and utilize the new key business information "customer credit rating", thereby improving the comprehensiveness and accuracy of big data compliance risk control.

[0079] In some embodiments of the application described above, when the review results are structured and the key features of schema evolution and semantic correction rules are extracted, pre-analysis of these new schema features and semantic correction rules is required to determine whether they have semantic conflicts with the data schema dictionary. Specifically, the pre-analysis of the new schema features and semantic correction rules identified in the review results to determine whether they have semantic conflicts with the data schema dictionary comprises:

[0080] determining the business domain identifier associated with the new schema features and semantic correction rules, and extracting relevant mode knowledge related to the business domain identifier and cross-domain mode knowledge associated with the relevant mode knowledge from the data schema dictionary according to the business domain identifier;

[0081] The semantic representation of the new mode feature and the semantic correction rule is compared with the semantic representation of the relevant mode knowledge and the cross-domain mode knowledge to determine whether a semantic conflict occurs.

[0082] The new mode feature and the semantic correction rule refer to new structures, new fields, new value domains or correction rules of existing modes that are identified from the review results after reviewing the original business data of the isolated area, reflecting the evolution of data mode. These new mode features and semantic correction rules need to be integrated into the data mode dictionary to maintain the real-time and accuracy of the dictionary. The business domain identifier can be understood as a specific business category or classification label related to the original business data or its mode features, such as "financial transaction", "user behavior", "logistics information", etc. By determining the business domain identifier associated with the new mode feature and the semantic correction rule, the scope of matching and comparison in the data mode dictionary can be effectively narrowed down, and the efficiency and accuracy of semantic conflict detection can be improved.

[0083] The data mode dictionary is a knowledge base that stores known data modes, field definitions, semantic rules and their associated relationships. It provides a benchmark and reference for big data compliance risk control. The relevant mode knowledge refers to the mode definition, field type, semantic constraint and other information directly related to the business domain identifier associated with the current new mode feature and the semantic correction rule in the data mode dictionary. The cross-domain mode knowledge refers to the mode knowledge related to the business domain identifier, but there is an association or sharing between different business domains, for example, some general data types or coding specifications may be common in multiple business domains. The purpose of extracting these knowledge is to comprehensively evaluate the semantic impact of the new mode feature and the semantic correction rule. The semantic representation refers to the conversion of new mode features, semantic correction rules and existing knowledge in the data mode dictionary into machine-understandable and comparable semantic forms through natural language processing, knowledge graph, vector embedding and other technologies. For example, field names, descriptions, enumerated values and other information can be encoded into vectors for similarity calculation. Comparing the semantic representation of the new mode feature and the semantic correction rule with the semantic representation of the relevant mode knowledge and the cross-domain mode knowledge aims to judge whether the introduction of new knowledge will lead to semantic inconsistency or ambiguity within the data mode dictionary by calculating the semantic similarity, consistency or conflict degree between them.

[0084] The scheme of the present application can accurately extract the most relevant schema knowledge and cross-domain schema knowledge from a large data schema dictionary by first determining the business domain identifier associated with the new schema feature and the semantic correction rule. This focused extraction method avoids blind comparison of the entire dictionary, significantly improving detection efficiency. Subsequently, by comparing the semantic representation of the new schema feature and the semantic correction rule with the semantic representation of the extracted relevant schema knowledge and cross-domain schema knowledge, it can be determined whether there is inconsistency, contradiction or ambiguity between the new and old knowledge at the semantic level. For example, if the new schema feature defines a field with the same name but completely different meaning as in the existing schema, or the semantic correction rule conflicts with the existing rule, potential semantic conflicts can be identified in time. Thus, it ensures that the newly introduced schema knowledge is fully semantically verified before updating the data schema dictionary, thereby maintaining the internal consistency and accuracy of the data schema dictionary.

[0085] In some embodiments of the present application described above, when the newly introduced schema feature and semantic correction rule determined in the review result are pre-analyzed, it is necessary to determine whether there is a semantic conflict with the data schema dictionary. Specifically, in order to accurately determine the occurrence of semantic conflict, the present application proposes a more detailed semantic conflict determination method.

[0086] The method comprises: comparing the semantic representation of the new schema feature and the semantic correction rule with the semantic representation of the relevant schema knowledge and the cross-domain schema knowledge to determine whether a semantic conflict occurs, comprising: determining whether the new schema feature or the semantic correction rule and the relevant schema knowledge, the cross-domain schema knowledge exist any one of data type mismatch, enumeration value range conflict, key field semantic ambiguity, if there is, it is determined that there is a semantic conflict.

[0087] Specifically, the data type mismatch refers to that the data type (e.g., integer, string, date, etc.) expected by a field in the new mode feature or the semantic correction rule is incompatible with the data type defined for the corresponding field in the data schema dictionary. For example, the new rule may define a certain "quantity" field as a string type, while the field is defined as an integer type in the existing dictionary. The enumeration value range conflict refers to that when a field is defined to only accept a preset set of enumeration values, the new mode feature or the semantic correction rule introduces a value that is beyond or contradictory to the existing enumeration value range. For example, the existing "status" field allows "active", "inactive", and "pending", while the new rule introduces "completed" which is not defined as an allowed value. The key field semantic ambiguity refers to that even if the data type of the key field is compatible, the meaning or interpretation of the key field in the new mode feature or the semantic correction rule is different from the meaning established in the data schema dictionary, thereby causing ambiguity. For example, a "code" field refers to a product code in one context and refers to a region code in another context.

[0088] The scheme of the present application defines the specific types of semantic conflicts, so that the detection process of semantic conflicts is more accurate and automated. When the semantic representation of the new mode feature or the semantic correction rule is compared with the related mode knowledge in the data schema dictionary and the cross-domain mode knowledge, it is no longer a general judgment of "whether there is a conflict", but a specific check of whether there are three types of core conflicts: data type mismatch, enumeration value range conflict, or key field semantic ambiguity. This detailed checking mechanism can effectively identify potential data parsing errors, inconsistent data conversion, and business logic misunderstanding, thereby providing accurate conflict positioning information for subsequent conflict resolution and data schema dictionary update.

[0089] The present application further proposes that the above-mentioned semantic conflicts determined according to the preset priority rules are resolved to obtain the new mode feature and the semantic correction rule after resolution, comprising:

[0090] determining the type of the semantic conflict, calculating the resolution priority of the semantic conflict according to the type of the semantic conflict, the business domain associated with the semantic conflict, the amount of data involved in the semantic conflict, and the expected compliance risk impact degree of the semantic conflict;

[0091] According to the resolution priority, a resolution strategy is selected to resolve the semantic conflict to obtain the new mode feature and the semantic correction rule after resolution;

[0092] monitoring the processing effect feedback of the resolution strategy on the business data flow, and optimizing the calculation rule of the resolution priority according to the processing effect feedback.

[0093] Specifically, the types of semantic conflicts can include, but are not limited to, any of data type mismatch, enumeration value range conflict, and ambiguity of key field semantics, which have been determined in the above embodiments. When calculating the resolution priority of a semantic conflict, multiple dimensions are considered comprehensively. For example, the business domain associated with the semantic conflict can indicate the importance of the conflict to a specific business process or data asset; the amount of data involved in the semantic conflict can reflect the breadth of the impact of the conflict; and the expected compliance risk impact of the semantic conflict directly quantifies the potential risks that the conflict can bring, such as data leakage, data misuse, or violation of regulations, etc. By weighting these factors, a quantitative resolution priority can be obtained.

[0094] The selection of the resolution strategy is based on the calculated resolution priority. For example, for conflicts with high priority, automatic correction or immediate triggering of manual intervention strategies can be preferred; for conflicts with medium priority, a pre-warning and waiting for manual confirmation strategy can be used; and for conflicts with low priority, only recording or batch processing under certain conditions can be performed. The goal of resolution processing is to generate new schema features and semantic correction rules that are compatible with existing data schema dictionaries and meet business needs.

[0095] In practical applications, in order to ensure the effectiveness and adaptability of the resolution strategy, the processing effect of the resolution strategy on the business data flow needs to be continuously monitored. The monitoring data can include data parsing success rate, data conversion consistency, field value distribution deviation, and abnormal data proportion, etc. According to these processing effect feedbacks, the calculation rules of the resolution priority can be optimized, such as adjusting the weights of various influencing factors or introducing new consideration factors, so that the priority calculation is more accurate and the resolution processing is more efficient.

[0096] The scheme of the present application can finely evaluate the identified semantic conflicts according to multiple dimensions such as the type of conflict, business impact, data volume, and compliance risk, thereby ensuring that limited review resources are preferentially allocated to conflicts that have the greatest impact on business or the highest compliance risk. In addition, by selecting appropriate resolution strategies according to the resolution priority, the "one-size-fits-all" processing method can be avoided, so that conflicts of different natures can be solved most appropriately. Furthermore, by continuously monitoring the processing effect feedback of the resolution strategy on the business data flow and optimizing the calculation rules of the resolution priority accordingly, the scheme of the present application can realize self-learning and adaptation, so that the big data compliance risk control system can continuously improve its conflict resolution capability as the business data schema evolves, thereby significantly improving the robustness and accuracy of the system in dealing with complex data environments.

[0097] In some preferred embodiments, the following is illustrated by a specific example. Assume in a big data compliance risk control scenario of a financial institution, the system continuously receives raw business data from different business systems (e.g. transaction system, customer management system).

[0098] When the system detects semantic conflicts in the data schema dictionary, for example: 1, Conflict A: the data type of the "transaction amount" field in the transaction system changes from integer to float, and part of the historical data is still in integer format. 2, Conflict B: the coding rules of the "customer ID" field in the customer management system change, resulting in semantic ambiguity with another "account ID" field in some cases. 3, Conflict C: the newly introduced regulatory requirements stipulate that all transaction data must contain the "anti-money laundering risk level" field, but the existing data schema is missing this field.

[0099] At this time, the scheme of the present application will prioritize the resolution of these semantic conflicts: for Conflict A, its type is data type mismatch, the associated business domain is transaction, the involved data volume may be large, but the expected compliance risk impact is relatively low (usually only affects accuracy, does not directly cause major violations). Therefore, its resolution priority may be calculated as medium to low. The system may choose an automatic conversion or warning resolution strategy when conversion fails.

[0100] For Conflict B, its type is semantic ambiguity of key field, the associated business domain is customer management and transaction, the involved data volume may be medium, but the expected compliance risk impact is high (may cause customer identity recognition error, trigger fraud or compliance loopholes). Therefore, its resolution priority may be calculated as high. The system may choose to trigger manual review immediately and require the business department to provide explicit semantic correction rules for resolution strategy.

[0101] For Conflict C, its type is missing key field, the associated business domain is compliance supervision, the involved data volume may be all transaction data, and the expected compliance risk impact is extremely high (directly violates regulatory requirements). Therefore, its resolution priority will be calculated as the highest. The system may choose to immediately block related data flow and force the business system to supplement the field or provide alternative solutions for resolution strategy.

[0102] After the execution of the resolution strategy, the system continuously monitors the processing effect feedback of the business data stream. For example, for the automatic conversion strategy of conflict A, the system monitors the data parsing success rate and field value distribution deviation. If it is found that a large number of data parsing fails or the distribution is abnormal after conversion, it indicates that the strategy effect is not good, and the system will optimize the calculation rules of resolution priority according to the feedback, such as increasing the priority of data type mismatch conflict or adjusting the threshold of automatic conversion. For the manual review strategy of conflict B, the system monitors the structured processing of review results and the integration effect of new mode features. If it is found that the manual review efficiency is low or the correction rules are not clear, the priority calculation rules will be optimized, such as increasing the consideration of business department response speed. Through this continuous monitoring and optimization, the big data compliance risk control system can continuously improve its identification, priority sorting and resolution ability of complex semantic conflicts, and ensure the accuracy and compliance of the data mode dictionary.

[0103] In some embodiments of the present application, the resolution priority of the semantic conflict is calculated according to the type of semantic conflict, the associated business field, the involved data volume and the expected compliance risk impact degree, and the resolution strategy is selected for processing to obtain the new mode features and semantic correction rules after resolution. In order to ensure the effectiveness of the selected resolution strategy and continuously optimize it, it is necessary to monitor the processing effect of the resolution strategy in the actual business data stream. Specifically, the above-mentioned monitoring the processing effect feedback of the resolution strategy on the business data stream can include the following steps:

[0104] Deploying probes at key nodes of the business data stream; wherein the probes collect monitoring data in a non-blocking manner; the monitoring data includes data parsing success rate, data conversion consistency, field value distribution deviation and abnormal data proportion;

[0105] Preprocessing and aggregating the monitoring data locally to obtain aggregated index data, and sending the aggregated index data to the monitoring center;

[0106] The monitoring center performs real-time stream analysis on the aggregated index data to obtain analysis results; according to the analysis results and the preset baseline, if it exceeds the preset baseline, an alarm is triggered, and a snapshot of the abnormal data stream is recorded to evaluate the actual processing effect of the resolution strategy.

[0107] Specifically, the key nodes can refer to important processing links in the whole link of business data from source generation, cleaning, conversion, storage to final application, such as data access layer, data processing layer, data storage layer or data consumption layer. Probes are deployed at these key nodes, aiming to comprehensively cover all stages of data flow and ensure the integrity of monitoring. The probes collect monitoring data in a non-blocking manner, meaning that the data collection process will not cause delay or interruption to normal business data flow processing, thereby ensuring stable operation of the business system. The monitoring data, such as data parsing success rate, is used to measure whether the data can be correctly identified and structured; data conversion consistency is used to evaluate whether the data maintains semantic and content accuracy when converted between different formats or structures; field value distribution deviation is used to detect whether the value of a specific field deviates from the expected range or pattern, which may indicate data anomalies; and the proportion of abnormal data directly reflects the amount of data in the data flow that does not conform to the preset rules or patterns.

[0108] Further, the collected monitoring data will be preprocessed and aggregated locally. Local preprocessing can include data cleaning, format unification, deduplication and other operations to improve data quality. Aggregation is to summarize scattered monitoring data according to time window, business type or key node dimensions to form more macro-insightful aggregated index data. For example, the data parsing success rate and average field value distribution deviation of a certain key node within a period of time can be calculated. These aggregated index data are then sent to the monitoring center for centralized analysis and management.

[0109] In the monitoring center, the aggregated index data are subjected to real-time stream analysis. Real-time stream analysis refers to the immediate processing and analysis of continuously flowing data, which can quickly discover anomalies or trend changes in the data stream. Through this analysis, analysis results about the actual operation effect of the resolution strategy can be obtained. The analysis results are compared with the preset baseline. The preset baseline is the normal operation index range set according to historical data, business requirements or compliance standards. If the analysis results exceed the preset baseline, for example, the data parsing success rate decreases significantly or the proportion of abnormal data rises sharply, an alarm will be triggered to notify relevant personnel for intervention in a timely manner. At the same time, the system will also record snapshots of abnormal data flow, i.e. detailed information of the relevant data flow is intercepted and saved at the time of the anomaly. These snapshots have important value for subsequent problem troubleshooting, root cause analysis and evaluation of the actual processing effect of the resolution strategy, which can help operation personnel to deeply understand the causes and impacts of the problem and provide data support for optimizing the resolution strategy.

[0110] The scheme of the present application realizes real-time and non-intrusive collection of key indicators such as data parsing success rate, data conversion consistency, field value distribution deviation, and abnormal data proportion by deploying non-blocking probes at key nodes of service data flow. Thus, the direct feedback of the data processing effect of the resolution strategy in the actual business scenario can be obtained. Through local preprocessing and aggregation of these monitoring data, the transmission and analysis load is effectively reduced, and the timeliness and accuracy of the data are ensured. Subsequently, the monitoring center performs real-time stream analysis on the aggregated index data and compares it with the preset baseline, so that any abnormal condition deviating from the normal range can be quickly identified and an alarm is triggered. At the same time, snapshots of abnormal data flow are recorded to provide detailed context information for subsequent in-depth analysis and problem positioning. It is precisely due to this end-to-end, real-time feedback mechanism that the actual processing effect of the resolution strategy can be objectively and accurately evaluated, and a solid data foundation is provided for subsequent strategy optimization.

[0111] In addition, the present application proposes a big data compliance risk control system, as shown in Figure 3 The system comprises:

[0112] An acquisition and packaging module 301 is configured to acquire original business data, add metadata information to the original business data, package and send the original business data; the metadata information comprises data source identifier, data type, and data version number;

[0113] A matching module 302 is configured to perform preliminary matching in a data mode dictionary according to the received metadata information, and determine a data mode matched with the metadata information;

[0114] A probability inference module 303 is configured to infer data structure and field type in the metadata information under the data mode according to a probability model, and output an inference confidence;

[0115] A risk shunting module 304 is configured to determine that the original business data has a risk and shunt the original business data to an isolation area in response to the inference confidence being lower than a preset threshold, and review the original business data in the isolation area, and update the data mode dictionary according to a review result.

[0116] The above only describes embodiments of the present application and is not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A big data compliance risk control method, characterized in that, The method comprises: Obtaining original business data, adding metadata information to the original business data, encapsulating and sending; The metadata information includes data source identification, data type, data version number; According to the received metadata information, preliminary matching is performed in the data mode dictionary, and the data mode matched with the metadata information is determined; According to the probability model, the data structure and field type in the metadata information under the data mode are inferred, and the inference confidence is output; In response to the inference confidence being lower than a preset threshold, it is determined that the original business data is at risk and is diverted to an isolation area, and the original business data in the isolation area is reviewed, and the data mode dictionary is updated according to the review result; The review of the original business data in the isolation area and the updating of the data mode dictionary according to the review result comprises: According to the frequency and complexity of the mode evolution corresponding to the original business data in the isolation area, the allocation priority of the review resources is adjusted; Based on manual review of the original business data, the review result is structured, the key features of the mode evolution and the semantic correction rules are extracted; The structured review result is compared with the data mode dictionary to determine the lag part in the data mode dictionary; According to the lag part, the update information of the data mode dictionary is generated, and the data mode dictionary is updated.

2. The big data compliance risk control method of claim 1, wherein, The preliminary matching of the received metadata information in the data mode dictionary to determine the data mode matched with the metadata information comprises: Taking the data source identification, data type and data version number in the metadata information as the key, matching in the data mode dictionary, extracting the historical field list, prior type and nested relationship matched with the metadata information, and forming the data mode of the current data.

3. The big data compliance risk control method of claim 2, wherein, The probability model comprises a Bayesian inference model or a conditional random field model; According to the probability model, the data structure and field type in the metadata information under the data mode are inferred, and the inference confidence is output, which comprises: According to the field under the data mode and the corresponding prior type, the structural features of the original business data are analyzed and obtained; The structural features include separator, field order, value type distribution and nesting level. According to the structural features, the posterior confidence of each field type and semantic is calculated by probability analysis, and the structural inference result with the posterior confidence is generated.

4. The big data compliance risk control method of claim 1, wherein, The structured review result is structured, the key features of the mode evolution and the semantic correction rules are extracted, which comprises: The new mode features and semantic correction rules determined in the review result are pre-analyzed to determine whether there is a semantic conflict with the data mode dictionary; According to the preset priority rule, the determined semantic conflict is eliminated to obtain the new mode features and semantic correction rules after elimination. label the key features or semantic rules determined in the review result and missing in the data schema dictionary, and prioritize according to business importance, data frequency and potential impact on the schema inference engine to obtain sorted new knowledge to be integrated; According to the priority, the new mode feature and semantic correction rule after resolution and the sorted new knowledge to be integrated are fused with the data schema dictionary, and the semantic compatibility is checked to obtain the updated data schema dictionary.

5. The big data compliance risk control method of claim 4, wherein, The pre-analysis of the new mode feature and semantic correction rule determined in the review result includes: determine the business domain identifier associated with the new mode feature and semantic correction rule, and extract relevant mode knowledge related to the business domain identifier and cross-domain mode knowledge associated with the relevant mode knowledge from the data schema dictionary; Compare the semantic representation of the new mode feature and semantic correction rule with the semantic representation of the relevant mode knowledge and the cross-domain mode knowledge to determine whether a semantic conflict occurs.

6. The big data compliance risk control method of claim 5, wherein, The comparison of the semantic representation of the new mode feature and semantic correction rule with the semantic representation of the relevant mode knowledge and the cross-domain mode knowledge to determine whether a semantic conflict occurs includes: determine whether the new mode feature or the semantic correction rule has any of the following: data type mismatch, enumeration value range conflict, and key field semantic ambiguity with the relevant mode knowledge and the cross-domain mode knowledge, if any, then determine that there is a semantic conflict.

7. The big data compliance risk control method of claim 4, wherein, The resolution processing of the determined semantic conflict according to the preset priority rule to obtain the new mode feature and semantic correction rule after resolution includes: determine the type of the semantic conflict, according to the type of the semantic conflict, the business domain associated with the semantic conflict, the amount of data involved in the semantic conflict, and the expected compliance risk impact degree of the semantic conflict, calculate the resolution priority of the semantic conflict; According to the resolution priority, select the resolution strategy to resolve the semantic conflict to obtain the new mode feature and semantic correction rule after resolution; monitor the processing effect feedback of the resolution strategy on the business data stream, and optimize the calculation rule of the resolution priority according to the processing effect feedback.

8. The big data compliance risk control method of claim 7, wherein, The monitoring of the processing effect feedback of the resolution strategy on the business data stream includes: deploy a probe at a key node of the business data stream; wherein the probe collects monitoring data in a non-blocking manner; the The monitoring data includes data parsing success rate, data conversion consistency, field value distribution deviation and abnormal data proportion; The monitoring data is preprocessed and aggregated locally to obtain aggregated index data, and the aggregated index data is sent to a monitoring center; The monitoring center performs real-time stream analysis on the aggregated index data to obtain an analysis result; according to the analysis result and a preset baseline, if the analysis result exceeds the preset baseline, an alarm is triggered, and a snapshot of the abnormal data stream is recorded to evaluate the actual processing effect of the resolution strategy.

9. A big data compliance risk control system, characterized in that, The system comprises: An acquisition and packaging module, configured to acquire original business data, add metadata information to the original business data, package and send the original business data; the metadata information comprises data source identification, data type, data version number; A matching module, configured to perform preliminary matching in a data mode dictionary according to the received metadata information, and determine a data mode matched with the metadata information; A probability inference module, configured to infer data structure and field type in the metadata information under the data mode according to a probability model, and output an inference confidence; A risk shunting module, configured to determine that the original business data has a risk and shunt the original business data to an isolation area in response to the inference confidence being lower than a preset threshold, review the original business data in the isolation area, update the data mode dictionary according to a review result, and adjust allocation priority of review resources according to frequency and complexity of mode evolution corresponding to the original business data in the isolation area; The original business data is reviewed based on human, the review result is structured, key features of the mode evolution and semantic correction rules are extracted; The structured review result is compared with the data mode dictionary to determine a lag part in the data mode dictionary; Update information of the data mode dictionary is generated according to the lag part, and data mode dictionary updating is performed. ​

Images