An automated data compliance verification method based on structured processing of policies and regulations

Abstract

The present invention discloses an automated data compliance verification method based on structured processing of policies and regulations, including the following steps: 1) obtaining the policy and regulatory text, and extracting structured rules therefrom; 2) selecting and configuring data verification rules for the policies and regulations after the structured rules are extracted based on a large model; 3) inputting the data source to be verified, performing data compliance verification, generating a detailed compliance verification report, and providing the compliance status of the data and related analysis results. The present invention extracts rules from the regulatory text through an automated mechanism, intelligently selects artifacts that meet the requirements in the verification component library, and combines them into a verification artifact collection template, thereby improving the accuracy and efficiency of verification; using an advanced rule engine, automatically matching and applying verification rules, and ensuring the adaptability of compliance verification.

Description

Technical Field

[0001] The present invention relates to the field of data compliance, and in particular to an automated data compliance verification method based on structured processing of policies and regulations. Background Art

[0002] Compliance verification refers to the inspection and review of the compliance of an enterprise or organization to ensure that its business activities comply with the requirements of laws, regulations, rules and regulations, industry standards and international practices. This process is an important part of ensuring the legal, standardized, ethical and sustainable operation of an enterprise or organization. Existing compliance verification mainly includes two methods: manual review verification and automated compliance verification.

[0003] Among them, manual review and verification mainly involves reviewers reviewing and comparing relevant laws, regulations, policies and data content. This method has good adaptability and is convenient for handling complex regulatory clauses; and with the update of laws and regulations, manual review can timely adjust and apply the latest compliance requirements to ensure the accuracy and compliance of the review results. Automated compliance verification uses a predefined rule base and compliance framework to help companies automatically perform compliance reviews through pre-mapped regulatory requirements and control measures. Automated compliance verification improves the efficiency and consistency of compliance reviews.

[0004] However, manual review and verification has the following drawbacks:

[0005] Inefficiency: Manual review requires a lot of manpower, especially when dealing with large data sets and complex regulations. The review process is time-consuming and costly. This approach is difficult to meet the needs of efficient and real-time data compliance.

[0006] Highly subjective: Since the review relies on personal understanding and judgment, the standards of different reviewers may be inconsistent, resulting in subjectivity and differences in the results of compliance judgments.

[0007] Susceptible to human errors: Manual review is susceptible to human errors, such as missed reviews, deviations in regulatory understanding, etc., which may lead to compliance risks.

[0008] Automated compliance verification also has the following drawbacks:

[0009] Insufficient flexibility: Automated compliance verification usually relies on predefined rule bases and frameworks, which are difficult to flexibly respond to new regulatory changes or process unstructured data. Such pre-set rule sets are difficult to adapt to regulations with complex rules or frequently updated rules.

[0010] Limited deep semantic understanding: Existing automated systems have limitations in deep semantic understanding and contextual parsing of regulatory texts, making it difficult to fully capture the subtle meanings of regulatory provisions, which may lead to misjudgments or omissions.

[0011] High cost: Usually, the initial configuration needs to be customized according to the specific compliance requirements of the enterprise, which makes the implementation of compliance verification difficult and time-consuming. In addition, if the initial configuration is not appropriate, it may result in the phenomenon that compliance verification cannot meet expectations.

[0012] Therefore, it is necessary to improve the above-mentioned prior art to overcome the above-mentioned defects. Summary of the invention

[0013] The purpose of the present invention is to provide an automated data compliance verification method based on structured processing of policies and regulations. By constructing an intelligent data compliance review system and adopting domain-specific large language models and big data technologies, it can not only deeply understand complex policies and regulations, but also realize real-time automated review, ensuring more accurate and timely data compliance verification, and effectively solving the shortcomings of the existing technology.

[0014] The above technical objectives of the present invention are achieved through the following technical solutions:

[0015] An automated data compliance verification method based on structured processing of policies and regulations includes the following steps:

[0016] 1) Obtain policy and regulatory texts and extract structured rules from them;

[0017] 2) Select and configure data validation rules for policies and regulations after structured rule extraction;

[0018] 3) Enter the data source to be verified and perform data compliance verification.

[0019] Furthermore, the specific method of step 1) is as follows:

[0020] 1.1) Normalized text generation: Convert the input text into normalized text through text preprocessing and cleaning;

[0021] 1.2) Rule Identification: The standardized text is parsed by the intelligent text analysis engine to automatically identify the compliance requirements in the standardized text;

[0022] 1.3) Rule conversion: converting the identified rules into a structured standard format;

[0023] 1.4) Rule storage: Through distributed storage technology, structured policies, regulations and rules are stored in a distributed object storage system;

[0024] 1.5) Regulatory Updates: Automatically verify and update compliance requirements in the rule base when new regulations are enacted or existing regulations change.

[0025] Further, the specific method of step 2) is as follows:

[0026] 2.2) Dynamic matching of artifacts: The rule engine automatically selects and matches the most appropriate rule artifact in the rule artifact library based on the characteristics of the data;

[0027] 2.3) Domain-based big model verification: The system uses pre-trained big models to deeply verify complex data and identify potential abnormal rule artifacts;

[0028] 2.4) Model parameter adjustment: The system continuously optimizes large model parameters to improve verification accuracy;

[0029] 2.5) Combination of minimum verification units: The system intelligently combines minimum verification units through algorithm optimization and meta-learning technology;

[0030] 2.6) Scheduling of large-scale artifacts: A distributed architecture is used to schedule and execute large-scale verification artifacts;

[0031] 2.7) The system designs an automated configuration generation process, and uses a modular framework to enable rapid development and reuse of verification units, ensuring the efficiency and reliability of the verification process.

[0032] Further, the specific method of step 3) is as follows:

[0033] 3.1) Data source input and formatting: The system converts the input data source into a unified format;

[0034] 3.2) Policy and regulation structured rule selection and configuration: Select applicable policies and regulations based on the characteristics of the data source, and configure corresponding validation rules;

[0035] 3.3) Data quality verification: Use the configured verification rules to conduct comprehensive quality verification of data sources, check whether the data complies with policies and regulations, and speed up the processing of large-scale data through distributed systems;

[0036] 3.4) Verification report generation: After the data quality verification is completed, the system generates a detailed compliance verification report, providing the compliance status of the data and related analysis results;

[0037] 3.5) Report Storage: The generated verification reports are stored in a distributed object storage system to ensure secure storage and convenient access to the reports.

[0038] Furthermore, the standardized text generation adopts a self-attention mechanism for context understanding and text generation, as shown in formula (1):

[0039] (1)

[0040] in, is the query matrix of the policy and regulation text fragments to be queried or focused on, is the key matrix of the feature set of all policy and regulation fragments in the text collection, is the value matrix of the actual value of the matched text feature set, is the dimension of the key vector, which is a constant used to scale the inner product result when performing attention mechanism calculations. is the self-attention mechanism formula, Represents the query matrix With key matrix Transpose Perform inner product operation, is the normalization function.

[0041] Furthermore, the rule recognition uses the BERT model to realize rule recognition, as shown in formula (4):

[0042] (4)

[0043] in, Indicates the target word currently being processed; Indicates the target word The context weight matrix of is the target word The transposed matrix of the weight matrix; Target word The context of is the target word Semantic embedding representation in a specific context, is the normalization function.

[0044] Furthermore, the regulatory update adopts the regulatory dynamic update algorithm RDUA and the adaptive learning update algorithm ALUA to ensure the continuous accuracy and timeliness of compliance verification. Formula (10) realizes the dynamic update of rules, and formula (11) realizes adaptive learning.

[0045] (10)

[0046] Represents the updated rules processed by the GPT-4 model; Represents the large language model used; Indicates new regulations or regulatory changes, and is the input data for rule updates;

[0047] (11)

[0048] in, and are the old parameters and new parameters of the model, corresponding to the parameter values ​​of the current model before the application of the new regulations or regulatory changes and the adjusted parameter values ​​after the application of the new regulations or regulatory changes; Represents the learning rate, which is used to control parameter updates; (.) is the loss function, which is used to measure the prediction error of the model for new data. Represents the loss function For model parameters The gradient of the loss function reflects the effect of parameter changes on the loss function impact.

[0049] Furthermore, the calculation method of the weighted average of node features in the workpiece dynamic matching rule is shown in formula (12):

[0050] (12)

[0051] in, It is a graph neural network The node feature matrix of the layer; It is a normalized adjacency matrix, which is used to represent the connection relationship between nodes in the graph and the relationship between structured regulatory clauses and artifacts; It is a graph neural network The node feature matrix of the layer, which contains the representation of structured data and workpiece features of the previous layer and is used to calculate the feature representation of the current layer; It is a graph neural network The weight matrix of the layer is used to linearly transform the structured regulations and artifact features; is an activation function used to introduce nonlinear characteristics to capture the complex relationship between structured regulations and artifact features.

[0052] Furthermore, the calculation of node feature aggregation in the dynamic matching of workpieces adopts formula (13):

[0053] (13)

[0054] in, is a node in a graph neural network In the A hierarchical representation of features, which enables the characteristics of each structured regulatory clause or artifact to take into account information about other clauses or artifacts with which it is associated; It is an activation function, used to introduce nonlinear characteristics; It is a node The set of adjacent points of Directly connected structured regulatory provisions or artifacts; It is a node To Node The normalization coefficient is used to adjust the weight of node features to ensure that the weighted average of features is reasonable; It is The weight matrix of the layer is used to adjust the adjacent points The features are linearly transformed to extract more representative features; It is a node In the The feature representation of the layer reflects the node The state of information at the current level;

[0055] Formula (14) is used for artifact selection, which optimizes artifact matching under different regulatory scenarios by balancing the source task loss and domain adversarial loss;

[0056] (14)

[0057] in, represents the total loss of transfer learning, which consists of the source task loss and the domain adversarial loss, and represents the ability to optimize rule matching; represents the loss of the source task, reflecting the prediction error of the model in the source domain; represents the domain-adversarial loss, which aims to reduce the difference between the source and target domains; is the weight coefficient used to balance the loss term and .

[0058] Furthermore, the combination of the minimum verification unit adopts the IDUO algorithm, and the implementation of the IDUO algorithm is shown in formula (16) to formula (18):

[0059] Formula (16) is used to calculate the weighted average of node features;

[0060] (16)

[0061] in, are the model parameters of the meta-learning model MAML, representing the model settings used to map structured policies and regulations to artifacts; is the learning rate, which indicates the adaptability of the MAML model to new regulations; It is The loss function of each task is used to evaluate the matching effect between artifacts and regulatory structured data; Indicates the model output after gradient update under the current parameters; Represents finding the parameter value that minimizes the sum of the loss functions , returns the optimal parameters , ensuring that the model can effectively adapt to new regulatory requirements;

[0062] Formula (17) is used to calculate the aggregation of node features;

[0063] (17)

[0064] in, Yes Status and actions of value, which indicates the expected effect of selecting an artifact under certain regulatory conditions; is an immediate reward, which represents the direct feedback obtained after selecting an artifact under a specific policy regulation; is a discount factor, which indicates the importance of evaluating and considering the possible effects in future steps when selecting artifacts. The higher the value, the more attention the algorithm pays to the long-term impact of the artifacts selected in the future. The lower the value, the more attention is paid to the immediate effects. is the learning rate, which indicates the flexibility and adjustment speed of the strategy in selecting artifacts, and is used to control The update amplitude of the value ensures that the model can quickly adapt to new feedback information; Indicates the new state All possible actions of The maximum value among the values ​​reflects the expected effect of the optimal strategy in this state;

[0065] Formula (18) is used to estimate the optimal parameters for workpiece selection;

[0066] (18)

[0067] in, Is input and observations The objective function under The posterior distribution of , which represents the probability of the optimal parameter configuration for workpiece selection; Represents Gaussian distribution, which is used to describe the probability distribution of random variables; is the mean function, indicating that the known structured policies and regulations , the expected value of artifact selection; Covariance function measures the correlation between different policy and regulatory characteristics.

[0068] In summary, the present invention has the following beneficial effects:

[0069] 1) Policy and regulation structuring technology: The core lies in structuring the text of policies and regulations. Through natural language processing and intelligent rule engine technology, the present invention can automatically parse the text of regulations, extract key rules from them, and convert these rules into an operable structured format. This structured rule format can be directly used in the automated process of data compliance verification, effectively reducing the need for manual participation and significantly improving the efficiency and accuracy of compliance verification.

[0070] 2) Automatic verification artifact selection technology: By analyzing the structured rules in the regulatory text, the artifacts that meet the specified requirements are automatically selected from the preset verification component library. The selection process is dynamic and intelligent, and can automatically combine the most appropriate verification artifact collection template according to different regulations and compliance requirements, thereby ensuring the comprehensiveness and accuracy of compliance verification.

[0071] 3) Compliance verification report generation technology: This method can automatically generate a detailed compliance verification report after completing data compliance verification. The report content not only includes the verification results of the data, but also covers the detailed analysis of each verification artifact and the matching of corresponding regulations and rules. This report can be used as a reference for auditors and can also be used for subsequent compliance processes and decision support. Automated report generation technology improves the transparency of information and the traceability of the verification process, making compliance audits more efficient and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a flow chart of the automated data compliance verification method described in the present invention.

[0073] Figure 2 It is a schematic diagram of the structured extraction of policies and regulations described in the present invention.

[0074] Figure 3 It is a schematic diagram of selecting and configuring data validation rules based on a large model according to the present invention.

[0075] Figure 4 It is a schematic diagram of the high-accuracy compliance verification method and report generation described in the present invention.

[0076] Figure 5 It is an application flow chart of the automated data compliance verification method described in the present invention. DETAILED DESCRIPTION

[0077] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further described below in conjunction with illustrations and specific embodiments.

[0078] like Figures 1 to 5As shown in the figure, the automated data compliance verification method proposed in the present invention uses novel intelligent text analysis technology to automatically analyze policy and regulatory texts, extract rules from them and structure them, thereby providing efficient support for data compliance verification. This process uses advanced deep learning models and incremental learning technologies to ensure accurate analysis of regulatory clauses and rule conversion, thereby achieving efficient and accurate compliance verification.

[0079] The specific steps are as follows:

[0080] 1) Standardized text generation

[0081] This step receives multiple text input types, including but not limited to PDF, Word documents, web page text, etc. The input text is converted into a standardized format (plain text format). This format conversion uses the new technology developed by the present invention, combining the large language model (LLM) and preprocessing technology, and specifically includes the following technical details:

[0082] LLM preprocessing model: This paper adopts the GPT (Generative Pre-trained Transformer) model based on policy and regulatory text normalization to preprocess and normalize the input text, aiming to improve the accuracy and consistency of text processing and ensure that regulatory clauses can be effectively mapped to structured data formats. The GPT model uses the Self-Attention Mechanism to perform context understanding and text generation, as shown in formula (1):

[0083] (1)

[0084] in, and denote the query matrix, key matrix and value matrix respectively, represents the dimension of the key vector. In this text normalization process, It represents the policy and regulation text fragments that need to be queried or concerned, and is the part that needs to be processed and converted. The query matrix is ​​refined to better capture and process the common legal terms and expressions in the regulations; A feature set representing all policy and regulatory fragments in a text collection, used to Used to match and find the corresponding context information; Represents the actual values ​​of the matched text feature set, which will be used to generate the normalized text for the key matrix Sum Matrix Customized adjustments were made to enable the model to more accurately match and generate standardized text that meets regulatory requirements; It represents the constant used to scale the inner product result when performing attention mechanism calculation to ensure the stability of the calculation. According to the length and complexity of the regulatory text, the dimension of the key vector is optimized to make it more suitable for feature extraction of regulatory text fragments.

[0085] Preprocessing step: Use data cleaning algorithm to remove noise and redundant information in the text. Specifically include the following steps:

[0086] Noise removal: The present invention uses a text filter algorithm to clean up the text. The core idea of ​​the algorithm is to first remove noise from the normalized text, and then convert the text after the noise is removed into pure text. The implementation process is reflected in formulas (2) and (3).

[0087] (2)

[0088] (3)

[0089] in, Represents a text that is closer to the target normalized text after removing redundant quotations, lengthy comments, and other information that may interfere with understanding; Represents the original input policy and regulation text, that is, the policy, law or regulation text that has not been processed in any way; Noise represents the content in the policy and regulation text that is irrelevant to the rules to be extracted; It refers to the standardized operation of converting the policy and regulatory text into a normalized text format to ensure the consistency and standardization of the text. Normalized Text refers to the text that converts the input policy and regulatory text into its normalized version.

[0090] 2) Rule Identification

[0091] The present invention uses the LLM-BERT semantic analysis algorithm to implement an intelligent text analysis engine, conducts deep semantic analysis and identification of compliance requirements and clauses in standardized texts, and extracts key rules in standardized texts. The LLM-BERT semantic analysis algorithm is optimized specifically for the characteristics of regulatory texts, uses a large language model (LLM) to implement deep semantic analysis, accurately locates compliance requirements and clauses through contextual relationships, and uses the BERT (Bidirectional Encoder Representations from Transformers) model to implement rule recognition. The calculation of the algorithm when implementing deep semantic understanding of standardized texts is shown in formula (4).

[0092] (4)

[0093] in, Indicates the target word currently being processed, that is, a key term or rule word extracted from the policy and regulation text; Indicates the target word The context weight matrix of w, these context words are other words associated with the target word w, used to provide semantic background; is the transposed matrix of the target word weight matrix. This weight matrix is ​​related to the target word Directly related weights are used to calculate the semantic representation of the target word in a specific context; Target word The context, that is, the sequence of words or phrases surrounding the target word in the regulatory text, helps understand the semantics of the target word. is the target word The semantic embedding representation in a specific context reflects the semantic meaning of the word in the current context of the regulatory text. The normalization function converts the calculation results into probability distribution to ensure that the semantic representation of the target word in a specific context can reasonably reflect its importance.

[0094] Contextual Embedding: Apply the contextual embedding model CE to convert text into vectors for further semantic analysis. The core idea of ​​the contextual embedding model CE is shown in formula (6).

[0095] (6)

[0096] in, Represents the context embedding vector of the text. It is formed by embedding the target word and its contextual information The vector representation generated by the combination reflects the semantic characteristics of the word in a given context; are target words, i.e., specific words or terms that need to be analyzed in the regulatory text; represents the context of the target word 𝑤, i.e., the sequence of words or phrases surrounding the target word in the regulatory text; It means to combine the target word 𝑤 and its context The process function that performs the conversion.

[0097] 3) Rule conversion

[0098] The LLM-Driven Rule Structuring algorithm is used to achieve the structural transformation of normalized text. The proposed algorithm consists of two steps: 1) extracting rule information from the normalized text, including conditions, logical expressions, and verification criteria; 2) generating a structured format. The core idea of ​​the LLM-Driven Rule Structuring algorithm is reflected in formulas (7)-(9).

[0099] (7)

[0100] in, Indicates the rules that have been processed by the GPT-4 model and converted into a structured format that complies with the specification for further compliance verification; Represents a large language model used by the present invention; It indicates that the rule information including conditions, logical expressions and validation criteria extracted by the proposed model from the normalized text is the input data of the transformation.

[0101] The specific steps to convert rule information into a structured format include:

[0102] Conditional extraction: Apply rule extraction algorithms Extraction conditions, the algorithm idea is shown in formula (8):

[0103] (8)

[0104] in, Represents structured condition information, that is, conditions extracted from rules and converted into a structured format; represents the rule extraction algorithm, whose function is to identify and extract conditional information from the extracted rules; Represents the original rule information, which is the input data for conditional extraction;

[0105] Logical Expression Generation: Using Expression Generation Model Generate a logical expression, the idea is shown in formula (9).

[0106] (9)

[0107] in, Represents the generated, compliant logical expression for further compliance verification; It is an expression generation model, whose function is to convert conditional information into logical expressions; Represents the extracted conditional information, which is the input data generated by the logical expression.

[0108] 4) Update regulations

[0109] The present invention adopts the Regulation Dynamic Update Algorithm (RDUA) to realize the automatic update of regulations, and the Adaptive Learning Update Algorithm (ALUA) to ensure the continuous accuracy and timeliness of compliance verification. The core idea of ​​the RDUA algorithm is mainly reflected in formula (10), and the core idea of ​​the ALUA algorithm is mainly reflected in formula (11).

[0110] (10)

[0111] in, Indicates updated rules processed by the GPT-4 model, reflecting the latest status of new regulations or regulatory changes; Represents a large language model used by the present invention; Indicates new regulations or information about changes in regulations, and is input data for rule updates.

[0112] (11)

[0113] in, and are the old parameters and new parameters of the model, corresponding to the parameter values ​​of the current model before the application of the new regulations or regulatory changes and the adjusted parameter values ​​after the application of the new regulations or regulatory changes; Represents the learning rate, which is used to control parameter updates; (.) is the loss function, which is used to measure the prediction error of the model for new data. In the context of regulatory changes, the learning rate controls how quickly the model adapts to new regulations; It represents the gradient of the loss function, reflecting the sensitivity of the model parameters to new data. The gradient reflects the accuracy and real-time response of the model to regulatory changes.

[0114] 5) Rule storage

[0115] Finally, the extracted and transformed structured rules are stored in the specified MINIOBucket through distributed storage technology. As a high-performance distributed object storage system, MINIO is compatible with the Amazon S3 API. Compared with traditional distributed file systems, it provides higher storage efficiency and flexibility. MINIO has built-in efficient redundancy management and data backup mechanisms, supports distributed data redundancy, automatically manages high availability and persistence of data, and reduces maintenance complexity. In addition, MINIO supports a variety of storage strategies, including data lifecycle management and version control, which effectively reduces costs and improves compatibility with cloud-native applications. Through these advantages, MINIO ensures the safe and reliable storage and efficient reading of rule data, and supports the continuity and traceability of compliance verification.

[0116] II. Data validation rule selection and configuration based on large models

[0117] Based on the extraction rules, a data verification selection method and process based on a large model is designed to handle complex and changing regulatory provisions. The steps are as follows:

[0118] 1) Establishment of a standardized rule artifact library

[0119] First, a standardized rule artifact library is established, covering various verification functions such as data format verification, value range check, missing value identification, etc. By standardizing different verification functions, it is ensured that the rule artifact library can support various data verification requirements, providing a basis for subsequent verification selection and application.

[0120] 2) Dynamic matching of workpieces

[0121] The present invention adopts the rule adaptation algorithm GraphRuleNet to capture and express the correlation between regulatory rules (such as structured data of policies and regulations) and artifacts (such as verification units, verification tools, etc.) to achieve dynamic matching of artifacts; the rule optimization algorithm DeepTransferRule combines the source task loss and domain adversarial loss to optimize the selection and application of rule artifacts and improve the rule matching ability; the function of the dynamic rule adjustment algorithm AttentionRuleAdjust is to use the self-attention mechanism to dynamically adjust the focus on key features of the data, so as to more accurately match and apply rule artifacts. Through the comprehensive application of these three algorithms, the comprehensiveness and accuracy of the compliance verification process are ensured. The core ideas of the three proposed algorithms are respectively embodied in formulas (12)-(15). Among them, formula (12) is used to calculate the weighted average of node features; formula (13) is used to calculate the aggregation of node features; formula (14) is used for artifact selection, and by balancing the source task loss and domain adversarial loss, the artifact matching in different regulatory scenarios is optimized; formula (15) is used to calculate the attention weight.

[0122] (12)

[0123] in, It is a graph neural network The node feature matrix of the layer contains the node information extracted by the graph convolutional network. It contains a comprehensive representation of structured regulatory clauses, conditions, and logical expressions. The graph convolution operation captures the features and contextual information of related artifacts in the artifact library, which facilitates subsequent matching and analysis. It is a normalized adjacency matrix, which is used to represent the connection relationship between nodes in the graph and the relationship between structured regulatory clauses and artifacts; It is a graph neural network The node feature matrix of the layer, which contains the representation of structured data and workpiece features of the previous layer and is used to calculate the feature representation of the current layer; It is a graph neural network The weight matrix of the layer is used to linearly transform the structured regulations and artifact features; is an activation function used to introduce nonlinear characteristics to capture the complex relationship between structured regulations and artifact features, enabling the model to learn complex feature representations.

[0124] (13)

[0125] in, is a node in a graph neural network In the A hierarchical representation of features, which enables the characteristics of each structured regulatory clause or artifact to take into account information about other clauses or artifacts with which it is associated; It is a node The set of adjacent points of Directly connected structured regulatory provisions or artifacts; It is a node To Node The normalization coefficient is used to adjust the weight of the node features to ensure that the weighted average of the features is reasonable.

[0126] (14)

[0127] in, represents the total loss of transfer learning, which consists of the source task loss and the domain adversarial loss, and represents the ability to optimize rule matching; represents the loss of the source task, reflecting the prediction error of the model in the source domain; represents the domain-adversarial loss, which aims to reduce the difference between the source and target domains; is the weight coefficient used to balance the loss term and .

[0128] (15)

[0129] in, is the query matrix, which represents the characteristics of the structured regulatory clause or artifact currently being processed; Represents all artifact features or structured regulatory clause features to be matched in the artifact library; is a value matrix, representing the feature information used for extraction during the matching process; is the dimension of the key vector, used to scale the dot product to ensure the stability of the calculation; is a normalization function, which means converting the dot product result into a probability distribution, and converting the calculated similarity into a probability distribution, so as to assign a suitable weight to each artifact or regulatory clause, ensure the focus is reasonable, and thus improve the accuracy of matching.

[0130] In summary, through the application of these three algorithms, the present invention can dynamically select verification artifacts that match data characteristics, ensuring the comprehensiveness and accuracy of the verification process.

[0131] 3) Verification based on domain-wide large models

[0132] For complex data, the system uses pre-trained domain models to efficiently verify regulatory data for anomalies through transfer learning technology. The large model has deep semantic understanding and cross-modal analysis capabilities, and can process complex data combined with different data sources (such as regulatory texts and financial statements) to identify hidden abnormal patterns.

[0133] 4) Model parameter adjustment

[0134] The system continuously optimizes the parameters of the domain model, including analysis depth, context window size, semantic similarity threshold, etc., to improve the performance of the model in data verification. By adjusting the model parameters, the system can improve the accuracy and efficiency of anomaly verification and adapt to the ever-changing data characteristics and business needs.

[0135] 5) Combination of minimum verification units

[0136] The comprehensive optimization verification unit algorithm IDUO is used to achieve the combination of the minimum verification unit. The proposed algorithm integrates technologies such as meta-learning, reinforcement learning and Bayesian model, has the adaptability of regulations and business scenarios, and ensures the efficiency of compliance verification. Among them, the meta-learning technology is implemented using the MAML model to ensure that the IDUO algorithm can dynamically adjust the combination of verification units based on historical data and feedback, and optimize the performance of the model in different tasks. Reinforcement learning technology uses the deep Q network (DQN) to achieve continuous improvement of the strategy, ensuring that the IDUO algorithm can dynamically select the most suitable data verification artifact. The Bayesian model is implemented using Gaussian process regression (GPR), and the optimal parameters are selected through the posterior distribution to ensure that the IDUO algorithm can find the optimal combination of verification units.

[0137] The core of the IDUO algorithm is mainly reflected in formulas (16)-(18). Among them, formula (16) is used to calculate the weighted average of node features; formula (17) is used to calculate the aggregation of node features; formula (18) is used to estimate the optimal parameters for workpiece selection.

[0138] (16)

[0139] in, are the model parameters of the meta-learning model MAML, representing the model settings used to map structured policies and regulations to artifacts; is the learning rate, which indicates the adaptability of the MAML model to new regulations; It is The loss function of each task is used to evaluate the matching effect between artifacts and regulatory structured data;

[0140] is the MAML model prediction function, which means based on the current parameters Map the input structured policies and regulations to generate corresponding artifact selections.

[0141] (17)

[0142] in, Yes Status and actions of value, which indicates the expected effect of selecting an artifact under certain regulatory conditions; is an immediate reward, which represents the direct feedback obtained after selecting an artifact under a specific policy regulation; is the discount factor, which indicates the importance attached to the effect of subsequent artifact selection; is the learning rate, which indicates the flexibility and speed of adjustment of the strategy in selecting artifacts.

[0143] (18)

[0144] in, is the mean function, indicating that the known structured policies and regulations , the expected value of artifact selection; Covariance function, which measures the correlation between different policy and regulatory characteristics; Is input and the objective function under the observed value y The posterior distribution of , represents the probability of the optimal parameter configuration for the workpiece selection.

[0145] Through the comprehensive application of the above three algorithms, the present invention can intelligently adjust and optimize the combination of verification units to adapt to different regulations and business scenarios, thereby enhancing the flexibility and adaptability of compliance verification.

[0146] 6) Scheduling of large-scale workpieces

[0147] In data verification tasks, the system uses a distributed architecture to schedule and execute large-scale artifacts. Through advanced resource management technology, computing resources are reasonably allocated to ensure efficient processing and rapid response of large-scale data verification tasks.

[0148] 7) Generate automated configuration

[0149] Design an automated configuration generation process to achieve rapid development and reuse of verification units through a modular framework. The automated process reduces the need for manual configuration, improves the system's responsiveness and flexibility, and ensures the efficiency and reliability of the verification process.

[0150] III. High-accuracy compliance verification method and report generation

[0151] Based on step II, the present invention designs and implements a highly accurate compliance verification method to ensure the compliance and reliability of the data. The steps are as follows:

[0152] 1) Data source input and formatting

[0153] First, the system receives the data source to be verified and supports multiple data formats (including but not limited to RDBMS, txt, json, csv, xlsx). The input data will be converted into a unified Parquet format to ensure consistency and efficiency of data processing. This process ensures that data from different sources can be processed uniformly, improving the accuracy of subsequent data analysis.

[0154] 2) Selection and configuration of structured rules for policies and regulations

[0155] Based on the characteristics of the input data, the system selects applicable structured rules and automatically configures the data validation suite for compliance verification, creating a template for a collection of artifacts for a specific data source. Based on business needs and regulatory requirements, users can adjust the parameters of the rules to ensure the flexibility and pertinence of the verification process.

[0156] 3) Data quality verification

[0157] Use the generated data verification suite to conduct comprehensive quality verification of the data. The system uses a distributed architecture to schedule and execute data verification tasks, supports rapid processing of large-scale data, and ensures the efficiency and accuracy of data quality verification.

[0158] 4) Verification report generation

[0159] After completing data verification, the system generates a detailed verification report in HTML format. The report includes the compliance status of the input data, detailed information on non-compliant data, and corresponding rectification suggestions, providing users with a clear compliance status and improvement measures.

[0160] 5) Report storage

[0161] Each verification report generated will be stored in MINIO, and users can download and browse the report at any time. This function ensures long-term preservation and convenient access to the report, providing users with effective compliance verification records.

[0162] Example

[0163] See also Figure 1In this embodiment, the automated data compliance verification method of the present invention includes three main steps: first, structural extraction of policy and regulatory texts. Next, the selection and configuration of data verification rules are performed using big models and big data technologies. Finally, data compliance verification is performed, comprehensive data quality verification is performed, and an HTML format report containing rectification suggestions is generated, and the report is stored in MINIO for easy download and browsing at any time.

[0164] See also Figure 2 The process of structured extraction of policies and regulations of the present invention is as follows:

[0165] Normalized text generation: The system receives policy and regulatory texts from multiple sources, removes irrelevant information, converts them into clearly structured and complete texts, and stores them in plain text format. For example, if a company needs to comply with a new data privacy regulation, the system will receive the PDF document of the regulation, remove footnotes and irrelevant instructions, convert it into plain text format, and generate a standardized text that is convenient for subsequent analysis.

[0166] Rule identification: The system uses an intelligent text analysis engine to deeply analyze the normalized text and automatically identify the compliance requirements in the regulations. In this example, the system automatically identifies the key clause in the new data privacy regulations that requires data encryption to be ensured during data storage.

[0167] Rule conversion: The identified rules are converted into a structured standard format. For the requirements of the above data privacy regulations, the system will convert them into structured rules of "IF data is stored THEN encrypt data". This format is convenient for the system to use directly in subsequent compliance verification.

[0168] Update regulations: When new regulations are enacted or existing regulations change, the system automatically verifies these changes and updates the relevant clauses in the rule base in real time. For example, over time, if the data privacy regulation adds a requirement for multi-factor authentication, the system automatically recognizes this new requirement and updates the rule base accordingly, possibly adding a new rule such as "IF user login THEN require multi-factor authentication".

[0169] Rule storage: All generated structured rules are securely stored in the MINIO Bucket. In this example, the rules for data encryption and multi-factor authentication are stored in the MINIO Bucket. These rules can be called at any time to support the compliance verification needs of enterprises at different time periods, such as periodic data security audits.

[0170] See also Figure 3 The data validation rule selection and configuration process based on the big model of the present invention is as follows:

[0171] Establishment of a standardized rule artifact library: The system collects and standardizes various verification functions and establishes a rule artifact library. For example, for the data compliance verification needs of an enterprise, the system will establish a standardized rule artifact library covering various verification functions, including regulatory compliance, financial statement analysis and other functions.

[0172] Dynamic matching of artifacts: During the verification process, the rule engine automatically matches the corresponding rule artifacts according to the characteristics of the data. For example, when the system receives complex data containing regulatory texts and financial statements, the rule engine automatically matches the rule artifacts that process these two data types.

[0173] Verification based on domain big models: The system uses big models to deeply verify complex data and identify abnormal patterns. In this example, the system uses pre-trained domain big models to perform deep semantic analysis on regulatory texts and financial statements to identify possible hidden abnormal patterns.

[0174] Model parameter adjustment: The system continuously optimizes the parameters of the big model to improve verification accuracy. According to the data characteristics of different enterprises, the system will continuously adjust and optimize the parameters of the big model to ensure that various compliance issues can be accurately identified.

[0175] Combination of minimum verification units: The system intelligently combines minimum verification units through algorithms and meta-learning. When processing complex data of an enterprise, the system automatically adjusts and combines minimum verification units through meta-learning technology to adapt to different data characteristics and compliance requirements.

[0176] Scheduling of large-scale artifacts: A distributed architecture is used to schedule and execute large-scale verification artifacts. For large enterprises with large amounts of data, the system uses a distributed architecture to efficiently schedule verification artifacts to ensure that comprehensive compliance checks can be completed within a reasonable time.

[0177] Generate automated configuration: The system generates automated configuration to ensure the efficiency and reliability of the verification process. Ultimately, the system will generate automated configuration based on the specific needs and data characteristics of the enterprise to ensure the efficiency, accuracy and reliability of the entire compliance verification process.

[0178] See also Figure 4 The high-accuracy compliance verification method and report generation steps of the present invention are as follows:

[0179] Data source input and formatting: The input financial statement data has multiple formats (such as Excel tables and JSON files). The system formats this data into Parquet format to uniformly handle the selection and configuration of structured rules for policies and regulations. For example, the system analyzes the characteristics of financial data, automatically selects applicable tax regulations, and configures corresponding validation rules. Users can manually adjust some parameters according to business needs, such as adjusting the upper and lower limits of tax rates;

[0180] Data quality verification: The system uses configured verification rules to conduct comprehensive quality verification of financial statements to check whether they meet tax compliance requirements and speed up the processing of large-scale data through distributed systems;

[0181] Verification report generation: After verification is completed, the system generates a detailed data compliance verification report in HTML format, listing compliant data, non-compliant data and their corresponding modification suggestions;

[0182] Report storage: The generated verification report is stored in MINIO. Users can log in to the system at any time, download and view the report, and retain the historical record for subsequent review or regulatory needs.

[0183] See also Figure 5 , the application process of the present invention is as follows:

[0184] 1) Legal document preparation

[0185] Prepare, including but not limited to, PDF, Word documents, web page texts, etc. as input sources for policies and regulations;

[0186] 2) Structured extraction of policies and regulations

[0187] Extract structured rules based on the input legal text and save the results to MINIO;

[0188] 3) Data validation rule selection and configuration based on large models

[0189] Read the rules generated in step 2 to select and configure data validation rules;

[0190] 4) Enter the data source to be verified

[0191] Enter the data source to be verified by the enterprise, including RDBMS, file data, etc.;

[0192] 5) High-accuracy compliance verification methods and report generation

[0193] Perform data quality verification to check compliance with data protection measures as required by applicable regulations; generate reports showing data compliance, flagging non-compliant records, and providing rectification recommendations.

[0194] In this document, the directions or positional relationships indicated by terms such as "up", "down", "front", "back", "left", "right", "top", "bottom", "inside", "outside", "vertical", and "horizontal" are based on the directions or positional relationships shown in the accompanying drawings and are only for the clarity of expressing the technical solutions and the convenience of description, and therefore should not be understood as limitations of the present invention.

[0195] In this document, the terms "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion of elements other than those listed and may also include additional elements not expressly listed.

[0196] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An automated data compliance verification method based on structured processing of policies and regulations, characterized in that: The steps include: 1) Obtain policy and regulatory texts and extract structured rules from them; 2) Select and configure data validation rules for policies and regulations after structured rule extraction; 3) Input the data source to be verified and conduct data compliance verification; The specific method of step 1) is as follows: 1.1) Normalized text generation: Convert the input text into normalized text through text preprocessing and cleaning; 1.2) Rule Identification: The standardized text is parsed by the intelligent text analysis engine to automatically identify the compliance requirements in the standardized text; 1.3) Rule conversion: converting the identified rules into a structured standard format; 1.4) Rule storage: Through distributed storage technology, structured policies, regulations and rules are stored in a distributed object storage system; 1.5) Regulatory updates: Automatically verify and update compliance requirements in the rule base when new regulations are enacted or existing regulations change; The regulatory update adopts the regulatory dynamic update algorithm RDUA and the adaptive learning update algorithm ALUA to ensure the continuous accuracy and timeliness of compliance verification. Formula (10) realizes the dynamic update of rules, and formula (11) realizes adaptive learning. (10) Represents the updated rules processed by the GPT-4 model; Represents the large language model used; Indicates new regulations or regulatory changes, and is the input data for rule updates; (11) in, and are the old parameters and new parameters of the model, corresponding to the parameter values ​​of the current model before the application of the new regulations or regulatory changes and the adjusted parameter values ​​after the application of the new regulations or regulatory changes; Represents the learning rate, which is used to control parameter updates; (.) is the loss function, which is used to measure the model's performance on new data. The prediction error, Represents the loss function For model parameters The gradient of the loss function reflects the effect of parameter changes on the loss function impact.

2. The automated data compliance verification method based on structured processing of policies and regulations according to claim 1 is characterized in that: The specific method of step 2) is as follows: 2.1) Establishment of a standardized rule artifact library: Collect and standardize various verification functions and establish a rule artifact library; 2.2) Dynamic matching of artifacts: The rule engine automatically selects and matches the most appropriate rule artifact in the rule artifact library based on the characteristics of the data; 2.3) Domain-based big model verification: The system uses pre-trained big models to deeply verify complex data and identify potential abnormal rule artifacts; 2.4) Model parameter adjustment: The system continuously optimizes large model parameters to improve verification accuracy; 2.5) Combination of minimum verification units: The system intelligently combines minimum verification units through algorithm optimization and meta-learning technology; 2.6) Scheduling of large-scale artifacts: A distributed architecture is used to schedule and execute large-scale verification artifacts; 2.7) The system designs an automated configuration generation process, and uses a modular framework to enable rapid development and reuse of verification units, ensuring the efficiency and reliability of the verification process.

3. The automated data compliance verification method based on structured processing of policies and regulations according to claim 1 is characterized in that: The specific method of step 3) is as follows: 3.1) Data source input and formatting: The system converts the input data source into a unified format; 3.2) Policy and regulation structured rule selection and configuration: Select applicable policies and regulations based on the characteristics of the data source, and configure corresponding validation rules; 3.3) Data quality verification: Use the configured verification rules to conduct comprehensive quality verification of data sources, check whether the data complies with policies and regulations, and speed up the processing of large-scale data through distributed systems; 3.4) Verification report generation: After the data quality verification is completed, the system generates a detailed compliance verification report, providing the compliance status of the data and related analysis results; 3.5) Report Storage: The generated verification reports are stored in a distributed object storage system to ensure secure storage and convenient access to the reports.

4. The automated data compliance verification method based on structured processing of policies and regulations according to claim 1 is characterized in that: The standardized text generation adopts the self-attention mechanism for context understanding and text generation, as shown in formula (1): (1) in, is the query matrix of the policy and regulation text fragments to be queried or focused on, is the key matrix of the feature set of all policy and regulation fragments in the text collection, is the value matrix of the actual value of the matched text feature set, is the dimension of the key vector, which is a constant used to scale the inner product result when performing attention mechanism calculations. is the self-attention mechanism formula, Represents the query matrix With key matrix Transpose Perform inner product operation, is the normalization function.

5. The automated data compliance verification method based on structured processing of policies and regulations according to claim 1 is characterized in that: The rule recognition uses the BERT model to realize rule recognition, as shown in formula (4): (4) in, Indicates the target word currently being processed; Indicates the target word The context weight matrix of is the target word The transposed matrix of the weight matrix; Target word The context of is the target word Semantic embedding representation in a specific context, is the normalization function.

6. The automated data compliance verification method based on structured processing of policies and regulations according to claim 2 is characterized in that: The calculation method of the weighted average of node features in the workpiece dynamic matching rule is shown in formula (12); (12) in, It is a graph neural network The node feature matrix of the layer; It is a normalized adjacency matrix, which is used to represent the connection relationship between nodes in the graph and the relationship between structured regulatory clauses and artifacts; It is a graph neural network The node feature matrix of the layer, which contains the representation of structured data and workpiece features of the previous layer and is used to calculate the feature representation of the current layer; It is a graph neural network The weight matrix of the layer is used to linearly transform the structured regulations and artifact features; is an activation function used to introduce nonlinear characteristics to capture the complex relationship between structured regulations and artifact features.

7. The automated data compliance verification method based on policy and regulation structured processing according to claim 2 is characterized in that: The calculation of node feature aggregation in the workpiece dynamic matching is shown in formula (13); (13) in, is a node in a graph neural network In the A hierarchical representation of features, which enables the characteristics of each structured regulatory clause or artifact to take into account information about other clauses or artifacts with which it is associated; It is an activation function, used to introduce nonlinear characteristics; It is a node The set of adjacent points of Directly connected structured regulatory provisions or artifacts; It is a node To Node The normalization coefficient is used to adjust the weight of node features to ensure that the weighted average of features is reasonable; It is The weight matrix of the layer is used to adjust the adjacent points The features are linearly transformed to extract more representative features; It is a node In the The feature representation of the layer reflects the node The state of information at the current level; Formula (14) is used for artifact selection, which optimizes artifact matching under different regulatory scenarios by balancing the source task loss and domain adversarial loss; (14) in, represents the total loss of transfer learning, which consists of the source task loss and the domain adversarial loss, and represents the ability to optimize rule matching; represents the loss of the source task, reflecting the prediction error of the model in the source domain; represents the domain-adversarial loss, which aims to reduce the difference between the source and target domains; is the weight coefficient used to balance the loss term and .

8. The automated data compliance verification method based on structured processing of policies and regulations according to claim 2 is characterized in that: The combination of the minimum verification unit adopts the IDUO algorithm, and the IDUO algorithm is shown in formula (16) to formula (18): Formula (16) is used to calculate the weighted average of node features; (16) in, are the model parameters of the meta-learning model MAML, representing the model settings used to map structured policies and regulations to artifacts; is the learning rate, which indicates the adaptability of the MAML model to new regulations; It is The loss function of each task is used to evaluate the matching effect between artifacts and regulatory structured data; Indicates the model output after gradient update under the current parameters; Represents finding the parameter value that minimizes the sum of the loss functions , returns the optimal parameters , ensuring that the model can effectively adapt to new regulatory requirements; Formula (17) is used to calculate the aggregation of node features; (17) in, Yes Status and actions of value, which indicates the expected effect of selecting an artifact under certain regulatory conditions; is an immediate reward, which represents the direct feedback obtained after selecting an artifact under a specific policy regulation; is a discount factor, which indicates the importance of evaluating and considering the possible effects in future steps when selecting artifacts. The higher the value, the more attention the algorithm pays to the long-term impact of the artifacts selected in the future. The lower the value, the more attention is paid to the immediate effects. is the learning rate, which indicates the flexibility and adjustment speed of the strategy in selecting artifacts, and is used to control The update amplitude of the value ensures that the model can quickly adapt to new feedback information; Indicates the new state All possible actions of The maximum value among the values ​​reflects the expected effect of the optimal strategy in this state; Formula (18) is used to estimate the optimal parameters for workpiece selection; (18) in, Is input and observations The objective function under The posterior distribution of , which represents the probability of the optimal parameter configuration for workpiece selection; Represents Gaussian distribution, which is used to describe the probability distribution of random variables; is the mean function, indicating that the known structured policies and regulations , the expected value of artifact selection; Covariance function measures the correlation between different policy and regulatory characteristics.

Images