A compliance data set construction method and system for vertical AI models

Abstract

The application discloses a compliance data set construction method and system for vertical AI models, and belongs to the technical field of artificial intelligence. The method comprises the following steps: performing semantic analysis on legal provisions by a legal analysis engine, and converting the analysis result into a Boolean logic expression representing legal constraint conditions based on a semantic rule template library, and then constructing a strategy mapping association table between compliance feature labels and desensitization operators; calling a named entity recognition model fused with a vertical domain knowledge graph to perform semantic scanning on an original data set to identify sensitive entities and implied indirect sensitive risks; dynamically calling corresponding desensitization operators to perform differentiated desensitization processing according to the identification result; performing re-identification risk assessment on the processed data set, and triggering feedback adjustment to upgrade the processing strength of the operator when the risk exceeds a threshold value until the compliance requirement is met. The application maximizes the available value of data under the premise of ensuring data security compliance.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a method and system for constructing compliant datasets for vertical AI models. Background Technology

[0002] With the increasing application of artificial intelligence in vertical fields such as healthcare, finance, and law, building compliant training datasets that meet regulatory requirements has become a crucial step in ensuring the security and reliability of models. This process necessitates identifying and anonymizing sensitive information in the raw data according to current laws and regulations to eliminate the risk of privacy breaches, while preserving as much of the data's business usability as possible.

[0003] Currently, when translating legal and regulatory requirements into specific data anonymization strategies, hard-coded rules or static keyword matching are commonly used. The compliance processing logic of such methods is highly coupled with the underlying code implementation. Once the legal provisions or industry standards on which they are based are revised or updated, technical personnel must re-parse the legal text, manually modify the data anonymization rules, and redeploy the system. This results in long response cycles to dynamic regulatory changes, high maintenance costs, and makes it difficult to meet the stringent requirements for compliance timeliness in the data governance field.

[0004] Furthermore, traditional methods lack the ability to quantitatively assess the overall compliance of the de-identified dataset and to automatically adjust policies based on the assessment results, making it difficult to ensure that data utility is maximized while meeting security thresholds.

[0005] Therefore, how to construct a compliance dataset construction mechanism that can efficiently transform legal provisions into executable compliance strategies, support manual-assisted parsing and rule accumulation, and conduct risk assessment and feedback adjustment for the relabeling of anonymized datasets is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for constructing compliant datasets for vertical AI models, which can effectively solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for constructing compliant datasets for vertical AI models includes the following steps: Step S1: The legal provisions and industry regulatory documents are parsed by the legal parsing engine. The pre-configured semantic rule template library is called and the parsing results are matched with the trigger condition patterns in the template library to generate Boolean logic expressions that represent legal constraints. Based on the Boolean logic expressions, the compliance feature tags are divided into different risk levels, and a strategy mapping association table is established between the compliance feature tags of each risk level and the de-identification operators in the preset operator library. Step S2: Call the pre-trained vertical domain named entity recognition model to perform semantic scanning on the original dataset to identify the compliance feature labels corresponding to sensitive entities; the model integrates the predefined topological relationships between entities in the vertical domain knowledge graph to identify the indirect sensitive risks hidden in the text based on the topological relationships and label them as the corresponding compliance feature labels. Step S3: Based on the compliance feature label query strategy mapping association table output in step S2, dynamically load the corresponding de-identification operator from the preset operator library, and perform differentiated de-identification processing on the corresponding fields in the original dataset. Step S4: Perform re-identification risk detection on the dataset processed in step S3 to calculate the maximum re-identification risk value. When the maximum re-identification risk value exceeds the preset compliance risk threshold, trigger feedback adjustment to upgrade the processing intensity of the corresponding de-identification operator and re-execute step S3 until the maximum re-identification risk value meets the preset requirements.

[0008] Furthermore, the process in step S1 where the legal parsing engine transforms the legal text into a Boolean logic expression specifically includes: The pre-built legal terminology ontology library is invoked to perform word segmentation and part-of-speech tagging on the semantic units of the pre-processed legal text; Dependency parsing and semantic role labeling are performed on semantic units that have undergone word segmentation and part-of-speech tagging to identify the actors, actions, and constraints acting on those actions in legal provisions. The pre-configured semantic rule template library is invoked, and the results of dependency parsing and semantic role labeling are matched with the triggering condition patterns in the semantic rule template library. When a match is successful, the corresponding logic generation template is extracted, and the parameter values ​​extracted from the legal text are filled into the placeholders of the logic generation template to generate a Boolean logic expression representing the legal constraint conditions. The semantic rule template library defines several conversion rules that map legal language patterns to program logic fragments. These conversion rules include at least: rules for mapping words representing numerical comparisons to corresponding numerical comparison operators; rules for mapping words representing legal obligations or rights to constraints in logical flows; and rules for mapping words representing logical connections to logical operations.

[0009] Furthermore, in step S1, compliance feature labels are divided into different risk levels, and a strategy mapping association table is established between compliance feature labels of each risk level and de-identification operators in the preset operator library, specifically including: Compliance feature labels are divided into three risk levels: direct identifiers, indirect identifiers, and quasi-identifiers. Establish a mapping association between the direct identifier and the masking operator to perform character-level masking processing on the direct identifier; Establish a mapping association between the indirect identifier and the generalization operator or perturbation operator to perform data resolution reduction or random noise injection on the indirect identifier; A mapping relationship is established between the quasi-identifier and the permutation operator to perform random value exchange processing on the quasi-identifier while maintaining the overall statistical distribution characteristics of the data.

[0010] Furthermore, in step S2, the vertical domain named entity recognition model integrates predefined topological relationships between entities from the vertical domain knowledge graph, specifically including: By using knowledge fusion technology, entity embedding vectors in the vertical domain knowledge graph are fused with context feature vectors output by the bidirectional long short-term memory network layer that constitutes the vertical domain named entity recognition model, so that the model has domain prior knowledge. During the identification process, when the first entity is identified, based on the predefined topological relationships between entities in the vertical domain knowledge graph, the second entity that is related to the first entity is identified and labeled as an indirect identifier.

[0011] Furthermore, the differential desensitization process in step S3 also includes: After the field-by-field desensitization operator is called, a K-anonymization constraint is applied to the entire dataset. The K-anonymization constraint includes: scanning the equivalence classes composed of all quasi-identifier attributes, checking the number of records contained in each equivalence class, and if the number of records in an equivalence class is less than a pre-set positive integer threshold K, then further generalization is performed on the quasi-identifier attributes involved in that equivalence class until the value of any record in the dataset on the quasi-identifier attribute is exactly the same as that of at least K-1 other records.

[0012] Further, step S4 calculates the maximum reidentification risk value of the dataset, specifically including: Let E be the set of all equivalence classes formed by the quasi-identifier attributes in the dataset. For each equivalence class e in set E, obtain the total number of records |e| contained in that equivalence class e. Maximum relabeling risk value The calculation formula is: ,in, denoted as the maximum reidentification risk value, max indicates the maximum value operation, e represents a single equivalence class formed after the dataset is partitioned by the quasi-identifier, E represents the set of all equivalence classes, |e| represents the number of records contained in equivalence class e, and 1 / |e| represents the upper limit of the probability that any record in a single equivalence class is uniquely identified.

[0013] Furthermore, step S4 triggers feedback adjustment to upgrade the processing intensity of the corresponding desensitization operator, specifically including: The feature tag combination that causes the maximum re-identification risk value to exceed the preset compliance risk threshold is extracted and transmitted as a feedback signal to step S1 to trigger an upgrade in the processing intensity of the corresponding de-identification operator in the policy mapping association table; the processing intensity upgrade includes: Increase the aggregation level of the generalization operator by 1 to 2 preset levels; Alternatively, the number of masking characters in the masking operator can be increased by a predetermined number of bits; Alternatively, enable the suppression operator on the relevant fields and delete them directly.

[0014] Furthermore, the method also includes: After the dataset processed in step S4 passes the compliance verification, a structured audit report is generated. The audit report records at least the following: the original data source identifier, the version number of the legal provisions on which this processing is based, the parameter version of the named entity recognition model used in step S2, the list of operators actually called in step S3 and their execution parameters, and the calculated risk assessment values. Meanwhile, a source fingerprint is generated for the processed dataset using a preset algorithm. The source fingerprint binds the hash value of the audit report content to the metadata of the dataset using encrypted hash chain technology, or embeds the core features of the processing process into the non-significant numerical bits of the dataset using robust digital watermarking technology, so as to ensure that the compliant processing status of the dataset is traceable.

[0015] Furthermore, the method is implemented through a distributed computing architecture, specifically including: The system is deployed using a master-slave architecture. The master node is responsible for receiving the original large-scale dataset and logically dividing the dataset into multiple independent data shards according to a preset data sharding strategy. The master node uses a distributed task distribution framework to schedule each data shard to multiple computing nodes. After receiving the fragmented data, each computing node performs feature extraction in step S2 and operator scheduling and cleaning in step S3 in parallel on its local machine, and after processing, it sends the desensitized data fragments back to the master node. The master node performs data merging and uniformly executes the compliance verification in step S4.

[0016] A compliant dataset construction system for vertical AI models, including The compliance strategy mapping model building module is used to execute step S1 in the compliance dataset building method for vertical AI models to build a compliance strategy mapping model containing a strategy mapping association table. The feature extraction and recognition module is used to perform step S2 in the compliant dataset construction method for vertical AI models, in order to call a pre-trained vertical domain named entity recognition model to perform semantic scanning on the original dataset; The dynamic operator scheduling and cleaning module is used to execute step S3 in the compliant dataset construction method for vertical AI models, so as to dynamically call the corresponding desensitization operators to perform differentiated desensitization processing on the original dataset based on the sensitive entity identification results; The compliance verification and feedback adjustment module is used to execute step S4 in the compliance dataset construction method for vertical AI models, to perform re-identification risk detection and audit on the processed dataset, and to trigger the feedback adjustment process to dynamically optimize the de-identification strategy.

[0017] In summary, this application includes at least one of the following beneficial technical effects: 1. This invention separates the legal text parsing process from the underlying data cleaning technology by constructing a compliance strategy mapping model. When the external regulatory environment or laws and regulations change, the system only needs to update the strategy mapping table through the legal parsing engine, without rewriting the underlying logic code for data cleaning. This architecture significantly improves the system's responsiveness to changes in compliance policies, enabling the dataset construction process to adapt to frequently updated industry standards in real time, and greatly reducing the manpower and time costs of compliance maintenance.

[0018] 2. Compared to traditional hard-coded regular expression matching or simple keyword filtering, this invention utilizes a pre-trained vertical domain named entity recognition model, particularly by introducing a deep learning architecture, to accurately capture specialized entities with complex contextual semantics. By integrating vertical domain knowledge graphs, this invention further achieves deep perception of implicit logical association risks, enabling it to identify and block privacy leakage paths inferred through multi-dimensional feature cross-referencing, ensuring the comprehensiveness and accuracy of sensitive information identification.

[0019] 3. This invention introduces a quantitative re-identification risk assessment mechanism through simulated attacks. This design not only provides measurable security indicators but also enables automated feedback adjustment from verification results to policy configuration. By dynamically adjusting the operator scheduling logic, the system can find the optimal balance between compliance requirements and data research value. While ensuring that the re-identification risk is below a set threshold, it maximizes the preservation of the statistical characteristics and semantic integrity of the dataset, providing high-quality and compliant data asset support for the development of AI models in vertical domains.

[0020] 4. By generating audit reports with traceability fingerprints, this invention achieves end-to-end traceability from the original legal basis to the final processing result, meeting stringent data security regulatory requirements. The introduction of a distributed computing architecture ensures high performance in large-scale dataset construction scenarios, providing a solid technical guarantee for the rapid deployment of industry-level artificial intelligence applications. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall technical solution for constructing compliant datasets for vertical AI models; Figure 2 This is a schematic diagram illustrating the core principles of the compliance policy mapping model and dynamic operator scheduling; Figure 3 This is a flowchart illustrating the logic of legal analysis engine execution and compliance feature tag conversion; Figure 4 This is a schematic diagram of a named entity recognition model that integrates a vertical domain knowledge graph; Figure 5 It is a diagram illustrating the multi-level interaction and data flow between compliance verification and policy feedback adjustment. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the following description is provided in conjunction with the appendix. Figure 1 To be continued Figure 5 The present invention will be further described in detail below with reference to specific embodiments.

[0023] Firstly, the method for constructing a compliant dataset for vertical AI models disclosed in this application is implemented according to the following steps: The first step, S1, involves constructing a compliance strategy mapping model. This step transforms external legal provisions and industry regulatory standards into structured compliance strategies that the system can call upon. Due to the inherent difficulty in semantic parsing of natural language legal texts, this step employs a semi-automated assisted parsing scheme to reduce the requirements for fully automated understanding capabilities, while ensuring the reliability of the parsing results through manual review and data recording. The specific implementation process is divided into the following sub-steps.

[0024] Step S101: Access and preprocess the original legal text source. The system receives external legal provisions and industry regulatory documents through a standardized interface, such as Article 28 of the Personal Information Protection Law regarding the processing of sensitive personal information, or specific provisions in the Guidelines for Health and Medical Data Security. File formats supported include plain text, PDF, and web pages. The system has a built-in legal parsing engine that first performs format cleaning and text structure recognition on the text, removing noise information such as headers, footers, and footer numbers, and dividing it into semantic units by clause or paragraph. For clauses that are difficult to parse, such as those containing multiple nested conditions or ambiguous expressions, the system supports manual intervention for correction and supplementation.

[0025] Step S102 involves calling a legal terminology ontology for word segmentation and part-of-speech tagging. This step uses a legal terminology ontology built for a vertical domain. This library uses a five-tuple structure to store each legal term, where each tuple includes the term's concept, attribute characteristics, relation type, constraint condition, and hierarchical position. For example, the term "must not" is tagged as a legal adverb with a semantic type of prohibitive obligation, and is associated with the inverse term "shall." The library is constructed as follows: First, high-frequency professional terms are statistically analyzed from publicly available legal texts, covering entity words such as "patient," action words such as "handle," and constraint words such as "shall." Then, the statistical results are merged with an existing structured legal dictionary to remove duplicates. Finally, the database is reviewed and confirmed by domain experts, and incremental updates are supported. Taking the medical data compliance field as an example, the library contains approximately 500 core terms, with a tested word segmentation accuracy of 87.3%. During word segmentation, terms in the library are matched first; if no match is found, the process reverts to a general word segmentation tool such as jieba. Part-of-speech tagging uses a legal domain-specific tag set, such as... Legal terms, This indicates a legal adverb. The construction method references existing technologies such as the legal ontology knowledge base construction approach disclosed in CN107133283A, but optimizes it for vertical domain adaptation and incremental updates.

[0026] Step S103 involves performing dependency parsing and core element extraction. Based on keyword matching, the engine performs dependency parsing on the text to identify the subject, action, and constraints. Considering that the accuracy of fully automated dependency analysis on Chinese legal texts is approximately 78% in actual tests, and based on test results from a corpus of 500 medical legal sentences, the system allows legal experts to review and correct the analysis results. For example, for the text "When processing the personal information of minors under the age of fourteen, the consent of their guardians must be obtained," the engine can identify "under the age of fourteen" as an age constraint and associate it with the entity "minor." For texts that cannot be accurately parsed, accounting for approximately 22%, the system records the original text and forwards it to manual processing.

[0027] Step S104: Perform compliance logic transformation based on the semantic rule template library. This step calls a pre-configured semantic rule template library, which contains approximately 120 transformation rules, covering common scenarios such as numerical comparison (e.g., "dissatisfied" mapped to "<"), obligation judgment (e.g., "shall" mapped to a necessary condition), and logical connection (e.g., "and" mapped to "AND"). The rule library is built by continuously accumulating manually annotated legal clauses. Each rule contains triggering keywords, dependent syntactic patterns, and corresponding logical fragment templates. The parsing process is as follows: The engine matches the dependency parsing results with the triggering condition patterns in the template library. If a match is successful, the corresponding template is extracted, and the parameter values ​​extracted from the text, such as the age value "14" and the field name "guardian consent", are filled into placeholders to generate a Boolean logical expression, for example: if (age<14 and guardian_consent ==false) then reject. Of the 200 legal clauses tested, this step correctly generated logical expressions in 72% of cases. The remaining 28% of clauses, such as those containing indirect references or nested conditions, required manual assistance in generating expressions. The system records the manual corrections and feeds them back to the template library as new training data to achieve gradual expansion of the rule base.

[0028] Through the aforementioned semi-automated legal analysis process, the system transforms legal provisions into actionable compliance strategies. Simultaneously, the system records the original clauses analyzed, the rule templates used, the generated expressions, and any manual corrections throughout the process, providing data support for subsequent audits and rule optimization. In a pilot project for medical data compliance, this system processed 320 clauses from five regulations, including the Personal Information Protection Law and the Guidelines for Health and Medical Data Security. It successfully constructed a strategy mapping table containing 156 compliance feature tags. Independently verified by three legal experts, the overall accuracy rate of the strategy mapping was 89.6%, meeting the practical requirements of the project.

[0029] Step S105: Define and categorize compliance feature tags. After completing the legal logic transformation, the system needs to clarify which specific fields in the dataset are subject to the aforementioned logical constraints. Therefore, the system systematically divides compliance feature tags into three risk level categories.

[0030] The first category is direct identifiers, which refer to features that can uniquely and directly determine the identity of a natural person. In engineering implementation, these features explicitly include name, citizen identification number, telephone number, biometric information, and residential address down to the house number.

[0031] The second category is indirect identifiers, which refer to features that, when combined with specific background knowledge or external databases, have a high probability of inferring an individual's identity. These features are usually highly scarce within a vertical field, such as the name of a rare disease, a specific medical appointment time accurate to the minute, a job description extremely rare in a particular region, or a record of a huge transaction with a significantly deviated amount from the average.

[0032] The third category is quasi-identifiers, which refer to features that cannot be located to a specific individual under a single dimension, but may lead to the risk of re-identification when correlated with data sources from multiple dimensions. Common quasi-identifiers include gender, age, ethnicity, administrative division, and educational background.

[0033] Step S106: Establish a strategy mapping association table between feature labels and data processing operators, and clarify the collaboration mechanism between the permutation operator and K-anonymization. The system has a built-in processing operator library, which encapsulates various callable algorithm units that perform specific de-identification operations. This operator library specifically includes the following 5 types of operators.

[0034] Masking operators are used to partially mask characters, supporting both fixed-length and variable-length masking modes. A typical execution logic is to replace several consecutive characters in the middle of the sensitive string with preset placeholders, such as processing the name "Zhang Wei" as "Zhang*".

[0035] Generalization operators are used to perform hierarchical upward aggregation on numerical values ​​or timestamps, essentially reducing the resolution of data in a specific dimension. For example, a specific birth date "May 20, 1990" can be generalized to an age range "30 to 35 years old".

[0036] Suppression operators are used to perform full column deletion or complete nulling on specific high-risk fields. They are suitable for extremely sensitive fields where the risk of re-identification cannot be reduced to an acceptable level even after de-identification.

[0037] Perturbation operators are used to inject random noise with a preset distribution into raw numerical data. A common implementation is to add differential noise that conforms to a Laplace distribution to statistical values, thereby protecting individual privacy when the data is published.

[0038] Permutation operators are used to randomly swap the values ​​of records in the same equivalence class on a specific field while keeping the overall statistical distribution characteristics of the data unchanged, so as to sever the direct correspondence between an individual and its quasi-identifier attribute value.

[0039] When constructing the policy mapping table, the system establishes binding relationships based on the following association rules: Direct identifiers are strongly mapped to masking operators to ensure that any information directly pointing to an individual's identity is physically masked in the dataset. Indirect identifiers are mapped to generalization or perturbation operators to reduce the risk of inference based on background knowledge by obfuscating or adding noise to the data. Quasi-identifiers are mapped to multidimensional random permutation operators to increase the difficulty for attackers to re-identify specific individuals through multidimensional association analysis.

[0040] The permutation operator and the subsequent K-anonymization mechanism collaborate as follows: In the field-by-field processing stage of step S3, the permutation operator performs random row-level permutations within each quasi-identifier attribute, effectively severing the direct mapping chain between individual records and their original quasi-identifier values. Subsequently, in the K-anonymization scanning stage of step S303, the system reconstructs equivalence classes based on the permuted quasi-identifier attribute values ​​and counts the number of records in each equivalence class. Since the permutation operation has disrupted the original correspondence between records and attribute values, equivalence classes that might otherwise fail to meet the K-anonymization requirement due to insufficient record count will have their statistical distribution changed after the permutation, thereby reducing the further generalization strength required for subsequent K-anonymization. Conversely, if the number of records in an equivalence class is still below the threshold K after the permutation, the K-anonymization mechanism will trigger additional generalization processing for the quasi-identifiers involved in that equivalence class. The two mechanisms form a collaborative chain of "scrambling first, then aggregating": the permutation operator reduces individual distinguishability, creating a more uniform equivalence class distribution for K-anonymization; K-anonymization, on the other hand, applies a population constraint based on the permutation, ultimately ensuring that each record in the dataset is indistinguishable from at least K-1 other records in terms of quasi-identifiers. This collaborative mechanism avoids the problems of redundant computation or excessive anonymization that may occur when the two are used independently, achieving a joint optimization of security and data utility.

[0041] The final generated strategy mapping association table is organized serially using key-value pairs or document-based data structures.

[0042] Step S107: Store the mapping association table and establish a dynamic update mechanism. The strategy mapping association table is stored in the form of a relational database table, as shown in the example below:

[0043] When executing step S3, the system queries the table based on the identified tags and dynamically loads the corresponding operators. The system has a built-in monitoring module that periodically polls the legal publishing platform. When changes to legal text are detected, the system wakes up the legal parsing engine to perform incremental parsing and writes the newly added or revised mapping entries into the database in real time, enabling dynamic updates of the strategy without downtime.

[0044] In summary, step S1 constructs a compliance strategy mapping model that dynamically couples legal provisions with data cleaning techniques. The structured strategy mapping table output by this model provides a directly usable decision-making basis for subsequent steps S2 (sensitive entity identification) and S3 (operator scheduling and data cleaning).

[0045] The next step is S2, feature extraction and recognition. This step calls a pre-trained vertical domain named entity recognition model to perform semantic scanning on the original dataset to accurately locate sensitive entities. The implementation process is as follows.

[0046] Step S201, Model Architecture Overview: The named entity recognition model used in this step is based on a deep learning architecture that combines bidirectional long short-term memory networks with conditional random fields.

[0047] After the input text is vectorized by the embedding layer, semantic features incorporating contextual information are extracted by the bidirectional long short-term memory network layer; the conditional random field layer then learns the transition constraints between labels to output the globally optimal entity label sequence. This architecture is a well-known technique in the field of sequence labeling, and this paper builds upon it with the following specific designs for compliance scenarios in vertical domains.

[0048] Step S202 involves constructing training data for compliance scenarios. To enable the model to identify sensitive entities in a vertical domain, it needs to be trained using domain-specific labeled data. The training data is constructed as follows: First, de-identified text corpora are collected from vertical business systems such as healthcare and finance. Then, annotation personnel use the BIO annotation method to perform sequence annotation on the text according to predefined sensitive entity type specifications.

[0049] The annotation specifications clearly define entity categories such as direct identifiers, indirect identifiers, and quasi-identifiers. For example, citizen identification numbers and names are labeled as direct identifiers; rare disease names and specific transaction amounts are labeled as indirect identifiers. The annotated data is then divided proportionally into training, validation, and test sets.

[0050] Step S203, Model Training Objective and Loss Function: The objective of model training is to minimize the difference between the model-predicted label sequence and the manually labeled real sequence by optimizing the network weights. To achieve this objective, the negative log-likelihood of the conditional random field layer is used as the loss function, and its expression is:

[0051] Where: L is the loss value; N is the number of samples in the batch; The feature vector sequence of the i-th input text after being encoded by a bidirectional long short-term memory network; Let i be the true label sequence of the i-th sample; For a conditional random field layer under a given condition Time prediction The probability of this is determined. By minimizing this loss function through backpropagation, the model gradually learns the semantic and label mapping rules of the vertical domain.

[0052] Step S204: Integrating implicit risk perception with knowledge graphs. Unlike general named entity recognition models, this invention introduces vertical domain knowledge graphs in the application stage to enhance the model's implicit risk identification capability. This knowledge graph predefines the topological relationships between entities within a domain, such as the "disease-symptom-drug-department" association network in the medical field.

[0053] The model uses knowledge fusion technology to fuse entity embedding vectors from a knowledge graph with contextual feature vectors output by a bidirectional long short-term memory network. This fusion operation enables the model to possess domain prior knowledge.

[0054] In actual identification, when the text contains the name of a specific treatment drug for a rare disease, even if the disease name is not mentioned, the model can identify the associated drug information as a second-class indirect identifier label and label it based on the drug-indication relationship path in the knowledge graph, thereby blocking the privacy leakage path of inferring the disease from the medication information.

[0055] In summary, through the above design, step S2 completes the deep semantic recognition and annotation of sensitive information in the original data. The recognition result will be passed to step S3 as the basis for triggering subsequent differentiated de-identification operations.

[0056] In the above method, regarding step S3, dynamic operator scheduling and cleaning are performed. Step S3, based on the sensitive entity identification results output in step S2 and combined with the policy mapping table established in step S1, dynamically calls the corresponding desensitization operators to perform differential cleaning on the original dataset. The implementation process is as follows.

[0057] Step S301: Matching and operator loading by the scheduling engine. The scheduling engine internally includes a feature matching module and an operator loading module. The feature matching module receives the sensitive entity tags passed in step S2 and performs a high-speed comparison with the policy mapping table generated and stored in the distributed database in step S1. Once a tag matches a preset entry in the mapping table, the scheduling engine reads the operator number corresponding to that entry.

[0058] The operator loading module dynamically loads the corresponding algorithm implementation from a predefined operator library based on the operator number. Each operator in the operator library exists as an independent script or dynamic link library, supporting hot-plugin mode. When the de-identification algorithm needs to be updated or added, only the new version of the operator needs to be deployed to the operator library, and the number in the policy mapping table needs to be updated, without stopping the system service.

[0059] Step S302, Classification and execution of desensitization operators: The system calls the corresponding operators for different categories of compliance feature tags. The specific execution logic is as follows.

[0060] For direct identifiers, the masking operator is invoked to perform character-level masking. The masking operator replaces several consecutive characters in the middle of the sensitive string with preset placeholders. For example, it replaces the first character of the name field with a placeholder, and replaces several characters in the middle of the ID number with a fixed mask.

[0061] For indirect identifiers, either the generalization operator or the perturbation operator is invoked. The generalization operator performs hierarchical upward aggregation on numerical or time fields to reduce data resolution, such as generalizing the exact time of a medical visit to the corresponding quarterly interval, or converting a specific age into an age range. The perturbation operator injects random noise with a pre-defined distribution into the numerical data.

[0062] For the quasi-identifier, the permutation operator is invoked. This operator, while maintaining the overall statistical distribution characteristics of the data, randomly swaps the values ​​of records in a specific field that are located in the same equivalence class, thereby severing the direct correspondence between the individual and its attribute values.

[0063] Step S303: Global Anonymization Constraint Processing. After the field-by-field operator processing is completed, the system applies K-anonymization constraints to the entire dataset. The system scans the equivalence classes composed of all quasi-identifier attributes and checks the number of records contained in each equivalence class. If the number of records in an equivalence class is less than a pre-set positive integer threshold K, further generalization is performed on the quasi-identifier attributes involved in that equivalence class until the value of any record in the dataset on the quasi-identifier attribute is exactly the same as at least K minus 1 other records. The specific value of K is pre-configured according to compliance requirements.

[0064] Optionally, in addition to satisfying K-anonymization, the system can further perform L-diversity processing. This processing ensures that within each equivalence class, the sensitive attribute has at least L distinct values, thereby preventing attackers from inferring the sensitive attribute values ​​of individuals with high confidence through homogeneity attacks.

[0065] In summary, through the above steps, step S3 completes the automated process from sensitive label identification to differentiated de-identification. The processed dataset will then be transferred to step S4 for compliance verification and re-identification risk assessment.

[0066] In the above method, the final step is compliance verification (S4). Step S4 performs re-identification risk detection and compliance audit on the dataset processed in step S3, and dynamically optimizes the de-identification strategy through a feedback mechanism. Its implementation process is divided into the following sub-steps.

[0067] Step S401: Simulate a re-identification risk attack. The system has a built-in verification module, which is configured with a known background knowledge base and a re-identification attack algorithm library. The background knowledge base stores auxiliary information that attackers may obtain from public channels, such as population distribution in a specific region, frequency tables of common surnames, or publicly available disease incidence data. The re-identification attack algorithm library includes various simulated attack logics such as record linking attacks and attribute inference attacks.

[0068] The verification module uses the dataset output in step S3 as the attack target and launches a simulated attack on the dataset using the aforementioned background knowledge and attack algorithm. Specifically, the verification module attempts to match the quasi-identifier combinations in the dataset with external data sources in the background knowledge base, calculating the proportion of records successfully associated with a unique natural person. This process is repeated multiple times, each time using different background knowledge sampling or attack parameters to obtain stable risk assessment results.

[0069] Step S402: Calculate the maximum re-identification risk value. Based on the simulated attack results, the verification module calculates the key security indicators of the dataset. The maximum re-identification risk value reflects the upper limit of the privacy leakage risk faced by the most vulnerable individuals in the dataset.

[0070] The calculation process is as follows: Let set E represent all equivalence classes formed by partitioning the dataset using the quasi-identifier attribute. For each equivalence class e in set E, obtain the total number of records contained in that equivalence class. Maximum re-identification risk value The maximum value of the reciprocals of the record counts of all equivalence classes is calculated using the following formula:

[0071] in: represents the maximum re-identification risk value; the smaller the value, the stronger the overall resistance to re-identification of the dataset. max indicates the maximum value operation; e represents a single equivalence class formed after the dataset is partitioned by the quasi-identifier. E represents the set of all equivalence classes; This indicates the number of records contained in equivalence class e; This represents the upper probability that any record within a single equivalence class can be uniquely identified.

[0072] The verification module will calculate the results. Compare with a preset compliance risk threshold. This compliance risk threshold is preset by the system administrator according to regulatory requirements, for example, set to 0.01. If the threshold is exceeded, the current dataset is deemed to have insufficient privacy protection, and the verification fails.

[0073] Step S403: Calculate the information loss degree. While assessing security, the verification module also calculates the information loss degree index to measure the extent to which the anonymization process damages the research value of the data. The information loss degree is calculated by comparing the distribution differences of key statistical features of the dataset before and after anonymization.

[0074] Specific metrics that can be used include the mean shift and variance change rate for numerical attributes, or the reduction in mutual information for categorical attributes. For example, for an age field, calculate the relative deviation between the mean of the original age column before anonymization and the mean of the median column of the age range after anonymization; for a disease diagnosis field, calculate the proportion of change in standardized mutual information between this field and the symptom field before and after anonymization. The lower the information loss value, the higher the degree to which the anonymization process preserves the inherent patterns and analytical value of the data.

[0075] Step S404, Feedback Adjustment and Strategy Upgrade: When step S402 determines that the verification fails, the system automatically triggers the feedback adjustment process. The verification module extracts the specific feature label combinations that lead to excessive risk, such as an equivalence class consisting of gender + age + rare disease name with too few records, and transmits this combination information back to step S1 as a feedback signal.

[0076] After receiving the feedback signal, the policy mapping model in step S1 automatically upgrades the strength of the operator corresponding to the feature label combination. The upgrade measures include: increasing the aggregation level of the generalization operator by 1 to 2 preset levels, for example, adjusting the age range from 5 years old to 10 years old; or increasing the number of masking characters in the masking operator by a predetermined number; or enabling the suppression operator to directly delete the relevant fields.

[0077] After the strategy upgrade, the system re-executes the operator scheduling and cleaning process in step S3, and verifies the newly generated dataset again using steps S401 to S403. This iterative process continues until... The risk level is reduced below the compliance risk threshold, while achieving a predetermined balance between information loss and security.

[0078] Step S405: Generate an audit report with traceability fingerprints.

[0079] After the dataset passes compliance verification, the system generates a structured audit report. The report is stored in a machine-readable format such as JSON or XML, and records the following: the original data source identifier, the version number of the legal provisions on which this processing is based, the parameter version of the named entity recognition model used in step S2, the list of operators actually called in step S3 and their execution parameters, and the various risk assessment values ​​calculated in steps S402 and S403.

[0080] Simultaneously, the system generates a traceability fingerprint for the dataset using a preset algorithm. One implementation method is to use encrypted hash chain technology to bind the hash value of the aforementioned audit report content to the dataset's metadata; another method is to use robust digital watermarking technology to encode the core features of the processing flow and embed them into the non-significant numerical bits of the dataset. This traceability fingerprint ensures that the compliant processing status of the dataset can be verified and traced by a third party at any time when it is subsequently distributed, transferred, or used for model training, meeting the data security regulatory requirements for end-to-end auditability.

[0081] Step S4 completes the verification and dynamic optimization of the anonymized dataset. This mechanism integrates security risk assessment, data utility measurement, and policy feedback adjustment into an automated verification and dynamic optimization process, enabling the system to retain the statistical characteristics and business analysis value of the dataset to the maximum extent while ensuring compliance. The verification results of step S4 provide the final quality assurance for the entire compliant dataset construction process, and the processed dataset can be directly used for training and inference of vertical domain artificial intelligence models.

[0082] The method of this invention is implemented through a distributed computing architecture. Specifically, the system is deployed using a master-slave architecture. The master node is responsible for receiving the original large-scale dataset and logically dividing the dataset into multiple independent data shards according to a preset data sharding strategy—such as sharding by the range of the number of records or sharding by hash modulo. The master node schedules each shard to multiple computing nodes through a distributed task distribution framework, which can implement task distribution and status tracking based on message queues or remote procedure call protocols.

[0083] After receiving the fragmented data, each computing node performs feature extraction in step S2 and operator scheduling and cleaning in step S3 locally in parallel. After each node completes its processing, it sends the de-identified data fragments back to the master node, which then performs data merging and performs compliance verification in step S4.

[0084] The data transmission channels between each compute node and the master node are encrypted and protected using secure data transmission protocols, such as TLS 1.3, to ensure that the original data and intermediate processing results are not stolen or tampered with in the distributed network environment.

[0085] On the other hand, the system for building compliant datasets for vertical AI models disclosed in this application includes: The compliance strategy mapping model building module is used to execute step S1 in the compliance dataset building method for vertical AI models to build a compliance strategy mapping model containing a strategy mapping association table. The feature extraction and recognition module is used to perform step S2 in the compliant dataset construction method for vertical AI models, in order to call a pre-trained vertical domain named entity recognition model to perform semantic scanning on the original dataset; The dynamic operator scheduling and cleaning module is used to execute step S3 in the compliant dataset construction method for vertical AI models, so as to dynamically call the corresponding desensitization operators to perform differentiated desensitization processing on the original dataset based on the sensitive entity identification results; The compliance verification and feedback adjustment module is used to execute step S4 in the compliance dataset construction method for vertical AI models, to perform re-identification risk detection and audit on the processed dataset, and to trigger the feedback adjustment process to dynamically optimize the de-identification strategy.

[0086] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0087] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for constructing a compliant dataset for vertical AI models, characterized in that, Includes the following steps: Step S1: The legal provisions and industry regulatory documents are parsed by the legal parsing engine. The pre-configured semantic rule template library is called and the parsing results are matched with the trigger condition patterns in the template library to generate Boolean logic expressions that represent legal constraints. Based on the Boolean logic expressions, the compliance feature tags are divided into different risk levels, and a strategy mapping association table is established between the compliance feature tags of each risk level and the de-identification operators in the preset operator library. Step S1 involves classifying compliance feature tags into different risk levels and establishing a strategy mapping association table between compliance feature tags for each risk level and de-identification operators in the preset operator library. Specifically, this includes: Compliance feature labels are divided into three risk levels: direct identifiers, indirect identifiers, and quasi-identifiers. Establish a mapping association between the direct identifier and the masking operator to perform character-level masking processing on the direct identifier; Establish a mapping association between the indirect identifier and the generalization operator or perturbation operator to perform data resolution reduction or random noise injection on the indirect identifier; Establish a mapping relationship between the quasi-identifier and the permutation operator so as to perform random value exchange processing on the quasi-identifier while maintaining the overall statistical distribution characteristics of the data; Step S2: Call the pre-trained vertical domain named entity recognition model to perform semantic scanning on the original dataset to identify the compliance feature labels corresponding to sensitive entities; the model integrates the predefined topological relationships between entities in the vertical domain knowledge graph to identify the indirect sensitive risks hidden in the text based on the topological relationships and label them as the corresponding compliance feature labels. Step S3: Based on the compliance feature label query strategy mapping association table output in step S2, dynamically load the corresponding de-identification operator from the preset operator library, and perform differentiated de-identification processing on the corresponding fields in the original dataset. Step S4: Perform re-identification risk detection on the dataset processed in step S3 to calculate the maximum re-identification risk value. When the maximum re-identification risk value exceeds the preset compliance risk threshold, trigger feedback adjustment to upgrade the processing intensity of the corresponding de-identification operator and re-execute step S3 until the maximum re-identification risk value meets the preset requirements. The maximum reidentification risk value is calculated. Based on the simulated attack results, the verification module calculates the key security indicators of the dataset. The maximum reidentification risk value reflects the upper limit of privacy leakage risk faced by the most vulnerable individuals in the dataset.

2. The method according to claim 1, characterized in that, Step S1, where the legal parsing engine transforms legal text into Boolean logic expressions, specifically includes: The system invokes a pre-built legal terminology ontology to perform word segmentation and part-of-speech tagging on the semantic units of the preprocessed legal text. The legal terminology ontology uses a five-tuple structured storage, where each five-tuple includes a term concept, attribute features, relation type, constraint condition, and hierarchical position. The library is constructed using a three-step method: domain high-frequency word statistics, merging of structured legal dictionaries, and expert review, and supports incremental updates. Dependency parsing and semantic role labeling are performed on semantic units that have undergone word segmentation and part-of-speech tagging to identify the actors, actions, and constraints acting on those actions in legal provisions. The pre-configured semantic rule template library is invoked, and the results of dependency parsing and semantic role labeling are matched with the triggering condition patterns in the semantic rule template library. When a match is successful, the corresponding logic generation template is extracted, and the parameter values ​​extracted from the legal text are filled into the placeholders of the logic generation template to generate a Boolean logic expression representing the legal constraint conditions. The semantic rule template library defines several conversion rules that map legal language patterns to program logic fragments. These conversion rules include at least: rules for mapping words representing numerical comparisons to corresponding numerical comparison operators; rules for mapping words representing legal obligations or rights to constraints in logical flows; and rules for mapping words representing logical connections to logical operations.

3. The method according to claim 1, characterized in that, In step S2, the vertical domain named entity recognition model integrates predefined topological relationships between entities from the vertical domain knowledge graph, specifically including: By using knowledge fusion technology, entity embedding vectors in the vertical domain knowledge graph are fused with context feature vectors output by the bidirectional long short-term memory network layer that constitutes the vertical domain named entity recognition model, so that the model has domain prior knowledge. During the identification process, when the first entity is identified, based on the predefined topological relationships between entities in the vertical domain knowledge graph, the second entity that is related to the first entity is identified and labeled as an indirect identifier.

4. The method according to claim 1, characterized in that, The differential desensitization process in step S3 also includes: After the field-by-field desensitization operator is called, a K-anonymization constraint is applied to the entire dataset. The K-anonymization constraint includes: scanning the equivalence classes composed of all quasi-identifier attributes, checking the number of records contained in each equivalence class, and if the number of records in an equivalence class is less than a pre-set positive integer threshold K, then further generalization is performed on the quasi-identifier attributes involved in that equivalence class until the value of any record in the dataset on the quasi-identifier attribute is exactly the same as that of at least K-1 other records.

5. The method according to claim 1, characterized in that, Step S4 calculates the maximum reidentification risk value of the dataset, specifically including: Let E be the set of all equivalence classes formed by the quasi-identifier attributes in the dataset. For each equivalence class e in set E, obtain the total number of records |e| contained in that equivalence class e. Maximum relabeling risk value The calculation formula is: ,in, denoted as the maximum reidentification risk value, max indicates the maximum value operation, e represents a single equivalence class formed after the dataset is partitioned by the quasi-identifier, E represents the set of all equivalence classes, |e| represents the number of records contained in equivalence class e, and 1 / |e| represents the upper limit of the probability that any record in a single equivalence class is uniquely identified.

6. The method according to claim 5, characterized in that, Step S4 triggers feedback adjustment to upgrade the processing strength of the corresponding desensitization operator, specifically including: The feature tag combination that causes the maximum re-identification risk value to exceed the preset compliance risk threshold is extracted and transmitted as a feedback signal to step S1 to trigger an upgrade in the processing intensity of the corresponding de-identification operator in the policy mapping association table; the processing intensity upgrade includes: Increase the aggregation level of the generalization operator by 1 to 2 preset levels; Alternatively, the number of masking characters in the masking operator can be increased by a predetermined number of bits; Alternatively, enable the suppression operator on the relevant fields and delete them directly.

7. The method according to claim 1, characterized in that, The method also includes: After the dataset processed in step S4 passes the compliance verification, a structured audit report is generated. The audit report records at least the following: the original data source identifier, the version number of the legal provisions on which this processing is based, the parameter version of the named entity recognition model used in step S2, the list of operators actually called in step S3 and their execution parameters, and the calculated risk assessment values. Meanwhile, a source fingerprint is generated for the processed dataset using a preset algorithm. The source fingerprint binds the hash value of the audit report content to the metadata of the dataset using encrypted hash chain technology, or embeds the core features of the processing process into the non-significant numerical bits of the dataset using robust digital watermarking technology, so as to ensure that the compliant processing status of the dataset is traceable.

8. The method according to claim 1, characterized in that, The method is implemented through a distributed computing architecture, specifically including: The system is deployed using a master-slave architecture. The master node is responsible for receiving the original large-scale dataset and logically dividing the dataset into multiple independent data shards according to a preset data sharding strategy. The master node uses a distributed task distribution framework to schedule each data shard to multiple computing nodes. After receiving the fragmented data, each computing node performs feature extraction in step S2 and operator scheduling and cleaning in step S3 in parallel on its local machine, and after processing, it sends the desensitized data fragments back to the master node. The master node performs data merging and uniformly executes the compliance verification in step S4.

9. A compliant dataset construction system for vertical AI models, applied to the method described in any one of claims 1 to 8, characterized in that, include: The compliance strategy mapping model building module is used to execute step S1 to build a compliance strategy mapping model containing a strategy mapping association table. The feature extraction and recognition module is used to perform step S2, which calls a pre-trained vertical domain named entity recognition model to perform semantic scanning on the original dataset; The dynamic operator scheduling and cleaning module is used to execute step S3, so as to dynamically call the corresponding desensitization operator to perform differential desensitization processing on the original dataset according to the sensitive entity identification results; The compliance verification and feedback remediation module is used to execute step S4 to perform reidentification risk detection and audit on the processed dataset, and trigger the feedback remediation process to dynamically optimize the de-identification strategy.

Images