An agent system for open data compliance verification

Abstract

This invention relates to the field of data security and compliance review, and discloses an intelligent agent system for open data compliance verification. The intelligent agent system receives a data catalog to be made publicly available, processes it through a data catalog preprocessing module to generate standardized text, extracts core semantics and key information, and inputs the standardized text along with metadata in parallel to a structured entity sensitivity identification module, an industry policy compliance rule analysis module, and a content semantic compliance risk analysis module. These modules output field-level risk classification results, a compliance assessment report, and risk assessment results, respectively. A comprehensive risk assessment module performs dynamic weight allocation and data analysis based on core semantics and key information to determine the overall risk level of the data table and generate a tiered closed-loop handling plan. The system analyzes the tiered closed-loop handling plan, executes differentiated control operations, and establishes a security control closed loop by combining digital watermarking and traceability modules. This invention improves the accuracy of data compliance verification.

Description

Technical Field

[0001] This invention relates to the field of data security and compliance review, and in particular to an intelligent agent system for verifying the compliance of open data. Background Technology

[0002] With the advancement of data openness and sharing, ensuring the compliance and security of publicly released data has become a crucial aspect of data management. Before data is released externally, it is typically necessary to verify the compliance of data catalogs and tables to prevent the leakage of sensitive data or the risk of unauthorized release. However, existing data compliance verification solutions still have the following technical shortcomings in practical applications:

[0003] Existing verification systems typically employ a single-dimensional review approach, or their review modules operate independently. Faced with entity sensitivity risks, industry policy risks, and semantic compliance risks inherent in the data, current technologies lack effective multi-dimensional result fusion mechanisms, making it difficult to quantify and uniformly summarize discrete risk classification results and text-based assessment reports. Furthermore, existing methods cannot dynamically allocate weights to various risk assessment results based on the specific business domain to which the data table belongs, limiting the accuracy of the system in determining the overall risk level of the data table.

[0004] When conducting semantic review of data content, existing technologies mostly rely on surface-level keyword matching or fixed rule validation. This approach is insufficient to effectively identify implicit risks such as politically sensitive or commercially confidential attributes hidden within data fields. Due to the lack of technical means to conduct in-depth quantitative risk assessment of individual data fields, existing systems cannot calculate a comprehensive semantic risk score for the entire data table, making it difficult to prevent hidden compliance risks in complex text.

[0005] Current data auditing systems often lack differentiated control actions and comprehensive traceability mechanisms after identifying risks. When high-risk data is discovered and blocked, the system can usually only perform simple blocking operations. Due to the lack of effective data traceability and analysis methods, the system cannot further extract the identity and timestamp information of the original data provider after blocking, resulting in the inability to generate accurate information on the responsible party and making it difficult to establish an effective security control loop at the end of the data flow. Summary of the Invention

[0006] To overcome the above shortcomings, this invention provides an intelligent agent system for open data compliance verification, which aims to improve the existing data compliance verification schemes, which mainly suffer from the lack of quantitative integration and dynamic weight allocation of multi-dimensional evaluation results, difficulty in effectively identifying implicit semantic risks, and lack of data traceability and analysis methods, which makes it impossible to establish a closed loop of security control.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: the intelligent agent system for open data compliance verification includes a data catalog preprocessing module, a structured entity sensitive identification module, an industry policy compliance rule analysis module, a content semantic compliance risk analysis module, and a comprehensive risk assessment module; The intelligent agent system for open data compliance verification is configured to receive a data catalog to be made public, wherein the data catalog includes at least one data table, and the data table includes at least one data field. The data catalog preprocessing module is used to process the data catalog to be made public to generate standardized text, extract the core semantics and key information contained in the standardized text, and determine the industry attributes and business areas corresponding to the data table based on the core semantics and key information. The structured entity sensitive identification module, the industry policy compliance rule judgment module, and the content semantic compliance risk judgment module are used to receive the standardized text in parallel, and trigger compliance judgment operations of corresponding dimensions according to the industry attributes and business domains, so as to output field-level risk classification results, compliance assessment reports, and risk judgment results respectively. The comprehensive risk assessment module is used to receive the field-level risk classification results, the compliance assessment report, and the risk judgment results, and to perform weight allocation and data analysis based on the core semantics and key information to determine the overall risk level of the data table, and generate a graded closed-loop handling plan for the data table.

[0008] As a further description of the above technical solution: The data catalog preprocessing module includes a data catalog pre-identification and judgment unit, a structured text fusion and splicing unit, and a deep semantic parsing unit; The data catalog pre-identification and judgment unit is used to filter the data tables and data fields contained in the data catalog to be opened to the public according to the preset data hierarchical classification and label system, and to perform sensitive data identification operation on the data fields to remove the data fields containing any of the features of document codes, communication contact information, biometric features and asset confidentiality key fields; The structured text fusion and splicing unit is used to receive the data table that has been screened, take all the data fields in each data table as the smallest semantic splicing unit, configure a unified and standardized description prefix and attach pre-attribute information containing basic attributes to construct structured text, and perform data cleaning operation on the structured text to output the standardized text. The deep semantic parsing unit is used to extract the core semantics and key information contained in the standardized text using natural language processing technology, and output the classification results for the data table.

[0009] As a further description of the above technical solution: The structured entity sensitive recognition module includes a BERT embedding layer, a BiLSTM layer, and a CRF layer; The BERT embedding layer is used to convert the standardized text into a character sequence, perform dual embedding processing on each character in the character sequence to generate a word embedding vector and a position embedding vector, perform a summation operation on the word embedding vector and the position embedding vector to generate an initial embedding representation, and use the Transformer encoder structure contained in the BERT embedding layer to extract contextual features and output a word vector matrix. The BiLSTM layer is used to receive the word vector matrix. The BiLSTM layer includes a forward long short-term memory network structure and a backward long short-term memory network structure. The forward long short-term memory network structure and the backward long short-term memory network structure process the word vector matrix in parallel from opposite temporal directions, perform concatenation and fusion to generate bidirectional feature vectors, and output a set of sequence feature vectors composed of all the bidirectional feature vectors.

[0010] As a further description of the above technical solution: The CRF layer is used to receive the sequence feature vector set and perform sequence labeling and risk classification decoding. The CRF layer constructs a label transition matrix, and introduces field-level label uniformity constraints into the label transition matrix. By configuring the transition constraint score, it is determined that character sequence segments belonging to the same data field within the standardized text must have the same sensitivity risk classification label. The CRF layer defines a matching score function between the potential label sequence and the sequence feature vector set, calculates the conditional probability based on the matching score function, and uses the Viterbi algorithm to perform optimal path decoding to output the optimal label sequence. The CRF layer aligns and matches the optimal label sequence with the character sequence in the standardized text, and extracts the uniform character labels within the text boundary as the field-level risk classification result corresponding to the data field.

[0011] As a further description of the above technical solution: The industry policy compliance rule analysis module is used to obtain metadata corresponding to the data directory to be opened to the public, business tags corresponding to the data table, and basic attributes corresponding to the data field, generate multi-dimensional search terms, and expand to generate a target search term set; The industry policy compliance rule analysis module performs a web search operation based on the target search term set to obtain authoritative policy standard documents, uses natural language processing technology to perform structured parsing on the authoritative policy standard documents to extract data openness rule elements, and transforms the extracted data openness rule elements into structured compliance entries containing specific clause content; The industry policy compliance rule judgment module constructs a multi-dimensional rule template based on the structured compliance items, including industry matching templates, sensitive data control templates, and open scenario matching templates. It uses a pre-trained large language model text encoder to map the metadata and business tags to the clause content of the structured compliance items into high-dimensional semantic vectors, and calculates the cosine similarity between the two sets of high-dimensional semantic vectors.

[0012] As a further description of the above technical solution: The industry policy compliance rule analysis module is configured to perform a fusion judgment operation to establish a target matching association between the data table and the structured compliance entries, specifically: When the data table and the multi-dimensional rule template are successfully matched, the matching relationship is directly confirmed. When the data table and the multi-dimensional rule template fail to match, the cosine similarity is compared with a preset similarity threshold. When the cosine similarity is greater than or equal to the similarity threshold, the matching relationship is confirmed. The industry policy compliance rule analysis module performs a comprehensive compliance assessment on the data table based on the target matching association, determines the data openness level corresponding to the data table, and generates a compliance assessment report containing the compliance basis and the data openness level.

[0013] As a further description of the above technical solution: The content semantic compliance risk assessment module includes a RoBERTa-large semantic encoding model and a risk mapping unit; The content semantic compliance risk assessment module constructs a tag and risk prefix prompt template, which includes a mask mark. The mask mark is concatenated with the standardized text to generate a prompt concatenation sequence. The RoBERTa-large semantic coding model receives the cue concatenation sequence and superimposes field boundary embedding vectors used to identify boundary positions in the embedding input layer included in the RoBERTa-large semantic coding model. It performs feature encoding on the cue concatenation sequence and outputs a mask embedding vector containing sensitive category mask embedding vectors and risk level mask embedding vectors at the position corresponding to the mask mark.

[0014] As a further description of the above technical solution: The risk mapping unit receives the mask embedding vector and maps the sensitive category mask embedding vector and the risk level mask embedding vector to the corresponding label feature space through the linear mapping layer contained in the risk mapping unit, and performs a nonlinear transformation operation. The risk mapping unit calculates the cosine similarity between the feature vector after the nonlinear transformation operation and the pre-initialized preset label embedding vector, and extracts the preset label corresponding to the highest cosine similarity value to determine the implicit sensitive category and quantified risk level of each data field. The risk mapping unit performs a weighted summation operation on the quantitative risk level of all the data fields based on the set risk weight coefficient, generates a table-level comprehensive semantic risk score, and outputs the risk assessment result containing the quantitative risk level and the table-level comprehensive semantic risk score to the comprehensive risk assessment module.

[0015] As a further description of the above technical solution: The comprehensive risk assessment module includes a compliance rules engine; The comprehensive risk assessment module transforms the discrete field-level risk classification results and the text-formatted compliance assessment report into continuous standardized risk values ​​through a preset risk mapping table. Based on the core semantics and key information, it determines the risk emphasis dimension and performs dynamic weight allocation. Through weighted calculation, it obtains a comprehensive risk score and determines the overall risk level of the data table. The overall risk level includes low risk level, medium risk level, and high risk level. The comprehensive risk assessment module schedules the compliance rule engine to perform matching and comparison based on the core semantics and key information to extract a special rule library, configures the special rule library as verification logic, and performs targeted compliance verification operations on the data table based on the verification logic to output the rule verification results; The comprehensive risk assessment module integrates the core semantics and key information, the overall risk level, and the rule verification results to perform decision analysis and generate the hierarchical closed-loop disposal plan for the data table.

[0016] As a further description of the above technical solution: The intelligent agent system for open data compliance verification also includes a digital watermarking and traceability module and a manual review module; When the overall risk level is low, the comprehensive risk assessment module triggers a fast pass rule chain to schedule the digital watermarking and traceability module. The digital watermarking and traceability module obtains the identity and timestamp information of the original data provider, extracts the metadata corresponding to the data directory to be opened to the public, and the basic attributes corresponding to the data table. It uses an encryption algorithm to fuse the identity of the original data provider, the timestamp information, the metadata, and the basic attributes to generate an encrypted sequence as a watermark identifier. The watermark identifier is then embedded in the data table to generate a digital watermark, and the data table carrying the digital watermark is allowed to pass. When the overall risk level is medium risk, the comprehensive risk assessment module triggers the early warning review rule chain, freezes the permissions of the data table, extracts specific risk points from the field-level risk classification results, the compliance assessment report and the risk judgment results, integrates them to generate a review task and pushes it to the manual review module, and unfreezes or generates targeted rectification suggestions based on the returned review confirmation results. When the overall risk level is high, the comprehensive risk assessment module triggers the mandatory control rule chain, blocks the release process and performs isolation and offline operations, generates a detailed alarm report, and schedules the digital watermarking and traceability module to extract the identity and timestamp information of the original data provider by parsing the attached digital watermark in order to generate the location information of the responsible entity.

[0017] The present invention has the following beneficial effects: 1. This invention utilizes an intelligent agent system for open data compliance verification. Standardized text, along with metadata from the data directory to be publicly released, is input in parallel to a structured entity sensitivity identification module, an industry policy compliance rule assessment module, and a content semantic compliance risk assessment module. This enables multi-dimensional, simultaneous compliance assessment. Subsequently, a comprehensive risk assessment module uses a pre-defined risk mapping table to transform discrete field-level risk grading results and text-formatted compliance assessment reports into continuous, standardized risk values. This quantifies and summarizes the field-level risk grading results, compliance assessment reports, and risk assessment results. By dynamically weighting the summarized field-level risk grading results, compliance assessment reports, and risk assessment results based on risk emphasis dimensions, this mechanism effectively improves the accuracy of determining the overall risk level of the data table.

[0018] 2. The content semantic compliance risk assessment module of this invention constructs a tag and risk prefix prompt template containing mask markers, transforming the document-level semantic review task for standardized text into a joint mask prediction task. Utilizing the RoBERTa-large semantic coding model, mask embedding vectors are output at the positions corresponding to sensitive category mask markers and risk level mask markers. The risk mapping unit extracts the preset tag corresponding to the highest cosine similarity value to determine the implicit sensitive category and quantified risk level of each data field. Based on a set risk weight coefficient, this scheme performs a weighted summation operation on the quantified risk levels of all data fields to generate a table-level comprehensive semantic risk score for the data table, thereby improving the reliability of the system in identifying hidden content semantic compliance risks.

[0019] 3. The present invention provides an intelligent agent system for open data compliance verification that analyzes a hierarchical closed-loop handling scheme and executes differentiated control operations. When the comprehensive risk assessment module determines that the overall risk level of the data table is high-risk, the comprehensive risk assessment module triggers a mandatory control rule chain. The intelligent agent system for open data compliance verification blocks the data table's release process according to the mandatory control rule chain, determines that the data table's review status is not passed, and performs an isolation and offline operation on the data table. On this basis, the digital watermarking and traceability module extracts the digital watermark attached to the data table, executes a data traceability parsing algorithm, extracts the identity and timestamp information of the original data provider by parsing the digital watermark, and generates the location information of the responsible entity based on the identity and timestamp information, thereby establishing a security control closed loop for the data table. Attached Figure Description

[0020] Figure 1 This is a structural block diagram of the intelligent agent system for open data compliance verification according to the present invention; Figure 2 This is a flowchart of the data catalog preprocessing module of the present invention; Figure 3 This is a schematic diagram of the network structure of the structured entity sensitive recognition module of the present invention; Figure 4 This is a flowchart illustrating the judgment process of the industry policy compliance rule judgment module of this invention. Figure 5 This is a processing logic diagram of the semantic compliance risk assessment module for the present invention. Figure 6 This is a flowchart illustrating the hierarchical closed-loop handling process of the comprehensive risk assessment module of this invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Reference Figures 1-6 This invention provides an intelligent agent system for open data compliance verification, including a data catalog preprocessing module, a structured entity sensitive identification module, an industry policy compliance rule analysis module, a content semantic compliance risk analysis module, a comprehensive risk assessment module, a digital watermarking and traceability module, and a manual review module.

[0023] The intelligent agent system for open data compliance verification uses an intelligent agent as the scheduling core and is configured to receive a data catalog to be made public. The data catalog includes at least one data table, and the data table includes at least one data field. The intelligent agent system for open data compliance verification calls the data catalog preprocessing module to process the data catalog to be made public and generate standardized text.

[0024] The intelligent agent system for open data compliance verification inputs standardized text along with metadata from the data catalog to be made public into the structured entity sensitive identification module, the industry policy compliance rule judgment module, and the content semantic compliance risk judgment module in parallel. This triggers compliance judgment operations in the corresponding dimensions and outputs the corresponding compliance judgment results. The intelligent agent system for open data compliance verification then aggregates the compliance judgment results output by the above modules into the comprehensive risk assessment module, which performs risk rating and closed-loop management and disposal.

[0025] The data catalog preprocessing module includes a data catalog pre-identification and judgment unit, a structured text fusion and splicing unit, and a deep semantic parsing unit. The data catalog pre-identification and judgment unit receives the data catalog to be made public and filters the data tables and data fields contained in the data catalog to be made public according to the preset data hierarchical classification and labeling system, so as to filter out the data tables and data fields that meet the preset control scope.

[0026] In addition, the data catalog pre-identification and analysis unit performs sensitive data screening operations on data fields. The scope of the sensitive data screening operation covers document codes, communication contact information, biometric features and key fields involving asset confidentiality. The data catalog pre-identification and analysis unit intercepts and removes data fields containing features corresponding to the above-mentioned identification scope.

[0027] After completing the above screening and sensitive data identification, the structured text fusion and splicing unit performs structured text fusion and splicing operations on the data tables that have passed the identification. Specifically, it uses all the data fields in a single data table as the smallest semantic splicing unit to construct structured text.

[0028] In the process of constructing structured text, the structured text fusion and splicing unit attaches pre-attribute information at the beginning of the structured text. The pre-attribute information includes the ownership attribute of the data directory to be made public and the basic attributes of the data table, which are used to mark the data source of a single data table.

[0029] The structured text fusion and splicing unit lists all data fields in a single data table according to the set sorting sequence rules, and configures a unified and standardized description prefix for each data field. The description prefix includes fixed identifier text used to indicate the field name and the English name of the field. The structured text fusion and splicing unit integrates the diverse elements of the structured fields in a single data table to construct a coherent structured text.

[0030] Subsequently, the structured text fusion and splicing unit performs data cleaning operations on the structured text to remove null values ​​and invalid redundant content. At the same time, the structured text fusion and splicing unit retains the business tags, field comments, enumeration values ​​and data item descriptions corresponding to the data table, and finally outputs standardized text.

[0031] The deep semantic parsing unit receives standardized text, uses natural language processing technology to extract the core semantics and key information contained in the standardized text, and determines the industry attributes and business domains corresponding to the data table based on the core semantics and key information.

[0032] The intelligent agent system for open data compliance verification receives industry attributes and business domains output by the deep semantic parsing unit. Based on these attributes and domains, it schedules the structured entity sensitivity identification module, the industry policy compliance rule judgment module, and the content semantic compliance risk judgment module to provide structured input and scheduling parameters for the three major review modules. When the intelligent agent system for open data compliance verification calls the data catalog preprocessing module, the data catalog pre-identification and judgment unit within the data catalog preprocessing module receives the data catalog to be made public and obtains the preset data hierarchical classification label system corresponding to the data catalog to be made public.

[0033] The data catalog pre-identification and analysis unit filters the data tables and data fields contained in the data catalog to be made public based on the data hierarchical classification and labeling system, in order to select the data tables and data fields that meet the preset control scope.

[0034] Subsequently, the data catalog pre-identification and analysis unit performs sensitive data screening operations on the screened data fields. The data catalog pre-identification and analysis unit extracts the feature information of the data fields and compares and identifies key fields containing document codes, communication contact information, biometric features and asset confidentiality attributes through preset regular expressions and keyword matching algorithms, thereby intercepting and eliminating data fields that match the above features.

[0035] The structured text fusion and splicing unit receives the data tables identified by the data catalog pre-recognition and judgment unit, and performs structured text fusion and splicing operations on the data tables. The structured text fusion and splicing unit uses all the data fields in a single data table as the smallest semantic splicing unit to construct structured text.

[0036] During the splicing process, the structured text fusion splicing unit extracts the attribution attributes of the data directory to be made public and the basic attributes of the data tables, and attaches the attribution attributes and basic attributes as pre-attribute information to the beginning of the structured text, with the pre-attribute information marking the data source dimension of a single data table.

[0037] Meanwhile, the structured text fusion and splicing unit serializes and arranges all data fields in a single data table according to preset sorting rules. The sorting rules include arranging according to the physical storage order of the fields or the alphabetical order of the field names. The structured text fusion and splicing unit configures a unified and standardized description prefix for each data field. The unified and standardized description prefix contains fixed identifier text indicating the Chinese name and English name of the field.

[0038] The structured text fusion and splicing unit integrates multiple elements of structured fields within a single data table to generate structured text, and performs data cleaning operations on the structured text to remove null values ​​and redundant characters.

[0039] The structured text fusion and splicing unit retains the business tags, field comments, enumeration values, and data item descriptions corresponding to the data table in the structured text, thereby outputting clear and standardized text that retains multi-dimensional semantic information.

[0040] The deep semantic parsing unit receives the standardized text output by the structured text fusion and splicing unit, and uses natural language processing technology to extract the core semantics and key information contained in the standardized text. The natural language processing technology includes a keyword extraction algorithm based on word frequency-inverse text frequency index. The deep semantic parsing unit determines the industry attributes and business domains corresponding to the data table based on the core semantics and key information, and finally outputs the classification results for the data table. When the intelligent agent system for open data compliance verification schedules the structured entity sensitive recognition module according to the industry attributes and business domain information, the structured entity sensitive recognition module receives the standardized text output by the data catalog preprocessing module as structured input. The structured entity sensitive recognition module includes a BERT embedding layer, a BiLSTM layer, and a CRF layer.

[0041] The structured entity sensitive recognition module converts standardized text into character sequences. and character sequence The input is fed into the BERT embedding layer, where Represents a character sequence The first in One character, Represents a character sequence The total length.

[0042] BERT embedding layers for character sequences Each character in Perform dual embedding processing, specifically including: querying a pre-trained vocabulary mapping table for characters. Perform word embedding mapping calculations to generate word embedding vectors that contain basic semantics. ; and for characters Perform position embedding mapping calculations to generate characters. Location embedding vectors of location features in standardized text .

[0043] Subsequently, the BERT embedding layer embeds the word vectors. With position embedding vector Perform element-wise summation to generate a string representation. initial embedding representation The initial embedding representation operation formula is:

[0044] BERT embedding layer includes The BERT embedding layer, a layer-stacked Transformer encoder structure, contains an initial embedding sequence for the entire character sequence. The input is fed into the Transformer encoder structure, and the multi-head self-attention mechanism inside the Transformer encoder structure is used to extract the contextual features between each character in the initial embedding sequence.

[0045] go through The operation of the stacked Transformer encoder structure, BERT embedding layer 21 outputs word vector matrix. Its output formula is:

[0046] in, Indicates the first The hidden state feature matrix output by the layer Transformer encoder.

[0047] Word vector matrix The dimension is ,in For the defined word vector dimensions, the word vector matrix The global contextual features and local descriptive features of standardized text are integrated to extract entity sequence dependencies by the BiLSTM layer within the structured entity sensitive recognition module. The BiLSTM layer within the structured entity sensitive recognition module receives the word vector matrix output by the BERT embedding layer. To extract entity sequence dependencies, the BiLSTM layer contains a forward long short-term memory network structure and a backward long short-term memory network structure. The forward and backward long short-term memory network structures operate independently and in parallel, processing the word vector matrix from opposite temporal directions. The sequence of word vectors contained therein.

[0048] Forward Long Short-Term Memory Network Structure from Character Sequence Perform a forward scan calculation from the starting position to the ending position, at any time step The feedforward long short-term memory network structure receives the word vector input corresponding to the current time step. Simultaneously, it receives the forward hidden state features from the previous time step. It updates the cell state through the synergistic effect of the internal forget gate, input gate, and output gate, and calculates the forward hidden state features of the current time step. The formula for calculating the features of the forward hidden state is:

[0049] in, The standard operational functions for representing Long Short-Term Memory (LSTM) networks.

[0050] Backward Long Short-Term Memory network structure from character sequence The calculation is performed in reverse from the end position to the start position, within the same time step. The feedforward long short-term memory network structure receives the word vector input corresponding to the current time step. Simultaneously, it receives the backward hidden state features from the previous time step in the backward processing direction. And perform the same gating update operation, calculate the output backward hidden state features of the current time step. The formula for calculating the features of the backward hidden state is:

[0051] The BiLSTM layer performs concatenation computation on the forward hidden state features and the backward hidden state features, specifically for each time step. The BiLSTM layer hides the state features in the forward pass. and backward hidden state features The vectors are concatenated and fused along their vector dimensions to generate a bidirectional feature vector for the current time step. The formula for calculating bidirectional eigenvectors is:

[0052] BiLSTM layer traverses character sequence For all time steps, the output is a set of sequence feature vectors consisting of all bidirectional feature vectors. Because the scanning process of the forward long short-term memory network structure and the backward long short-term memory network structure crosses the boundaries of each data field of the standardized text concatenation, the set of sequence feature vectors... It fully maps the sensitive information association features between different data fields within a single data table.

[0053] The BiLSTM layer converts the sequence feature vector set The output is sent to the CRF layer within the structured entity sensitive recognition module for sequence labeling and risk classification decoding. To perform sequence labeling and risk classification decoding, the CRF layer within the structured entity sensitive recognition module receives the sequence feature vector set output from the BiLSTM layer. And construct the label transition matrix Label transition matrix The dimension is ,in This represents the total number of preset sensitive risk classification labels, and the label transition matrix. elements in Indicates from the first The label corresponding to each character Transfer to the The label corresponding to each character The transition constraint score.

[0054] CRF layer in label transition matrix The system introduces field-level label uniformity constraints, specifically by configuring transfer constraint scores. This restricts character sequence segments belonging to the same data field within standardized text to have the same sensitivity risk classification label, prohibiting the transfer of different sensitivity labels within the same data field, thereby eliminating label transfer ambiguity within a single data field.

[0055] Subsequently, the CRF layer defines the potential label sequence. With sequence feature vector set Matching score function Its calculation formula is:

[0056] in, Represents the set of sequence feature vectors The Middle Bidirectional feature vectors at each time step Representation and Label The corresponding weight vector operation factor.

[0057] The CRF layer calculates the character sequence for the standardized text based on the matching score function. Corresponding latent label sequences conditional probability This is used to measure the fit between the potential label sequence and the current character feature sequence. The calculation formula is as follows:

[0058] in, Represents a character sequence The set of all corresponding potential label sequences, Based on the above conditional probabilities, the CRF layer uses the Viterbi algorithm to select from all potential label sequences. Perform optimal path decoding and output the optimal label sequence with the highest probability value. The optimal path decoding calculation formula is as follows:

[0059] After decoding, the CRF layer will select the optimal tag sequence. The individual character tags and character sequences within the standardized text Alignment matching is performed, and the text boundaries corresponding to the data fields are defined based on the unified normalized description prefix contained in the standardized text.

[0060] The CRF layer is based on the optimal label sequence within the text boundaries. The system determines the sensitive attributes of the corresponding data fields, extracts uniform character tags within the text boundaries as the field-level risk classification results for the data fields, and finally, the structured entity sensitivity identification module summarizes and outputs the compliance assessment results containing the field-level risk classification results to the comprehensive risk assessment module, completing the verification flow of structured entities. When the intelligent agent system for open data compliance verification schedules the industry policy compliance rule assessment module based on industry attributes and business domain information, the industry policy compliance rule assessment module obtains the metadata of the data catalog to be opened to the public, the business tags of the data tables, and the basic attributes of the data fields retained through the aforementioned screening.

[0061] Based on the aforementioned data, the industry policy compliance rule analysis module generates multi-dimensional search terms using a pre-defined search term template. These multi-dimensional search terms include basic search terms, domain search terms, compliance search terms, open attribute search terms, and standard search terms. Subsequently, the industry policy compliance rule analysis module performs synonym rewriting and expansion operations on the multi-dimensional search terms based on a pre-defined thesaurus and domain knowledge graph to generate a target search term set.

[0062] The industry policy compliance rule assessment module performs web search operations based on the target search term set. Specifically, it filters initial policy documents related to industry attributes and business areas through global search, and obtains authoritative policy and standard documents through targeted search on government agency websites and national standardization platforms. During the search process, the industry policy compliance rule assessment module filters invalid web pages based on the set domain priority and file format conditions, and retains authoritative policy and standard documents that are valid.

[0063] The industry policy compliance rule assessment module uses natural language processing technology to perform structured analysis on authoritative policy and standard documents. Through layout analysis and sequence labeling algorithms, it breaks down the chapters and clauses of authoritative policy and standard documents to identify the scope of application, prohibitive clauses, and restrictive clauses. Then, the industry policy compliance rule assessment module extracts data classification and grading requirements and data openness rule elements, and transforms the extracted elements into structured compliance entries. The structured compliance entries include the document source, specific clause content, and compliance requirements.

[0064] The industry policy compliance rule assessment module constructs multi-dimensional rule templates based on structured compliance items. These templates include industry matching templates, sensitive data management templates, and open scenario matching templates. Simultaneously, the module utilizes a pre-trained large language model text encoder to map the metadata and business tags of the data catalog to be made public to the clauses of the structured compliance items into high-dimensional semantic vectors, and calculates the cosine similarity between the two sets of high-dimensional semantic vectors.

[0065] To establish the target matching association between the data table and the structured compliance items, the industry policy compliance rule analysis module performs a fusion judgment operation: when the multi-dimensional rule template is matched, the matching relationship is directly confirmed; when it is not matched, the cosine similarity is compared with the preset similarity threshold, and the matching relationship is confirmed when the cosine similarity is greater than or equal to the similarity threshold.

[0066] The industry policy compliance rule assessment module performs a comprehensive compliance assessment on the data table based on target matching and association. The assessment dimensions cover legality assessment, industry compliance assessment, sensitive data control assessment, and open procedure compliance assessment. The industry policy compliance rule assessment module determines the data openness level of the data table according to the assessment results. The data openness level is divided into prohibited openness, restricted openness, and unconditional openness.

[0067] Finally, the industry policy compliance rule analysis module generates a compliance assessment report containing compliance basis and data openness level, and outputs the compliance assessment report to the comprehensive risk assessment module for subsequent hierarchical control decisions. When the intelligent agent system for open data compliance verification schedules the content semantic compliance risk analysis module according to industry attributes and business domain information, the content semantic compliance risk analysis module receives standardized text output by the data catalog preprocessing module.

[0068] The content semantic compliance risk assessment module constructs a tag and risk prefix prompt template containing mask markers, transforming the document-level semantic review task for standardized text into a joint mask prediction task. This joint mask prediction task includes sensitive category prediction and risk level prediction. Subsequently, the content semantic compliance risk assessment module concatenates the mask markers with the standardized text according to the set mask arrangement rules, generating a prompt concatenation sequence. Its calculation formula is:

[0069] in, This indicates the total number of data fields contained in the standardized text. Indicates the first Sensitive category mask markers for each data field Indicates the first Risk level mask markers corresponding to each data field This represents the sequence concatenation operator. This represents the sequence of text characters corresponding to standardized text.

[0070] The content semantic compliance risk assessment module includes a RoBERTa-large semantic encoding model and a risk mapping unit. This module will provide suggestions for the concatenation sequence. The input is fed into the RoBERTa-large semantic coding model, and field boundary embedding vectors are superimposed in the embedding input layer of the RoBERTa-large semantic coding model. The field boundary embedding vectors are used to identify the hierarchical structure and boundary position of each data field in the standardized text.

[0071] The RoBERTa-large semantic encoding model utilizes its internal multi-head self-attention layer to extract contextual semantic association features between different data fields and to concatenate the cue sequences. Feature encoding is performed, and then the RoBERTa-large semantic encoding model outputs mask embedding vectors at the positions corresponding to the sensitive category mask markers and the risk level mask markers. The mask embedding vectors contain sensitive category mask embedding vectors and risk level mask embedding vectors.

[0072] The risk mapping unit receives the mask embedding vector and maps the sensitive category mask embedding vector and the risk level mask embedding vector to the corresponding label feature space through a linear mapping layer. Then, the risk mapping unit performs a nonlinear transformation operation on the mapped feature vector through a batch normalization function and a Tanh activation function.

[0073] The risk mapping unit calculates the cosine similarity between the feature vector after nonlinear transformation and the preset label embedding vector. The preset label embedding vector is a pre-initialized vector representation of sensitive categories and risk levels. The risk mapping unit extracts the preset label corresponding to the highest cosine similarity value to determine the implicit sensitive category and quantified risk level of each data field. Implicit sensitive categories include politically sensitive attributes and commercially confidential attributes.

[0074] Finally, the risk mapping unit performs a weighted summation operation on the quantitative risk levels of all data fields based on the set risk weight coefficients, and generates a table-level comprehensive semantic risk score for the data table based on the result of the weighted summation operation. The content semantic compliance risk assessment module outputs the risk assessment results, which include the quantitative risk level and the table-level comprehensive semantic risk score, to the comprehensive risk assessment module. When the content semantic compliance risk assessment module outputs the risk assessment results, which include the quantitative risk level and the table-level comprehensive semantic risk score, to the comprehensive risk assessment module, the comprehensive risk assessment module simultaneously receives the field-level risk classification results output by the structured entity sensitivity identification module, as well as the compliance assessment report output by the industry policy compliance rule assessment module.

[0075] The comprehensive risk assessment module uses a pre-defined risk mapping table (e.g., mapping the "prohibited access" level in the compliance assessment report to the highest risk value of 1.0, and "restricted access" to 0.5) to transform discrete field-level risk classification results and text-formatted compliance assessment reports into continuous standardized risk values. This quantifies and summarizes the field-level risk classification results, compliance assessment reports, and risk assessment results. Simultaneously, the comprehensive risk assessment module acquires the core semantics and key information output by the data catalog preprocessing module during the parsing phase.

[0076] The comprehensive risk assessment module determines the risk emphasis dimension of the data table's domain based on core semantics and key information. It then dynamically assigns weights to the summarized field-level risk classification results, compliance assessment reports, and risk judgment results based on these risk emphasis dimensions. For example, when the core semantics and key information indicate the healthcare domain, the weight coefficient for biometric features in the field-level risk classification results is dynamically increased. Subsequently, the comprehensive risk assessment module obtains a comprehensive risk score through weighted calculation and determines the overall risk level of the data table based on the comprehensive risk score. The overall risk level is divided into low-risk, medium-risk, and high-risk levels.

[0077] The comprehensive risk assessment module includes a compliance rule engine. The comprehensive risk assessment module schedules the compliance rule engine to perform matching and comparison in a pre-built rule database based on core semantics and key information in order to extract a special rule library corresponding to the core semantics and key information.

[0078] The comprehensive risk assessment module configures the special rule base as the verification logic for the data table and sends instructions to the compliance rule engine. The compliance rule engine performs targeted compliance verification operations on the data table according to the verification logic and outputs the rule verification results.

[0079] Finally, the comprehensive risk assessment module integrates core semantics and key information, overall risk level, and rule verification results to perform decision analysis. Based on the results of the decision analysis, it generates a hierarchical closed-loop handling plan for the data table, thereby establishing an execution path for the intelligent agent system for open data compliance verification to perform hierarchical handling actions. Based on the execution path established by the comprehensive risk assessment module for the intelligent agent system for open data compliance verification, the intelligent agent system for open data compliance verification analyzes the hierarchical closed-loop handling plan and executes differentiated control operations.

[0080] When the comprehensive risk assessment module determines that the overall risk level of the data table is low, the comprehensive risk assessment module triggers a rapid pass through the rule chain, extracts core semantics and key information as well as rule verification results to generate a release instruction, the intelligent agent system for open data compliance verification receives the release instruction, and schedules the digital watermarking and traceability module.

[0081] The digital watermarking and traceability module obtains the identity and timestamp information of the original data provider, extracts the metadata and basic attributes of the data table of the data directory to be made public, and uses an encryption algorithm to fuse the identity, timestamp information and basic attributes of the metadata to generate an encrypted sequence as a watermark identifier. The watermark identifier is then embedded into the data table to generate a digital watermark. Specific embedding methods include, but are not limited to, inserting a zero-width character sequence into the non-numeric text field of the data table, or mapping the watermark identifier to the least important bit of the numeric field, so as to complete the watermark concealment without compromising the usability of the original data. Subsequently, the intelligent agent system for open data compliance verification allows the data table carrying the digital watermark to enter the data publishing node.

[0082] When the comprehensive risk assessment module determines that the overall risk level of the data table is medium risk, the comprehensive risk assessment module triggers the early warning review rule chain. The intelligent agent system for open data compliance verification performs an access freeze operation on the data table and marks the data table as pending review by modifying the status field.

[0083] Simultaneously, the comprehensive risk assessment module integrates core semantics and key information, as well as specific risk points extracted from field-level risk classification results, compliance assessment reports, and risk judgment results, to generate tasks to be reviewed. These tasks are then pushed to the manual review module. Upon receiving the review confirmation result from the manual review module, the comprehensive risk assessment module executes branch decisions based on the review confirmation result. When the review and confirmation result indicates a risk-free status, the comprehensive risk assessment module instructs the intelligent agent system for open data compliance verification to lift the permission freeze and allow the data table to pass through. When the review and confirmation result indicates a risk status, the comprehensive risk assessment module generates targeted rectification suggestions and instructs the intelligent agent system for open data compliance verification to send the targeted rectification suggestions to the corresponding data provider terminal. The intelligent agent system for open data compliance verification then obtains the rectified data table and re-triggers the compliance assessment operation for the data table.

[0084] When the comprehensive risk assessment module determines that the overall risk level of the data table is high, the comprehensive risk assessment module triggers the mandatory control rule chain. The intelligent agent system for open data compliance verification blocks the data table release process according to the mandatory control rule chain, determines that the data table's review status is not passed, and performs an isolation and offline operation on the data table.

[0085] Furthermore, the comprehensive risk assessment module extracts core semantics and key information, as well as high-risk characteristic data that triggers the mandatory control rule chain extracted from field-level risk classification results, compliance assessment reports, and risk judgment results, to generate detailed alarm reports. It then instructs the intelligent agent system for open data compliance verification to send the detailed alarm reports to the preset regulatory control platform.

[0086] For data tables that have triggered a violation release status, the intelligent agent system for open data compliance verification schedules the digital watermarking and traceability module. The digital watermarking and traceability module extracts the digital watermark attached to the data table, executes the data traceability parsing algorithm, extracts the identity and timestamp information of the original data provider by parsing the digital watermark, and generates the location information of the responsible entity based on the identity and timestamp information, thereby establishing a closed loop for security control of the data table.

[0087] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An intelligent agent system for open data compliance verification, characterized in that, The intelligent agent system for open data compliance verification includes a data catalog preprocessing module, a structured entity sensitive identification module, an industry policy compliance rule analysis module, a content semantic compliance risk analysis module, and a comprehensive risk assessment module. The intelligent agent system for open data compliance verification is configured to receive a data catalog to be made public, wherein the data catalog includes at least one data table, and the data table includes at least one data field. The data catalog preprocessing module is used to process the data catalog to be made public to generate standardized text, extract the core semantics and key information contained in the standardized text, and determine the industry attributes and business areas corresponding to the data table based on the core semantics and key information. The structured entity sensitive identification module, the industry policy compliance rule judgment module, and the content semantic compliance risk judgment module are used to receive the standardized text in parallel, and trigger compliance judgment operations of corresponding dimensions according to the industry attributes and business domains, so as to output field-level risk classification results, compliance assessment reports, and risk judgment results respectively. The comprehensive risk assessment module is used to receive the field-level risk classification results, the compliance assessment report, and the risk judgment results, and to perform weight allocation and data analysis based on the core semantics and key information to determine the overall risk level of the data table, and generate a graded closed-loop handling plan for the data table.

2. The intelligent agent system for open data compliance verification according to claim 1, characterized in that, The data catalog preprocessing module includes a data catalog pre-identification and judgment unit, a structured text fusion and splicing unit, and a deep semantic parsing unit; The data catalog pre-identification and judgment unit is used to filter the data tables and data fields contained in the data catalog to be opened to the public according to the preset data hierarchical classification and label system, and to perform sensitive data identification operation on the data fields to remove the data fields containing any of the features of document codes, communication contact information, biometric features and asset confidentiality key fields; The structured text fusion and splicing unit is used to receive the data table that has been screened, take all the data fields in each data table as the smallest semantic splicing unit, configure a unified and standardized description prefix and attach pre-attribute information containing basic attributes to construct structured text, and perform data cleaning operation on the structured text to output the standardized text. The deep semantic parsing unit is used to extract the core semantics and key information contained in the standardized text using natural language processing technology, and output the classification results for the data table.

3. The intelligent agent system for open data compliance verification according to claim 1, characterized in that, The structured entity sensitive recognition module includes a BERT embedding layer, a BiLSTM layer, and a CRF layer; The BERT embedding layer is used to convert the standardized text into a character sequence, perform dual embedding processing on each character in the character sequence to generate a word embedding vector and a position embedding vector, perform a summation operation on the word embedding vector and the position embedding vector to generate an initial embedding representation, and use the Transformer encoder structure contained in the BERT embedding layer to extract contextual features and output a word vector matrix. The BiLSTM layer is used to receive the word vector matrix. The BiLSTM layer includes a forward long short-term memory network structure and a backward long short-term memory network structure. The forward long short-term memory network structure and the backward long short-term memory network structure process the word vector matrix in parallel from opposite temporal directions, perform concatenation and fusion to generate bidirectional feature vectors, and output a set of sequence feature vectors composed of all the bidirectional feature vectors.

4. The intelligent agent system for open data compliance verification according to claim 3, characterized in that, The CRF layer is used to receive the sequence feature vector set and perform sequence labeling and risk classification decoding. The CRF layer constructs a label transition matrix, and introduces field-level label uniformity constraints into the label transition matrix. By configuring the transition constraint score, it is determined that character sequence segments belonging to the same data field within the standardized text must have the same sensitivity risk classification label. The CRF layer defines a matching score function between the potential label sequence and the sequence feature vector set, calculates the conditional probability based on the matching score function, and uses the Viterbi algorithm to perform optimal path decoding to output the optimal label sequence. The CRF layer aligns and matches the optimal label sequence with the character sequence in the standardized text, and extracts the uniform character labels within the text boundary as the field-level risk classification result corresponding to the data field.

5. The intelligent agent system for open data compliance verification according to claim 1, characterized in that, The industry policy compliance rule analysis module is used to obtain metadata corresponding to the data directory to be opened to the public, business tags corresponding to the data table, and basic attributes corresponding to the data field, generate multi-dimensional search terms, and expand to generate a target search term set; The industry policy compliance rule analysis module performs a web search operation based on the target search term set to obtain authoritative policy standard documents, uses natural language processing technology to perform structured parsing on the authoritative policy standard documents to extract data openness rule elements, and transforms the extracted data openness rule elements into structured compliance entries containing specific clause content; The industry policy compliance rule judgment module constructs a multi-dimensional rule template based on the structured compliance items, including industry matching templates, sensitive data control templates, and open scenario matching templates. It uses a pre-trained large language model text encoder to map the metadata and business tags to the clause content of the structured compliance items into high-dimensional semantic vectors, and calculates the cosine similarity between the two sets of high-dimensional semantic vectors.

6. The intelligent agent system for open data compliance verification according to claim 5, characterized in that, The industry policy compliance rule analysis module is configured to perform a fusion judgment operation to establish a target matching association between the data table and the structured compliance entries, specifically: When the data table and the multi-dimensional rule template are successfully matched, the matching relationship is directly confirmed. When the data table and the multi-dimensional rule template fail to match, the cosine similarity is compared with a preset similarity threshold. When the cosine similarity is greater than or equal to the similarity threshold, the matching relationship is confirmed. The industry policy compliance rule analysis module performs a comprehensive compliance assessment on the data table based on the target matching association, determines the data openness level corresponding to the data table, and generates a compliance assessment report containing the compliance basis and the data openness level.

7. The intelligent agent system for open data compliance verification according to claim 1, characterized in that, The content semantic compliance risk assessment module includes a RoBERTa-large semantic encoding model and a risk mapping unit; The content semantic compliance risk assessment module constructs a tag and risk prefix prompt template, which includes a mask mark. The mask mark is concatenated with the standardized text to generate a prompt concatenation sequence. The RoBERTa-large semantic coding model receives the cue concatenation sequence and superimposes field boundary embedding vectors used to identify boundary positions in the embedding input layer included in the RoBERTa-large semantic coding model. It performs feature encoding on the cue concatenation sequence and outputs a mask embedding vector containing sensitive category mask embedding vectors and risk level mask embedding vectors at the position corresponding to the mask mark.

8. The intelligent agent system for open data compliance verification according to claim 7, characterized in that, The risk mapping unit receives the mask embedding vector and maps the sensitive category mask embedding vector and the risk level mask embedding vector to the corresponding label feature space through the linear mapping layer contained in the risk mapping unit, and performs a nonlinear transformation operation. The risk mapping unit calculates the cosine similarity between the feature vector after the nonlinear transformation operation and the pre-initialized preset label embedding vector, and extracts the preset label corresponding to the highest cosine similarity value to determine the implicit sensitive category and quantified risk level of each data field. The risk mapping unit performs a weighted summation operation on the quantitative risk level of all the data fields based on the set risk weight coefficient, generates a table-level comprehensive semantic risk score, and outputs the risk assessment result containing the quantitative risk level and the table-level comprehensive semantic risk score to the comprehensive risk assessment module.

9. The intelligent agent system for open data compliance verification according to claim 1, characterized in that, The comprehensive risk assessment module includes a compliance rules engine; The comprehensive risk assessment module transforms the discrete field-level risk classification results and the text-formatted compliance assessment report into continuous standardized risk values ​​through a preset risk mapping table. Based on the core semantics and key information, it determines the risk emphasis dimension and performs dynamic weight allocation. Through weighted calculation, it obtains a comprehensive risk score and determines the overall risk level of the data table. The overall risk level includes low risk level, medium risk level, and high risk level. The comprehensive risk assessment module schedules the compliance rule engine to perform matching and comparison based on the core semantics and key information to extract a special rule library, configures the special rule library as verification logic, and performs targeted compliance verification operations on the data table based on the verification logic to output the rule verification results; The comprehensive risk assessment module integrates the core semantics and key information, the overall risk level, and the rule verification results to perform decision analysis and generate the hierarchical closed-loop disposal plan for the data table.

10. The intelligent agent system for open data compliance verification according to claim 9, characterized in that, The intelligent agent system for open data compliance verification also includes a digital watermarking and traceability module and a manual review module; When the overall risk level is low, the comprehensive risk assessment module triggers a fast pass rule chain to schedule the digital watermarking and traceability module. The digital watermarking and traceability module obtains the identity and timestamp information of the original data provider, extracts the metadata corresponding to the data directory to be opened to the public, and the basic attributes corresponding to the data table. It uses an encryption algorithm to fuse the identity of the original data provider, the timestamp information, the metadata, and the basic attributes to generate an encrypted sequence as a watermark identifier. The watermark identifier is then embedded in the data table to generate a digital watermark, and the data table carrying the digital watermark is allowed to pass. When the overall risk level is medium risk, the comprehensive risk assessment module triggers the early warning review rule chain, freezes the permissions of the data table, extracts specific risk points from the field-level risk classification results, the compliance assessment report and the risk judgment results, integrates them to generate a review task and pushes it to the manual review module, and unfreezes or generates targeted rectification suggestions based on the returned review confirmation results. When the overall risk level is high, the comprehensive risk assessment module triggers the mandatory control rule chain, blocks the release process and performs isolation and offline operations, generates a detailed alarm report, and schedules the digital watermarking and traceability module to extract the identity and timestamp information of the original data provider by parsing the attached digital watermark in order to generate the location information of the responsible entity.

Images