A method for identifying shared attributes of public data with multi-dimensional information enhancement

Abstract

The present invention provides a method for identifying shared attributes of public data enhanced by multi-dimensional information, which belongs to the field of natural language processing, and includes: obtaining guidance basis data, formatting and cleaning it, and obtaining a set of guidance domain knowledge; inputting the set of guidance domain knowledge into a pre-trained language model, fine-tuning the model parameters of the pre-trained language model by a masking method, and obtaining a pre-trained language model enhanced with domain knowledge; splicing the data class name, data source department, and application scenario description of the data to be identified in the data to be identified, and obtaining training data enhanced with context information; designing a discriminant model, using the training data enhanced with context information to perform model training on the pre-trained language model enhanced with domain knowledge and the discriminant model, and using the trained discriminant model to identify the shared attributes of the data to be identified. The present invention can accurately distinguish the shared attributes of data fields, and is convenient for carrying out the shared exchange of public data.

Description

Technical Field

[0001] The invention relates to a method for identifying shared attributes of public data enhanced by multi-dimensional information, and belongs to the technical field of natural language processing. Background Art

[0002] Public data sharing refers to the act of a public agency using data from other public agencies, or providing data to other public agencies, in accordance with the law in order to perform its statutory duties or meet public service needs.

[0003] In order to standardize the data sharing process, it is necessary to confirm the sharing attributes of each field in the public data, and divide the data into conditional sharing, unconditional sharing, and non-sharing, so as to provide a practical basis for public data sharing. In the actual application process, it is difficult to carry out actual data sharing and exchange because the public institutions have not clearly defined the data sharing conditions at this stage. Faced with the huge amount of data, manual judgment is not only time-consuming and laborious, but also has the risk of delaying subsequent work.

[0004] This task has two characteristics: first, there must be a basis for judging the sharing conditions of the fields, such as the requirements of laws, regulations and other documents. Judgment without a basis cannot be recognized by the data source unit and the data user unit; second, direct judgment of the fields is prone to misjudgment. For example, for the "address" and "contact information" fields, if they are personal information, they should be conditionally contributed, and if they are corporate registration information, they should be unconditionally shared.

[0005] At present, text classification models have been widely used. However, considering the characteristics of this task, when using text classification models directly to solve this task, on the one hand, the model lacks the basis for judging shared attributes; on the other hand, the existence of different shared attributes of the same text in the data can easily interfere with the model's classification ability during the model training process, resulting in a large number of errors in the final result. Summary of the invention

[0006] The present invention provides a method for identifying shared attributes of public data with multi-dimensional information enhancement, which is used to determine the sharing conditions of various fields of public data, and solves the problem that the confirmation of the shared attributes of public data is prone to misjudgment due to lack of judgment basis.

[0007] The technical solution of the present invention is: a method for identifying shared attributes of public data with multi-dimensional information enhancement, the method comprising the following steps:

[0008] Step 1: Obtain guidance basis data, format and clean it, and obtain a set of guidance domain knowledge;

[0009] Step 2: Input the guidance domain knowledge set into the pre-trained language model, and fine-tune the model parameters of the pre-trained language model using a masking method to learn the domain knowledge during the fine-tuning process, thereby obtaining a pre-trained language model enhanced with domain knowledge;

[0010] Step 3: The data class name, data source department, and application scenario description of the data to be identified are combined in the data to be identified to obtain training data enhanced with context information;

[0011] Step 4: Design a discriminant model, use the training data enhanced with contextual information to train the pre-trained language model and discriminant model enhanced with domain knowledge, and use the trained discriminant model to identify the shared attributes of the data to be identified.

[0012] Furthermore, the step 1 comprises:

[0013] Step 1.1: Use crawler technology to crawl the guidance data on the Internet;

[0014] Step 1.2: Format and clean the guidance data, treat each item or section of each chapter as a unit sequence, and form a set of guidance domain knowledge.

[0015] Furthermore, the step 2 comprises:

[0016] Step 2.1: Based on the open source pre-trained language model, take each sequence in the obtained guidance domain knowledge set as a unit, and use the dictionary that comes with the model to segment the cleaned guidance basis data;

[0017] Step 2.2: Randomly mask the results of word segmentation, replace the randomly selected content with the "[mask]" marker, and construct the input of the pre-trained language model fine-tuning task;

[0018] Step 2.3: Predict the original word at the position of the "[mask]" mark in the input sequence, use the cross entropy loss function to calculate the loss of the pre-trained language model, and use the adaptive moment estimation algorithm as the optimizer to fine-tune the pre-trained language model;

[0019] Step 2.4: After the model loss value of the task to be fine-tuned no longer decreases within several training batches, stop the fine-tuning task and save the parameters.

[0020] Furthermore, the step 3 comprises:

[0021] Step 3.1: Use the “three-person blind judgment intersection” method to mark a batch of training samples for the recognition data;

[0022] Step 3.2, modify the pre-trained language model's built-in dictionary, and treat "conditionally shared", "unconditionally shared" and "not shared" as three separate words;

[0023] Step 3.3: For the pre-trained language model input text, the data table name, field name, data source department and application scenario description are concatenated by designing standardized text. The concatenated text input is: "When {data source department} handles {application scenario description} business, a data table {data table name} is generated, which contains the field {field name}. The sharing condition of this field should be [mask]", where the content in "{}" is filled with the corresponding content, and the "[mask]" position is "conditionally shared", "unconditionally shared" and "not shared" according to the different labels of the training data.

[0024] Furthermore, in step 4, the discriminant model is designed by adding a fully connected layer and a softmax layer to the pre-trained language model enhanced with domain knowledge.

[0025] Furthermore, the step 4 comprises:

[0026] Step 4.1: Use the word segmenter of the pre-trained language model and the dictionary modified in step 3.2 to segment and convert the input text constructed in step 3.3 into word sequence numbers, and convert the text sequence into a vector that can be processed by a computer;

[0027] Step 4.2: Use the vector obtained in step 4.1 to perform semantic representation using the pre-trained language model enhanced with the domain knowledge obtained in step 2;

[0028] Step 4.3: Pass the semantic representation obtained in step 4.2 through the fully connected layer and then the softmax layer to obtain the predicted value of the model, and use the predicted value and the true label to calculate the cross entropy loss function;

[0029] Step 4.4: During training, the first several encoding layers of the pre-trained language model enhanced with domain knowledge are fixed, and the parameters of the last encoding layer of the model are optimized using the adaptive moment estimation optimizer and the calculated loss value;

[0030] Step 4.5: Use the above optimized domain knowledge-enhanced pre-trained language model, fully connected layer, and softmax layer to identify the shared attributes of the data to be identified.

[0031] The beneficial effects of the present invention are as follows: the present invention is used to solve the problem that the confirmation of public data sharing conditions is prone to misjudgment due to lack of judgment basis, and can more accurately distinguish the sharing attributes of data fields, thereby facilitating the sharing and exchange of public data. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the process of the present invention;

[0033] Figure 2This is a schematic diagram of the training of the pre-trained language model fine-tuning task in the present invention. DETAILED DESCRIPTION

[0034] Example 1: Figure 1-Figure 2 As shown, a method for identifying shared attributes of public data enhanced by multi-dimensional information comprises the following steps:

[0035] Step 1: Obtain guidance basis data, format and clean it, and obtain a set of guidance domain knowledge;

[0036] The step 1 comprises:

[0037] Step 1.1: Use crawler technology to crawl the guidance data on the Internet;

[0038] Step 1.2: Format and clean the guidance data, treat each item or section of each chapter as a unit sequence, and form a set of guidance domain knowledge.

[0039] The guidance data for Internet crawling include: data sharing and exchange, data security, and relevant laws, regulations, and standards and specifications on data classification and grading crawled from the Internet (main sources: National Standard Information Public Service Platform https: / / std.samr.gov.cn / , National Laws and Regulations Database https: / / flk.npc.gov.cn / fl.html, etc.).

[0040] Step 2: Input the guidance domain knowledge set into the pre-trained language model, and fine-tune the model parameters of the pre-trained language model using a masking method to learn the domain knowledge during the fine-tuning process, thereby obtaining a pre-trained language model enhanced with domain knowledge;

[0041] The step 2 comprises:

[0042] Step 2.1: Use the Chinese pre-trained language model BERT-wwm (12 encoding layers in total) based on the whole word masking technology as the baseline model, and use the dictionary that comes with the model to segment the cleaned guidance basis data based on each sequence in the obtained guidance domain knowledge set;

[0043] Step 2.2: Randomly mask the results of word segmentation, replace the randomly selected content with the "[mask]" marker, and construct the input of the pre-trained language model fine-tuning task;

[0044] Step 2.3: Pre-trained language model fine-tuning task Figure 2As shown in the figure, for the input sequence of the model, the original word at the position of the "[mask]" tag in the input sequence is predicted, the loss of the pre-trained language model is calculated using the cross entropy loss function, and the pre-trained language model is fine-tuned using the adaptive moment estimation algorithm as the optimizer;

[0045] Step 2.4: After the model loss value of the task to be fine-tuned no longer decreases within 5 training batches, stop the fine-tuning task and save the parameters.

[0046] Step 3: The data class name, data source department, and application scenario description of the data to be identified are combined in the data to be identified to obtain training data enhanced with context information;

[0047] The step 3 comprises:

[0048] Step 3.1: The “three-person blind judgment intersection” method is used to label the data fields provided by various public institutions. The sample categories are “conditionally shared”, “unconditionally shared” and “not shared”;

[0049] Step 3.2, modify the pre-trained language model's built-in dictionary, and treat "conditional sharing", "unconditional sharing" and "not to be shared" as three separate words; in the subsequent word segmentation, "conditional sharing", "unconditional sharing" and "not to be shared" will be marked as separate words by the word segmenter;

[0050] Step 3.3, for the input text of the pre-trained language model, by designing standardized text to splice the data table name, field name, data source department and application scenario description, the contextual information of the data is enhanced, which is convenient for the model to eliminate ambiguity and better judge the sharing category. The spliced ​​text input is: "When {data source department} handles {application scenario description} business, a data table {data table name} is generated, which contains the field {field name}, and the sharing condition of this field should be [mask]", where the content in "{}" is filled with the corresponding content, and the "[mask]" position is "conditionally shared", "unconditionally shared" and "not shared" according to the different labels of the training data. For example: the original data sample is: "Unified social credit code: unconditionally shared", after the context information is enhanced, the sample changes to: "When handling the enterprise registration business, a data table of basic enterprise information is generated, which contains the field unified social credit code, and the sharing condition of this field should be unconditionally shared.";

[0051] Step 4: Design a discriminative model. The discriminative model is designed by adding a fully connected layer and a softmax layer to the pre-trained language model enhanced with domain knowledge. The pre-trained language model enhanced with domain knowledge and the discriminative model are trained using training data enhanced with contextual information. The trained discriminative model is used to identify the shared attributes of the data to be identified.

[0052] The step 4 comprises:

[0053] Step 4.1: Use the word segmenter of the pre-trained language model and the dictionary modified in step 3.2 to segment and convert the input text constructed in step 3.3 into word sequences, and convert the text sequence into a vector that can be processed by a computer; replace the label position with "[mask]", and use the position of the original label "conditionally shared" or "unconditionally shared" or "not shared" in the vocabulary as the label used for training;

[0054] Step 4.2: Use the vector obtained in step 4.1 to perform semantic representation using the pre-trained language model enhanced with the domain knowledge obtained in step 2;

[0055] Step 4.3: Select the "[mask]" position of the semantic representation obtained in step 4.2 and pass it through the fully connected layer and then the softmax layer to obtain the model's predicted value, and use the predicted value and the true label to calculate the cross entropy loss function;

[0056] Step 4.4: In order to retain domain knowledge and reduce the amount of training, the first several encoding layers of the pre-trained language model enhanced with domain knowledge are fixed during training, and the parameters of the last encoding layer of the model are optimized using the adaptive moment estimation optimizer and the calculated loss value;

[0057] Step 4.5: Use the above optimized domain knowledge-enhanced pre-trained language model, fully connected layer, and softmax layer to identify the shared attributes of the data to be identified.

[0058] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A method for identifying shared attributes of public data with multi-dimensional information enhancement, characterized in that: The method comprises the following steps: Step 1: Obtain guidance basis data, format and clean it, and obtain a set of guidance domain knowledge; Step 2: Input the guidance domain knowledge set into the pre-trained language model, and fine-tune the model parameters of the pre-trained language model using a masking method to learn the domain knowledge during the fine-tuning process, thereby obtaining a pre-trained language model enhanced with domain knowledge; Step 3: The data class name, data source department, and application scenario description of the data to be identified are combined in the data to be identified to obtain training data enhanced with context information; Step 4: Design a discriminant model, use the training data enhanced by context information to train the pre-trained language model and discriminant model enhanced by domain knowledge, and use the trained discriminant model to identify the shared attributes of the data to be identified; The step 3 comprises: Step 3.1: Use the "three-person blind judgment intersection" method to mark a batch of training samples; Step 3.2, modify the pre-trained language model's built-in dictionary, and treat "conditionally shared", "unconditionally shared" and "not shared" as three separate words; Step 3.3: For the pre-trained language model input text, a standardized text is designed to concatenate the data table name, field name, data source department, and application scenario description. The concatenated text input is: "When {data source department} handles {application scenario description} business, a data table {data table name} is generated, which contains the field {field name}. The sharing condition of this field should be [mask]", where the content in "{}" is filled with the corresponding content, and the "[mask]" position is "conditionally shared", "unconditionally shared", and "not shared" according to the different labels of the training data.

2. According to the multi-dimensional information enhanced public data sharing attribute identification method of claim 1, it is characterized in that: The step 1 comprises: Step 1.1: Use crawler technology to crawl the guidance data on the Internet; Step 1.2: Format and clean the guidance data, treat each item or section of each chapter as a unit sequence, and form a set of guidance domain knowledge.

3. The method for identifying shared attributes of public data enhanced by multi-dimensional information according to claim 1, characterized in that: The step 2 comprises: Step 2.1: Based on the open source pre-trained language model, take each sequence in the obtained guidance domain knowledge set as a unit, and use the dictionary that comes with the model to segment the cleaned guidance basis data; Step 2.2: Randomly mask the results of word segmentation, replace the randomly selected content with the "[mask]" marker, and construct the input of the pre-trained language model fine-tuning task; Step 2.3, predict the original word at the "[mask]" mark position in the input sequence, use the cross entropy loss function to calculate the loss of the pre-trained language model, and use the adaptive moment estimation algorithm as the optimizer to fine-tune the pre-trained language model; Step 2.4: After the model loss value of the task to be fine-tuned no longer decreases within several training batches, stop the fine-tuning task and save the parameters.

4. The method for identifying shared attributes of public data enhanced by multi-dimensional information according to claim 1, characterized in that: In step 4, the discriminant model is designed by adding a fully connected layer and a softmax layer to the pre-trained language model enhanced with domain knowledge.

5. The method for identifying shared attributes of public data enhanced by multi-dimensional information according to claim 1, characterized in that: The step 4 comprises: Step 4.1: Use the word segmenter of the pre-trained language model and the dictionary modified in step 3.2 to segment and convert the input text constructed in step 3.3 into word sequence numbers, and convert the text sequence into a vector that can be processed by a computer; Step 4.2: Use the vector obtained in step 4.1 to perform semantic representation using the pre-trained language model enhanced with the domain knowledge obtained in step 2; Step 4.3: Pass the semantic representation obtained in step 4.2 through the fully connected layer and then the softmax layer to obtain the predicted value of the model, and use the predicted value and the true label to calculate the cross entropy loss function; Step 4.4: During training, the first several encoding layers of the pre-trained language model enhanced with domain knowledge are fixed, and the parameters of the last encoding layer of the model are optimized using the adaptive moment estimation optimizer and the calculated loss value; Step 4.5: Use the above optimized domain knowledge-enhanced pre-trained language model, fully connected layer, and softmax layer to identify the shared attributes of the data to be identified.

Images