System for dynamically generating cue words through policy document information extraction

Abstract

The invention discloses a policy document information extraction and prompt word dynamic generation system. The system comprises a policy management module, a data set management module, a diversity screening module, a Prompt management module and a schema design module. Policy information data is quantized through a diversity screening module, and a representative given number of sample result sets are output. According to the method, a large model is introduced, the large model is guided to apply new prompts generated by the large model through prompt words of a schema design module, and a prompt word with higher quality is optimized and generated according to an example of a representative policy sample which is randomly selected, so that a current task is better completed, and the score of verification calculation is higher. Therefore, accurate policy pushing is achieved, policy bonuses are boosted to be directly and quickly shared, and it is ensured that a market subject wants to completely know the latest policies and wants to enjoy the latest policies quickly.

Description

Technical Field

[0001] The present invention relates to the technical field of information extraction, and more specifically, to a policy document information extraction and dynamic generation prompt word system. Background Art

[0002] Precise policy push is an innovative service measure for the tax system to push policies in a full-process, progressive and differentiated manner based on tax big data. To ensure the implementation of policies, local departments continue to optimize and upgrade the "policy finds people" work mechanism to help market entities enjoy policies in a timely manner, promote the direct and quick enjoyment of policy dividends, and ensure that market entities know everything they should know about the latest policies and enjoy them quickly.

[0003] However, policies are of various types and complex in content. How to efficiently extract effective information from various policy contents is an urgent problem to be solved.

[0004] Traditional information extraction methods include: 1. Rule matching. Identify key elements based on predefined rules and patterns. The disadvantage is that it may not adapt to complex and diverse text data. It is suitable for texts with relatively simple structures and clear rules and patterns. 2. Machine learning. Learn the key elements in the text by training models. The disadvantage is that a large amount of training data needs to be annotated, the text structure is relatively simple, and the rules and patterns are unclear. 3. Deep learning. Handle complex key element extraction tasks. The disadvantage is that a large amount of annotated data and computing resources are required.

[0005] With the rapid development of big models, it has demonstrated its powerful capabilities in all walks of life. It has performed well in text classification, text analysis, text summarization, and text extraction. At present, the method of using big models for information extraction only requires an accurate prompt word for the target task. It is suitable for various complex information extraction tasks, greatly reducing the difficulty of implementing information extraction.

[0006] When faced with complex and diverse policy document contents, the information that needs to be extracted is different, which requires manual design of prompt words for different policy document contents. As we all know, the quality of prompt words is directly related to the effect of the large model's response. Since the large model is sensitive and consistent to the prompt words, sensitivity refers to the degree to which the large model responds to subtle changes in the prompt words. High sensitivity means that the behavior of the large model is extremely unstable and is more susceptible to the wording of the prompt words; low sensitivity means that the large model is more robust to subtle changes in the prompt words. Consistency refers to the similarity of the output distribution of the large model when it processes different samples of the same category. High consistency means that the large model can stably identify and process samples of the same category without producing very different results due to subtle differences in samples.

[0007] It can be seen that the quality of the prompt words is directly related to the stability and reliability of the results. The design of prompt words for information extraction of different policy contents is usually manually designed and then obtained through repeated experiments. Manually designed prompt words rely heavily on empirical knowledge and have obvious human subjectivity. Manually designed prompt words may be different from the prompt words used in the training of large models, and there are differences between the human problem-solving method and the neural network method. In other words, the interpretability of large models is still a topic that needs to be explored continuously.

[0008] When faced with information extraction from different models and different policy documents, it is time-consuming and laborious to manually try to design high-quality prompt words. Therefore, it is becoming increasingly important to design a method to automatically generate high-quality information extraction prompt words for different policy types. Summary of the invention

[0009] The present invention overcomes the shortcomings of the prior art and provides a policy document information extraction and dynamic generation prompt word system which realizes a method for automatically generating high-quality information extraction prompt words for different policy types.

[0010] The technical solution of the present invention is as follows:

[0011] A policy document information extraction and dynamic generation prompt word system, including a policy management module, a data set management module, a diversity screening module, a prompt management module, and a schema design module;

[0012] The policy management module automatically captures policy information from various official policy release platforms and unofficial platforms, deduplicates the collected policies, and stores the policy information in a structured form in the corresponding database;

[0013] The data set management module manually labels different types of policy information in the database and divides different types of policy information into training sets and test sets;

[0014] The diversity screening module forms a set of samples by concatenating the titles and contents of the policies in the training set, and then forms a two-dimensional array of embedded vectors by vectorizing each sample. The two-dimensional array is input into the K-Center-Greedy algorithm module program. Given the desired number of center samples, a representative sample result set of the given number will be output.

[0015] Among them, the goal of the K-Center-Greedy algorithm module program is to select a set of k samples so that it can best represent the entire data set;

[0016] Prompt management module, used to record the initial prompt words and historical prompt words in the entire automatic iteration process;

[0017] The schema design module uses the big model to generate new prompts and guides how to apply the new prompts it generates, optimizing and generating higher quality prompts based on examples of randomly selected representative policy samples.

[0018] Furthermore, the training set provides sufficient samples for the diversity screening module, and the test set samples are used to verify the quality of the prompt words.

[0019] Furthermore, it also includes a verification calculation module, which is responsible for evaluating the quality of the new prompt words output at the current iteration number.

[0020] Furthermore, the goal of the K-Center-Greedy algorithm module program is to select a set of k samples that can best represent the entire data set; that is, to select k samples from the complete data set T as the core set S so that the maximum distance between the data points in T and the nearest data points in S is minimized.

[0021] Furthermore, the vectorized representation is achieved through a policy embedding vector model, which can map high-dimensional data into a low-dimensional space to form a continuous vector representation, whereby similar data points will be closer in the low-dimensional space through the embedded vector.

[0022] Furthermore, the initial prompt word is a preset prompt word for different policy types, and is used to obtain an initial prompt word score by verifying the calculation module using the preset prompt word before the iteration starts;

[0023] The historical prompt words are used to record the specific content, score, and ranking calculated based on the score of the prompt words at the current iteration number in the iteration process.

[0024] Furthermore, the verification calculation module is responsible for evaluating the quality of the new prompt words output by the current iteration number, that is, designing a quantifiable calculation method based on the test set samples of the current policy type for the current policy type, and standardizing the score calculated by the calculation method to obtain the score of the new prompt words output by the current iteration number.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] The present invention quantifies policy information data through a diversity screening module and outputs a representative sample result set of a given number. The prompt words of the schema design module guide the large model on how to apply the new prompts it generates, and optimize and generate a higher quality prompt word based on the example of randomly selected representative policy samples, so as to better complete the current task and make the verification calculation score higher. This achieves accurate policy push, promotes the direct and quick enjoyment of policy dividends, and ensures that market entities know everything they should know about the latest policies and enjoy them quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the overall framework of the present invention;

[0028] Figure 2 It is a schematic diagram of the overall process framework of the diversity screening module of the present invention;

[0029] Figure 3 It is a schematic diagram of the process framework of the Prompt management module of the present invention;

[0030] Figure 4 A schematic diagram of the process framework of the verification calculation method of the present invention;

[0031] Figure 5 This is a schematic diagram of the Prompt automated optimization training process framework of the present invention. DETAILED DESCRIPTION

[0032] The embodiments of the present invention are described in detail below, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and cannot be used as a limitation of the present invention.

[0033] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0034] The numbers of the steps mentioned in each embodiment are only for the convenience of description and have no substantial sequence relationship. Different steps in each specific embodiment can be combined in different sequences to achieve the purpose of the invention.

[0035] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0036] like Figure 1-Figure 5 As shown, a policy document information extraction and dynamic prompt word generation system includes a policy management module, a data set management module, a diversity screening module, a prompt management module, and a schema design module.

[0037] The policy management module realizes multi-source collection and automatically captures policy information from various official policy release platforms and unofficial platforms to ensure the breadth and timeliness of information sources. It also deduplicates and integrates information, that is, deduplicates the collected policies to ensure the uniqueness of the policies in the database. Finally, the policy information is stored in a structured form in the database. The specific attributes of the policy include: source, type, title, content, validity period, etc.

[0038] The data set management module manually annotates different types of policy information in the database and divides different types of policy information into training sets and test sets. The annotated result is the key information that we want to extract from a given type of policy. The training set provides enough samples for the diversity screening module, and the test set samples are used to verify the quality of the prompt words.

[0039] The diversity screening module forms a group of samples by concatenating the titles and contents of the policies in the training set, and then vectorizes each sample through the policy embedding vector model to form a two-dimensional array of embedding vectors; the two-dimensional array is input into the K-Center-Greedy algorithm module program, and a representative sample result set of the given number of desired center samples is output.

[0040] Among them, the goal of the K-Center-Greedy algorithm module program is to select a set of k samples that can best represent the entire data set; that is, to select k samples from the complete data set T as the core set S so that the maximum distance between the data points in T and the nearest data points in S is minimized.

[0041] The K-Center-Greedy algorithm ensures the selection of instruction data with wide coverage and diversity. The specific steps are as follows:

[0042] (1) Initial center point selection: Select an initial center point from the training set, which is usually the data point farthest from other center points.

[0043] (2) Iteratively adding new center points: The data points farthest from the current center point set are then iteratively selected as new center points to ensure a wider coverage.

[0044] (3) Diversity and coverage: The algorithm iteratively selects the point farthest from the current center point set, which ensures that the selected data is as dispersed as possible and covers all aspects of instruction types, thus ensuring the diversity and coverage of the selected instructions.

[0045] The policy embedding vector model can map high-dimensional data into a low-dimensional space to form a continuous vector representation, which helps reduce the complexity of the data and the demand for computing resources while retaining key information. Through the embedding vector, similar data points will be closer in the low-dimensional space, which helps the model better understand the semantic relationship between the data. In natural language processing, similar words or sentences will be closer in the embedding space.

[0046] The prompt management module is used to record the initial prompt words and historical prompt words in the entire automatic iteration process. The initial prompt words are preset prompt words for different policy types, which are used to obtain the initial prompt word score through the verification calculation module using the preset prompt words before the iteration begins. The historical prompt words are mainly used to record the specific content, score, and ranking calculated based on the score of the prompt words at the current iteration number in the iteration process.

[0047] The schema design module is the core component of the entire iterative optimization process. It uses the big model as an optimizer to generate new prompt words based on the prompt template. The prompt template consists of two parts: one is the specific content and scores of a certain number of top-ranked historical prompts obtained from the Prompt Management Module, and the other is the representative policy sample content and expected annotation results obtained from the Diversity Screening Module.

[0048] Among them, the prompt words of the schema design module will guide the large model on how to apply the new prompts it generates, and optimize and generate a higher quality prompt word based on the examples of randomly selected representative policy samples, so as to better complete the current task and make the verification calculation score higher.

[0049] Preferably, a verification calculation module is also included, which is responsible for evaluating the quality of the new prompt words output at the current iteration number. The details are as follows:

[0050] A quantifiable calculation method based on the test set samples of the current policy type is designed for the current policy type, and the score calculated by the calculation method is standardized to obtain the score of the new prompt word output at the current iteration number.

[0051] Here, we take the tax policy as an example and push this policy to more types of enterprises. First, we need to extract enterprise-related tags from the policy content, which are called policy tags. Then, we match these policy tags with enterprise tags. The enterprise tags include registration type (state-owned enterprises, collective enterprises, private enterprises, limited liability companies, etc.) and industry type (automotive industry, banking industry, catering, services, real estate, etc.).

[0052] The labels of policies are usually different from those of enterprises. They can be matched by rule matching or fuzzy matching, but this method only matches the surface label content and cannot match the deep semantics. Here, the labels of policies and enterprises are vectorized, and then the cosine similarity between their respective vectors is calculated. In this way, the enterprise labels can be matched according to the threshold of cosine similarity. This method can judge whether the labels of policies and enterprises match from the deep semantics.

[0053] If the policy label can match the enterprise label under the specified cosine similarity threshold, the policy label is said to be able to recall the enterprise label. Otherwise, the policy label is said to be unable to recall the enterprise label.

[0054] The score is calculated by the ratio of the number of enterprise labels that can be recalled to the total number of policy labels. This ratio is normalized to a percentage integer to obtain the final score.

[0055] In summary, we will take tax policy as an example to illustrate the implementation steps as follows:

[0056] 1) Preprocessing step of tax policy information: obtain information related to tax preferential policies from public websites. This time, we took the source information, title, and content of 582 tax preferential policies, and used regular expressions to remove irrelevant characters such as web page tags, special characters, and extra spaces in the titles and contents.

[0057] The cleaned tax incentive policies were labeled with policy labels to obtain a label list corresponding to each policy. Then 552 tax incentive policies were randomly selected as training set samples, and the remaining 30 were used as test set samples.

[0058] 2) Preprocessing steps of enterprise label library: 2331 enterprise labels are obtained from the existing enterprise registration type code table and industry type code table. The registration types include: domestic enterprises, state-owned enterprises, collective enterprises, joint-stock cooperative enterprises, limited liability companies, joint-stock companies, etc. The industry types include computer, software and auxiliary equipment wholesale, application software development, real estate agency services, etc.

[0059] Considering the possibility of duplication of registration type and industry type, duplicate enterprise tags are merged to ensure the uniqueness of the tags. At the same time, the enterprise tags after deduplication are manually verified, such as deleting similar tags of other industries, removing redundant descriptive characters in the tags, etc. After the above processing, a total of 2085 enterprise tags were obtained.

[0060] 3) Steps for formulating the initial prompt: To extract information about the types of enterprises and industries involved in the tax incentive policy, write prompt words so that the big model can accurately extract the desired information from the policy content and return it in a specific format to facilitate subsequent result processing. The following is a specific initial prompt example:

[0061] """

[0062] ##Goal:

[0063] Given the announcement content of the tax incentives provided, focus on pinpointing the specific product names that can be developed from the relevant industries, sectors, and industries in the policy.

[0064] ## Output format example:

[0065] ["Industry 1","Industry 2","Industry 1","Product 1",...]

[0066] ##Example (based on hypothetical announcement content)

[0067] Example 1: Announcement content: For enterprises engaged in core products (technology) in key areas such as integrated circuits, artificial intelligence, biomedicine, and civil aviation in the new area

[0068] Output: ["Integrated Circuit", "Artificial Intelligence", "Biomedicine", "Civil Aviation"]

[0069] ##Notice:

[0070] 1. Strictly follow the output format to ensure that the output is in the form of a list.

[0071] """

[0072] 4) Prompt automatic optimization training steps: Tax preferential policy label extraction Prompt automatic optimization training process follows the steps of each module above. The overall training steps are as follows:

[0073] First, set the number of iterations, start the iteration, and create a json file of the prompt management module with the current training start time. The json file includes:

[0074] {"type":"tax"

[0075] "Initial Prompt":""

[0076] "HistoryPrompt":}

[0077] {"rank":1,

[0078] "Number of generations":0,

[0079] "content":""

[0080] "Score":0}

[0081] Start to determine whether the length of the historical prompt list is equal to 1. If it is equal to 1, use the initial prompt to extract labels for the test set policy information, and start the policy embedding vector model to vectorize the policy labels. There may be multiple policy labels here. The vectors of individual policy labels will be taken one by one to calculate the cosine similarity with the vectors of 2085 enterprise labels. The enterprise labels will be recalled with a cosine similarity of 0.6 as the threshold, and saved in the training record. Among them, when the cosine similarity value is greater than or equal to 0.6, it is defined that the policy label can recall the enterprise label (regardless of the number of enterprise labels matched by the policy label); when the cosine similarity value between the policy label vector and each enterprise label vector is less than 0.6, it is defined that the policy label cannot recall the enterprise label. Then count and calculate the score of the label, and update the initial prompt score in the historical prompt. Specifically, the number of enterprise labels that can be recalled and the number of enterprise labels that cannot be recalled in the extracted policy label list when the cosine similarity value is 0.6 will be counted. The final score calculation is as follows:

[0082] Score = (the number of enterprise labels that can be recalled / (the number of enterprise labels that can be recalled + the number of enterprise labels that cannot be recalled)) × 100.

[0083] If the length of the historical prompt list is not equal to 1, then take out the top 3 prompt contents and scores from the historical prompt list. Use the diversity screening module to take out 3 representative policy samples from the training set, construct the prompt of the schema design module, generate a new prompt through the large model, and perform verification calculations and statistical scores on the new prompt, and save it in the training record according to the current number of iterations. Then re-rank the new prompt and the historical prompt, and determine whether the current number of iterations is greater than the set number of iterations. If so, the iteration is terminated. If not, repeat the steps to take out the top 3 prompt contents and scores from the historical prompt list.

[0084] 5) Results and analysis steps: Baidu Wenxin Yiyan's ERNIE-4.0-8K and Zhipu AI's GLM-4 are used as Prompt optimizers, and the number of iterations is defined as 3. The following Tables 1 and 2 show their Prompt and corresponding scores under different numbers of iterations.

[0085]

[0086]

[0087] Table 1

[0088]

[0089]

[0090]

[0091] Table 2

[0092] It can be seen from Tables 1 and 2 that the large model, as an optimizer, can automatically optimize and generate prompts based on historical prompts, scores, and provided example samples, and can autonomously determine the direction of optimization without human intervention, so that the current prompt performs better on the current task.

[0093] In summary, this solution improves the quality of prompt words. The extracted prompt words are automatically optimized and designed based on the information of the policy content. The prompt words largely avoid relying on personal experience and knowledge, and are more objective and accurate. This method realizes the automatic generation of high-quality information extraction prompt words for different policy types.

[0094] The above is only a preferred embodiment of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and modifications without departing from the concept of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A system for extracting policy document information and dynamically generating prompt words, characterized by: Including policy management module, data set management module, diversity screening module, prompt management module, schema design module; The policy management module automatically captures policy information from various official policy release platforms and unofficial platforms, deduplicates the collected policies, and stores the policy information in a structured form in the corresponding database; The data set management module manually labels different types of policy information in the database and divides different types of policy information into training sets and test sets; The diversity screening module forms a set of samples by concatenating the titles and contents of the policies in the training set, and then forms a two-dimensional array of embedded vectors by vectorizing each sample. The two-dimensional array is input into the K-Center-Greedy algorithm module program. Given the desired number of center samples, a representative sample result set of the given number will be output. Among them, the goal of the K-Center-Greedy algorithm module program is to select a set of k samples so that it can best represent the entire data set; Prompt management module, used to record the initial prompt words and historical prompt words in the entire automatic iteration process; The schema design module uses the big model to generate new prompts and guides how to apply the new prompts it generates, optimizing and generating higher quality prompts based on examples of randomly selected representative policy samples.

2. A system for extracting policy document information and dynamically generating prompt words according to claim 1, characterized in that: The training set provides sufficient samples for the diversity screening module, and the test set samples are used to verify the quality of the prompt words.

3. A system for extracting policy document information and dynamically generating prompt words according to claim 1, characterized in that: It also includes a verification calculation module, which is responsible for evaluating the quality of the new prompt words output at the current iteration number.

4. A system for extracting policy document information and dynamically generating prompt words according to claim 1, characterized in that: The goal of the K-Center-Greedy algorithm module program is to select a set of k samples that can best represent the entire data set; that is, to select k samples from the complete data set T as the core set S so that the maximum distance between the data points in T and the nearest data points in S is minimized.

5. A system for extracting policy document information and dynamically generating prompt words according to claim 1, characterized in that: The vectorized representation is achieved through a policy embedding vector model, which can map high-dimensional data into a low-dimensional space to form a continuous vector representation, so that similar data points will be closer in the low-dimensional space through the embedded vector.

6. A system for extracting policy document information and dynamically generating prompt words according to claim 1, characterized in that: The initial prompt word is a preset prompt word for different policy types, and is used to obtain the initial prompt word score by the verification calculation module using the preset prompt word before the iteration starts; The historical prompt words are used to record the specific content, score, and ranking calculated based on the score of the prompt words at the current iteration number in the iteration process.

7. A system for extracting policy document information and dynamically generating prompt words according to claim 6, characterized in that: The verification calculation module is responsible for evaluating the quality of the new prompt words output by the current iteration, that is, designing a quantifiable calculation method based on the test set samples of the current policy type, and standardizing the scores calculated by the calculation method to obtain the scores of the new prompt words output by the current iteration.

Images