Rule-enhanced large language model auxiliary dense point labeling method and system
By using a rule-enhanced large language model to assist in dense point annotation, the problem of poor adaptability and security of existing dense point annotation methods is solved, achieving efficient and interpretable dense point annotation that can adapt to the dynamic requirements of density determination tasks.
Patent Information
- Application Number
- CN202511470922.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-15
- Publication Date
- 2026-01-23
AI Technical Summary
Existing intelligent dense point annotation methods are difficult to adapt to dynamically changing dense points, lack unified annotation standards, affect the efficiency and accuracy of density determination work, and pose security risks.
A rule-enhanced large language model-assisted dense point annotation method is adopted. Through four steps, namely data preprocessing, fine-tuning of density determination instructions, accumulation of density determination rules, and dense point annotation, a unified dense point annotation standard is constructed by combining the interpretability of large language models and rules, so as to realize the preservation and inheritance of expert experience.
It improves the accuracy and efficiency of dense point annotation, establishes a unified annotation standard, enhances the interpretability of large models, adapts to the ever-changing requirements of density determination tasks, and ensures data security.
Smart Images
Figure CN121390001A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of security management, and particularly relates to a rule-enhanced large language model assisted secret point labeling method and system. BACKGROUND
[0002] Secret classification work is the basic and source work of security work. People play an irreplaceable role in secret classification work. Classified files are generated by people, and the file classification level is ultimately determined by people. However, manual classification also has corresponding problems, such as difficulty in grasping the classification scale, difficulty in refining the classification granularity, and difficulty in inheriting the classification experience.
[0003] In existing intelligent secret point labeling work, methods based on statistical features or semantic features usually preset a secret point library or a secret point sentence library, and realize the labeling of secret point information in the text by calculating the statistical features of the secret text content such as TF_IDF and confidence, or the similarity between the text content and the secret point vector. The method based on the deep learning model needs to train the model by using the collected secret point information to realize the auxiliary classification of the document. However, the secret point dynamically changes with time and the development of the project, and the effect of the existing secret point labeling method is limited by the content covered by the secret point knowledge base, which is difficult to adapt to the constantly updated secret point. At the same time, these methods are difficult to establish a unified secret point labeling standard, and lack of explainability of classification. These problems not only affect the efficiency and accuracy of the classification work, but also increase the security risk. SUMMARY
[0004] In view of the deficiencies of the existing intelligent secret point labeling technology, the application provides a rule-enhanced large language model assisted secret point labeling method. The method combines the natural language understanding ability of the large language model (referred to as the large model) and the explainability of the rule, and designs four steps of data preprocessing, secret classification instruction fine-tuning, secret classification rule accumulation and secret point labeling to realize the secret point labeling of the classified file. The method can ensure the accuracy of the task, improve the efficiency of the secret point labeling work, and also summarize the secret classification rules according to the expert experience, construct a unified secret point labeling standard, improve the explainability of the large model labeling, and make the expert classification experience be preserved and inherited, which has important application value in the field of security management.
[0005] To achieve the above purpose, the technical scheme adopted by the application is:
[0006] According to one aspect of the application, a rule-enhanced large language model assisted secret point labeling method is provided, which includes the following steps:
[0007] The data preprocessing step: the secret text data is divided into paragraphs, and structured secret point manual labeling processing is performed to obtain a paragraph labeling data set including paragraph text, secret point, secret reason and secret level;
[0008] The secret marking instruction fine-tuning step: constructing a secret marking instruction template according to the secret data and the secret marking details, combining the paragraph annotation data set to obtain fine-tuning data, forming a secret point annotation instruction data set, and obtaining a secret point annotation model after fine-tuning of the large language model;
[0009] The secret marking rule accumulation step: using the secret point annotation model to annotate the secret text data, comparing the manual annotation and the model annotation results, guiding the large model to summarize, organize and verify the secret point annotation experience, forming a symbolic natural language rule, and finally storing it in the secret marking rule database;
[0010] The secret point annotation step: constructing an instruction template based on a few-shot prompt learning technology, searching the secret marking rule database and the paragraph annotation data set, guiding the large model to annotate the text to be annotated, thereby improving the annotation effect of the large model in the secret point annotation task.
[0011] As a further technical solution, the data preprocessing step includes: marking and structurally storing the secret text, including paragraph text, secret points, and secret levels, obtaining a paragraph annotation data set D, which is used for subsequent large model fine-tuning and generation of a secret marking rule database.
[0012] Further, the data preprocessing step includes:
[0013] Constructing a sequence for storing secret text paragraph data After paragraph segmentation of all texts in the secret text data set, store them in ; expert annotation is performed on , including paragraph text, secret points, secret reasons, and secret levels; if there are inconsistent annotated paragraphs, a third expert confirms; and a paragraph annotation data set is obtained.
[0014] Specifically, the data preprocessing step is as follows:
[0015] Step 1.1 Constructing a sequence for storing secret text paragraph data After paragraph segmentation of all texts in the secret text data set, store them in , and go to step 1.2;
[0016] Step 1.2 For each paragraph in , two experts perform annotation, including secret points in the paragraph and paragraph secret levels, wherein the information of the secret points includes content content, secret level level, and secret marking reason reason, and the secret level level includes first level (top secret), second level (secret), third level (secret), and no secret, and go to step 1.3;
[0017] Step 1.3 If the two experts' annotations differ in terms of classified points or classified levels, a third expert is called in to confirm, ensuring that each paragraph has a unique paragraph classified level and correct classified points. Finally, the annotation results are stored in the paragraph annotation dataset .
[0018] After data preprocessing, a structured paragraph set D storing all the classified point information will be obtained, including paragraph text, classified points, and classified levels.
[0019] As a further technical solution, the classified instruction fine-tuning step includes: constructing multiple classified task-related instruction templates (classified instruction templates), generating a classified point annotation instruction dataset based on the paragraph annotation dataset D , fine-tuning the large model (including but not limited to Qwen, DeepSeek, etc.) to obtain the classified point annotation model ISPA.
[0020] Further, the classified instruction fine-tuning step includes: setting a classified instruction template set I; for each paragraph and its annotation information in the paragraph annotation dataset constructing instruction data to obtain the classified point annotation instruction dataset; based on the classified point annotation instruction dataset, adjusting the hyperparameters of the large model according to the task , fine-tuning to obtain the classified point annotation model ISPA.
[0021] Specifically, the classified instruction fine-tuning step is as follows:
[0022] Step 2.1 According to the needs of the classified point annotation task, design a set of classified instruction templates I, as shown in Table 1, go to step 2.2;
[0023] Table 1 Classified instruction template example
[0024]
[0025] Step 2.2 For each paragraph and its annotation information in the paragraph set , fill in the prompt template in set I to obtain instruction data for fine-tuning.
[0026]
[0027] wherein, represents the paragraph adapted prompt template, , represents the classified point annotation information of the paragraph , and is a fill operator;
[0028] By combining different prompt templates and labeled data, a dense point labeling instruction data set is obtained , wherein each instruction data is converted to step 2.3.
[0029] Step 2.3 fine-tunes the large model using the constructed instruction data set, adjusts the hyperparameters of the model according to the task , to obtain the dense point labeling model ISPA.
[0030] As a further technical solution, the dense point rule accumulation step includes: by means of the large model, realizing rule generation, verification and summary on the data in the paragraph labeling data set D, to obtain the final dense point rule database R. Wherein, the rule generation, verification and summary includes: based on the paragraph labeling data set D and the designed rule accumulation instruction template, calling the large model to obtain the dense point rule; according to the dense point rule, the original paragraph is labeled and verified to obtain the rule verification result; analyze the relationship between the correct rules and the existing rules, and iterate and summarize the rules that produce redundancy or conflict to obtain the final dense point rule database R.
[0031] Further, the dense point rule accumulation step includes:
[0032] Using the dense point labeling model ISPA to label each paragraph in the test paragraph set;
[0033] From the paragraph labeling data set , the paragraph corresponding correct labeling result is obtained.
[0034] Based on the few-shot learning technology, a rule accumulation instruction template is constructed .
[0035] Guiding the large model to summarize the dense point labeling experience from the artificial labeling and model labeling results, and generating rules , wherein represents the th generated rule.
[0036] Using the rules in the rule to label and verify the corresponding paragraph, and eliminating the rules that are not correct in dense point labeling;
[0037] Judge whether the correct rules and existing rules have conflict or redundancy; if the rules have conflict or redundancy, store them in the conflict rule set or the redundancy rule set, construct the prompt word, and guide the large model to summarize the new rules from the conflicting or redundant rules to update .
[0038] If there is no conflict or redundancy in the rules, store the secret marking rules in the database .
[0039] Specifically, the secret marking rule accumulation step is as follows:
[0040] Step 3.1 uses the secret point marking model ISPA to mark each paragraph in the secret text paragraph data sequence For each paragraph in , first, build different prompt words, and use the ISPA model to extract secret information and mark secret points:
[0041]
[0042] Where S represents secret information, including project type T and secret point set M, C represents the result of model marking, and each secret point in the marking result includes content content, secret level level, and secret marking reason reason. After model marking, go to step 3.2;
[0043] Step 3.2 obtains the correct marking result corresponding to the paragraph from the paragraph marking data set , and goes to step 3.3;
[0044] Step 3.3 constructs a rule accumulation instruction template based on the idea of few-shot learning , as shown in Table 2, and goes to step 3.4;
[0045] Table 2 Rule accumulation instruction template
[0046]
[0047] Step 3.4 guides the large model to summarize secret point marking experience from artificial marking and model marking results, and generates rules :
[0048]
[0049] Wherein, represents the th generated rule. Go to step 3.5;
[0050] Step 3.5 checks the generated rules , constructs a prompt word combination paragraph and rule , calls the ISPA model to re-mark the original paragraph, and if the secret point marking is correct, it means that the rule is generated correctly, go to step 3.6, otherwise discard the rule, return to step 3.1;
[0051] Step 3.6 If there is a rule in the rule set or redundancy, it is determined whether it is related to any existing rule r e R according to the following formula:
[0052]
[0053] wherein, represents that the newly generated rule has a redundant relationship with the existing rule, represents that the newly generated rule has a conflict relationship with the existing rule, E represents embedding representation of the rule, represents the type (classification) of the rule. The similarity calculation formula is as follows:
[0054]
[0055] If the generated rule conflicts with the existing rule in the rule base, the rule is added to the conflict rule set ; if the generated rule is redundant with the rule in the rule base, the rule is added to the redundant rule set If and , that is, the number of elements in the set is 0, the new rule does not produce conflict or redundancy, and step 3.8 is converted; otherwise, step 3.7 is converted.
[0056] Step 3.7 Construct a prompt word to guide the large model to summarize a new rule from the conflicting or redundant rule update :
[0057]
[0058] Generate a new rule Still need to be checked, step 3.5-3.6 is converted.
[0059] Step 3.8 Store the rules in to the classified security rule database . If the paragraph annotation in is completed, return the final classified security rule database ; otherwise, return to step 3.1.
[0060] As a further technical solution, the classified security point annotation step comprises: constructing an instruction template based on a few-shot prompt learning technology, providing external knowledge for the classified security point annotation model ISPA by searching the paragraph annotation data set D and the classified security rule database R, and realizing paragraph classified security point annotation of the classified security file.
[0061] Further, the classified security point annotation step is specifically as follows:
[0062] Split the text F to be labeled by the user into a paragraph set Y ;
[0063] Each paragraph y in Y Retrieves similar text y from the paragraph labeling dataset D as an example ;
[0064] Uses the secret point labeling model ISPA to extract secret information from the paragraph y to obtain secret information S
[0065] Retrieves relevant rules from the secret classification rule database R according to the project type T and the secret point set M and stores them in the rule set
[0066] Designs a secret point labeling prompt template, splices the rule set rules and the example into the template, and the ISPA outputs the labeling result , which is stored in the labeling result list result
[0067] If the paragraph y is not the last paragraph in the text F, continue to retrieve similar text
[0068] If the paragraph y is the last paragraph in the text F, traverse all the labeling results in result, and take the highest secret level of the paragraph as the final secret level of the text F.
[0069] Specifically, the secret point labeling steps are as follows:
[0070] Step 4.1 After the user issues a secret point labeling request, pre-process the text F to be labeled sent by the user, split the file into a paragraph set Y , and go to step 4.2
[0071] Step 4.2 For each paragraph y in Y , retrieve similar text y from D. Specifically, calculate the similarity between
[0072]
[0073] and y according to the formula , sort them according to the similarity from high to low, and select the most similar paragraph and labeling result as an example , and go to step 4.3
[0074] Step 4.3 Uses the ISPA model to extract secret information from the paragraph y to obtain secret information S, and goes to step 4.4
[0075]
[0076] Step 4.4 Based on the type of items involved in the paragraph and collection of classified information In the database of confidentiality rules The relevant annotation rules are retrieved from the database. Specifically, the relevant rules are retrieved by calculating the relevance score between the classified information S and rule r.
[0077]
[0078] in, It is the relevance score between classified information and the rules; and Refers to project type and classified points In the rules The frequency in, among which ; and It refers to and Inverse document frequency; It is a rule Length; This represents the average length of all rules; and It is an adjustable parameter. It is usually between 1.2 and 2. It is usually 0.75.
[0079] Among them, project types Inverse document frequency The calculation formula is:
[0080]
[0081] in, For a set of rules The total number of rules in; It includes The number of rules. Classified points. Inverse document frequency Calculation process and same.
[0082] Ultimately based on Sort the rules, select the 5 most relevant rules and store them in the rule set, then proceed to step 4.5;
[0083] Step 4.5 Design a dense point annotation prompt template, as shown in Table 3, and combine the retrieved rule set and examples. The paragraphs are then stitched into the template and finally output by the ISPA model. Annotation results , and the result is stored in a result list result. If the paragraph is the last paragraph in the text F, go to step 4.6; otherwise, go to step 4.2.
[0084] Table 3: Classified point labeling prompt template
[0085]
[0086] Step 4.6: Traverse all the labeling results in result, and take the highest classified level of the paragraph as the final classified level of the text F, and complete the classified point labeling.
[0087] According to an aspect of the present application, a rule-enhanced large language model assisted classified point labeling system is provided, comprising:
[0088] A data preprocessing module: used for splitting classified text data into paragraphs and performing structured classified point manual labeling processing to obtain a paragraph labeling data set including paragraph text, classified points, classified reasons, and classified levels;
[0089] A classified instruction fine-tuning module: used for manually constructing a task instruction template according to classified data, classified details, and expert experience, combining the paragraph labeling data set to obtain a classified point labeling instruction data set, and obtaining a classified point labeling model after fine-tuning of a large language model;
[0090] A classified rule accumulation module: used for comparing manual labeling and model labeling results to guide the large model to summarize, organize, and verify classified point labeling experience, form symbolic natural language rules, and finally store them in a classified rule database;
[0091] A classified point labeling module: used for constructing an instruction template based on a few-shot prompt learning technology, searching the classified rule database and the paragraph labeling data set to guide the large model to perform classified point labeling on the text to be labeled, and improving the labeling effect of the large model in the classified point labeling task.
[0092] According to an aspect of the present application, a large language model based auxiliary classified point labeling device is provided, comprising a memory and a processor, the memory stores program instructions executed by the processor, and the processor calls the program instructions to perform the above-mentioned large language model based auxiliary classified point labeling.
[0093] According to an aspect of the present application, a non-transitory computer readable storage medium is provided, which stores computer instructions, and the computer instructions make the computer execute the above-mentioned large language model based auxiliary classified point labeling.
[0094] According to one aspect of the present invention, a computer program product is provided, wherein instructions in the computer program product, when executed by a processor of an electronic device, cause the electronic device to perform the aforementioned auxiliary dense point annotation based on a large language model.
[0095] Beneficial effects:
[0096] The advantages of this invention are as follows: First, the classification rules explicitly represent expert classification experience in the form of natural language, establishing a unified standard for classification point annotation. This provides a basis and constraints for the classification point annotation task of large models, improving the accuracy of classification point annotation. Simultaneously, it also enables the preservation and inheritance of expert experience. Second, the large model has strong natural language processing capabilities, enabling rapid updates and optimization of classification rules to adapt to the ever-changing needs of classification tasks. Third, the method combines manual work with the large model to fuse classification point information, integrating the large model and rules. This ensures that classified content is stored in the rule base rather than in the model parameters, preventing attacks on the large model from leading to the leakage of classified content. Fourth, it eliminates the need to pre-inject prior knowledge into the large model. Instead, it identifies and accumulates effective rules through manual feedback and model-summarized annotation experience, dynamically optimizing the annotation rules and generating different rule bases. This method enables plug-and-play rule bases to adapt to classification point annotation tasks in different industries and fields. Furthermore, it also enables the large model to be adapted to tasks in other fields. Compared with existing methods, the method of the present invention has higher annotation accuracy, stronger interpretability and data security, and can meet the needs of practical dense point annotation applications. Attached Figure Description
[0097] Figure 1 This is a flowchart illustrating the implementation of the present invention;
[0098] Figure 2 This is a flowchart of the data preprocessing process of the present invention;
[0099] Figure 3 This is a flowchart of the fine-tuning process for the security command of the present invention;
[0100] Figure 4 This is a flowchart illustrating the cumulative confidentiality rules of the present invention.
[0101] Figure 5 This is a flowchart of the dense dot annotation process of the present invention. Detailed Implementation
[0102] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort are within the protection scope of the present application.
[0103] According to an embodiment of the present application, an auxiliary secret point labeling method based on a large language model is provided, as shown in Figure 1 , comprising the following steps:
[0104] Data preprocessing step: select patent data, design simulation secret point details, and label secret point information manually, split the data into paragraphs, and obtain paragraph labeling data set by manual labeling , so as to realize automatic rule mining and generation. Referring to Figure 2 , the specific steps of the data preprocessing step are as follows:
[0105] Step 1.1: Construct a sequence for storing secret-related text paragraph data After the paragraph division of all the text in the secret-related text data set is completed, store it in , and go to step 1.2;
[0106] Step 1.2: For each paragraph in , two experts label the secret-related points and the paragraph secret level, wherein the information of the secret-related points includes content, secret level and secret classification reason, and the secret level includes first class (top secret), second class (secret), third class (secret) and non-secret, and go to step 1.3;
[0107] Step 1.3: If the two experts label different secret-related points or different secret levels, a third expert is invited to confirm, so as to ensure that each paragraph has a unique paragraph secret level and correct secret-related points. Finally, the labeling result is stored in the paragraph labeling data set .
[0108] After the data preprocessing is completed, a structured paragraph labeling data set D storing all the labeled secret point information is obtained, which includes paragraph text, secret-related points and secret level.
[0109] The secret instruction fine-tuning step is used to solve the problems that the basic large model does not understand the secret point labeling task and the secret level classification is not accurate. In the secret point labeling process, the error rate of the secret point labeling of the general large model is high, the labeling accuracy is low, the secret point provided in the labeling process is fuzzy and almost covers all the text, and the secret level labeling of the secret point has randomness and no definite basis and rules. In order to enable the model to have specific field ability, understand the secret point labeling task and the secret management knowledge, it is necessary to fine-tune the basic model to inject knowledge into the model.
[0110] According to the actual needs of the secret point labeling task, the fine-tuning instruction data set is constructed, the large language model is fine-tuned, and the understanding ability of the large model to the secret point labeling task is enhanced.
[0111] Specifically, the secret instruction fine-tuning step is as follows, see Figure 3 :
[0112] Step 2.1, according to the needs of the secret point labeling task, a set of secret instruction templates I is designed, for example, as shown in Table 1, go to step 2.2;
[0113] Table 1 Secret instruction template example
[0114]
[0115] Step 2.2, for each paragraph and its labeling information in the paragraph set (paragraph labeling data set) , fill in the prompt template in set I to obtain the instruction data for fine-tuning.
[0116]
[0117] Wherein, represents the paragraph adapted prompt template, , represents the secret point labeling information of the paragraph , and is a filling operator;
[0118] By combining different prompt templates and labeling data, a secret point labeling instruction data set is obtained, wherein each instruction data, go to step 2.3;
[0119] Step 2.3, fine-tune the large model using the constructed secret point labeling instruction data set, adjust the hyperparameters of the large model (such as Top-k, Top-p, Temperature, etc.) according to the secret point labeling task to obtain a secret point labeling model ISPA.
[0120] The secret marking rule accumulation step is used to guide the large model to summarize the secret point marking rules. In the marking process, the large model is provided with correct or incorrect marking instances through artificial feedback, and the large model continuously summarizes the marking experience according to these feedbacks, identifies and accumulates effective rules, so as to improve the marking ability and robustness of the large model. The rule accumulation does not need to inject priori knowledge into the large model in advance, but dynamically optimizes the marking rules through artificial feedback and model self-summarization, and can adapt to the changing secret point marking requirements. In addition, the rule accumulation step can form natural language description of the marking rules, establish a unified marking standard, and is beneficial to the retention and inheritance of expert secret marking experience.
[0121] The secret marking rule accumulation step is specifically as follows, please refer to Figure 4 :
[0122] Step 3.1, using the secret point marking model ISPA to mark each paragraph in the secret text paragraph data sequence . For each paragraph in , first, build different prompt words, and use the ISPA model to extract secret information and mark secret points:
[0123]
[0124] Wherein, S represents secret information, including project type T and secret point set M, C represents the result of model marking, and each secret point in the marking result includes content content, secret level level and secret marking reason reason. After the model marking is completed, go to step 3.2;
[0125] Step 3.2, obtaining the artificial marking correct marking result corresponding to the paragraph from the paragraph marking data set (step 1 artificial marking result), go to step 3.3;
[0126] Step 3.3, constructing a rule accumulation instruction template based on the idea of few-shot learning , as shown in Table 2, go to step 3.4;
[0127] Table 2 Rule accumulation instruction template
[0128]
[0129] Step 3.4, guiding the large model to summarize secret point marking experience from artificial marking and model marking results, and generating rules :
[0130]
[0131] Wherein, represents the i-th rule, and the rule is generated by the large model according to the rule accumulation instruction template and the artificial marking correct marking result. LLM represents a large closed-source language model with large parameters, such as ChatGPT, etc. Step 3.5;
[0132] Step 3.5 verifies the generated rule , constructs a prompt word combination paragraph and rule , and calls the ISPA model to re-label; If the point labeling is correct, it means that the rule generation is correct, go to step 3.6, otherwise discard the rule and return to step 3.1;
[0133] Step 3.6 For the case of conflict or redundancy of rules in the rule set, it is judged whether it is related to any existing rule r∈R according to the following formula:
[0134]
[0135] Among them, indicates that the newly generated rule has a redundant relationship with the existing rule, indicates that the newly generated rule has a conflict relationship with the existing rule, E represents the embedding representation of the rule, indicates the type of rule (secret level). The similarity calculation formula is as follows:
[0136]
[0137] If the generated rule conflicts with the rules in the rule base, add the rule to the conflict rule set ; If the generated rule produces redundancy with the rules in the rule base, add the rule to the redundancy rule set If and , the new rule does not produce conflict or redundancy, go to step 3.8; Otherwise, go to step 3.7;
[0138] Step 3.7 Construct a prompt word to guide the large model to summarize a new rule from the conflicting or redundant rule Update :
[0139]
[0140] Generate a new rule still needs to be verified, go to step 3.5-3.6.
[0141] Step 3.8 Store the rules in to the secret rule database . If the paragraph labeling in is complete, return the final secret rule database Otherwise, return to step 3.1.
[0142] The dense point annotation step involves retrieving external knowledge related to the paragraph from a pre-established knowledge base. Specifically, it involves using cosine similarity to retrieve similar annotation data from already annotated paragraphs and from a density rule database. The rules corresponding to the paragraphs to be labeled are retrieved from the database, guiding the large model to follow the relevant labeling rules and imitate the examples to perform dense point labeling. The paragraph labeling dataset D obtained in the previous steps and the density rule database are then combined. As external knowledge, it helps the model to better complete the dense point annotation task.
[0143] The specific steps for dense dot annotation are as follows, see details below. Figure 5 :
[0144] Step 4.1 After the user sends a dense point annotation request, preprocess the text F to be annotated sent by the user, splitting the file into a set of paragraphs. Proceed to step 4.2;
[0145] Step 4.2 For each paragraph in the paragraph set Y To retrieve similar text y from D, specifically, using the formula...
[0146]
[0147] calculate The similarity between y and y is sorted from highest to lowest, and the most similar paragraphs and their annotations are selected as examples. Proceed to step 4.3;
[0148] Step 4.3 Use the ISPA model to analyze paragraphs Extract classified information to obtain classified information S, then proceed to step 4.4;
[0149]
[0150] Step 4.4 Based on the type of items involved in the paragraph and collection of classified information In the database of confidentiality rules The relevant annotation rules are retrieved from the database. Specifically, the relevant rules are retrieved by calculating the relevance score between the classified information S and rule r.
[0151]
[0152] in, It is the relevance score between classified information and the rules; and Refers to project type and classified points In the rules The frequency in, among which ; and It refers to and Inverse document frequency; It is a rule Length; This represents the average length of all rules; and It is an adjustable parameter. It is usually between 1.2 and 2. It is usually 0.75.
[0153] Among them, project types Inverse document frequency The calculation formula is:
[0154]
[0155] in, For a set of rules The total number of rules in; It includes The number of rules. Classified points. Inverse document frequency Calculation process and same.
[0156] Ultimately based on Sort the rules, select the 5 most relevant rules and store them in the rule set, then proceed to step 4.5;
[0157] Step 4.5 Design a dense point annotation prompt template, as shown in Table 3, and combine the retrieved rule set and examples. The paragraphs are then stitched into the template and finally output by the ISPA model. Annotation results Store the annotation results in the `result` list. If the paragraph... If it is the last paragraph in text F, proceed to step 4.6; otherwise, proceed to step 4.2.
[0158] Table 3. Dense Dot Annotation Template
[0159]
[0160] Step 4.6 Iterate through all the annotation results in result, take the highest security level of the paragraph as the final security level of text F, and complete the security point annotation.
[0161] Based on the same inventive concept as the foregoing embodiments, the embodiments of the present application also provide a rule-enhanced large language model assisted secret point labeling system, comprising:
[0162] A data preprocessing module is configured to split classified text data into paragraphs and perform structured secret point manual labeling processing to obtain a paragraph labeling data set including paragraph text, secret points, secret reasons, and secret levels;
[0163] A classified instruction fine-tuning module is configured to construct a classified instruction template based on classified data and classified items, combine the paragraph labeling data set, obtain a secret point labeling instruction data set, and obtain a secret point labeling model after fine-tuning of a large language model;
[0164] A classified rule accumulation module is configured to compare manual labeling and model labeling results, guide the large model to summarize, organize, and verify secret point labeling experience, form symbolic natural language rules, and finally store the rules in a classified rule database;
[0165] A secret point labeling module is configured to construct an instruction template based on a few-shot prompt learning technique, search the classified rule database and the paragraph labeling data set, guide the large model to label secret points of a text to be labeled, and improve the labeling effect of the large model in the secret point labeling task.
[0166] Based on the same inventive concept as the foregoing embodiments, the embodiments of the present application also provide a large language model based auxiliary secret point labeling method and device, comprising a memory and a processor, the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the large language model based auxiliary secret point labeling method.
[0167] Based on the same inventive concept as the foregoing embodiments, the embodiments of the present application also provide a non-transitory computer readable storage medium storing computer instructions, which cause the computer to execute the large language model based auxiliary secret point labeling method.
[0168] Based on the same inventive concept as the foregoing embodiments, the embodiments of the present application also provide a computer program product, the instructions in the computer program product are executed by a processor of an electronic device, so that the electronic device executes the large language model based auxiliary secret point labeling method.
[0169] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In one
[0170] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks.
[0171] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks.
[0172] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks.
[0173] While the forgoing specification illustrates illustrative embodiments of the present application, it will be apparent to those skilled in the art that various modifications can be made within the scope of the present application, and the present application is not intended to be limited to the above described embodiments. Rather, it is intended that the present application cover modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.
Claims
1. A rule-enhanced method for dense point annotation in large language models, characterized in that, Includes the following steps: Data preprocessing: The classified text data is split into paragraphs and structured manual annotation of classified points is performed to obtain a paragraph annotation dataset that includes paragraph text, classified points, reasons for classification, and classification level; Fine-tuning of classification instructions: Based on classified data and classification details, a classification instruction template is constructed. Combined with the paragraph annotation dataset, a dense point annotation instruction dataset is obtained. After fine-tuning the large model, a dense point annotation model is obtained. Accumulation of classification rules: Use the key point annotation model to annotate classified text data, compare the results of manual annotation and model annotation, guide the large model to summarize, organize and verify key point annotation experience, form symbolic natural language rules, and finally store them in the classification rule database; Dense dot annotation: Based on few-shot cue learning technology, an instruction template is constructed. By searching the density rule database and paragraph annotation dataset, the large model is guided to perform dense dot annotation on the text to be annotated.
2. The rule-enhanced large language model-assisted dense point annotation method according to claim 1, characterized in that: The data preprocessing steps include: Construct a sequence to store classified text paragraph data The classified text dataset is segmented into paragraphs and then stored. middle; right Expert annotations were provided, including paragraph text, classified information, reasons for classification, and classification level. If there are inconsistent paragraphs, they will be confirmed by a third expert; Obtain the paragraph annotation dataset.
3. The rule-enhanced large language model-assisted dense point annotation method according to claim 1, characterized in that: The fine-tuning steps for the security declaration instruction include: Set the set of classified instruction templates I; For each paragraph and its annotation information in the paragraph annotation dataset Construct instruction data to obtain a densely labeled instruction dataset; Based on a dense point annotation instruction dataset, the hyperparameters of a large model are adjusted according to the task. Fine-tuning yields the dense point annotation model ISPA.
4. The rule-enhanced large language model-assisted dense point annotation method according to claim 1, characterized in that: The steps for accumulating confidentiality rules include: using the paragraph annotation dataset D and the designed rule accumulation instruction template, calling the large model to obtain initial confidentiality rules; verifying the annotations of the original paragraphs according to the confidentiality rules to obtain rule verification results; analyzing the relationship between the verified rules and existing rules, iteratively summarizing the rules that generate redundancy or conflict, and obtaining the final confidentiality rule database R.
5. The rule-enhanced large language model-assisted dense point annotation method according to claim 4, characterized in that: The accumulation step of the confidentiality rule includes: The dense dot annotation model ISPA was used to annotate each paragraph in the test paragraph set; Obtain paragraphs from paragraph annotation dataset D. The corresponding correct annotation result ; Constructing a rule-accumulating instruction template based on few-shot learning techniques ; Guide large models to summarize dense point annotation experience from manual and model annotation results, and generate rules. ,in, Indicates the first One generated rule; Utilizing rules The rules in the document are used to mark and validate the corresponding paragraphs; Determine if the validated rule conflicts with or is redundant with existing rules: If a rule conflicts or is redundant, store it in the conflict rule set or redundant rule set, construct prompt words, and guide the large model to summarize new rules from the conflicting or redundant rules for update. ; If there are no conflicts or redundancies in the rules, they are stored in the fixed-secret rule database. .
6. The rule-enhanced large language model-assisted dense point annotation method according to claim 1, characterized in that: The dense dot annotation step includes: Split the text F to be annotated into a set of paragraphs. ; Each paragraph in Y Retrieve similar text y from paragraph annotation dataset D as examples. ; The dense dot annotation model ISPA for paragraphs Classified information is extracted to obtain classified information S; Based on the project type T and the set of classified points M, relevant rules are retrieved from the classification rule database R and stored in the rule set; Design a dense point annotation prompt template, including the rule set and examples. When spliced into the template, ISPA outputs the annotation results. Store the annotation results in the result list; If paragraph If it is not the last paragraph in text F, return to continue searching for similar text; If paragraph If it is the last paragraph in text F, then iterate through all the labeled results in result and take the highest security level of the paragraph as the final security level of text F.
7. A rule-enhanced large language model-assisted dense point annotation system, characterized in that, include: Data preprocessing module: used to split classified text data into paragraphs and perform structured manual annotation of classified points to obtain a paragraph annotation dataset including paragraph text, classified points, reasons for classification, and classification level; Classification Instruction Fine-tuning Module: This module is used to construct classification instruction templates based on classified data and classification details, combine them with paragraph annotation datasets to obtain a dense point annotation instruction dataset, and then fine-tune the large model to obtain a dense point annotation model. The density rule accumulation module is used to guide large models to summarize, organize, and verify the density point annotation experience by comparing the results of manual annotation and model annotation, forming symbolic natural language rules, and finally storing them in the density rule database. Dense dot annotation module: Used to build instruction templates based on few-shot cue learning technology. By searching the density rule database and paragraph annotation dataset, it guides the large model to annotate the text to be annotated with dense dots.
8. A rule-enhanced large language model-assisted dense point annotation device, characterized in that, The method includes a memory and a processor, wherein the memory stores program instructions that are executed by the processor, and the processor invokes the program instructions to execute a rule-enhanced large language model-assisted dense point annotation method as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to execute a rule-enhanced large language model-assisted dense point annotation method as described in any one of claims 1 to 6.
10. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs a rule-enhanced large language model-assisted dense point annotation method as described in any one of claims 1 to 6.