Data labeling method and device based on rule engine and large language model cooperation

Abstract

The application discloses a data labeling method and device based on rule engine and large language model cooperation. The method comprises the following steps: analyzing inputted text features to be labeled and business requirement parameters; automatically selecting one from multiple modes including only rule labeling, only large language model labeling, rule first and large model second labeling, and large model first and rule second labeling, according to the text features to be labeled and the business requirement parameters by a decision engine; and labeling the text to be labeled by using the selected mode and outputting a result file. Wherein, the rule engine performs rigid verification and matching based on dictionary classification, sentence breaking rules, black and white lists, dictionary order, bottom-up rules and confidence; the large language model module performs semantic understanding and label prediction based on prompt words and knowledge base reference examples, and extracts labeling reasons and original text basis. The application realizes flexible deployment and cooperation of multiple labeling modes, and takes into account labeling efficiency and recognition accuracy.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and data annotation technology, specifically relating to a data annotation method and apparatus based on the collaboration of a rule engine and a large language model. Background Technology

[0002] The information disclosure element data production process includes obtaining announcement documents from information disclosure data sources, sequentially completing announcement cleaning, data structuring, data labeling, and finally outputting interface services. Currently, data labeling mainly uses rule-based methods such as dictionary matching, regular expression matching, and logical rule matching to tag the data and output structured tagged data.

[0003] However, existing rule-based data annotation technologies have the following problems in practical applications: 1. Risks of missed or incorrect identification: There are many existing matching rules, all of which are formed manually based on daily supervision work and are highly dependent on human experience, which can easily lead to risks of missed or incorrect identification. At the same time, the current rules only support single-sentence dimension matching and identification, and cannot be combined with the context and the overall semantics of the paragraph for comprehensive judgment, resulting in insufficient semantic recognition accuracy and thus causing prominent problems of missing labels.

[0004] 2. Insufficient universality and scenario adaptability: Due to individual differences in the summary standards and understanding of personnel, the rule system lacks a unified standard and cannot adapt well to complex and ever-changing business scenarios.

[0005] 3. Labeling basis cannot be explained to the public in a standardized manner: Traditional rule matching methods lack a standardized explanation mechanism for the labeling basis and rule generation, resulting in the labeling basis being unable to be explained to the public in a standardized manner and lacking transparency. Summary of the Invention

[0006] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form to prepare for the more detailed descriptions that follow.

[0007] The purpose of this invention is to solve the above-mentioned problems and provide a data annotation method and apparatus based on the collaboration of a rule engine and a large language model. It can support the on-demand switching and combination of two annotation modes, accurately extract the original text basis with the help of the large language model and automatically generate annotation basis to achieve traceability of results, use the initial annotation of the large language model combined with the rigid verification of the rule engine to meet compliance review requirements, rely on the semantic understanding and generalization ability of the large language model to properly cover boundary scenarios and reduce omissions and errors in annotation, and flexibly allocate the processing capabilities of rule annotation and large language model to balance data annotation efficiency and recognition accuracy.

[0008] The technical solution of this invention is as follows: This invention discloses a data annotation method based on the collaboration of a rule engine and a large language model, comprising the following steps: Analyze the features of the input text to be labeled and the parameters of business requirements; The decision engine automatically selects an annotation method from multiple annotation methods based on the features of the text to be annotated and the business requirement parameters; The selected annotation method is used to annotate the text to be annotated, and an annotation result file is output. If the mode including rule annotation is selected, the rule engine is used for annotation. The rule engine defines the range of tags to be matched, splits the text to be annotated into annotation statements, constructs regular expressions, and uses the regular expressions to match the annotation statements to obtain candidate annotation tags. A fallback rule is used to determine whether to trigger the fallback logic to output the corresponding annotation result. If the mode including large language model annotation is selected, the large language model annotation module is used for annotation. The data annotation task is determined based on pre-configured system prompt words, and semantic understanding and tag prediction are performed on the data to be annotated based on user prompt words and reference examples containing preset tags and annotation examples in the knowledge base to obtain annotation results.

[0009] According to an embodiment of the data annotation method based on the collaboration of rule engine and large language model according to the present invention, the multiple annotation methods include: rule-only annotation mode, large language model-only annotation mode, rule-first annotation followed by large language model annotation mode, and large language model-first annotation followed by rule annotation mode.

[0010] According to an embodiment of the data annotation method based on the collaboration of rule engine and large language model of the present invention, the output fields of the annotation result file include: source data primary key, source data content, tag primary key, tag name, annotation reason and annotation original text basis.

[0011] According to an embodiment of the data annotation method based on the collaboration of a rule engine and a large language model according to the present invention, the selection process of the decision engine is as follows: When the text to be labeled has a standardized structure, clear keywords, and the confidence level of the rule base in judging this type of sample exceeds a preset threshold, select the rule-only labeling mode; When the semantics of the text to be labeled are open and the rule base has zero coverage of this type of text, select the large language model only labeling mode; When candidate labels can be quickly filtered through the rule base, but the judgment boundary is ambiguous and the labeling basis is required, choose the rule labeling first and then large language model labeling mode. When business operations have constraints that require initial labeling results to be generated from a large language model and then subject to compliance review by rule annotation, the mode of first annotating the large language model and then annotating the rules should be selected.

[0012] According to an embodiment of the data annotation method based on the collaboration of a rule engine and a large language model according to the present invention, the processing of the rule engine is as follows: Determine the label range: Based on dictionary classification, clarify the label range corresponding to the text to be labeled; Split annotation sentences: Split the text to be annotated into annotation sentences according to the punctuation marks in the sentence segmentation rules; Blacklist filtering: Determine whether there are prohibited phrases in the marked statement based on the dictionary blacklist; if so, discard the marked statement. Dictionary matching and regular expression matching: If no prohibited phrases exist, determine whether there is an order requirement based on the dictionary order. If there is an order requirement, match the dictionary in order; and combine the dictionary whitelist with regular expression matching for tags. Fallback rule determination: Determine whether a fallback tag is involved based on the fallback rule; if so, further determine whether a parent tag exists and whether it is the same parent tag; if it is the same parent tag, the fallback tag is not executed; if not, the tag is matched according to regular expressions.

[0013] According to an embodiment of the data annotation method based on the collaboration of rule engine and large language model of the present invention, the processing of the rule engine further includes determining whether to output the corresponding annotation result based on the confidence level.

[0014] This invention also discloses a data annotation device based on the collaboration of a rule engine and a large language model, the device comprising: The text feature and business requirement analysis module is used to analyze the features and business requirement parameters of the input text to be labeled. The annotation mode selection module is used by the decision engine to automatically select an annotation mode from multiple annotation methods based on the features of the text to be annotated and business requirement parameters; The multi-mode collaborative annotation and result output module is used to annotate the text to be annotated using the selected annotation method and output the annotation result file. If the mode including rule annotation is selected, the rule engine is used for annotation. The rule engine defines the range of tags to be matched, splits the text to be annotated into annotation statements, constructs regular expressions, and uses these regular expressions to match the annotation statements to obtain candidate annotation tags. A fallback rule is used to determine whether to trigger fallback logic, and a confidence level is used to determine whether to output the corresponding annotation result. If the mode including large language model annotation is selected, the large language model annotation module is used for annotation. The module determines the data annotation task based on pre-configured system prompts and performs semantic understanding and tag prediction on the data to be annotated based on user prompts and reference examples containing preset tags and annotation examples in the knowledge base, to obtain the annotation result.

[0015] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the data annotation method based on the collaborative operation of a rule engine and a large language model as described above.

[0016] The present invention also discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data annotation method based on the collaboration of a rule engine and a large language model as described above.

[0017] The present invention also discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the data annotation method based on the collaborative approach of a rule engine and a large language model as described above.

[0018] Compared with the prior art, the present invention has the following advantages: 1. This invention discloses a data annotation method that integrates a rule engine and a large language model, supporting multiple annotation modes (including rule-only annotation mode, large language model-only annotation mode, rule-first annotation followed by large language model annotation mode, and large language model-first annotation followed by rule annotation mode) that can be switched and combined as needed. In business applications, both capabilities can be flexibly deployed: rule annotation handles high-certainty samples, while the large language model processes complex text, balancing annotation efficiency and recognition accuracy.

[0019] 2. Compared to pure rule-based annotation schemes, this invention leverages a large model to accurately extract the original textual basis (i.e., the basis for annotation) to achieve traceability of annotation results. It also automatically generates annotation basis (i.e., annotation reasons) based on laws, regulations, and business specifications, ensuring the annotation process is transparent and verifiable. Simultaneously, relying on the semantic understanding capabilities of the large model, it effectively covers boundary scenarios such as synonymous expressions, implicit logic, and multiple conditional intersections that rules cannot exhaustively cover, reducing missed and incorrect annotations and improving annotation accuracy. Furthermore, the large model possesses strong generalization capabilities, adapting to newly added semantic scenarios and reducing the manual maintenance costs of the rule base.

[0020] 3. This invention supports a sequential process of initial model specification followed by rule engine review, relying on rigid rule verification as a fallback to meet the hard standards of compliance review. Attached Figure Description

[0021] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.

[0022] Figure 1 A flowchart of an embodiment of the data annotation method based on the collaboration of rule engine and large language model of the present invention is shown.

[0023] Figure 2 A schematic diagram of the processing flow of the decision engine of the present invention is shown.

[0024] Figure 3 A schematic diagram of the processing flow of the rule engine of the present invention is shown.

[0025] Figure 4 A schematic diagram of an embodiment of the data annotation device based on the collaboration of a rule engine and a large language model of the present invention is shown. Detailed Implementation

[0026] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should be noted that the aspects described below with reference to the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0027] Figure 1 The flowchart of an embodiment of the data annotation method based on the collaborative operation of a rule engine and a large language model according to the present invention is shown. See also Figure 1 The implementation steps of the data annotation method in this embodiment are as follows.

[0028] First, analyze the features of the input text to be labeled and the parameters of business requirements.

[0029] The text to be annotated has characteristics such as Markdown format or structured / unstructured documents, such as listed company announcements or inquiry letters. Business requirement parameters are usually configured by the user.

[0030] The processing of analyzing the features of the text to be annotated mainly involves: Text structure level: Evaluate the standardization of the text layout, formatting, and contextual logic of the text to be annotated; Keyword clarity and semantic openness: Identify whether the text contains explicit industry / business keywords, or whether the text semantics are relatively open and require deep semantic understanding; Rule base coverage and decision confidence: Evaluate the extent to which the existing rule base covers this type of text sample (e.g., whether the coverage is zero), and whether the decision confidence of the rule base in labeling and predicting this type of sample reaches or exceeds a preset threshold.

[0031] The processing of business requirement parameters mainly involves: Compliance review requirements: Analyze whether the business scenario imposes strict compliance constraints or hard review standards on the annotation results (such as whether a high level of compliance review is required). Provide annotation basis requirements: Analyze whether the business requirements require the output of detailed annotation reasons and original text basis (such as whether traceability and explanation are required).

[0032] Then, the decision engine automatically selects an annotation method from multiple annotation methods based on the features of the text to be annotated and the business requirement parameters.

[0033] In this embodiment, as Figure 2 As shown, various annotation methods include: rule-only annotation mode, large language model-only annotation mode, rule-first annotation followed by large language model annotation mode, and large language model-first annotation followed by rule-first annotation mode.

[0034] The decision engine's selection process is as follows: When the text to be labeled has a standardized structure, clear keywords, and the confidence level of the rule base in judging this type of sample exceeds a preset threshold, select the rule-only labeling mode; When the semantics of the text to be labeled are open and the rule base has zero coverage of this type of text, select the large language model only labeling mode; When candidate labels can be quickly filtered through the rule base, but the judgment boundary is ambiguous and the labeling basis is required, choose the rule labeling first and then large language model labeling mode. When business operations have constraints that require initial labeling results to be generated from a large language model and then subject to compliance review by rule annotation, the mode of first annotating the large language model and then annotating the rules should be selected.

[0035] Finally, the selected annotation method is used to annotate the text to be annotated, and the annotation result file is output.

[0036] The output fields of the annotation result file include: source data primary key, source data content, tag primary key, tag name, annotation reason, and original text basis for annotation.

[0037] If the mode including rule annotation is selected, the rule engine is used for annotation. The rule engine defines the range of tags to be matched, splits the text to be annotated into annotation statements, constructs regular expressions, and uses the regular expressions to match the annotation statements to obtain candidate annotation tags. Based on the fallback rules, it determines whether to trigger the fallback logic and outputs the corresponding annotation results.

[0038] Specifically, such as Figure 3 As shown, the rule engine processes the rules as follows: Determine the tag scope: Based on dictionary classification, clarify the tag scope corresponding to the text to be tagged; use dictionary classification as a preliminary filtering condition. The rule engine first obtains the business attributes of the text to be tagged (such as the business area and topic section to which the announcement or letter belongs), and reads the dictionary classification fields pre-configured for each candidate tag in the rule base (e.g., operation for business, legal / compliance for legal, etc.).

[0039] Segmenting and Labeling Statements: The text to be labeled is split into labeled statements according to the punctuation marks in the sentence segmentation rules. After clarifying the range of tags to be matched, the rule engine calls the sentence segmentation rule with priority 0 to perform sentence segmentation processing on the input text to be labeled (such as cleaned and structured announcement or inquiry letter paragraph data). Specifically, the rule engine reads the pre-configured set of punctuation marks in the sentence segmentation rules (such as period, semicolon, exclamation mark, question mark and combined punctuation marks, etc., text segmentation separators), and uses the punctuation marks as the boundary features of text segmentation to accurately decompose continuous long chapters or paragraphs into multiple independent, smallest-granularity labeled statements (i.e., single-sentence-dimensional text units to be matched), providing basic slice data for subsequent blacklist filtering, dictionary and regular expression matching.

[0040] Blacklist filtering: The system determines whether prohibited phrases exist in the labeled statements based on a dictionary blacklist; if so, the labeled statement is discarded. After splitting the text to be labeled into labeled statements, the rule engine calls the dictionary blacklist logic with priority 1 to perform exclusive filtering verification on each of the split labeled statements in sequence. 1. Disable phrase search and matching: The rule engine reads the dictionary blacklist field configured for the target tag (such as negative or exclusionary phrases configured in the blacklist, such as "does not exist" or "does not constitute"), and uses the phrases in the dictionary blacklist as prohibited phrases to perform full-text search matching with the text content of the currently labeled statement; 2. Exclusivity determination and handling: If the current labeled statement contains any prohibited phrase from the dictionary blacklist (i.e., it hits the prohibited phrase condition), it indicates that the statement has negative semantics or exclusion conditions (e.g., the text is expressed as "there is no competition in the same industry" or "does not constitute competition in the same industry"), and does not meet the conditions for hitting the label; the rule engine immediately terminates the subsequent matching process of the labeled statement under the label and directly discards the labeled statement (i.e., abandons the current labeling of the statement).

[0041] If it is determined that the current labeled statement does not contain any prohibited phrases, it means that the statement has passed the blacklist filtering verification. The rule engine then sends the labeled statement to the next priority dictionary order and whitelist matching process.

[0042] Dictionary matching and regular expression matching: If no prohibited phrases exist, the dictionary order is used to determine if there are any order requirements. If so, the dictionary is matched in that order. This is combined with regular expression matching based on the dictionary whitelist. Once the tagged statement successfully passes the dictionary blacklist filter (i.e., it is determined that no prohibited phrases exist), the rule engine sequentially calls the dictionary order (priority 2) and the dictionary whitelist (priority 3) for keyword matching and regular expression construction. 1. Dictionary order verification and order matching (priority 2): The rule engine reads the dictionary order configuration field (e.g., to determine whether the dictionary order is configured as "yes" or "no"); if it is determined that there is a dictionary order requirement, the rule engine performs strict order matching verification on the text in the labeled statement according to the preset keyword order (ensuring that the target keywords appear in the statement in the specified order); if it is determined that there is no dictionary order requirement, it directly enters the dictionary whitelist matching stage.

[0043] 2. Construct tag matching using regular expressions based on dictionary whitelists (priority 3): The rule engine reads the dictionary whitelist field corresponding to the target tag (the whitelist contains the core keywords that constitute the tag, as well as logical relationships composed of logical connectors such as "+" for "AND" and "|" for "OR", such as "holding + industry competition" or "customer | supplier + industry competition"). The rule engine parses and constructs standard regular expressions from the keywords in the dictionary whitelist and their AND, OR, and NOT logical relationships. The constructed regular expression is used to perform pattern matching with the current annotation statement; if the match is successful, the annotation statement is determined to have matched the whitelist rule, and the corresponding candidate annotation label is obtained.

[0044] Fallback rule determination: The fallback rule determines whether a fallback tag is involved. If so, it further checks for the existence of a parent tag and whether it shares the same parent tag. If they share the same parent tag, the fallback tag is not executed; otherwise, the tag is matched using regular expressions. During or after dictionary and regular expression matching, the rule engine invokes the fallback rule logic with priority 4 to perform conflict prevention and multi-level tag validation on the currently matched or triggered tag. 1. Catch-all label determination: The rule engine reads the catch-all rule field in the target tag configuration (for example, to determine whether the tag is configured as a catch-all tag, such as "yes" or "no"; catch-all tags usually contain generalized or broad keywords, such as "other related to industry competition" and other catch-all categories).

[0045] 2. Parent tag and sibling tag conflict judgment: If it is determined that the current tag does not involve a catch-all rule (i.e., it is not a catch-all tag), then the regular expression matching is executed directly to obtain the corresponding tag; If the current tag is determined to be a catch-all tag (i.e., it involves a catch-all rule), the rule engine will further detect whether there is a parent tag (or a more specific / higher priority business tag of the same category) in the currently matched or same context level. If no parent tag exists, the catch label is matched and output according to the regular expression; If a parent tag exists, the rule engine continues to compare and determine whether the parent tag and the current catch-all tag belong to the same parent category / parent relationship: If they are determined to be the same parent tag, it means that a more precise specific tag already covers the business semantics (e.g., the specific tag "competition with the controlling shareholder" has been successfully matched). To avoid tag redundancy and mis-applying generalized tags, the rule engine determines not to execute the catch-all tag (i.e., discard or block the output of the catch-all tag); If they are determined not to be the same parent tag (or there is no conflict), the rule engine continues to match normally according to the regular expression and output the catch-all tag to complete the catch-all coverage of the generalized scenario.

[0046] Ideally, after the rule engine obtains candidate labels, confidence levels are used to perform quality control and output filtering on the labeling results: First, the pre-set confidence threshold and label confidence are read: The rule engine reads the confidence field of the target label in the rule base (for example, the labeling confidence is configured to be 95% or 90% for a certain type of label); at the same time, it reads the pre-set confidence threshold set by the system or business scenario for the current data labeling task (for example, the business requires the confidence of the output label to be no less than 80%).

[0047] Then, confidence comparison and output decision: The rule engine compares and verifies the confidence of the hit label with the preset confidence threshold. Meeting the threshold requirement: If the confidence level of the current candidate label reaches or exceeds a preset confidence threshold (e.g., 95%). (80%), indicating that the annotation result has high certainty and reliability, it is allowed to be output to downstream services as a valid annotation result through a standardized interface; Threshold requirement not met: If the confidence level of a candidate label is lower than the preset confidence level threshold, the labeling result is determined to have a risk of mislabeling or insufficient confidence. The rule engine will intercept and discard the candidate label and will not output it to the outside world.

[0048] If the mode including large language model annotation is selected, the large language model annotation module is used for annotation. The data annotation task is determined based on the pre-configured system prompt words, and the semantic understanding and label prediction of the data to be annotated are performed based on the user prompt words and the reference examples containing preset labels and annotation examples in the knowledge base, so as to obtain the annotation results.

[0049] The specific processing for determining the data annotation task is as follows: Read the pre-configured system prompt (SystemPrompt) to clarify and limit the current role and core task domain of the large language model (for example, setting the large model as a professional data annotation agent in the domain of announcements and inquiry letters); at the same time, read the user-configured user prompt (UserPrompt) to instruct the large model to execute specific text parsing instructions, output format requirements, and context judgment criteria.

[0050] The specific processing of retrieving and loading reference examples is as follows: access the pre-built intelligent agent knowledge base; retrieve and load preset tags (such as tag primary keys, tag names and dictionary classifications) related to the business domain of the current text to be labeled, as well as corresponding standard annotation examples (such as real examples containing specific questions, reply content and corresponding industry backgrounds) from the knowledge base, and inject the preset tags and annotation examples as reference examples for Few-Shot learning into the contextual prompts of the large model.

[0051] The specific processing of semantic understanding, tag prediction, and annotation basis extraction is as follows: The large language model integrates the system prompts, user prompts, and reference examples in the knowledge base to perform deep semantic understanding and tag prediction on the data to be annotated; it can not only accurately identify the semantics of a single sentence, but also make a comprehensive judgment by combining the overall semantics of the paragraph and the contextual logic, accurately covering complex boundary scenarios that are difficult for rules to cover, such as synonyms, implicit logic, and multiple conditions; while generating candidate tags, the large language model accurately extracts the original text basis (i.e., the original text basis for annotation) based on laws, regulations, and business specifications and automatically generates an explainable annotation reason, finally forming a standard annotation result that includes the source data primary key, source data content, tag primary key, tag name, annotation reason, and the original text basis for annotation, realizing the transparency and traceability of the entire annotation process.

[0052] Figure 4 The principle of an embodiment of the data annotation device based on the collaborative operation of a rule engine and a large language model of the present invention is illustrated. For example... Figure 4 As shown, the device in this embodiment includes the following modules.

[0053] The text feature and business requirement analysis module is used to analyze the features and business requirement parameters of the input text to be labeled.

[0054] The text to be annotated has characteristics such as Markdown format or structured / unstructured documents, such as listed company announcements or inquiry letters. Business requirement parameters are usually configured by the user.

[0055] The processing of analyzing the features of the text to be annotated mainly involves: Text structure level: Evaluate the standardization of the text layout, formatting, and contextual logic of the text to be annotated; Keyword clarity and semantic openness: Identify whether the text contains explicit industry / business keywords, or whether the text semantics are relatively open and require deep semantic understanding; Rule base coverage and decision confidence: Evaluate the extent to which the existing rule base covers this type of text sample (e.g., whether the coverage is zero), and whether the decision confidence of the rule base in labeling and predicting this type of sample reaches or exceeds a preset threshold.

[0056] The processing of business requirement parameters mainly involves: Compliance review requirements: Analyze whether the business scenario imposes strict compliance constraints or hard review standards on the annotation results (such as whether a high level of compliance review is required). Provide annotation basis requirements: Analyze whether the business requirements require the output of detailed annotation reasons and original text basis (such as whether traceability and explanation are required).

[0057] The annotation mode selection module is used by the decision engine to automatically select an annotation mode from multiple annotation methods based on the features of the text to be annotated and business requirement parameters.

[0058] Multiple annotation methods are available, including: rule-only annotation mode, large language model-only annotation mode, rule-first annotation followed by large language model annotation mode, and large language model-first annotation followed by rule-first annotation mode.

[0059] The decision engine's selection process is as follows: When the text to be labeled has a standardized structure, clear keywords, and the confidence level of the rule base in judging this type of sample exceeds a preset threshold, select the rule-only labeling mode; When the semantics of the text to be labeled are open and the rule base has zero coverage of this type of text, select the large language model only labeling mode; When candidate labels can be quickly filtered through the rule base, but the judgment boundary is ambiguous and the labeling basis is required, choose the rule labeling first and then large language model labeling mode. When business operations have constraints that require initial labeling results to be generated from a large language model and then subject to compliance review by rule annotation, the mode of first annotating the large language model and then annotating the rules should be selected.

[0060] The multi-mode collaborative annotation and result output module is used to annotate the text to be annotated using the selected annotation method and output the annotation result file.

[0061] The output fields of the annotation result file include: source data primary key, source data content, tag primary key, tag name, annotation reason, and original text basis for annotation.

[0062] If the mode including rule annotation is selected, the rule engine is used for annotation. The rule engine defines the range of tags to be matched, splits the text to be annotated into annotation statements, constructs regular expressions, and uses the regular expressions to match the annotation statements to obtain candidate annotation tags. Based on the fallback rule, it determines whether the fallback logic is triggered to output the corresponding annotation result.

[0063] The rules engine processes it as follows: Determine the tag scope: Based on dictionary classification, clarify the tag scope corresponding to the text to be tagged; use dictionary classification as a preliminary filtering condition. The rule engine first obtains the business attributes of the text to be tagged (such as the business area and topic section to which the announcement or letter belongs), and reads the dictionary classification fields pre-configured for each candidate tag in the rule base (e.g., operation for business, legal / compliance for legal, etc.).

[0064] Segmenting and Labeling Statements: The text to be labeled is split into labeled statements according to the punctuation marks in the sentence segmentation rules. After clarifying the range of tags to be matched, the rule engine calls the sentence segmentation rule with priority 0 to perform sentence segmentation processing on the input text to be labeled (such as cleaned and structured announcement or inquiry letter paragraph data). Specifically, the rule engine reads the pre-configured set of punctuation marks in the sentence segmentation rules (such as period, semicolon, exclamation mark, question mark and combined punctuation marks, etc., text segmentation separators), and uses the punctuation marks as the boundary features of text segmentation to accurately decompose continuous long chapters or paragraphs into multiple independent, smallest-granularity labeled statements (i.e., single-sentence-dimensional text units to be matched), providing basic slice data for subsequent blacklist filtering, dictionary and regular expression matching.

[0065] Blacklist filtering: The system determines whether prohibited phrases exist in the labeled statements based on a dictionary blacklist; if so, the labeled statement is discarded. After splitting the text to be labeled into labeled statements, the rule engine calls the dictionary blacklist logic with priority 1 to perform exclusive filtering verification on each of the split labeled statements in sequence. 1. Disable phrase search and matching: The rule engine reads the dictionary blacklist field configured for the target tag (such as negative or exclusionary phrases configured in the blacklist, such as "does not exist" or "does not constitute"), and uses the phrases in the dictionary blacklist as prohibited phrases to perform full-text search matching with the text content of the currently labeled statement; 2. Exclusivity determination and handling: If the current labeled statement contains any prohibited phrase from the dictionary blacklist (i.e., it hits the prohibited phrase condition), it indicates that the statement has negative semantics or exclusion conditions (e.g., the text is expressed as "there is no competition in the same industry" or "does not constitute competition in the same industry"), and does not meet the conditions for hitting the label; the rule engine immediately terminates the subsequent matching process of the labeled statement under the label and directly discards the labeled statement (i.e., abandons the current labeling of the statement).

[0066] If it is determined that the current labeled statement does not contain any prohibited phrases, it means that the statement has passed the blacklist filtering verification. The rule engine then sends the labeled statement to the next priority dictionary order and whitelist matching process.

[0067] Dictionary matching and regular expression matching: If no prohibited phrases exist, the dictionary order is used to determine if there are any order requirements. If so, the dictionary is matched in that order. This is combined with regular expression matching based on the dictionary whitelist. Once the tagged statement successfully passes the dictionary blacklist filter (i.e., it is determined that no prohibited phrases exist), the rule engine sequentially calls the dictionary order (priority 2) and the dictionary whitelist (priority 3) for keyword matching and regular expression construction. 1. Dictionary order verification and order matching (priority 2): The rule engine reads the dictionary order configuration field (e.g., to determine whether the dictionary order is configured as "yes" or "no"); if it is determined that there is a dictionary order requirement, the rule engine performs strict order matching verification on the text in the labeled statement according to the preset keyword order (ensuring that the target keywords appear in the statement in the specified order); if it is determined that there is no dictionary order requirement, it directly enters the dictionary whitelist matching stage.

[0068] 2. Construct tag matching using regular expressions based on dictionary whitelists (priority 3): The rule engine reads the dictionary whitelist field corresponding to the target tag (the whitelist contains the core keywords that constitute the tag, as well as logical relationships composed of logical connectors such as "+" for "AND" and "|" for "OR", such as "holding + industry competition" or "customer | supplier + industry competition"). The rule engine parses and constructs standard regular expressions from the keywords in the dictionary whitelist and their AND, OR, and NOT logical relationships. The constructed regular expression is used to perform pattern matching with the current annotation statement; if the match is successful, the annotation statement is determined to have matched the whitelist rule, and the corresponding candidate annotation label is obtained.

[0069] Fallback rule determination: The fallback rule determines whether a fallback tag is involved. If so, it further checks for the existence of a parent tag and whether it shares the same parent tag. If they share the same parent tag, the fallback tag is not executed; otherwise, the tag is matched using regular expressions. During or after dictionary and regular expression matching, the rule engine invokes the fallback rule logic with priority 4 to perform conflict prevention and multi-level tag validation on the currently matched or triggered tag. 1. Catch-all label determination: The rule engine reads the catch-all rule field in the target tag configuration (for example, to determine whether the tag is configured as a catch-all tag, such as "yes" or "no"; catch-all tags usually contain generalized or broad keywords, such as "other related to industry competition" and other catch-all categories).

[0070] 2. Parent tag and sibling tag conflict judgment: If it is determined that the current tag does not involve a catch-all rule (i.e., it is not a catch-all tag), then the regular expression matching is executed directly to obtain the corresponding tag; If the current tag is determined to be a catch-all tag (i.e., it involves a catch-all rule), the rule engine will further detect whether there is a parent tag (or a more specific / higher priority business tag of the same category) in the currently matched or same context level. If no parent tag exists, the catch label is matched and output according to the regular expression; If a parent tag exists, the rule engine continues to compare and determine whether the parent tag and the current catch-all tag belong to the same parent category / parent relationship: If they are determined to be the same parent tag, it means that a more precise specific tag already covers the business semantics (e.g., the specific tag "competition with the controlling shareholder" has been successfully matched). To avoid tag redundancy and mis-applying generalized tags, the rule engine determines not to execute the catch-all tag (i.e., discard or block the output of the catch-all tag); If they are determined not to be the same parent tag (or there is no conflict), the rule engine continues to match normally according to the regular expression and output the catch-all tag to complete the catch-all coverage of the generalized scenario.

[0071] Ideally, after the rule engine obtains candidate labels, confidence levels are used to perform quality control and output filtering on the labeling results: First, the pre-set confidence threshold and label confidence are read: The rule engine reads the confidence field of the target label in the rule base (for example, the labeling confidence is configured to be 95% or 90% for a certain type of label); at the same time, it reads the pre-set confidence threshold set by the system or business scenario for the current data labeling task (for example, the business requires the confidence of the output label to be no less than 80%).

[0072] Then, confidence comparison and output decision: The rule engine compares and verifies the confidence of the hit label with the preset confidence threshold. Meeting the threshold requirement: If the confidence level of the current candidate label reaches or exceeds a preset confidence threshold (e.g., 95%). (80%), indicating that the annotation result has high certainty and reliability, it is allowed to be output to downstream services as a valid annotation result through a standardized interface; Threshold requirement not met: If the confidence level of a candidate label is lower than the preset confidence level threshold, the labeling result is determined to have a risk of mislabeling or insufficient confidence. The rule engine will intercept and discard the candidate label and will not output it to the outside world.

[0073] If the mode including large language model annotation is selected, the large language model annotation module is used for annotation. The data annotation task is determined based on the pre-configured system prompt words, and the semantic understanding and label prediction of the data to be annotated are performed based on the user prompt words and the reference examples containing preset labels and annotation examples in the knowledge base, so as to obtain the annotation results.

[0074] The specific processing for determining the data annotation task is as follows: Read the pre-configured system prompt (SystemPrompt) to clarify and limit the current role and core task domain of the large language model (for example, setting the large model as a professional data annotation agent in the domain of announcements and inquiry letters); at the same time, read the user-configured user prompt (UserPrompt) to instruct the large model to execute specific text parsing instructions, output format requirements, and context judgment criteria.

[0075] The specific processing of retrieving and loading reference examples is as follows: access the pre-built intelligent agent knowledge base; retrieve and load preset tags (such as tag primary keys, tag names and dictionary classifications) related to the business domain of the current text to be labeled, as well as corresponding standard annotation examples (such as real examples containing specific questions, reply content and corresponding industry backgrounds) from the knowledge base, and inject the preset tags and annotation examples as reference examples for Few-Shot learning into the contextual prompts of the large model.

[0076] The specific processing of semantic understanding, tag prediction, and annotation basis extraction is as follows: The large language model integrates the system prompts, user prompts, and reference examples in the knowledge base to perform deep semantic understanding and tag prediction on the data to be annotated; it can not only accurately identify the semantics of a single sentence, but also make a comprehensive judgment by combining the overall semantics of the paragraph and the contextual logic, accurately covering complex boundary scenarios that are difficult for rules to cover, such as synonyms, implicit logic, and multiple conditions; while generating candidate tags, the large language model accurately extracts the original text basis (i.e., the original text basis for annotation) based on laws, regulations, and business specifications and automatically generates an explainable annotation reason, finally forming a standard annotation result that includes the source data primary key, source data content, tag primary key, tag name, annotation reason, and the original text basis for annotation, realizing the transparency and traceability of the entire annotation process.

[0077] Furthermore, this invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements... Figure 1 The steps of an embodiment of the data annotation method based on the collaboration of rule engine and large language model are shown.

[0078] Furthermore, this invention also discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements as follows: Figure 1 The steps of an embodiment of the data annotation method based on the collaboration of rule engine and large language model are shown.

[0079] Furthermore, this invention also discloses a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, they implement... Figure 1The steps of an embodiment of the data annotation method based on the collaboration of rule engine and large language model are shown.

[0080] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0081] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.

[0082] The various illustrative logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein can be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0083] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read and write information to / from the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0084] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functionality may be stored or transmitted as one or more instructions or code on or through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, encompassing any medium that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium accessible to a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible to a computer. Any connection is also legitimately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. As used in this article, disk and disc include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs. Disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.

[0085] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data annotation method based on the collaboration of a rule engine and a large language model, characterized in that, Includes the following steps: Analyze the features of the input text to be labeled and the parameters of business requirements; The decision engine automatically selects an annotation method from multiple annotation methods based on the features of the text to be annotated and the business requirement parameters; The selected annotation method is used to annotate the text to be annotated and an annotation result file is output. If the mode including rule annotation is selected, the rule engine is used for annotation. The rule engine defines the range of tags to be matched, splits the text to be annotated into annotation statements, constructs regular expressions, and uses the regular expressions to match the annotation statements to obtain candidate annotation tags. The fallback rule determines whether to trigger the fallback logic to output the corresponding annotation result. If the mode including large language model annotation is selected, the large language model annotation module is used for annotation. The data annotation task is determined based on the pre-configured system prompt words, and the semantic understanding and label prediction of the data to be annotated are performed based on the user prompt words and the reference examples containing preset labels and annotation examples in the knowledge base, so as to obtain the annotation results.

2. The data annotation method based on the collaboration of rule engine and large language model according to claim 1, characterized in that, The various annotation methods include: rule-only annotation mode, large language model-only annotation mode, rule-first annotation followed by large language model annotation mode, and large language model-first annotation followed by rule-first annotation mode.

3. The data annotation method based on the collaboration of rule engine and large language model according to claim 1, characterized in that, The output fields of the annotation result file include: source data primary key, source data content, tag primary key, tag name, annotation reason, and original text basis for annotation.

4. The data annotation method based on the collaborative operation of a rule engine and a large language model according to claim 2, characterized in that, The decision engine's selection process is as follows: When the text to be labeled has a standardized structure, clear keywords, and the confidence level of the rule base in judging this type of sample exceeds a preset threshold, select the rule-only labeling mode; When the semantics of the text to be labeled are open and the rule base has zero coverage of this type of text, select the large language model only labeling mode; When candidate labels can be quickly filtered through the rule base, but the judgment boundary is ambiguous and the labeling basis is required, choose the rule labeling first and then large language model labeling mode. When business operations have constraints that require initial labeling results to be generated from a large language model and then subject to compliance review by rule annotation, the mode of first annotating the large language model and then annotating the rules should be selected.

5. The data annotation method based on the collaboration of rule engine and large language model according to claim 2, characterized in that, The rules engine processes them as follows: Determine the label range: Based on dictionary classification, clarify the label range corresponding to the text to be labeled; Split annotation sentences: Split the text to be annotated into annotation sentences according to the punctuation marks in the sentence segmentation rules; Blacklist filtering: Determine whether prohibited phrases exist in the marked statements based on the dictionary blacklist; If it exists, discard the annotation statement; Dictionary matching and regular expression matching: If there are no prohibited phrases, determine whether there is an order requirement based on the dictionary order. If there is an order requirement, match the dictionary in order. It also combines a dictionary whitelist with regular expression matching for tags; Fallback rule determination: Determine whether a fallback tag is involved based on the fallback rule; if so, further determine whether a parent tag exists and whether it is the same parent tag; if it is the same parent tag, the fallback tag is not executed; if not, the tag is matched according to regular expressions.

6. The data annotation method based on the collaboration of rule engine and large language model according to claim 5, characterized in that, The rule engine's processing also includes determining whether to output the corresponding annotation result based on the confidence level.

7. A data annotation device based on the collaboration of a rule engine and a large language model, characterized in that, The device includes: The text feature and business requirement analysis module is used to analyze the features and business requirement parameters of the input text to be labeled. The annotation mode selection module is used by the decision engine to automatically select an annotation mode from multiple annotation methods based on the features of the text to be annotated and business requirement parameters; The multi-mode collaborative annotation and result output module is used to annotate the text to be annotated using the selected annotation method and output the annotation result file. If the mode including rule annotation is selected, the rule engine is used for annotation. The rule engine defines the range of tags to be matched, splits the text to be annotated into annotation statements, constructs regular expressions, and uses the regular expressions to match the annotation statements to obtain candidate annotation tags. Based on the fallback rule, it determines whether to trigger the fallback logic to output the corresponding annotation result. If the mode including large language model annotation is selected, the large language model annotation module is used for annotation. The data annotation task is determined based on the pre-configured system prompt words, and the semantic understanding and label prediction of the data to be annotated are performed based on the user prompt words and the reference examples containing preset labels and annotation examples in the knowledge base, so as to obtain the annotation results.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the data annotation method based on the collaboration of rule engine and large language model as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the data annotation method based on the collaboration of rule engine and large language model as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the data annotation method based on the collaboration of rule engine and large language model as described in any one of claims 1 to 6.

Images