A labeling method and device, electronic equipment and storage medium

Abstract

The application provides a labeling method and device, electronic equipment and a storage medium. The method is suitable for an intelligent question and answer system, performs intent recognition on a natural language question input by a user to identify a target intent of the user, fills the target intent, a first label associated with the target intent and the natural language question into a preset prompt word template to construct a first prompt word, inputs the first prompt word into a first large language model to obtain to-be-verified content output by the first large language model, and determines a label of the target intent through the to-be-verified content. In the scheme, the target intent is identified through intent recognition on the natural language question, and the first large language model is called to obtain the to-be-verified content according to the target intent. The label of the target intent is determined through the to-be-verified content, data mining is performed based on the label of the target intent, and thus the business strategy of the intelligent question and answer system is optimized.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to an annotation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, intelligent question-answering systems are widely used in scenarios such as equipment consultation, marketing services, and business inquiries. These systems identify the intent behind a user's question and then route it to the appropriate business module based on that intent. However, the core capabilities of current intelligent question-answering systems are concentrated on "intent recognition - business routing - response generation," lacking data mining capabilities and thus unable to use precise data to optimize business strategies. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a labeling method, apparatus, electronic device, and storage medium to realize data mining.

[0004] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0005] The first aspect of this invention discloses an annotation method applicable to intelligent question-answering systems, the method comprising:

[0006] The user's input natural language question is subjected to intent recognition in order to identify the user's target intent, which is any preset basic intent;

[0007] The target intent, the first tag associated with the target intent, and the natural language question are filled into a preset prompt word template to construct the first prompt word;

[0008] Input the first prompt word into the first large language model to obtain the content to be verified output by the first large language model;

[0009] The tag of the target intent is determined by the content to be verified.

[0010] Preferably, determining the tag of the target intent through the content to be verified includes:

[0011] Determine whether the content to be verified is an empty value or a tag to be verified;

[0012] If the content to be verified is an empty value, the tag of the target intent is determined to be the second tag;

[0013] If the content to be verified is the tag to be verified, then the tag to be verified is filtered.

[0014] Determine whether the filtered tag to be verified is any of the first tag;

[0015] If the filtered tag to be verified is any of the first tag, then the tag of the target intent is determined to be the filtered tag to be verified.

[0016] If the filtered tag to be verified is not any of the first tags, the tag of the target intent is determined to be the second tag.

[0017] Preferably, after determining the tag of the target intent through the content to be verified, the method further includes:

[0018] The target intent and its tag are stored in the database.

[0019] Preferably, after storing the target intent and the tag of the target intent in the database, the method further includes:

[0020] In response to a data statistics command, the data stored in the database is statistically analyzed to obtain the data statistics results;

[0021] Data charts are generated and displayed based on the statistical results.

[0022] Preferably, intent recognition is performed on the natural language questions input by the user to identify the user's target intent, including:

[0023] Perform keyword matching or regular expression matching on the natural language questions input by the user;

[0024] If any basic intent is matched, the matched basic intent is determined to be the user's target intent;

[0025] If no basic intent is matched, the natural language question and the user's historical chat history are combined into a second prompt word;

[0026] Input the second prompt word into the second language model to obtain the confidence level of each of the basic intentions output by the second language model;

[0027] The basic intent with the highest confidence level is selected as the user's target intent.

[0028] Preferably, after performing intent recognition on the natural language question input by the user, the method further includes:

[0029] The natural language question is routed to the business module associated with the target intent for a business response.

[0030] A second aspect of this invention discloses an annotation device, which is suitable for intelligent question-answering systems. The device includes:

[0031] The recognition unit is used to perform intent recognition on the natural language questions input by the user, so as to identify the user's target intent, which is any preset basic intent;

[0032] A construction unit is used to fill the target intent, the first tag associated with the target intent, and the natural language question into a preset prompt word template to construct the first prompt word;

[0033] The processing unit is used to input the first prompt word into the first large language model and obtain the content to be verified output by the first large language model.

[0034] A determining unit is used to determine the tag of the target intent based on the content to be verified.

[0035] Preferably, the determining unit includes:

[0036] The first judgment subunit is used to determine whether the content to be verified is an empty value or a tag to be verified; if the content to be verified is an empty value, the tag of the target intent is determined to be a second tag; if the content to be verified is the tag to be verified, the tag to be verified is filtered.

[0037] The second judgment subunit is used to determine whether the filtered tag to be verified is any of the first tags; if the filtered tag to be verified is any of the first tags, the tag of the target intent is determined to be the filtered tag to be verified; if the filtered tag to be verified is not any of the first tags, the tag of the target intent is determined to be the second tag.

[0038] A third aspect of the present invention discloses an electronic device, comprising: a processor and a memory, the processor and the memory being connected via a bus; wherein, the processor is used to call and execute a program stored in the memory; the memory is used to store the program, the program being used to implement the annotation method disclosed in the first aspect of the present invention.

[0039] The fourth aspect of the present invention discloses a storage medium storing computer-executable instructions, wherein the computer-executable instructions are the annotation method disclosed in the first aspect of the present invention.

[0040] Based on the above-described annotation method, apparatus, electronic device, and storage medium provided by embodiments of the present invention, this method is applicable to intelligent question-answering systems. It involves performing intent recognition on natural language questions input by users to identify the user's target intent; filling the target intent, a first tag associated with the target intent, and the natural language question into a preset prompt word template to construct a first prompt word; inputting the first prompt word into a first large language model to obtain the content to be verified output by the first large language model; and determining the tag of the target intent based on the content to be verified. In this solution, intent recognition is performed on natural language questions to identify the target intent, and the first large language model is invoked to obtain the content to be verified based on the target intent. The tag of the target intent is determined based on the content to be verified, and data mining is performed based on the tag of the target intent to optimize the business strategy of the intelligent question-answering system. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 A flowchart of an annotation method provided in an embodiment of the present invention;

[0043] Figure 2 A flowchart for determining a tag for a target intent provided in an embodiment of the present invention;

[0044] Figure 3 A flowchart for identifying a user's target intent provided in an embodiment of the present invention;

[0045] Figure 4 This is an example architecture diagram of an intelligent question-answering system provided in an embodiment of the present invention;

[0046] Figure 5 Another flowchart of an annotation method provided in an embodiment of the present invention;

[0047] Figure 6 This is a structural block diagram of a labeling device provided in an embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0050] Currently, intelligent question-answering systems are widely used in scenarios such as equipment consultation, marketing services, and business inquiries. These systems identify the intent behind a user's question and then route it to the appropriate business module based on that intent. However, the core capabilities of current intelligent question-answering systems are concentrated on "intent recognition - business routing - response generation," lacking data mining capabilities and thus unable to use precise data to optimize business strategies.

[0051] Furthermore, research has revealed that existing intelligent question-answering systems lack a full-scale labeling step after the intent recognition stage. This deficiency directly prevents intelligent question-answering systems from breaking down user needs under various basic intents, leading to problems such as single data statistics dimensions and idle massive amounts of user interaction data, making it impossible to use accurate data to optimize business strategies.

[0052] To address the aforementioned issues, this invention proposes a labeling method, apparatus, electronic device, and storage medium. The method involves performing intent recognition on natural language questions to identify the target intent, and then invoking a first major language model to obtain the content to be verified based on the target intent. The target intent is then labeled using the content to be verified, and data mining is performed based on these labels to optimize the business strategy of the intelligent question-answering system.

[0053] See Figure 1 The diagram illustrates a flowchart of an annotation method provided by an embodiment of the present invention. This annotation method is applicable to intelligent question-answering systems and includes:

[0054] Step S101: Perform intent recognition on the natural language question input by the user to identify the user's target intent, which is any preset basic intent.

[0055] It should be noted that "going beyond the scope" specifically refers to exceeding the service range of the intelligent question-answering system. Natural language questions posed by users are typically either "questions related to the service range of the intelligent question-answering system" or "questions beyond the service range of the intelligent question-answering system." Questions exceeding the service range of the intelligent question-answering system can be termed "boundary-crossing questions."

[0056] In the specific implementation step S101, the user obtains a conversational, unformatted natural language question input by the user through a terminal (e.g., mobile phone, PC, or tablet).

[0057] For example, a user might input a natural language question like "What should I do if my X-431 PAD9 Bluetooth keeps disconnecting?" or "Are there any discounts on diagnostic equipment for new energy vehicles recently?"

[0058] In practical applications, the "request carrying a natural language question" is encapsulated in JSON format into core information, which includes at least the user ID, natural language question, terminal type, and request timestamp. This core information is then transmitted to the intelligent question answering system through a unified access interface.

[0059] Predefine multiple basic intents, such as the four basic intents of "device Q&A", "marketing support", "car repair Q&A" and "crossing boundaries".

[0060] The intent of the user's natural language input is identified, thereby identifying the user's target intent, which is any number (one or more) basic intents.

[0061] It should be noted that the intelligent question-and-answer system includes multiple business modules, such as a marketing support module, an equipment question-and-answer module, an overstepping of boundaries handling module, and a vehicle repair question-and-answer module.

[0062] The marketing support module's functions and responsibilities include answering questions related to product features and pricing. It primarily uses marketing materials to build a knowledge base to address questions about product selling points, specifications, and comparisons. The module handles inquiries about product characteristics, pricing policies, promotional activities, and purchase inquiries.

[0063] The equipment Q&A module is responsible for handling equipment-related issues and troubleshooting. The marketing support module covers product functions, technical parameters, troubleshooting, and user guides.

[0064] The function and responsibility of the car repair Q&A module is to solve car repair and maintenance problems. The scope of the car repair Q&A module includes: fault diagnosis, repair solutions, maintenance suggestions, and parts information.

[0065] The function and responsibility of the boundary crossing handling module is to handle issues that exceed the service scope of other business modules. The scope of the boundary crossing handling module includes: amicably refusing to answer, reporting the issue, and providing suggestions.

[0066] In some specific embodiments, after identifying the user's target intent, the user's natural language question is routed to the business module associated with the target intent for a business response. The step of "routing the user's natural language question to the business module associated with the target intent for a business response" is executed in parallel with subsequent steps and does not affect the response speed of the intelligent question answering system.

[0067] Step S102: Fill the target intent, the first label associated with the target intent, and the natural language question into the preset prompt word template to construct the first prompt word.

[0068] It should be noted that each basic intent has a predefined tag range, and the tag range of a certain basic intent covers multiple tags associated with that basic intent.

[0069] For example, the basic intent "Device Q&A" is associated with several tags, namely "Bluetooth Operation", "Fault Code Inquiry", "Device Model" and "Operation Steps".

[0070] In the specific implementation step S102, the label range of the target intent is matched. The label range of the target intent covers multiple labels associated with the target intent. The label associated with the target intent is called the first label.

[0071] The first prompt word is constructed by filling the prompt word template with the target intent, the first label associated with the target intent, and the natural language question.

[0072] Specifically, a structured prompt template (stored in a JSON file) is pre-configured. This prompt template includes fixed task requirements, input information placeholders, and output format specifications. The target intent, the first label associated with the target intent, and the natural language question are filled into the input information placeholders of the prompt template to generate standardized first prompts. This ensures that the output of the first language model is controllable and free of redundancy.

[0073] For example, the input information and content examples for the prompt word template are shown in Table 1.

[0074] Table 1:

[0075] Target Intent Equipment FAQ Tag range [Bluetooth Operation, Fault Code Inquiry, Device Model, Operation Steps] Natural Language Problems "What should I do if my X-431 PAD9 Bluetooth keeps disconnecting?"

[0076] The first prompt words generated based on Table 1 above are as follows:

[0077] "You are a labeling expert."

[0078] Basic intent: Device Q&A;

[0079] Optional tag range: Bluetooth operation, fault code query, device model, operation steps;

[0080] Natural Language Issue: What to do if the Bluetooth connection to my X-431 PAD9 keeps dropping?

[0081] Please select one or more of the best-matching tags from the available tag range for output.

[0082] Step S103: Input the first prompt word into the first large language model to obtain the content to be verified output by the first large language model.

[0083] It should be noted that the first language model is a lightweight, large-scale model deployed locally or in a private cloud. This model is capable of accurately matching tags with vague or colloquial expressions based on semantic understanding. For example, the first language model can accurately match the tag "Bluetooth operation" from the information "constantly disconnecting".

[0084] In the specific implementation step S103, the first prompt word is input into the first large language model through the API interface, and the content to be verified is obtained from the output of the first large language model.

[0085] Understandably, the results returned by the first language model are divided into two cases: "empty value" or "label". Therefore, after inputting the first prompt word into the first language model, the content to be verified by the first language model will be either an empty value or a label to be verified.

[0086] Step S104: Determine the tag of the target intent through the content to be verified.

[0087] In the specific implementation step S104, the tag of the target intent is determined by the content to be verified, wherein the tag of the target intent is any first tag or second tag.

[0088] It should be noted that the second label can be called a catch-all label, for example, the second label is the "Other" label. The second label can prevent the situation of "label loss".

[0089] In some specific embodiments, after determining the tag of the target intent, the target intent and the tag of the target intent are stored in a database, which can be a MySQL database or other types of databases, without limitation.

[0090] It should be noted that the target intent can be any basic intent. The database creates a structured data table containing fields such as issue ID, user ID, basic intent, tag, original issue text, terminal type, and request timestamp. Indexes are created for "basic intent, creation time, and tag" to accelerate subsequent statistical queries.

[0091] When storing target intents and their tags in the database, an ORM framework can be used to write the data, ensuring the standardization and security of data storage and avoiding data redundancy.

[0092] In other specific embodiments, in response to a data statistics instruction, the data stored in the database is statistically analyzed to obtain data statistics results; and data charts are generated and displayed according to the data statistics results.

[0093] In practical applications, data statistics commands can be SQL aggregation query commands, which can achieve multiple core statistics such as "tag dimension statistics", "basic intent + tag dimension statistics" and "time trend dimension statistics".

[0094] Among them, tag-level statistics: count the total number of inquiries for a single tag. For example: count the number of inquiries for the tag "Bluetooth operation".

[0095] Basic Intent + Tag Dimension Statistics: Statistics on the proportion of each tag under a certain basic intent. For example: Statistics on the proportion of the "Bluetooth Operation" tag within the basic intent of "Device Q&A".

[0096] Time trend statistics: This section tracks the weekly / monthly trends in inquiry volume for different tags. For example, it tracks the growth of inquiries related to the "Promotional Activities" tag over the past 30 days.

[0097] After obtaining the statistical results, a data chart is generated based on the lightweight chart library and the statistical results, and the data chart is displayed to relevant personnel.

[0098] For example, the generated data charts are bar charts and pie charts. The bar chart shows the consultation volume ranking of the top 10 high-frequency tags, and the pie chart shows the consultation proportion distribution of each tag under a certain basic intent.

[0099] It should be noted that the present invention also provides a time filtering function (for example, data from the last 7 days or the last 30 days can be filtered), and supports displaying the changing trend of tag consultation volume in tabular form, so that relevant personnel can quickly locate high-frequency needs and changing trends.

[0100] Furthermore, by displaying the generated data charts on the back-end terminal, relevant personnel can quickly identify the segmented needs of users' frequently asked questions based on the data charts, and optimize the response content of the intelligent question-and-answer system accordingly (such as supplementing the troubleshooting guide for "Bluetooth operation") and adjust business strategies (such as launching promotional activities for the "price inquiry" tag).

[0101] In this embodiment of the invention, intent recognition is performed on natural language questions to identify the target intent, and a first major language model is invoked to obtain the content to be verified based on the target intent. The target intent is labeled using the content to be verified, and data mining is performed based on these labels to optimize the business strategy of the intelligent question-answering system.

[0102] Regarding the above embodiments of the present invention Figure 1 The content of step S104, "determining the tag of the target intent through the content to be verified," can be found in [link to relevant documentation]. Figure 2 This illustrates a flowchart of a label for determining a target intent, provided in an embodiment of the present invention. Figure 2 Includes the following steps:

[0103] Step S201: Determine whether the content to be verified is an empty value or a tag to be verified. If the content to be verified is an empty value, proceed to step S202. If the content to be verified is a tag to be verified, proceed to step S203.

[0104] It should be noted that the results returned by the first major language model are divided into two categories: "empty value" and "label". "Empty value" specifically refers to empty strings, null, empty objects, empty arrays, etc.

[0105] In the specific implementation step S201, it is determined whether the content to be verified returned by the first language model is an empty value or a label to be verified.

[0106] If the content to be verified is empty, it indicates that the first language model did not return a valid label (or returned an invalid label), and step S202 is executed. If the content to be verified is a label to be verified, step S203 is executed to continue the subsequent verification of the label to be verified.

[0107] Step S202: If the content to be verified is empty, determine the label of the target intent as the second label.

[0108] In the specific implementation step S202, if the content to be verified is empty, the label for determining the target intent is the second label, which can be called a catch-all label, for example, the second label is the "other" label. Using the second label avoids the situation of "label loss" and also ensures 100% coverage of the natural language questions raised by the user.

[0109] In practical applications, if the format of the content to be verified returned by the first language model is incorrect, it also indicates that the first language model has not returned a valid label, and the label for determining the target intent is the second label.

[0110] Output the target intent tags in the format of "target intent + second tag" to avoid missing tags.

[0111] Step S203: If the content to be verified is a tag to be verified, filter the tag to be verified.

[0112] In the specific implementation of step S203, if the content to be verified is a tag to be verified, the tag to be verified is filtered to remove null values ​​and invalid characters.

[0113] Step S204: Determine whether the filtered tag to be verified is any first tag. If the filtered tag to be verified is any first tag, proceed to step S205. If the filtered tag to be verified is not any first tag, proceed to step S206.

[0114] It should be noted that the scope of the target intent's tags covers multiple tags associated with the target intent, and the tags associated with the target intent are called the first tags.

[0115] In the specific implementation step S204, it is determined whether the filtered tag to be verified is any first tag, that is, whether the filtered tag to be verified is within the tag range of the target intent.

[0116] If the filtered tag to be verified is any first tag (within the tag range of the target intent), it indicates that the filtered tag to be verified is a valid tag, and step S205 is executed.

[0117] If the filtered tag to be verified is not an arbitrary first tag (not within the target intent tag range), it indicates that the first language model returned an invalid tag, and step S206 is executed.

[0118] Step S205: Determine the target intent label as the filtered label to be verified.

[0119] In the specific implementation of step S205, if the filtered tag to be verified is any first tag (one or more first tags), the tag of the target intent can be determined as the filtered tag to be verified.

[0120] Output the tags of the target intent in the form of "target intent + tag set", where the tag set contains the tags of the target intent.

[0121] For example, when outputting the information "Device Q&A + Bluetooth Operation + X-431 PAD9", "Device Q&A" is the target intent, and "Bluetooth Operation" and "X-431 PAD9" are the labels of the target intent.

[0122] Step S206: Determine the label of the target intent as the second label.

[0123] In the specific implementation step S206, if the filtered tag to be verified is not an arbitrary first tag, the tag that determines the target intent is the second tag.

[0124] Output the target intent tags in the format of "target intent + second tag" to avoid missing tags.

[0125] To optimize the prompt words and tag library, after determining that the tag for the target intent is the second tag, invalid tag cases are recorded in the log file. The invalid tag case includes the question ID of the natural language question received this time, the target intent, the tag to be verified, and the filtered tag to be verified.

[0126] As can be seen from steps S201 to S206 above, this scheme uses a dual verification logic of "tag existence + tag range matching" to determine whether the content to be verified returned by the first language model is a valid tag. If the first language model returns an invalid tag, the tag of the target intent is determined to be the second tag; if the first language model returns a valid tag, the tag of the target intent is determined to be the filtered tag to be verified.

[0127] Regarding the above embodiments of the present invention Figure 1 The content related to "identifying the user's target intent" in step S101 can be found in [link to relevant documentation]. Figure 3 This illustrates a flowchart of identifying a user's target intent according to an embodiment of the present invention. Figure 3 Includes the following steps:

[0128] Step S301: Perform keyword matching or regular expression matching on the natural language question input by the user.

[0129] It should be noted that a lightweight rule base is set up in advance, which includes keyword matching rules and regular expression rules.

[0130] For example, the keyword matching rules are as follows: for the basic intent of "device Q&A", keywords such as "X-431" and "cannot connect" are configured; for the basic intent of "marketing support", keywords such as "price" and "discount" are configured.

[0131] For example, the regular expression rule is: "Out of bounds problem" as the basic intention to configure regular expression patterns such as "weather." and "today.news".

[0132] In the specific implementation step S301, based on the natural language question input by the user, the rule base is quickly retrieved through keyword matching or regular expression matching to obtain the corresponding matching result.

[0133] If the matching result indicates that any basic intent is matched (equivalent to hitting any basic intent), proceed to step S302.

[0134] If the matching result indicates that no basic intent was matched (equivalent to no basic intent was hit), proceed to step S303.

[0135] Step S302: If any basic intent is matched, determine that the matched basic intent is the user's target intent.

[0136] In the specific implementation step S302, if any basic intent is matched, the matched basic intent is determined as the user's target intent.

[0137] Step S303: If no basic intent is matched, the natural language question and the user's chat history are assembled into a second prompt word.

[0138] In the specific implementation step S303, if no basic intent is matched, the natural language question and the user's historical chat history are assembled into a standardized second prompt word.

[0139] Step S304: Input the second prompt word into the second language model to obtain the confidence scores of each basic intent output by the second language model.

[0140] It should be noted that the second large language model can be a lightweight natural language understanding (NLU) model or other types of large language models.

[0141] In the specific implementation step S304, the second prompt word is input into the second language model, which then parses the user's natural language questions and historical chat records (equivalent to context) to obtain the confidence of each basic intent output by the second language model.

[0142] In some embodiments, the second language model outputs the confidence scores of each underlying intent in a JSON structure.

[0143] For example, some of the output from the second language model is: "Device Q&A: 0.87", "Marketing Support: 0.03", "Car Repair Q&A: 0.92", "Boundary Overstepping Issues: 0.0".

[0144] Step S305: Select the basic intent with the highest confidence as the user's target intent.

[0145] In the specific implementation step S305, the basic intent with the highest confidence level is selected as the user's target intent.

[0146] In practical applications, a basic intent with a confidence level higher than a threshold can also be selected as the user's target intent. For example, a basic intent with a confidence level higher than 0.5 can be selected as the user's target intent.

[0147] The above embodiments of the present invention Figure 3 This is an explanation of how to determine a user's target intent.

[0148] In practical applications, this solution can be applied to the existing architecture of the intelligent question-answering system without intruding on the original core logic, thus reducing the difficulty of modifying the intelligent question-answering system. For example... Figure 4 As shown in the architectural example diagram of the intelligent question-answering system provided in the embodiment of the present invention, the intelligent question-answering system includes: a user terminal 401, an intent recognition component 402, a service routing component 403, a model invocation component 404, a tag verification component 405, a data storage component 406, a statistics component 407, and a visualization component 408.

[0149] Intent recognition component 402: performs intent recognition on the natural language questions input by the user terminal 401 to identify the user's target intent.

[0150] Business routing component 403: routes the user's natural language question to the business module associated with the target intent for a business response.

[0151] Model call component 404: The first language model was called to output the content to be verified.

[0152] Tag validation component 405: Validates the content to be validated to determine the tag of the target intent.

[0153] Data storage component 406: Stores the target intent and its tag in the database.

[0154] Statistical component 407: Performs statistical analysis on the data stored in the database to obtain statistical results.

[0155] Visualization component 408: Generates and displays data charts based on statistical results.

[0156] Natural language issues resolved by the business routing component 403 are seamlessly transferred to the newly added model call component 404 and label verification component 405, enabling the handling of complex, ambiguous, and colloquial natural language problems.

[0157] It should be noted that, Figure 4 The execution principles of each component have been explained in detail in the above embodiments and will not be repeated here.

[0158] To better understand the above Figure 4 The workflow of the intelligent question-answering system shown is as follows: Figure 5 Another flowchart illustrating one annotation method is shown below. Figure 5 Includes the following steps:

[0159] Step S501: Perform intent recognition on the natural language question input by the user to identify the user's target intent.

[0160] Step S502: The user's natural language question is routed to the business module associated with the target intent for a business response.

[0161] Step S503: Call the first major language model to output the content to be verified.

[0162] Step S504: Perform double verification on the content to be verified to determine the label of the target intent.

[0163] Step S505: Store the target intent and its tag in the database.

[0164] Step S506: Perform statistics on the data stored in the database to obtain the statistical results.

[0165] Step S507: Generate and display data charts based on the data statistics results.

[0166] It should be noted that, Figure 5 The execution principle of each step has been explained in detail in the above embodiments, and will not be repeated here.

[0167] As can be seen from the above embodiments, this solution, through the full coverage mechanism of "label range matching" and "second label", ensures that all natural language questions that have undergone intent recognition can enter the labeling process without omission, thus guaranteeing the integrity of statistical data and realizing the true presentation of user demand distribution.

[0168] This solution can reuse the architecture of existing intelligent question-answering systems without requiring large-scale modifications to the architecture, thus reducing modification costs and shortening the deployment cycle.

[0169] This solution achieves a complete data loop of "tag labeling - data storage - data statistics - visualization display", thereby efficiently mining data value and ultimately realizing data-driven precise business optimization.

[0170] Corresponding to the annotation method provided in the above embodiments of the present invention, see also... Figure 6 The present invention also provides a structural block diagram of an annotation device, which is suitable for intelligent question-and-answer systems. The annotation device includes: an identification unit 601, a construction unit 602, a processing unit 603, and a determination unit 604.

[0171] The recognition unit 601 is used to perform intent recognition on the natural language questions input by the user, so as to identify the user's target intent, which is any preset basic intent.

[0172] The construction unit 602 is used to fill the target intent, the first tag associated with the target intent, and the natural language question into a preset prompt word template to construct the first prompt word.

[0173] The processing unit 603 is used to input the first prompt word into the first large language model and obtain the content to be verified output by the first large language model.

[0174] The determination unit 604 is used to determine the tag of the target intent through the content to be verified.

[0175] In this embodiment of the invention, intent recognition is performed on natural language questions to identify the target intent, and a first major language model is invoked to obtain the content to be verified based on the target intent. The target intent is labeled using the content to be verified, and data mining is performed based on these labels to optimize the business strategy of the intelligent question-answering system.

[0176] Preferred, combined Figure 6 The content shown indicates that the determining unit 604 includes a first determining subunit and a second determining subunit. The execution principle of each subunit is as follows:

[0177] The first judgment subunit is used to determine whether the content to be verified is an empty value or a tag to be verified; if the content to be verified is an empty value, the tag of the target intent is determined to be the second tag; if the content to be verified is a tag to be verified, the tag to be verified is filtered.

[0178] The second judgment subunit is used to determine whether the filtered tag to be verified is any first tag; if the filtered tag to be verified is any first tag, the tag of the target intent is determined to be the filtered tag to be verified; if the filtered tag to be verified is not any first tag, the tag of the target intent is determined to be the second tag.

[0179] Preferred, combined Figure 6 The annotation device also includes the following:

[0180] The storage unit is used to store the target intent and its tag in the database.

[0181] The statistics unit is used to respond to data statistics instructions and perform statistics on the data stored in the database to obtain the data statistics results.

[0182] The display unit is used to generate and display data charts based on the statistical results.

[0183] Preferred, combined Figure 6 The content shown indicates that the identification unit 601 includes a matching subunit, an assembly subunit, and a selection subunit; the execution principle of each subunit is as follows:

[0184] The matching subunit is used to perform keyword matching or regular expression matching on the natural language question input by the user. If any basic intent is matched, the matched basic intent is determined to be the user's target intent.

[0185] The assembly subunit is used to assemble the natural language question and the user's chat history into a second prompt if no basic intent is matched.

[0186] The selection sub-unit is used to input the second prompt word into the second language model to obtain the confidence of each basic intent output by the second language model; the basic intent with the highest confidence is selected as the user's target intent.

[0187] Preferred, combined Figure 6 The annotation device also includes the following:

[0188] The routing unit is used to route natural language questions to the business modules associated with the target intent for business response.

[0189] Preferably, the present invention also provides an electronic device, including: a processor and a memory, the processor and the memory being connected via a bus; wherein, the processor is used to call and execute a program stored in the memory; the memory is used to store the program, the program being used to implement the annotation method provided in the above method embodiments.

[0190] Preferably, the present invention also provides a storage medium storing computer-executable instructions for executing the annotation method provided in the above-described method embodiments.

[0191] In summary, the embodiments of the present invention provide a labeling method, apparatus, electronic device, and storage medium that perform intent recognition on natural language questions to identify the target intent, invoke a first major language model to obtain the content to be verified based on the target intent, determine the tag of the target intent through the content to be verified, and perform data mining based on the tag of the target intent to optimize the business strategy of the intelligent question answering system.

[0192] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0193] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0194] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A labeling method characterized by comprising: The method is applicable to intelligent question-answering systems, and the method includes: The user's input natural language question is subjected to intent recognition in order to identify the user's target intent, which is any preset basic intent; The target intent, the first tag associated with the target intent, and the natural language question are filled into a preset prompt word template to construct the first prompt word; Input the first prompt word into the first large language model to obtain the content to be verified output by the first large language model; The tag of the target intent is determined by the content to be verified.

2. The method of claim 1, wherein, Determining the tags of the target intent through the content to be verified includes: Determine whether the content to be verified is an empty value or a tag to be verified; If the content to be verified is an empty value, the tag of the target intent is determined to be the second tag; If the content to be verified is the tag to be verified, then the tag to be verified is filtered. Determine whether the filtered tag to be verified is any of the first tag; If the filtered tag to be verified is any of the first tag, then the tag of the target intent is determined to be the filtered tag to be verified. If the filtered tag to be verified is not any of the first tags, the tag of the target intent is determined to be the second tag.

3. The method according to claim 1, characterized in that, After determining the tag of the target intent through the content to be verified, the method further includes: The target intent and its tag are stored in the database.

4. The method according to claim 3, characterized in that, After storing the target intent and its tag in the database, the process also includes: In response to a data statistics command, the data stored in the database is statistically analyzed to obtain the data statistics results; Data charts are generated and displayed based on the statistical results.

5. The method according to claim 1, characterized in that, Perform intent recognition on natural language questions input by the user to identify the user's target intent, including: Perform keyword matching or regular expression matching on the natural language questions input by the user; If any basic intent is matched, the matched basic intent is determined to be the user's target intent; If no basic intent is matched, the natural language question and the user's historical chat history are combined into a second prompt word; Input the second prompt word into the second language model to obtain the confidence level of each of the basic intentions output by the second language model; The basic intent with the highest confidence level is selected as the user's target intent.

6. The method according to claim 1, characterized in that, After performing intent recognition on the natural language questions input by the user, the process also includes: The natural language question is routed to the business module associated with the target intent for a business response.

7. A labeling device, characterized in that, The device is suitable for intelligent question-answering systems, and the device includes: The recognition unit is used to perform intent recognition on the natural language questions input by the user, so as to identify the user's target intent, which is any preset basic intent; A construction unit is used to fill the target intent, the first tag associated with the target intent, and the natural language question into a preset prompt word template to construct the first prompt word; The processing unit is used to input the first prompt word into the first large language model and obtain the content to be verified output by the first large language model. A determining unit is used to determine the tag of the target intent based on the content to be verified.

8. The apparatus according to claim 7, characterized in that, The determining unit includes: The first judgment subunit is used to determine whether the content to be verified is an empty value or a tag to be verified; if the content to be verified is an empty value, the tag of the target intent is determined to be a second tag; if the content to be verified is the tag to be verified, the tag to be verified is filtered. The second judgment subunit is used to determine whether the filtered tag to be verified is any of the first tags; if the filtered tag to be verified is any of the first tags, the tag of the target intent is determined to be the filtered tag to be verified; if the filtered tag to be verified is not any of the first tags, the tag of the target intent is determined to be the second tag.

9. An electronic device, characterized in that, include: A processor and a memory are connected via a bus; wherein the processor is used to call and execute a program stored in the memory; The memory is used to store a program for implementing the annotation method as described in any one of claims 1-6.

10. A storage medium, characterized in that, The storage medium stores computer-executable instructions for performing the annotation method as described in any one of claims 1-6.

Images