Training set construction method and device for intention recognition, equipment and medium

Abstract

The embodiment of the invention provides a training set construction method and device for intention recognition, equipment and a medium, and is used for solving the problem that when large-scale sample recognition information is manually constructed in related technologies, the sample recognition information is relatively single, and deviation exists between a corresponding intention and a real intention, so that the accuracy of model training is influenced. In the embodiment of the invention, the electronic equipment constructs the sample identification information based on the accurate identification information, the corresponding labeling intention and the large language model, so that the accuracy of the sample identification information and the corresponding target labeling intention can be improved, and the accuracy of the target labeling intention is improved. According to the technical scheme of the invention, the large language model carries out the construction of the sample recognition information based on each pre-stored intention and the description information of each intention, so that the understanding of the large language model on the intention can be improved, the accuracy of the sample recognition information and the corresponding target annotation intention can be improved, and the accuracy of model training can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, equipment and medium for constructing a training set for intent recognition. Background Art

[0002] In the product intent recognition scenario, when a user enters a keyword or question in a specific product, the built-in system of the specific product can recognize the intent according to the user's needs, and jump to the corresponding page or modify the configuration based on the recognized intent, thereby improving the intelligence level of the product. For example, when a user enters "I want to buy XXX", the system can directly jump to the search page of XXX product; when a user enters "change nickname to XXX", the system will automatically call the nickname modification API to help the user complete the nickname modification. The accuracy of intent recognition determines the user's experience of using the product and the degree of product intelligence. However, large open source or closed source models directly applied to intent recognition have poor results.

[0003] In order to improve the accuracy of intent recognition, the model needs to be fine-tuned or specially trained for a specific training set. The traditional approach relies on pure manual annotation to construct sample recognition information. Although the data quality is reliable, the efficiency is low. Especially when constructing large-scale sample recognition information, not only is the workload large and the repetition is high, but the evaluation standards followed by different business personnel may be different, resulting in relatively simple sample recognition information, and there is a deviation between the corresponding intent and the real intent, which in turn affects the accuracy of model training. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device and medium for constructing a training set for intent recognition, which is used to solve the problem in the related technology that when large-scale sample recognition information is manually constructed, the sample recognition information is relatively simple and there is a deviation between the corresponding intent and the actual intent, which in turn affects the accuracy of model training.

[0005] In a first aspect, an embodiment of the present application provides a method for constructing a training set for intent recognition, the method comprising:

[0006] Receive each piece of identification information that has been constructed and the marking intent of each piece of identification information; obtain each pre-saved intent and description information of each intent;

[0007] Inputting each of the obtained identification information and the corresponding annotation intent, each pre-saved intent and the corresponding description information, and any preset command statement into the first large language model, obtaining each sample identification information output by the first large language model and the target annotation intent corresponding to each sample identification information, wherein the command statement is used to prompt the first large language model to construct sample identification information based on each of the identification information and the corresponding annotation intent;

[0008] Each sample identification information and the target labeling intent corresponding to each sample identification information are added to the training set and saved.

[0009] In a second aspect, an embodiment of the present application further provides a training set construction device for intent recognition, the device comprising:

[0010] A receiving and obtaining module is used to receive each identification information constructed and the marking intention of each identification information; obtain each intention saved in advance and the description information of each intention;

[0011] A processing module is used to input each of the obtained identification information and the corresponding labeling intent, each pre-saved intent and the corresponding description information, and any preset command statement into the first large language model, and obtain each sample identification information output by the first large language model and the target labeling intent corresponding to each sample identification information, wherein the command statement is used to prompt the first large language model to construct sample identification information based on each identification information and the corresponding labeling intent; each sample identification information and the target labeling intent corresponding to each sample identification information are added to a training set and saved.

[0012] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0013] A memory for storing program instructions;

[0014] A processor is used to call the program instructions stored in the memory, and execute the steps included in the method of constructing a training set for intent recognition described in any of the above items according to the obtained program instructions.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes a method for constructing a training set for intent recognition as described in any of the above items.

[0016] In an embodiment of the present application, the electronic device receives each identification information and the labeling intention of each identification information that has been constructed; obtains each pre-saved intention and the description information of each intention; inputs each identification information and the corresponding labeling intention, each pre-saved intention and the corresponding description information, and any preset command statement into the first large language model, and obtains each sample identification information output by the first large language model and the target labeling intention corresponding to each sample identification information, wherein the command statement is used to prompt the first large language model to construct sample identification information based on each identification information and the corresponding labeling intention; each sample identification information and the target labeling intention corresponding to each sample identification information are added to the training set and saved. Because in the embodiment of the present application, the electronic device constructs the sample identification information based on accurate identification information and the corresponding labeling intention, as well as the large language model, the accuracy of the sample identification information and the corresponding target labeling intention can be improved, and the large language model constructs the sample identification information based on each pre-saved intention and the description information of each intention, which can improve the understanding of the large language model on the intention, thereby improving the accuracy of the sample identification information and the corresponding target labeling intention, thereby improving the accuracy of the model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic diagram of the process of constructing a training set for intent recognition provided in an embodiment of the present application;

[0019] Figure 2 A detailed process diagram of constructing a training set provided in an embodiment of the present application;

[0020] Figure 3 A structural diagram of a training set construction device for intent recognition provided in an embodiment of the present application;

[0021] Figure 4 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0023] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0024] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0025] The words "connected", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. The terms "first", "second", "third" and the like involved in this application are merely used to distinguish similar objects, and do not represent a specific ordering of objects.

[0026] In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0027] In order to solve the problem in the related technology that when large-scale sample identification information is manually constructed, the sample identification information is relatively simple, and there is a deviation between the corresponding intention and the actual intention, which in turn affects the accuracy of model training, the embodiments of the present application provide a method, device, equipment and medium for constructing a training set for intent recognition.

[0028] The method for constructing a training set for intent recognition includes: an electronic device receives each identification information and the labeling intent of each identification information that has been constructed; obtains each intention and the description information of each intention that has been saved in advance; inputs each identification information obtained and the corresponding labeling intent, each intention saved in advance and the corresponding description information, and any preset command statement into a first large language model, obtains each sample identification information output by the first large language model and the target labeling intent corresponding to each sample identification information, wherein the command statement is used to prompt the first large language model to construct sample identification information based on each identification information and the corresponding labeling intent; and adds each sample identification information and the target labeling intent corresponding to each sample identification information to a training set and saves it.

[0029] Embodiment 1:

[0030] Figure 1 A schematic diagram of a process for constructing a training set for intent recognition provided in an embodiment of the present application, the process comprising the following steps:

[0031] S101: Receive each piece of identification information that has been constructed and the annotation intent of each piece of identification information; obtain each pre-saved intent and description information of each intent.

[0032] The method for constructing a training set for intent recognition provided in an embodiment of the present application is applied to an electronic device, which may be a smart device such as a PC or a server.

[0033] In order to construct a training set, the electronic device may first receive each piece of identification information that has been constructed and the labeling intention of each piece of identification information.

[0034] The identification information and the corresponding annotation intent can be referred to as seed data. In the embodiment of the present application, manual annotation is used to generate several seed data for the intent. The manually annotated annotation intent can also be referred to as a label.

[0035] For example, the identification information "Please help me change my nickname to XXX" has the corresponding annotation intention: "Modify nickname"; the identification information "Please help me check my points balance" has the corresponding annotation intention: "Check balance".

[0036] In order to realize the construction of the training set, the electronic device also obtains each pre-saved intention and the description information of each intention, wherein the description information of the intention includes a paragraph describing the specific meaning of the intention and the triggering scene of the intention.

[0037] S102: Input each of the obtained recognition information and the corresponding annotation intent, each pre-saved intent and the corresponding description information, and any preset command statement into the first large language model, and obtain each sample recognition information output by the first large language model and the target annotation intent corresponding to each sample recognition information, wherein the command statement is used to prompt the first large language model to construct sample recognition information based on each recognition information and the corresponding annotation intent.

[0038] In order to construct a training set, the electronic device can input each acquired recognition information and the corresponding annotation intent, each pre-saved intent and the corresponding description information into a large language model. For easy distinction, the large language model can be called the first large language model.

[0039] In order to enable the first language model to construct sample recognition information, the electronic device also stores a plurality of preset command statements, wherein the command statements are used to prompt the first language model to construct sample recognition information based on each recognition information and the corresponding annotation intention. For example, the command statement may be: construct similar recognition information and corresponding intention based on the example.

[0040] The electronic device can input any command sentence, each acquired identification information and the corresponding annotation intention, each pre-stored intention and the corresponding description information into the first language model.

[0041] The electronic device can obtain the output of the first large language model, where the output of the first large language model is each generated sample identification information and the target labeling intention corresponding to each sample identification information.

[0042] It should be noted that in order to improve the accuracy of the sample recognition information and the corresponding target labeling intention constructed by the first language model, the electronic device can also input a sentence that presets the first language model into the first language model. For example, the sentence can be "Take a deep breath and think carefully..."

[0043] S103: Add each sample identification information and the target labeling intent corresponding to each sample identification information to a training set and save them.

[0044] The electronic device can add each sample identification information output by the first language model and the target labeling intent corresponding to each sample identification information to the training set and save them.

[0045] In an embodiment of the present application, the intent recognition model may be trained based on the sample recognition information in the training set and the corresponding target annotation intent. For example, the information input into the intent recognition model may be:

[0046] "As a smart assistant for home services, please return the most matching intent in the following intent list based on the user input.

[0047] Intent List

[0048] {INTENT_LIST}

[0049]

important

[0050] Take a deep breath and think carefully….

[0051] Returns the most matching intent in the intent list

[0052]

Return format

[0053] Returns the name of one and only one intent. No other text is allowed.

[0054] [Example]

[0055] Question: How to improve body immunity in daily life?

[0056] Answer: Health knowledge. ”

[0057] The embodiment of the present application is equivalent to providing a method for constructing an industry intention classification training set, and also provides a method for constructing industry instruction fine-tuning data.

[0058] Currently, large language models are empowering thousands of industries. However, these models are usually trained based on a large amount of public data, and face the challenge of a lack of massive industry training sets. Although large original models perform well in common fields, industry data often contains unique knowledge, terminology, rules, processes, and logic, which makes it difficult to be fully covered in general training sets. Therefore, in specific industry scenarios, the effect of large models is poor. This application proposes a method for constructing an industry field intent recognition training set, which aims to efficiently fine-tune large language models and improve their performance in specific industries and downstream tasks. In this way, industry-specific knowledge and rules can be better covered, thereby improving the performance of large language models in practical applications.

[0059] Moreover, in order to improve the performance and generalization ability of the first language model, diversified recognition information can be constructed, such as multiple-choice questions and judgment questions. Among them, the multiple-choice question can be "please select the intent closest to the question from the following intents", and the judgment question can be "how to improve physical immunity in daily life should be classified as health knowledge or as modifying the nickname".

[0060] Since in the embodiments of the present application, the electronic device constructs sample recognition information based on accurate recognition information and corresponding labeling intentions, as well as a large language model, the accuracy of the sample recognition information and the corresponding target labeling intentions can be improved, and the large language model constructs sample recognition information based on pre-saved description information of each intent and each intent, which can improve the large language model's understanding of the intent, thereby improving the accuracy of the sample recognition information and the corresponding target labeling intentions, thereby improving the accuracy of the model training.

[0061] Embodiment 2:

[0062] In order to accurately and effectively construct a training set, based on the above embodiment, in an embodiment of the present application, after obtaining each sample identification information output by the first language model and the target labeling intent corresponding to each sample identification information, before adding each sample identification information and the target labeling intent corresponding to each sample identification information to the training set and saving it, the method further includes:

[0063] Inputting each sample identification information and the target labeling intention corresponding to each sample identification information into the second largest language model; so that the second largest language model modifies and filters each sample identification information and the target labeling intention corresponding to each sample identification information;

[0064] Based on each sample identification information and the target labeling intention corresponding to each sample identification information output by the second largest language model, a subsequent step of adding each sample identification information and the target labeling intention corresponding to each sample identification information to a training set and saving the steps is performed.

[0065] Since the accuracy of the sample identification information and the corresponding target labeling intention output by the first largest language model is low, in order to improve the accuracy of the sample information and the corresponding target labeling intention, the electronic device can modify and filter each sample identification information and the corresponding target labeling intention based on the second largest language model.

[0066] Specifically, after obtaining each sample identification information and the corresponding target labeling intent output by the first large language model, the electronic device can input each sample identification information and the target labeling intent corresponding to each sample identification information into the second large language model, so that the second large language model modifies and filters each sample identification information and the target labeling intent corresponding to each sample identification information. The second large language model and the first large language model can be the same large language model or different large language models.

[0067] The electronic device can obtain each sample identification information output by the second largest language model and the target labeling intention corresponding to each sample identification information, and based on each sample identification information output by the second largest language model and the target labeling intention corresponding to each sample identification information, perform subsequent steps of adding each sample identification information and the target labeling intention corresponding to each sample identification information to a training set and saving them.

[0068] Figure 2 A detailed process diagram of constructing a training set provided in an embodiment of the present application, the process includes the following steps:

[0069] S201: Receive each piece of identification information that has been constructed and the annotation intention of each piece of identification information.

[0070] S202: Acquire each pre-saved intent and description information of each intent.

[0071] S203: Input each acquired recognition information and the corresponding annotation intent, each pre-stored intent and the corresponding description information, and any preset command sentence into the first language model, wherein the command sentence is used to prompt the first language model to construct sample recognition information based on each recognition information and the corresponding annotation intent.

[0072] S204: Obtain each sample identification information output by the first largest language model and the target labeling intention corresponding to each sample identification information.

[0073] S205: Input each sample identification information and the target labeling intention corresponding to each sample identification information into the second largest language model; so that the second largest language model modifies and filters each sample identification information and the target labeling intention corresponding to each sample identification information.

[0074] S206: Add each sample identification information output by the second largest language model and the target labeling intention corresponding to each sample identification information to the training set and save them.

[0075] Embodiment 3:

[0076] In order to accurately and effectively construct a training set, based on the above embodiments, in the embodiment of the present application, before inputting each of the acquired identification information and the corresponding annotation intent, each pre-stored intent and the corresponding description information, and any preset command statement into the first language model, the method further includes:

[0077] Receive a task description of a model to be trained corresponding to a training set to be constructed;

[0078] The step of inputting each of the obtained identification information and the corresponding annotation intention, each of the pre-stored intentions and the corresponding description information, and any preset command statement into the first language model includes:

[0079] Input each of the obtained identification information and the corresponding annotation intention, each pre-saved intention and the corresponding description information, the task description and any preset command statement into the first language model.

[0080] In order to make each sample identification information constructed by the first language model fit the actual usage scenario, the electronic device can also input the task description of the model trained corresponding to the training set into the first language model, so that each sample identification information output by the first language model fits the corresponding task description.

[0081] Specifically, before the electronic device inputs each acquired identification information and the corresponding annotation intent, each pre-saved intent and the corresponding description information, and any preset command statement into the first language model, the electronic device receives the task description of the model to be trained corresponding to the training set to be constructed. Specifically, the business personnel can select the task description of the model to be trained corresponding to the training set to be constructed on the preset page of the device they use or the preset device, and click a preset button, such as the "Generate" button, and the electronic device can receive the task description of the model to be trained corresponding to the training set to be constructed.

[0082] After receiving the task description, the electronic device inputs each acquired identification information and the corresponding annotation intention, each pre-saved intention and the corresponding description information, the task description and any preset command statement into the first large language model, so that the first large language model can input each sample identification information that fits the task description and the target annotation intention corresponding to the sample identification information.

[0083] In the embodiment of the present application, the electronic device uses a large language model to generate and expand new recognition information based on seed data and task descriptions, multiple intents and descriptions of intents, etc., and modifies and filters the generated recognition information through the large language model to ensure data quality. This process is iterated multiple times until there is enough seed data.

[0084] Embodiment 4:

[0085] In order to accurately and effectively construct a training set, based on the above embodiments, in the embodiment of the present application, the method further includes:

[0086] For each sample identification information, the sample identification information is input into a third language model to obtain the recognition labeling intent output by the third language model; if the recognition labeling information is different from the target labeling intent corresponding to the sample identification information, the sample identification information, the recognition labeling intent and the target labeling intent corresponding to the sample identification information are sent to a preset device; the standard recognition intent returned by the preset device is received, and if the standard recognition intent is different from the target labeling intent corresponding to the sample identification information, the standard recognition intent is used to replace the target labeling intent corresponding to the sample identification information in the training set.

[0087] Since the target annotation intent corresponding to the sample annotation information may be inaccurate, in order to improve the accuracy of the target annotation intent, the electronic device can determine the target annotation intent based on a large language model and business personnel.

[0088] Specifically, the electronic device can input the currently processed sample identification information into the large language model for each sample identification information. For the sake of distinction, the large language model can be called the third language model, and the electronic device waits for and receives the intent output by the third language model. For the sake of distinction, the intent can be called the recognition annotation intent. This annotation intent is the third language model's understanding of the input information, and it is expressed in a structured manner.

[0089] The electronic device compares the recognition labeling intent output by the third language model with the target labeling intent corresponding to the obtained sample recognition information. If a difference is found between the two, it can be determined that the target labeling intent may be wrong. In order to further determine the accurate intent corresponding to the sample recognition information, the business personnel can determine the accurate intent based on the preset device. If the two are found to be consistent, it can be determined that the target labeling intent corresponding to the sample recognition information is accurate.

[0090] If it is determined that the target annotation intent corresponding to the sample identification information is not consistent with the identification annotation intent output by the third language model, the electronic device can send the inconsistent sample identification information, the identification annotation intent output by the third language model, and the target annotation intent originally corresponding to the sample identification information to a preset review or processing device. This preset device may be a manual review system or another more advanced language processing model for further analysis and judgment. It can be reviewed by business personnel at the preset device. Specifically, the business personnel can select the accurate intent at the preset device and click the "OK" button. At this time, the electronic device can receive the standard recognition intent received and returned from the preset device. This standard recognition intent is the final understanding and annotation of the sample identification information by the preset device.

[0091] The electronic device may compare the received standard recognition intent with the original target labeling intent again. If the standard recognition intent is still different from the original target labeling intent, the standard recognition intent is used to replace the target labeling intent corresponding to the sample recognition information in the training set. After replacing the target labeling intent, the training set is updated to ensure that the subsequent training process can be based on more accurate data.

[0092] Embodiment 5:

[0093] In order to accurately and effectively construct a training set, based on the above embodiments, in an embodiment of the present application, if the standard recognition intent and the target labeling intent corresponding to the sample recognition information are different, the method further includes:

[0094] Inputting the standard recognition intent, the sample recognition information, and a preset expansion sentence into the first large language model; wherein the expansion sentence is used to prompt the first large language model to output a batch of recognition information and corresponding intents whose similarity with the sample recognition information is higher than a threshold;

[0095] Each output recognition information and corresponding intent output by the first large language model are added to the training set and saved.

[0096] When the standard recognition intent is inconsistent with the target labeling intent corresponding to the given sample recognition information, in order to enrich the recognition diversity related to the sample recognition information, the electronic device can input the standard recognition intent, the sample recognition information and a preset expansion sentence into the first language model. This expansion sentence is intended to guide the first language model to generate a series of recognition information and its corresponding intent whose similarity with the sample recognition information exceeds a preset threshold, thereby achieving effective expansion of the recognition information.

[0097] After successfully obtaining the output results of the first language model, the electronic device will further process the data. Specifically, each output recognition information and corresponding intent output by the first language model are added to the training set and saved. This process not only ensures that the newly generated recognition information and its intent can be fully utilized, but also provides a richer and more diverse training set for subsequent model training. At the same time, in order to ensure the persistence and accessibility of the data, the electronic device will also properly save the newly added training data to the storage device.

[0098] That is to say, in the embodiment of the present application, the performance of the large language model (LLM) on the training set is continuously evaluated, data that is prone to errors or confusion by the LLM is selected, and similar data of the same type is continued to be expanded using the large language model and added to the training set again, and the iteration is repeated continuously.

[0099] Embodiment 6:

[0100] In order to improve the learning ability of the first language model and thus realize the construction of the training set, on the basis of the above embodiments, in the embodiment of the present application, after receiving each identification information and the annotation intention of each identification information that has been constructed, before inputting each identification information and the corresponding annotation intention, each intention and the corresponding description information saved in advance, and any preset command statement into the first language model, the method further includes:

[0101] Identify the similarity of each identification information; determine the identification information with a similarity higher than a threshold as the identification information in the target information group, and determine the identification information with a similarity lower than the threshold to each identification information as the identification information in the candidate information group;

[0102] Arrange each target information group at least according to the number of identification information contained in each target information group; arrange the identification information in the candidate information group between two identification information in the target information group;

[0103] Based on each of the identified information after arrangement, a subsequent step of inputting each of the identified information and the corresponding annotation intention obtained, each of the intentions and the corresponding description information saved in advance, and any preset command statement into the first language model is performed.

[0104] Since some input examples may be ignored by the first language model during processing. To overcome this problem, the electronic device can adopt a strategy to adjust the structure of the input examples. Specifically, by carefully arranging the position of high-similarity recognition information in the example, the electronic device aims to reduce the risk of intermediate information discarding that may occur when the first language model processes long sequences or complex documents. This strategic position adjustment is intended to optimize the flow of information and ensure that high-similarity problems can be fully captured and processed by the first language model, thereby enhancing the model's learning ability and output accuracy.

[0105] Specifically, the electronic device can identify the similarity of each identification information. In a possible implementation, the electronic device can determine the vector corresponding to each identification information based on a variety of encoding methods, such as One-Hot encoding, etc. Specifically, these encoding methods can convert text or identification information into digital vectors with a specific structure. Once each identification information is converted into a vector form, the electronic device can use these vectors to calculate the similarity between them. Among them, cosine similarity is a commonly used measurement method. This method evaluates the similarity between two vectors by calculating the cosine value of the angle between them. The closer the cosine value is to 1, the more consistent the directions of the two vectors are, that is, the higher the similarity between them; conversely, the closer the cosine value is to -1, the more opposite the directions of the two vectors are, that is, the lower the similarity between them.

[0106] The electronic device can set a similarity threshold to determine which identification information should be included in the target information group. Specifically, if the similarity between any two identification information is higher than the threshold, the two identification information will be considered similar and included in the same target information group. In this way, the electronic device can identify and group multiple such target information groups, and the number of identification information in each target information group may be different. In addition, for those identification information whose similarity with all other identification information is lower than the threshold, the electronic device classifies them into a candidate information group separately. In this way, it can ensure that the identification information with high similarity is correctly classified, and can also identify those relatively independent identification information that are greatly different from other information.

[0107] The electronic device can sort the target information group from most to least according to the number of identification information contained in each target information group. When arranging, the electronic device will insert the identification information in the candidate information group between the identification information of the target information group to achieve orderly display or processing of the information. Specifically, the identification information in the candidate information group can be flexibly inserted between any two identification information in the target information group. This insertion can be uniform, that is, the same number of candidate information is inserted between the identification information of each target information group; it can also be distributed according to the number of identification information in the candidate information group. If the number of identification information in the candidate information group is large, multiple identification information in the candidate information group can be arranged between the identification information of multiple target information groups; if the number of identification information in the candidate information group is small, it can be preferentially arranged between the identification information of the first few target information groups.

[0108] It should be noted that, in practical applications, large language models usually use retrieval-augmented generation (RAG) to detect and extract relevant recognition information and intent as sample data in instruction data. In order to improve the accuracy of the large language model response, the embodiment of the present application will carefully construct a large number of RAG instances containing positive and negative samples in the training set construction stage. Each instance not only contains recognition information, but also adds interference information with low similarity to the recognition information. Among them, a group of high-similarity recognition information can help the model infer the intention of the recognition information. For P% of the recognition information, its high similarity problem and interference problem will be retained, and the position of the high-similarity problem in the example will be strategically adjusted to reduce the intermediate information discarding phenomenon that may occur when the large language model processes long sequences or complex documents. For the remaining (1-P)% of the recognition information, only interference information is included, and the large language model is trained to use only high-similarity recognition information for reasoning, thereby reducing the interference of irrelevant problems and improving the accuracy and efficiency of the model. Such a design is intended to make full use of the advantages of RAG technology and improve the performance of large language models in practical applications by constructing rich and diverse training examples.

[0109] Embodiment 7:

[0110] In order to accurately and effectively construct a training set, based on the above embodiments, in an embodiment of the present application, after obtaining each sample identification information output by the first language model and the target labeling intent corresponding to each sample identification information, before adding each sample identification information and the target labeling intent corresponding to each sample identification information to the training set and saving it, the method further includes:

[0111] For each sample identification information, determine whether the length of the sample identification information exceeds a preset length. If so, delete the sample identification information and the corresponding target labeling intent.

[0112] During the model training process, if the identification information is too long, it may lead to poor training results. Therefore, the electronic device can determine whether the length of each sample identification information exceeds the preset length. If the length of the sample identification information exceeds the preset length, the sample identification information and the corresponding target labeling intention will be deleted.

[0113] Among them, in the process of fine-tuning the large language model based on the training set, if only the training set constructed above is used, it may cause catastrophic forgetting of the large language model, resulting in a serious decline in general ability and reasoning ability. To prevent this problem, some open source instruction data needs to be added. Therefore, high-quality recognition information that is beneficial to fine-tuning can be screened out from a large number of open source instruction training sets. Specifically, a large number of public instruction training sets are collected, and some indicators such as instruction length and instruction diversity are first defined to filter out most of the data. Then, through an open source small language model, the perplexity of each recognition information is calculated, thereby screening out a small amount of high-quality recognition information, and these recognition information and corresponding intents are also added to the training set.

[0114] It should be noted that traditional methods include pure manual annotation, constructing identification information and corresponding intentions based on existing texts with the help of large language models, and performing multiple iterations and expansions based on existing seed data. However, the quality of manually annotated data is high, but the efficiency is low. This method of constructing identification information and corresponding intentions based on existing texts with the help of large language models is efficient, but the data quality is relatively low. Performing multiple iterations and expansions based on existing seed data is more applied to the construction of public data and is rarely combined with specific industry scenarios. The method provided in the embodiments of the present application can be combined with specific industry products and improve the quality of data and generation efficiency.

[0115] Embodiment 8:

[0116] Based on the same inventive concept, the present application embodiment provides a training set construction device for intent recognition, please refer to Figure 3 , the device comprises:

[0117] The receiving and obtaining module 301 is used to receive each identification information constructed and the marking intention of each identification information; obtain each intention and the description information of each intention saved in advance;

[0118] The processing module 302 is used to input each of the obtained identification information and the corresponding labeling intent, each pre-saved intent and the corresponding description information, and any preset command statement into the first large language model, and obtain each sample identification information output by the first large language model and the target labeling intent corresponding to each sample identification information, wherein the command statement is used to prompt the first large language model to construct sample identification information based on each identification information and the corresponding labeling intent; each sample identification information and the target labeling intent corresponding to each sample identification information are added to the training set and saved.

[0119] In a possible implementation, the processing module 302 is further used to input each sample identification information and the target labeling intent corresponding to each sample identification information into a second largest language model; so that the second largest language model can modify and filter each sample identification information and the target labeling intent corresponding to each sample identification information; based on each sample identification information and the target labeling intent corresponding to each sample identification information output by the second largest language model, perform subsequent steps of adding each sample identification information and the target labeling intent corresponding to each sample identification information to a training set and saving the steps.

[0120] In a possible implementation, the processing module 302 is further configured to receive a task description of a model to be trained corresponding to the training set to be constructed;

[0121] The processing module 302 is specifically used to input each of the obtained identification information and the corresponding annotation intention, each pre-saved intention and the corresponding description information, the task description and any preset command statement into the first large language model.

[0122] In a possible implementation, the processing module 302 is also used to input each sample identification information into a third language model to obtain the recognition labeling intent output by the third language model; if the recognition labeling information is different from the target labeling intent corresponding to the sample identification information, the sample identification information, the recognition labeling intent and the target labeling intent corresponding to the sample identification information are sent to a preset device; receive the standard recognition intent returned by the preset device, and if the standard recognition intent is different from the target labeling intent corresponding to the sample identification information, use the standard recognition intent to replace the target labeling intent corresponding to the sample identification information in the training set.

[0123] In a possible implementation, the processing module 302 is further used to input the standard recognition intent, the sample recognition information and a preset expansion sentence into the first large language model if the standard recognition intent and the target labeling intent corresponding to the sample recognition information are different; wherein the expansion sentence is used to prompt the first large language model to output a batch of recognition information and corresponding intents whose similarity with the sample recognition information is higher than a threshold; and each output recognition information and corresponding intent output by the first large language model is added to the training set and saved.

[0124] In a possible implementation, the processing module 302 is also used to identify the similarity of each identification information; determine the identification information with a similarity higher than a threshold as the identification information in the target information group, and determine the identification information with a similarity with each identification information lower than the threshold as the identification information in the candidate information group; arrange each target information group by at least a number of identification information contained in each target information group; arrange the identification information in the candidate information group between two identification information in the target information group; and based on each identification information after arrangement, execute the subsequent step of inputting each obtained identification information and the corresponding annotation intention, each pre-saved intention and the corresponding description information, and any preset command statement into the first language model.

[0125] In a possible implementation, the processing module 302 is further configured to determine, for each piece of sample identification information, whether the length of the sample identification information exceeds a preset length, and if so, delete the sample identification information and the corresponding target labeling intent.

[0126] Embodiment 9:

[0127] Based on the same inventive concept, the present application embodiment provides an electronic device, which can realize the function of constructing a training set for the above-mentioned intent recognition. Please refer to Figure 4 The device includes a processor 401, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0128] The memory 403 stores a computer program. When the program is executed by the processor 401, the processor 401 performs the following steps:

[0129] Receive each piece of identification information that has been constructed and the marking intent of each piece of identification information; obtain each pre-saved intent and description information of each intent;

[0130] Inputting each of the obtained identification information and the corresponding annotation intent, each pre-saved intent and the corresponding description information, and any preset command statement into the first large language model, obtaining each sample identification information output by the first large language model and the target annotation intent corresponding to each sample identification information, wherein the command statement is used to prompt the first large language model to construct sample identification information based on each of the identification information and the corresponding annotation intent;

[0131] Each sample identification information and the target labeling intent corresponding to each sample identification information are added to the training set and saved.

[0132] In a possible implementation, after obtaining each sample identification information output by the first large language model and the target labeling intent corresponding to each sample identification information, and before adding each sample identification information and the target labeling intent corresponding to each sample identification information to a training set and saving the information, the method further includes:

[0133] Inputting each sample identification information and the target labeling intention corresponding to each sample identification information into the second largest language model; so that the second largest language model modifies and filters each sample identification information and the target labeling intention corresponding to each sample identification information;

[0134] Based on each sample identification information and the target labeling intention corresponding to each sample identification information output by the second largest language model, a subsequent step of adding each sample identification information and the target labeling intention corresponding to each sample identification information to a training set and saving the steps is performed.

[0135] In a possible implementation, before inputting each of the acquired identification information and the corresponding annotation intent, each pre-stored intent and the corresponding description information, and any preset command statement into the first language model, the method further includes:

[0136] Receive a task description of a model to be trained corresponding to a training set to be constructed;

[0137] The step of inputting each of the obtained identification information and the corresponding annotation intention, each of the pre-stored intentions and the corresponding description information, and any preset command statement into the first language model includes:

[0138] Input each of the obtained identification information and the corresponding annotation intention, each pre-saved intention and the corresponding description information, the task description and any preset command statement into the first language model.

[0139] In a possible implementation, the method further includes:

[0140] For each sample identification information, the sample identification information is input into a third language model to obtain the recognition labeling intent output by the third language model; if the recognition labeling information is different from the target labeling intent corresponding to the sample identification information, the sample identification information, the recognition labeling intent and the target labeling intent corresponding to the sample identification information are sent to a preset device; the standard recognition intent returned by the preset device is received, and if the standard recognition intent is different from the target labeling intent corresponding to the sample identification information, the standard recognition intent is used to replace the target labeling intent corresponding to the sample identification information in the training set.

[0141] In a possible implementation manner, if the standard recognition intent and the target labeling intent corresponding to the sample recognition information are different, the method further includes:

[0142] Inputting the standard recognition intent, the sample recognition information, and a preset expansion sentence into the first large language model; wherein the expansion sentence is used to prompt the first large language model to output a batch of recognition information and corresponding intents whose similarity with the sample recognition information is higher than a threshold value;

[0143] Each output recognition information and corresponding intent output by the first large language model are added to the training set and saved.

[0144] In a possible implementation, after receiving each identification information and the annotation intent of each identification information, and before inputting each identification information and the corresponding annotation intent, each pre-stored intent and the corresponding description information, and any preset command statement into the first language model, the method further includes:

[0145] Identify the similarity of each identification information; determine the identification information with a similarity higher than a threshold as the identification information in the target information group, and determine the identification information with a similarity lower than the threshold to each identification information as the identification information in the candidate information group;

[0146] Arrange each target information group at least according to the number of identification information contained in each target information group; arrange the identification information in the candidate information group between two identification information in the target information group;

[0147] Based on each of the identified information after arrangement, a subsequent step of inputting each of the identified information and the corresponding annotation intention obtained, each of the intentions and the corresponding description information saved in advance, and any preset command statement into the first language model is performed.

[0148] In a possible implementation, after obtaining each sample identification information output by the first large language model and the target labeling intent corresponding to each sample identification information, and before adding each sample identification information and the target labeling intent corresponding to each sample identification information to a training set and saving the information, the method further includes:

[0149] For each sample identification information, determine whether the length of the sample identification information exceeds a preset length. If so, delete the sample identification information and the corresponding target labeling intent.

[0150] The communication bus mentioned in the above server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0151] The communication interface 402 is used for communication between the electronic device and other devices.

[0152] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0153] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0154] Embodiment 10:

[0155] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device implements the following steps when executing:

[0156] Receive each piece of identification information that has been constructed and the marking intent of each piece of identification information; obtain each pre-saved intent and description information of each intent;

[0157] Inputting each of the obtained identification information and the corresponding annotation intent, each pre-saved intent and the corresponding description information, and any preset command statement into the first large language model, obtaining each sample identification information output by the first large language model and the target annotation intent corresponding to each sample identification information, wherein the command statement is used to prompt the first large language model to construct sample identification information based on each of the identification information and the corresponding annotation intent;

[0158] Each sample identification information and the target labeling intent corresponding to each sample identification information are added to the training set and saved.

[0159] In a possible implementation, after obtaining each sample identification information output by the first large language model and the target labeling intent corresponding to each sample identification information, and before adding each sample identification information and the target labeling intent corresponding to each sample identification information to a training set and saving the information, the method further includes:

[0160] Inputting each sample identification information and the target labeling intention corresponding to each sample identification information into the second largest language model; so that the second largest language model modifies and filters each sample identification information and the target labeling intention corresponding to each sample identification information;

[0161] Based on each sample identification information and the target labeling intention corresponding to each sample identification information output by the second largest language model, a subsequent step of adding each sample identification information and the target labeling intention corresponding to each sample identification information to a training set and saving the steps is performed.

[0162] In a possible implementation, before inputting each of the acquired identification information and the corresponding annotation intent, each pre-stored intent and the corresponding description information, and any preset command statement into the first language model, the method further includes:

[0163] Receive a task description of a model to be trained corresponding to a training set to be constructed;

[0164] The step of inputting each of the obtained identification information and the corresponding annotation intention, each of the pre-stored intentions and the corresponding description information, and any preset command statement into the first language model includes:

[0165] Input each of the obtained identification information and the corresponding annotation intention, each pre-saved intention and the corresponding description information, the task description and any preset command statement into the first language model.

[0166] In a possible implementation, the method further includes:

[0167] For each sample identification information, the sample identification information is input into a third language model to obtain the recognition labeling intent output by the third language model; if the recognition labeling information is different from the target labeling intent corresponding to the sample identification information, the sample identification information, the recognition labeling intent and the target labeling intent corresponding to the sample identification information are sent to a preset device; the standard recognition intent returned by the preset device is received, and if the standard recognition intent is different from the target labeling intent corresponding to the sample identification information, the standard recognition intent is used to replace the target labeling intent corresponding to the sample identification information in the training set.

[0168] In a possible implementation manner, if the standard recognition intent and the target labeling intent corresponding to the sample recognition information are different, the method further includes:

[0169] Inputting the standard recognition intent, the sample recognition information, and a preset expansion sentence into the first large language model; wherein the expansion sentence is used to prompt the first large language model to output a batch of recognition information and corresponding intents whose similarity with the sample recognition information is higher than a threshold;

[0170] Each output recognition information and corresponding intent output by the first large language model are added to the training set and saved.

[0171] In a possible implementation, after receiving each identification information and the annotation intent of each identification information, and before inputting each identification information and the corresponding annotation intent, each pre-stored intent and the corresponding description information, and any preset command statement into the first language model, the method further includes:

[0172] Identify the similarity of each identification information; determine the identification information with a similarity higher than a threshold as the identification information in the target information group, and determine the identification information with a similarity lower than the threshold to each identification information as the identification information in the candidate information group;

[0173] Arrange each target information group at least according to the number of identification information contained in each target information group; arrange the identification information in the candidate information group between two identification information in the target information group;

[0174] Based on each of the identified information after arrangement, a subsequent step of inputting each of the identified information and the corresponding annotation intention obtained, each of the intentions and the corresponding description information saved in advance, and any preset command statement into the first language model is performed.

[0175] In a possible implementation, after obtaining each sample identification information output by the first large language model and the target labeling intent corresponding to each sample identification information, and before adding each sample identification information and the target labeling intent corresponding to each sample identification information to a training set and saving the information, the method further includes:

[0176] For each sample identification information, determine whether the length of the sample identification information exceeds a preset length. If so, delete the sample identification information and the corresponding target labeling intent.

[0177] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0178] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0179] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0180] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.

[0181] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for constructing a training set for intent recognition, characterized in that: The method comprises: Receive each piece of identification information that has been constructed and the marking intent of each piece of identification information; obtain each pre-saved intent and description information of each intent; Inputting each of the obtained identification information and the corresponding annotation intent, each pre-saved intent and the corresponding description information, and any preset command statement into the first large language model, obtaining each sample identification information output by the first large language model and the target annotation intent corresponding to each sample identification information, wherein the command statement is used to prompt the first large language model to construct sample identification information based on each of the identification information and the corresponding annotation intent; Each sample identification information and the target labeling intent corresponding to each sample identification information are added to the training set and saved.

2. The method according to claim 1, characterized in that: After obtaining each sample identification information output by the first large language model and the target labeling intent corresponding to each sample identification information, and before adding each sample identification information and the target labeling intent corresponding to each sample identification information to a training set and saving the information, the method further includes: Inputting each sample identification information and the target labeling intention corresponding to each sample identification information into the second largest language model; so that the second largest language model modifies and filters each sample identification information and the target labeling intention corresponding to each sample identification information; Based on each sample identification information and the target labeling intention corresponding to each sample identification information output by the second largest language model, a subsequent step of adding each sample identification information and the target labeling intention corresponding to each sample identification information to a training set and saving the steps is performed.

3. The method according to claim 1, characterized in that Before inputting each of the acquired identification information and the corresponding annotation intent, each of the pre-stored intents and the corresponding description information, and any preset command statement into the first language model, the method further includes: Receive a task description of a model to be trained corresponding to a training set to be constructed; The step of inputting each of the obtained identification information and the corresponding annotation intention, each of the pre-stored intentions and the corresponding description information, and any preset command statement into the first language model includes: Input each of the obtained identification information and the corresponding annotation intention, each pre-saved intention and the corresponding description information, the task description and any preset command statement into the first language model.

4. The method according to claim 1, characterized in that The method further comprises: For each sample identification information, the sample identification information is input into a third language model to obtain the recognition labeling intent output by the third language model; if the recognition labeling information is different from the target labeling intent corresponding to the sample identification information, the sample identification information, the recognition labeling intent and the target labeling intent corresponding to the sample identification information are sent to a preset device; the standard recognition intent returned by the preset device is received, and if the standard recognition intent is different from the target labeling intent corresponding to the sample identification information, the standard recognition intent is used to replace the target labeling intent corresponding to the sample identification information in the training set.

5. The method according to claim 4, characterized in that If the standard identification intention is different from the target mark corresponding to the sample identification information, the method further includes: Inputting the standard recognition intent, the sample recognition information, and a preset expansion sentence into the first large language model; wherein the expansion sentence is used to prompt the first large language model to output a batch of recognition information and corresponding intents whose similarity with the sample recognition information is higher than a threshold; Each output recognition information and corresponding intent output by the first large language model are added to the training set and saved.

6. The method according to claim 1, characterized in that After receiving each identification information and the annotation intent of each identification information, and before inputting each identification information and the corresponding annotation intent, each pre-stored intent and the corresponding description information, and any preset command statement into the first language model, the method further includes: Identify the similarity of each identification information; determine the identification information with a similarity higher than a threshold as the identification information in the target information group, and determine the identification information with a similarity lower than the threshold to each identification information as the identification information in the candidate information group; Arrange each target information group at least according to the number of identification information contained in each target information group; arrange the identification information in the candidate information group between two identification information in the target information group; Based on each of the identified information after arrangement, a subsequent step of inputting each of the identified information and the corresponding annotation intention obtained, each of the intentions and the corresponding description information saved in advance, and any preset command statement into the first language model is performed.

7. The method according to claim 1, characterized in that After obtaining each sample identification information output by the first large language model and the target labeling intent corresponding to each sample identification information, and before adding each sample identification information and the target labeling intent corresponding to each sample identification information to a training set and saving the information, the method further includes: For each sample identification information, it is determined whether the length of the sample identification information exceeds the preset length. If so, the sample identification information and the corresponding target marker are deleted.

8. A training set construction device for intent recognition, characterized in that: The device comprises: The receiving and obtaining module is used to receive each identification information completed and a marker of the identification information; to obtain pre-save description information of each intention and the description information of each intention; A processing module is used to input each of the obtained identification information and the corresponding labeling intent, each pre-saved intent and the corresponding description information, and any preset command statement into the first large language model, and obtain each sample identification information output by the first large language model and the target labeling intent corresponding to each sample identification information, wherein the command statement is used to prompt the first large language model to construct sample identification information based on each identification information and the corresponding labeling intent; each sample identification information and the target labeling intent corresponding to each sample identification information are added to a training set and saved.

9. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call program instructions stored in the memory and perform the steps of a method of constructing the intent-identified training set as claimed in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a computer, the computer executes the method for constructing a training set for intent recognition as described in any one of claims 1 to 7.

Images