Instruction intent recognition method and apparatus, computing device, storage medium

Abstract

The embodiment of the present disclosure provides a kind of instruction intention identification method and device, computing device, storage medium.The instruction intention identification method includes: using rule identification method to process first instruction to obtain rule identification intention, and using large language model identification method to process first instruction to obtain model identification intention;With the artificial mark intention corresponding to first instruction as reference, determine the preferred identification method associated with each intention type;Using rule identification method and large language model identification method to process second instruction, obtain corresponding two selected identification intentions;Based on the intention type of two selected identification intentions and the preferred identification method associated with each intention type, determine the selected intention in two selected identification intentions, and the selected intention is as the instruction intention of second instruction.The embodiment of the present disclosure provides the evaluation method suitable for various intention types, and in actual application, the instruction intention of second instruction can be determined by selecting the intention recognition result of two identification methods.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning technology, specifically to an instruction intent recognition method and apparatus, computing device, and storage medium. Background Technology

[0002] Intent recognition is a key technology used in human-computer interaction and natural language understanding. Currently, existing intent recognition methods include rule extraction methods based on keywords and semantic similarity calculations, and methods based on large oracle models for direct recognition. While rule-based extraction methods are accurate and effective in specific scenarios, their generalization ability is insufficient because manual methods cannot exhaust all possible rules, making them unsuitable for large-scale, multi-intent scenarios. Model-based recognition methods, while significantly improving the flexibility and breadth of model classification, lose the accuracy of manually constructed rules. Model-based classification methods suffer from instability, even in simple tasks, and exhibit poor recognition of pointers in long texts. For these reasons, it is not possible to directly compare the advantages and disadvantages of rule-based and model-based recognition methods in related technologies, thus hindering the achievement of accurate intent recognition results for instructions with uncertain intent in automated recognition. Summary of the Invention

[0003] To address the problem that existing automated recognition methods struggle to obtain accurate intent recognition results for instructions with uncertain intents, this disclosure provides an instruction intent recognition method and apparatus, a computing device, and a storage medium.

[0004] In a first aspect, embodiments of this disclosure provide a method for identifying instruction intent, including:

[0005] The first instruction in the first instruction set is processed using a rule recognition method to obtain a rule-recognized intent, and the first instruction is processed using a large language model recognition method to obtain a model-recognized intent; the first instruction has a corresponding manually labeled intent; the intent types of the rule-recognized intent, the model-recognized intent, and the manually labeled intent are all intents from a pre-constructed intent type set;

[0006] Taking the manually labeled intent corresponding to the first instruction as a reference, a preferred identification method is determined based on the rule-based identification intent and the model-based identification intent corresponding to the first instruction, wherein the preferred identification method is the rule-based identification method or the large language model identification method.

[0007] The second instruction is processed using the rule recognition method and the large language model recognition method respectively to obtain two candidate recognition intentions corresponding to the second instruction;

[0008] Based on the intent types of the two candidate intents and the preferred identification method associated with each intent type, a selected intent is determined from the two candidate intents, and the selected intent is used as the instruction intent of the second instruction.

[0009] Optionally, the step of processing the first instruction in the first instruction set using a rule recognition method to obtain the rule-recognized intent includes:

[0010] Extract instruction keywords from the first instruction, construct a regular expression based on the instruction keywords, and determine the core semantic vector corresponding to the first instruction based on the regular expression;

[0011] Based on the core semantic vector corresponding to the first instruction and the semantic vectors associated with each intent type, the rule recognition intent corresponding to the first instruction is determined.

[0012] Optionally, the extraction of instruction keywords from the first instruction includes:

[0013] Using pre-constructed keyword extraction prompts and the first instruction as model inputs, the first large language model is used for keyword recognition to obtain the instruction keywords.

[0014] Optionally, constructing a regular expression based on the instruction keywords includes:

[0015] Using pre-built regular expression prompts and the instruction keywords as model input, the second largest language model is used to construct the expression, resulting in the regular expression.

[0016] Optionally, after determining the rule recognition intent corresponding to the first instruction, the method further includes:

[0017] The core semantic vector of the first instruction is associated with the corresponding intent type, and this is used as the semantic vector associated with the corresponding intent type.

[0018] Optionally, after determining the rule recognition intent corresponding to the first instruction, the method further includes:

[0019] Based on the regular expression corresponding to the first instruction and the regular expression associated with the corresponding intent type, expression matching is performed to determine a new regular expression associated with the intent type, and a new semantic vector associated with the corresponding intent type is constructed based on the new regular expression; or,

[0020] Based on the core semantic vector of the first instruction, cluster analysis is performed on the semantic vectors associated with each intent type and the number of intent types to determine a new cluster center vector, which is then used as the semantic vector associated with the corresponding intent type.

[0021] Optionally, the step of processing the first instruction using a large language model recognition method to obtain the model recognition intent includes:

[0022] Using pre-built intent prompts and the first instruction as model input, a third language model is used for intent recognition to obtain the model-recognized intent of the first instruction. The pre-built intent prompts include all intent types in the intent type set.

[0023] Optionally, the preferred identification method for determining the association of each intent type in the intent type set based on the manually labeled intent and the model-identified intent, with reference to the manually labeled intent, includes:

[0024] Based on the rule recognition intent corresponding to the first instruction, determine the number of rules recognized for each intent type, and compare the rule recognition intent corresponding to the first instruction with the corresponding manually labeled intent to determine the number of rules correctly recognized for each intent type.

[0025] Based on the model recognition intent corresponding to the first instruction, determine the number of models recognized for each intent type, and compare the model recognition intent corresponding to the first instruction with the corresponding manually labeled intent to determine the number of models correctly recognized for each intent type.

[0026] Based on the number of rules recognized and the number of rules correctly recognized for each intent type, calculate the rule recognition accuracy for each intent type; and based on the number of models recognized and the number of models correctly recognized for each intent type, calculate the model recognition accuracy for each intent type.

[0027] The rule recognition accuracy and model recognition accuracy for each intent type are compared to determine the one with the higher recognition accuracy. The intent recognition method with the higher recognition accuracy is then selected as the preferred recognition method for that intent type.

[0028] Secondly, embodiments of this disclosure provide an instruction intent recognition device, comprising:

[0029] The intent prediction unit is used to process the first instruction in the first instruction set using a rule recognition method to obtain a rule-recognized intent, and to process the first instruction using a large language model recognition method to obtain a model-recognized intent; the first instruction has a corresponding manually labeled intent; the intent types of the rule-recognized intent, the model-recognized intent, and the manually labeled intent are all intents in a pre-constructed intent type set;

[0030] The method selection unit is used to determine the preferred identification method for each intent type in the intent type set based on the rule-based identification intent and the model-based identification intent, with reference to the manually annotated intent. The preferred identification method is either the rule-based identification method or the large language model identification method.

[0031] The intent prediction unit is also used to process the second instruction using the rule recognition method and the large language model recognition method respectively, to obtain two candidate recognition intents corresponding to the second instruction;

[0032] The result determination unit is used to determine the selected intent from the two candidate identification intents based on the intent types of the two candidate identification intents and the preferred identification method associated with each intent type, and to use the selected intent as the instruction intent of the second instruction.

[0033] Thirdly, embodiments of this disclosure provide a computing device, characterized in that it includes a processor and a memory, the memory being used to store a computer program; when the computer program is loaded by the processor, it causes the processor to execute the instruction intent recognition method as described above.

[0034] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the instruction intent recognition method as described above.

[0035] By employing the scheme of this embodiment, after obtaining the rule-recognized intent and model-recognized intent corresponding to multiple first instructions using the rule recognition method and the large language model recognition method, the two recognized intents are judged by using manually labeled intents as a reference. This allows it to determine which recognition method is more suitable for a specific intent recognition type, and thus the corresponding intent recognition method can be selected as the preferred recognition method for this intent type. Subsequently, after processing new second instructions using the rule recognition method and the large language model recognition method to obtain the intent to be selected, the selected intent can be determined based on the previously determined preferred recognition method, and the selected intent is used as the instruction intent of the second instruction.

[0036] In other words, the present disclosure provides a method for evaluating the adaptability of rule recognition method and large language model recognition method to various intent types. In practical applications, the intent recognition results of the two recognition methods can be selected to determine the instruction intent of the second instruction. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0038] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without any creative effort, wherein...

[0039] Figure 1 This is a flowchart of the instruction intent recognition method provided in the embodiments of this disclosure;

[0040] Figure 2 This is a flowchart of the preferred identification method in some embodiments of this disclosure;

[0041] Figure 3 This is a flowchart of a method for determining rule recognition intent according to some embodiments of this disclosure;

[0042] Figure 4 This is a schematic diagram of the instruction intent recognition device provided in the embodiments of this disclosure;

[0043] Figure 5 This is a schematic diagram of the structure of a computing device provided in an embodiment of this disclosure. Detailed Implementation

[0044] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0045] The term "comprising" and its variations as used herein are open-ended inclusion, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply that these entities or operations are separate entities or operations.

[0046] To address the problem that it is impossible to compare different types of intent recognition methods horizontally, thus making it impossible to determine which method yields more accurate intent recognition and consequently, to accurately determine the intent recognition result, this disclosure provides a new instruction intent recognition method. The instruction intent recognition method provided in this disclosure is executed by a computing device.

[0047] Figure 1 This is a flowchart of the instruction intent recognition method provided in the embodiments of this disclosure. Figure 1 As shown, the instruction intent recognition method provided in this embodiment includes S110-S140.

[0048] S110: The first instruction in the first instruction set is processed using a rule recognition method to obtain the rule recognition intent, and the first instruction is processed using a large language model recognition method to obtain the model recognition intent.

[0049] The first instruction set is a dataset containing a large number of first instructions. The aforementioned first instructions are instructions that contain a specific intent.

[0050] The correct intents corresponding to all first instructions in the first instruction set are contained in the intent type set, and the true intents corresponding to all first instructions in the first instruction set constitute the complete set of the intent type set. The intent type set is a dataset constructed using all possible intents.

[0051] In some embodiments of this disclosure, annotators can construct a multi-level intent system by analyzing user commands and add all intent types in the aforementioned multi-level intent system to an intent type set. For example, in some embodiments, the intent system includes a first-level intent, a second-level intent, and a third-level intent, wherein a second-level intent is a sub-intent of a first-level intent, and a third-level intent is a sub-intent of a second-level intent.

[0052] In some embodiments, while constructing a multi-level intent system, annotators can also provide explanations for the intents in various three-level intent systems. For example, when the three-level intent included in the intent system is word formation and sentence construction, the corresponding explanation can be "select Chinese characters within a given range to perform word formation tasks, or perform word formation or sentence construction given certain words." When the three-level intent included in the intent system is sentence sorting, the corresponding explanation can be "including tasks related to sentence order adjustment such as restoring disordered sentences and reversing the output order; the reorganization of scattered and disordered sentences is also considered sentence sorting."

[0053] Provided that the aforementioned explanations are included in the intent system, the aforementioned explanations can be added to the intent type set as part of the multi-level intent system.

[0054] Furthermore, in this embodiment of the disclosure, all first instructions in the first instruction set have corresponding manually labeled intentions, which are intentions manually labeled. Based on the foregoing analysis, it can be determined that the true intentions corresponding to the first instructions in the first instruction set constitute the complete set of intention type sets.

[0055] In specific implementation, the computing device uses a rule recognition method to process the first instruction in the first instruction set to obtain the rule recognition intent. This can be done by using existing recognition methods to process the first instruction and obtain the rule recognition intent corresponding to each first instruction. Alternatively, the computing device can use a large language model recognition method to process the first instruction in the first instruction set to obtain the model recognition intent. This can also be done by inputting the first instruction into a large language model, as in existing technical solutions, and then using the large language model to process the first instruction to obtain the model recognition intent.

[0056] It should be noted here that the rule recognition intent and model recognition intent obtained by processing the first instruction according to the aforementioned method may be correct or incorrect. Furthermore, the rule recognition intent and model recognition intent corresponding to the same first instruction may be the same or different.

[0057] S120: Using the manually labeled intent corresponding to the first instruction as a reference, determine the preferred identification method for the association of each intent type in the intent type set based on the rule-based intent identification and model-based intent identification corresponding to the first instruction.

[0058] As analyzed earlier, the manually labeled intent is the pre-determined correct intent corresponding to each first instruction. Accordingly, the manually labeled intent corresponding to each first instruction can be used as a reference intent to determine whether the corresponding rule recognition intent and the intent recognized by the model are correct.

[0059] In this embodiment of the disclosure, the preferred identification method for each intent type in the intent type set is determined based on the rule-based identification intent and the model-based identification intent corresponding to the first instruction. This method determines whether the rule-based identification method or the large language model identification method is better for identifying the first instruction containing a specific intent type.

[0060] For example, for translation-related intents, keywords and rules are relatively limited, making rule-based identification methods more accurate. In terms of both accuracy and efficiency, rule-based identification is more effective. However, for question-answering and writing-related intents, the relevant keywords are broader, and the patterns of the symbols are more diverse, making large-scale oracle model identification methods more suitable for identifying intent types. Accordingly, the optimal identification method can be determined.

[0061] Figure 2 This is a flowchart illustrating the preferred identification method in some embodiments of this disclosure. For example... Figure 2 As shown, in some embodiments, the computing device may use the following S121-S124 to determine the preferred identification method.

[0062] S121: Based on the rule recognition intent corresponding to the first instruction, determine the number of rules to be recognized for each intent type, and compare the rule recognition intent corresponding to the first instruction with the corresponding manually labeled intent to determine the number of rules correctly recognized for each intent type.

[0063] Determining the number of rules recognized for each intent type based on the rule recognition intent corresponding to the first instruction involves counting the rule recognition intents for each intent type separately to determine the recognition quantity for each intent type. According to the quantitative relationship, the sum of the rule recognition quantities for each intent type is equal to the number of first instructions contained in the first instruction set.

[0064] Based on the aforementioned analysis, since the manually labeled intent is considered the correct intent for the first instruction, comparing the rule-recognized intent corresponding to each first instruction with the corresponding manually labeled intent can determine whether the rule-recognized intent corresponding to the first instruction is correct. After determining whether the rule-recognized intent corresponding to each first instruction is correct, counting the correct first instructions can determine the number of correctly recognized rules for the corresponding intent type.

[0065] S122: Based on the model recognition intent corresponding to the first instruction, determine the number of models recognized for each intent type, and compare the model recognition intent corresponding to the first instruction with the corresponding manually labeled intent to determine the number of models correctly recognized for each intent type.

[0066] Determining the number of model recognitions for each intent type based on the model recognition intent corresponding to the first instruction involves counting the model recognition intents for each intent type separately to determine the recognition count for each intent type. According to the quantitative relationship, the sum of the model recognition counts for each intent type equals the number of first instructions contained in the first instruction set.

[0067] Based on the aforementioned analysis, since the manually labeled intent is considered the correct intent for the first instruction, comparing the model-recognized intent corresponding to each first instruction with the corresponding manually labeled intent can determine whether the model-recognized intent corresponding to the first instruction is correct. After determining whether the model-recognized intent corresponding to each first instruction is correct, counting the correct first instructions can determine the number of correctly recognized intent types by the model.

[0068] S123: Calculate the rule recognition accuracy for each intent type based on the number of rules recognized and the number of rules correctly recognized for each intent type; and calculate the model recognition accuracy for each intent type based on the number of models recognized and the number of models correctly recognized for each intent type.

[0069] In practice, the computing device divides the number of correctly recognized rules for each intent type by the total number of correctly recognized rules; the resulting ratio can be used as the rule recognition accuracy for that intent type. Similarly, the computing device divides the number of correctly recognized models for each intent type by the total number of correctly recognized models; the resulting ratio can be used as the model recognition accuracy for that intent type.

[0070] S124: Compare the rule recognition accuracy and model recognition accuracy for each intent type, determine the larger recognition accuracy, and select the intent recognition method with the larger recognition accuracy as the preferred recognition method for the corresponding intent type.

[0071] As the analysis above shows, if the rule recognition accuracy for a certain intent type is greater than the model recognition accuracy, then for the first instruction in the first instruction set, the rule recognition method is more suitable, and thus the rule recognition method can be considered the preferred recognition method for that intent type. Conversely, the model recognition method can be considered the preferred recognition method for that intent type.

[0072] It should be noted that in some cases, the rule recognition accuracy and model recognition accuracy determined according to the aforementioned scheme are the same (of course, this situation should be based on ensuring that the number of first instructions in the first instruction set is sufficient). In this case, either the rule recognition method or the model recognition method can be determined as the preferred recognition method.

[0073] In this embodiment of the disclosure, determining the preferred identification method is not merely to determine the absolute superiority or inferiority of the two intent identification methods, but rather to determine which intent identification method has higher reliability in the final adoption (of course, correspondingly, under reasonable data conditions, the preferred identification method will have higher accuracy in identifying the intent of the new instruction).

[0074] S130: The second instruction is processed using both rule-based recognition and large language model recognition methods to obtain two candidate recognition intentions corresponding to the second instruction.

[0075] The second instruction is one without a corresponding manually labeled instruction. In practice, both rule-based recognition and large oracle model recognition methods are used to process the second instruction. Specifically, the same rule-based recognition method and large oracle model recognition method are used as in S110 to process the second instruction, resulting in two candidate recognition intentions corresponding to the second instruction. It can be imagined that the two candidate recognition intentions... Figure 1 One is the rule-based intent recognition, and the other is the model-based intent recognition.

[0076] It should be noted here that the two candidate intentions may be the same or different, and cannot be absolutely determined in advance. In addition, the two candidate intentions may be correct (assuming they are the same), or only one of them may be correct, or neither of them may be correct.

[0077] S140: Based on the intent types of two candidate intents and the preferred identification method associated with each intent type, the selected intent is determined from the two candidate intents and the selected intent is used as the instruction intent of the second instruction.

[0078] After obtaining two candidate intents, the intent type obtained under the corresponding intent recognition method can be determined based on the aforementioned candidate intents. If the recognition method for the candidate intent type is determined to be the preferred recognition method corresponding to this intent type, then this candidate intent is selected as the chosen intent; conversely, if the recognition method for the candidate intent type is determined to be not the preferred recognition method corresponding to this intent type, then this candidate intent is not selected as the chosen intent.

[0079] Once the selected intent is determined, it can be used as the instruction intent for the second instruction.

[0080] As analyzed above, in the scheme of this embodiment, after obtaining the rule-recognized intent and model-recognized intent corresponding to multiple first instructions using the rule recognition method and the large language model recognition method, the two recognized intents are judged by using manually labeled intents as a reference. This allows it to determine which recognition method is more suitable for a specific intent recognition type, and thus the corresponding intent recognition method can be selected as the preferred recognition method for this intent type. Subsequently, after processing new second instructions using the rule recognition method and the large language model recognition method to obtain the intent to be selected, the selected intent can be determined based on the previously determined preferred recognition method, and the selected intent can be used as the instruction intent of the second instruction.

[0081] In other words, the present disclosure provides a method for evaluating the adaptability of rule recognition method and large language model recognition method to various intent types. In practical applications, the intent recognition results of the two recognition methods can be selected to determine the instruction intent of the second instruction.

[0082] As mentioned in S110 above, S110 uses a rule recognition method to process the first instruction in the first instruction set to obtain the rule recognition intent. Figure 3 This is a flowchart illustrating a method for determining rule-based intent using some embodiments of this disclosure. Figure 3 As shown, in some embodiments of this disclosure, the method for determining the rule recognition intent includes S111-S112.

[0083] S111: Extract instruction keywords from the first instruction, construct a regular expression based on the instruction keywords, and determine the core semantic vector corresponding to the first instruction based on the regular expression.

[0084] Instruction keywords are keywords used to characterize the intent of a first instruction. In specific implementations, instruction keywords include verbs and nouns. In specific implementations, the computing device can use a pre-trained first language model to process the first instruction and extract the instruction keywords. The aforementioned first language model can be a model specifically designed for extracting instruction keywords, or it can be a general model; this disclosure does not limit the specific model.

[0085] When the primary language model is a general-purpose model, it is necessary to ensure that the primary language model understands and processes the task. In this case, the computing device can use pre-constructed keywords to extract prompts and the first instruction as model input, and then use the primary language model to perform keyword recognition to obtain the instruction keywords.

[0086] Keyword extraction prompts are used to prompt large language models about their tasks and rules, indicating the processing goals and methods of the large language model for the first instruction.

[0087] In some embodiments, the keyword prompts used are as follows: "You are an intent keyword extractor. For instructions of any length, you need to do the following. Step 1: Identify the different components of the instruction, including: main task, constraints, example cases, and text materials. Step 2: Intent keywords only exist in the main task. If the instruction is long, ignore the text materials, example cases, constraints, etc., and only focus on the language of the main task part, extracting the core verbs and objects in the main task."

[0088] In some embodiments, to facilitate the acquisition of instruction keywords, additional prompt keywords targeting various types of intents can be added to the keyword extraction prompts, enabling the model to identify instruction keywords more quickly. For example, for information extraction, the prompt keywords added to the keyword prompts are: "r" (extract | extract | list | extract | list).? (key | core | relevant | content | element | field | option | answer | analysis | key point)", "r" (combine | according to | from | read).*? (below | above) (text | information).*? (give | find | output | extract | answer).*? (answer)"]

[0089] After obtaining the instruction keywords, a regular expression can be constructed based on them. In practice, pre-built regular expression prompts and instruction keywords can be used as model input, and the second language model is used to construct the expression. In real-world applications, the first and second language models mentioned above can be the same or different models.

[0090] Considering that constructing a regular expression essentially involves combining verbs, restrictive modifiers (such as quantity modifiers and quantifier modifiers), nouns, and other instruction keywords to form verb-object phrases (including verb-object phrases with various restrictive modifiers), the aforementioned regular expression construction prompts can be added after the keyword extraction words, and the regular expression can be directly obtained using the first language model.

[0091] After obtaining the regular expression, it can then be vectorized to obtain the core semantic vector corresponding to the first instruction.

[0092] S112: Based on the core semantic vector corresponding to the first instruction and the semantic vector associated with each intent type, determine the rule recognition intent corresponding to the first instruction.

[0093] The semantic vectors associated with each intent type can be semantic vectors obtained by semantically vectorizing the regular expressions of various predetermined intent types. For example, for a specific intent type, if the expression constructed using keywords is to extract the answer, the regular expression corresponding to this intent type can be a regular expression obtained by processing the aforementioned intent type.

[0094] After obtaining the core semantic vector corresponding to the first instruction, the core semantic vector can be compared with the semantic vectors associated with each intent type to determine the closest semantic vector, and the recognition intent corresponding to the closest semantic vector is used as the rule recognition intent.

[0095] In practical applications, keyword extraction is a very labor-intensive task, resulting in a limited number of pre-built regular expressions for each intent type, and consequently, a limited number of semantic vectors for each intent type. To address this issue, in some embodiments, after determining the rule-based intent corresponding to the first instruction, the computing device may also execute the following step S150.

[0096] S150: Associate the core semantic vector of the first instruction with the corresponding intent type, and use it as the semantic vector associated with the corresponding intent type.

[0097] Specifically, after obtaining the core semantic vector of the first instruction and the intent type of its rule-based intent recognition, the core semantic vector can be directly assigned to the corresponding intent type, increasing the number of core semantic vectors associated with the corresponding intent type.

[0098] In other embodiments, to address the issue of fewer semantic vectors corresponding to each intent type, the computing device may execute the following S160 after determining the core semantic vector corresponding to the first instruction.

[0099] S160: Perform expression matching based on the regular expression corresponding to the first instruction and the regular expression associated with the corresponding intent type to determine a new regular expression associated with the intent type, and construct a new semantic vector associated with the corresponding intent type based on the new regular expression.

[0100] The expression matching is performed based on the regular expression corresponding to the first instruction and the regular expression associated with the corresponding intent type. The regular expression that appears most frequently is matched, and then it is used as a new regular expression associated with the intent type.

[0101] After determining the regular expression associated with the intent type, the aforementioned new regular expression can then be vectorized to obtain a semantic vector, which can then be used as the semantic vector associated with the corresponding intent type.

[0102] As analyzed above, adding corresponding instruction keywords or regular expressions to keyword suggestions or regular expression construction suggestions can help the model better determine instruction keywords or write regular expressions. To solve this problem, in some embodiments of this disclosure, after obtaining the regular expression, the regular expression can be added to the regular expression construction suggestions, or the instruction keywords in the regular expression can be added to the keyword suggestions.

[0103] In some other embodiments, to address the issue of fewer semantic vectors corresponding to each intent type, the computing device may execute the following S170 after determining the core semantic vector corresponding to the first instruction.

[0104] S170: Based on the core semantic vector of the first instruction, the semantic vectors associated with each intent type and the number of intent types are clustered to determine a new cluster center vector, and the new cluster center vector is used as the semantic vector associated with the corresponding intent type.

[0105] After obtaining the core semantic vectors of each first instruction, cluster analysis can be used to redetermine the number of clusters for each intent type, and the cluster center vectors can be determined based on the semantic vectors contained in each cluster. After obtaining the cluster center vectors, the semantic vectors associated with each intent type can be compared with the cluster center vectors to determine the intent type corresponding to each cluster center vector, and this is used as the new associated semantic vector for the corresponding intent type.

[0106] By adopting the aforementioned S150, S160, or S170, the characteristics of large language models in summarizing intent rules can be utilized to overcome the problem of time-consuming and labor-intensive rule summarization. At the same time, it can also solve the problem of rule loopholes caused by small-scale manual rule summarization.

[0107] As mentioned earlier, a large language model recognition method is needed to process the first instruction to obtain the model recognition intent. In specific implementation, the computing device can obtain the model recognition intent using the following S113.

[0108] S113: Using pre-built intent prompts and the first instruction as model input, the third language model is used for intent recognition to obtain the model-recognized intent of the first instruction.

[0109] In some embodiments of this disclosure, the pre-built intent prompt words include all intent types from the intent type set.

[0110] In some embodiments, the intent prompt may also include descriptions of each intent type. For example, the intent prompt could be: "You are now a user task intent classifier for a prompt. The knowledge base stores a list of tags to be classified and their corresponding explanations, as follows: class_definition_str = {'Language understanding': 'Language understanding and related tasks, including words, sentences, and paragraphs, are prioritized in this category',}, system_prompt = f "You are now a user task intent classifier for a prompt. The knowledge base stores a list of tags to be classified and their corresponding explanations, as follows:\n{class_definition_str}"—this is where the three-level intent system is placed; user_prompt = f "Your output can only be selected from the tag list. You cannot output tags outside the list. If there is no suitable tag in the list, please output something else. Do not output any extra content." Correspondingly, the first instruction could be: "The content of the prompt entered by the user is: "{question}".

[0111] The third major language model in this embodiment can be the same as the first or second major language model mentioned above, or it can be a different major language model.

[0112] In some embodiments, the second instruction is much more numerous than the instructions in the first instruction set. After determining the intent type of the second instruction using the aforementioned method, the computing device can train the large language model based on the second instruction and the corresponding intent type, enabling the large language model to perform intent recognition more accurately.

[0113] In addition to providing the aforementioned method for recognizing instruction intent, this disclosure also provides an apparatus for recognizing instruction intent. Figure 4 This is a schematic diagram of the instruction intent recognition device provided in an embodiment of this disclosure. Figure 4 As shown, the instruction intent recognition device 400 provided in this embodiment includes an intent prediction unit 401, a method selection unit 402, and a result determination unit 403.

[0114] The intent prediction unit 401 is used to process the first instruction in the first instruction set using a rule recognition method to obtain a rule-recognized intent, and to process the first instruction using a large language model recognition method to obtain a model-recognized intent; the first instruction has a corresponding manually labeled intent; the intent types of the rule-recognized intent, the model-recognized intent, and the manually labeled intent are all intents in a pre-constructed intent type set.

[0115] The method selection unit 402 is used to determine the preferred identification method for each intention type association in the intention type set based on rule-based intention identification and model-based intention identification, with reference to manually annotated intentions. The preferred identification method is either a rule-based identification method or a large language model identification method.

[0116] The intent prediction unit 401 is also used to process the second instruction using a rule recognition method and a large language model recognition method respectively, to obtain two candidate recognition intents corresponding to the second instruction.

[0117] The result determination unit 403 is used to determine the selected intent among the two candidate intents based on the intent types of the two candidate intents and the preferred identification method associated with each intent type, and to use the selected intent as the instruction intent of the second instruction.

[0118] In some embodiments, the intent prediction unit 401 uses a rule recognition method to process the first instruction in the first instruction set to obtain the rule-recognized intent, including: extracting instruction keywords from the first instruction, constructing a regular expression based on the instruction keywords, and determining the core semantic vector corresponding to the first instruction based on the regular expression; and determining the rule-recognized intent corresponding to the first instruction based on the core semantic vector corresponding to the first instruction and the semantic vector associated with each intent type.

[0119] In some embodiments, the intent prediction unit 401 uses pre-constructed keywords to extract prompt words and a first instruction as model input, and uses a first large language model to perform keyword recognition to obtain instruction keywords.

[0120] In some embodiments, the intent prediction unit 401 uses pre-built regular expression prompts and instruction keywords as model inputs, and uses a second language model to construct expressions to obtain regular expressions.

[0121] In some embodiments, after determining the rule recognition intent corresponding to the first instruction, the intent determination unit associates the core semantic vector of the first instruction with the corresponding intent type as a semantic vector associated with the corresponding intent type.

[0122] In some embodiments, after determining the rule recognition intent corresponding to the first instruction, the intent determination unit is further configured to perform expression matching based on the regular expression corresponding to the first instruction and the regular expression associated with the corresponding intent type, determine a new regular expression associated with the intent type, and construct a new semantic vector associated with the corresponding intent type based on the new regular expression; or, perform cluster analysis based on the core semantic vector of the first instruction, the semantic vectors associated with each intent type, and the number of intent types, determine a new cluster center vector, and use the new cluster center vector as the semantic vector associated with the corresponding intent type.

[0123] In some embodiments, the intent determination unit uses pre-built intent prompts and a first instruction as model input, employs a third language model for intent recognition, and obtains the model-recognized intent of the first instruction. The pre-built intent prompts include all intent types in the intent type set. In practical applications, the intent prompts can be: `class_definition_str = {'Language Understanding': 'Including three levels of language understanding (words, sentences, and paragraphs), and processing-related tasks are preferentially classified into this category',}`; `system_prompt = f"Now you are a user task intent classifier for prompts. The knowledge base stores a list of tags to be classified and their corresponding explanations, as follows: \n{class_definition_str}"—this is where the three-level intent system is placed; `user_prompt = f"Your output can only be selected from the tag list. You cannot output tags outside the list. If there is no suitable tag in the list, please output something else. Do not output any extra content. The content of the prompt input by the user is: "{question}". The prompt intent-related keywords are: {keywords}. The tag that best matches its task intent is: ".

[0124] In some embodiments, the method selection unit 402 determines the number of rules recognized for each intent type based on the rule recognition intent corresponding to the first instruction, and compares the rule recognition intent corresponding to the first instruction with the corresponding manually labeled intent to determine the number of correctly recognized rules for each intent type; determines the number of models recognized for each intent type based on the model recognition intent corresponding to the first instruction, and compares the model recognition intent corresponding to the first instruction with the corresponding manually labeled intent to determine the number of correctly recognized models for each intent type; calculates the rule recognition accuracy for each intent type based on the number of rules recognized and the number of correctly recognized rules for each intent type; and calculates the model recognition accuracy for each intent type based on the number of models recognized and the number of correctly recognized models for each intent type; compares the rule recognition accuracy and model recognition accuracy for each intent type respectively, determines the larger recognition accuracy, and selects the intent recognition method corresponding to the larger recognition accuracy as the preferred recognition method for the corresponding intent type.

[0125] This disclosure also provides a computing device for implementing the aforementioned method. Figure 5 This is a schematic diagram of the structure of a computing device provided in an embodiment of this disclosure. See below for details. Figure 5 It shows a schematic diagram of a structure suitable for implementing the computing device 500 in the embodiments of this disclosure. Figure 5 The computing device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0126] like Figure 5 As shown, the computing device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory ROM 502 or a program loaded from a storage device 508 into a random access memory RAM 503. The RAM 503 also stores various programs and data required for the operation of the computing device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0127] Typically, the following devices can be connected to I / O interface 505: input devices 505 including, for example, touchscreens, touchpads, cameras, microphones, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows computing device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5A computing device 500 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0128] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0129] It should be noted that the computer-readable medium described above in this disclosure may be a computer-readable storage medium, a computer-readable signal medium, or any combination thereof.

[0130] Computer-readable storage media can be, for example—but not limited to—electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0131] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0132] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0133] The aforementioned computer-readable medium may be included in the aforementioned computing device; or it may exist independently and not assembled into the computing device.

[0134] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the tester's computer, partially on the tester's computer, as a standalone software package, partially on the tester's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the tester's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0135] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0136] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not necessarily limiting in certain circumstances. The functions described above can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), etc.

[0137] The above are merely specific embodiments of this disclosure, enabling those skilled in the art to understand or implement this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for recognizing instruction intent, characterized in that, include: The method involves processing a first instruction in a first instruction set using a rule recognition method to obtain a rule-recognized intent, and processing the first instruction using a large language model recognition method to obtain a model-recognized intent. The first instruction has a corresponding manually labeled intent. The intent types of the rule-recognized intent, the model-recognized intent, and the manually labeled intent are all intents from a pre-constructed intent type set. Specifically, processing the first instruction in the first instruction set using the rule recognition method to obtain a rule-recognized intent includes: extracting instruction keywords from the first instruction, constructing a regular expression based on the instruction keywords, and determining the core semantic vector corresponding to the first instruction based on the regular expression; determining the rule-recognized intent corresponding to the first instruction based on the core semantic vector corresponding to the first instruction and the semantic vectors associated with each intent type. Processing the first instruction using the large language model recognition method to obtain a model-recognized intent includes: using pre-constructed intent prompts and the first instruction as model input, performing intent recognition using a third large language model to obtain the model-recognized intent of the first instruction. The pre-constructed intent prompts include all intent types from the intent type set. Taking the manually labeled intent corresponding to the first instruction as a reference, a preferred identification method is determined based on the rule-based identification intent and the model-based identification intent corresponding to the first instruction, wherein the preferred identification method is the rule-based identification method or the large language model identification method. The second instruction is processed using the rule recognition method and the large language model recognition method respectively to obtain two candidate recognition intentions corresponding to the second instruction; Based on the intent types of the two candidate intents and the preferred identification method associated with each intent type, a selected intent is determined from the two candidate intents, and the selected intent is used as the instruction intent of the second instruction.

2. The method according to claim 1, characterized in that, The extraction of instruction keywords from the first instruction includes: Using pre-constructed keyword extraction prompts and the first instruction as model inputs, the first large language model is used for keyword recognition to obtain the instruction keywords.

3. The method according to claim 1, characterized in that, The construction of regular expressions based on the instruction keywords includes: Using pre-built regular expression prompts and the instruction keywords as model input, the second largest language model is used to construct the expression, resulting in the regular expression.

4. The method according to claim 1, characterized in that, After determining the rule recognition intent corresponding to the first instruction, the method further includes: The core semantic vector of the first instruction is associated with the corresponding intent type, and this is used as the semantic vector associated with the corresponding intent type.

5. The method according to claim 1, characterized in that, After determining the rule recognition intent corresponding to the first instruction, the method further includes: Based on the regular expression corresponding to the first instruction and the regular expression associated with the corresponding intent type, expression matching is performed to determine a new regular expression associated with the intent type, and a new semantic vector associated with the corresponding intent type is constructed based on the new regular expression; or, Based on the core semantic vector of the first instruction, cluster analysis is performed on the semantic vectors associated with each intent type and the number of intent types to determine a new cluster center vector, which is then used as the semantic vector associated with the corresponding intent type.

6. The method according to any one of claims 1-5, characterized in that, The preferred identification method for determining the association of each intent type in the intent type set based on the manually labeled intent and the rule-based intent identification and the model-based intent identification, using the manually labeled intent as a reference, includes: Based on the rule recognition intent corresponding to the first instruction, determine the number of rules recognized for each intent type, and compare the rule recognition intent corresponding to the first instruction with the corresponding manually labeled intent to determine the number of rules correctly recognized for each intent type. Based on the model recognition intent corresponding to the first instruction, determine the number of models recognized for each intent type, and compare the model recognition intent corresponding to the first instruction with the corresponding manually labeled intent to determine the number of models correctly recognized for each intent type. Based on the number of rules recognized and the number of rules correctly recognized for each intent type, calculate the rule recognition accuracy for each intent type; and based on the number of models recognized and the number of models correctly recognized for each intent type, calculate the model recognition accuracy for each intent type. The rule recognition accuracy and model recognition accuracy for each intent type are compared to determine the one with the higher recognition accuracy. The intent recognition method with the higher recognition accuracy is then selected as the preferred recognition method for that intent type.

7. A command intent recognition device, characterized in that, include: An intent prediction unit is used to process a first instruction in a first instruction set using a rule recognition method to obtain a rule-recognized intent, and to process the first instruction using a large language model recognition method to obtain a model-recognized intent; the first instruction has a corresponding manually labeled intent; the intent types of the rule-recognized intent, the model-recognized intent, and the manually labeled intent are all intents in a pre-constructed intent type set; wherein, processing the first instruction in the first instruction set using a rule recognition method to obtain a rule-recognized intent includes: extracting instruction keywords from the first instruction, constructing a regular expression based on the instruction keywords, and determining the core semantic vector corresponding to the first instruction based on the regular expression; determining the rule-recognized intent corresponding to the first instruction based on the core semantic vector corresponding to the first instruction and the semantic vectors associated with each intent type; processing the first instruction using a large language model recognition method to obtain a model-recognized intent includes: using pre-constructed intent prompt words and the first instruction as model input, performing intent recognition using a third large language model to obtain the model-recognized intent of the first instruction, wherein the pre-constructed intent prompt words include all intent types in the intent type set; The method selection unit is used to determine the preferred identification method for each intent type in the intent type set based on the rule-based identification intent and the model-based identification intent, with reference to the manually annotated intent. The preferred identification method is either the rule-based identification method or the large language model identification method. The intent prediction unit is also used to process the second instruction using the rule recognition method and the large language model recognition method respectively, to obtain two candidate recognition intents corresponding to the second instruction; The result determination unit is used to determine the selected intent from the two candidate identification intents based on the intent types of the two candidate identification intents and the preferred identification method associated with each intent type, and to use the selected intent as the instruction intent of the second instruction.

8. A computing device, characterized in that, It includes a processor and a memory, the memory being used to store a computer program; when the computer program is loaded by the processor, it causes the processor to execute the instruction intent recognition method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, causes the processor to implement the instruction intent recognition method as described in any one of claims 1-6.

Images