Instruction intention recognition method and device, computing equipment and storage medium

Abstract

The embodiment of the invention provides an instruction intention recognition method and device, computing equipment and a storage medium. The instruction intention recognition method comprises the steps of processing a first instruction by adopting a rule recognition method to obtain a rule recognition intention, and processing the first instruction by adopting a large language model recognition method to obtain a model recognition intention; determining a preferable identification method associated with each intention type by taking a manual annotation intention corresponding to the first instruction as a reference; processing the second instruction by adopting a rule identification method and a large language model identification method to obtain two corresponding to-be-selected identification intentions; and based on the intention types of the two to-be-selected identification intentions and the preferable identification method associated with each intention type, determining a selected intention in the two to-be-selected identification intentions, and taking the selected intention as an instruction intention of the second instruction. According to the scheme of the invention, the method for judging the adaptability of various intention types is provided, and the intention recognition results of the two recognition methods can be selected in a targeted manner in practical application to determine the instruction intention of the second instruction.

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning technology, and in particular to a method and apparatus for identifying instruction intent, a computing device, and a storage medium. Background Art

[0002] Intent recognition is a key technology used in the fields of human-computer interaction and natural language understanding. At present, existing intent recognition methods include rule extraction methods based on keyword and semantic similarity calculations and methods based on direct recognition of large prediction models. Although the aforementioned rule-based extraction methods are accurate and effective in specific scenarios, because manual methods cannot exhaust all possible rules, the rule-based methods are insufficient in generalization ability and are difficult to cope with large-scale, multi-intent scenarios. Although the model-based recognition method will greatly improve the flexibility and breadth of model classification and recognition, it will also lose the accuracy of manually constructed rules. The classification method of model recognition has problems such as the inability to stably and accurately recognize simple tasks, unstable performance, and poor recognition of instructions in long texts. Due to the aforementioned reasons, the relevant technology cannot compare the advantages and disadvantages of rule-based recognition methods and model-based recognition methods horizontally, and it is impossible to obtain more accurate intent recognition results for instructions with uncertain intent in automatic recognition. Summary of the invention

[0003] In order to solve the problem that it is difficult to obtain relatively accurate intent recognition results for instructions with uncertain intent in the existing automatic recognition, the embodiments of the present disclosure provide a method and apparatus for instruction intent recognition, a computing device, and a storage medium.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for identifying instruction intent, including:

[0005] A rule recognition method is used to process a first instruction in a first instruction set to obtain a rule recognition intent, and a large language model recognition method is used to process the first instruction to obtain a model recognition intent; the first instruction has a corresponding manually annotated intent; the intent types of the rule recognition intent, the model recognition intent and the manually annotated intent are all intents in a pre-constructed intent type set;

[0006] Taking the manually annotated intent corresponding to the first instruction as a reference, determining a preferred recognition method associated with each intent type in the intent type set based on the rule recognition intent and the model recognition intent corresponding to the first instruction, wherein the preferred recognition method is the rule recognition method or the large language model recognition method;

[0007] The second instruction is processed by the rule recognition method and the large language model recognition method respectively, so as to obtain two to-be-selected recognition intentions corresponding to the second instruction;

[0008] Based on the intention types of the two to-be-selected recognition intentions and the preferred recognition methods associated with the respective intention types, a selected intention is determined among the two to-be-selected recognition intentions, and the selected intention is used as the instruction intention of the second instruction.

[0009] Optionally, the adopting a rule recognition method to process the first instruction in the first instruction set to obtain a rule recognition intent includes:

[0010] Extracting instruction keywords from the first instruction, constructing a regular expression based on the instruction keywords, and determining a core semantic vector corresponding to the first instruction based on the regular expression;

[0011] Based on the core semantic vector corresponding to the first instruction and the semantic vectors associated with each intent type, the rule recognition intent corresponding to the first instruction is determined.

[0012] Optionally, extracting the instruction keyword in the first instruction includes:

[0013] The pre-constructed keyword extraction prompt word and the first instruction are used as model inputs, and the first language model is used to perform keyword recognition to obtain the instruction keyword.

[0014] Optionally, constructing a regular expression based on the instruction keyword includes:

[0015] The pre-constructed regular expression construction prompt word and the instruction keyword are used as model input, and the second largest language model is used to construct the expression to obtain the regular expression.

[0016] Optionally, after determining the rule recognition intention corresponding to the first instruction, the method further includes:

[0017] The core semantic vector of the first instruction is associated with the corresponding intent type as a semantic vector associated with the corresponding intent type.

[0018] Optionally, after determining the rule recognition intention corresponding to the first instruction, the method further includes:

[0019] Perform expression matching according to the regular expression corresponding to the first instruction and the regular expression associated with the corresponding intent type, determine a new regular expression associated with the intent type, and construct a new semantic vector associated with the corresponding intent type based on the new regular expression; or,

[0020] According to the core semantic vector of the first instruction, the semantic vectors associated with each of the intent types and the number of intent types are clustered and analyzed to determine a new cluster center vector, which is used as the semantic vector associated with the corresponding intent type.

[0021] Optionally, the adopting a large language model recognition method to process the first instruction to obtain a model recognition intent includes:

[0022] Taking the pre-constructed intent prompt words and the first instruction as model input, a third language model is used to perform intent recognition to obtain the model recognition intent of the first instruction, wherein the pre-constructed intent prompt words include all intent types in the intent type set.

[0023] Optionally, the preferred identification method for determining the association between each intent type in the intent type set based on the rule identification intent and the model identification intent with reference to the manually annotated intent includes:

[0024] Determine the number of rule recognitions corresponding to each intent type based on the rule recognition intent corresponding to the first instruction, and compare the rule recognition intent corresponding to the first instruction with the corresponding manually labeled intent to determine the number of correct rule recognitions corresponding to each intent type;

[0025] Determine the number of model recognitions corresponding to each intent type based on the model recognition intent corresponding to the first instruction, and compare the model recognition intent corresponding to the first instruction with the corresponding manually labeled intent to determine the number of correct model recognitions corresponding to each intent type;

[0026] Based on the number of rule recognitions and the number of correct rule recognitions corresponding to each intent type, the rule recognition accuracy corresponding to each intent type is calculated; and based on the number of model recognitions and the number of correct model recognitions corresponding to each intent type, the model recognition accuracy corresponding to each intent type is calculated;

[0027] The rule recognition accuracy and model recognition accuracy corresponding to each intent type are compared respectively, the larger recognition accuracy is determined, and the intent recognition method corresponding to the larger recognition accuracy is used as the preferred recognition method for the corresponding intent type.

[0028] In a second aspect, an embodiment of the present disclosure provides a device for identifying instruction intention, including:

[0029] An intention prediction unit, configured to process a first instruction in a first instruction set by a rule recognition method to obtain a rule recognition intention, and to process the first instruction by a large language model recognition method to obtain a model recognition intention; the first instruction has a corresponding manually annotated intention; the intention types of the rule recognition intention, the model recognition intention and the manually annotated intention are all intentions in a pre-constructed intention type set;

[0030] A method selection unit, used to determine a preferred recognition method associated with each intent type in an intent type set based on the rule recognition intent and the model recognition intent, with reference to the manually annotated intent, wherein the preferred recognition method is the rule recognition method or the large language model recognition method;

[0031] The intention prediction unit is further used to process the second instruction by using the rule recognition method and the large language model recognition method respectively, to obtain two to-be-selected recognition intentions corresponding to the second instruction;

[0032] A result determination unit is used to determine a selected intention among the two to-be-selected recognition intentions based on the intention types of the two to-be-selected recognition intentions and the preferred recognition methods associated with each intention type, and to use the selected intention as the instruction intention of the second instruction.

[0033] In a third aspect, an embodiment of the present disclosure provides a computing device, characterized in that it includes a processor and a memory, wherein the memory is used to store a computer program; when the computer program is loaded by the processor, the processor executes the instruction intent recognition method as described above.

[0034] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the storage medium stores a computer program. When the computer program is executed by a processor, the processor implements the instruction intent recognition method as described above.

[0035] By adopting the embodiment of the present disclosure, after obtaining the rule recognition intent and model recognition intent corresponding to multiple first instructions by using the rule recognition method and the large language model recognition method, the two recognition intents are judged by using the manually labeled intent as a reference, and it can be determined which recognition method is more suitable for a specific intent recognition type, and then the corresponding intent recognition method can be selected as the preferred recognition method corresponding to this intent type. Subsequently, after the new second instruction is processed by the rule recognition method and the large language model recognition method to obtain the intent to be selected, it can be determined which intent is used as the determined selected intent according to the previously determined preferred recognition method, and the determined selected intent is used as the instruction intent of the second instruction.

[0036] That is to say, the embodiment scheme of the present disclosure provides a method for judging the adaptability of the rule recognition method and the large language model recognition method to various intent types. In practical applications, the intent recognition results of the two recognition methods can be selected to determine the instruction intent of the second instruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor, including

[0039] Figure 1 is a flow chart of a method for identifying instruction intent provided by an embodiment of the present disclosure;

[0040] Figure 2 is a flow chart of determining a preferred identification method in some embodiments of the present disclosure;

[0041] Figure 3 is a flow chart of a method for determining rule recognition intent in some embodiments of the present disclosure;

[0042] Figure 4 is a schematic diagram of the structure of a command intention recognition device provided by an embodiment of the present disclosure;

[0043] Figure 5 It is a schematic diagram of the structure of a computing device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0045] The term "including" and its variations used in this document are open-ended, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description. In this document, relational terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or operations are the same.

[0046] In order to solve the problem that different types of intent recognition methods cannot be compared horizontally, and thus it is impossible to determine which method is more accurate in intent recognition, and it is impossible to determine the intent recognition result more accurately, the embodiment of the present disclosure provides a new instruction intent recognition method. The instruction intent recognition method provided by the embodiment of the present disclosure is executed by a computing device.

[0047] Figure 1 is a flow chart of the instruction intention recognition method provided by the embodiment of the present disclosure. Figure 1 As shown, the instruction intent recognition method provided by the embodiment of the present disclosure includes S110-S140.

[0048] S110: Using a rule recognition method to process the first instruction in the first instruction set to obtain a rule recognition intent, and using a large language model recognition method to process the first instruction to obtain a model recognition intent.

[0049] The first instruction set is a data set containing a large number of first instructions. The aforementioned first instructions are instructions containing specific intentions.

[0050] The correct intents corresponding to all the first instructions in the first instruction set are included in the intent type set, and the real intents corresponding to all the first instructions in the first instruction set constitute the full set of the intent type set. The intent type set is a data set constructed using various possible intents.

[0051] In some embodiments of the present disclosure, the annotator can construct a multi-level intent system by analyzing user instructions, and add all intent types in the aforementioned multi-level intent system to the intent type set. For example, in some embodiments, the intent system includes primary intent, secondary intent, and tertiary intent, wherein the secondary intent is a sub-intent of the primary intent, and the tertiary intent is a sub-intent of the secondary intent.

[0052] In some embodiments, while constructing a multi-level intention system, the annotator can also explain the intentions in various three-level intention systems. For example, when the third-level intention included in the intention system is word formation and sentence formation, the corresponding explanation can be "select Chinese characters within a given range for word formation tasks, and give certain words to form words or sentences", and when the third-level intention included in the intention system is sentence sorting, the corresponding explanation can be "including tasks related to sentence word order adjustment such as chaotic sentence recovery and reverse output, and scattered and chaotic sentence reorganization is also considered sentence sorting".

[0053] On the premise that the aforementioned explanation is included in the intent system, the aforementioned explanation can be added to the intent type set as part of the multi-level intent system.

[0054] In addition, in the disclosed embodiment, all first instructions in the first instruction set have corresponding manually annotated intentions, and the manually annotated intentions are manually annotated intentions. Based on the above analysis, it can be determined that the real intentions corresponding to the first instructions in the first instruction set constitute the full set of intention type sets.

[0055] In a specific implementation, the computing device uses a rule recognition method to process the first instruction in the first instruction set to obtain a rule recognition intent, which may be to use an existing recognition method to process the first instruction to obtain a rule recognition intent corresponding to each first instruction. The computing device uses a large language model recognition method to process the first instruction in the first instruction set to obtain a model recognition intent, or it may be to input the first instruction into a large language model as in the prior art solution, and use the large language model to process the first instruction to obtain a model recognition intent.

[0056] It should be noted here that the rule recognition intent and model recognition intent obtained by processing the first instruction according to the above method may be correct intent or wrong intent. Moreover, the rule recognition intent and model recognition intent corresponding to the same first instruction may be the same intent or different intent.

[0057] S120: Taking the manually marked intention corresponding to the first instruction as a reference, determine the preferred recognition method associated with each intention type in the intention type set based on the rule recognition intention and model recognition intention corresponding to the first instruction.

[0058] As analyzed above, the manually labeled intent is a predetermined correct intent corresponding to each first instruction. Accordingly, the manually labeled intent corresponding to each first instruction can be used as a reference intent to determine whether the corresponding rule recognition intent and model recognition intent are correct or not.

[0059] In the embodiment of the present disclosure, the preferred recognition method for each intent type in the intent type set is determined based on the rule recognition intention and model recognition intention corresponding to the first instruction, whether it is better to use the rule recognition method or the large language model recognition method to determine the first instruction containing the specific intent type.

[0060] For example, for translation intent, keywords and rules are relatively limited, and the accuracy of rule-based recognition methods is relatively high. In terms of accuracy and efficiency, rule-based recognition methods are more effective. For question-answering and writing intent, the relevant keywords are relatively broad, and the patterns of instructions are more diverse, which makes it more suitable to use the large prediction model recognition method to identify the intent type. Accordingly, the corresponding preferred recognition method can be determined.

[0061] Figure 2 is a flow chart of determining a preferred identification method in some embodiments of the present disclosure. Figure 2 As shown, in some embodiments, the computing device may determine the preferred identification method using the following S121 - S124 .

[0062] S121: Determine the number of rule recognitions corresponding to each intent type based on the rule recognition intent corresponding to the first instruction, and compare the rule recognition intent corresponding to the first instruction with the corresponding manually labeled intent to determine the number of correct rule recognitions corresponding to each intent type.

[0063] Based on the rule recognition intention corresponding to the first instruction, the number of rule recognitions corresponding to each intention type is determined, and the rule recognition intentions corresponding to various intention types are counted respectively to determine the number of recognitions corresponding to each intention type. According to the quantitative relationship, the sum of the number of rule recognitions corresponding to each intention type is the number of first instructions included in the first instruction set.

[0064] According to the above analysis, because the manually annotated intention is identified as the correct intention of the first instruction, by comparing the rule recognition intention corresponding to each first instruction with the corresponding manually annotated intention, it is possible to determine whether the rule recognition intention corresponding to the first instruction is correct. After determining whether the rule recognition intention corresponding to each first instruction is correct, the correct first instructions are counted to determine the number of correct rule recognitions corresponding to the corresponding intention type.

[0065] S122: Determine the number of model recognitions corresponding to each intent type based on the model recognition intent corresponding to the first instruction, and compare the model recognition intent corresponding to the first instruction with the corresponding manually labeled intent to determine the number of correct model recognitions corresponding to each intent type.

[0066] The number of model recognitions corresponding to each intention type is determined based on the model recognition intent corresponding to the first instruction, and the model recognition intents corresponding to various intention types are counted respectively to determine the number of recognitions corresponding to each intention type. According to the quantitative relationship, the sum of the number of model recognitions corresponding to each intention type is the number of first instructions included in the first instruction set.

[0067] According to the above analysis, because the manually annotated intent is identified as the correct intent of the first instruction, by comparing the model recognition intent corresponding to each first instruction with the corresponding manually annotated intent, it is possible to determine whether the model recognition intent corresponding to the first instruction is correct. After determining whether the model recognition intent corresponding to each first instruction is correct, the correct first instructions are counted to determine the number of correct model recognitions corresponding to the corresponding intent type.

[0068] S123: Based on the number of rule recognitions and the number of correct rule recognitions corresponding to each intent type, calculate the rule recognition accuracy corresponding to each intent type; and, based on the number of model recognitions and the number of correct model recognitions corresponding to each intent type, calculate the model recognition accuracy corresponding to each intent type.

[0069] In a specific implementation, the computing device divides the number of correctly identified rules corresponding to each intent type by the number of corresponding rule identifications, and the obtained ratio can be used as the rule identification accuracy rate of the corresponding intent type. Similarly, the computing device divides the number of correctly identified models corresponding to each intent type by the number of model identifications, and the obtained ratio can be used as the model identification accuracy rate of the corresponding intent type.

[0070] S124: Compare the rule recognition accuracy and model recognition accuracy corresponding to each intent type respectively, determine the larger recognition accuracy, and use the intent recognition method corresponding to the larger recognition accuracy as the preferred recognition method for the corresponding intent type.

[0071] According to the above analysis, if the rule recognition accuracy corresponding to a certain intent type is greater than the model recognition accuracy, it can be determined that the rule recognition method is more appropriate for the first instruction in the first instruction set, and the rule recognition method can be used as the preferred recognition method for the corresponding intent type. Correspondingly, the model recognition method can be used as the preferred recognition method for the corresponding intent type.

[0072] It should be noted that in some cases the rule recognition accuracy and model recognition accuracy determined according to the above scheme are the same (of course, the premise for this situation is to ensure that the number of first instructions in the first instruction set is sufficient). At this time, it can be determined that either the rule recognition method or the model recognition method will be used as the preferred recognition method.

[0073] In the disclosed embodiment, determining the preferred recognition method is not simply to determine the absolute superiority or inferiority of the two intent recognition methods, but to determine which intent recognition method is ultimately adopted with higher credibility (of course, correspondingly, under reasonable data conditions, the corresponding preferred recognition method will have a higher accuracy in recognizing the intent of new instructions).

[0074] S130: The second instruction is processed by using a rule recognition method and a large language model recognition method respectively to obtain two to-be-selected recognition intentions corresponding to the second instruction.

[0075] The second instruction is an instruction without a corresponding manually labeled label. In the specific implementation, the rule recognition method and the large language model recognition processing method are used to process the second instruction respectively, that is, the rule recognition method and the large prediction model recognition method used in S110 are used to process the second instruction to obtain two candidate recognition intentions corresponding to the second instruction. It can be imagined that the two candidate recognition intentions Figure 1 One is rule recognition intent, and the other is model recognition intent.

[0076] It should be noted here that the two to-be-selected recognition intentions may be the same intention or different intentions, and cannot be absolutely determined in advance. In addition, the two to-be-selected recognition intentions may be correct intentions (under the premise that they are the same), or only one may be correct, or both may be incorrect.

[0077] S140: Based on the intention types of the two to-be-selected recognition intentions and the preferred recognition methods associated with the respective intention types, a selected intention is determined from the two to-be-selected recognition intentions, and the selected intention is used as the instruction intention of the second instruction.

[0078] After obtaining two potential recognition intentions, the intent type obtained under the corresponding intent recognition method can be determined based on the foregoing potential recognition intentions. If the recognition method of the potential recognition intention type is determined to be the preferred recognition method corresponding to this intent type, then this potential recognition intention is taken as the selected intent; on the contrary, if the recognition method of the potential recognition intention type is determined not to be the preferred recognition method corresponding to this intent type, then this potential recognition intention will not be taken as the selected intent.

[0079] After the selected intention is determined, the aforementioned selected intention can be used as the instruction intention of the second instruction.

[0080] As analyzed above, by using the embodiment of the present disclosure, after obtaining the rule recognition intent and model recognition intent corresponding to multiple first instructions by using the rule recognition method and the large language model recognition method, the two recognition intents are judged by using the manually labeled intent as a reference, and it can be determined which recognition method is more suitable for a specific intent recognition type, and then the corresponding intent recognition method can be selected as the preferred recognition method corresponding to this intent type. Subsequently, after the new second instruction is processed by the rule recognition method and the large language model recognition method to obtain the intent to be selected, it can be determined which intent is used as the determined selected intent according to the previously determined preferred recognition method, and the determined selected intent is used as the instruction intent of the second instruction.

[0081] That is to say, the embodiment of the present disclosure provides a method for judging the adaptability of the rule recognition method and the large language model recognition method to various intent types. In practical applications, the intent recognition results of the two recognition methods can be specifically selected to determine the instruction intent of the second instruction.

[0082] The foregoing S110 mentioned that S110 uses a rule recognition method to process the first instruction in the first instruction set to obtain a rule recognition intent. Figure 3 is a flow chart of a method for determining rule recognition intent in some embodiments of the present disclosure. Figure 3 As shown, in some embodiments of the present disclosure, the method for determining rule recognition intent includes S111-S112.

[0083] S111: extracting instruction keywords from the first instruction, constructing a regular expression based on the instruction keywords, and determining a core semantic vector corresponding to the first instruction based on the regular expression.

[0084] The instruction keyword is a keyword used to characterize the intent of the first instruction. In a specific implementation, the instruction keyword includes a verb and a noun. In a specific implementation, the computing device may use a pre-trained first language model to process the first instruction and extract the instruction keyword. The aforementioned first language model may be a model specifically used to extract instruction keywords, or it may be a general model, which is not limited in the embodiments of the present disclosure.

[0085] In the case where the first language model is a general model, it is necessary to ensure that the large language model understands the processing task. In this case, the computing device can use the pre-built keyword extraction prompt word and the first instruction as model input, use the first language model to perform keyword recognition, and obtain the instruction keyword.

[0086] The keyword extraction prompt word is a prompt word used to prompt the large language model's job tasks and job rules, and is used to indicate the large language model's processing target and processing method for the first instruction.

[0087] The keyword prompt words used in some embodiments are as follows: "You are an intent keyword extractor. For instructions of any length, you need to do the following. Step 1: Identify the different components of the instruction, including: main task, constraints, demonstration cases, and text materials. Step 2: The intent keywords only exist in the main task. If the instruction is long, ignore the text materials, demonstration cases, constraints, etc., and only focus on the language of the main task part to extract the core verbs and objects in the main task."

[0088] In some embodiments, in order to facilitate obtaining instruction keywords, prompt keywords for various types of intent can also be added to the keyword extraction prompt words, so that the model can recognize instruction keywords more quickly. For example, the prompt keywords added to the keyword prompt words for information extraction are "r" (extract|extract|list|extract|list).? (key|core|related|content|element|field|option|answer|analysis|key point)", r" (combine|according to|from|read).*? (below|above)(text|information).*? (give|find|output|extract|answer).*? (answer)"]"

[0089] After obtaining the instruction keywords, a regular expression can be constructed based on the instruction keywords. In a specific implementation, the pre-constructed regular construction prompt words and instruction keywords can be used as model inputs, and the second largest language model can be used to construct the expression to obtain a regular expression. In practical applications, the aforementioned first largest language model and the second largest language model can use the same model or different models.

[0090] Considering that constructing a regular expression is essentially to combine instruction keywords such as verbs, restrictive adjectives (such as quantitative adjectives and quantifier adjectives), nouns, etc. to form phrases with structures such as verb-object phrases (including verb-object phrases of various restrictive adjectives), the aforementioned regular construction prompt words can be added to the keyword extraction words, and the regular expression can be directly obtained using the first largest language model.

[0091] After obtaining the regular expression, the regular expression may then be vectorized to obtain a core semantic vector corresponding to the first instruction.

[0092] S112: Determine the rule recognition intent corresponding to the first instruction based on the core semantic vector corresponding to the first instruction and the semantic vectors associated with each intent type.

[0093] The semantic vectors associated with each intent type may be semantic vectors obtained by semantically vectorizing regular expressions of various predetermined intent types. For example, for a certain intent type, an expression constructed with keywords is used to extract the answer, and the regular expression corresponding to this intent type may be a regular expression obtained by processing the aforementioned intent type.

[0094] After obtaining the core semantic vector corresponding to the first instruction, the aforementioned core semantic vector can then be compared with the semantic vectors associated with each intent type to determine the closest semantic vector, and the recognition intent corresponding to the closest semantic vector can be used as the rule recognition intent.

[0095] In actual applications, because keyword extraction tasks are very labor-intensive, there are fewer regular expressions corresponding to each intent type pre-built, and accordingly fewer semantic vectors corresponding to each intent type obtained. To solve this problem, in some embodiments, after determining the rule recognition intent corresponding to the first instruction, the computing device may also perform the following S150.

[0096] S150: Associating the core semantic vector of the first instruction with the corresponding intent type as the semantic vector associated with the corresponding intent type.

[0097] Specifically, after obtaining the core semantic vector of the first instruction and the intent type of its rule recognition intent, the core semantic vector can be directly assigned to the corresponding intent type to increase the number of core semantic vectors associated with the corresponding intent type.

[0098] In some other embodiments, also to solve the problem of fewer semantic vectors corresponding to each intent type, after determining the core semantic vector corresponding to the first instruction, the computing device may then execute the following S160.

[0099] S160: Perform expression matching according to the regular expression corresponding to the first instruction and the regular expression associated with the corresponding intent type, determine a new regular expression associated with the intent type, and construct a new semantic vector associated with the corresponding intent type based on the new regular expression.

[0100] Expression matching is performed based on the regular expression corresponding to the first instruction and the regular expression associated with the corresponding intent type, which matches the regular expression with a typical number of occurrences (specifically, the expression with a large number of occurrences), and then uses it as a new regular expression associated with the intent type.

[0101] After determining the regular expression associated with the intent type, the new regular expression may then be vectorized to obtain a semantic vector, and used as the semantic vector associated with the corresponding intent type.

[0102] As analyzed above, adding corresponding instruction keywords or regular expressions to keyword prompts or regular construction prompts can help the model better determine instruction keywords or write regular expressions. To solve this problem, in some embodiments of the present disclosure, after obtaining a regular expression, the regular expression can also be added to the regular construction prompt, or the instruction keyword in the regular expression can be added to the keyword prompt.

[0103] In some further embodiments, also to solve the problem of fewer semantic vectors corresponding to each intent type, after determining the core semantic vector corresponding to the first instruction, the computing device may then execute the following S170.

[0104] S170: Perform cluster analysis based on the core semantic vector of the first instruction, the semantic vectors associated with each intent type and the number of intent types, determine a new cluster center vector, and use the new cluster center vector as the semantic vector associated with the corresponding intent type.

[0105] After obtaining the core semantic vectors of each first instruction, the clustering set with the number of intent types can be re-determined by cluster analysis, and the cluster center vector can be determined based on the semantic vectors contained in each clustering set. After obtaining the cluster center vector, the semantic vector associated with each intent type can be compared with the cluster center vector to determine the intent type corresponding to each cluster center vector, and use it as the new associated semantic vector of the corresponding intent type.

[0106] The use of the aforementioned S150, S160 or S170 can utilize the characteristics of the large language model's self-summary of intention rules to overcome the problem of time-consuming and labor-intensive rule summarization, and at the same time can also solve the rule loopholes caused by small-scale manual summary of rules.

[0107] As mentioned above, a large language model recognition method needs to be used to process the first instruction to obtain the model recognition intent. In a specific implementation, the computing device may use the following S113 to obtain the model recognition intent.

[0108] S113: Using the pre-constructed intention prompt word and the first instruction as model input, adopting the third largest language model to perform intention recognition, and obtaining the model recognition intention of the first instruction.

[0109] In some embodiments of the present disclosure, the pre-constructed intent prompt words include all intent types in the intent type set.

[0110] In some embodiments, the intent prompt word may also include a description of each intent type. For example, the intent prompt word may be "Now you are a user task intent classifier for prompt. The knowledge base stores a list of labels to be classified and their corresponding explanations, as follows: class_definition_str = {'language understanding': 'including three levels of language understanding, words, sentences, and chapters, and processing-related tasks are prioritized to this category',}, system_prompt = f"Now you are a user task intent classifier for prompt. The knowledge base stores a list of labels to be classified and their corresponding explanations, as follows: \n{class_definition_str}"-put the three-level intent system here; user_prompt = f"You can only select your output from the label list, and you cannot output labels outside the list. If there is no suitable label in the list, please output others, and do not output additional content." Correspondingly, the first instruction can be "The content of the prompt entered by the user is: "{question}".

[0111] The third largest language model in the embodiment of the present disclosure may be the same large language model as the first large language model or the second large language model mentioned above, or may be a different large language model.

[0112] In some embodiments, the second instructions are far more numerous than the instructions in the first instruction set. After determining the intent type of the second instruction using the aforementioned method, the computing device can train the large language model based on the second instruction and the corresponding intent type, so that the large language model can perform intent recognition more accurately.

[0113] In addition to providing the aforementioned instruction intent recognition method, the embodiment of the present disclosure also provides an instruction intent recognition device. Figure 4 Schematic diagram of the structure of the instruction intention recognition device provided by the embodiment of the present disclosure. Figure 4 As shown, the instruction intention recognition device 400 provided in the embodiment of the present disclosure includes an intention prediction unit 401, a method selection unit 402 and a result determination unit 403.

[0114] The intention prediction unit 401 is used to process the first instruction in the first instruction set by adopting a rule recognition method to obtain a rule recognition intention, and to process the first instruction by adopting a large language model recognition method to obtain a model recognition intention; the first instruction has a corresponding manually annotated intention; the intention types of the rule recognition intention, the model recognition intention and the manually annotated intention are all intentions in the pre-constructed intention type set.

[0115] The method selection unit 402 is used to determine the preferred recognition method associated with each intent type in the intent type set based on the rule recognition intent and the model recognition intent, with reference to the manually annotated intent, wherein the preferred recognition method is the rule recognition method or the large language model recognition method;

[0116] The intention prediction unit 401 is also used to process the second instruction using a rule recognition method and a large language model recognition method respectively, to obtain two candidate recognition intentions corresponding to the second instruction.

[0117] The result determination unit 403 is used to determine the selected intention among the two to-be-selected recognition intentions based on the intention types of the two to-be-selected recognition intentions and the preferred recognition methods associated with each intention type, and use the selected intention as the instruction intention of the second instruction.

[0118] In some embodiments, the intent prediction unit 401 uses a rule recognition method to process the first instruction in the first instruction set to obtain a rule recognition intent, including: extracting instruction keywords in the first instruction, constructing a regular expression based on the instruction keywords, and determining the core semantic vector corresponding to the first instruction based on the regular expression; determining the rule recognition intent corresponding to the first instruction based on the core semantic vector corresponding to the first instruction and the semantic vectors associated with each intent type.

[0119] In some embodiments, the intention prediction unit 401 uses the pre-constructed keyword extraction prompt word and the first instruction as model input, and uses the first large language model to perform keyword recognition to obtain the instruction keyword.

[0120] In some embodiments, the intention prediction unit 401 uses pre-constructed regular expression construction prompt words and instruction keywords as model input, adopts the second largest language model to construct the expression, and obtains a regular expression.

[0121] In some embodiments, after determining the rule recognition intent corresponding to the first instruction, the intent determination unit associates the core semantic vector of the first instruction with the corresponding intent type as a semantic vector associated with the corresponding intent type.

[0122] In some embodiments, after determining the rule recognition intention corresponding to the first instruction, the intention determination unit is also used to perform expression matching according to the regular expression corresponding to the first instruction and the regular expression associated with the corresponding intent type, determine a new regular expression associated with the intent type, and construct a new semantic vector associated with the corresponding intent type based on the new regular expression; or, perform cluster analysis based on the core semantic vector of the first instruction, the semantic vectors associated with each intent type, and the number of intent types, determine a new cluster center vector, and use the new cluster center vector as the semantic vector associated with the corresponding intent type.

[0123] In some embodiments, the intent determination unit uses the pre-constructed intent prompt word and the first instruction as model input, uses the third language model to perform intent recognition, and obtains the model recognition intent of the first instruction. The pre-constructed intent prompt word includes all intent types in the intent type set. In practical applications, the intent prompt word can be class_definition_str = {'language understanding':'including three levels of language understanding, words, sentences, and chapters, and processing-related tasks are prioritized to this category',}, system_prompt = f"Now you are a user task intent classifier for prompt. The knowledge base stores a list of labels to be classified and their corresponding explanations, as follows: \n{class_definition_str}"-put the three-level intent system here; user_prompt = f"You can only select your output from the label list, and you cannot output labels outside the list. If there is no suitable label in the list, please output others, and do not output additional content. The content of the prompt entered by the user is: "{question}". The keywords related to prompt intent are: {keywords} The label that best matches its task intent is: ".

[0124] In some embodiments, the method selection unit 402 determines the number of rule recognitions corresponding to each intent type based on the rule recognition intent corresponding to the first instruction, and compares the rule recognition intent corresponding to the first instruction and the corresponding manually labeled intent to determine the number of correct rule recognitions corresponding to each intent type; determines the number of model recognitions corresponding to each intent type based on the model recognition intent corresponding to the first instruction, and compares the model recognition intent corresponding to the first instruction and the corresponding manually labeled intent to determine the number of correct model recognitions corresponding to each intent type; calculates the rule recognition accuracy corresponding to each intent type based on the number of rule recognitions and the number of correct rule recognitions corresponding to each intent type; and calculates the model recognition accuracy corresponding to each intent type based on the number of model recognitions and the number of correct model recognitions corresponding to each intent type; respectively compares the rule recognition accuracy and the model recognition accuracy corresponding to each intent type, determines the larger recognition accuracy, and uses the intent recognition method corresponding to the larger recognition accuracy as the preferred recognition method for the corresponding intent type.

[0125] The embodiment of the present disclosure also provides a computing device for implementing the aforementioned method. Figure 5 Schematic diagram of the structure of the computing device provided by the embodiment of the present disclosure. Figure 5 , which shows a schematic diagram of the structure of a computing device 500 suitable for implementing the embodiments of the present disclosure. Figure 5 The computing device shown is only an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present disclosure.

[0126] like Figure 5 As shown, the computing device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory ROM 502 or a program loaded from a storage device 508 to a random access memory RAM 503. In the RAM 503, various programs and data required for the operation of the computing device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output I / O interface 505 is also connected to the bus 504.

[0127] Typically, the following devices may be connected to the I / O interface 505: input devices 505 including, for example, a touch screen, a touch pad, a camera, a microphone, etc.; output devices 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 508 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 509. The communication devices 509 may allow the computing device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5The computing device 500 having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0128] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0129] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable storage medium, a computer-readable signal medium, or any combination of the above two.

[0130] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0131] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer readable program codes. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0132] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0133] The computer-readable medium may be included in the computing device, or may exist independently without being incorporated into the computing device.

[0134] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the tester computer, partially on the tester computer, as a separate software package, partially on the tester computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the tester computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0135] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to each embodiment of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system based on hardware that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0136] The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. The name of the unit does not, in some cases, constitute a limitation on the unit itself. The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0137] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying instruction intention, characterized in that: include: A rule recognition method is used to process a first instruction in a first instruction set to obtain a rule recognition intent, and a large language model recognition method is used to process the first instruction to obtain a model recognition intent; the first instruction has a corresponding manually annotated intent; the intent types of the rule recognition intent, the model recognition intent and the manually annotated intent are all intents in a pre-constructed intent type set; Taking the manually annotated intent corresponding to the first instruction as a reference, determining a preferred recognition method associated with each intent type in the intent type set based on the rule recognition intent and the model recognition intent corresponding to the first instruction, wherein the preferred recognition method is the rule recognition method or the large language model recognition method; The second instruction is processed by the rule recognition method and the large language model recognition method respectively, so as to obtain two to-be-selected recognition intentions corresponding to the second instruction; Based on the intention types of the two to-be-selected recognition intentions and the preferred recognition methods associated with the respective intention types, a selected intention is determined among the two to-be-selected recognition intentions, and the selected intention is used as the instruction intention of the second instruction.

2. The method according to claim 1, characterized in that The step of using the rule recognition method to process the first instruction in the first instruction set to obtain the rule recognition intention includes: Extracting instruction keywords from the first instruction, constructing a regular expression based on the instruction keywords, and determining a core semantic vector corresponding to the first instruction based on the regular expression; Based on the core semantic vector corresponding to the first instruction and the semantic vectors associated with each intent type, the rule recognition intent corresponding to the first instruction is determined.

3. The method according to claim 2, characterized in that The extracting the instruction keyword in the first instruction includes: The pre-constructed keyword extraction prompt word and the first instruction are used as model inputs, and the first language model is used to perform keyword recognition to obtain the instruction keyword.

4. The method according to claim 2, characterized in that: The constructing a regular expression based on the instruction keyword includes: The pre-constructed regular expression construction prompt word and the instruction keyword are used as model input, and the second largest language model is used to construct the expression to obtain the regular expression.

5. The method according to claim 2, characterized in that: After determining the rule recognition intention corresponding to the first instruction, the method further includes: The core semantic vector of the first instruction is associated with the corresponding intent type as a semantic vector associated with the corresponding intent type.

6. The method according to claim 2, characterized in that After determining the rule recognition intention corresponding to the first instruction, the method further includes: Perform expression matching according to the regular expression corresponding to the first instruction and the regular expression associated with the corresponding intent type, determine a new regular expression associated with the intent type, and construct a new semantic vector associated with the corresponding intent type based on the new regular expression; or, According to the core semantic vector of the first instruction, the semantic vectors associated with each of the intent types and the number of intent types are clustered and analyzed to determine a new cluster center vector, which is used as the semantic vector associated with the corresponding intent type.

7. The method according to any one of claims 1 to 6, characterized in that: The method of using a large language model to process the first instruction to obtain a model recognition intent includes: using a pre-constructed intent prompt word and the first instruction as a model input, using a third large language model to perform intent recognition, and obtaining a model recognition intent of the first instruction, wherein the pre-constructed intent prompt word includes all intent types in the intent type set; or The preferred identification method for determining the association between each intent type in the intent type set based on the rule identification intent and the model identification intent with reference to the manually annotated intent includes: Determine the number of rule recognitions corresponding to each intent type based on the rule recognition intent corresponding to the first instruction, and compare the rule recognition intent corresponding to the first instruction with the corresponding manually labeled intent to determine the number of correct rule recognitions corresponding to each intent type; Determine the number of model recognitions corresponding to each intent type based on the model recognition intent corresponding to the first instruction, and compare the model recognition intent corresponding to the first instruction with the corresponding manually labeled intent to determine the number of correct model recognitions corresponding to each intent type; Based on the number of rule recognitions and the number of correct rule recognitions corresponding to each intent type, the rule recognition accuracy corresponding to each intent type is calculated; and based on the number of model recognitions and the number of correct model recognitions corresponding to each intent type, the model recognition accuracy corresponding to each intent type is calculated; The rule recognition accuracy and model recognition accuracy corresponding to each intent type are compared respectively, the larger recognition accuracy is determined, and the intent recognition method corresponding to the larger recognition accuracy is used as the preferred recognition method for the corresponding intent type.

8. A command intention recognition device, characterized in that: include: An intention prediction unit, configured to process a first instruction in a first instruction set by a rule recognition method to obtain a rule recognition intention, and to process the first instruction by a large language model recognition method to obtain a model recognition intention; the first instruction has a corresponding manually annotated intention; the intention types of the rule recognition intention, the model recognition intention and the manually annotated intention are all intentions in a pre-constructed intention type set; A method selection unit, used to determine a preferred recognition method associated with each intent type in an intent type set based on the rule recognition intent and the model recognition intent, with reference to the manually annotated intent, wherein the preferred recognition method is the rule recognition method or the large language model recognition method; The intention prediction unit is further used to process the second instruction by using the rule recognition method and the large language model recognition method respectively, to obtain two to-be-selected recognition intentions corresponding to the second instruction; A result determination unit is used to determine a selected intention among the two to-be-selected recognition intentions based on the intention types of the two to-be-selected recognition intentions and the preferred recognition methods associated with each intention type, and to use the selected intention as the instruction intention of the second instruction.

9. A computing device, characterized in that It comprises a processor and a memory, wherein the memory is used to store a computer program; when the computer program is loaded by the processor, the processor executes the instruction intent recognition method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the processor implements the instruction intent recognition method as described in any one of claims 1-7.

Images