Plug-in calling method and device, equipment and storage medium

The plug-in calling method addresses the accuracy and efficiency issues of large language models by using a memory mechanism to store user feedback for quick and consistent intent recognition and parameter extraction, improving user experience.

CN119088919APending Publication Date: 2024-12-06THE FOURTH PARADIGM BEIJING TECH CO LTD

Patent Information

Application Number
CN202411124667.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Large language models face challenges in accurately and efficiently performing intent recognition and parameter extraction in complex engineering scenarios, leading to inconsistent responses and delayed feedback, which affects user experience.

Method used

A plug-in calling method that utilizes a memory mechanism to store and analyze user feedback, enabling quick retrieval of trusted memories for intent recognition and parameter extraction, thereby improving accuracy and efficiency.

Benefits of technology

The method ensures consistent and timely responses by leveraging trusted memories for plug-in distribution and parameter extraction, enhancing user experience and reducing latency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088919A_ABST
    Figure CN119088919A_ABST
Patent Text Reader

Abstract

The invention relates to a plug-in calling method and device, equipment and a storage medium. In response to first questioning information of the user, whether target memory data matched with the first questioning information exists in a database is judged, the database is used for storing multiple pieces of memory data, and the memory data comprises historical questioning information and second plug-in calling information corresponding to the historical questioning information; and if the target memory data exists in the database, determining first plug-in calling information corresponding to the first question information based on the target memory data. Therefore, the consistency problem and the feedback timeliness problem can be solved while the delay is low, and the user experience is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a plug-in calling method, device, equipment and storage medium. Background Art

[0002] With the continuous development of Large Language Model (LLM) technology, how to automatically apply large language models to solve problems in engineering scenarios has become a technical focus. Large language models themselves lack the ability to perform actions or actively obtain information, so plug-ins are needed to perform standardized engineering tasks, such as weather query tasks and hotel reservation tasks. In this context, OpenAI ChatGPT launched a plug-in mechanism.

[0003] In actual engineering scenarios, it is often necessary to enable multiple plug-ins at the same time, especially multiple plug-ins that may have similar functions. This requires selecting the appropriate plug-in based on the user's questions and supplementing the corresponding parameters.

[0004] Existing solutions often use large models to select intent recognition and parameter extraction prompts and build application processes. The above process faces challenges in both accuracy and efficiency. From the perspective of accuracy, in more complex actual engineering scenarios, there are often difficult intent recognition or parameter extraction problems that are difficult for large models to solve, such as the presence of a large number of similar intents, close or nested parameter meanings, etc. For similar scenarios, large models often cannot complete the task accurately and need to use other means to solve the problem, which limits the flexibility and scalability of the application. From the perspective of efficiency, even when resources are sufficient, it takes seconds for a large model to perform a simple intent recognition and parameter extraction task. If the output is long or resources are tight and the model load is high, the response time may be extended to 10 seconds or even longer, which greatly affects the user experience.

[0005] In addition, users may have some user experience problems during use, among which the more obvious problems are consistency problems and feedback timeliness problems. For example, for consistency problems, due to the instability of the effect of the large model itself, a user may get completely different answers after asking two very similar questions. Similar scenarios can easily confuse users and question the reliability of large model applications. For another example, in terms of feedback, after users give feedback on satisfactory / unsatisfactory answers, manual operations are often required to evaluate the data and iterate the strategy. This process takes a certain amount of time and cannot immediately improve the user experience.

[0006] Therefore, a solution is needed that can improve accuracy and efficiency while improving user experience problems (such as consistency problems and feedback timeliness problems) during user use. Summary of the invention

[0007] A technical problem to be solved by the present disclosure is how to improve the user experience while improving accuracy and efficiency in application scenarios combining large language models with plug-ins.

[0008] According to a first aspect of the present disclosure, a plug-in calling method is provided, comprising: in response to a user's first question information, determining whether there is target memory data matching the first question information in a database, the database being used to store multiple memory data, the memory data comprising historical question information and second plug-in calling information corresponding to the historical question information; if the target memory data exists in the database, determining first plug-in calling information corresponding to the first question information based on the target memory data.

[0009] Optionally, before determining whether there is target memory data matching the first question information in the database, the method further includes: selecting part of the memory data from the database for the first question information to obtain a candidate memory data set; determining whether there is target memory data matching the first question information in the database, including: determining whether there is target memory data matching the first question information in the candidate memory data set.

[0010] Optionally, selecting part of the memory data from the database for the first question information includes: selecting memory data that meets the following conditions from the database: the historical question information and the first question information correspond to the same organization; and / or the historical question information and the first question information correspond to the same enabled plug-in.

[0011] Optionally, determining whether there is target memory data matching the first question information in the database includes: determining whether there is second question information identical to the first question information in the database; if the second question information exists in the database, taking the memory data corresponding to the second question information as the target memory data.

[0012] Optionally, determining first plug-in calling information corresponding to the first question information based on the target memory data includes: using second plug-in calling information corresponding to second question information in the target memory data as first plug-in calling information corresponding to the first question information.

[0013] Optionally, determining whether there is target memory data matching the first question information in the database further includes: if the second question information does not exist in the database, determining whether there is third question information in the database having the same word segmentation result as the first question information, the word segmentation result being used to characterize the part of speech of each word segmented after word segmentation of the question information; if the third question information exists in the database, calculating the similarity between the first question information and the third question information; and taking the memory data corresponding to the third question information whose similarity is greater than or equal to the first threshold and whose similarity ranking is the highest as the target memory data.

[0014] Optionally, calculating the similarity between the first question information and the third question information includes: obtaining a first vectorized representation of the first question information, the first vectorized representation being used to represent a vectorized representation of a sentence after at least some words in the first question information are replaced with equivalent meanings; obtaining a second vectorized representation of the third question information, the second vectorized representation being used to represent a vectorized representation of a sentence after at least some words in the three question information are replaced with equivalent meanings; and calculating the similarity between the first vectorized representation and the second vectorized representation.

[0015] Optionally, determining the first plug-in calling information corresponding to the first question information based on the target memory data includes: using the plug-in distribution result in the second plug-in calling information of the target memory data as the plug-in distribution result corresponding to the first question information; and / or based on the word segmentation part corresponding to each plug-in parameter in the parameter extraction result in the second plug-in calling information of the target memory data, performing parameter extraction on the corresponding word segmentation part in the first question information to obtain the parameter extraction result corresponding to the first question information.

[0016] Optionally, the method also includes: if there is no target memory data matching the first question information, determining the first plug-in call information through a plug-in distribution and / or parameter extraction process; and / or calling a plug-in based on the first plug-in call information, and generating response information for the first question information based on the plug-in call result.

[0017] Optionally, the method further includes: obtaining feedback results from the user regarding the response information; if the feedback results indicate that the user is satisfied with the response information, generating the memory data based on the first question information and the first plug-in call information and storing it in the database.

[0018] According to a second aspect of the present disclosure, a plug-in calling device is provided, comprising: a judgment module, for judging, in response to a user's first question information, whether there is target memory data matching the first question information in a database, the database being used to store multiple memory data, the memory data comprising historical question information and second plug-in calling information corresponding to the historical question information; a processing module, for determining, based on the target memory data, first plug-in calling information corresponding to the first question information if the target memory data exists in the database.

[0019] Optionally, the method further includes: a reading module, used to select part of the memory data from the database according to the first question information to obtain a candidate memory data set, and the judgment module is used to judge whether there is target memory data matching the first question information in the candidate memory data set.

[0020] Optionally, the reading module selects memory data that meets the following conditions from the database: the historical question information and the first question information correspond to the same organization; and / or the historical question information and the first question information correspond to the same enabled plug-in.

[0021] Optionally, the judgment module includes: a first judgment module, used to judge whether there is second question information identical to the first question information in the database; and a first determination module, used to use the memory data corresponding to the second question information as the target memory data if the second question information exists in the database.

[0022] Optionally, the processing module uses the second plug-in calling information corresponding to the second question information in the target memory data as the first plug-in calling information corresponding to the first question information.

[0023] Optionally, the judgment module also includes: a second judgment module, which is used to judge whether there is a third question information in the database whose segmentation result is the same as the first question information if the second question information does not exist in the database, and the segmentation result is used to characterize the part of speech of each segmented word obtained after segmenting the question information; a calculation module, which is used to calculate the similarity between the first question information and the third question information if the third question information exists in the database; and a second determination module, which is used to take the memory data corresponding to the third question information whose similarity is greater than or equal to the first threshold and whose similarity ranking is the highest as the target memory data.

[0024] Optionally, the calculation module includes: a first acquisition module, used to acquire a first vectorized representation of the first question information, the first vectorized representation is used to represent a vectorized representation of a sentence after at least part of the words in the first question information are replaced with equivalent meanings; a second acquisition module, used to acquire a second vectorized representation of the third question information, the second vectorized representation is used to represent a vectorized representation of a sentence after at least part of the words in the three question information are replaced with equivalent meanings; and a calculation submodule, used to calculate the similarity between the first vectorized representation and the second vectorized representation.

[0025] Optionally, the processing module uses the plug-in distribution result in the second plug-in calling information of the target memory data as the plug-in distribution result corresponding to the first question information; and / or the processing module extracts parameters of the corresponding word segmentation parts in the first question information based on the word segmentation parts corresponding to each plug-in parameter in the parameter extraction result in the second plug-in calling information of the target memory data, to obtain the parameter extraction result corresponding to the first question information.

[0026] Optionally, the method also includes: a distribution and parameter extraction module, which is used to determine the first plug-in call information through a plug-in distribution and / or parameter extraction process if there is no target memory data matching the first question information; and / or a response module, which is used to call the plug-in based on the first plug-in call information, and generate response information for the first question information based on the plug-in call result.

[0027] Optionally, the method also includes: a third acquisition module, used to obtain the user's feedback result on the response information; a generation module, used to generate the memory data and store it in the database based on the first question information and the first plug-in call information if the feedback result indicates that the user is satisfied with the response information.

[0028] According to a third aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory on which executable codes are stored, and when the executable codes are executed by the processor, the processor executes the method described in the first aspect above.

[0029] According to a fourth aspect of the present disclosure, a computer program product is provided, comprising an executable code, which, when executed by a processor of an electronic device, causes the processor to execute the method described in the first aspect above.

[0030] According to a fifth aspect of the present disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the first aspect above.

[0031] Therefore, the present invention determines whether there is target memory data matching the first question information in the database, and determines the first plug-in call information corresponding to the first question information based on the target memory data, so as to solve the consistency problem and feedback timeliness problem with low latency and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0033] Figure 1 is a schematic flow chart showing a plug-in calling method according to an embodiment of the present disclosure.

[0034] Figure 2 is a schematic diagram showing a flow chart of determining whether target memory data exists in a database according to an embodiment of the present disclosure

[0035] Figure 3 2 is a schematic diagram showing how to determine the plug-in calling information of the current question based on the plug-in calling information of the memory data.

[0036] Figure 4 It is a schematic diagram showing the construction process of the memory database.

[0037] Figure 5 The figure shows the plug-in calling process intention based on the memory mechanism.

[0038] Figure 6 It is a schematic diagram showing the structure of a plug-in calling device according to an embodiment of the present disclosure.

[0039] Figure 7 A schematic diagram of the structure of a computing device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0040] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0041] The plug-ins mentioned in the present disclosure refer to function modules that are used to implement specific functions and can be called.

[0042] The present disclosure is particularly applicable to multi-plugin application scenarios.

[0043] In a multi-plugin application scenario, it is necessary to select a plug-in that matches the current user's question information from multiple plug-ins, call the selected plug-in, and provide services based on the component call results. Multi-plugin application scenarios can be, but are not limited to, LLM applications that support the plug-in mechanism. LLM applications that support the plug-in mechanism mainly use the natural language understanding and output capabilities of LLM to connect with specific plug-ins to provide answer services to users.

[0044] The plug-in calling task in the multi-plug-in application scenario mainly includes two subtasks: plug-in distribution and parameter extraction.

[0045] Plugin distribution refers to selecting a target plugin suitable for processing the current question information from multiple plugins (such as multiple enabled plugins). Each plugin can correspond to an intent. Plugin distribution can also be called intent recognition. The plugin distribution result can also be called intent recognition result. In other words, the plugin distribution result can refer to the determined plugin to be called, or the determined intent corresponding to the plugin to be called.

[0046] Parameter extraction refers to extracting parameter values ​​that meet the input parameter definition of the target plug-in from the current question information.

[0047] For plug-in calling tasks in multi-plug-in application scenarios, plug-in distribution and parameter extraction can be achieved with the help of LLM, rule processing or other means. This method has a high latency and cannot solve the consistency problem and feedback timeliness problem. The consistency problem means that the user may get different or even completely different answers to two very similar questions. The feedback timeliness problem means that after the user gives feedback on the satisfactory / unsatisfactory answers, the user experience cannot be improved immediately.

[0048] In response to this, the present disclosure proposes a memory mechanism to solve consistency problems and feedback timeliness problems while reducing latency. Simply put, the memory mechanism refers to the ability to extract trusted memories based on user feedback, and use trusted memories for subsequent user questions to improve the accuracy and efficiency of plug-in distribution (i.e., intent recognition) and parameter extraction. In addition, since the currently extracted trusted memories can be used for the next question, the timeliness problem of feedback during user use can be solved. Moreover, the memory mechanism can ensure that when a user asks a very similar question, if the previous question has been marked as a trusted memory, the new question can also get a reliable result, thereby also solving the consistency problem.

[0049] The following is a further description of various details of the present disclosure.

[0050] Figure 1 is a schematic flow chart showing a plug-in calling method according to an embodiment of the present disclosure. Figure 1The method shown can be implemented in software via a computer program, and can also be executed via a specifically configured computing device.

[0051] See also Figure 1 In step S110, in response to the user's first question information, it is determined whether there is target memory data matching the first question information in the database.

[0052] The database is used to store multiple memory data. Therefore, the database can also be called a memory database.

[0053] Each piece of memory data includes historical question information and second plug-in calling information corresponding to the historical question information.

[0054] The first question information refers to the question asked by the current user at the current moment, which requires LLM to obtain corresponding answer information by calling the plug-in. The historical question information refers to the question asked by the past user at the past moment, which has been answered by LLM by calling the plug-in. The current user and the past user can be the same user or different users.

[0055] The second plug-in calling information refers to the plug-in calling information obtained by performing plug-in distribution and / or parameter extraction for the historical question information. That is, the second plug-in calling information may include the second plug-in distribution result and / or the second parameter extraction result. The second plug-in distribution result is the plug-in to be called determined by performing plug-in distribution for the historical question information. The second parameter extraction result is the parameter value extracted from the historical question information that meets the input parameter definition of the plug-in to be called.

[0056] The memory data stored in the database can all be credible memory data. Credible memory data refers to memory data constructed based on historical question information and plug-in call results corresponding to historical answer information that the user is satisfied with. It is possible to determine whether the user is satisfied with the answer information generated for the question information based on user feedback, and construct credible memory data based on the question information corresponding to the answer information that the user is satisfied with (i.e., historical question information).

[0057] Determine whether there is target memory data matching the first question information in the database, that is, determine whether there is available memory data for the first question information in the database. Available memory data refers to memory data that can skip the conventional plug-in distribution and / or parameter extraction process for the first question information and determine the first plug-in call information (i.e., plug-in distribution result and / or parameter extraction result) of the first question information.

[0058] Figure 2 is a schematic diagram showing a flow chart of determining whether target memory data exists in a database according to an embodiment of the present disclosure. Step S110 may include Figure 2The steps S111 to S119 shown above.

[0059] Refer to Figure 2 , first, step S111 can be executed to determine whether there is a second question information in the database that is the same as the first question information. For example, the first question information can be compared with the historical question information included in each memory data in the database respectively to determine whether there is a second question information in the database that is the same as the first question information.

[0060] If the determination result of step S111 is yes, that is, there is a second question information that is the same as the first question information, then step S112 can be executed to use the memory data corresponding to the second question information as the target memory data.

[0061] If the determination result of step S111 is no, that is, there is no second question information that is the same as the first question information, then step S113 can be executed to determine whether there is a third question information in the database whose word segmentation result is the same as the first question information. For example, the first word segmentation result of the first question information can be compared with the second word segmentation results of the historical question information included in each memory data in the database respectively to determine whether there is a third question information in the database whose word segmentation result is the same as the first question information.

[0062] The word segmentation results (the first word segmentation result and the second word segmentation result) are used to represent the parts of speech of each word obtained after word segmentation of the question information (the first question information and the historical question information). That is, the word segmentation result refers to the parts of speech of each word obtained after word segmentation processing of the question information, rather than the word segmentation itself.

[0063] For example, the word segmentation result of the question sentence "Draw a cat" can be expressed as an ordered part-of-speech sequence ["v", "num", "n"]. Among them, "v" represents a verb, "num" represents a quantifier, and "n" represents a noun. Among them, in order to ensure that the extracted parts of speech are all the main contents with meanings in the original sentence and exclude the interference of meaningless parts of speech such as modal particles and auxiliary words, the meaningless part of speech (such as the part of speech corresponding to "de") can be removed, and only the remaining meaningful part of speech is retained.

[0064] In addition, in specific application scenarios, it is difficult to map two different professional terms to the same part of speech, resulting in difficulties in applying memory in corresponding problems. To solve this problem, professional terms can be marked as a thesaurus with the same part of speech and added to the word segmentation configuration to ensure that they are assigned to the same type of words and improve the versatility of the memory mechanism. That is, when performing word segmentation, various professional terms can be mapped to the same type of part of speech.

[0065] Assuming that the first question information is: How is the weather in Beijing?, then the parts of speech of each word segmented by word segmentation processing of the question information (i.e., the first word segmentation result) can be recorded as: ["ns", "n", "vi"], where "ns" represents a place name; a certain historical question information is: How is the weather in Shanghai?, then the parts of speech of each word segmented by word segmentation processing of the historical question information (i.e., the second word segmentation result) can be recorded as: ["v", "ns", "n", "vi"]. It can be seen that the two are not completely consistent, so the second word segmentation result of the historical question information is different from the first word segmentation result of the first question information, and the historical question information is not the third question information that is the same as the first question information.

[0066] If the judgment result of step S113 is no, that is, there is no third question information in the database whose word segmentation result is the same as the first question information, then it can be considered that there is no target memory data matching the first question information in the database. In other words, there is no memory data available for the first question information in the database. At this time, the conventional plug-in calling process (i.e., the conventional plug-in distribution and / or parameter extraction process) can be executed for the first question information. The conventional plug-in calling process may refer to an LLM-based plug-in distribution and / or parameter extraction process, or a rule-based plug-in distribution and / or parameter extraction process, or other methods (such as combining LLM and rules) of plug-in calling processes.

[0067] If the judgment result of step S113 is yes, that is, there is a third question information in the database whose word segmentation result is the same as the first question information, it indicates that the target memory data may exist in the database. That is, there may be memory data available for the first question information in the database. The number of third question information whose word segmentation result is the same as the first question information may be one or more. In the case that there is a third question information whose word segmentation result is the same as the first question information, steps S115 to S119 can be executed to further determine whether there is target memory data in the third question information whose word segmentation result is the same as the first question information from the perspective of semantic similarity.

[0068] In step S115, the similarity between the first question information and the third question information is calculated.

[0069] In order to ensure the generalization of parameters under the memory mechanism and eliminate the influence of some complex professional term parameters that account for too high a proportion in the semantics, when calculating the similarity between the first question information and the third question information, the similarity between the first question information and the third question information after equivalent replacement can be calculated. Equivalent replacement, also known as parameter equivalent replacement, refers to equivalent replacement of some words (such as professional term parameters) in the question information (the first question information and the third question information). For example, the word "Beijing" can be replaced with "place name."

[0070] Exemplarily, a first vectorized representation of the first question information and a third vectorized representation of the third question information can be obtained. The first vectorized representation is used to represent the vectorized representation of the sentence after at least some words (such as nouns, especially professional noun parameters) in the first question information are replaced with equivalent meanings. The second vectorized representation is used to represent the vectorized representation of the sentence after at least some words in the three question information are replaced with equivalent meanings. Among them, the vectorized representation can be generated using an embedding model (i.e., an embedding model), and the embedding model can adopt but is not limited to the m3e-base model. The first vectorized representation can be obtained by real-time processing. The second vectorized representation can be pre-stored in a database, and the second vectorized representation can be directly obtained from the database. After obtaining the first vectorized representation and the second vectorized representation, the similarity between the first vectorized representation and the second vectorized representation can be calculated. The similarity calculation method can adopt but is not limited to cosine similarity, Pearson correlation coefficient, two norm, Jaccard similarity coefficient, etc.

[0071] In step S117, it is determined whether there is a similarity greater than or equal to a first threshold.

[0072] If the determination result of step S117 is no, that is, there is no similarity greater than or equal to the first threshold, it can be considered that all third question information is unavailable, that is, there is no target memory data matching the first question information in the database. The specific value of the first threshold can be flexibly set according to actual conditions, such as 0.95. If the determination result of step S117 is no, the conventional plug-in call process can be executed for the first question information.

[0073] If the determination result of step S117 is yes, that is, there is a similarity greater than or equal to the first threshold, step S119 can be executed to take the memory data corresponding to the third question information whose similarity is greater than or equal to the first threshold and whose similarity ranking is the highest as the target memory data.

[0074] In step S120, if the target memory data exists in the database, first plug-in calling information corresponding to the first question information is determined based on the target memory data.

[0075] As described above, the target memory data matching the first question information mainly includes the following two types. One is the memory data whose historical question information is exactly the same as the first question information. The other is the memory data whose historical question information is not exactly the same as the first question information, but whose sentence backbone parts of speech are the same and only the specific parameter values ​​are different.

[0076] For the first type, since the historical question information (i.e., the second question information) is exactly the same as the first question information, the second plug-in call information corresponding to the second question information in the target memory data can be directly used as the first plug-in call information corresponding to the first question information. That is, the steps of plug-in distribution and / or parameter extraction for the first question information can be omitted, and the second plug-in call information obtained by plug-in distribution and / or parameter extraction for the historical question information can be directly used as the plug-in distribution result and / or parameter extraction result (i.e., the first plug-in call information) of the first question information.

[0077] For the second type, since the historical question information (i.e., the third question information) has the same intention as the first question information, only the parameter values ​​are different. Therefore, the plug-in distribution result (i.e., the plug-in that needs to be called) in the second plug-in call information corresponding to the third question information in the target memory data can be used as the plug-in distribution result of the second question information. As for parameter extraction, the parameter extraction result in the second plug-in call information in the target memory data can be used to assist in parameter extraction of the second question information. Specifically, although the parameter extraction process has not been performed for the first question information and the parameter position cannot be directly determined, since the word segmentation results of the first question information and the third question information are completely consistent, the word segmentation parts corresponding to the parameters (i.e., plug-in parameters) in the parameter extraction results in the second plug-in call information in the target memory data can be used to extract parameters for the corresponding (i.e., the same) word segmentation parts in the first question information to obtain the parameter extraction results corresponding to the first question information.

[0078] Figure 3 2 is a schematic diagram showing how to determine the plug-in calling information of the current question based on the plug-in calling information of the memory data.

[0079] See also Figure 3 , the historical question sentence "What are the orders of the Wuxi factory today?" and the current question sentence "What are the orders of the Foshan factory tomorrow?" have the same word segmentation results and the similarity is greater than the predetermined threshold. The memory data corresponding to the historical question sentence can be used as the target memory data available for the current question sentence. The plug-in distribution result in the memory data is the plug-in that needs to be called corresponding to the memory intent of "query order records". The parameter extraction result in the memory data is {time: today; location: Wuxi factory}. For the current question sentence, the plug-in that needs to be called corresponding to the memory intent of "query order records" can be directly used as the plug-in distribution result of the current question sentence. For the parameter extraction part, the corresponding part of speech part (i.e. time and place) in the current question sentence can be extracted according to the part of speech part (i.e. time and place) corresponding to the parameter value in the memory data (today, Wuxi factory), and the parameter extraction result of {time: today; location: Wuxi factory} is obtained.

[0080] In some embodiments, before executing step S110, some memory data may be selected from the database for the first question information to obtain a candidate memory data set. Candidate memory data refers to memory data that is available for backup for the first question information. Exemplarily, memory data that meets the following conditions may be selected from the database: the historical question information corresponds to the same organization as the first question information; and / or the historical question information corresponds to the same enabled plug-in as the first question information, and each enabled plug-in may be regarded as an alternative intention. Thus, when executing step S110 (for example Figure 2 When performing the steps shown in the figure, it can be determined whether there is target memory data matching the first question information in the candidate memory data set (rather than the entire database). By reading the candidate memory data, the accuracy can be improved while reducing the computing cost.

[0081] After obtaining the first plug-in call information, the plug-in can be called based on the first plug-in call information, and response information for the first question information can be generated based on the plug-in call result. For example, an input prompt (i.e., a prompt) can be generated based on the first question information and the plug-in call result, the input prompt can be input into a generative model (such as LLM), and the output of the generative model can be provided to the user as response information.

[0082] In some embodiments, the user's feedback result for the response information can also be obtained. If the feedback result shows that the user is satisfied with the response information, memory data can be generated and stored in the database based on the first question information and the first plug-in call information. Among them, if there is memory data in the database that is exactly the same as the first question information, memory data can no longer be generated based on the first question information and the first plug-in call information to avoid repeated storage. Optionally, if the feedback result shows that the user is not satisfied with the response information, the target memory data corresponding to the first question information can also be deleted from the database.

[0083] Figure 4 It is a schematic diagram showing the construction process of the memory database.

[0084] See also Figure 4 In the process of LLM application providing question-and-answer services to users, a large amount of behavior data will be generated, including questions asked by users, intents routed to by LLM application (corresponding to the plug-in to be called), corresponding parameters extracted, time of corresponding behavior execution, user feedback on the results, etc. In order to analyze user behavior and feedback and extract reliable memory content, it is necessary to record user behavior data.

[0085] The following is an example of a record of user behavior data (some fields are omitted):

[0086]

[0087]

[0088] Among them, orgId represents the user's organization, userId represents the user's nickname, selected_plugins represents the plug-ins that the user turns on (allowing intent recognition) when asking questions, query represents the user's original question content, operation_selected represents the intent recognized by the LLM application (the intent has a clear corresponding plug-in), the parameters field and the requestBody field represent the parameters extracted by the LLM application, userBehavior represents the user's behavior or feedback, and the timestamp field represents the time when the user behavior is recorded.

[0089] In the actual engineering development process, the user's behavior may also involve more dimensional data including the above fields to record the user's behavior more comprehensively. The above fields will be used to describe the user's behavior and provide assistance for the memory mechanism proposed in this disclosure.

[0090] like Figure 4 As shown, the user's behavior data will be saved and may be used to generate memories. Memories can be generated according to a set of configured rules, and operators can manage the rules for memory generation. For example, under the most stringent rules, only behavior data manually approved by operators will be considered trusted memories. Conversely, operators can enable different degrees of automatic generation of trusted memories. For example: if a user likes an answer, the corresponding behavior will be added to the trusted memory; if the user clicks "Confirm" in a confirmation intent window, the corresponding behavior will be added to the trusted memory.

[0091] Depending on the configuration, the logic of converting the immediate action into memory changes accordingly.

[0092] When it is confirmed that a certain behavior data needs to be converted into memory data, it can be processed by combining natural language processing (NLP) technology and embedding technology to generate the following data based on a certain user behavior data:

[0093]

[0094] The tokenization field can be the result of segmenting and tagging the original query of the user using, but not limited to, the jieba library. The result is a list in which each element represents the part of speech of each segment of the original sentence after segmentation. In order to ensure that the extracted parts of speech are the main content with meaning in the original sentence, and exclude interference from modal particles, auxiliary words, etc., the corresponding meaningless parts of speech will be removed, and only the rest will be retained.

[0095] The embedding field is a vector value obtained by performing embedding vectorization after replacing the original sentence with equivalent parameters. It can be stored in the pgvector type data of the postgres database.

[0096] The explanation of parameter equivalent replacement is as follows: the user's behavior data includes the user's original question and the parameters obtained from the question. In order to ensure the generalization of parameters under the memory mechanism and eliminate the influence of some complex professional ranking parameters that account for too high a proportion in the semantics, when vectorizing user questions, the parameter source part in the original sentence will be replaced based on part-of-speech equivalents before vectorization and preservation. For example, the query "How is the weather in Beijing?" after parameter equivalent replacement can be "How is the weather in a certain place?" When vectorizing, the query based on parameter equivalent replacement can be vectorized.

[0097] After building the memory database, each time a user makes a new query, it will first try to use the memory mechanism to implement the plug-in call. Figure 5 The figure shows the plug-in calling process intention based on the memory mechanism.

[0098] See also Figure 5 After a user asks a question, the corresponding memory can be read first according to the user's information. For example, only the memory content under the same organization, with the same plug-in enabled, and containing the same alternative intention will be added to the alternative.

[0099] After the memory is read, a completely consistent memory judgment is performed first. The execution logic of this part is similar to getting the result from the cache. If and only if the user's question and the query in the memory are exactly the same, the memory is equivalent to a special cache. At this time, the intent recognition (i.e., plug-in distribution) and parameter extraction results of the corresponding memory are directly used as the intent recognition and parameter extraction results obtained by this user query, and the regular extraction and distribution process is skipped.

[0100] For example, for the query "What is the weather in Beijing?", if there is a completely identical memory query, the memory intent / parameters (such as weather plug-in / city: Beijing) can be used as the result of this plug-in distribution and parameter extraction, that is, the result this time is also: weather plug-in / city: Beijing.

[0101] If there is no completely identical memory, the user's query can be called Jieba's word segmentation and part-of-speech tagging model, and the tokenization field result can be obtained. After that, the tokenization result of the user's query (i.e., the word segmentation result) is compared with the tokenization result of the available memory, and only the memory with completely identical word segmentation results is considered valid.

[0102] For example, after segmenting and tagging the query "How is the weather in Beijing?", the segmentation result (memory_tokenization) for the historical question "How is the weather in Shanghai?" can be recorded as ["v", "ns", "n", "vi"]. These two tokenizations are not completely consistent, and the memory match fails.

[0103] If there is a completely identical memory, the query can be replaced with equivalent parameters, and then the embedding similarity can be compared with the qualified memory (that is, the memory with the exactly identical word segmentation results). After that, the embedding vectorization result is calculated, and the similarity with the embedding field of the memory data is calculated. If the highest similarity exceeds 0.95, the intent recognition and parameter extraction results corresponding to the memory data can be used as the intent recognition and parameter extraction results obtained by this user query, and the regular parameter extraction and distribution process can be skipped. At this time, since the query has not undergone the parameter extraction process, the parameter position cannot be directly determined. However, since the word segmentation results of the query and the memory are exactly the same, the word segmentation part corresponding to the parameter in the memory is also used as the parameter part in the query, and the parameter is replaced with equivalent meaning.

[0104] In summary, the present disclosure proposes a memory mechanism in the LLM plug-in scenario, which can store and analyze high-quality feedback from users, and can quickly decompose, identify and locate new intentions of users. The technical fields involved in the present disclosure include natural language processing technologies such as word segmentation and part-of-speech analysis, embedding vector calculation and storage technology, etc. The present disclosure can be applied to question-and-answer services based on large models, and participate in the core intent recognition process of the plug-in engine in the question-and-answer service. The present disclosure proposes a memory mechanism based on user feedback and interaction history, and provides a new intent recognition recall link, which greatly improves the response efficiency in actual user usage scenarios, solves the problem that similar problems in high-difficulty intent recognition scenarios are difficult to handle, and greatly improves the user experience.

[0105] The disclosure assists in the intention recognition scenario of LLM, improves the accuracy and efficiency of the overall process, effectively solves the consistency problem and feedback timeliness problem in the user's use process, and greatly improves the user's experience in the use process. In specific practice, the disclosure proposes a memory mechanism to record the questions asked by users and the corresponding matched intentions and parameters, as well as user feedback, and efficiently collect historical data. According to the configurable judgment rules, the memory mechanism stores the credible historical data after word segmentation, part-of-speech tagging, embedding vectorization, as the "memory" of the LLM application, and is applied to the subsequent user question processing process. When there is available memory, the memory mechanism processes and preprocesses the user's question with the corresponding natural language technology, and performs semantic comparison and credibility verification with the existing memory. If there is a matching memory, the intent and parameters corresponding to the matching memory are returned to the user as the result of this intent recognition and parameter extraction.

[0106] The memory mechanism proposed in the present disclosure extracts credible conclusions by processing user feedback, and participates in the construction of the LLM application loop based on existing credible data. The corresponding results can ensure accuracy and can be judged quickly, which can effectively improve the accuracy and efficiency of LLM application intent recognition and parameter extraction. In addition, the memory mechanism proposed in the present disclosure instantly and efficiently collects users' real-time conversations and feedback data, and uses the processed results for the user's next question, solving the timeliness problem of feedback during user use. The memory mechanism proposed in the present disclosure can ensure that when a user asks a very similar question, if the previous question has been marked as a credible memory, the new question can also get a reliable result, solving the consistency problem.

[0107] The plug-in calling method disclosed in the present invention may also be implemented as a plug-in calling device. Figure 61 is a schematic diagram showing the structure of a plug-in calling device according to an embodiment of the present disclosure. The functional units of the plug-in calling device can be implemented by hardware, software, or a combination of hardware and software that implements the principles of the present disclosure. It can be understood by those skilled in the art that Figure 6 The functional units described can be combined or divided into sub-units to implement the principles of the above invention. Therefore, the description herein can support any possible combination, division, or further limitation of the functional units described herein. The following is a brief description of the functional units that the plug-in calling device can have and the operations that each functional unit can perform. For the details involved, please refer to the relevant description above, which will not be repeated here.

[0108] See also Figure 6 The plug-in calling device 600 may include a determination module 610 and a processing module 620 .

[0109] The judgment module 610 is used to respond to the user's first question information and judge whether there is target memory data matching the first question information in the database, and the database is used to store multiple memory data, including historical question information and second plug-in call information corresponding to the historical question information.

[0110] The processing module 620 is configured to determine first plug-in calling information corresponding to the first question information based on the target memory data if the target memory data exists in the database.

[0111] In some embodiments, the plug-in calling device 600 may further include a reading module. The reading module is used to select part of the memory data from the database for the first question information to obtain a candidate memory data set. Exemplarily, the reading module may select memory data that meets the following conditions from the database: the historical question information corresponds to the same organization as the first question information; and / or the historical question information corresponds to the same enabled plug-in as the first question information. The judgment module 610 may judge whether there is target memory data matching the first question information in the candidate memory data set.

[0112] In some embodiments, the judgment module 610 may include a first judgment module and a first determination module. The first judgment module is used to determine whether there is a second question information that is the same as the first question information in the database. The first determination module is used to use the memory data corresponding to the second question information as the target memory data if the second question information exists in the database. The processing module 620 can use the second plug-in call information corresponding to the second question information in the target memory data as the first plug-in call information corresponding to the first question information.

[0113] The judgment module 610 may also include a second judgment module, a calculation module, and a second determination module. The second judgment module is used to determine whether there is a third question information in the database whose word segmentation result is the same as the first question information if the second question information does not exist in the database, and the word segmentation result is used to characterize the part of speech of each word segmentation obtained after the question information is segmented. The calculation module is used to calculate the similarity between the first question information and the third question information if the third question information exists in the database. The second determination module is used to take the memory data corresponding to the third question information whose similarity is greater than or equal to the first threshold and whose similarity ranking is the highest as the target memory data. Exemplarily, the calculation module may include a first acquisition module, a second acquisition module, and a calculation submodule. The first acquisition module is used to acquire a first vectorized representation of the first question information, and the first vectorized representation is used to characterize the vectorized representation of the sentence after at least part of the words in the first question information are replaced with equivalent meanings. The second acquisition module is used to acquire a second vectorized representation of the third question information, and the second vectorized representation is used to characterize the vectorized representation of the sentence after at least part of the words in the three question information are replaced with equivalent meanings. The calculation submodule is used to calculate the similarity between the first vectorized representation and the second vectorized representation. The second determination module may use the plug-in distribution result in the second plug-in call information of the target memory data as the plug-in distribution result corresponding to the first question information; and / or the second determination module may perform parameter extraction on the corresponding word segmentation part in the first question information based on the word segmentation part corresponding to each plug-in parameter in the parameter extraction result in the second plug-in call information of the target memory data, to obtain the parameter extraction result corresponding to the first question information.

[0114] In some embodiments, the plug-in calling device 600 may further include a distribution and parameter extraction module and / or a response module. The distribution and parameter extraction module is used to determine the first plug-in calling information through the plug-in distribution and / or parameter extraction process if there is no target memory data matching the first question information. The response module is used to call the plug-in based on the first plug-in calling information, and generate response information for the first question information based on the plug-in calling result.

[0115] In some embodiments, the plug-in calling device 600 may further include a third acquisition module and a generation module. The third acquisition module is used to obtain the user's feedback result on the response information. The generation module is used to generate memory data based on the first question information and the first plug-in calling information and store it in the database if the feedback result shows that the user is satisfied with the response information.

[0116] Figure 7 A schematic diagram of the structure of a computing device that can be used to implement the above-mentioned plug-in calling method according to an embodiment of the present disclosure is shown.

[0117] See also Figure 7, the computing device 700 includes a memory 710 and a processor 720 .

[0118] The processor 720 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 720 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, the processor 720 may be implemented using a customized circuit, such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0119] The memory 710 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 720 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 710 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 710 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0120] The memory 710 stores executable codes. When the executable codes are processed by the processor 720, the processor 720 can execute the plug-in calling method mentioned above.

[0121] The plug-in calling method, apparatus and device according to the present disclosure have been described above in detail with reference to the accompanying drawings.

[0122] In addition, the method according to the present disclosure may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present disclosure.

[0123] Alternatively, the present disclosure may also be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes the various steps of the above-mentioned method according to the present disclosure.

[0124] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both.

[0125] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the system and method according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0126] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A plug-in calling method, characterized in that: include: In response to a first question information of a user, determining whether there is target memory data matching the first question information in a database, the database being used to store a plurality of memory data, the memory data including historical question information and second plug-in calling information corresponding to the historical question information; If the target memory data exists in the database, first plug-in calling information corresponding to the first question information is determined based on the target memory data.

2. The method according to claim 1, characterized in that Before determining whether there is target memory data matching the first question information in the database, the method further includes: Selecting part of the memory data from the database according to the first question information to obtain a candidate memory data set; Determining whether there is target memory data matching the first question information in the database includes: determining whether there is target memory data matching the first question information in the candidate memory data set.

3. The method according to claim 2, characterized in that Selecting part of the memory data from the database according to the first question information includes: selecting the memory data that meets the following conditions from the database: The historical question information and the first question information correspond to the same organization; and / or The historical question information and the first question information correspond to the same enabled plug-in.

4. The method according to claim 1, characterized in that: Determining whether there is target memory data matching the first question information in the database includes: Determine whether there is a second question information identical to the first question information in the database, If the second question information exists in the database, the memory data corresponding to the second question information is used as the target memory data.

5. The method according to claim 4, characterized in that Determining whether there is target memory data matching the first question information in the database further includes: If the second question information does not exist in the database, determining whether there is third question information in the database whose word segmentation result is the same as that of the first question information, wherein the word segmentation result is used to represent the part of speech of each word segmented after the question information is segmented; If the third question information exists in the database, calculating the similarity between the first question information and the third question information; The memory data corresponding to the third question information whose similarity is greater than or equal to the first threshold and whose similarity ranking is the highest is used as the target memory data.

6. The method according to claim 5, characterized in that Calculating the similarity between the first question information and the third question information includes: Acquire a first vectorized representation of the first question information, where the first vectorized representation is used to represent a vectorized representation of a sentence after equivalently replacing at least part of the words in the first question information; Acquire a second vectorized representation of the third question information, where the second vectorized representation is used to represent a vectorized representation of a sentence after equivalently replacing at least part of the words in the third question information; A similarity between the first vectorized representation and the second vectorized representation is calculated.

7. The method according to claim 5, characterized in that Determining first plug-in calling information corresponding to the first question information based on the target memory data includes: using the plug-in distribution result in the second plug-in calling information of the target memory data as the plug-in distribution result corresponding to the first question information; and / or Based on the word segmentation parts corresponding to each plug-in parameter in the parameter extraction result in the second plug-in call information of the target memory data, parameter extraction is performed on the corresponding word segmentation parts in the first question information to obtain a parameter extraction result corresponding to the first question information.

8. A plug-in calling device, characterized in that: include: a judgment module, configured to judge, in response to a first question information of a user, whether there is target memory data matching the first question information in a database, wherein the database is configured to store a plurality of memory data, the memory data including historical question information and second plug-in calling information corresponding to the historical question information; A processing module is used to determine first plug-in calling information corresponding to the first question information based on the target memory data if the target memory data exists in the database.

9. A computing device comprising: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 7.

10. A non-transitory machine-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Intelligent session method and device

    CN112069830A

  • Knowledge base optimization method and device, electronic equipment and storage medium

    CN115048501A

  • Intelligent customer service response method and system based on automobile part field and storage medium

    CN117291616A

  • User intention response method, device and equipment of intelligent robot and storage medium

    CN118171658A

  • Interaction data processing method and device, equipment, storage medium and program product

    CN118410137A

Cited By

  • Dialogue management method, electronic equipment and computer readable storage medium

    CN121581067A

  • A dialogue management method, an electronic device and a computer readable storage medium

    CN121581067B