A large model-based plug-in calling method, an electronic device, and a storage medium

Abstract

The application relates to the technical field of large language models, in particular to a large-model-based plug-in calling method, an electronic device and a storage medium, the method comprises the following steps: matching a preset plug-in keyword in a preset plug-in keyword library with any request keyword in a target request text input by a user, directly determining a target plug-in according to the preset plug-in matched or determining a plurality of alternative plug-ins, when the request keyword is not matched with the preset plug-in, taking all the preset plug-ins as the alternative plug-ins, screening the target plug-in from the alternative plug-ins by using a given large language model, extracting target parameter information from the target request text and sending the target parameter information to the target plug-in to obtain a plug-in feedback result, and taking the plug-in feedback result as a request feedback text when the judgment result of the plug-in feedback result meets preset requirements; the application can improve the accuracy of target plug-in selection in a dialogue system, and the selected target plug-in can meet the requirements of the user.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language model, in particular to a plug-in calling method based on a large model, an electronic device and a storage medium. BACKGROUND

[0002] The large language model is a language model based on deep learning, which is constructed through the training of a large corpus and can automatically learn language rules and generate text. After a complete process of pre-training, fine-tuning and reinforcement learning, the large language model has very strong capabilities and has been used in human-computer dialogue systems. However, it is found in practice that the large language model still has some weaknesses. Since the request problem input by the user involves content in multiple fields and various demand scenarios, such as obtaining real-time or future weather information in a certain city, performing complex mathematical operations, generating formulas, drawing charts for display, and filtering network sources, etc., the dialogue system has low accuracy in returning results when encountering these demand scenarios, which affects the user's experience. SUMMARY

[0003] To solve the above technical problems, the present application provides a plug-in calling method based on a large model, an electronic device and a storage medium, which can improve the accuracy of selecting target plug-ins in the dialogue system and make the selected target plug-ins more in line with the user's needs.

[0004] According to a first aspect of the present application, a plug-in calling method based on a large model is provided, comprising the following steps:

[0005] Receiving a target request text input by a user to a given large language model, and extracting a plurality of request keywords from the target request text.

[0006] Matching each request keyword with a preset plug-in keyword in a preset plug-in keyword library. Based on the keyword matching result, when the number of matched preset plug-ins is 1, the matched preset plug-in is selected as the target plug-in; when the number of matched preset plug-ins is greater than 1, the selected plug-in corresponding to each matched request keyword is determined as a candidate plug-in; and when no preset plug-in is matched, all preset plug-ins are selected as candidate plug-ins.

[0007] Performing semantic analysis on the target request text, the plug-in description text corresponding to each candidate plug-in and the input parameter description text by the given large language model, and selecting the target plug-in from the plurality of candidate plug-ins according to the semantic analysis result, so as to call the selected target plug-in.

[0008] According to the input parameter description text corresponding to each target plug-in of the call, the target parameter information is extracted from the target request text by the given large language model, and the target parameter information is converted into the preset parameter format of the corresponding target plug-in and input into the corresponding target plug-in to obtain a plug-in feedback result.

[0009] The plug-in feedback result is input into the given large language model for judgment, and when the judgment result of the plug-in feedback result meets the preset requirement, the plug-in feedback result is taken as a request feedback text and output.

[0010] According to a second aspect of the present application, a non-transitory computer readable storage medium is provided, the storage medium stores at least one instruction or at least one program, the at least one instruction or the at least one program is loaded and executed by a processor to realize the above-mentioned large model-based plug-in calling method.

[0011] According to a third aspect of the present application, an electronic device is provided, comprising a processor and the above-mentioned non-transitory computer readable storage medium.

[0012] The present application has at least the following beneficial effects:

[0013] The present application provides a large model-based plug-in calling method, first, the preset plug-in keywords in the preset plug-in keyword library are matched with any request keyword in the target request text input by the user, when there is no matching request keyword, all preset plug-ins are taken as alternative plug-ins, when there is only one matching preset plug-in, the preset plug-in is taken as a target plug-in, when the number of matching preset plug-ins is multiple, all matching preset plug-ins are taken as alternative plug-ins, then for several alternative plug-ins, the large language model performs semantic analysis on the target request text, the plug-in description text and the input parameter description text corresponding to each alternative plug-in, and according to the semantic analysis result, the target plug-in is screened out, so as to call the screened target plug-in, and through the content of the target request text and the input parameter description of the target plug-in, further screening of several target plug-ins is realized, which can improve the accuracy of target plug-in selection in the dialogue system, and the selected target plug-in is more in line with the user's demand. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0015] Figure 1 The flowchart of the large model-based plug-in calling method provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present application will be apparently and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person skilled in the art without creative work belong to the protection scope of the present application.

[0017] The embodiments of the present application provide a plug-in calling method based on a large model, as shown in Figure 1 The method comprises the following steps:

[0018] S100, receiving a target request text input by a user to a given large language model, and extracting a plurality of request keywords from the target request text. For example, a keyword extraction tool is used to extract request keywords from the target request text.

[0019] Specifically, the target request text comprises a plurality of rounds of dialogue information text input by the user to the dialogue system; it can be understood that the target request text comprises but is not limited to the current round of information text input by the user to the dialogue system. In a specific implementation, the sensitivity of the request text input by the user needs to be judged and whether the sentence meaning is clear needs to be judged. When the request text involves sensitive words, the model directly replies politely; when the sentence meaning of the request text is not clear, for example, the request text is "what information is contained herein", for multi-round dialogue, the context text needs to be obtained, and the given large language model is used to rewrite complete information. For single-round dialogue or multi-round dialogue that cannot rewrite complete information through context, the given large language model outputs a preset reply text, for example, "please further supplement information", to obtain a complete question with clear meaning. In judging whether the sentence meaning of the request text is clear, one embodiment is to detect whether the request text contains specified words, such as "it", "he", "this", "that", etc. If so, it is judged that the sentence meaning of the request text is not clear. Another embodiment is to combine the questions in the actual application scenario to annotate whether the meaning is complete and then train a binary classification model to obtain a better judgment result with high accuracy and generalization.

[0020] In one embodiment, the step S100 further comprises the following steps before the step S100:

[0021] S101, receiving an initial request text input by a user to a given large language model, and extracting a plurality of initial keywords from the initial request text.

[0022] S102, when there is an initial keyword corresponding to any preset first type keyword, intercepting and processing the initial request text; it can be understood that the first type keyword refers to a preset keyword that must be intercepted. For example, some high-sensitivity words.

[0023] Specifically, the risk levels of the first type keyword, the second type keyword, and the third type keyword decrease in turn.

[0024] S103, when there is an initial keyword corresponding to any preset second type keyword, obtaining a detection result of the initial request text by a preset anomaly detection model, when the detection result is abnormal, intercepting and processing the initial request text, when the detection result is non-abnormal but the confidence of the detection result is less than a first preset threshold, intercepting and processing the initial request text; it can be understood that the anomaly detection model is a binary classification model trained in advance to evaluate whether the text is an abnormal text. For example, when detecting the initial keyword in the initial request text, if the risk of the initial keyword exceeds a preset degree, it is considered as an abnormal text, otherwise it is considered as a non-abnormal text.

[0025] Further, the first preset threshold is set by the person skilled in the art according to actual needs, for example, 80%.

[0026] S104, when there is an initial keyword corresponding to any preset third type keyword, obtaining a detection result of the initial request text by the anomaly detection model, when the detection result is abnormal and the confidence of the detection result is greater than a second preset threshold, intercepting and processing the initial request text.

[0027] Further, the second preset threshold is set by the person skilled in the art according to actual needs, for example, 95%.

[0028] In the above, when receiving any request text sent by a user, the risk of the request text needs to be evaluated and screened first, when the risk degree is too high, the request text is intercepted, when the risk degree is not high, the detection result of the anomaly detection model needs to be considered comprehensively, and the final detection result is determined by voting, which improves the rationality of judging whether the text is abnormal. The text that passes the detection is taken as a target request text and is subjected to subsequent selection and calling of plug-ins.

[0029] S200, match each request keyword with a preset plug-in keyword in a preset plug-in keyword library, and based on the keyword matching result, when the number of matched preset plug-ins is 1, the matched preset plug-in is taken as the target plug-in; when the number of matched preset plug-ins is greater than 1, the candidate plug-in corresponding to each matched request keyword is determined; when no preset plug-in is matched, all preset plug-ins are taken as candidate plug-ins; it can be understood that: in the process of determining the candidate plug-in corresponding to each matched request keyword, a plurality of target plug-ins corresponding to the matched request keyword are determined from the preset word correspondence table.

[0030] Specifically, the preset word correspondence table includes a plurality of preset plug-ins and a plurality of plug-in keywords corresponding to each preset plug-in. Wherein, the same plug-in keyword can also correspond to a plurality of preset plug-ins. The following is illustrated: when the target request text involves keywords corresponding to calculation problems, the preset plug-in is a calculator, a python executor, etc.; when the target request text involves keywords corresponding to formula writing, the preset plug-in is a math tool, etc.; when the target request text involves real-time information data, the preset plug-in is an information media plug-in, etc.; when the target request text involves keywords related to the drawing of statistical pie charts, column charts, flowcharts, the preset plug-in is a drawing tool, etc.; when the target request text involves weather or earthquake related messages, the preset plug-in is a weather tool. In specific implementation, one of the tools can be used according to actual needs, or multiple tools can be used in parallel or in series, or customized selection can be made according to project requirements.

[0031] It also needs to be explained that for the target request text containing not less than 1 question, the target request text needs to be disassembled and the plug-in is called in steps, for example, the target request text input by the user is: please help me search for the recent stock price of Maotai and draw a line chart. It needs to be split into two steps: a. search for the recent stock price of Maotai, b. draw a line chart, and the plug-ins called respectively are search tool and statistical chart drawing tool.

[0032] Further, the method further screens the candidate plug-in corresponding to each matched request keyword by the following steps:

[0033] S201, for any request keyword matched, when the number of preset plug-ins corresponding to the request keyword is 1, the preset plug-in corresponding to the request keyword is determined as the candidate plug-in.

[0034] S202, when the number of preset plug-ins corresponding to the request keyword is greater than 1, the preset interface stability score of each preset plug-in corresponding to the request keyword and the number of request keywords corresponding to each preset plug-in are obtained; for example, when the request keyword is calculation, there are a plurality of tools that can realize the calculation function corresponding to the preset plug-in.

[0035] Further, the preset interface stability score of the preset plug-in refers to a human score obtained according to an API test result of the preset plug-in. The API test includes correctness test and stability test of the API interface.

[0036] S203, the preset plug-in corresponding to the maximum number of request keywords and the preset plug-in corresponding to the second maximum number of request keywords are both determined as the key plug-in.

[0037] S204, when D max / D min <η, the preset plug-in corresponding to D max is determined as the candidate plug-in, wherein D max is a numerical value corresponding to the maximum number of request keywords, D min is a numerical value corresponding to the second maximum number of request keywords, and η is a preset proportion threshold. For example, η can be set to 2 / 3.

[0038] S205, when D max / D min ≥η, a target score corresponding to each preset plug-in is obtained by calculating the weighted sum according to the preset scoring weight, the quantity weight, the preset interface stability score of each preset plug-in corresponding to the request keyword, and the number of request keywords corresponding to each preset plug-in, and the preset plug-in corresponding to the maximum target score is taken as the candidate plug-in. In specific implementation, since the number of keywords corresponding to the preset plug-in is generally a single digit, the preset interface stability score can be normalized to between 1 and 5.

[0039] As described above, first, the number of matched keywords is determined, when no request keyword matches, all preset plug-ins are taken as candidate plug-ins, when a request keyword matches, according to a plurality of preset plug-ins corresponding to one request keyword, the interface stability score of each preset plug-in and the number of corresponding request keywords are comprehensively considered to select one candidate plug-in corresponding to one request keyword, in this way, the accuracy of selecting the candidate plug-in is improved, and the selected candidate plug-in is more in line with the user's demand in probability.

[0040] S300, the target request text, the plug-in description text and the input parameter description text corresponding to each candidate plug-in are subjected to semantic analysis by a given large language model, and a target plug-in is selected from a plurality of candidate plug-ins according to the semantic analysis result, so as to call the selected target plug-in; it can be understood that the plug-in description text is a plug-in introduction text generated according to the function range text and the use method text of the plug-in.

[0041] Specifically, the input parameter description text corresponding to the alternative plug-in refers to the text combined by the input parameter structure description text and the parameter format description text extracted from the preset parameter information required by the alternative plug-in. For example, the input parameter structure description text includes optional parameter and mandatory parameter descriptions; the parameter format description text includes the format requirements of each parameter, for example, the date type parameter needs to specify whether the specific format is month, day, or needs to be time, minute, and second, etc.

[0042] After obtaining the alternative plug-in, the above-obtained several alternative plug-ins need to be further screened. Through semantic analysis of the target request text content by the large language model and semantic analysis of the plug-in description text and the input parameter description text, the final required target plug-in is screened out by comparing whether the semantics are consistent, thereby improving the accuracy and reliability of the plug-in call.

[0043] S400, according to the input parameter description text corresponding to each target plug-in, the target parameter information is extracted from the target request text by the given large language model, and the target parameter information is converted according to the preset parameter format of the corresponding target plug-in and input into the corresponding target plug-in to obtain the plug-in feedback result. For example, including but not limited to outputting the obtained target parameter information in JSON format.

[0044] In one specific embodiment, the S400 step further includes the following steps:

[0045] S401, according to the parameter format description text corresponding to the target plug-in, the parameter format of the target parameter information converted according to the preset parameter format is verified.

[0046] Further, the target parameter information corresponding to the target plug-in refers to the corresponding parameter information extracted from the target request text according to the input parameter description text of the target plug-in.

[0047] S402, when the verification is consistent, the converted parameter is input into the corresponding target plug-in through the corresponding API interface to obtain the output result of the target plug-in.

[0048] S403, when the verification is inconsistent, the obtained parameter format is corrected according to the parameter format description text corresponding to the target plug-in to obtain a new parameter, and the new parameter is input into the corresponding target plug-in through the corresponding API interface to obtain the output result of the target plug-in.

[0049] S404, the output results of all target plug-ins are combined to obtain the plug-in feedback result; it can be understood that: according to the output result of each target plug-in, all output results are post-processed and sent to the large language processing model.

[0050] The above checks the format of the parameter before inputting the target parameter information into the target plug-in, prevents the case that the target plug-in cannot be successfully input or the target plug-in cannot output accurate results due to format problems, and ensures normal use of the target plug-in and accurate output of data.

[0051] S500, input the plug-in feedback result into the given large language model for judgment, and when the judgment result of the plug-in feedback result meets the preset requirement, take the plug-in feedback result as the request feedback text and output; it can be understood that: the given large language model judges whether the plug-in feedback result can answer the user question through the reflection function, and when it can answer, the large language model sorts and organizes the combined result, that is, the feedback results of multiple plug-ins are written into the prompt together to let the given large language model make the final output to obtain the feedback result of the current round question, and the feedback result is displayed on the interface of the dialogue system. In another embodiment, the target request text and the plug-in feedback result corresponding thereto can also be sent to the base model, and whether the plug-in feedback result can answer the user question is judged through the combination of the base model and the answerable prompt. When the base model cannot judge, the instruction data for plug-in selection is constructed as a training set and a test set to fine-tune and iterate the base model, wherein each piece of instruction data constructed is the request prompt actually input into the given large language model in actual use and the ideal output result, for example, the output result at least contains the selected plug-in and plug-in parameters; in a specific implementation, the plug-in parameters need to be defined clearly in combination with the characteristics of each plug-in, such as a search tool, which tends to search with keywords and expand as many questions as possible.

[0052] Further, the method further comprises the following steps:

[0053] S001, calculate the text similarity between the received target request text and each historical request text in the historical plug-in selection library; it can be understood that: the plug-in selection result of the historical request question is stored in the historical plug-in selection library.

[0054] S002, when the maximum text similarity exceeds the preset similarity threshold, send the historical plug-in selection result corresponding to the maximum text similarity to the given large language model, so that the given large language model determines the plug-in corresponding to the historical plug-in selection result as the target plug-in; it can be understood that: when the maximum text similarity exceeds the preset similarity threshold, it indicates that the current request question has completed plug-in selection, and the selection result is introduced into the plug-in selection prompt for few shot to guide the model to preferentially select this result.

[0055] Further, the method further comprises the following steps:

[0056] S10, when the judgment result of the plug-in feedback result is not in line with the preset requirement, inputting each preset plug-in corresponding training data set to the given large language model to obtain a model output result; the training data set refers to a data set obtained after sample expansion and prompt word addition operation; it can be understood that: increasing sample data in actual business scenarios, and adding prompt words in different business fields.

[0057] S20, adjusting the given large language model according to the model output result and the real result in the training data set to obtain a trained target large language model; those skilled in the art know the specific process of training the large language model according to the data set, which will not be repeated here.

[0058] S30, taking the target large language model as a new given large language model, and returning to execute the step of performing semantic analysis on the target request text and the plug-in description text and the input parameter description text corresponding to each candidate plug-in by the given large language model, and screening the target plug-in from the several candidate plug-ins according to the semantic analysis result until the judgment result of the plug-in feedback result is in line with the preset requirement.

[0059] The above, when the obtained request feedback result does not meet the requirement, it indicates that the accuracy of the large language model is not enough, so the sample data in the actual scene needs to be increased, and the large language model is further trained through sample expansion to update the large language model and improve the accuracy of the model output result.

[0060] In summary, the application provides a plug-in calling method based on a large model, first, the preset plug-in keywords in the preset plug-in keyword library are matched with any request keyword in the target request text input by the user, when there is no matching request keyword, all preset plug-ins are used as candidate plug-ins, when there is only one matching preset plug-in, the preset plug-in is used as the target plug-in, when the number of matching preset plug-ins is more than one, all matching preset plug-ins are used as candidate plug-ins, then for several candidate plug-ins, the large language model performs semantic analysis on the target request text and the plug-in description text and the input parameter description text corresponding to each candidate plug-in, and the target plug-in is screened according to the semantic analysis result, so as to call the screened target plug-in, further screen several target plug-ins according to the content of the target request text and the input parameter description of the target plug-in, which can improve the accuracy of selecting the target plug-in in the dialogue system, and the selected target plug-in is more in line with the user's demand.

[0061] The embodiment of the present application further provides a non-transitory computer readable storage medium, which can be arranged in an electronic device to store at least one instruction or at least one program related to a method in the method embodiment, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided by the above embodiment.

[0062] The embodiment of the present application further provides an electronic device, comprising a processor and the aforementioned non-transitory computer readable storage medium.

[0063] Although some specific embodiments of the present application have been described in detail by examples, those skilled in the art should understand that the above examples are only for illustration, but not for limiting the scope of the present application. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.

Claims

1. A plugin invocation method based on a large model, characterized in that, The method includes the following steps: Receive the target request text input by the user into a given large language model, and extract several request keywords from the target request text; Each request keyword is matched with the preset plugin keywords in the preset plugin keyword library. Based on the keyword matching results, when the number of matched preset plugins is 1, the matched preset plugin is used as the target plugin; when the number of matched preset plugins is greater than 1, the candidate plugins corresponding to each matched request keyword are determined; when no preset plugin is matched, all preset plugins are used as candidate plugins. Semantic analysis is performed on the target request text and the plugin description text and input parameter description text corresponding to each candidate plugin by using a given large language model. Based on the semantic analysis results, the target plugin is selected from several candidate plugins and then invoked. Based on the input parameter description text corresponding to each target plugin, target parameter information is extracted from the target request text using a given large language model. The target parameter information is then converted according to the preset parameter format of the corresponding target plugin and input into the corresponding target plugin to obtain the plugin feedback result. The plugin feedback result is input into a given large language model for judgment. When the judgment result of the plugin feedback result meets the preset requirements, the plugin feedback result is output as a request feedback text. Before receiving the target request text input by the user into a given large language model as described in step [1], the method further includes the following steps: Iterate through several initial keywords in the initial request text input by the user into the given large language model. When there is an initial keyword that corresponds to any preset first type keyword, the initial request text is intercepted. When an initial keyword exists that corresponds to any preset second-type keyword or only to any preset third-type keyword, the detection result of the anomaly detection model on the initial request text is obtained. When the detection result meets the preset interception conditions corresponding to the second-type keyword itself or the third-type keyword itself, the initial request text is intercepted.

2. The plugin invocation method based on a large model according to claim 1, characterized in that, The method also determines the candidate plugins corresponding to each matched request keyword through the following steps: For any matching request keyword, if the number of preset plugins corresponding to the request keyword is 1, the preset plugin corresponding to the request keyword will be determined as a candidate plugin. When the number of preset plugins corresponding to the request keyword is greater than 1, obtain the preset interface stability score of each preset plugin corresponding to the request keyword and the number of request keywords corresponding to each preset plugin. Both the preset plugin corresponding to the highest number of requested keywords and the preset plugin corresponding to the second highest number of requested keywords are identified as key plugins. When D max / D min When <η, D max The corresponding preset plugins were determined as candidate plugins, among which, D max D represents the numerical value corresponding to the maximum number of requested keywords. min η is the value corresponding to the number of keywords in the second most requested request, and η is the preset ratio threshold. When D max / D min When ≥η, the target score for each preset plugin is calculated by weighting the preset scoring weight, quantity weight, preset interface stability score of each preset plugin corresponding to the request keyword, and the number of request keywords corresponding to each preset plugin. The preset plugin with the maximum target score is then selected as the alternative plugin.

3. The plugin invocation method based on a large model according to claim 1, characterized in that, The input parameter description text corresponding to the alternative plugin refers to the text that is a combination of input parameter structure description text and parameter format description text extracted from the preset parameter information required by the alternative plugin.

4. The plugin invocation method based on a large model according to claim 3, characterized in that, The process of converting the target parameter information according to the preset parameter format of the corresponding target plugin and inputting it into the corresponding target plugin to obtain the plugin feedback result includes the following steps: The target parameter information is validated according to the parameter format description text corresponding to the target plugin, and the parameter format is converted according to the preset parameter format. When the verification is successful, the converted parameters are input into the target plugin through the corresponding API interface to obtain the output result of the target plugin; When the verification is inconsistent, the obtained parameter format is corrected according to the parameter format description text corresponding to the target plugin to obtain new parameters. The new parameters are then input into the target plugin through the corresponding API interface to obtain the output result of the target plugin. The outputs of all target plugins are combined and processed to obtain the plugin feedback results.

5. The plugin invocation method based on a large model according to claim 1, characterized in that, The method further includes the following steps: Calculate the text similarity between the received target request text and each historical request text in the historical plugin selection library; When the maximum text similarity exceeds a preset similarity threshold, the historical plugin selection result corresponding to the maximum text similarity is sent to the given large language model so that the given large language model determines the plugin corresponding to the historical plugin selection result as the target plugin.

6. The plugin invocation method based on a large model according to claim 1, characterized in that, The method further includes the following steps: When the feedback result of the plugin does not meet the preset requirements, the training dataset corresponding to each preset plugin is constructed and input into the given large language model to obtain the model output result; the training dataset refers to the dataset obtained after expanding the input of samples and adding prompt words; Based on the model output and the real results in the training dataset, the given large language model is adjusted to obtain the trained target large language model. The target large language model is used as the new given large language model, and the execution step is returned to perform semantic analysis on the target request text and the plugin description text and input parameter description text corresponding to each candidate plugin through the given large language model. Based on the semantic analysis results, the target plugin is selected from several candidate plugins until the judgment result of the plugin feedback result meets the preset requirements.

7. A non-transitory computer-readable storage medium, wherein the storage medium stores at least one instruction or at least one program segment, characterized in that, The at least one instruction or the at least one program segment is loaded and executed by the processor to implement the large model-based plug-in invocation method as described in any one of claims 1-6.

8. An electronic device, characterized in that, Includes a processor and the non-transitory computer-readable storage medium as described in claim 7.

Images