Language model-based dialogue system root cause positioning method and related device

Abstract

The invention provides a root cause positioning method of a dialogue system based on a language model and a related device, the dialogue system comprises a plurality of modules, the plurality of modules respectively provide different information processing functions, and the method comprises the following steps: obtaining sample data of the dialogue system, the sample data comprises first input information, first reply information corresponding to the first input information and a first link log of the first reply information generated by the dialogue system, and the first reply information meets a negative sample condition; for each module in at least part of modules in the plurality of modules, executing the following steps: determining a root cause analysis result of the module according to at least one of first input information, first reply information and a target link log related to the module in the first link log, the root cause analysis result is used for representing whether the information processing function provided by the module has defects or not. According to the method, a specific function with defects in the dialogue system can be accurately positioned, and root cause positioning of module granularity is realized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a root cause localization method, device, electronic device, computer-readable storage medium, and computer program product for a language model-based dialogue system. Background Art

[0002] With the continuous development of computer technology, especially language modeling, dialogue systems based on language models have emerged. These systems, also known as conversational robots or digital assistants, provide intelligent conversational services, allowing users to interact with them through human-computer dialogue.

[0003] In practical applications, a dialogue system can include multiple modules that work together to provide intelligent dialogue services. This type of dialogue system can also be referred to as a multi-module dialogue system. For example, a dialogue system can include an intent distribution module, an online search module, and a programming module. For a piece of input information, the intent distribution module identifies the intent, which is then processed by the online search module or the programming module to generate a response message. Each module in the dialogue system can implement its corresponding information processing function by invoking a language model.

[0004] To improve the quality of intelligent conversations, conversation system service providers need to test and diagnose the accuracy of the conversation system's questions and answers. Related technologies typically combine input information to determine whether the conversation system's response meets user needs. However, these methods struggle to pinpoint specific root causes and lack fine-grained module diagnostic capabilities. Summary of the Invention

[0005] This application provides a method for locating the root cause of a dialog system based on a language model. This method can accurately locate a specific function with a defect in the dialog system, achieving root cause location at a module-level granularity. This application also provides a device, electronic device, computer-readable storage medium, and computer program product corresponding to the above method.

[0006] In a first aspect, the present application provides a root cause location method for a dialogue system based on a language model, wherein the dialogue system is configured to generate corresponding reply information based on the language model for input information, the dialogue system comprising multiple modules, each of which provides a different information processing function. The method comprises:

[0007] Obtaining sample data of the dialogue system; wherein the sample data includes: first input information, first reply information corresponding to the first input information, and a first link log of the dialogue system generating the first reply information, wherein the first reply information satisfies a negative sample condition;

[0008] For each module of at least some of the multiple modules, the following steps are performed: determining a root cause analysis result of the module based on the first input information, the first reply information, and at least one item in the target link log related to the module in the first link log; wherein the root cause analysis result is used to characterize whether there is a defect in the information processing function provided by the module.

[0009] In a second aspect, the present application provides a root cause location device for a dialogue system based on a language model, wherein the dialogue system is configured to generate corresponding reply information based on the language model for input information, the dialogue system comprising multiple modules, each of which provides a different information processing function. The device comprises:

[0010] an acquisition module, configured to acquire sample data of the dialogue system; wherein the sample data includes: a first input message, a first reply message corresponding to the first input message, and a first link log of the dialogue system generating the first reply message, wherein the first reply message satisfies a negative sample condition;

[0011] An analysis module is configured to perform the following steps for each module of at least some of the multiple modules: determining a root cause analysis result of the module based on the first input information, the first reply information, and at least one item in the target link log related to the module in the first link log; wherein the root cause analysis result is used to characterize whether there is a defect in the information processing function provided by the module.

[0012] In a third aspect, the present application provides an electronic device comprising a processor and a memory. The processor and the memory communicate with each other. The processor is configured to execute instructions stored in the memory to cause the electronic device to perform the root cause localization method for a language model-based dialogue system as described in the first aspect or any implementation of the first aspect.

[0013] In a fourth aspect, the present application provides a computer-readable storage medium, which stores instructions, and the instructions instruct an electronic device to execute the root cause localization method of a language model-based dialogue system in the above-mentioned first aspect or any implementation of the first aspect.

[0014] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the root cause localization method for a language model-based dialogue system in the above-mentioned first aspect or any one of the implementations of the first aspect.

[0015] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods.

[0016] It can be seen from the above technical solutions that this application has the following advantages:

[0017] The present application provides a root cause location method for a dialogue system based on a language model, wherein the dialogue system is used to generate corresponding reply information based on a language model for input information, and the dialogue system includes multiple modules, each of which provides different information processing functions. The method first obtains sample data of the dialogue system, wherein the sample data includes: first input information, first reply information corresponding to the first input information, and a first link log in which the dialogue system generates the first reply information, and the first reply information satisfies a negative sample condition. For each module in at least some of the multiple modules, the following steps are performed: based on the first input information, the first reply information, and at least one item in the target link log related to the module in the first link log, a root cause analysis result of the module is determined; wherein the root cause analysis result is used to characterize whether there is a defect in the information processing function provided by the module.

[0018] This method diagnoses the information processing functions provided by each module in a multi-module dialogue system. For each module, this method leverages the dialogue system's input and output information and the link logs associated with each module to conduct a targeted analysis of its information processing function, resulting in module-level root cause identification. Because each module is responsible for a specific function within the dialogue system's intelligent dialogue service, this method can accurately pinpoint the specific function in the dialogue system experiencing a defect, achieving module-level root cause identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical methods of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments.

[0020] Figure 1 A schematic diagram of the structure of a dialogue system provided in an embodiment of the present application;

[0021] Figure 2 A flowchart of a root cause location method for a dialogue system based on a language model provided in an embodiment of the present application;

[0022] Figure 3 A flowchart of a root cause location method for a dialogue system based on a language model provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of the structure of a root cause location device for a dialogue system based on a language model provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The terms "first" and "second" in the embodiments of this application are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0026] First, some technical terms and application scenarios involved in the embodiments of this application are introduced.

[0027] Dialogue systems, also known as conversational robots or digital assistants, typically have natural language processing capabilities and are used to provide intelligent conversational services. Users can interact with these systems through human-computer dialogue. Specifically, users send input information (queries) to the dialogue system, which analyzes the input and generates responses to it, thus enabling human-computer dialogue.

[0028] The dialogue system can provide intelligent dialogue services based on the language model. That is, the dialogue system can be connected to the language model, and by calling the language model, the language model can perform natural language processing on the input information and receive the reply information returned by the language model.

[0029] Typically, users can engage in various types of intelligent conversations with a dialogue system. For example, a user can send a question related to general knowledge to the dialogue system, and the dialogue system will generate an answer to the question, completing a question-and-answer type of intelligent conversation. Another example is a user can send a command related to natural language processing to the dialogue system, and the dialogue system will perform natural language processing on the input information and generate a processed response, completing a natural language processing type of intelligent conversation.

[0030] The dialogue system can provide services in various forms. In some examples, the dialogue system can be an independent software system, such as an independent application (APP). By installing the dialogue system on a terminal, intelligent dialogue with the dialogue system can be achieved. In other examples, the dialogue system can also be deployed in other systems. For example, the dialogue system can be deployed in office collaboration systems, instant messaging systems, document systems, etc., providing intelligent dialogue services as a functional component of other systems.

[0031] As dialogue systems continue to develop, their structures have become increasingly complex. In practical applications, dialogue systems can include multiple modules that work together to provide intelligent dialogue services. These systems are also referred to as multi-module dialogue systems. Specifically, in a multi-module dialogue system, multiple modules collaborate to process input information and ultimately generate corresponding responses. For example, multiple modules can implement corresponding information processing functions by invoking language models.

[0032] For an example, see Figure 1 The diagram shows a structural diagram of a multi-module dialogue system. Input information sent by the user is first processed by the intent distribution module, which identifies the intent of the input information, determines the intent of the input information, and sends the input information to different agents in the dialogue system for processing based on the intent of the input information. For example, when the intent representation of the input information is processed online, the intent distribution module can send the input information to the networked agent. When the intent representation of the input information requires programmatic processing, the intent distribution module can send the input information to the programmed agent.

[0033] The networked agent includes a sentence rewriting module, an online search module, and a summary module. The sentence rewriting module rewrites input information to make it easier to search online. The online search module uses the rewritten input information to search the internet and obtain relevant knowledge. The summary module combines this knowledge with the input information to generate a final response.

[0034] The programming agent includes a requirements extraction module and a code generation module. The requirements extraction module extracts programming-related requirements from input information, such as the programming language and the functions to be implemented. The code generation module combines the input information and the programming-related requirements information to generate the corresponding code to obtain the response information.

[0035] Because each module can focus on its own information processing function, it can more accurately process input information and improve the accuracy of the dialogue system. Furthermore, the modular design enhances the scalability and maintainability of the dialogue system, making it easier to upgrade and optimize the dialogue system to adapt to different dialogue scenarios and needs.

[0036] To improve the quality of intelligent conversations, conversation system service providers need to test and diagnose the accuracy of the conversation system's questions and answers. In related technologies, when testing and diagnosing conversation systems with multi-module architectures, preset sample data (e.g., different input messages) is usually input into the conversation system to obtain the conversation system's output response information. The system then combines this sample data to determine whether the conversation system's response information meets user needs. For example, whether there are issues with poor timeliness, whether there are questions that are not answered as asked, or whether the response information does not match the input information.

[0037] However, the above method can only perform a macro-diagnosis of the intelligent dialogue capability of the dialogue system by analyzing the reply information. It is difficult to locate the specific root cause and lacks fine-grained module diagnosis capabilities.

[0038] In view of this, the present application provides a root cause location method for a dialogue system based on a language model, wherein the dialogue system is used to generate corresponding reply information based on a language model for input information, and the dialogue system includes multiple modules, each of which provides different information processing functions. The method first obtains sample data of the dialogue system, wherein the sample data includes: first input information, first reply information corresponding to the first input information, and a first link log for the dialogue system to generate the first reply information, wherein the first reply information meets the negative sample condition, and for each module of at least some of the multiple modules, the following steps are performed: based on the first input information, the first reply information, and at least one item in the target link log related to the module in the first link log, the root cause analysis result of the module is determined; wherein the root cause analysis result is used to characterize whether there is a defect in the information processing function provided by the module.

[0039] This method diagnoses the information processing functions provided by each module in a multi-module dialogue system. For each module, this method leverages the dialogue system's input and output information and the link logs associated with each module to conduct a targeted analysis of its information processing function, resulting in module-level root cause identification. Because each module is responsible for a specific function within the dialogue system's intelligent dialogue service, this method can accurately pinpoint the specific function in the dialogue system experiencing a defect, achieving module-level root cause identification.

[0040] To facilitate understanding of the technical solutions provided in the embodiments of the present application, the following description will be made with reference to the accompanying drawings. Figure 2 The figure shows a flow chart of a root cause localization method for a dialogue system based on a language model.

[0041] First, the dialogue system in the embodiments of this application will be described. The dialogue system is used to generate corresponding responses based on a language model for input information. Both the input information and the responses are described in natural language. In other words, the dialogue system has natural language processing capabilities, analyzing input information described in natural language to generate responses described in natural language. The dialogue system can be understood as a dialogue system that provides intelligent dialogue services and requires testing and diagnosing the quality of these services.

[0042] In the embodiments of the present application, the dialogue system includes multiple modules, each of which provides different information processing functions. In other words, the dialogue system is a dialogue system with a multi-module architecture. Each module in the dialogue system can be understood as a functional module for implementing different information processing functions. Through the coordinated processing of multiple modules, intelligent dialogue capabilities are achieved.

[0043] The embodiments of this application do not limit the deployment method of the dialogue system. In some embodiments, the dialogue system can be a locally deployed dialogue system. In this case, the dialogue system is obtained locally for testing and diagnosis of the dialogue system. In other embodiments, the dialogue system can be a cloud-based dialogue system. In this case, the dialogue system is obtained from the cloud for testing and diagnosis of the dialogue system.

[0044] The method specifically comprises the following steps:

[0045] S201: Obtain sample data of the dialogue system.

[0046] Among them, the sample data can be understood as test samples used to test and diagnose the intelligent dialogue quality of the dialogue system. That is to say, combined with the sample data, it is determined which module or modules in the dialogue system have defects, such as a fault, or the information processing function provided does not meet expectations.

[0047] Specifically, the sample data includes the first input information, the first reply information corresponding to the first input information, and the first link log of the dialogue system generating the first reply information. The first input information can be understood as the information input into the dialogue system, and the first reply information corresponding to the first input information can be understood as the reply information generated by the dialogue system after analyzing and processing the first input information.

[0048] Among them, the first reply information meets the negative sample condition, that is, the quality of the question and answer corresponding to the first reply information does not meet the requirements. In some embodiments, the negative sample condition can be related to the user's feedback on the first reply information. For example, after outputting the first reply information to the user, the dialogue system can also provide feedback controls (for example, a control indicating satisfaction with the first reply information and a control indicating dissatisfaction with the first reply information). Through the user's selection operation on the feedback control, it is determined whether the first reply information meets the negative sample condition. For example, when the user selects the control indicating dissatisfaction with the first reply information, the first reply information meets the negative sample condition.

[0049] It can be understood that since root cause location is required in the embodiment of the present application, the first reply information should meet the negative sample condition, and then through the root cause location process, the specific reason why the first reply information is a negative sample can be determined.

[0050] The embodiments of the present application do not limit the method for obtaining the first input information and the first reply information. For example, the first input information can be input information actually sent by a user to the dialogue system during operation of the dialogue system, and the corresponding first reply information can be the reply information actually output by the dialogue system in response to the first input information during operation of the dialogue system. For another example, the first input information can be configured by a tester. After manually configuring the first input information, the first input information is input into the dialogue system, and the first reply information output by the dialogue system is obtained.

[0051] The first link log can be understood as the log information of the dialogue system, and the first link log can be used to describe the process of the dialogue system analyzing the first input information and generating the first reply information. In some embodiments, the first link log may include the input data and output data of multiple modules in the dialogue system, the execution status of multiple modules, and the execution sequence of multiple modules. Among them, the execution sequence can be understood as the order in which the modules execute related functions in the process of the dialogue system generating the first reply information. For example, in the process of the dialogue system generating the first reply information, it is first processed by module A, then by module C, and then by module D. In this case, the execution sequence of module A is 1, the execution sequence of module C is 2, and the execution sequence of module D is 3.

[0052] S202: For each module of at least some of the multiple modules, perform the following steps: determine a root cause analysis result of the module based on the first input information, the first reply information, and at least one item in the target link log related to the module in the first link log.

[0053] See also Figure 3The figure shows a flow chart of a root cause location method for a dialogue system. Since the purpose of the embodiment of the present application is to achieve module-level root cause location, in the process of testing and diagnosing the dialogue system using sample data, analysis is performed on a module-by-module basis to determine the root cause analysis results of each module. The root cause analysis results can be used to characterize whether there are defects in the information processing function provided by the module.

[0054] Since each module in the dialogue system is used to provide different information processing functions, when performing root cause analysis on a module basis, it is necessary to filter out the target link log related to the module from the first link log. In this way, by combining the target link log related to the module with the first input information and the first reply information, it is possible to accurately determine whether there are defects in the module's processing of the first input information.

[0055] In some embodiments, the target link log associated with the module in the first link log may include the module's input data and output data. In other words, the first link log is filtered to identify the data actually input to each module and the data actually output by each module when processing the first input information. This allows for subsequent determination of whether the information processing function provided by the module is defective based on the module's input data and output data.

[0056] In some possible implementations, root cause analysis is performed on each module based on assertion rules. Specifically, at least one assertion rule corresponding to the module is determined, and target link logs related to the module in the first link log are verified based on the at least one assertion rule to obtain a root cause analysis result for the module.

[0057] Among them, assertion rules can be understood as rules that verify whether a certain condition is met. In the embodiment of the present application, assertion rules can be used to verify whether the information processing function provided by the module is defective. In other words, the target link logs related to each module are verified using assertion rules. Since the target link logs can represent the information processing functions performed by each module in the process of generating the first reply information, the target link logs are verified using assertion rules, and root cause analysis is performed on each module to determine the root cause analysis results of each module.

[0058] In the embodiments of the present application, assertion rules are associated with modules. In other words, each module is configured with an assertion rule specifically for verifying that module. The embodiments of the present application do not limit the method for generating assertion rules. For example, based on the high frequency of defects in each module, an assertion rule associated with the module is formulated to verify whether the information processing function provided by the module has defects.

[0059] For example, the dialogue system includes module A, which is used to search the Internet to obtain knowledge information related to the input information. The assertion rules associated with module A can be as follows:

[0060]

[0061] Among them, the assertion rule can be used to verify whether the uniform resource locator (URL) in the knowledge information searched by module A on the Internet is available. In this way, the target link log related to module A in the first link log is verified by the assertion rule, and it is determined whether there is a defect in the target link log related to module A that the knowledge information searched by module A on the Internet has the URL unavailable. If so, the root cause analysis result is determined to indicate that the information processing function provided by module A has a defect; if not, the root cause analysis result is determined to indicate that the information processing function provided by module A does not have a defect.

[0062] In this way, by presetting multiple assertion rules, targeted judgments can be made on common defects in various modules of the dialogue system, and then the root cause analysis results of each module can be determined.

[0063] In some other possible implementations, a language model is used to perform root cause analysis on each module. Specifically, a first prompt word is generated, the first prompt word is sent to the language model, and a root cause analysis result of the module is received from the language model.

[0064] A language model can be understood as a natural language processing model based on deep learning technology. It typically has the ability to understand, process, and generate natural language. Prompts can be used to guide the language model to specific outputs in generative tasks (such as text generation, question-answering, and conversational tasks). By configuring prompts, the language model understands the context and requirements of the task, enabling it to handle different types of natural language processing tasks without retraining, increasing its scalability and flexibility.

[0065] In an embodiment of the present application, the first prompt word includes: root cause analysis information and prompt information used to indicate whether there is a defect in the information processing function provided by the module based on the root cause analysis information. The root cause analysis information includes the first input information, the first reply information and at least one of the target link logs related to the module in the first link log.

[0066] Since the first prompt word is configured with the above information, the language model can perform root cause analysis on each module based on the prompt capability of the first prompt word and the root cause analysis information, determine whether the information processing function provided by each module has defects, and output the root cause analysis results of each module.

[0067] For example, the modules of the dialogue system include module B, which is used to identify the intent of the input information and judge the intention of the input information. In this case, the root cause analysis information may include a first prompt word, which may include the first input information and the target link log related to module B in the first link log. The first prompt word may include the aforementioned root cause analysis information and prompt information used to indicate whether the intention recognition of module B is correct.

[0068] The language model recognizes the intent of the first input information based on the prompt ability of the first prompt word, determines the correct intent of the first input information, and then combines the target link log related to module B to determine whether the intent of the first input information output by module B in the target link log is consistent with the correct intent. Figure 1 If they are consistent, the language model can output a root cause analysis result that the information processing function provided by the representation module B is not defective; if they are inconsistent, the language model can output a root cause analysis result that the information processing function provided by the representation module B is defective.

[0069] Continuing with the above example, in some dialogue systems, module B's intention recognition of input information may include two intention recognition processes. In the first intention recognition process, module B preliminarily recognizes the intention of the input information, and determines to which agent in the dialogue system the input information is to be sent based on the preliminarily recognized intention of the input information. In the second intention recognition process, a more detailed intention recognition is performed based on the determined agent. For example, when the determined agent is an information search agent, it further identifies whether the input information requires an online information search, or whether the input information involves a rich media information search.

[0070] In this case, the prompt information in the first prompt word indicating whether module B's intent recognition is correct can be further refined to indicate whether module B's preliminary intent recognition is correct and, if the preliminary intent recognition is correct, whether module B's detailed intent recognition is correct. In this way, the first language model can perform a hierarchical root cause analysis on module B and output the root cause identification results for module B.

[0071] For example, when the initial intention recognition of module B is incorrect, the root cause recognition result can indicate that module B divides the input information into the wrong intelligent agent. When the initial intention recognition of module B is correct, but the detailed intention recognition is incorrect, the root cause recognition result can indicate that module B has missed a call, made a wrong call, over-called, or made an incorrect judgment on the information type of the input information.

[0072] In some embodiments, the root cause analysis results can also characterize specific defects in the module, and the information processing functions provided by a module can have different types of defects. For example, for a sentence rewriting module, the target link log associated with the module may include the rewritten first input information. Based on the first input information and the rewritten first input information, it is determined whether the sentence rewriting module has defects. Specific defects in the module may include information loss, poor timeliness, information errors, information redundancy, etc. For an online search module, the target link log associated with the module may include the rewritten first input information and search results. Based on the target link log associated with the module, it is determined whether the online search module has defects. Specific defects in the module may include sorting problems, poor authority, unavailable images, unavailable videos, low relevance between search results and the first input information, etc. For a summary module, the target link log associated with the module may include search results. Based on the first input information, the target link log associated with the module, and the first reply information, it is determined whether the summary module has defects. Specific defects in the module may include format abnormalities, duplication and redundancy, language mismatch, etc.

[0073] After obtaining the root cause analysis results of each module, identification information of a target module in the dialogue system may also be output, wherein the root cause analysis result of the target module indicates that an information processing function provided by the module has defects.

[0074] By performing root cause analysis on each module in the dialogue system, the root cause analysis results of each module are obtained, and the module that represents the function provided by the module with defects is identified as the target module. In this way, from the various modules of the dialogue system, the modules with defects and that can be optimized are found, and fine-grained module root cause positioning is achieved. By outputting the identification information of the target module, such as the name of the target module, the test results of the dialogue system are presented to the tester.

[0075] Furthermore, after finding the defective target module in the dialogue system, training data may be generated based on the first input information and the first reply information, and the target module may be trained using the training data.

[0076] That is to say, the dialogue system is tested through sample data and defects in the target module are diagnosed. For the target module, when the first input information is used as the input of the dialogue system, the first reply information is the output of the dialogue system that does not meet expectations and needs to be optimized.

[0077] Therefore, by using the first input message and the first reply message as training data, the target module with a defective information processing function in the dialogue system is retrained. This allows for targeted, module-level training and optimization of the target module corresponding to this sample data. Furthermore, a closed-loop link between the fault attribution process and the module training process is established, shortening the dialogue system's optimization cycle and enabling root cause identification and system optimization using a single sample data set.

[0078] Continue to see Figure 3 ,After testing the dialogue system and outputting the identification information of the ,defective target module, the hierarchical label corresponding to the target module can also be ,output. The hierarchical label describes the hierarchical relationship of the target ,module in the dialogue system.

[0079] Among them, the hierarchical label includes at least one of the following: a label indicating the user intention corresponding to the first input information for analyzing the first input information, a label indicating the target capability provided together with other modules in the dialogue system, a label indicating the information processing function provided, and a label indicating the module parameters.

[0080] That is to say, in the embodiment of the present application, a multi-level tag is constructed for the dialogue system with a multi-module architecture. Figure 1 Taking the dialogue system in as an example, the intention distribution module is the first level, the networked agent and the programming agent are the second level, the modules in the networked agent (i.e., the sentence rewriting module, the networked search module, and the summary module) and the modules in the programming agent (i.e., the requirement extraction module and the code generation module) are the third level, and the specific module parameters of each module are the fourth level.

[0081] When the target module is an intent distribution module, since the target module is at the first level, the hierarchical tags associated with the target module only include tags indicating the user intent for analyzing the first input information. When the target module is a sentence rewriting module, since the target module is at the third level, the hierarchical tags associated with the target module include tags indicating the user intent for analyzing the first input information, tags indicating that the target capability is provided together with other modules in the dialogue system, and tags indicating the provided functions. In other words, the tags associated with a module include tags at a level higher than or equal to that module's level.

[0082] In this way, after performing a root cause analysis on the dialogue system, the output presents the hierarchical labels associated with the target module. Testers can intuitively determine the hierarchical relationship of the target module in the dialogue system by viewing the hierarchical labels, and clarify which level in the dialogue system has defects.

[0083] In addition, for the service provider of the dialogue system, a tester or a testing team can be responsible for related services at a level in the dialogue system. By outputting the level label associated with the target module, the service provider can quickly locate the tester or testing team corresponding to the target module that needs to be optimized and repaired, thereby improving the efficiency of optimizing the dialogue system.

[0084] In some embodiments, the target module is multiple target modules. That is, after root cause location is performed on the dialogue system, multiple defective target modules are found in the dialogue system. The root cause analysis results include a first root cause analysis result determined based on the first assertion rule and a second root cause analysis result determined using the first language model. That is, for each target module, the first root cause analysis result is determined by verifying the target link log associated with the target module using the first assertion rule, and the second root cause analysis result is determined by analyzing the target link log associated with the target module using the first language model.

[0085] In this case, the severity of the target modules is determined by ranking them. Specifically, the target modules are ranked based on their execution sequence in the first link log, the first root cause analysis results, and the second root cause analysis results, to determine their order. The order of the target modules is used to measure the severity of the defects in the target modules.

[0086] In other words, the severity of defects in each target module is judged and ranked based on three dimensions: execution timing, root cause analysis results based on assertion rules, and root cause analysis results based on language models. Since the severity of defects in each target module can, to a certain extent, indicate the urgency of repair or optimization, by ranking multiple target modules, testers can repair and optimize them in an orderly manner, from the most severe defect to the least severe, minimizing the impact on the normal operation of the dialogue system.

[0087] In some possible implementations, defect degree assessment values of multiple target modules are determined based on the first ranking value and the first weight value corresponding to the execution timing of the target module in the first link log, the second ranking value and the second weight value corresponding to the first root cause analysis result, and the third ranking value and the third weight value corresponding to the second root cause analysis result; and the order of the multiple target modules is determined based on the defect degree assessment values of the multiple target modules.

[0088] Based on the foregoing, the defect degree evaluation value may be: first ranking value×first weight value+second ranking value×second weight value+third ranking value×third weight value.

[0089] That is, by configuring different weights for execution timing, root cause analysis results based on assertion rules, and root cause analysis results based on language models, the severity of defects in each target module is quantified, and then multiple target modules are ranked.

[0090] The execution sequence of the target module in the first link log is negatively correlated with the first ranking value. Understandably, since the module that is executed first in the process of generating the first reply message by the dialogue system has a greater impact on the first reply message, the target module that is executed earlier in the first link log is assigned a higher ranking value.

[0091] The second ranking value corresponds to the confidence level of the first assertion rule, and the third ranking value corresponds to the confidence level of the first language model. In other words, each assertion rule has a corresponding confidence level, and each language model has a corresponding confidence level. The confidence level of the assertion rule can be understood as the credibility of the first root cause analysis result, and the confidence level of the language model can be understood as the credibility of the second root cause analysis result.

[0092] Furthermore, the second weight is greater than the third weight. Considering that assertion rules are more interpretable and accurate, while language models may suffer from hallucinations when processing complex semantics, a larger second weight and a smaller third weight are configured to make the first root cause analysis results based on assertion rules more decisive.

[0093] This method diagnoses the information processing functions provided by each module in a multi-module dialogue system. For each module, this method leverages the dialogue system's input and output information and the link logs associated with each module to conduct a targeted analysis of its information processing function, resulting in module-level root cause identification. Because each module is responsible for a specific function within the dialogue system's intelligent dialogue service, this method can accurately pinpoint the specific function in the dialogue system experiencing a defect, achieving module-level root cause identification.

[0094] Combined with the above Figures 1 to 3 The root cause localization method of the language model-based dialogue system provided in the embodiment of the present application is introduced in detail. The following will introduce the apparatus and equipment provided in the embodiment of the present application in conjunction with the accompanying drawings.

[0095] See also Figure 4 The diagram shows a schematic diagram of a root cause location device for a language model-based dialogue system. The dialogue system is configured to generate corresponding reply information based on a language model for input information. The dialogue system includes multiple modules, each of which provides a different information processing function. The device 40 includes:

[0096] Acquisition module 401 is configured to acquire sample data of the dialogue system; wherein the sample data includes: a first input message, a first reply message corresponding to the first input message, and a first link log of the dialogue system generating the first reply message, wherein the first reply message satisfies a negative sample condition;

[0097] The analysis module 402 is used to perform the following steps for each module of at least some of the multiple modules: determining a root cause analysis result of the module based on the first input information, the first reply information, and at least one item in the target link log related to the module in the first link log; wherein the root cause analysis result is used to indicate whether there is a defect in the information processing function provided by the module.

[0098] In some possible implementations, the analysis module 402 is specifically configured to:

[0099] Determine at least one assertion rule corresponding to the module; wherein the assertion rule is used to verify whether there are defects in the function provided by the module;

[0100] According to the at least one assertion rule, the target link log related to the module in the first link log is verified to obtain a root cause analysis result of the module.

[0101] In some possible implementations, the analysis module 402 is specifically configured to:

[0102] Generate a first prompt word; wherein the first prompt word includes: root cause analysis information and prompt information for indicating whether an information processing function provided by the module has a defect based on the root cause analysis information, the root cause analysis information including at least one of the first input information, the first reply information, and a target link log related to the module in the first link log;

[0103] The first prompt word is sent to a language model, and a root cause analysis result of the module returned by the language model is received.

[0104] In some possible implementations, the apparatus 40 further includes an output module, and the output module is configured to:

[0105] The identification information of the target module in the dialogue system is output, and the root cause analysis result of the target module indicates that the information processing function provided by the module has defects.

[0106] In some possible implementations, the apparatus 40 further includes a training module, wherein the training module is configured to:

[0107] generating training data according to the first input information and the first reply information;

[0108] The target module is trained using the training data.

[0109] In some possible implementations, the output module is further configured to:

[0110] Outputting a hierarchical label associated with the target module, the hierarchical label describing a hierarchical relationship of the target module in the dialogue system;

[0111] Among them, the hierarchical label includes at least one of the following: a label indicating the user intention corresponding to the first input information for analyzing the first input information, a label indicating the target capability provided together with other modules in the dialogue system, a label indicating the information processing function provided, and a label indicating module parameters.

[0112] In some possible implementations, the target module includes multiple target modules, the root cause analysis result includes a first root cause analysis result determined based on the first assertion rule and a second root cause analysis result determined using the first language model; and the apparatus 40 further includes a determination module configured to:

[0113] The multiple target modules are sorted according to the execution timing of the target modules in the first link log, the first root cause analysis result, and the second root cause analysis result to determine the order of the multiple target modules; wherein the order of the multiple target modules is used to measure the severity of the defects existing in the target modules.

[0114] In some possible implementations, the determining module is specifically configured to:

[0115] Determining defect degree assessment values of the plurality of target modules based on a first ranking value and a first weight value corresponding to the execution timing of the target module in the first link log, a second ranking value and a second weight value corresponding to the first root cause analysis result, and a third ranking value and a third weight value corresponding to the second root cause analysis result;

[0116] determining an order of the plurality of target modules according to the defect degree evaluation values of the plurality of target modules;

[0117] Among them, the execution timing of the target module in the first link log is negatively correlated with the first sorting value, the second sorting value corresponds to the confidence of the first assertion rule, the third sorting value corresponds to the confidence of the first language model, and the second weight value is greater than the third weight value.

[0118] In some possible implementations, the target link log related to the module in the first link log includes: input data and output data of the module.

[0119] The root cause location device 40 of the dialogue system based on the language model according to the embodiment of the present application may correspond to the method described in the embodiment of the present application, and the above and other operations and / or functions of the various modules / units of the root cause location device 40 of the dialogue system based on the language model are respectively to achieve Figure 2 For the sake of brevity, the corresponding processes of the various methods in the illustrated embodiments are not described here in detail.

[0120] The embodiment of the present application also provides an electronic device. The electronic device is specifically used to implement Figure 4 The functions of the root cause localization device 40 of the language model-based dialogue system in the illustrated embodiment.

[0121] Figure 5 A schematic diagram of the structure of an electronic device 500 is provided. Figure 5 As shown, the electronic device 400 includes a bus 501, a processor 502, a communication interface 503, and a memory 504. The processor 502, the memory 504, and the communication interface 503 communicate with each other via the bus 501.

[0122] The bus 501 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0123] The processor 502 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0124] The communication interface 503 is used for communicating with the outside, for example, the communication interface 503 can be used for communicating with a terminal.

[0125] The memory 504 may include volatile memory, such as random access memory (RAM), or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0126] The memory 504 stores executable codes, and the processor 502 executes the executable codes to perform the aforementioned root cause localization method for the language model-based dialogue system.

[0127] Specifically, in the implementation Figure 4 In the case of the embodiment shown, and Figure 4 In the embodiment, when each module or unit of the root cause location device 40 of the dialogue system is implemented by software, the execution Figure 4 The software or program code required for the functions of each module / unit in the system may be partially or completely stored in the memory 404. The processor 502 executes the program code corresponding to each unit stored in the memory 504 to perform the root cause location method of the dialogue system based on the language model.

[0128] Embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, hard disk, or magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the root cause localization method for a language model-based dialogue system as described above, as applied to the root cause localization apparatus 40 for the language model-based dialogue system.

[0129] The present application also provides a computer program product comprising one or more computer instructions that, when loaded and executed on a computing device, fully or partially generate the process or function described in the present application.

[0130] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0131] When the computer program product is executed by a computer, the computer performs any of the aforementioned root cause localization methods for a language model-based dialogue system. The computer program product may be a software installation package. When any of the aforementioned root cause localization methods for a language model-based dialogue system is needed, the computer program product may be downloaded and executed on the computer.

[0132] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to the various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the prescribed logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0133] The units involved in the embodiments described in this application may be implemented in software or hardware, wherein the name of a unit / module does not, in some cases, constitute a limitation on the unit itself.

[0134] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0135] In the context of the present application embodiment, machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include but is not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0136] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0137] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0138] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0139] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0140] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A root cause location method for a dialogue system based on a language model, characterized in that: The dialogue system is used to generate corresponding reply information based on a language model for input information, the dialogue system includes multiple modules, and the multiple modules respectively provide different information processing functions. The method includes: Obtaining sample data of the dialogue system; wherein the sample data includes: first input information, first reply information corresponding to the first input information, and a first link log of the dialogue system generating the first reply information, wherein the first reply information satisfies a negative sample condition; For each module of at least some of the multiple modules, the following steps are performed: determining a root cause analysis result of the module based on the first input information, the first reply information, and at least one item in the target link log related to the module in the first link log; wherein the root cause analysis result is used to characterize whether there is a defect in the information processing function provided by the module.

2. The method according to claim 1, characterized in that The determining a root cause analysis result of the module according to the first input information, the first reply information, and at least one of the target link logs related to the module in the first link log includes: Determining at least one assertion rule corresponding to the module; wherein the assertion rule is used to verify whether an information processing function provided by the module has defects; According to the at least one assertion rule, the target link log related to the module in the first link log is verified to obtain a root cause analysis result of the module.

3. The method according to claim 1, characterized in that The determining a root cause analysis result of the module according to the first input information, the first reply information, and at least one of the target link logs related to the module in the first link log includes: Generate a first prompt word; wherein the first prompt word includes: root cause analysis information and prompt information for indicating whether an information processing function provided by the module has a defect based on the root cause analysis information, the root cause analysis information including at least one of the first input information, the first reply information, and a target link log related to the module in the first link log; The first prompt word is sent to a language model, and a root cause analysis result of the module returned by the language model is received.

4. The method according to claim 1, wherein The method further comprises: The identification information of the target module in the dialogue system is output, and the root cause analysis result of the target module indicates that the information processing function provided by the module has defects.

5. The method according to claim 4, characterized in that The method further comprises: generating training data according to the first input information and the first reply information; The target module is trained using the training data.

6. The method according to claim 4, characterized in that The method further comprises: Outputting a hierarchical label associated with the target module, the hierarchical label describing a hierarchical relationship of the target module in the dialogue system; Among them, the hierarchical label includes at least one of the following: a label indicating the user intention corresponding to the first input information for analyzing the first input information, a label indicating the target capability provided together with other modules in the dialogue system, a label indicating the information processing function provided, and a label indicating module parameters.

7. The method according to claim 4, characterized in that The target module is a plurality of target modules, and the root cause analysis result includes a first root cause analysis result determined based on the first assertion rule and a second root cause analysis result determined using the first language model; The method further comprises: The multiple target modules are sorted according to the execution timing of the target modules in the first link log, the first root cause analysis result, and the second root cause analysis result to determine the order of the multiple target modules; wherein the order of the multiple target modules is used to measure the severity of the defects existing in the target modules.

8. The method according to claim 7, characterized in that Sorting the multiple target modules according to the execution timing of the target modules in the first link log, the first root cause analysis result, and the second root cause analysis result to determine the order of the multiple target modules includes: Determining defect degree assessment values of the plurality of target modules based on a first ranking value and a first weight value corresponding to the execution timing of the target module in the first link log, a second ranking value and a second weight value corresponding to the first root cause analysis result, and a third ranking value and a third weight value corresponding to the second root cause analysis result; determining an order of the plurality of target modules according to the defect degree evaluation values of the plurality of target modules; Among them, the execution timing of the target module in the first link log is negatively correlated with the first sorting value, the second sorting value corresponds to the confidence of the first assertion rule, the third sorting value corresponds to the confidence of the first language model, and the second weight value is greater than the third weight value.

9. The method according to any one of claims 1 to 8, characterized in that The target link log related to the module in the first link log includes: input data and output data of the module.

10. A root cause location device for a dialogue system based on a language model, characterized in that: The dialogue system is used to generate corresponding reply information based on a language model for input information. The dialogue system includes multiple modules, each of which provides different information processing functions. The device includes: an acquisition module, configured to acquire sample data of the dialogue system; wherein the sample data includes: a first input message, a first reply message corresponding to the first input message, and a first link log of the dialogue system generating the first reply message, wherein the first reply message satisfies a negative sample condition; An analysis module is configured to perform the following steps for each module of at least some of the multiple modules: determining a root cause analysis result of the module based on the first input information, the first reply information, and at least one item in the target link log related to the module in the first link log; wherein the root cause analysis result is used to characterize whether there is a defect in the information processing function provided by the module.

11. An electronic device, characterized in that: The electronic device includes a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that The method comprises instructions, wherein the instructions instruct an electronic device to execute the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product comprises computer-readable instructions for implementing the method according to any one of claims 1 to 9.

Images