Abnormal log data detection method and device, computer equipment and storage medium

Abstract

The invention relates to an abnormal log data detection method and device, computer equipment and a storage medium. The method comprises the steps of obtaining to-be-processed log data of a dialogue system; the to-be-processed log data comprises dialogue input data and dialogue output data; inputting the dialogue input data into a preset dialogue model to obtain model output data; determining candidate log data according to the dialogue output data and the model output data; and obtaining an analysis result input by the user for the candidate log data, and determining abnormal log data according to the analysis result. The abnormal log data detection method provided by the invention is low in manpower consumption and high in efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer device, and storage medium for detecting abnormal log data. Background Art

[0002] With the rapid development of the Internet, dialogue systems have been integrated into people's daily lives, such as the intelligent customer service of e-commerce and in-car language assistants used in daily life.

[0003] In traditional technology, the iterative working mode of the dialogue system includes R&D personnel obtaining online log data or test set data, manually annotating the obtained log data or test set data to filter out data that may have problems in the dialogue system, and then the R&D personnel improve the problematic data to achieve positive feedback for the online dialogue system.

[0004] However, traditional technology relies on manual detection of problematic data in the dialogue system, which has the problems of high manpower consumption and low efficiency. Summary of the Invention

[0005] Based on this, it is necessary to provide a method, device, computer equipment and storage medium for detecting abnormal log data that can reduce manpower consumption and improve efficiency in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for detecting abnormal log data, the method comprising:

[0007] Obtaining log data to be processed by the dialogue system; the log data to be processed includes dialogue input data and dialogue output data;

[0008] Input the dialogue input data into the preset dialogue model to obtain the model output data;

[0009] Determine candidate log data based on conversation output data and model output data;

[0010] Obtain the analysis results of the user's input for the candidate log data, and determine the abnormal log data based on the analysis results.

[0011] In a second aspect, the present application further provides a device for detecting abnormal log data, the device comprising:

[0012] An acquisition module is used to acquire log data to be processed by the dialogue system; the log data to be processed includes dialogue input data and dialogue output data;

[0013] An input module, used to input dialogue input data into a preset dialogue model to obtain model output data;

[0014] A determination module is used to determine candidate log data based on the dialogue output data and the model output data;

[0015] The determination module is further used to obtain the analysis results input by the user for the candidate log data and determine the abnormal log data based on the analysis results.

[0016] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method provided in the first aspect when executing the computer program.

[0017] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method provided in the first aspect when the computer program is executed by a processor.

[0018] In a fifth aspect, the present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method provided in the first aspect above.

[0019] The above-described abnormal log data detection method, apparatus, computer device, and storage medium obtain log data to be processed from a dialogue system; the log data to be processed includes dialogue input data and dialogue output data; the dialogue input data is input into a preset dialogue model to obtain model output data; candidate log data is determined based on the dialogue output data and the model output data; the user's analysis results of the candidate log data input are obtained, and abnormal log data is determined based on the analysis results. In this embodiment, the dialogue model is first used to screen candidate log data that may contain problems in the log data to be processed. The user then analyzes the candidate log data to determine abnormal log data. This eliminates the need for the user to analyze all the log data to be processed, only a small amount of candidate log data, which reduces labor consumption and improves the efficiency of determining abnormal log data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a schematic structural diagram of a computer device in one embodiment;

[0021] Figure 2 A schematic diagram of a process flow of abnormal log data detection method according to an embodiment;

[0022] Figure 3 A schematic diagram of a process flow of a method for detecting abnormal log data in another embodiment;

[0023] Figure 4 A schematic diagram of a process flow of a method for detecting abnormal log data in another embodiment;

[0024] Figure 5A schematic diagram of a process flow of a method for detecting abnormal log data in another embodiment;

[0025] Figure 6 A schematic diagram of a process flow of a method for detecting abnormal log data in another embodiment;

[0026] Figure 7 A schematic diagram of a process flow of a method for detecting abnormal log data in another embodiment;

[0027] Figure 8 1 is a schematic diagram of the structure of an abnormal log data detection device in one embodiment;

[0028] Figure 9 Schematic diagram of the structure of an abnormal log data detection device in another embodiment. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0030] Before specifically introducing the technical solutions of the disclosed embodiments of this application, we will first introduce the background technology or technological evolution context underlying the embodiments of this application. With the rapid development of the internet, conversational systems have become integrated into people's daily lives, such as in e-commerce intelligent customer service and in-car voice assistants. A conversational system enables humans and machines to interact through natural language conversations. A conversational system includes multiple functional modules, such as a conversation state tracking module, a conversation strategy module, and a natural language generation module. To ensure the correct operation of a conversational system, each functional module in the conversational system, as well as the entire conversational system, requires continuous iteration to improve conversational effectiveness. Currently, a widely adopted model used by R&D teams for e-commerce intelligent customer service and in-car voice assistants involves manual offline full or sample log annotation, log analysis, and test set analysis. Specifically, R&D personnel obtain log data or test set data and manually annotate it offline to screen for log data that may contain issues within the conversational system. The R&D personnel then improve the problematic log data, providing positive feedback to the online conversational system and improving the conversational effectiveness of the conversational system. However, in traditional technologies, the process of manually detecting and iterating the dialogue system suffers from problems such as high labor consumption and low efficiency. In other words, the method of detecting problematic data in the dialogue system during the iterative process suffers from high labor consumption and low efficiency. To address this issue, the present application provides a method for detecting abnormal log data.

[0031] The following describes in detail the technical solution of the present application and how the technical solution of the present application solves the technical problem with specific embodiments.

[0032] The abnormal log data detection method provided in this application can be applied to computer devices, which can be, but are not limited to, industrial computers, laptops, and tablet computers. The internal structure of the computer device can be as follows: Figure 1 As shown. The computer device includes a processor, memory, a communication interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal via wired or wireless communication. The wireless communication can be achieved via Wi-Fi, a mobile cellular network, NFC (near-field communication), or other technologies. When executed by the processor, the computer program implements a method for detecting abnormal log data. The display screen of the computer device can be a liquid crystal display or an electronic ink display. The input device of the computer device can be a touch layer covering the display screen, or keys, a trackball, or a touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse.

[0033] In one embodiment, Figure 2 As shown, a method for detecting abnormal log data is provided. This embodiment uses the method applied to a computer device as an example. In this embodiment, the method includes the following steps:

[0034] Step 200: Obtain log data to be processed of the dialogue system; the log data to be processed includes dialogue input data and dialogue output data.

[0035] The application process of a dialogue system generates a large amount of log data. This log data includes dialogue input data and dialogue output data. Dialogue input data refers to the data entered by the user into the dialogue system when using it; dialogue output data refers to the data output by the dialogue system to the user after analyzing the user's input data. The dialogue input data and dialogue output data in the log data to be processed are in a one-to-one correspondence. In other words, the log data to be processed can include multiple sub-log data to be processed, each of which includes dialogue input data and the dialogue output data corresponding to the dialogue input data.

[0036] The dialogue system includes multiple functional modules, such as the ASR (Automatic Speech Recognition) module, the NLU (Natural Language Understanding) module, the DM (Dialog Management) module, and the TTS (Text-To-Speech) module. Log data corresponding to different functional modules is stored in different databases, and the computer device can store the pending log data corresponding to each functional module in the dialogue system in different databases. The pending log data of the dialogue system can also be pre-stored in the computer device's memory. When the pending log data is needed, the computer device directly retrieves it from the memory.

[0037] Step 210: Input the dialogue input data into the preset dialogue model to obtain model output data.

[0038] The conversation model can be a pre-trained neural network model stored in a computer device. Optionally, the conversation model is a generative model (llama)—a language model using adaptive attention. This generative model takes string input and outputs strings. Alternatively, the conversation model can be a Chat Generative Pre-trained Transformer (ChatGPT).

[0039] In an optional embodiment, the training process of the dialogue model includes: obtaining log data samples; the log data samples include dialogue input samples, and labeled dialogue output samples corresponding to the dialogue input samples; inputting the dialogue input samples into the neural network model to obtain data output by the neural network model; calculating the loss based on the data output by the neural network model and the dialogue output samples; updating the network weights of the neural network model based on the loss to achieve training of the neural network model. When the loss converges, the training stops and the dialogue model is obtained.

[0040] After obtaining the log data to be processed of the dialogue system, the computer device inputs the dialogue input data in the log data to be processed into the dialogue model, analyzes and processes the dialogue input data through the dialogue model, and obtains model output data corresponding to the dialogue input data.

[0041] Step 220: Determine candidate log data based on the dialogue output data and the model output data.

[0042] After the conversation model outputs the model output data corresponding to the conversation input data, the computer device compares the model output data with the conversation output data corresponding to the conversation input data to determine whether the conversation output data and the model output data are consistent. The computer device then identifies the conversation output data that is inconsistent with the model output data and the conversation input data corresponding to the conversation output data as candidate log data. The candidate log data includes conversation output data that may be abnormal and the conversation input data corresponding to the conversation output data.

[0043] In one optional embodiment, taking the NLU module in a dialogue system as an example, the dialogue input data of the log data to be processed by the NLU module includes input information "query." The NLU module outputs dialogue output data corresponding to the dialogue input data. The dialogue output data includes an output scenario (domain), intent (intent), and slots (slots). The computer device compares the output scenario, intent, and slots in the model output data with those in the dialogue output data. If any of the output scenarios, intents, or slots differ, the dialogue output data and the model output data are inconsistent. For example, if the model output data and the dialogue output data have the same output scenario and intent, but the same slot names but different slot values, the model output data and the dialogue output data are determined to be inconsistent. Suppose the query in the dialogue input data is "Navigate to A," and the domain, intent, and slots in the dialogue output data are "map," "navigate," and "location: A," respectively. After the conversation input data is fed into the conversation model, the model output data is: map#navigate#location: B. The domain, intent, and slots fields in the model output data are map, navigate, and location: B, respectively. Comparing the conversation output data with the model output data reveals that the slot values ​​in the model output data differ from those in the conversation output data. Therefore, the conversation output data and the corresponding conversation input data are identified as candidate log data.

[0044] Step 230: Obtain analysis results input by the user for the candidate log data, and determine abnormal log data based on the analysis results.

[0045] The computer device includes a labeling platform. After obtaining candidate log data, the computer device sends the candidate log data to the labeling platform for a user to analyze the candidate log data. The computer device also receives analysis results of the candidate log data input by the user. The analysis results may include issues with the dialogue output data or model output data in the candidate log data.

[0046] In an optional embodiment, the user may annotate the conversation input data in the candidate log data to obtain an annotation result; and compare the annotation result with the conversation output data in the candidate log data to obtain an analysis result.

[0047] After receiving the analysis results, the computer device determines abnormal log data within the candidate log data based on the analysis results. If the candidate log data includes multiple log data items, the abnormal log data determined based on the analysis results may include all or part of the candidate log data. The abnormal log data includes the problematic output data of the conversation and the corresponding input data of the conversation.

[0048] The abnormal log data detection method provided in an embodiment of the present application obtains log data to be processed from a dialogue system; the log data to be processed includes dialogue input data and dialogue output data; the dialogue input data is input into a preset dialogue model to obtain model output data; candidate log data is determined based on the dialogue output data and the model output data; the analysis results of the user's input of the candidate log data are obtained, and abnormal log data is determined based on the analysis results. In this embodiment, the dialogue model is first used to screen candidate log data that may have problems in the log data to be processed, and the user analyzes the candidate log data to determine abnormal log data. This method eliminates the need for the user to analyze all the log data to be processed, only a small amount of candidate log data, which can reduce manpower consumption and improve the efficiency of determining abnormal log data.

[0049] In addition, using the abnormal log data detection method provided by the embodiment of the present application in the iterative process of the dialogue system can improve the iteration efficiency. Compared with the manual labeling method in the traditional technology, the trigger cycle of the iterative process of the dialogue system can be increased to the hour level, and the abnormal log data discovery capability at the hour level can be achieved. The model output data is compared with the dialogue output data to determine the candidate log data, which can recall the long-tail log data on a larger scale, thereby solving the log data that is easily missed by the long-tail problem, and thus can improve the accuracy of determining the abnormal log data. Moreover, the abnormal log data detection method provided by the embodiment of the present application only needs to focus on the input data and output data of each functional module in the dialogue system, and does not require each functional module to go through complex self-iteration and program logic improvement. It can achieve the hourly iteration effect of the entire dialogue system based on the hourly abnormal log data discovery.

[0050] In one embodiment, a dialog model includes multiple sub-dialog models. The dialog system includes multiple functional modules, and the number of sub-dialog models is less than or equal to the number of functional modules. Log data to be processed by different functional modules may be processed using different sub-dialog models, or the same sub-dialog model. The types of sub-dialog models are the same as those of dialog models. For a description of sub-dialog models, refer to the description of dialog models in the above embodiments and will not be repeated here.

[0051] In the case where a dialogue model includes multiple sub-dialogue models, such as Figure 3 As shown, a method for inputting dialogue input data into a preset dialogue model to obtain model output data includes the following steps:

[0052] Step 300: Select a target sub-dialogue model from multiple sub-dialogue models according to the data category identifier in the log data to be processed.

[0053] The log data to be processed includes a data category identifier that can characterize the category of the log data to be processed. The data category identifier can be different numbers or different character strings. For example, if the data category identifier in the log data to be processed is "1", it means that the log data to be processed is a data category corresponding to the natural language module; if the data category in the log data to be processed is represented as "2", it means that the log data to be processed is a data category corresponding to the language synthesis module. The multiple sub-dialogue modules in the dialogue system are used to analyze and process different categories of dialogue input data.

[0054] The computer device can determine the category of the log data to be processed based on the data category identifier in the log data to be processed, and select a target sub-dialogue model from multiple sub-dialogue models based on the category of the log data to be processed.

[0055] In an optional embodiment, a first correspondence between categories and sub-dialogue models is stored in the computer device. The computer device searches the first correspondence for a category that is the same as the category of the log data to be processed, and determines the sub-dialogue model corresponding to the category in the first correspondence as the target sub-dialogue model.

[0056] Step 310: Input the dialogue input data into the target sub-dialogue model to obtain model output data.

[0057] After obtaining the target sub-dialogue model, the computer device inputs the dialogue input data in the log data to be processed into the target sub-dialogue model, analyzes and processes the dialogue input data through the target sub-dialogue model, and obtains model output data.

[0058] In this embodiment, a target sub-dialogue model is determined from among the multiple sub-dialogue models included in the dialogue model based on the data category identifier in the log data to be processed. Model output data corresponding to the dialogue input data is then obtained based on the target sub-dialogue model. This approach, taking into account the different logs to be processed corresponding to different functional modules in the dialogue system, uses different sub-dialogue models to process the dialogue input data in the corresponding log data to be processed. This improves the accuracy and reliability of the determined model output data, thereby increasing the accuracy of detecting abnormal log data.

[0059] In one embodiment, when a dialog model includes multiple sub-dialog models, such as Figure 4 As shown, another implementation method involves inputting dialogue input data into a preset dialogue model to obtain model output data, and the steps of this implementation method include:

[0060] Step 400: Classify the log data to be processed to obtain classification results corresponding to the log data to be processed.

[0061] After obtaining the log data to be processed, the computer device classifies the log data to be processed to obtain a classification result corresponding to the log data to be processed. The classification result is used to represent the category of the log data to be processed.

[0062] In an optional embodiment, a pre-trained classification model is stored in the computer device. The computer device inputs the acquired log data to be processed into the classification model, classifies the log data to be processed through the classification model, and outputs the classification results corresponding to the log data to be processed. The training process of the classification model may include: obtaining log data samples and the labeling results of the log data samples; the computer device inputs the log data samples into a neural network model to obtain the model classification results; comparing the model classification results with the labeling results to determine the error; adjusting the network weights of the neural network model according to the error to implement the training of the neural network model; when the error is less than or equal to the preset error threshold, the training is stopped to obtain the classification model.

[0063] Step 410: Select a target sub-dialogue model from multiple sub-dialogue models according to the classification result of the log data to be processed.

[0064] After obtaining the classification result of the log data to be processed, the computer device selects a target sub-dialogue model from the multiple sub-dialogue models of the dialogue model according to the classification result.

[0065] In an optional embodiment, a second correspondence between classification results and sub-dialogue models is pre-stored in the computer device. After obtaining the classification result of the log data to be processed, the computer device searches the second correspondence for a classification result identical to the classification result of the log data to be processed, and determines the sub-dialogue model corresponding to the classification result in the second correspondence as the target sub-dialogue model.

[0066] Step 420: Input the dialogue input data into the target sub-dialogue model to obtain model output data.

[0067] After obtaining the target sub-dialogue model, the computer device inputs the dialogue input data in the log data to be processed into the target sub-dialogue model, analyzes the dialogue input data through the target sub-dialogue model, and obtains model output data.

[0068] In this embodiment, the log data to be processed is first classified to obtain a classification result. A target sub-dialogue model is selected from multiple sub-dialogue models based on the classification result. The target sub-dialogue model is used to analyze the dialogue input data in the log data to be processed to obtain model output data. This method of determining the target sub-dialogue model is quick and easy to implement, and can improve the efficiency of determining the model output data.

[0069] In one embodiment, Figure 5 As shown, an implementation method for obtaining log data to be processed of a dialogue system includes the following steps:

[0070] Step 500: Collect initial log data of the dialogue system and store the initial log data in a search server.

[0071] The dialogue system includes multiple functional modules, and log data corresponding to different functional modules is stored in different databases. The computer device collects initial log data for the dialogue system, specifically, log data corresponding to each functional module in the dialogue system from different databases. After collecting the initial log data, the computer device stores the initial log data on a search server. The search server is an ES cluster.

[0072] In an optional embodiment, the computer device may collect the initial log data of the current line in real time during the application of the dialogue system. The computer device may also collect the initial log data of the dialogue system according to a preset time interval.

[0073] Step 510: Acquire initial log data within a preset time range from the search server according to a preset period as log data to be processed, and store the log data to be processed in the object storage system.

[0074] The preset period may be a time interval pre-set by the user based on the actual application scenario. The preset time range may be pre-set by the user and stored in the computer device. The search server stores initial log data collected by the computer device at various times during the application of the dialogue system.

[0075] The computer device obtains initial log data within a preset time range from the search server according to a preset period and uses the initial log data as the log data to be processed. After obtaining the log data to be processed, the computer device stores the log data to be processed in the Object Storage Service (OSS).

[0076] Step 520: When the log data to be processed in the object storage system meets the detection trigger condition, obtain the log data to be processed from the object storage system.

[0077] A detection trigger condition refers to a condition that requires detection of abnormal log data. This can include the presence of pending log data in the object storage system, the time interval since the last detection reaching a preset time interval threshold, or the amount of pending log data stored in the object storage system reaching a preset storage capacity. This embodiment does not limit the specific content of the detection trigger condition; users can customize it based on their actual application.

[0078] The computer device will regularly monitor whether the pending log data stored in the object storage system meets the detection trigger conditions. If it is detected that the pending log data in the object storage system meets the detection trigger conditions, the computer device will obtain the pending log from the object storage system and then detect any abnormal log data in the pending log.

[0079] In this embodiment, when obtaining log data to be processed, the initial log data of each functional module in the dialogue system is first collected and stored in a search server; then, the initial log data within a preset time range is obtained from the search server as the log data to be processed and stored in an object storage system; and when the detection trigger conditions are met, the log data to be processed is obtained from the object storage system. In this way, before performing anomaly detection on the log data in the dialogue system, the initial log data is collected and stored in the search server, and the log data to be processed is obtained from the search server and stored in the object storage system. This avoids obtaining the log data to be processed from each functional module in the dialogue system during the anomaly detection process, thereby improving the efficiency of anomaly log data detection. Furthermore, the log data to be processed can be persistently stored in the object storage system, avoiding the loss of the log data to be processed, thereby improving the reliability of the anomaly log data detection method.

[0080] In one embodiment, an implementation method for determining abnormal log data based on analysis results includes:

[0081] If the analysis result indicates that there is a problem with the candidate log data, the candidate log data is determined to be abnormal log data.

[0082] After obtaining the analysis results of the user's input of the candidate log data, the computer device determines the content of the analysis results. If the analysis results include a problem with the candidate log data, i.e., the user has determined that the conversation output data in the candidate log data is abnormal, the candidate log data is identified as abnormal log data. If the analysis results include a problem with the model output data, i.e., the user has compared the conversation output data in the candidate log data with the annotation results and determined that the conversation output data is normal.

[0083] In this embodiment, the computer device determines the candidate log data with problems as abnormal log data. This method of determining abnormal log data is fast and easy to implement.

[0084] In one embodiment, Figure 6 As shown, the steps of the abnormal log data detection method also include:

[0085] Step 600: Receive the user's correction processing of abnormal log data to obtain corrected log data.

[0086] When a user encounters abnormal log data, they will correct the abnormal log data and input the corrected log data into a computer device; the computer device then receives the modified log data. In other words, if the user determines that there is a problem with the conversation output data in the candidate log data, they will correct the conversation output data to obtain corrected conversation output data, and the corrected conversation output data and the corresponding conversation input data will be determined as the corrected log data.

[0087] In an optional embodiment, the user may modify the dialogue output data according to the model output data corresponding to the dialogue output data in the abnormal log data.

[0088] In an optional embodiment, the user inputs the computer device with the corrected conversation output data obtained by correcting the conversation output data in the abnormal log data. After receiving the corrected conversation output data, the computer device constructs the corrected log data based on the corrected conversation output data and the corresponding conversation input data.

[0089] Step 610: Store the modified log data in an online database in the dialogue system; the online database is used to obtain the target output dialogue corresponding to the target input dialogue from the online database during the application of the dialogue system.

[0090] After receiving the correction log data input by the user, the computer device stores the correction log data in the online database of the dialogue system. The online database can use Redis (Remote DIctionary Server, database), which is a key-value database with fast data query.

[0091] The online database in the dialogue system is connected in series with the functional modules in the dialogue system, and the online database is located in front of the series structure. That is, during the application of the dialogue system, the online database is accessed first, and then the functional modules are accessed.

[0092] The online database is used to retrieve the target output dialog corresponding to the target input dialog from the online database during the dialogue system application process. Specifically, when the user enters a target input dialog, the online database is first checked to see if a corresponding target output dialog exists. If so, the target output dialog is directly sent to the user. If not, the target output dialog is retrieved through various functional modules. For example, if the correction dialog input query in the correction log data stored in the online database is "Navigate to A," and the correction dialog output data is "map#navigate#location:A," then if the user enters the target input dialog "Navigate to A," the corresponding target output dialog "map#navigate#location:A" is directly retrieved from the online database. If the user enters the target input dialog "Navigate to B," and the online database does not contain any correction input data that matches the target input dialog, the target output dialog is determined through various functional modules in the dialogue system. For example, the NLU module in the dialogue system calculates the corresponding output data through methods such as template matching, network models, and rule parsing.

[0093] In this embodiment, after determining that there is abnormal log data in the log data to be processed, the corrected log data after the user corrects the abnormal log data is received, and the corrected log data is stored in the online database, so that the user can quickly find the target output dialogue from the online database during use, thereby improving the practicality of the dialogue system.

[0094] See Figure 7 An embodiment of the present application provides a method for detecting abnormal log data, the method comprising the following steps:

[0095] Step 700: Collect initial log data of the dialogue system and store the initial log data in the search server;

[0096] Step 710: Acquire initial log data within a preset time range from the search server according to a preset period as log data to be processed, and store the log data to be processed in the object storage system;

[0097] Step 720: If the log data to be processed in the object storage system meets the detection trigger condition, obtain the log data to be processed from the object storage system; the log data to be processed includes the conversation input data and the conversation output data;

[0098] Step 730: Select a target dialogue model from a plurality of sub-dialogue models included in a preset dialogue model according to a data category identifier in the log data to be processed;

[0099] Step 740: Input the dialogue input data into the target sub-dialogue model to obtain model output data;

[0100] Step 750: Determine the conversation output data that is inconsistent with the model output data, and the conversation input data corresponding to the conversation output data, as candidate log data;

[0101] Step 760: Receive analysis results input by the user for the candidate log data, and if the analysis results include problems with the dialogue output data in the candidate log data, determine the candidate log data as abnormal log data;

[0102] Step 770: Receive the user's correction processing of the abnormal log data to obtain the corrected log data;

[0103] Step 780: Store the modified log data in an online database in the dialogue storage system; the online database is used to obtain the target output dialogue corresponding to the target input dialogue from the online database during the application of the dialogue system.

[0104] In an optional embodiment, if Figure 8 As shown, the present application provides an abnormal log data detection device 10, which includes a log collection module 11, a log classification module 12, a large model module 13, a question selection module 14, a labeling platform 15, a data construction module 16 and an online database 17.

[0105] The process of iterating the dialogue system using the abnormal log data detection device includes: a log collection module 11 collects the initial log data of the dialogue system and stores it in the ES cluster; a scheduled task triggers the acquisition of initial log data within a preset time range from the ES cluster as pending log data, and stores this pending log data in the OSS system. A log classification module 12 regularly monitors whether the initial log data (pending log data) within the preset time range is in the OSS system. If so, it determines the category of the pending log data based on the data category identifier in the pending log data and stores the classified pending log data in the OSS system. A large model module 13 analyzes the pending log data and determines the model output data; a problem selection module 14 selects candidate log data whose dialogue output data is inconsistent with the model output data and pushes the candidate log data to the annotation platform 15. Manual annotation of the candidate log data in the annotation platform 15 is performed to identify problematic dialogue output data, thereby obtaining abnormal log data. The dialogue output data is then corrected based on the model output results to obtain the corrected output data. The data construction module constructs the correction log data according to the correction output data and the dialogue input data corresponding to the correction output data, and transmits the correction log data to the online database 17 .

[0106] In an optional embodiment, a detection experiment is performed using the abnormal log data detection method provided by this application, as shown in the following table.

[0107]

[0108] The PV value of the abnormal log data in the table is used to represent the proportion of abnormal log data in the log data to be processed. Using the abnormal log data detection method provided by this application can reduce the PV value of abnormal log data by about one-fifth. In other words, less and less log data needs to be manually labeled. As the accuracy of the dialogue system improves, the PV value of abnormal log data will be further reduced.

[0109] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0110] Based on the same inventive concept, the present application also provides an abnormal log data detection device for implementing the abnormal log data detection method described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more abnormal log data detection device embodiments provided below can be found in the above limitations of the abnormal log data detection method and will not be repeated here.

[0111] In one embodiment, Figure 9 As shown, an abnormal log data detection device 20 is provided, comprising: an acquisition module 21, an input module 22 and a determination module 23, wherein:

[0112] The acquisition module 21 is used to acquire the log data to be processed of the dialogue system; the log data to be processed includes dialogue input data and dialogue output data.

[0113] The input module 22 is used to input the dialogue input data into the preset dialogue model to obtain model output data.

[0114] A determination module 23 is used to determine candidate log data based on the dialogue output data and the model output data;

[0115] The determination module 23 is further configured to obtain analysis results input by the user for candidate log data, and determine abnormal log data based on the analysis results.

[0116] In one embodiment, the input module 22 includes a first selection unit and a first input unit. The first selection unit is configured to select a target sub-dialogue model from multiple sub-dialogue models based on a data category identifier in the log data to be processed. The first input unit is configured to input the dialogue input data into the target sub-dialogue model to obtain model output data.

[0117] In one embodiment, the input module 22 further includes a processing unit, a second selection unit, and a second input unit. The processing unit is configured to classify the log data to be processed and obtain a classification result corresponding to the log data to be processed. The second selection unit is configured to select a target sub-dialogue model from a plurality of sub-dialogue models based on the classification result of the log data to be processed. The second input unit is configured to input the dialogue input data into the target sub-dialogue model and obtain model output data.

[0118] In one embodiment, the acquisition module 21 is specifically used to collect the initial log data of the dialogue system and store the initial log data to the search server; obtain the initial log data within a preset time range from the search server according to a preset period as the log data to be processed, and store the log data to be processed in the object storage system; when the log data to be processed in the object storage system meets the detection trigger conditions, obtain the log data to be processed from the object storage system.

[0119] In one embodiment, the determination module 23 is specifically configured to determine the candidate log data as abnormal log data when the analysis result indicates that the candidate log data has a problem.

[0120] In one embodiment, the abnormal log data detection device 20 further includes a correction module and a storage module. The correction module is configured to receive user corrections to the abnormal log data and obtain corrected log data. The storage module is configured to store the corrected log data in an online database within the dialogue system. The online database is configured to retrieve target output dialogues corresponding to target input dialogues from the online database during application of the dialogue system.

[0121] Each module in the abnormal log data detection device described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0122] In one embodiment, a computer device is provided, whose internal structure diagram can be as follows: Figure 1 As shown. Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0123] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0124] Obtaining log data to be processed by the dialogue system; the log data to be processed includes dialogue input data and dialogue output data;

[0125] Input the dialogue input data into the preset dialogue model to obtain the model output data;

[0126] determining candidate log data based on the dialogue output data and the model output data;

[0127] Obtain the analysis results of the user's input for the candidate log data, and determine the abnormal log data based on the analysis results.

[0128] In one embodiment, when the processor executes the computer program, it further implements the following steps: selecting a target sub-dialogue model from multiple sub-dialogue models based on a data category identifier in the log data to be processed; and inputting the dialogue input data into the target sub-dialogue model to obtain model output data.

[0129] In one embodiment, when the processor executes the computer program, it further implements the following steps: classifying the log data to be processed to obtain a classification result corresponding to the log data to be processed; selecting a target sub-dialogue model from multiple sub-dialogue models based on the classification result of the log data to be processed; and inputting the dialogue input data into the target sub-dialogue model to obtain model output data.

[0130] In one embodiment, when the processor executes the computer program, it also implements the following steps: collecting initial log data of the dialogue system and storing the initial log data in a search server; obtaining initial log data within a preset time range from the search server according to a preset period as log data to be processed, and storing the log data to be processed in an object storage system; when the log data to be processed in the object storage system meets the detection trigger conditions, obtaining the log data to be processed from the object storage system.

[0131] In one embodiment, when the processor executes the computer program, the following steps are further implemented: if the analysis result includes that there is a problem with the candidate log data, the candidate log data is determined to be abnormal log data.

[0132] In one embodiment, when the processor executes the computer program, it also implements the following steps: receiving the user's correction processing of abnormal log data to obtain corrected log data; storing the corrected log data in an online database in the dialogue system; the online database is used to obtain the target output dialogue corresponding to the target input dialogue from the online database during the application of the dialogue system.

[0133] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0134] Obtaining log data to be processed by the dialogue system; the log data to be processed includes dialogue input data and dialogue output data;

[0135] Input the dialogue input data into the preset dialogue model to obtain the model output data;

[0136] determining candidate log data based on the dialogue output data and the model output data;

[0137] Obtain the analysis results of the user's input for the candidate log data, and determine the abnormal log data based on the analysis results.

[0138] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: selecting a target sub-dialogue model from multiple sub-dialogue models based on a data category identifier in the log data to be processed; and inputting the dialogue input data into the target sub-dialogue model to obtain model output data.

[0139] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: classifying the log data to be processed to obtain a classification result corresponding to the log data to be processed; selecting a target sub-dialogue model from multiple sub-dialogue models based on the classification result of the log data to be processed; and inputting the dialogue input data into the target sub-dialogue model to obtain model output data.

[0140] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: collecting the initial log data of the dialogue system and storing the initial log data in the search server; obtaining the initial log data within a preset time range from the search server according to a preset period as the log data to be processed, and storing the log data to be processed in the object storage system; when the log data to be processed in the object storage system meets the detection trigger conditions, obtaining the log data to be processed from the object storage system.

[0141] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: if the analysis result includes that there is a problem with the candidate log data, determining the candidate log data as abnormal log data.

[0142] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: receiving the user's correction processing of abnormal log data to obtain corrected log data; storing the corrected log data in an online database in the dialogue system; the online database is used to obtain the target output dialogue corresponding to the target input dialogue from the online database during the application of the dialogue system.

[0143] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0144] Obtaining log data to be processed by the dialogue system; the log data to be processed includes dialogue input data and dialogue output data;

[0145] Input the dialogue input data into the preset dialogue model to obtain the model output data;

[0146] determining candidate log data based on the dialogue output data and the model output data;

[0147] Obtain the analysis results of the user's input for the candidate log data, and determine the abnormal log data based on the analysis results.

[0148] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: selecting a target sub-dialogue model from multiple sub-dialogue models based on a data category identifier in the log data to be processed; and inputting the dialogue input data into the target sub-dialogue model to obtain model output data.

[0149] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: classifying the log data to be processed to obtain a classification result corresponding to the log data to be processed; selecting a target sub-dialogue model from multiple sub-dialogue models based on the classification result of the log data to be processed; and inputting the dialogue input data into the target sub-dialogue model to obtain model output data.

[0150] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: collecting the initial log data of the dialogue system and storing the initial log data in the search server; obtaining the initial log data within a preset time range from the search server according to a preset period as the log data to be processed, and storing the log data to be processed in the object storage system; when the log data to be processed in the object storage system meets the detection trigger conditions, obtaining the log data to be processed from the object storage system.

[0151] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: if the analysis result includes that there is a problem with the candidate log data, determining the candidate log data as abnormal log data.

[0152] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: receiving the user's correction processing of abnormal log data to obtain corrected log data; storing the corrected log data in an online database in the dialogue system; the online database is used to obtain the target output dialogue corresponding to the target input dialogue from the online database during the application of the dialogue system.

[0153] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0154] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0155] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for detecting abnormal log data, characterized in that: The method comprises: Obtaining log data to be processed of the dialogue system; the log data to be processed includes dialogue input data and dialogue output data; Inputting the dialogue input data into a preset dialogue model to obtain model output data; determining candidate log data according to the dialogue output data and the model output data; An analysis result input by a user for the candidate log data is obtained, and abnormal log data is determined according to the analysis result.

2. The method according to claim 1, characterized in that The dialogue model includes a plurality of sub-dialogue models, and the dialogue input data is input into a preset dialogue model to obtain model output data, including: selecting a target sub-dialogue model from the plurality of sub-dialogue models according to a data category identifier in the log data to be processed; The dialogue input data is input into the target sub-dialogue model to obtain the model output data.

3. The method according to claim 1, characterized in that The dialogue model includes a plurality of sub-dialogue models, and the dialogue input data is input into a preset dialogue model to obtain model output data, including: Classify the log data to be processed to obtain a classification result corresponding to the log data to be processed; selecting a target sub-dialogue model from the plurality of sub-dialogue models according to the classification result of the log data to be processed; The dialogue input data is input into the target sub-dialogue model to obtain the model output data.

4. The method according to claim 1, wherein The obtaining of log data to be processed by the dialogue system includes: Collecting initial log data of the dialogue system and storing the initial log data in a search server; Acquire initial log data within a preset time range from the search server according to a preset period as the log data to be processed, and store the log data to be processed in the object storage system; When the log data to be processed in the object storage system meets the detection trigger condition, the log data to be processed is obtained from the object storage system.

5. The method according to any one of claims 1 to 4, characterized in that Determining abnormal log data according to the analysis result includes: If the analysis result indicates that there is a problem with the candidate log data, the candidate log data is determined as the abnormal log data.

6. The method according to any one of claims 1 to 4, characterized in that The method further comprises: receiving a correction process performed by a user on the abnormal log data to obtain corrected log data; The correction log data is stored in an online database in the dialogue system; the online database is used to obtain the target output dialogue corresponding to the target input dialogue from the online database during the application of the dialogue system.

7. An abnormal log data detection device, characterized in that: The device comprises: An acquisition module, configured to acquire log data to be processed of a dialogue system; the log data to be processed includes dialogue input data and dialogue output data; An input module, configured to input the dialogue input data into a preset dialogue model to obtain model output data; a determination module, configured to determine candidate log data based on the dialogue output data and the model output data; The determination module is further configured to obtain an analysis result input by a user for the candidate log data, and determine abnormal log data according to the analysis result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images