Dialogue quality inspection method and device

Abstract

According to the dialogue quality inspection method and device provided by the embodiment of the invention, the first-level conventional question and answer library and the second-level professional knowledge library of the hierarchical knowledge base are preset, and the knowledge information corresponding to the user question is retrieved from the hierarchical knowledge base according to the hierarchical sequence; and performing quality inspection on question answers of the user questions based on the retrieved knowledge information. According to the technical scheme of the embodiment of the specification, quality inspection can be accurately carried out on different types of dialogue data to be inspected through a layered quality inspection method, and the risk of missing inspection is remarkably reduced.

Description

Technical Field

[0001] This specification relates to the field of large model technology, and in particular to a dialogue quality inspection method and dialogue quality inspection equipment. Background Technology

[0002] With the development of internet technology and the rapid growth of transaction volume and customer base on online platforms, the amount of service dialogue data, such as ticket service data, is increasing dramatically. Therefore, how to conduct quality inspection of service dialogue data on online platforms has become a key focus.

[0003] In existing technical solutions, some basic service specification rules are pre-set. Based on these rules, regular expressions are used to check whether the dialogue data to be inspected conforms to the service specifications. However, this approach can only detect fixed keywords using regular expressions, making it difficult to accurately inspect the entire dialogue data. Therefore, how to accurately inspect the dialogue data to be inspected has become a pressing technical challenge.

[0004] The information in the background section is merely information known only to the inventor and does not imply that such information had entered the public domain before the date of this application, nor does it imply that it can be considered prior art in this disclosure. Summary of the Invention

[0005] This manual provides a dialogue quality inspection method and equipment, which can accurately inspect the dialogue data to be inspected and reduce the risk of missed inspections.

[0006] Firstly, this specification provides a dialog-based quality inspection method, including: Obtain at least one question-and-answer pair to be inspected in the target scenario, wherein the question-and-answer pair to be inspected includes a user question in the target scenario and the question answer corresponding to the user question. Knowledge information corresponding to the user's question in the question-answer pair to be checked is retrieved from a hierarchical knowledge base in hierarchical order. The hierarchical knowledge base includes a first-level general question-answer base and a second-level professional knowledge base. The question answers of the question-answer pair to be inspected are quality checked based on the retrieved knowledge information.

[0007] In some example embodiments, based on the above scheme, retrieving knowledge information corresponding to the user question of the question-answer pair to be checked from the hierarchical knowledge base in hierarchical order includes: Retrieve regular questions from the regular question-and-answer database at the first level that correspond to the user's question in the question-and-answer pair to be tested; If the general question is not found, then the professional knowledge corresponding to the user question is retrieved from the professional knowledge base of the second level.

[0008] In some example embodiments, based on the above scheme, retrieving a regular question corresponding to the user question of the question-answer pair to be tested from the regular question-answer database at the first level includes: Determine the semantic similarity between the user question in the question-answer pair to be inspected and the regular question in the regular question-answer database; If the semantic similarity corresponding to the regular question is greater than or equal to a preset similarity threshold, then the answer to the regular question corresponding to the regular question is obtained from the regular question-and-answer database; If the semantic similarity of each of the aforementioned regular questions is less than the preset similarity threshold, then it is determined that no regular question was retrieved from the regular question-and-answer database.

[0009] In some example embodiments, based on the above scheme, determining the semantic similarity between the user question of the question-answer pair to be examined and the regular questions in the regular question-answer database further includes: Determine the preset weights for the common problems, where the preset weights are weights pre-set based on the importance of the common problems; and The semantic similarity between the user question and the regular question is weighted based on the preset weight.

[0010] In some example embodiments, based on the above scheme, the quality inspection of the question answer of the question-answer pair to be inspected based on the retrieved knowledge information includes: If the regular question is found, the answer to the question corresponding to the user question is inspected based on the regular question answer according to the preset quality inspection rules.

[0011] In some example embodiments, based on the above scheme, retrieving the professional knowledge corresponding to the user's question from the second-level professional knowledge base includes: The professional knowledge corresponding to the user's question is retrieved from the second-level professional knowledge base using a multi-hop retrieval method.

[0012] In some example embodiments, based on the above scheme, retrieving professional knowledge corresponding to the user's question from the second-level professional knowledge base through multi-hop retrieval includes: The user's question is broken down into several corresponding sub-questions; Based on the first sub-question, retrieve the sub-professional knowledge corresponding to the first sub-question from the professional knowledge base; The current query for the current round is generated based on the sub-problem of the current round and the sub-professional knowledge of the previous round; and Based on the current query, retrieve the sub-professional knowledge corresponding to the current query from the professional knowledge base.

[0013] In some example embodiments, based on the above scheme, the quality inspection of the question answer of the question-answer pair to be inspected based on the retrieved knowledge information includes: The retrieved knowledge information and the question-answer pair to be checked are input into a preset language model to guide the preset language model to perform quality checks on the question answers based on the knowledge information.

[0014] In some example embodiments, based on the above scheme, the step of inputting at least the retrieved knowledge information and the question-answer pair to be tested into a preset language model includes: The context information corresponding to the preset language model is obtained from the temporary storage area. At least the context information, the retrieved knowledge information, and the question-answer pair to be checked are input into the preset language model. The temporary storage area is used to temporarily store the context information in a vectorized manner.

[0015] In some example embodiments, based on the above scheme, guiding the preset language model to perform quality checks on the question answer based on the knowledge information includes: Based on the aforementioned knowledge information and the question-and-answer pairs to be inspected, quality inspection prompts are generated using a quality inspection prompt template. This template is a pre-configured prompt template that combines the aforementioned knowledge information and the question-and-answer pairs to generate quality inspection results. Based on the quality inspection prompts, the quality inspection results corresponding to the question answers are generated through the preset language model.

[0016] In some example embodiments, based on the above scheme, the preset language model is a language model obtained by knowledge distillation of a large language model.

[0017] In some example embodiments, based on the above scheme, the method further includes: Multiple question-answer pair samples for the target scenario are generated based on the large language model. The preset language model is trained based on the multiple question-answer pairs.

[0018] In some example embodiments, based on the above scheme, the output of the preset language model includes the confidence level of the quality inspection result for the question-answer pair to be inspected, and the method further includes: If the confidence level is less than the first confidence level threshold, then manual review of the question-answer pair to be inspected or quality inspection of the question answer of the question-answer pair to be inspected based on the large language model is triggered.

[0019] In some example embodiments, based on the above scheme, the method further includes A first corrected sample is generated based on the review results of the manual review, and the preset language model is trained using the first corrected sample.

[0020] In some example embodiments, based on the above scheme, after performing quality inspection on the question-answer pair based on the large language model, the method further includes: If the confidence level of the quality inspection result of the large language model for the question-answer pair sample is less than the second confidence level threshold, then the annotation information for the question-answer pair sample is obtained, and a second corrected sample is generated based on the annotation information, so as to train the preset language model through the second corrected sample; If the confidence level of the large language model for the quality inspection results of the question-answer pair samples is greater than or equal to the second confidence threshold, then the preset language model is fine-tuned based on the quality inspection results of the large language model for the question-answer pair samples.

[0021] In some example embodiments, based on the above scheme, the method further includes: Obtain historical dialogue records in the target scenario, the historical dialogue records including multiple question-answer pairs; Multi-dimensional clustering is performed on the multiple question-answer pairs to generate regular questions in the regular question-answer library.

[0022] In some example embodiments, based on the above scheme, the method further includes: Structured knowledge extraction is performed on the domain expertise in the target scenario to generate the expertise in the expertise base.

[0023] Secondly, this specification also provides a dialogue quality inspection device, comprising: at least one storage medium storing at least one instruction set for performing dialogue quality inspection processing; and at least one processor communicatively connected to the at least one storage medium, wherein, when the dialogue quality inspection device is running, the at least one processor reads the at least one instruction set and executes the dialogue quality inspection method described in the first aspect of this specification according to the instructions of the at least one instruction set.

[0024] As can be seen from the above technical solutions, the dialogue quality inspection method and equipment provided in the embodiments of this specification, on the one hand, pre-set a first-level conventional question-and-answer database and a second-level professional knowledge database in a hierarchical knowledge base. The conventional question-and-answer database can efficiently handle conventional questions and answers that occur frequently, while the professional knowledge database can cover professional domain knowledge of the target scenario. On the other hand, knowledge information corresponding to the user's question is retrieved from the hierarchical knowledge base in hierarchical order, and the question answers to the user's question are quality inspected based on the retrieved knowledge information. The hierarchical quality inspection method can accurately inspect different types of dialogue data to be inspected, such as dialogue data with semantic differences, which significantly reduces the risk of missed detections.

[0025] Other functions of the dialogue quality inspection methods and apparatus provided in this specification will be partially listed in the following description. The figures and examples described below will be readily apparent to those skilled in the art. The inventive aspects of the dialogue quality inspection methods and apparatus provided in this specification can be fully understood through practice or use of the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A schematic diagram of the implementation environment of a dialogue quality inspection method provided in an embodiment of this specification is shown; Figure 2 A hardware structure diagram of a dialog quality inspection device 200 provided according to an embodiment of this specification is shown; Figure 3 A flowchart of a dialog-based quality inspection method provided according to some embodiments of this specification is shown; Figure 4 A schematic diagram of a process for performing dual-model quality inspection according to some embodiments of this specification is shown; Figure 5 This specification illustrates a flowchart of a tiered quality inspection process according to some embodiments thereof; and Figure 6 A schematic diagram of the model optimization process provided according to some embodiments of this specification is shown. Detailed Implementation

[0028] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0029] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0030] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0031] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0032] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0033] Large Language Model (LLM): A large model is an artificial intelligence model that is trained on a large amount of text data and is capable of understanding and generating natural language text.

[0034] Prompt: A prompt or stimulus, typically used in large models to guide user input or elicit a specific program response. It is frequently used in large model programming and user interface design.

[0035] RAG (Retrieval-Augmented Generation): This method helps large models generate information-resource-based answers by retrieving relevant information resources before generating a response.

[0036] Fine-tuning large models: Further training a pre-trained language model using a domain-specific dataset to improve the performance of the language model in that specific domain.

[0037] In existing technical solutions, regular expressions are used to check whether the dialogue data to be inspected conforms to the service specifications according to preset service specification rules. However, this technical solution has two drawbacks. First, it can only detect fixed keywords through regular expressions, making it difficult to handle dialogue data with semantic differences. Second, due to insufficient coverage of professional domain knowledge, it is difficult to accurately inspect the dialogue data.

[0038] Based on the above, embodiments of this specification provide a dialogue quality inspection method and a dialogue quality inspection device. On one hand, a first-level conventional question-and-answer database and a second-level professional knowledge database are pre-set. The conventional question-and-answer database efficiently handles frequently occurring conventional questions and answers, while the professional knowledge database covers the professional domain knowledge of the target scenario. On the other hand, knowledge information corresponding to the user's question is retrieved from the hierarchical knowledge base according to hierarchical order. Based on the retrieved knowledge information, the answers to the user's question are quality inspected. This hierarchical quality inspection method can accurately inspect different types of dialogue data to be inspected, such as dialogue data with semantic differences, significantly reducing the risk of missed detections.

[0039] The technical solutions of the embodiments of this specification will now be described in detail with reference to the accompanying drawings.

[0040] Figure 1 A schematic diagram of the implementation environment of a dialog-based quality inspection method provided in an embodiment of this specification is shown.

[0041] See Figure 1 As shown, the implementation environment 100 may include a terminal 110, a server 130, and a database 140.

[0042] Terminal 110 is connected to server 130 via wireless network or wired network 120. Terminal 110 can be a tablet computer, laptop computer, or desktop computer, but is not limited to these.

[0043] Terminal 110 may store data or instructions for executing the dialogue quality inspection method described in this specification. Terminal 110 may include hardware devices with data processing capabilities and the necessary programs to drive the hardware devices.

[0044] Server 130 is a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Server 130 provides background services for applications running on terminal 110.

[0045] Server 130 is equipped with an integrated development platform (IDE). An IDE, also known as an integrated development environment, is an application that provides a program development environment, typically including tools such as a code editor, compiler, debugger, and graphical user interface. Developers can write program code (i.e., program development) on the IDE. The IDE server can be a dedicated computing device used to implement dialog-based quality control methods. Server 130 can communicate with both terminal 110 and database 140.

[0046] Furthermore, server 130 may store data or instructions for executing the dialogue quality inspection method described in this specification. Server 130 may include hardware devices with data processing capabilities and the necessary programs required to drive the hardware devices. Of course, server 130 may also be merely a hardware device with data processing capabilities, or simply a program running on the hardware device. In some embodiments, server 130 may also be deployed as a plug-in on terminal 110, in which case server 130 stores data or instructions for executing the dialogue quality inspection method corresponding to terminal 110 described in this specification.

[0047] Database 140 may store data and / or instructions. In some embodiments, database 140 may store historical conversation records, etc. In some embodiments, database 140 may store data and / or instructions executed by server 130 or used to execute the conversation quality inspection methods described in this specification. Terminal 110 and server 130 have access to database 140, and terminal 110 and server 130 may access data or instructions stored in database 140 via a network. In some embodiments, database 140 may be directly connected to terminal 110 and server 130. In some embodiments, database 140 may be part of server 130. In some embodiments, database 140 may include mass storage, removable storage, volatile read-write memory, read-only memory (ROM), or similar content, or any combination thereof. Exemplary mass storage may include non-transitory storage media such as disks, optical discs, and solid-state drives. Exemplary removable storage may include flash drives, floppy disks, optical discs, memory cards, zip disks, magnetic tapes, etc. Typical volatile read-write memory may include random access memory (RAM). Example RAMs may include dynamic RAM (DRAM), dual date rate synchronous dynamic RAM (DDRSDRAM), static RAM (SRAM), thyristor RAM (T-RAM), and zero-capacitance RAM (Z-RAM), etc. Exemplary ROMs may include mask ROM (MROM), programmable ROM (PROM), virtual programmable ROM (PEROM), electronically programmable ROM (EEPROM), optical disc ROM (CD ROM), and digital multifunction disk ROM, etc.

[0048] Those skilled in the art will understand that the number of terminals described above can be more or less. For example, there may be only one terminal, or there may be dozens or hundreds of terminals, or even more, in which case other terminals may also be included in the above implementation environment. This specification does not limit the number of terminals or the type of devices in the embodiments.

[0049] After introducing the implementation environment of the embodiments of this specification, the application scenarios of the embodiments of this specification will be described below in conjunction with the above implementation environment. In the following description, the terminal is also the terminal 110 in the above implementation environment, and the server is also the server 130 in the above implementation environment. The technical solutions provided by the embodiments of this specification can be applied in various customer service scenarios, such as insurance sales services, e-commerce customer service, and product after-sales service.

[0050] Taking the technical solution provided in the embodiments of this specification as an example in an insurance sales service scenario, terminal 110 obtains at least one question-and-answer pair to be inspected in the insurance sales scenario. The question-and-answer pair to be inspected includes user questions in the insurance sales scenario and the corresponding answers to the user questions. Knowledge information corresponding to the user questions in the question-and-answer pair to be inspected is retrieved from the hierarchical knowledge base in hierarchical order. The hierarchical knowledge base includes a first-level general question-and-answer base and a second-level professional knowledge base. And quality inspection is performed on the answers to the question-and-answer pair to be inspected based on the retrieved knowledge information.

[0051] It should be noted that the above description is based on the application of the technical solution provided in the embodiments of this specification to the insurance sales service scenario. The technical solution provided in the embodiments of this specification can also be applied to other appropriate scenarios, such as e-commerce customer service or product after-sales service, and the implementation process belongs to the same inventive concept as described above, and will not be repeated here.

[0052] It should be noted that the steps in the dialogue quality inspection method in the example embodiments of this specification may be partially performed by the client, partially performed by the server, or entirely performed by the server or entirely by the client. This specification does not impose any special limitations on this.

[0053] based on Figure 1 The implementation environment shown below will be combined with... Figures 2-6 This specification provides a detailed description of the dialogue quality inspection method and equipment provided in the embodiments. It should be noted that the above-described implementation environment is provided only to facilitate understanding of the spirit and principles of this specification, and the embodiments are not limited in any way. Rather, the embodiments can be applied to any applicable scenario.

[0054] Figure 2 This is a schematic diagram of a dialogue quality inspection device 200 provided according to some embodiments of this specification. The dialogue quality inspection device 200 can execute the dialogue quality inspection method described in this specification. The dialogue quality inspection method is described in other parts of this specification. The dialogue quality inspection device 200 can be a general-purpose computer or a special-purpose computer. For example, the dialogue quality inspection device 200 can be a server, a personal computer, a portable computer (such as a laptop computer, tablet computer, etc.), or other dialogue quality inspection devices with computing capabilities. Of course, the dialogue quality inspection device can be... Figure 1 The terminal 110 and / or server 130 can also be terminal devices used by multiple developers to develop programs on an integrated development platform.

[0055] The dialogue quality inspection device described in this specification may include one or more of the following components: processor 210, memory 220, input device 230, output device 240, and bus 250. The processor 210, memory 220, input device 230, and output device 240 may be connected to each other via bus 250.

[0056] Processor 210 may include one or more processing cores. Processor 210 connects to various parts within the entire dialogue quality inspection device using various interfaces and lines, and executes the dialogue quality inspection method described in this specification by running or executing instructions, programs, code sets, or instruction sets stored in memory 220, and by calling data stored in memory 220. Optionally, processor 210 may be implemented using at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). Processor 210 may integrate one or a combination of central processing unit (CPU), graphics processing unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 210, but may be implemented separately using a communication chip.

[0057] The memory 220 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 220 may include non-transitory computer-readable storage medium. The memory 220 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 220 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (e.g., touch function, sound playback function, image playback function, etc.), instructions for implementing the various method embodiments described below, etc. The operating system may be an Android system, including systems deeply developed based on the Android system, an iOS system, including systems deeply developed based on the iOS system, or other systems.

[0058] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to establish data communication between the third-party applications and the operating system. This would allow the operating system to obtain the current scenario information of the third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0059] The input device 230 is used to receive input instructions or data, and includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 240 is used to output instructions or data, and includes, but is not limited to, a display device and a speaker. In one example, the input device 230 and the output device 240 can be combined, and both the input device 230 and the output device 240 can be a touch display screen.

[0060] In addition, those skilled in the art will understand that the structure of the dialogue quality inspection device shown in the above figures does not constitute a limitation on the dialogue quality inspection device. The dialogue quality inspection device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the dialogue quality inspection device may also include radio frequency circuits, input units, sensors, audio circuits, Wireless Fidelity (WiFi) modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.

[0061] Figure 3 A flowchart of a dialogue quality inspection method according to an embodiment of this specification is shown. As previously described, the dialogue quality inspection device 200 can execute the dialogue quality inspection method of the embodiment of this specification. Specifically, the processor 210 can read the instruction set stored in its local storage medium and then execute the dialogue quality inspection method of the embodiment of this specification according to the instructions of the instruction set. Hereinafter, steps S310 to S330 of the dialogue quality inspection method will be described in detail with reference to the accompanying drawings.

[0062] Reference Figure 3 As shown, in step S310, at least one question-answer pair to be checked is obtained in the target scene.

[0063] In the example embodiment, the target scenario includes insurance sales service scenarios, e-commerce customer service scenarios, wealth management product service scenarios, or product after-sales service scenarios, etc. The dialogue quality inspection device 200 acquires at least one question-and-answer pair to be inspected within the target scenario. The question-and-answer pair includes a user question and its corresponding answer within the target scenario. For example, it acquires at least one question-and-answer pair from the database of the network platform corresponding to the target scenario. The network platform may include an insurance service platform, a financial service platform, or an e-commerce service platform, etc. The data format of the question-and-answer pair to be inspected may include text data or voice data.

[0064] The following is a detailed explanation of the case where the question-and-answer pair to be checked is text data or voice data, using specific embodiments as examples.

[0065] Example 1: In this example embodiment, the question-and-answer pair to be inspected is text data. The dialogue quality inspection device 200 obtains historical dialogue text data from the database of the network platform corresponding to the target scenario, divides the historical dialogue text data into independent "question-answer" text units according to rounds, and identifies the text unit data corresponding to the user and customer service through identity tags; based on the above text unit data, at least one question-and-answer pair to be inspected is constructed, for example, the text unit data is structured and stored in JSON format.

[0066] For example, the dialogue quality inspection device 200 collects historical dialogue text data from the database of the insurance sales platform. For instance, the following historical dialogue text data: "[10:10] [User] Does this critical illness insurance include cancer coverage? [10:13] [Customer Service] Yes, this product covers 100 critical illnesses, including all malignant tumors. [10:17] [User] If I get cancer, how much will I get in compensation? [10:00:20] [Customer Service] The maximum compensation is 500,000 yuan, and it's paid out immediately upon diagnosis, without waiting for treatment." Based on the user's identity tag and the dialogue time, the historical dialogue text data is divided into independent "question-answer" text units according to the rounds. At least one question-and-answer pair is constructed based on the text unit data. For example, question-and-answer pair 1: { Question: "Does this critical illness insurance include cancer coverage?"; Answer: "Yes, our product covers 100 critical illnesses, including all malignant tumors."; Question and answer to be checked 2: Question: "If I get cancer, how much will I get in compensation?"; Answer: "The maximum compensation is 500,000 yuan, and it's paid out as soon as I'm diagnosed, without waiting for treatment."

[0067] Example 2: In this example embodiment, the question-and-answer pair to be inspected is voice data. The dialogue quality inspection device 200 obtains historical dialogue voice data from the database of the network platform corresponding to the target scenario, converts the historical dialogue voice data into dialogue text data through a speech recognition model, and constructs at least one question-and-answer pair to be inspected based on the aforementioned dialogue text data. The implementation method of constructing the question-and-answer pair to be inspected based on text data has been described above and will not be repeated here. It should be noted that the speech recognition model can be a Transformer-based machine learning model, or a model with other appropriate structures, such as a recurrent neural network model, within the scope of the embodiments in this specification.

[0068] In step S320, knowledge information corresponding to the user's question and the question-answer pair to be checked is retrieved from the hierarchical knowledge base in hierarchical order.

[0069] In the example embodiment, the hierarchical knowledge base is a knowledge storage and retrieval architecture based on question complexity and knowledge depth. The hierarchical knowledge base includes a first-level general question-and-answer base and a second-level professional knowledge base. The general question-and-answer base refers to a structured collection of knowledge addressing high-frequency, general, and standardized questions, typically stored in the form of "question-answer" pairs, used to quickly respond to common user inquiries and improve interaction efficiency. The content of the general question-and-answer base includes at least one of the following: high-frequency questions, standard question-and-answer scripts, and common FAQs. Taking an insurance sales scenario as an example, the general question-and-answer base includes: Question 1: "Who can apply for insurance?"; Answer 1: "People aged 18-55 in good health can apply for insurance."; Question 2: "Does this critical illness insurance have a waiting period?"; Answer 2: "Yes, the waiting period is 90 days." The professional knowledge base refers to an unstructured or semi-structured collection of knowledge addressing complex, professional, and compliance-critical questions, typically derived from formal documents (such as contracts, regulations, and policy documents), used to support deep reasoning, accurate answers, and compliance verification. The professional knowledge base includes at least one of the following: insurance terms, laws and regulations, regulatory policies, product manuals, and claims details.

[0070] In the example embodiment, the dialogue quality inspection device 200 acquires the user's question for the question-and-answer pair to be inspected, and retrieves the corresponding general questions from the first-level general question-and-answer database of the hierarchical knowledge base. If no general questions are found, it retrieves the corresponding professional knowledge from the second-level professional knowledge base. This hierarchical retrieval method using the hierarchical knowledge base enables a "start with the easy, then move to the difficult, and progress step by step" retrieval strategy, ensuring both response speed and the accuracy and compliance of the answers.

[0071] For example, the dialogue quality inspection device 200 encodes the user question of the question-answer pair to be inspected into a user question vector, and matches the user question vector with the regular question vector of regular questions in the regular question-answer database. If the match is successful, the regular question answer corresponding to the regular question vector is returned; if all regular questions in the regular question-answer database fail to match, it is determined that no regular question was found in the regular question-answer database, and the professional knowledge corresponding to the user question is retrieved from the second-level professional knowledge base.

[0072] In some example embodiments, the dialogue quality inspection device 200 determines the semantic similarity between the user's question and regular questions in a regular question-and-answer database for the question-and-answer pair to be inspected. If the semantic similarity corresponding to a regular question is greater than or equal to a preset similarity threshold, the answer to the regular question is retrieved from the regular question-and-answer database. If the semantic similarity corresponding to each regular question is less than the preset similarity threshold, it is determined that no regular question was retrieved from the regular question-and-answer database. Semantic similarity refers to the degree of closeness between two text segments in terms of semantics or meaning. It measures whether two language expressions convey the same or similar ideas, intentions, or information, rather than just literal or vocabulary overlap. Semantic similarity is an indicator used in natural language processing to quantify the degree of similarity between two text segments at the semantic level. Its value is usually between 0 and 1, for example, 1 indicates that the semantics are completely identical, and 0 indicates that the semantics are completely unrelated. The preset similarity threshold can be 0.8, and the preset similarity threshold can be different for different target scenarios.

[0073] For example, assuming a predetermined similarity threshold of 0.8 and a semantic similarity of cosine similarity, the dialogue quality inspection device 200 encodes the user question in the question-answer pair to be inspected using a pre-trained natural language processing model such as Sentence-BERT, obtaining a user question vector V_user. It then determines the cosine similarity between the user question vector and regular question vectors in the regular question-answer database. If the cosine similarity is greater than 0.8, the regular question answer corresponding to that regular question is retrieved from the regular question-answer database. If the semantic similarity of each regular question is less than 0.8, it is determined that no regular question was retrieved from the regular question-answer database.

[0074] According to the technical solution in the above example embodiment, by determining the semantic similarity between user questions and regular questions in a regular question-and-answer database, it is possible to automatically identify questions that are "phrased but synonymous", and then retrieve and process user questions with semantic differences.

[0075] Furthermore, since different routine questions have different levels of importance, the dialogue quality inspection device 200 determines the preset weights of routine questions in the routine question-and-answer database. These preset weights are pre-set based on the importance of the routine questions. The device also performs a weighted calculation on the semantic similarity between user questions and routine questions based on these preset weights. For example, disclaimer questions and health disclosure questions have higher weights, while payment questions have lower weights.

[0076] According to the technical solution in the above example embodiment, by introducing a preset weighting mechanism, different weights are assigned to common problems, and weighted calculations are performed during semantic matching, thereby improving the identification priority and quality inspection accuracy of key problems.

[0077] Furthermore, in the example embodiment, if no regular question is found in the regular question-and-answer database, the dialogue quality inspection device 200 retrieves the professional knowledge corresponding to the user's question from the professional knowledge base, for example, obtaining the top 5 pieces of knowledge information retrieved from the professional knowledge base. For instance, the dialogue quality inspection device 200 retrieves the professional knowledge corresponding to the user's question from the second-level professional knowledge base using a multi-hop retrieval method. Multi-hop retrieval refers to the process of gradually retrieving relevant information from the knowledge base through multiple reasoning steps, and piecing together the information retrieved in each step to form a complete answer. In each hop query, a more accurate query is generated based on the currently known information; the results of the previous round of retrieval are used to enrich and reconstruct the next round of query.

[0078] For example, the dialogue quality inspection device 200 decomposes the user's question into multiple corresponding sub-questions; retrieves the sub-professional knowledge corresponding to the first sub-question from the professional knowledge base; generates the current query corresponding to the current round based on the sub-questions of the current round and the sub-professional knowledge of the previous round; and retrieves the sub-professional knowledge corresponding to the current query from the professional knowledge base based on the current query. Taking an insurance sales scenario as an example, consider the user's question: "I have mild fatty liver and hypertension, with my blood pressure controlled at around 138 / 88. Can I buy critical illness insurance?" The dialogue quality inspection device 200 breaks down the user's question into several sub-questions: Sub-question 1: "What are the underwriting standards for fatty liver and hypertension affecting critical illness insurance coverage?"; Sub-question 2: "What are the underwriting rules for the coexistence of fatty liver and hypertension?". The first hop: Based on the first sub-question, the device queries the relevant sub-professional knowledge from the professional knowledge base. For example, it searches the chapters on "fatty liver grading standards" and "hypertension grading standards" in the "Critical Illness Insurance Underwriting Manual." The retrieved sub-professional knowledge includes: fatty liver is divided into mild, moderate, and severe; the standard for grade 1 hypertension is 140-159 / 90-99. The second hop: Based on sub-question 2 and the sub-professional knowledge retrieved in the first hop, the device determines that the user has grade 1 hypertension and generates the current query for the current round: "What are the underwriting rules for the coexistence of mild fatty liver and grade 1 hypertension?" According to the "Underwriting Rules for Multiple Diseases", the sub-professional knowledge is that "the combination of mild fatty liver and grade 1 hypertension usually requires an increase in premium of 15%-25%".

[0079] According to the technical solution in the above example embodiment, the long-term linked analysis of professional knowledge is realized through RAG multi-hop retrieval, which improves the accuracy of retrieval of professional knowledge.

[0080] In step S330, the question answer of the question-answer pair to be checked is performed based on the retrieved knowledge information.

[0081] In some example embodiments, if a regular question is retrieved, the dialogue quality inspection device 200 performs quality inspection on the question-answered question based on the regular question's corresponding answer. For example, the dialogue quality inspection device 200 performs semantic consistency analysis between the customer service's question answer corresponding to the user's question and the retrieved regular question answer; if the semantics are consistent, the question answer corresponding to the user's question is determined to be correct, and the quality inspection passes; if the semantics are inconsistent, the question answer corresponding to the user's question is determined to be incorrect, and the quality inspection fails.

[0082] Furthermore, pre-set quality inspection rules are established. These rules are expressions composed of logical operators and conditions. Logical operators include AND, OR, NOT, equal to, and not equal to. Conditions consist of a check range and operators. The check range includes sentences M to N, where M and N range from 1 to N. Operators are used to evaluate each sentence within the check range; for example, operators include regular expressions and semantic matching. The dialogue quality inspection device 200 performs quality inspection on the answers to user questions based on the answers to regular questions, using the pre-set quality inspection rules. For example, if the customer service representative's first sentence is "Hello," the regular question's answer is "Hello," the pre-set quality inspection rule's check range is "the first sentence," and the operator is the semantic matching keyword "Hello." The pre-set quality inspection rules then perform quality inspection on the customer service representative's answer to the user question.

[0083] According to the technical solution in the above example embodiment, by pre-setting quality inspection rules to perform quality inspection on the answers to the questions to be inspected, it is possible to efficiently and accurately perform quality inspection on the answers to high-frequency questions.

[0084] In some example embodiments, if no regular questions are retrieved, the dialogue quality inspection device 200 at least inputs the retrieved knowledge information and the question-answer pair to be inspected into a preset language model to guide the preset language model to perform quality inspection on the question answers of the question-answer pair based on the knowledge information. The preset language model can be a large language model or a small language model obtained by distillation of a large language model. The preset language model can adopt the Transformer architecture or its variants (such as GPT, BERT, etc.), which utilizes the attention mechanism to achieve global modeling of sequence data and can efficiently handle long-distance dependencies. The core capabilities of the preset language model include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Its usage methods typically include two modes: direct inference and fine-tuning. In direct inference mode, the user guides the large language model to generate specific outputs by designing prompts. Prompts can be task descriptions or instructions in text form, used to stimulate the semantic understanding and generation capabilities of the large language model. In fine-tuning mode, the pre-defined language model is further trained on a small dataset within a specific domain to optimize its performance on a particular task. The pre-defined language model can also possess the ability to understand and generate data from other modalities (such as visual and audio data); in this case, the language model can be called a multimodal language model (MLLM). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound.

[0085] In an example embodiment, the dialogue quality inspection device 200 generates quality inspection prompts based on retrieved knowledge information and the question-answer pair to be inspected, combined with a quality inspection prompt template. The quality inspection prompt template is a pre-configured template that combines knowledge information and the question-answer pair to generate quality inspection results. Based on the quality inspection prompts, the device generates the corresponding quality inspection results for the question answers using a preset language model. Taking an insurance sales scenario as an example, suppose a user asks: "I have mild fatty liver and high blood pressure, with my blood pressure controlled at around 138 / 88. Can I buy critical illness insurance?" The answer is: "Yes, you can." The knowledge information states: "Mild fatty liver and stage 1 hypertension, two mild symptoms, usually require an increase in premium of 15%-25%." The quality inspection result of the preset language model is: "Partially correct, further explanation of the increased premium is needed." It should be noted that each target scenario can correspond to a preset prompt template. The prompt templates for different target scenarios are not always the same; that is, different target scenarios can have different prompt templates, but some similar scenarios can correspond to the same prompt template. The prompt template can contain multiple prompt fields, such as knowledge information, user questions, question answers, output requirements, background information, and task instructions.

[0086] In some example implementations, the preset language model is a language model obtained by knowledge distillation of a large language model. Knowledge distillation is a model compression and transfer learning technique that transfers knowledge from a large language model to a lighter, smaller model, allowing the smaller model to achieve performance close to that of the large model. If the large language model can correctly identify problems that the smaller language model (e.g., the preset language model) misidentifies, the judgment logic of the large language model can be used as knowledge to train the smaller model, enabling the smaller model to achieve the performance of the large model as closely as possible on the given sample set (where the smaller model misidentifies and the large model correctly identifies). Using the language model obtained through knowledge distillation for inference can reduce model deployment costs and improve the processing efficiency of inference tasks.

[0087] For example, the dialogue quality inspection device 200 generates multiple question-answer pair samples for a target scenario based on a large language model. These sample samples include the large language model's response strategies for questions within the target scenario. Based on these multiple question-answer pair samples, a pre-defined language model is trained using knowledge distillation. For instance, target scenarios might include insurance sales services, e-commerce customer service, financial product services, or after-sales service. The dialogue quality inspection device 200 generates multiple question-answer pair samples for each scenario using pre-defined prompt word templates. Based on these sample samples, the pre-defined language model is fine-tuned using knowledge distillation.

[0088] According to the technical solution in the above example embodiment, high-quality labeled data is generated by utilizing the reasoning ability of a large language model, and the ability of the large language model is transferred to a lightweight preset language model through knowledge distillation technology.

[0089] In some example embodiments, the output of the preset language model includes a confidence score for the question-and-answer pair to be tested. The confidence score is a reliability assessment metric of the language model's generated answers or judgment results, typically a floating-point number in the range [0, 1]. Further, a first confidence threshold is preset, for example, 0.8. This first confidence threshold can be set according to the needs of the target scenario; for example, the confidence threshold is higher for critical illness insurance and lower for life insurance. If the confidence score of the preset language model's quality inspection result for the question-and-answer pair to be tested is less than the first confidence threshold, the dialogue quality inspection device 200 triggers manual review of the question-and-answer pair; if the confidence score is greater than or equal to the first confidence threshold, it indicates that the question-and-answer pair to be tested passes the quality inspection.

[0090] For example, suppose the question-and-answer pair to be checked is "[User] Does this critical illness insurance include cancer coverage? [Customer Service] Yes, our product covers 100 critical illnesses, including all malignant tumors." The output of the preset semantic model is {"Result": "Correct", "Confidence": 0.91}. If the confidence level of 0.91 is greater than the first confidence threshold of 0.8, it means that the question-and-answer pair to be checked has passed the quality inspection and no manual intervention is required.

[0091] According to the technical solution in the above example embodiment, on the one hand, by setting the confidence level of the language model for the question-answer pair to be inspected, cases with lower confidence levels are handled manually to avoid the risk of missed detection caused by AI misjudgment; on the other hand, manual review is only performed on uncertain cases, which improves the accuracy of quality inspection while reducing the cost of manual review.

[0092] Furthermore, in the example embodiment, the dialogue quality inspection device 200 generates a first corrected sample based on the review results of manual review, and uses the first corrected sample to train the preset language model. For example, the dialogue quality inspection device 200 extracts the review results of manual review, structures the review results into a first corrected sample, and performs model fine-tuning on the preset language model based on the first corrected sample.

[0093] According to the technical solution in the above example embodiment, the preset language model is fine-tuned by the review results of manual review, which improves the accuracy of the quality inspection of the preset language model and thus reduces the manual review rate.

[0094] according to Figure 3 The technical solution in the example embodiment, on the one hand, pre-sets a first-level conventional question-and-answer database and a second-level professional knowledge database. The conventional question-and-answer database can efficiently handle frequently occurring conventional questions and answers, while the professional knowledge database can cover professional domain knowledge of the target scenario. On the other hand, knowledge information corresponding to the user's question is retrieved from the hierarchical knowledge database in hierarchical order, and the answers to the user's question are quality checked based on the retrieved knowledge information. The hierarchical quality check method can accurately check different types of dialogue data to be checked, such as dialogue data with semantic differences, which significantly reduces the risk of missed detections.

[0095] In some example embodiments, a hierarchical knowledge base is pre-built, containing a dynamically updated general question-and-answer base and a professional knowledge base. The general question-and-answer base includes at least one of the following: high-frequency questions, standard question-and-answer scripts, common FAQs, and a database of common errors in sales scripts. The database of common errors in sales scripts includes labeled examples of such errors, such as exaggerating benefits or concealing disclaimers. The dialogue quality inspection device 200 acquires historical dialogue records for the target scenario, including multiple question-and-answer pairs; it performs multi-dimensional clustering on these pairs to generate general questions from the general question-and-answer base. These multi-dimensional clusters include product type, question scenario, and intent category dimensions.

[0096] For example, suppose there are N question-answer pairs in the historical dialogue records of the target scenario. The dialogue quality inspection device 200 clusters each question-answer pair based on the distance between the embedding vectors corresponding to each dimension, obtaining clusters corresponding to each dimension. Based on the cluster center vectors of the corresponding clusters for each dimension, the regular questions in the regular question-answer library are determined. The value of the cluster center vector represents each dimension of the question. Taking a regular question table for critical illness insurance in chart form as an example, each field of the chart represents each dimension of the question. For example, the first field is critical illness insurance, the second field is the question scenario, and the third field is the product type.

[0097] According to the technical solution in the above example embodiment, a regular question and answer library can be constructed by multi-dimensional clustering, which can obtain the same questions from multiple users, thereby obtaining question and answer pairs that appear frequently, and realizing the dynamic and efficient construction and updating of the regular question library.

[0098] Furthermore, in some example embodiments, the dialogue quality inspection device 200 performs structured knowledge extraction on domain expertise in the target scenario to generate professional knowledge in a professional knowledge base. Structured knowledge extraction includes tree-structure extraction, matrix-form extraction, and graph-structure extraction. For example, it transforms the eligibility judgment logic for health insurance into an executable decision tree structure and determines the combination logic of various insurance liabilities in health insurance through a liability matrix; it establishes semantic relationships between insurance liabilities, exclusion clauses, cash value, etc., in life insurance terms to generate a relational graph. According to the technical solutions in the above example embodiments, by extracting structured knowledge from domain expertise in the target scenario, unstructured texts such as insurance terms and product manuals can be transformed into machine-understandable and reasonable professional knowledge, thereby improving the retrieval efficiency of knowledge retrieval and enabling the dynamic and efficient construction and updating of the professional knowledge base.

[0099] Furthermore, in some example embodiments, the dialogue quality inspection device 200 obtains context information corresponding to a preset language model from a temporary storage area. It inputs at least the context information, retrieved knowledge information, and the question-answer pair to be inspected into the preset language model. The temporary storage area is used to temporarily store the context information in a vectorized manner. The context information includes the multi-turn question-answer history between the current question-answer pair. Temporary vector storage is a technique for dynamically and short-term storing embedded vectors. It can temporarily save content of interest and enhance retrieval (e.g., temporarily storing the preceding text of the current sentence to assist the larger model in making judgments). Storing in a vectorized manner can improve data processing efficiency.

[0100] According to the technical solution of the above example embodiment, the temporary storage area is used to store context information in a vectorized manner. The context information corresponding to the preset language model is obtained from the temporary storage area to assist the language model in quality inspection, which can realize a context-aware quality inspection mechanism and thus improve the accuracy of quality inspection.

[0101] Figure 4 A schematic diagram of a dual-model quality inspection process provided according to some embodiments of this specification is shown.

[0102] Reference Figure 4 As shown, in step S410, if the confidence level of the quality inspection result of the preset language model for the question-answer pair to be inspected is less than the first confidence level threshold, then the question answer of the question-answer pair to be inspected is quality inspected based on the large language model.

[0103] In the example embodiment, the preset language model is a lightweight language model obtained by knowledge distillation of a large language model. The large language model is more accurate than the preset language model in terms of quality control results for the question-answer pairs. The first confidence threshold can be set according to the needs of the target scenario; for example, the confidence threshold is higher for critical illness insurance and lower for life insurance.

[0104] If the confidence level of the quality inspection result of the pre-defined language model for the question-answer pair to be inspected is less than the first confidence threshold, then the dialogue quality inspection device 200 performs quality inspection on the question answer of the question-answer pair to be inspected based on the large language model. For example, the dialogue quality inspection device 200 may input the retrieved knowledge information and the question-answer pair to be inspected into the large language model to guide the large language model to perform quality inspection on the question answer of the question-answer pair to be inspected based on the knowledge information.

[0105] In step S420, if the confidence level of the large language model for the quality inspection results of the question-answer pair samples is less than the second confidence level threshold, then the annotation information for the question-answer pair samples is obtained.

[0106] In the example embodiment, the second confidence threshold can also be set according to the needs of the target scenario. For example, the confidence threshold for critical illness insurance is higher, while the confidence threshold for life insurance is lower. If the confidence of the large language model's quality inspection result for the question-answer pair to be inspected is less than the second confidence threshold, it indicates that the quality inspection results of both the large language model and the distillation model are incorrect. In this case, the dialogue quality inspection device 200 triggers manual annotation of the question-answer pair to be inspected, and obtains the manual annotation information for the question-answer pair sample. In step S430, a second corrected sample is generated based on the annotation information, so as to train the preset language model using the second corrected sample.

[0107] In an example embodiment, the dialogue quality inspection device 200 acquires the annotation information for the question-answer pair samples, structures the annotation information into a second corrected sample, and performs model fine-tuning on the preset language model based on the second corrected sample.

[0108] In step S440, if the confidence level of the large language model for the quality inspection results of the question-answer pair samples is greater than or equal to the second confidence threshold, then the preset language model is fine-tuned based on the quality inspection results of the large language model for the question-answer pair samples.

[0109] In the example embodiment, if the confidence level of the large language model's quality inspection result for the question-answer pair sample is greater than or equal to the second confidence level threshold, it indicates that the large language model's quality inspection result is correct. Then, the dialogue quality inspection device 200 obtains the large language model's quality inspection result for the question-answer pair sample, constructs a fine-tuning sample based on the above quality inspection result, and uses the fine-tuning sample to fine-tune the preset language model. The fine-tuning sample includes the sample question, the sample answer, and the answer conception process.

[0110] According to the technical solution in the above example embodiment, a dual knowledge injection strategy is adopted in the model training and fine-tuning process. High-quality labeled data is generated by using the reasoning ability of the large language model first. The ability of the large language model is transferred to the lightweight preset language model through knowledge distillation technology. Then, samples that are judged incorrectly by both the large language model and the preset language model are corrected by manual annotation, so as to ensure the accuracy of the judgment of complex cases and achieve a balance between model deployment cost and model effect.

[0111] Figure 5 A schematic diagram of a stratified quality inspection process provided according to some embodiments of this specification is shown.

[0112] Reference Figure 5 As shown, in step S505, a regular problem is matched from the regular stable library of the first level.

[0113] The dialogue quality inspection device 200 encodes the user question of the question-answer pair to be inspected into a user question vector, and matches the user question vector with the regular question vectors of regular questions in a regular question-answer database. For example, the dialogue quality inspection device 200 encodes the user question of the question-answer pair to be inspected using a pre-trained natural language processing model such as Sentence-BERT to obtain the user question vector V_user, and determines the semantic similarity, such as cosine similarity, between the user question vector and the regular question vectors in the regular question-answer database.

[0114] In step S510, is it determined whether the match was successful? In the example embodiment, if a match is successful, the answer to the regular question corresponding to the regular question vector is returned; if all regular questions in the regular question-and-answer database fail to match, it is determined that no regular question was retrieved in the regular question-and-answer database.

[0115] For example, assuming a predetermined similarity threshold of 0.8 and semantic similarity as cosine similarity, if the cosine similarity between the user's question vector and the regular question vector in the regular question-answering database is greater than 0.8, then a successful match is determined, and the process proceeds to step S515. The regular question answer corresponding to the regular question is retrieved from the regular question-answering database; if the semantic similarity of each regular question is less than 0.8, then it is determined that no regular question was retrieved from the regular question-answering database.

[0116] In step S515, the answer to the question to be checked is determined by rules.

[0117] In the example embodiment, quality inspection rules are preset, which are expressions composed of logical operators and conditions. If a regular question is matched, the dialogue quality inspection device 200 performs quality inspection on the answer to the user's question based on the regular question's corresponding regular question answer using the preset quality inspection rules. For example, if the customer service representative's first sentence is "Hello," the regular question's corresponding regular question answer is "Hello," the detection range of the preset quality inspection rule's condition is "the first sentence," and the operator is the semantic matching keyword "Hello." The customer service representative's answer to the user's question is then inspected using the aforementioned preset quality inspection rules.

[0118] In step S520, professional knowledge is retrieved from the second-level professional knowledge base.

[0119] In the example embodiment, the dialogue quality inspection device 200 retrieves professional knowledge corresponding to the user's question from the second-level professional knowledge base through multi-hop retrieval. Multi-hop retrieval refers to the process of gradually retrieving relevant information from the knowledge base through multiple reasoning steps, and piecing together the information retrieved in each step to form a complete answer. In each hop query, a more accurate query is generated based on the currently known information; the results of the previous round of retrieval are used to enrich and reconstruct the next round of query.

[0120] In step S525, reasoning and judgment are performed using a preset language model.

[0121] In an example embodiment, the dialogue quality inspection device 200 inputs at least the retrieved knowledge information and the question-answer pair to be inspected into a preset language model to guide the preset language model to perform quality inspection on the question answers based on the knowledge information. The preset language model can be a large language model or a small language model obtained by distillation of a large language model. The preset language model can adopt the Transformer architecture or its variants (such as GPT, BERT, etc.). In step S530, is it determined whether the confidence level of the preset language model output meets the requirements? In the example embodiment, the output of the preset language model includes the confidence level for the question-answer pair to be tested. A first confidence level threshold is preset, for example, 0.8. If the confidence level of the preset language model for the question-answer pair to be tested is less than the first confidence level threshold, then step S535 is performed; if the confidence level is greater than or equal to the first confidence level threshold, then step S540 is performed.

[0122] In step S535, a manual review is performed.

[0123] In an example embodiment, the dialogue quality inspection device 200 triggers a manual review of the question-answer pair to be inspected. Further, the dialogue quality inspection device 200 extracts the review results from the manual review, structures the review results into a first correction sample, and performs model fine-tuning on the preset language model based on the first correction sample.

[0124] In addition, the retrieval strategy of the hierarchical knowledge base can be adjusted using the results of manual review. For example, if the initial retrieval strategy is set to retrieve the first 5 pieces of knowledge information retrieved from the hierarchical knowledge base, and manual review finds that the reason for the low confidence level is that the existing knowledge information in the hierarchical knowledge base was not utilized, such as the 8th piece of knowledge information retrieved, then the retrieval strategy can be set to retrieve the first 8 pieces of knowledge information.

[0125] In step S540, it is determined that the question-and-answer pair to be inspected has passed the quality inspection.

[0126] according to Figure 5The technical solution in the example embodiment, on the one hand, adopts high-frequency question multi-dimensional clustering and professional clause structured extraction, which can handle questions with semantic differences; on the other hand, it realizes long clause linkage analysis through RAG multi-hop retrieval, improving the accuracy of retrieval; furthermore, it conducts quality inspection on the question-answer pair of the question to be inspected through a hierarchical quality inspection method. The first layer realizes the rapid matching of high-frequency questions through semantic similarity matching of conventional question databases. The second layer uses dynamic RAG technology to retrieve professional knowledge from the professional knowledge base, realizes the retrieval enhancement generation of language model, and improves the accuracy of language model quality inspection. The third layer triggers manual review based on confidence threshold, which improves the quality inspection coverage.

[0127] Figure 6 A schematic flowchart illustrating the model optimization process provided according to some embodiments of this specification is shown. Reference Figure 6 As shown, in step S610, the large language model generates high-quality labeled data.

[0128] In an example embodiment, the dialogue quality inspection device 200 generates multiple question-answer pair samples for a target scenario based on a large language model. These sample samples include the large language model's approach to answering questions within the target scenario. For example, target scenarios may include insurance sales services, e-commerce customer service, financial product services, or after-sales product services. The dialogue quality inspection device 200 generates multiple question-answer pair samples for each scenario using preset prompt word templates.

[0129] In step S620, a preset language model is obtained through knowledge distillation.

[0130] In the example embodiment, knowledge distillation is a model compression and transfer learning technique that transfers knowledge from a large language model to a lighter, smaller model, allowing the smaller model to achieve performance close to that of the larger model. The dialogue quality inspection device 200 fine-tunes a preset language model based on question-answer pair samples from the target scenario using knowledge distillation.

[0131] In step S630, the question-answer pair to be checked is quality inspected using a large language model.

[0132] In an example embodiment, if the confidence level of the quality inspection result of the preset language model for the question-answer pair to be inspected is less than a first confidence threshold, the dialogue quality inspection device 200 performs quality inspection on the question answer of the question-answer pair to be inspected based on the large language model. For example, the dialogue quality inspection device 200 at least inputs the retrieved knowledge information and the question-answer pair to be inspected into the large language model to guide the large language model to perform quality inspection on the question answer of the question-answer pair to be inspected based on the knowledge information.

[0133] In step S640, is it determined whether the large language model correctly judges the question-answer pair samples? In the example embodiment, a second confidence threshold is preset, for example, 0.8. If the confidence of the large language model for the quality inspection result of the question-answer pair sample is less than the second confidence threshold, then proceed to step S660; if the confidence of the large language model for the quality inspection result of the question-answer pair sample is greater than or equal to the second confidence threshold, then proceed to step S650.

[0134] In step S650, the preset language model is fine-tuned.

[0135] In the example embodiment, the preset language model is a language model obtained by knowledge distillation of a large language model. If the confidence level of the large language model's quality inspection result for the question-answer pair samples is greater than or equal to a second confidence threshold, the dialogue quality inspection device 200 fine-tunes the preset language model based on the large language model's quality inspection result for the question-answer pair samples. For example, the dialogue quality inspection device 200 obtains the large language model's quality inspection result for the question-answer pair samples, constructs fine-tuned samples based on the above quality inspection result, and uses the fine-tuned samples to fine-tune the preset language model. The fine-tuned samples include sample questions, sample answers, and the answer conception process.

[0136] In step S660, a second corrected sample is generated through manual annotation.

[0137] In an example embodiment, the dialogue quality inspection device 200 acquires annotation information for question-answer pair samples, generates a second corrected sample based on the annotation information, and trains a preset language model using the second corrected sample.

[0138] In step S670, the preset language model is iteratively updated.

[0139] In an example embodiment, the dialogue quality inspection device 200 iteratively updates and trains a preset language model using a second corrected sample.

[0140] according to Figure 6 The technical solution in the example embodiment adopts a dual knowledge injection strategy. On the one hand, it uses the reasoning ability of the large model to generate high-quality labeled data and transfers the ability to a lightweight preset language model through knowledge distillation technology. On the other hand, it corrects samples that are judged incorrectly by both the large language model and the preset language model through manual annotation to ensure the accuracy of judgment in complex cases. Furthermore, it guides the fine-tuning of the lightweight preset language model through the large language model and combines manual correction of erroneous samples from both models to achieve a balance between model deployment cost and model performance.

[0141] This specification, in another aspect, provides a non-transitory storage medium storing at least one set of executable instructions for performing dialogue quality control processing. When these executable instructions are executed by a processor, they instruct the processor to implement the steps of the dialogue quality control method described in this specification. In some possible embodiments, various aspects of this specification can also be implemented as a program product comprising program code. When the program product is run on the dialogue quality control device 200, the program code causes the dialogue quality control device 200 to perform the steps of the dialogue quality control method described in this specification. The program product for implementing the above method may employ a portable compact disc read-only memory (CD-ROM) containing program code and may run on the dialogue quality control device 200. However, the program product of this specification is not limited thereto. In this specification, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. The aforementioned computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing the operations described herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the dialogue quality inspection device 200, partially on the dialogue quality inspection device 200, as a standalone software package, partially on the dialogue quality inspection device 200 and partially on a remote computing device, or entirely on a remote computing device.

[0142] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0143] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure is presented by way of example only and is not restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0144] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0145] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and aiding in the understanding of a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art may readily identify some of the devices as separate embodiments when reading this specification. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. It is also valid when each secondary embodiment contains fewer than all the features of a single foregoing disclosed embodiment.

[0146] Each patent, patent application, publication of a patent application, and other material such as articles, books, specifications, publications, documents, articles, etc., cited herein may be incorporated by reference, except for any identical content appearing in the related documents that may be inconsistent with or conflict with this document, or any identical document content that may have a limiting effect on the widest scope of the claims. For example, in the event of any inconsistency or conflict between the description, definition, and / or use of terms associated with any included material and those related to this document, the terminology herein shall prevail.

[0147] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. A dialogue-based quality inspection method, comprising: Obtain at least one question-and-answer pair to be inspected in the target scenario, wherein the question-and-answer pair to be inspected includes a user question in the target scenario and the question answer corresponding to the user question. Knowledge information corresponding to the user's question in the question-answer pair to be checked is retrieved from the hierarchical knowledge base in hierarchical order. The hierarchical knowledge base includes a first-level general question-answer base and a second-level professional knowledge base. as well as The question answers of the question-answer pair to be inspected are quality checked based on the retrieved knowledge information.

2. The method according to claim 1, wherein, The step of retrieving knowledge information corresponding to the user question in the hierarchical knowledge base according to hierarchical order includes: Retrieve regular questions from the regular question-and-answer database at the first level that correspond to the user's question in the question-and-answer pair to be tested; If the general question is not found, then the professional knowledge corresponding to the user question is retrieved from the professional knowledge base of the second level.

3. The method according to claim 2, wherein, The step of retrieving a regular question from the regular question-and-answer database at the first level that corresponds to the user question in the question-and-answer pair to be checked includes: Determine the semantic similarity between the user question in the question-answer pair to be inspected and the regular question in the regular question-answer database; If the semantic similarity corresponding to the regular question is greater than or equal to a preset similarity threshold, then the answer to the regular question corresponding to the regular question is obtained from the regular question-and-answer database; If the semantic similarity of each of the aforementioned regular questions is less than the preset similarity threshold, then it is determined that no regular question was retrieved from the regular question-and-answer database.

4. The method according to claim 3, wherein, Determining the semantic similarity between the user question of the question-answer pair to be tested and the regular questions in the regular question-answer database further includes: Determine the preset weights for the common problems, where the preset weights are weights pre-set based on the importance of the common problems; and The semantic similarity between the user question and the regular question is weighted based on the preset weight.

5. The method according to claim 2, wherein, The quality inspection of the question-answer pair based on the retrieved knowledge information includes: If the regular question is found, the answer to the question corresponding to the user question is inspected based on the regular question answer according to the preset quality inspection rules.

6. The method according to claim 3, wherein, The step of retrieving professional knowledge corresponding to the user's question from the professional knowledge base at the second level includes: The professional knowledge corresponding to the user's question is retrieved from the second-level professional knowledge base using a multi-hop retrieval method.

7. The method according to claim 6, wherein, The step of retrieving professional knowledge corresponding to the user's question from the second-level professional knowledge base using a multi-hop retrieval method includes: The user's question is broken down into several corresponding sub-questions; Based on the first sub-question, retrieve the sub-professional knowledge corresponding to the first sub-question from the professional knowledge base; The current query for the current round is generated based on the sub-problem of the current round and the sub-professional knowledge of the previous round; and Based on the current query, retrieve the sub-professional knowledge corresponding to the current query from the professional knowledge base.

8. The method according to claim 1, wherein, The quality inspection of the question-answer pair based on the retrieved knowledge information includes: The retrieved knowledge information and the question-answer pair to be checked are input into a preset language model to guide the preset language model to perform quality checks on the question answers based on the knowledge information.

9. The method according to claim 8, wherein, The step of inputting at least the retrieved knowledge information and the question-answer pair to be tested into a preset language model includes: The context information corresponding to the preset language model is obtained from the temporary storage area. At least the context information, the retrieved knowledge information, and the question-answer pair to be checked are input into the preset language model. The temporary storage area is used to temporarily store the context information in a vectorized manner.

10. The method of claim 8, wherein, The process of guiding the preset language model to perform quality checks on the question answers based on the knowledge information includes: Based on the aforementioned knowledge information and the question-and-answer pairs to be inspected, quality inspection prompts are generated using a quality inspection prompt template. This template is a pre-configured prompt template that combines the aforementioned knowledge information and the question-and-answer pairs to generate quality inspection results. Based on the quality inspection prompts, the quality inspection results corresponding to the question answers are generated through the preset language model.

11. The method according to claim 8, wherein, The preset language model is a language model obtained by knowledge distillation of a large language model.

12. The method according to claim 11, wherein, The method further includes: Multiple question-answer pair samples for the target scenario are generated based on the large language model. The preset language model is trained based on the multiple question-answer pairs.

13. The method according to claim 11, wherein, The output of the preset language model includes the confidence level of the quality inspection result for the question-answer pair to be inspected, and the method further includes: If the confidence level is less than the first confidence level threshold, then manual review of the question-answer pair to be inspected or quality inspection of the question answer of the question-answer pair to be inspected based on the large language model is triggered.

14. The method according to claim 13, wherein, The method also includes A first corrected sample is generated based on the review results of the manual review, and the preset language model is trained using the first corrected sample.

15. The method according to claim 13, wherein, After performing quality control on the question-answer pair based on the large language model, the method further includes: If the confidence level of the quality inspection result of the large language model for the question-answer pair sample is less than the second confidence level threshold, then the annotation information for the question-answer pair sample is obtained, and a second corrected sample is generated based on the annotation information, so as to train the preset language model through the second corrected sample; If the confidence level of the large language model for the quality inspection results of the question-answer pair samples is greater than or equal to the second confidence threshold, then the preset language model is fine-tuned based on the quality inspection results of the large language model for the question-answer pair samples.

16. The method according to claim 1, wherein, The method further includes: Obtain historical dialogue records in the target scenario, the historical dialogue records including multiple question-answer pairs; Multi-dimensional clustering is performed on the multiple question-answer pairs to generate regular questions in the regular question-answer library.

17. The method according to claim 1, wherein, The method further includes: Structured knowledge extraction is performed on the domain expertise in the target scenario to generate the expertise in the expertise base.

18. A dialogue-based quality inspection device, comprising: At least one storage medium storing at least one instruction set for performing dialogue quality inspection processing; as well as At least one processor is communicatively connected to the at least one storage medium. When the dialogue quality inspection device is running, the at least one processor reads the at least one instruction set and executes the dialogue quality inspection method according to any one of claims 1-17 according to the instructions of the at least one instruction set.

Images