Answer generation method and device for intelligent customer service question-answering system and storage medium

Abstract

The invention relates to an intelligent customer service question-answering system answer generation method and device and a storage medium. Belongs to the technical field of artificial intelligence. The method comprises the steps of obtaining text information input by a user; inputting the text information into the created knowledge base for retrieval recall, and determining a knowledge base slice recall result matched with the text information; inputting the text information and a knowledge base slice recall result into a large language model to obtain an information illusion cognition result; if the information illusion cognition result indicates that the illusion exists, not feeding back the question and answer based on retrieval to the user; if the information illusion cognition result indicates that illusion does not exist, the text information and the knowledge base slice recall result are input into a large language model, and a final answer result is obtained. The information illusion confirmation is performed based on the user input text and the knowledge base recall result, and the final reply result is output based on the confirmation result, so that the overall accuracy is improved, and the overall effect of the intelligent customer service question-answering system is improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a method, device and storage medium for generating answers for an intelligent customer service question and answer system. Background Art

[0002] With the rapid development of artificial intelligence and big data technologies, the new generation of artificial intelligence technology represented by large language models has brought revolutionary breakthroughs. How to reconstruct and upgrade traditional products and services based on large language models has become a hot research topic.

[0003] The intelligent customer service question-and-answer system is one of the most popular applications of natural language processing. The system generates corresponding answers based on user input to meet user needs. Human natural language expressions are very diverse, and traditional methods are difficult to deal with well. The large language model has a strong emergence ability, a general understanding of natural language, and extremely high cross-domain capabilities and flexibility. However, the traditional intelligent customer service question-and-answer system has low intent understanding accuracy and scene migration flexibility, and needs to be upgraded to improve customer experience.

[0004] Therefore, intelligent customer service question-answering systems based on large language models have become one of the hot research directions. Summary of the invention

[0005] In view of the above analysis, the embodiments of the present invention aim to provide a method, device and storage medium for generating answers for an intelligent customer service question and answer system, so as to solve the problem that the overall accuracy of the existing intelligent customer service question and answer system is limited.

[0006] In a first aspect of the present application, a method for generating answers for an intelligent customer service question and answer system is provided, comprising:

[0007] Get the text information entered by the user;

[0008] Input the text information into the created knowledge base for retrieval and recall, and determine the knowledge base slice recall result that matches the text information;

[0009] Inputting the text information and the knowledge base slice recall result into a large language model to obtain an information hallucination recognition result;

[0010] If the information hallucination recognition result indicates that a hallucination exists, no retrieval-based question and answer is fed back to the user; if the information hallucination recognition result indicates that a hallucination does not exist, the text information and the knowledge base slice recall result are input into the large language model to obtain a final answer result;

[0011] Among them, the large language model is obtained by fine-tuning the large language model in advance using an information hallucination cognitive alignment training dataset. The pre-creation of the information hallucination cognitive alignment training dataset includes collecting text, knowledge base slice text and corresponding information hallucination cognitive results to identify whether the information involved in the text and the knowledge base slice text exists in a given knowledge base.

[0012] Optionally, the obtaining text information input by the user includes:

[0013] Get the real-time text entered by the user and obtain the historical session records;

[0014] The real-time text is spliced ​​with the historical conversation record to obtain text information.

[0015] Optionally, inputting the text information into the created knowledge base for retrieval and recall, and determining the knowledge base slice recall result matching the text information includes:

[0016] Vectorize the text information using a vectorization model to obtain a text vector;

[0017] The text vector is input into the knowledge base for retrieval and recall, and according to the vector similarity between the text vector and the knowledge base slices in the knowledge base, the first N knowledge base slices with high relevance to the text vector are taken as the knowledge base slice recall results.

[0018] Optionally, the information hallucination cognitive alignment training data set is collected by automatically annotating using a large language model, or is collected by manually annotating.

[0019] Optionally, the information hallucination cognitive alignment training dataset is collected by automatically annotating using a large language model, including:

[0020] Collecting input samples, where the input samples include actual conversation texts between a user and a system and knowledge base slice texts;

[0021] Use a large language model to process each input sample; read the actual conversation text and knowledge base slices to predict whether the input sample has information hallucination;

[0022] Based on the prediction of the large language model, a labeling result is automatically generated; the labeling result includes result information of whether information hallucination exists and the content of the hallucinated information.

[0023] 6. The answer generation method according to claim 5, characterized in that after the prediction based on the large language model and the automatic generation of the annotation result, it also includes:

[0024] Obtaining manual correction data for the annotation result;

[0025] The manually corrected data is input into a large language model to fine-tune the large language model.

[0026] Optionally, any one of the LoRA efficient fine-tuning method, the full parameter fine-tuning method, and the Prefix Tuning fine-tuning method is used to perform fine-tuning training on the large language model.

[0027] Optionally, after the information hallucination recognition result indicates the presence of hallucination, the method further includes:

[0028] An answer result is generated based on a preset refusal to answer script, and the question and answer result is used to indicate that the user's current information is incorrect and / or request the user to input additional required information.

[0029] According to a second aspect of the present application, an intelligent customer service question and answer system is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, any one of the above-mentioned methods for generating answers for the intelligent customer service question and answer system is implemented.

[0030] According to a third aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned methods for generating answers for the intelligent customer service question and answer system is implemented.

[0031] The answer generation method of the intelligent customer service question and answer system provided in the present application obtains the text information input by the user; inputs the text information into the created knowledge base for retrieval and recall, and determines the knowledge base slice recall result that matches the text information; inputs the text information and the knowledge base slice recall result into the large language model to obtain the information hallucination cognition result; if the information hallucination cognition result indicates the existence of hallucination, no retrieval-based question and answer is fed back to the user; if the information hallucination cognition result indicates the absence of hallucination, the text information and the knowledge base slice recall result are input into the large language model to obtain the final answer result. Based on the knowledge base recall result, the present application uses a large language model to confirm the information hallucination based on the user input text and the knowledge base recall result, and passes the confirmation result to the intelligent customer service question and answer system, and outputs the final reply result. This process can identify and verify whether the key entity element information is correct, improve the overall accuracy, and thus improve the overall effect of the intelligent customer service question and answer system.

[0032] In addition, the present application also provides an intelligent customer service question and answer system and a computer-readable storage medium having the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0034] Figure 1 A flowchart of a specific implementation method of the method for generating answers to the intelligent customer service question and answer system provided in this application;

[0035] Figure 2 A flowchart for obtaining the information hallucination cognitive alignment training data set;

[0036] Figure 3 This is the structural block diagram of the intelligent customer service question and answer system. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. It should be noted that, in the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be combined, separated, interchanged and / or rearranged with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0038] The terms used here are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used here, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used here, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0039] Traditional intelligent customer service question-and-answer systems use models to understand user input needs or intentions, and match the best response content in a pre-built business knowledge base based on the understanding results. There is no identification and verification of the correctness of the information provided by the user, especially the key entity element information, which leads to limitations in the overall accuracy of the conversational question-and-answer system.

[0040] In view of this, a flowchart of a specific implementation method of the intelligent customer service question and answer system answer generation method provided in this application is as follows: Figure 1 As shown, refer to Figure 1 , the process specifically includes:

[0041] S101: Acquire text information input by a user.

[0042] The text information can be real-time text input by the user, or can be text information spliced ​​from real-time text and historical conversation records. The process can be specifically as follows: obtaining real-time text input by the user, obtaining historical conversation records; splicing the real-time text with the historical conversation records to obtain text information, which can be recorded as Q.

[0043] Optionally, the text information input by the user may be further preprocessed, including but not limited to text standardization, noise removal, keyword extraction and other processing steps, to prepare data for subsequent analysis.

[0044] The text information input by the user includes questions, requests or any other form of inquiry. The intelligent customer service question-answering system can obtain the text information input by the user through the human-computer interaction interface.

[0045] S102: Input the text information into the created knowledge base for retrieval and recall, and determine the knowledge base slice recall results that match the text information.

[0046] Match the user's input text with the pre-built knowledge base to recall relevant knowledge base slices. This step can use search algorithms or information retrieval technology to find the most relevant information to the user's question based on keywords, semantic matching, etc.

[0047] In a specific implementation, this step inputs the text information into the created knowledge base for retrieval and recall, and determines the knowledge base slice recall result that matches the text information, including: using a vectorization model to vectorize the text information to obtain a text vector; inputting the text vector into the knowledge base for retrieval and recall, and based on the vector similarity between the text vector and the knowledge base slices in the knowledge base, taking the top N knowledge base slices with a high correlation with the text vector as the knowledge base slice recall result.

[0048] In this embodiment, the text information Q is vectorized using a vectorization model to obtain a text vector, which is recorded as QV. The vectorization model can use a mainstream model, such as M3E.

[0049] Use the vector database for retrieval and recall. Recall the knowledge base slices with high relevance to the user input text according to the descending order of similarity between the input text vector QV and the knowledge base slice vector in the vector database, and perform regular deduplication processing. Take the Top N results as the knowledge base slice recall results, denoted as KR1, and N can be set to 50.

[0050] S103: Input the text information and the knowledge base slice recall result into a large language model to obtain an information hallucination recognition result.

[0051] The text information Q and the knowledge base slice recall result KR1 are input into the large language model, and the information hallucination recognition result is output. The large language model has been fine-tuned through the information hallucination recognition alignment training dataset, which is specifically used to evaluate the degree of match between the input information and the knowledge base content. The purpose of this step is to determine whether the user input is based on correct and existing information.

[0052] It can be understood that the process of hallucination recognition is to determine whether the information provided by the user (i.e., the facts or data in the question) exists in a known and reliable knowledge base. If the information contained in the query is inconsistent with the information in the knowledge base or the corresponding information cannot be found in the knowledge base, this situation is considered to be information hallucination. The result of information hallucination recognition is information generated based on the model's own training data that does not match the actual user input or the data in the business knowledge base.

[0053] S104: If the information hallucination recognition result indicates the existence of hallucination, no retrieval-based question and answer is fed back to the user; if the information hallucination recognition result indicates the absence of hallucination, the text information and the knowledge base slice recall result are input into the large language model to obtain the final answer result.

[0054] If the information hallucination recognition result indicates the existence of hallucination, it means that the user input information has hallucination, that is, the user's query is based on an incorrect or non-existent assumption, and it is not suitable to output business knowledge as an answer. At this time, the user will not be fed back the question and answer based on the retrieval, and the user will not get an answer based on incorrect information. As a specific implementation method, the refusal process can be automatically entered. The reply script can be pre-configured with rules or use a large language model to generate a solution for refusal and guidance of information confirmation.

[0055] After the information hallucination recognition result indicates the existence of hallucination, the system further includes: generating an answer result based on a preset refusal to answer script, wherein the question and answer result is used to indicate that the user's current information is incorrect and / or request the user to input additional required information. For example, a sample reply may be: "The information you provided is beyond my knowledge, and I cannot answer it for the time being. Please provide the information again or describe the problem, thank you!" This can avoid providing misleading or unfounded answers and maintain the reliability of the system.

[0056] If the information hallucination recognition result indicates that there is no hallucination, which means that the user input information is not hallucinated, then business knowledge can be output as an answer. At this time, the intelligent agent question-answering process is entered, and the text information Q and the knowledge base slice recall result KR1 are input into the large language model to output the final answer result. The final answer result is based on the actual business knowledge base information and has been semantically optimized by the model. The large language model uses existing models in the industry, such as ChatGLM, and does not require fine-tuning.

[0057] The large language model is obtained by fine-tuning the large language model in advance using the information illusion cognitive alignment training data set, so as to be able to identify and process whether the information involved in the user query exists in a given static knowledge base. This ensures that the answer provided to the user is not only based on correct information, but also ensures that the information is up-to-date and accurate. This method is suitable for intelligent question-answering systems or customer service automation systems that need to verify the authenticity of information. The model can be deployed on a cloud server.

[0058] Pre-creating the information hallucination cognitive alignment training data set includes collecting text and knowledge base slice text and corresponding information hallucination cognitive results to identify and process whether the information involved in the text and knowledge base slice text exists in a given knowledge base.

[0059] The information hallucination cognitive alignment training data set is collected by automatically annotating a large language model, or is collected by manually annotating the data set.

[0060] Based on the recall results of the knowledge base, this application uses a large language model to confirm the information illusion based on the user input text and the recall results of the knowledge base, and passes the confirmation results to the intelligent customer service question and answer system to output the final reply results. This process can identify and verify whether the key entity element information is correct, improve the overall accuracy of the intelligent customer service question and answer system, and thus improve the overall effect of the intelligent customer service question and answer system.

[0061] The information hallucination cognitive alignment training data set of the embodiment of the present invention can be collected in the form of automatic annotation using a large language model, or can be collected in the form of manual annotation, and the cost of manual annotation is relatively high.

[0062] Reference Figure 2The flowchart of obtaining the information hallucination cognitive alignment training data set is as follows: the process of using a large language model to automatically annotate and collect the information hallucination cognitive alignment training data set specifically includes:

[0063] S201: Collect input samples, where the input samples include actual conversation texts between the user and the system and knowledge base slice texts.

[0064] Collect data from existing conversation records, question-and-answer system logs, or related text data. This data can come from customer service logs, online question-and-answer platforms, professional forums, etc. For business content in professional fields, you can also extract information from professional literature or case libraries.

[0065] The input data includes conversation text and corresponding knowledge base slices. The actual conversation text is the actual conversation between the user and the system, while the knowledge base slice is the relevant reference information, which may be answers to questions, guidelines or other forms of structured knowledge.

[0066] S202: Use a large language model to process each input sample; read the actual conversation text and the knowledge base slice to predict whether the input sample has information hallucination.

[0067] Use a large language model to automatically annotate each input sample. Read the conversation text and knowledge base slices to predict possible information hallucinations in these texts. For example, the large language model annotates that some parts of the actual conversation text are inconsistent with the knowledge base slice or do not contain relevant information.

[0068] S203: Automatically generate annotation results based on the prediction of the large language model; the annotation results include result information of whether information hallucination exists and the content of the hallucinated information.

[0069] Based on the prediction of the large language model, the annotation results are automatically generated. This can specifically include whether there is information hallucination (isInStaticKnowledge) and the specific content of the hallucination information (analysis).

[0070] After the prediction based on the large language model automatically generates the annotation result, the method further includes: obtaining manual correction data for the annotation result; inputting the manual correction data into the large language model to fine-tune the large language model.

[0071] The model's annotation results can be checked by a professional annotation team, and necessary corrections can be made. The collected and corrected automatic annotation data can be used to further fine-tune the model to enhance its ability to identify and handle information hallucinations in practical applications. Through repeated iterations, the performance of the large language model can be gradually improved.

[0072] It is understandable that the large language model in this application uses existing models in the industry, such as ChatGLM, and other large language models, such as ChatGPT, GPT-4, etc., which does not affect the implementation of this application. Use the large language model to complete fine-tuning on the information illusion cognitive alignment training data set created above to enhance the information illusion cognitive alignment ability of the large language model. The loss function can choose cross entropy, and the fine-tuning method can use the mainstream large language model fine-tuning method, such as full parameter fine-tuning, LoRA efficient fine-tuning method, full parameter fine-tuning method, Prefix Tuning fine-tuning method, any one of which can fine-tune the large language model.

[0073] The training data set format example of the embodiment of the present invention is as follows:

[0074] {"input":" <duty><! --Role and Mission-->\n*As an information hallucination detector, you need to pay attention to whether the knowledge that the user's question or reply relies on exists in the given static knowledge information.

[0075] < / duty> \n\n <staticknowledge> <! --Known knowledge domain-->\nThe second-generation home charging pile cannot charge, please perform the following troubleshooting: 1. Press and hold the red RESET button on the left side of the charging pile to restart the charging pile; 2. Restart the home charging pile switch. 3. Make sure that your vehicle / APP does not enable timed charging\nThe third-generation home charging pile cannot charge or charging is interrupted. If the charging pile indicator light is not on, please perform the following troubleshooting: 1. Confirm that the charging pile power is turned on; 2. Restart the leakage protector: push the lever down to disconnect the leakage protector and cut off the power of the charging pile. After waiting for 2 minutes, push the lever up again to close the switch, and the home charging pile will restart immediately; 3. Check whether there is any arrears on the meter; 4. Contact the property to confirm whether the power circuit breaker is closed. The charging pile indicator light is always red, please try to perform the following troubleshooting: 1. Press the charging gun head button. If the charging pile resumes flashing green, please try to charge again. 2. Restart the leakage protector: push the lever down to disconnect the leakage protector and cut off the power of the charging pile. After waiting for 2 minutes, push the lever upwards again to close the switch, and the home charging pile will restart immediately. If the charging pile indicator light is always blue, please perform the following troubleshooting: 1. Restart the leakage protector: Push the lever downward to disconnect the leakage protector and cut off the power of the charging pile. After waiting for 2 minutes, push the lever upwards again to close the switch, and the home charging pile will restart immediately; 2. Please check the charging control permission function to make sure it is not turned on; 3. Make sure that your vehicle / APP does not enable timed charging. \n< / staticknowledge> \n\n <targetquery> <! --Query string waiting for knowledge domain detection-->\n```user:My home charger cannot charge\nassistant:What generation of home charger do you have?\n user:Fifth generation```\n< / targetquery> \n\n <format> <! --Unique output format and description-->\n```json{"targetQuery":...,"analysis":...,"isInStaticKnowledge":...}```\n\nField description:\n*targetQuery: string format, after removing rhetoric and expression, objectively describes the user session record, that is, the text to be detected;\n*analysis: string format, objectively describes the detected hallucination information content;\n*isInStaticKnowledge: Boolean format, describes whether staticKnowledge has knowledge that may answer targetQuery, if so, the value is True; otherwise, the value is False.\n< / format> \n\n <init> <! --After completing the following initialization, execute the given task-->\n*The output must start from "```json";\n*No need to output the thinking process.\n< / init> "

[0076] "target":"```json\n{\n"targetQuery":"My home charger cannot charge\nWhat generation of home charging pile is yours?\nFifth generation",\n"analysis":"Based on background knowledge, home charging piles only mention the second and third generations, not the fifth generation, so questions about the fifth generation of home charging piles cannot be answered",\n"isInStaticKnowledge":False\n}\n```"}

[0077] The training dataset consists of the following parts: Duty: The task of the information hallucination detector is to detect whether the information in the user's question or reply exists in a given static knowledge base.

[0078] Static Knowledge: Contains verified information in a specific field, such as troubleshooting information about second-generation and third-generation home charging piles.

[0079] Target Query (question to be tested): string format, after removing rhetoric and expression, objectively describes the user conversation record, that is, the question to be tested. This is an example of a conversation between an actual user and the intelligent customer service question-and-answer system. The user mentioned that the home charger could not charge, and specified in the conversation that it was a fifth-generation home charger.

[0080] Analysis: string format, describing the information hallucination content detected in the conversation, that is, the information identified that does not match the knowledge base.

[0081] isInStaticKnowledge (whether it is in the static library): Boolean format, describing whether staticKnowledge has knowledge that may answer targetQuery. If so, the value is True; otherwise, the value is False.

[0082] In this example, the system recognizes that the "fifth-generation home charging pile" mentioned in the user's query does not exist in the existing knowledge base (which only includes information on the second and third generations), so this is an information illusion.

[0083] In addition, this application also provides an intelligent customer service question and answer system, such as Figure 3 As shown in the structural block diagram of the intelligent customer service question and answer system, it includes a memory 31 and a processor 32. The memory 31 stores a computer program. When the computer program is executed by the processor 32, any of the above-mentioned intelligent customer service question and answer system answer generation methods is implemented. Figure 3 The intelligent customer service question and answer system shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0084] In addition, the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements any of the above-mentioned methods for generating answers for the intelligent customer service question and answer system.

[0085] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable storage medium, a computer-readable signal medium, or any combination of the above two.

[0086] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0087] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer readable program codes. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0088] The above-mentioned computer-readable medium may be included in the above-mentioned computing device; or it may exist independently without being assembled into the intelligent customer service question and answer system.

[0089] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the tester computer, partially on the tester computer, as a separate software package, partially on the tester computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the tester computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0090] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system based on hardware that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0091] It is understandable that the specific implementation process of the intelligent customer service question and answer system and computer-readable storage medium provided by this application can refer to the implementation described in the above-mentioned intelligent customer service question and answer system answer generation method, and will not be repeated here. Based on the knowledge base recall results, this application uses a large language model to confirm the information illusion based on the user input text and the knowledge base recall results, and passes the confirmation results to the intelligent customer service question and answer system, and outputs the final reply result. This process can identify and verify whether the key entity element information is correct, improve the overall accuracy, and thus improve the overall effect of the intelligent customer service question and answer system.

[0092] The professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0093] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0094] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for generating answers for an intelligent customer service question and answer system, characterized in that: include: Get the text information entered by the user; Input the text information into the created knowledge base for retrieval and recall, and determine the knowledge base slice recall result that matches the text information; Inputting the text information and the knowledge base slice recall result into a large language model to obtain an information hallucination recognition result; If the information hallucination recognition result indicates that a hallucination exists, no retrieval-based question and answer is fed back to the user; If the information hallucination recognition result indicates that there is no hallucination, the text information and the knowledge base slice recall result are input into the large language model to obtain a final answer result; Among them, the large language model is obtained by fine-tuning the large language model in advance using an information hallucination cognitive alignment training dataset. The pre-creation of the information hallucination cognitive alignment training dataset includes collecting text, knowledge base slice text and corresponding information hallucination cognitive results to identify whether the information involved in the text and the knowledge base slice text exists in a given knowledge base.

2. The answer generation method according to claim 1, characterized in that: The step of obtaining text information input by the user includes: Get the real-time text entered by the user and obtain the historical session records; The real-time text is spliced ​​with the historical conversation record to obtain text information.

3. The answer generation method according to claim 2, characterized in that: The step of inputting the text information into the created knowledge base for retrieval and recall, and determining the knowledge base slice recall result matching the text information includes: Vectorize the text information using a vectorization model to obtain a text vector; The text vector is input into the knowledge base for retrieval and recall, and according to the vector similarity between the text vector and the knowledge base slices in the knowledge base, the first N knowledge base slices with high relevance to the text vector are taken as the knowledge base slice recall results.

4. The answer generation method according to claim 2, characterized in that: The information hallucination cognitive alignment training data set is collected by automatically annotating a large language model, or is collected by manually annotating the data set.

5. The answer generation method according to claim 4, characterized in that: The information hallucination cognitive alignment training dataset obtained by automatically annotating the dataset using a large language model includes: Collecting input samples, where the input samples include actual conversation texts between a user and a system and knowledge base slice texts; Use a large language model to process each input sample; read the actual conversation text and knowledge base slices to predict whether the input sample has information hallucination; Based on the prediction of the large language model, a labeling result is automatically generated; the labeling result includes result information of whether information hallucination exists and the content of the hallucinated information.

6. The answer generation method according to claim 5, characterized in that: After the prediction based on the large language model and the automatic generation of the annotation results, the following steps are also included: Obtaining manual correction data for the annotation result; The manually corrected data is input into a large language model to fine-tune the large language model.

7. The answer generation method according to any one of claims 1 to 6, characterized in that: The large language model is fine-tuned using any one of the LoRA efficient fine-tuning method, the full parameter fine-tuning method, and the Prefix Tuning fine-tuning method.

8. The answer generation method according to claim 7, characterized in that: After the information hallucination cognitive result indicates the presence of hallucination, the method further includes: An answer result is generated based on a preset refusal to answer script, and the question and answer result is used to indicate that the user's current information is incorrect and / or request the user to input additional required information.

9. An intelligent customer service question and answer system, characterized in that: It includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the answer generation method of the intelligent customer service question and answer system according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the intelligent claim according to any one of claims 1 to 8 is implemented. ZSP240501CN Customer service question and answer system answer generation method.

Images