An intelligent customer service system based on AI question answering and large language models

Abstract

This application relates to the field of intelligent customer service technology, and discloses an intelligent customer service system based on AI question answering and a large language model. The system includes an answer generation module, an answer analysis module, and an answer output module. This application uses professional factors to quantitatively evaluate the standardization of the first answer generated by the large language model, ensuring that the output second answer conforms to industry terminology, sentence structure, and response framework requirements. It utilizes accuracy factors to perform multi-dimensional verification of knowledge points and uses a set of minimum knowledge units to falsify and replace risky knowledge points, ensuring the accuracy and credibility of the answers. This satisfies the core requirements of customer service scenarios—that information must be provided, cannot be wrong, and must conform to standards—effectively improving user experience and system credibility.

Description

Technical Field

[0001] This application relates to the field of intelligent customer service technology, specifically an intelligent customer service system based on AI question answering and large language models. Background Technology

[0002] In the field of intelligent customer service, systems based on AI question answering and large language models have become mainstream, but existing technologies have significant limitations in answer quality control. Current systems mostly focus on identifying and matching the intent of user questions, directly generating answers through large language models, lacking a refined evaluation mechanism for the professionalism of the output content and the accuracy of the answers.

[0003] Specifically, existing technologies lack effective quantitative standards for standardized language, resulting in generated answers that often contain colloquialisms and non-standard terminology, failing to meet the professional expression requirements of customer service scenarios. At the same time, there is insufficient verification of the accuracy of content generated by large language models, relying solely on static knowledge base retrieval results for simple judgment, without combining dynamic knowledge bases to deeply verify the authenticity of knowledge points, which easily leads to fabricated information.

[0004] Furthermore, existing technologies lack a closed-loop mechanism that uses knowledge bases as the basis for quality optimization, making it impossible to improve answer quality and failing to meet the core requirements of customer service scenarios where answers must be provided, cannot be wrong, and must conform to standards.

[0005] Chinese invention patent application CN119248918A discloses a method, device and storage medium for generating answers in an intelligent customer service question and answer system, but the accuracy of the answers is not good.

[0006] In conclusion, there is an urgent need for a new intelligent customer service solution based on AI question answering and large language models. Summary of the Invention

[0007] The purpose of this application is to provide an intelligent customer service system based on AI question answering and large language models to solve the technical problems mentioned in the background.

[0008] To achieve the above objectives, this application discloses the following technical solution: an intelligent customer service system based on AI question answering and large language models, the system comprising:

[0009] The answer generation module is used to analyze the user's question using AI question answering and large language models to obtain the first answer to the question.

[0010] The answer analysis module is used to obtain professional factors and accuracy factors from the customer service knowledge base. The professional factors are used to define the professionalism of the first answer's wording, and the accuracy factors are used to define the accuracy of the first answer's response.

[0011] The answer output module is used to optimize the wording professionalism and / or answer accuracy of the first answer based on the professional factor and the accuracy factor to obtain a second answer, and output the second answer as the answer to the user's question; wherein, optimizing the answer accuracy of the first answer based on the accuracy factor is to obtain the smallest knowledge unit set through the customer service knowledge base to replace the knowledge points used in the first answer.

[0012] Preferably, the step of analyzing the user's question using AI question answering and large language models to obtain a first answer to the question includes:

[0013] Perform semantic analysis on the problem to extract core demands and domain features;

[0014] Based on the aforementioned domain characteristics, retrieve the relevant knowledge set from the customer service knowledge base;

[0015] Input the core requirements and the related knowledge set into the large language model to generate the first answer;

[0016] The output of the large language model is configured to prioritize the use of content from the associated knowledge set.

[0017] Preferably, the professional factors are obtained through the customer service knowledge base, including:

[0018] Extract preset domain-specific dialogue templates corresponding to the questions from the customer service knowledge base. The domain-specific dialogue templates include an industry terminology system, sentence structure, and response framework.

[0019] Analyze the similarity between the first answer and the domain-specific language template, quantify the similarity to ensure that the similarity is positively correlated with the professionalism of the language, and define the similarity as the professional factor output.

[0020] Preferably, the accuracy factor is obtained through the customer service knowledge base, including:

[0021] Analyze the knowledge points in the first answer to obtain a set of knowledge points, which includes factual statements, data, and logical relationships.

[0022] Perform consistency verification between the knowledge point set and the authoritative knowledge in the customer service knowledge base, and count the percentage of matching knowledge points.

[0023] Analyze the correlation between the external knowledge called by the large language model when generating the first answer and the corresponding domain knowledge in the customer service knowledge base;

[0024] Based on the proportion of the knowledge points and the degree of correlation, an accuracy factor is obtained, and the accuracy factor is negatively correlated with the accuracy of the answer.

[0025] Preferably, the answer output module includes:

[0026] The factor analysis unit is used to analyze the first answer based on preset professional factor thresholds and accuracy factor thresholds to obtain factor analysis results. The factor analysis results are used to define the first answer's professionalism and accuracy.

[0027] The answer optimization unit is used to obtain a second answer that is both professional in terms of wording and free from the risk of answering inaccuracies, based on the factor analysis results and the first answer.

[0028] Preferably, the step of analyzing the first answer based on preset professional factor thresholds and accurate factor thresholds to obtain factor analysis results includes:

[0029] When the professional factor is greater than or equal to the professional factor threshold and the accuracy factor is greater than or equal to the accuracy factor threshold, the factor analysis result is the first result. The first result is used to define the first answer as professional and without accuracy risk.

[0030] When the professional factor is greater than or equal to the professional factor threshold and the accuracy factor is less than the accuracy factor threshold, the factor analysis result is the second result. The second result is used to define the first answer as professional but with an accuracy risk.

[0031] When the professional factor is less than the professional factor threshold and the accuracy factor is greater than or equal to the accuracy factor threshold, the factor analysis result is the third result, which is used to define the first answer as unprofessional but without accuracy risk.

[0032] When the professional factor is less than the professional factor threshold and the accuracy factor is less than the accuracy factor threshold, the factor analysis result is the fourth result, which is used to define the first answer as unprofessional and at risk of inaccuracy.

[0033] Preferably, a direct output instruction is obtained from the first result;

[0034] For the second result, a knowledge point falsification instruction is generated, which includes a list of risk knowledge points to be verified and reference knowledge in the corresponding field in the customer service knowledge base;

[0035] For the third result, a script reconstruction instruction is generated, which includes a script template matching scheme;

[0036] For the fourth result, a priority optimization instruction is generated, wherein the priority optimization instruction executes the knowledge point falsification instruction first, and then executes the speech reconstruction instruction.

[0037] Preferably, the knowledge point falsification instruction is to mark the knowledge points in the first answer that do not match the customer service knowledge base and determine the risk level. The risk level is divided based on the number of knowledge points that are not in the customer service knowledge base and the importance of the knowledge points that are not in the customer service knowledge base in the first answer.

[0038] The script reconstruction instruction analyzes the deviation between the script of the first answer and the customer service knowledge base.

[0039] Preferably, the knowledge point falsification instruction also includes:

[0040] The system retrieves basic knowledge points related to the domain of the problem from the customer service knowledge base to construct a minimum knowledge unit set. Based on the minimum knowledge unit set, it performs a consistency check on the risk knowledge points. If the check fails, it retrieves valid knowledge associated with the risk knowledge point from the minimum knowledge unit set to replace the risk knowledge point.

[0041] Preferably, obtaining the second answer based on the factor analysis results and the first answer includes:

[0042] For the first result, the first answer is directly output as the second answer;

[0043] For the second result, keep the wording structure of the first answer unchanged, and only perform the following operations on the risk knowledge point: if the falsification verification passes, retain the risk knowledge point; if the falsification verification fails but there is related replacement knowledge, replace it with related knowledge; if the falsification verification fails and there is no related replacement knowledge, generate a professional response framework based on the minimum knowledge unit set.

[0044] For the third result, the verbal structure of the first answer is modified based on the verbal reconstruction instruction, while keeping the knowledge points of the first answer unchanged;

[0045] For the fourth result, first process the risk knowledge points, then complete the script optimization according to the script reconstruction instruction, and output a second answer that conforms to the script specifications and whose knowledge points have been verified by the knowledge base.

[0046] Beneficial Effects: The intelligent customer service system based on AI question answering and a large language model in this application uses professional factors to quantitatively evaluate the standardization of the first answer generated by the large language model. Combined with domain-specific dialogue templates from the customer service knowledge base, it achieves precise matching, ensuring that the output second answer conforms to industry terminology, sentence structure, and response framework requirements. This solves the problems of colloquial language and non-standard terminology in existing systems, improving the professionalism and authority of customer service responses. Accurate factors are used to perform multi-dimensional verification of knowledge points, and risky knowledge points are falsified and replaced through the smallest knowledge unit set, ensuring the accuracy and credibility of the answers. Targeted processing of dual factors through factor analysis units forms a closed loop of generation, analysis, and optimization, ensuring that professional, compliant, accurate, and reliable responses are output in various scenarios. In particular, it meets the core requirements of customer service scenarios—that what cannot be omitted, what cannot be said incorrectly, and what is said must conform to standards—effectively improving user experience and system credibility. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a structural block diagram of an intelligent customer service system based on AI question answering and a large language model, provided for an embodiment of this application. Detailed Implementation

[0049] The technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0050] In this document, the term "comprising" is intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0051] To address the issues of non-standard wording and fabricated knowledge points in existing intelligent customer service systems when generating answers, this embodiment provides an intelligent customer service system based on AI question answering and a large language model. This system achieves a closed-loop process of generation, analysis, and optimization through modular design. Specifically, the system includes three core modules: an answer generation module for initial answer generation, an answer analysis module for quantitatively evaluating answer quality, and an answer output module for optimizing output results. These three modules work together to improve the professionalism and accuracy of customer service responses.

[0052] Specifically, such as Figure 1 As shown, this embodiment discloses an intelligent customer service system based on AI question answering and a large language model. The system includes:

[0053] The answer generation module is used to analyze the user's question using AI question answering and large language models to obtain the first answer to the question.

[0054] The answer analysis module is used to obtain professional factors and accuracy factors from the customer service knowledge base. Professional factors are used to define the professionalism of the first answer's wording, and accuracy factors are used to define the accuracy of the first answer's response.

[0055] The answer output module is used to optimize the wording professionalism and / or answer accuracy of the first answer based on the professional factor and the accuracy factor to obtain a second answer, and output the second answer as the answer to the user's question; wherein, optimizing the answer accuracy of the first answer based on the accuracy factor is to obtain the smallest knowledge unit set through the customer service knowledge base to replace the knowledge points used in the first answer.

[0056] This embodiment further defines the specific implementation process of how the answer generation module generates the first answer based on the user's question. To ensure that the generated initial answer is strongly relevant to the business domain, it is necessary to first analyze the core requirements and domain characteristics of the user's question, then retrieve relevant knowledge from the customer service knowledge base as constraints, and finally generate the first answer through a large language model. The model output is configured to prioritize calling knowledge base content to reduce the risk of generating answers without a basis, while retaining the advantages of the large language model's external knowledge base.

[0057] Specifically, by analyzing user questions through AI question answering and large language models, the system obtains the first answer to the question, including:

[0058] Perform semantic analysis on the problem to extract core demands and domain features;

[0059] Based on domain characteristics, retrieve related knowledge sets from the customer service knowledge base;

[0060] The core requirements and related knowledge set are input into the large language model to generate the first answer; the output of the large language model is configured to prioritize the use of content from the related knowledge set.

[0061] It should be noted that the AI ​​question-answering and large language model used in this embodiment can be any existing AI question-answering and large language model suitable for customer service systems.

[0062] To quantitatively assess the professionalism of the first answer's wording, the answer analysis module needs to calculate a professional factor. In this embodiment, the professional factor is obtained by relying on the domain-specific wording templates in the customer service knowledge base, which contain industry terminology, sentence structures, and frameworks. By analyzing the similarity between the first answer and the template and quantifying this similarity (the higher the similarity, the stronger the professionalism), the final quantitative result is defined as the professional factor, thereby achieving an objective assessment of the wording's standardization.

[0063] Specifically, professional factors are obtained through the customer service knowledge base, including:

[0064] Extract pre-set domain-specific dialogue templates corresponding to the questions from the customer service knowledge base. These templates include industry terminology, sentence structure, and response framework.

[0065] Analyze the similarity between the first answer and the domain-specific script template, quantify the similarity to ensure that the similarity is positively correlated with the professionalism of the script, and define the similarity as the professional factor output.

[0066] Regarding the calculation logic of the accuracy factor, this embodiment ensures the comprehensiveness of the evaluation through multi-dimensional verification. First, the knowledge point set of the first answer is analyzed, and then consistency verification is performed with authoritative knowledge in the knowledge base. At the same time, the correlation between the external knowledge called by the large language model and the knowledge base is analyzed. Finally, the accuracy factor is obtained by weighting the knowledge point matching ratio and correlation. This factor is negatively correlated with the accuracy of the answer (the lower the factor, the higher the risk of fabrication), thus achieving precise control over the authenticity of the knowledge points.

[0067] Specifically, accurate information is obtained through the customer service knowledge base, including:

[0068] Analyze the knowledge points in the first answer to obtain a set of knowledge points, which includes factual statements, data, and logical relationships.

[0069] Perform consistency checks between the knowledge point set and the authoritative knowledge in the customer service knowledge base, and count the percentage of matching knowledge points.

[0070] Analyze the correlation between the external knowledge called by the large language model when generating the first answer and the corresponding domain knowledge in the customer service knowledge base;

[0071] Based on the proportion and relevance of knowledge points, an accuracy factor is obtained, and the accuracy factor is negatively correlated with the accuracy of the answer.

[0072] The answer output module, as the final optimization stage of the system, focuses on generating reliable answers based on the results of a two-factor evaluation. This embodiment divides this module into a factor analysis unit and an answer optimization unit: the former determines whether the first answer meets the standards for professionalism and accuracy by using a preset threshold, and outputs the factor analysis results; the latter performs targeted optimization based on these results to ensure that the final output second answer conforms to the wording standards and has no accuracy risks.

[0073] Specifically, the answer output module includes:

[0074] The factor analysis unit is used to analyze the first answer based on preset professional factor thresholds and accurate factor thresholds, and obtain factor analysis results. The factor analysis results are used to define the first answer's level of professionalism and accuracy.

[0075] The answer optimization unit is used to obtain a second answer that is both professional in terms of wording and free from the risk of answering inaccuracies, based on the factor analysis results and the first answer.

[0076] To clarify the evaluation criteria for the factor analysis unit, this embodiment divides the first answer into four states. Based on preset professional factor thresholds and accurate factor thresholds, a combination of two factors is used to provide a clear basis for the selection of subsequent optimization strategies.

[0077] Specifically, based on the preset professional factor threshold and accurate factor threshold, the first answer is analyzed to obtain factor analysis results, including:

[0078] When the professional factor is greater than or equal to the professional factor threshold and the accuracy factor is greater than or equal to the accuracy factor threshold, the factor analysis result is the first result. The first result is used to define the first answer as professional and without accuracy risk.

[0079] When the professional factor is greater than or equal to the professional factor threshold and the accuracy factor is less than the accuracy factor threshold, the factor analysis result is the second result. The second result is used to define the first answer as having professional wording but with a risk of inaccuracy.

[0080] When the professional factor is less than the professional factor threshold and the accuracy factor is greater than or equal to the accuracy factor threshold, the factor analysis result is the third result. The third result is used to define the first answer as unprofessional but without the risk of inaccuracy.

[0081] When the professional factor is less than the professional factor threshold and the accuracy factor is less than the accuracy factor threshold, the factor analysis result is the fourth result. The fourth result is used to define the first answer as unprofessional and at risk of inaccuracy.

[0082] For the four results output by the factor analysis unit, this embodiment further defines the corresponding processing instructions. For the first answer in different states, the system generates differentiated instructions: directly output a risk-free answer, falsify risky knowledge points, reconstruct unprofessional statements, or perform combination optimization according to priority, ensuring that each type of result has a clear processing path.

[0083] Specifically, for the first result, a direct output instruction is obtained;

[0084] For the second result, generate knowledge point falsification instructions, which include a list of risk knowledge points to be verified and reference knowledge in the corresponding field in the customer service knowledge base;

[0085] For the third result, generate a script reconstruction instruction, which includes a script template matching scheme;

[0086] For the fourth result, a priority optimization instruction is generated. The priority optimization instruction is to execute the knowledge point falsification instruction first, and then execute the speech reconstruction instruction.

[0087] It should be noted that when generating answers based on existing large language models, the most common risk is that the large language model generates answers that are relevant to the question but do not conform to the knowledge polarity of the customer service system, i.e., answers based on pseudo-knowledge. This embodiment addresses this risk by proposing knowledge point falsification of the answers to ensure that customer service representatives do not respond based on incorrect knowledge. Unlike existing technologies that filter knowledge based on the question, this embodiment directly improves the accuracy of answers by using a dynamic customer service knowledge base and falsifying knowledge points based on the answers. It should be noted that the customer service knowledge base in this embodiment can be any existing online, dynamically updated customer service knowledge base, intended to provide a reference for answer falsification.

[0088] To refine the execution content of the instructions, this embodiment sets limitations on the knowledge point falsification instructions and the speech reconstruction instructions. The falsification instructions need to mark the risky knowledge points that do not match the knowledge base and classify them according to their quantity and importance; the speech reconstruction instructions need to analyze the deviation between the first answer and the knowledge base template to provide a quantitative basis for subsequent precise optimization.

[0089] Specifically, the knowledge point falsification instruction is to mark the knowledge points in the first answer that do not match the customer service knowledge base, determine the risk level, and classify the risk level based on the number of knowledge points outside the customer service knowledge base and the importance of the knowledge points outside the customer service knowledge base in the first answer;

[0090] Analysis of the deviation between the first answer of the script reconstruction instruction and the customer service knowledge base.

[0091] Regarding the specific execution process of knowledge point falsification instructions, this embodiment introduces a minimum set of knowledge units as a verification benchmark. A core fact base is constructed by retrieving fundamental domain knowledge points to perform consistency verification on risky knowledge points. If the verification fails, valid knowledge is retrieved from the latest version of the customer service knowledge base to replace the risky content, ensuring the reliability of the replacement process.

[0092] Specifically, the instructions for falsifying knowledge points also include:

[0093] Retrieve basic knowledge points related to the domain of the problem from the customer service knowledge base to construct a minimum knowledge unit set; perform consistency verification on risk knowledge points based on the minimum knowledge unit set; if the verification fails, call valid knowledge associated with the risk knowledge point from the minimum knowledge unit set to replace the risk knowledge point.

[0094] As the final execution logic of the answer optimization unit, this embodiment clarifies the second answer generation methods corresponding to the four factor analysis results. Risk-free answers are output directly; risky knowledge points are replaced or professional response frameworks are generated as needed; unprofessional phrases are structurally corrected or optimized by priority combination, ultimately ensuring that the output second answer simultaneously meets the requirements of both phrase standardization and accuracy.

[0095] Specifically, based on the factor analysis results and the first answer, the second answer is obtained, including:

[0096] For the first result, output the first answer directly as the second answer;

[0097] For the second result, the wording structure of the first answer remains unchanged, and the following operations are performed only on the risk knowledge points: if the falsification verification passes, the risk knowledge point is retained; if the falsification verification fails but there is related replacement knowledge, it is replaced with related knowledge; if the falsification verification fails and there is no related replacement knowledge, a professional response framework is generated based on the minimum knowledge unit set.

[0098] For the third result, the verbal structure of the first answer is modified based on the verbal reconstruction instruction, while keeping the knowledge points of the first answer unchanged;

[0099] For the fourth result, first process the risk knowledge points, then complete the script optimization according to the script reconstruction instructions, and output the second answer that conforms to the script specifications and whose knowledge points have been verified by the knowledge base.

[0100] In summary, the intelligent customer service system based on AI question answering and a large language model in this embodiment quantitatively evaluates the standardization of the first answer generated by the large language model through professional factors. It then achieves precise matching by combining domain-specific dialogue templates from the customer service knowledge base, ensuring that the output second answer conforms to industry terminology, sentence structure, and response framework requirements. This solves the problems of colloquial language and non-standard terminology in existing systems, enhancing the professionalism and authority of customer service responses. Furthermore, it uses accurate factors to perform multi-dimensional verification of knowledge points and achieves the falsification and replacement of risky knowledge points through the smallest knowledge unit set, ensuring the accuracy and credibility of the answers. Through targeted processing of dual factors by the factor analysis unit, a closed loop of generation, analysis, and optimization is formed, ensuring that professional, compliant, accurate, and reliable responses are output in various scenarios. In particular, it meets the core requirements of customer service scenarios—that what cannot be omitted, what cannot be said incorrectly, and what is said must conform to standards—effectively improving user experience and system credibility.

[0101] In the embodiments provided in this application, it should be understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, code, or any suitable combination thereof. For hardware implementation, the processor may be implemented in one or more of the following: application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, other electronic units designed to implement the functions described herein, or combinations thereof. For software implementation, some or all of the processes of the embodiments may be performed by a computer program instructing the associated hardware. During implementation, the program may be stored in a computer-readable storage medium or transmitted as one or more instructions or code on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of a computer program from one place to another. Storage media may be any available medium accessible to a computer. Computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code having the form of instructions or data structures and accessible to a computer.

[0102] Finally, it should be noted that the above description is only a preferred embodiment of this application and is not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An intelligent customer service system based on AI question answering and large language models, characterized in that, The system includes: The answer generation module is used to perform semantic analysis on the user's question through AI question answering and a large language model to extract the core demand and domain features. Based on the domain features, it retrieves the relevant knowledge set from the customer service knowledge base and inputs the core demand and the relevant knowledge set into the large language model to generate the first answer. The answer analysis module is used to obtain professional factors and accuracy factors from the customer service knowledge base. The professional factors are used to define the similarity between the first answer and the preset domain-specific script template. The accuracy factors are used to quantify the accuracy of the first answer based on the consistency verification results between the knowledge points in the first answer and the customer service knowledge base. The answer output module includes a factor analysis unit and an answer optimization unit. The factor analysis unit is used to analyze the first answer based on preset professional factor thresholds and accuracy factor thresholds to obtain factor analysis results. The factor analysis results are used to define the first answer's professionalism and accuracy levels. The answer optimization unit is used to obtain a second answer that meets the professionalism requirements and has no risk of inaccuracy based on the factor analysis results and the first answer. Specifically, optimizing the accuracy of the first answer based on the accuracy factor involves obtaining the smallest set of knowledge units from the customer service knowledge base to replace the knowledge points used in the first answer.

2. The intelligent customer service system based on AI question answering and large language models according to claim 1, characterized in that, The output of the large language model is configured to preferentially invoke content from the associated knowledge set.

3. The intelligent customer service system based on AI question answering and large language models according to claim 1, characterized in that, The industry-specific terminology template includes an industry terminology system, sentence structure, and response framework.

4. The intelligent customer service system based on AI question answering and large language models according to claim 1, characterized in that, The accuracy factors are obtained through the customer service knowledge base, including: Analyze the knowledge points in the first answer to obtain a set of knowledge points, which includes factual statements, data, and logical relationships. Perform consistency verification between the knowledge point set and the authoritative knowledge in the customer service knowledge base, and count the percentage of matching knowledge points. Analyze the correlation between the external knowledge called by the large language model when generating the first answer and the corresponding domain knowledge in the customer service knowledge base; Based on the proportion of the knowledge points and the degree of correlation, an accuracy factor is obtained, and the accuracy factor is negatively correlated with the accuracy of the answer.

5. The intelligent customer service system based on AI question answering and large language models according to claim 1, characterized in that, The first answer is analyzed based on preset professional factor thresholds and accurate factor thresholds to obtain factor analysis results, including: When the professional factor is greater than or equal to the professional factor threshold and the accuracy factor is greater than or equal to the accuracy factor threshold, the factor analysis result is the first result. The first result is used to define the first answer as professional and without accuracy risk. When the professional factor is greater than or equal to the professional factor threshold and the accuracy factor is less than the accuracy factor threshold, the factor analysis result is the second result. The second result is used to define the first answer as professional but with an accuracy risk. When the professional factor is less than the professional factor threshold and the accuracy factor is greater than or equal to the accuracy factor threshold, the factor analysis result is the third result, which is used to define the first answer as unprofessional but without accuracy risk. When the professional factor is less than the professional factor threshold and the accuracy factor is less than the accuracy factor threshold, the factor analysis result is the fourth result, which is used to define the first answer as unprofessional and at risk of inaccuracy.

6. The intelligent customer service system based on AI question answering and large language models according to claim 5, characterized in that, For the first result, a direct output instruction is obtained; For the second result, a knowledge point falsification instruction is generated, which includes a list of risk knowledge points to be verified and reference knowledge in the corresponding field in the customer service knowledge base; For the third result, a script reconstruction instruction is generated, which includes a script template matching scheme; For the fourth result, a priority optimization instruction is generated, wherein the priority optimization instruction executes the knowledge point falsification instruction first, and then executes the speech reconstruction instruction.

7. The intelligent customer service system based on AI question answering and large language models according to claim 6, characterized in that, The knowledge point falsification instruction is to mark the knowledge points in the first answer that do not match the customer service knowledge base, and determine the risk level. The risk level is divided based on the number of knowledge points that are not in the customer service knowledge base and the importance of the knowledge points that are not in the customer service knowledge base in the first answer. The script reconstruction instruction analyzes the deviation between the script of the first answer and the customer service knowledge base.

8. The intelligent customer service system based on AI question answering and large language models according to claim 7, characterized in that, The knowledge point falsification instructions also include: Retrieve basic knowledge points related to the domain of the problem from the customer service knowledge base to construct the smallest set of knowledge units; Based on the set of minimum knowledge units, consistency verification is performed on the risk knowledge points; If the verification fails, valid knowledge associated with the risk knowledge point is retrieved from the set of smallest knowledge units to replace the risk knowledge point.

9. The intelligent customer service system based on AI question answering and large language models according to claim 8, characterized in that, The second answer is obtained by: For the first result, the first answer is directly output as the second answer; For the second result, keep the wording structure of the first answer unchanged, and only perform the following operations on the risk knowledge point: if the falsification verification passes, retain the risk knowledge point; if the falsification verification fails but there is related replacement knowledge, replace it with related knowledge; if the falsification verification fails and there is no related replacement knowledge, generate a professional response framework based on the minimum knowledge unit set. For the third result, the verbal structure of the first answer is modified based on the verbal reconstruction instruction, while keeping the knowledge points of the first answer unchanged; For the fourth result, first process the risk knowledge points, then complete the script optimization according to the script reconstruction instruction, and output a second answer that conforms to the script specifications and whose knowledge points have been verified by the knowledge base.

Images