Customer service response correction method and device, equipment and medium

Abstract

The invention relates to a customer service response correction method and device, equipment and medium in the technical field of e-commerce, and the method comprises the steps: carrying out the preprocessing of each manual customer service session in a manual customer service session set, and obtaining a target session set; adopting a large language model to determine each standard consultation text describing a corresponding user consultation intention in the target session set; when customer service knowledge materials related to the standard consultation text exist in a preset customer service knowledge base, all the customer service knowledge materials are recalled and used for generating a corresponding to-be-evaluated response text; and when the answer assessment text of the to-be-assessed response text represents an answer error, executing a corresponding repair process according to an error reason in the answer assessment text so as to regenerate a corresponding correct response text after repair. According to the invention, response defects can be accurately repaired, so that high-quality intelligent customer service is provided.

Description

Technical Field

[0001] This application relates to the field of e-commerce technology, and in particular to a customer service response correction method and corresponding apparatus, computer equipment, and computer-readable storage medium. Background Technology

[0002] In the practice of intelligent customer service, when faced with inaccurate response text generated by automated customer service, the existing solution is to directly add the content of human responses from conversations with human-provided customer service as new knowledge to a pre-set customer service knowledge base. The core logic of this method is to attribute the error in the generated response to the lack of corresponding answer material in the knowledge base, attempting to improve the accuracy of responses by expanding the data scale of the knowledge base to cover similar problems in the future.

[0003] However, this solution has significant drawbacks in principle. It essentially attributes the complex defects that may occur during response generation to a lack of knowledge base material and attempts to solve all types of errors through a single data expansion method. This approach lacks in-depth exploration of the root causes of errors and is a general and inefficient optimization method. Because it fails to distinguish the specific technical aspects that cause errors, blindly and continuously adding knowledge material leads to disordered growth of the knowledge base, potentially resulting in redundant, contradictory, or fragmented information. This, in turn, makes subsequent knowledge retrieval and response generation face greater noise interference and consistency challenges, ultimately reducing overall efficiency rather than improving it.

[0004] Therefore, the main drawback of the prior art is that its correction mechanism is general and unspecific. In view of the shortcomings of the prior art, this application takes a different approach. Summary of the Invention

[0005] The primary objective of this application is to solve at least one of the aforementioned problems by providing a customer service response correction method and corresponding apparatus, computer equipment, and computer-readable storage medium.

[0006] To achieve the various objectives of this application, the following technical solution is adopted: A customer service response correction method provided for one of the purposes of this application includes the following steps: Preprocess each human customer service session in the human customer service session set to obtain the target session set; A large language model is used to determine the standard consultation texts in the target conversation set that describe the corresponding user's consultation intent; When customer service knowledge materials related to the standard consultation text exist in the preset customer service knowledge base, all customer service knowledge materials are recalled to generate the corresponding response text to be evaluated. When the solution evaluation text of the response text to be evaluated indicates an error, the corresponding repair process is executed according to the reason for the error in the solution evaluation text, so as to regenerate the corresponding correct response text after the repair.

[0007] On the other hand, a customer service response correction device provided to meet one of the purposes of this application includes a session preprocessing module, a consultation determination module, a response generation module, and an error repair module. The session preprocessing module is used to preprocess each human customer service session in a set of human customer service sessions to obtain a target session set. The consultation determination module is used to determine, using a large language model, each standard consultation text in the target session set that describes the corresponding user's consultation intent. The response generation module is used to recall all customer service knowledge materials related to the standard consultation text in a preset customer service knowledge base, and to generate a corresponding response text to be evaluated. The error repair module is used to execute a corresponding repair process based on the error cause in the response text to be evaluated when the answer evaluation text of the response text to be evaluated indicates an error, so as to regenerate the corresponding correct response text after repair.

[0008] In another aspect, a computer device provided for one of the purposes of this application includes a central processing unit and a memory, the central processing unit being configured to invoke and run a computer program stored in the memory to perform the steps of the customer service response correction method described in this application.

[0009] In another aspect, a computer program product provided for another purpose of this application includes a computer program / instructions that, when executed by a processor, implement the steps of the method described in any embodiment of this application.

[0010] The technical solution of this application has many advantages, including but not limited to the following aspects: This application first generates a response text to be evaluated based on the existing customer service knowledge base, and uses the accuracy of the final answer in this text as the starting point and guide for evaluation. When an answer is determined to be incorrect, the approach is not to attribute the cause in a general way, but to trace back to the specific technical steps that caused the error, and to perform targeted repairs based on the precisely located cause in order to regenerate the correct answer. It is evident that by transforming the correction logic from "simple data appending" to "result-oriented effective attribution and targeted repair," the disorderly expansion and content contradictions of the knowledge base are effectively suppressed, fundamentally improving the accuracy and overall effectiveness of customer service optimization.

[0011] Secondly, the original conversation is abstracted and standardized using a large language model to generate standard consultation text that accurately describes the user's consultation intent. This establishes a unified and clear semantic benchmark for subsequent knowledge retrieval and response evaluation, significantly enhancing the understanding and generalization ability of diverse natural language expressions used by users. The retrieval and generation process based on standard consultation effectively reduces semantic noise and improves the relevance and accuracy of the initial response, thus laying a reliable foundation for efficient backward reasoning and attribution based on incorrect results.

[0012] Most importantly, by introducing a closed-loop process of "assessment-attribution-correction," the intelligent customer service system achieves dynamic self-optimization. When a generated response is judged to be incorrect, it doesn't simply append data; instead, it triggers corresponding correction strategies based on the identified specific cause of the error and iteratively generates the correct result. This process enables continuous accumulation of corrective experience, precisely strengthening weak points. While ensuring the refinement of the knowledge base, it achieves stable and sustainable improvement in response quality, providing an efficient technical path for the long-term evolution of intelligent customer service. Attached Figure Description

[0013] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 The network architecture of the e-commerce platform exemplified in this application; Figure 2 This is a flowchart illustrating a typical embodiment of the customer service response correction method of this application; Figure 3 This is a schematic block diagram of the customer service response correction device of this application; Figure 4 This is a schematic diagram of the structure of a computer device used in this application. Detailed Implementation

[0014] The following describes in detail Embodiment 1 of this application. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0015] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0016] Those skilled in the art will understand that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0017] like Figure 1 In the network architecture shown, the e-commerce platform 82 is deployed on the Internet to provide corresponding services to its users. Similarly, the devices 80 of the merchant users and the devices 81 of the consumer users of the e-commerce platform 82 are also connected to the Internet to use the services provided by the e-commerce platform.

[0018] An exemplary e-commerce platform 82 provides supply and demand matching of products and / or services to the general public through the Internet infrastructure. In e-commerce platform 82, products and / or services are provided as commodity information. For the sake of simplicity, the concepts of commodity and product are used in this application to refer to the products and / or services in e-commerce platform 82. Specifically, these may be physical products, digital products, tickets, service subscriptions, other offline services, etc.

[0019] In reality, various entities can access e-commerce platform 82 as users and utilize its online services to participate in the business activities facilitated by the platform. These entities can be natural persons, legal persons, or social organizations. Corresponding to the two types of entities in business activities—merchants and consumers—e-commerce platform 82 has two corresponding categories of users: merchant users and consumer users. Entities involved in the product distribution chain in business activities, including manufacturers, sellers, retailers, and logistics providers, can all use online services on e-commerce platform 82 as merchant users. Similarly, consumers in business activities, including actual or potential consumers, can use online services on e-commerce platform 82 as consumer users. In actual business activities, the same entity can operate as both a merchant user and a consumer user; this should be interpreted flexibly.

[0020] The infrastructure used to deploy the e-commerce platform 82 mainly includes the backend architecture and frontend devices. The backend architecture runs various online services through a service cluster, including middleware or frontend services for the platform, services for consumers, and services for merchants, to enrich and improve its service functions. The frontend devices mainly cover the terminal devices used by users as clients to access the e-commerce platform 82, including but not limited to various mobile terminals, personal computers, and point-of-sale devices. For example, merchant users can use their terminal device 80 to enter product information for their online stores or use the interfaces opened by the e-commerce platform to generate their product information; consumer users can use their terminal device 81 to access the webpage of the online store implemented by the e-commerce platform 82, trigger the shopping process by clicking the shopping button provided on the webpage, and call various online services provided by the e-commerce platform 82 during the shopping process to achieve the purpose of placing an order.

[0021] In some embodiments, the e-commerce platform 82 may be implemented via a processing facility including a processor and memory, which stores a set of instructions that, when executed, cause the e-commerce platform 82 to perform the e-commerce and support functions as described in this application. The processing facility may be part of a server, client, network infrastructure, mobile computing platform, cloud computing platform, fixed computing platform, or other computing platform, and may provide electronic components, merchant devices, payment gateways, application developers, marketing channels, transportation providers, customer devices, point-of-sale devices, etc., for the e-commerce platform 82.

[0022] E-commerce platform 82 can provide online services such as cloud computing services, Software as a Service (SaaS), Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Desktop as a Service (DaaS), Hosted Software as a Service, Mobile Backend as a Service (MBaaS), and Information Technology Management as a Service (ITMaaS). In some embodiments, the various functional components of e-commerce platform 82 can be implemented to operate on various platforms and operating systems. For example, for an online store, its administrator user enjoys the same or similar functions regardless of whether it is on iOS, Android, HomonyOS, or a web page.

[0023] E-commerce platform 82 enables merchants to create their own independent websites to run their online stores. It provides merchants with corresponding business management engine instances, allowing them to establish, maintain, and operate one or more online stores across these independent websites. The business management engine instance can be used for content management, task automation, and data management for one or more online stores. It can be configured through interfaces or built-in components to support various specific business processes in the online store, supporting business activities. Independent websites are the infrastructure of e-commerce platform 82, which offers cross-border services. Merchants can maintain their online stores relatively independently and centrally based on these independent websites. Independent websites typically have dedicated domain names and storage space, and different independent websites are relatively independent. E-commerce platform 82 can provide standardized or customized technical support for a large number of independent websites, allowing merchants to customize a business management engine instance that suits their needs and use it to maintain one or more online stores.

[0024] Online stores can be configured and maintained in the backend by merchant users logging into their Business Management Engine instance as administrators. Supported by the various online services provided by the e-commerce platform 82's infrastructure, merchant users can configure various functions within their online stores and view various data as administrators. For example, merchant users can manage various aspects of their online stores, such as viewing recent online store activities, updating the online store's product catalog, managing orders, recent visit activity, and total order activity. Merchant users can also view more detailed information about their business and visitors to their online store by obtaining reports or metrics, such as displaying a sales summary of the merchant's overall business, specific sales and engagement data from promotional sales and marketing channels, etc.

[0025] E-commerce platforms 82 can provide communication facilities and associated merchant interfaces for electronic communication and marketing. For example, they can utilize electronic messaging aggregation facilities to collect and analyze communication interactions between merchants, consumers, merchant devices, customer devices, point-of-sale devices, etc., aggregating and analyzing communications to increase the potential for product sales. For instance, a consumer may have product-related questions, which could lead to a dialogue between the consumer and the merchant (or an automated processor-based agent representing the merchant), where the communication facilities handle the interaction and provide the merchant with analysis on how to increase the probability of a sale.

[0026] In some embodiments, applications suitable for installation on devices can be provided to serve the access needs of different users, enabling various users to access the e-commerce platform 82 by running the application on their terminal devices. Examples include the merchant backend module of online stores within the e-commerce platform 82. During the process of conducting business activities through these functions, the e-commerce platform 82 can implement various functions related to business activities as middleware or online services and expose corresponding interfaces. Then, toolkits corresponding to the interface access functions are embedded into the application to achieve functional expansion and task completion. The business management engine can include a series of basic functions and expose these functions to online services and / or applications via APIs. Online services and applications use the corresponding functions by remotely calling the corresponding APIs.

[0027] With the support of various components of the Business Management Engine instance, E-commerce Platform 82 can provide online shopping functionality, enabling merchants to connect with customers in a flexible and transparent manner. Consumers can select items online, create orders, provide delivery addresses in the orders, and complete payment confirmation. Merchants can then review and complete or cancel orders.

[0028] The customer service response correction method of this application can be programmed into a computer program product and deployed on a client or server for execution. For example, in an exemplary application scenario of this application, it can be deployed on the server of an e-commerce customer service platform. In this way, the method can be executed by human-computer interaction with the process of the computer program product through a graphical user interface by accessing the interface opened after the computer program product is running.

[0029] Please see Figure 2 The customer service response correction method of this application, in its typical embodiment, includes the following steps: Step S1100: Preprocess each human customer service session in the human customer service session set to obtain the target session set; In e-commerce customer service scenarios, human customer service conversation data serves as a high-quality optimization data source. Its core value lies in its ability to handle user inquiries that intelligent customer service cannot effectively address. These conversations are answered by humans because intelligent customer service cannot automate the process. However, not all human conversations have optimization value. Specifically, human customer service conversations can be categorized into four types: those where the human customer service representative did not resolve the user inquiry, those where the human customer service representative did not answer the user inquiry, those where the human customer service representative provided an incomplete answer, and those where the human customer service representative provided a correct and complete answer. Among these, only conversations where the human customer service representative provided a correct and complete answer have optimization value and can be used in subsequent response correction processes. The other three types of conversations, lacking a basis for a correct answer, have no substantial significance for the intelligent customer service optimization process and must be screened and eliminated before preprocessing.

[0030] The screening process employs a large language model for automated evaluation. The specific implementation includes: First, constructing an evaluation standard system. This system includes clearly defined criteria for determining correct answers, such as the answer content perfectly matching the user's question, containing necessary information points, complying with e-commerce platform rules, and avoiding misleading statements. Based on this, the large language model is used to perform semantic analysis on the human customer service conversation to evaluate whether the human response meets the above criteria. During evaluation, the large language model analyzes the user's inquiry content and the human response content to determine whether the response fully covers the core needs of the user's inquiry and provides an accurate and effective solution. Those skilled in the art can flexibly adapt the screening process based on this disclosure; this step is not detailed here.

[0031] After filtering, the effective human customer service conversations need to be preprocessed. The core objective is to transform the raw conversation data into a structured, denoised, and standardized format, laying the foundation for subsequent large language model processing.

[0032] Key preprocessing steps include removing invalid text, text denoising, sentence segmentation and structuring, and deleting sensitive information. Implementation methods for removing invalid text include building a standard invalid text library containing common greetings (such as "Hello," "Please wait a moment," "Thank you," and "Goodbye") and transitional phrases (such as "Let me check for you," and "I need to confirm this"). Batch filtering is then performed using string matching or regular expressions. For example, the original conversation fragment "Hello, I'd like to check my order status" is processed into "I'd like to check my order status."

[0033] Text denoising primarily targets non-text content in the conversation, including removing emoticons (such as kaomoji characters), special characters (such as "@", "#", and "!"), and illegal characters (such as control characters and garbled text). Implementation can employ regular expressions to match specific character sets or use predefined character filtering lists for cleanup. For example, the original conversation "Order number 123456, logistics status (≧-≦) shipped" becomes "Order number 123456, logistics status shipped" after processing.

[0034] Sentence segmentation and structured processing aim to break down long dialogues into logically complete sentence units, ensuring that each sentence contains independent semantic information. Implementation methods include punctuation-based segmentation (such as periods, question marks, and exclamation marks) and semantic-based segmentation (such as using sentence segmentation models). For example, the original conversation "I haven't received the phone I bought yesterday. Can you help me check the logistics? Also, I need to apply for a refund." is segmented into two structured sentences: "I haven't received the phone I bought yesterday. Can you help me check the logistics?" and "I need to apply for a refund." Deleting sensitive information is a necessary step in preprocessing to ensure data privacy and security. Implementation methods include using regular expressions to match common sensitive information patterns (such as phone numbers, email addresses, and ID card numbers) and replacing them with placeholders (such as whitespace or the * character). For example, the original conversation "My phone number is 13800138000, order number 123456" becomes "My phone number is, order number 123456" after processing.

[0035] During preprocessing, the temporal relationship of the conversation needs to be preserved to ensure the contextual coherence of multi-turn dialogues. In implementation, a timestamp or session number is added to each processed statement to maintain the chronological order of the dialogue. For example, the original conversation "User: Order status? Customer service: Shipped, expected delivery tomorrow. User: Can you help me track the logistics?" is processed to "1. Order status? 2. Shipped, expected delivery tomorrow. 3. Can you help me track the logistics?".

[0036] The filtered and preprocessed dataset contains structured, valid conversation content. Each statement has been standardized to remove noise and irrelevant information, preserving the core intent of the user's inquiry while ensuring data privacy and security. This dataset can be directly used for subsequent large language model processing, providing high-quality input for generating standard inquiry text, thereby improving the accuracy of customer service response correction. The resulting target conversation set contains multiple processed target human customer service conversations.

[0037] Step S1200: Use a large language model to determine the standard consultation texts in the target conversation set that describe the corresponding user's consultation intent; The Large Language Model is a model trained using deep learning techniques that can understand and generate natural language text. This model is pre-trained on a large amount of text data and has powerful language understanding and generation capabilities.

[0038] A large language model is used to perform in-depth semantic analysis and abstraction on the preprocessed multi-turn human customer service conversations in the target conversation set, in order to generate accurate, concise, and independent standard consultation texts. The core of this process is to transform the scattered, redundant, and unstructured natural language expressions in the original conversations into standardized consultation statements under a unified semantic benchmark, thereby providing a clear input basis for subsequent knowledge base-based response generation and evaluation.

[0039] Specifically, the large language model first receives the entire target conversation as input. This conversation has been stripped of greetings, transitional phrases, emoticons, special characters, and sensitive information, while retaining the original dialogue's temporal structure and contextual logic. Since human customer service conversations typically exhibit complex interaction patterns such as "one question with multiple answers," "multiple questions with one answer," or "interspersed questions and answers," a single statement often cannot fully reflect the user's true inquiry intent. Therefore, the large language model needs to comprehensively understand the entire conversation context, identifying the core requests, key entities (such as order numbers, payment methods, product names, and time points), and the context in which the inquiry occurred (such as "recharge not received," "logistics stalled," "refund rejected," etc.). It then synthesizes and refines a comprehensive inquiry expression that encompasses all valid information, eliminates irrelevant details, and conforms to common expression habits.

[0040] Understandably, large language models can leverage their pre-trained language understanding and generation capabilities to perform three key operations: consultation abstraction, redundant content elimination, and consultation standardization. Consultation abstraction refers to generalizing personalized descriptions from specific instances (such as "I recharged 100 yuan with Alipay yesterday but it hasn't arrived") into a general consultation template (such as "What should I do if the funds haven't arrived after recharging 100 yuan with Alipay yesterday?"), stripping away unnecessary variables related to user identity and retaining the essential structure of the question. Redundant content elimination refers to filtering out dialogue fragments that do not affect the core intent, such as repeated questions, probing statements, and confirmatory replies. For example, if a user repeatedly asks "Is it not ready yet?" or customer service repeatedly confirms "Are you referring to order 123456?", these will not be included in the standard consultation text during the summary. Question standardization ensures that the output question sentences conform to grammatical norms, are semantically complete, and unambiguous, and adopts the terminology system commonly used on e-commerce platforms, such as uniformly using "refund application" instead of "refund" and "logistics status" instead of "where is the package?".

[0041] The above process can be achieved by invoking the prompt engineering mechanism of the large language model. Structured input containing the complete conversation history is sent to the large language model, along with explicit instruction prompts, such as: "Based on the following customer service dialogue, summarize a standard question that best represents the user's core consultation intent. The question should be clearly and completely stated, without specific personal information, and suitable for knowledge base retrieval. Customer service dialogue {embedded target human customer service dialogue}." The large language model then generates a single, focused standard consultation text. For example, for the original dialogue: "User: I haven't received the phone I bought yesterday. Can you help me check the logistics? Also, I need to apply for a refund. Customer Service: The order has been shipped and is expected to arrive tomorrow. Regarding the refund, please provide the reason for not receiving the goods. User: I don't want to wait anymore, I want a refund now." After processing, two standard consultation texts can be generated: "What should be done if the order has been shipped but the user requests an early refund?" or more precisely, "How do I check the logistics status of a shipped order?" and "What is the process for applying for a refund before receiving the goods?", depending on the granularity of the conversation clustering and intent segmentation design. Those skilled in the art can configure the prompt text here as needed based on the disclosure herein.

[0042] The aforementioned technical approach enables the efficient conversion of massive, disorganized human customer service conversations into high-quality standard consultation texts. These texts not only retain the semantic essence of the original user intent but also possess excellent structure and searchability, laying a reliable foundation for subsequent steps to generate evaluation response texts based on the existing customer service knowledge base. Simultaneously, it avoids knowledge recall bias or semantic distortion caused by problems in the original expression of the consultation intent.

[0043] Step S1300: When there are customer service knowledge materials related to the standard consultation text in the preset customer service knowledge base, recall all customer service knowledge materials to generate the corresponding response text to be evaluated. Based on a pre-defined customer service knowledge base, a knowledge retrieval operation is performed on the standard consultation text generated from previous steps to obtain relevant customer service knowledge materials. These materials are then used to generate the response text to be evaluated. This process employs a Retrieval-Augmented Generation (RAG) algorithm specifically implemented in an e-commerce customer service scenario. Its purpose is to support the generation of accurate, compliant, and context-consistent responses by a large language model through structured and / or unstructured knowledge content. Understandably, this implementation is fundamental to supporting intelligent customer service.

[0044] The customer service knowledge base can contain various types or any single type of customer service knowledge documents. These documents, as the original data source, are fully preserved in the knowledge base to maintain the integrity and traceability of the knowledge content. Retaining the original documents helps provide necessary contextual basis during subsequent knowledge maintenance, updates, or corrections, avoiding modification deviations or semantic breaks caused by relying solely on fragmented data. Simultaneously, to adapt to the efficient retrieval requirements of the RAG mechanism, the document content is preprocessed while retaining the original documents to extract knowledge point units that can be used for semantic matching and response generation. These knowledge point units do not replace the original documents but exist as a retrieval index structure to accelerate similarity calculation and related knowledge retrieval.

[0045] Specifically, document types can include the following three structural types: The first type is a knowledge point document, in which multiple independent knowledge points are explicitly organized. Each knowledge point contains a historical standard consultation text, several similar consultation texts with different expressions but semantic equivalence, and the corresponding standard response text; the second type is a document with a hierarchical chapter structure, such as using Markdown, HTML, or rich text format, and organizing content through structural elements such as headings and subheadings; the third type is a continuous text document without explicit structure, in which the entire text does not have clearly defined knowledge boundaries, and the content is presented in the form of paragraphs or sentence flow.

[0046] For the above-mentioned different document types, corresponding preprocessing strategies are adopted when constructing the retrieval index. For knowledge point-type documents, each knowledge point can be directly extracted as an index unit, and each unit retains its internal consultation-response structure relationship. For chapter-structured documents, logical segmentation is performed according to their hierarchical structure. For example, each title and its subordinate text content can be treated as a knowledge point unit. The segmentation granularity can be configured according to the actual application scenario, such as by first-level title, second-level title, or mixed hierarchy. For unstructured documents, they are first initially segmented by sentences or semantic paragraphs. Then, a BERT-like model pre-trained to convergence in the customer service vertical domain is used to determine whether there is semantic coherence between adjacent text segments. Semantic breaks are used as segmentation points to generate semantically cohesive knowledge point units.

[0047] During the recall phase, using the current standard consultation text as the query input, its semantic relevance to various knowledge point units is calculated, and candidate customer service knowledge materials are selected based on the relevance scores. For units originating from knowledge point-type documents, the historical standard consultation texts and similar consultation texts within them are semantically encoded, and the vector similarity between the current standard consultation text and these texts is calculated. The target similarity of the knowledge point is obtained through weighted fusion (e.g., assigning higher weights to historical standard consultation texts and taking the maximum similarity of similar consultation texts before weighting; this can be configured as needed by those skilled in the art). For units originating from chapter-structured documents, the title (summary text) and body text (detailed text) are encoded separately, and the similarity between the current standard consultation text and both is calculated and weighted summed. For units originating from unstructured documents, a semantic encoding model suitable for long texts is used to vectorize the entire knowledge point content. This is usually achieved by averaging the vectors of all tokens or using a pooling strategy to obtain the overall representation, and then calculating its similarity to the current standard consultation text.

[0048] After similarity calculation, recall is performed according to a preset threshold strategy: For units generated from knowledge point-based documents, if their target similarity exceeds the first preset threshold, their corresponding standard response text is recalled as customer service knowledge material; for units generated from chapter-structured and unstructured documents, if their target similarity exceeds the second preset threshold (this threshold may be the same as or different from the first threshold, depending on the actual accuracy and recall requirements), their complete knowledge point content is recalled as customer service knowledge material. All recalled customer service knowledge material constitutes the contextual basis required for generating a response.

[0049] When generating the response text to be evaluated, a structured prompt is constructed. This prompt includes the current standard consultation text and all recalled customer service knowledge materials, along with explicit instructions requiring the large language model to strictly reason and generate based on the provided knowledge materials, without introducing external knowledge or subjective speculation. The text output by the large language model based on this prompt is the response text to be evaluated, used for subsequent error assessment and targeted remediation processes.

[0050] Step S1400: When the solution evaluation text of the response text to be evaluated indicates an error, a corresponding repair process is executed based on the error cause in the solution evaluation text to regenerate the corresponding correct response text after repair. This includes the following steps: Step S2400: From the target human customer service conversation from which the standard consultation text in the target conversation set originates, the corresponding human response text is determined using a large language model; When it is necessary to correct the responses generated by the intelligent customer service, the target human customer service conversation corresponding to the standard consultation text is first extracted from the target conversation set. Within this conversation, a large language model is used to semantically parse the multi-turn dialogue content, accurately locating the final response provided by the human customer service representative. This process is achieved by constructing structured prompts, which include the complete dialogue history and explicit instructions, such as: "Please extract the customer service representative's final answer to the user's core question from the following customer service dialogue, requiring only substantive solutions and excluding non-substantive content such as confirmations. Dialogue content: {complete conversation text}; User's core question: {standard consultation text}". The large language model outputs a structured human response text based on this prompt. Those skilled in the art can configure this prompt text as needed based on the disclosure herein.

[0051] In the recommended embodiment, a multi-model voting mechanism can also be adopted, which calls multiple large language models with different parameter scales to independently extract responses, performs consistency verification on the results, and directly adopts the response when the outputs of each model are highly consistent, and initiates a manual review process when there are discrepancies to ensure the accuracy of the manual response text.

[0052] Step S2410: When there is a substantial difference in the corresponding answer content when the manual response text and the response text to be evaluated are used to answer the standard consultation text, the answer judgment result in the answer evaluation text of the response text to be evaluated is determined to be an answer error.

[0053] After obtaining the human response text, a large language model is used to determine the substantive content differences between the human response text and the response text to be evaluated. This process guides the large language model to focus on the essential content of the answer rather than its superficial expression through pre-designed prompt templates. Those skilled in the art can further configure the prompt templates as needed based on this information. In practical implementation, a structured prompt is constructed, using standard consultation text, human response text, and response text to be evaluated as contextual input, with explicit instructions: "Please strictly compare the following two customer service responses in terms of effectiveness and substantive content differences in resolving user inquiries. Judgment criteria: When the solutions, key steps, applicable conditions, limiting rules, key information content, or final results provided by the two responses have irreconcilable contradictions or significant omissions, they are considered to have substantive differences; when only non-core differences exist such as expression methods, example details, and language style, they are considered not to have substantive differences. User inquiry: {standard consultation text}; Human response: {human response text}; Response to be evaluated: {response text to be evaluated}. Please output the judgment result, including the 'has_substantial_difference' (Boolean value) field." The large language model outputs this result directly indicating whether there is a substantive difference (true indicates a substantive difference or false indicates no substantive difference). In a recommended further embodiment, to enhance the accuracy of the judgment, a multi-round verification mechanism can be implemented. First, a general large language model is used for preliminary judgment, and then a professional large language model fine-tuned for the customer service domain is called for verification for boundary cases. In another recommended embodiment, business rule constraints can be added to the prompt template, such as: "Special Note: When judging issues involving user rights such as refunds, exchanges, and compensation, differences in elements such as time limits, amount limits, and necessary conditions must be considered substantial differences." This design makes the judgment of the large language model more in line with actual business needs. When the large language model determines that there is a substantial difference, the response text to be evaluated is marked as an incorrect answer, and the subsequent error attribution and repair process is triggered.

[0054] Based on the determination of incorrect responses, a targeted repair process is executed, which is implemented according to different error causes. Error causes are mainly divided into four categories: insufficient response generation capability, inadequate knowledge retrieval, missing knowledge base content, and outdated knowledge base content. In one embodiment, human customer service representatives can label the specific type of error attribution. When the error stems from insufficient response generation capability, it indicates that the existing customer service knowledge material is sufficient, but the large language model has failed to correctly reason and generate a response. Repair can be achieved by improving the precision of the large language model's prompting engineering, increasing the model's reasoning step size, introducing thought chain technology, switching to a higher-performance model, or fine-tuning the large language model using all standard consultation texts and their correct and incorrect human response texts. For example, the prompt template can be reconstructed to explicitly require the model to reason step-by-step, providing intermediate thought processes, or adding example demonstrations; another example is to customize dedicated prompt templates for specific business types of consultations to enhance the model's understanding and adherence to business rules. When errors stem from insufficient knowledge retrieval, it indicates that while the knowledge base contains relevant material, the retrieval mechanism has failed to retrieve all of it. This can be addressed by optimizing the coding model's ability to accurately understand and represent semantics, optimizing retrieval algorithm parameters, introducing multi-stage retrieval strategies, integrating multiple vector representation methods, or enhancing query comprehension. For example, adjusting the weight allocation of features in similarity calculation, adding a secondary retrieval channel based on precise keyword matching, automatically expanding queries to cover synonyms, or fine-tuning the coding model using all standard consultation texts and their correct and incorrect human responses. When errors arise from missing knowledge base content, it indicates a lack of necessary material to answer such questions. This can be addressed by constructing new knowledge units from standard consultation texts and corresponding human responses, verifying their quality, and then adding them to the knowledge base. In practice, the coverage and contextual relationships between new and existing knowledge should be checked first, and similar knowledge should be merged and optimized to avoid simple, repetitive additions. When an error stems from outdated knowledge base content, it indicates the presence of outdated knowledge that conflicts with current business rules. By utilizing a large language model and the complete context within the corresponding customer service knowledge document, the conflicting knowledge fragments are identified and replaced with updated, standardized content. A knowledge version tracking mechanism is also established. For example, time-sensitive tags are added to knowledge materials and linked to business rule versions, and potentially outdated content is scanned periodically to proactively maintain the knowledge base. After executing any repair process, the response text is regenerated and validated to ensure the effectiveness of the repair. For cases with multiple failed repairs, an escalation mechanism is initiated, transferring the case to professional knowledge engineers for in-depth analysis and manual repair. Simultaneously, error cases are accumulated for continuous optimization and correction.

[0055] As can be seen from the typical embodiments of this application, the technical solution of this application has many advantages, including but not limited to the following aspects: This application first generates a response text to be evaluated based on the existing customer service knowledge base, and uses the accuracy of the final answer in this text as the starting point and guide for evaluation. When an answer is determined to be incorrect, the approach is not to attribute the cause in a general way, but to trace back to the specific technical steps that caused the error, and to perform targeted repairs based on the precisely located cause in order to regenerate the correct answer. It is evident that by transforming the correction logic from "simple data appending" to "result-oriented effective attribution and targeted repair," the disorderly expansion and content contradictions of the knowledge base are effectively suppressed, fundamentally improving the accuracy and overall effectiveness of customer service optimization.

[0056] Secondly, the original conversation is abstracted and standardized using a large language model to generate standard consultation text that accurately describes the user's consultation intent. This establishes a unified and clear semantic benchmark for subsequent knowledge retrieval and response evaluation, significantly enhancing the understanding and generalization ability of diverse natural language expressions used by users. The retrieval and generation process based on standard consultation effectively reduces semantic noise and improves the relevance and accuracy of the initial response, thus laying a reliable foundation for efficient backward reasoning and attribution based on incorrect results.

[0057] Most importantly, by introducing a closed-loop process of "assessment-attribution-correction," the intelligent customer service system achieves dynamic self-optimization. When a generated response is judged to be incorrect, it doesn't simply append data; instead, it triggers corresponding correction strategies based on the identified specific cause of the error and iteratively generates the correct result. This process enables continuous accumulation of corrective experience, precisely strengthening weak points. While ensuring the refinement of the knowledge base, it achieves stable and sustainable improvement in response quality, providing an efficient technical path for the long-term evolution of intelligent customer service.

[0058] In a further embodiment, after step S1200, which uses a large language model to determine the standard consultation texts in the target session set that describe the corresponding user's consultation intent, the following steps are included: Step S2300: When there is no customer service knowledge material related to the standard consultation text in the preset customer service knowledge base, the corresponding human response text is determined from the target human customer service conversation from which the standard consultation text originates in the target conversation set using a large language model. When the pre-defined customer service knowledge base lacks customer service knowledge materials related to the standard inquiry text, valid answers need to be extracted from the original dialogue data as a source of supplementary knowledge. This judgment process is implemented through the retrieval stage of the retrieval enhancement generation algorithm. Specifically, the standard inquiry text is used as the query input, and its semantic similarity with all knowledge point units in the customer service knowledge base is calculated. If all similarity scores are lower than a preset threshold (e.g., 0.35), it is determined that no relevant customer service knowledge materials exist. This threshold can be adjusted by domain experts according to the actual business scenario. For example, it can be set to 0.45 in scenarios with high precision requirements and reduced to 0.25 in scenarios with high recall requirements. The retrieval calculation can use different semantic encoding models, including but not limited to dense retrieval models such as Sentence-BERT, DPR dual-tower model, or ColBERT, or it can be combined with sparse retrieval algorithms such as BM25 to form a hybrid retrieval strategy. In actual e-commerce customer service scenarios, for example, if the standard inquiry text is "How to handle cross-border goods detained by customs", and a search reveals that there are no relevant knowledge points in the knowledge base, the subsequent knowledge supplementation process is triggered.

[0059] From the target human customer service conversation from which the standard consultation text in the target conversation set originates, a large language model is used to determine the corresponding human response text. This process requires accurately identifying the valid answers given by the customer service personnel in the conversation. The complete target human customer service conversation record is input into the large language model, with the following structured instruction prompt: "Please extract the customer service personnel's final answer to the user's core question from the following customer service dialogue. The answer should only include substantive solutions, excluding non-substantive content such as duplicate confirmations. Dialogue content: {Complete conversation text}; User's core question: {Standard consultation text}". Based on the dialogue sequence and semantic coherence, the large language model identifies the start and end positions of the customer service response, summarizing the complete answer text as the human response text corresponding to the standard consultation text.

[0060] Step S2310: Construct new customer service knowledge material based on the standard consultation text and the manual response text, and add the customer service knowledge material to the customer service knowledge base.

[0061] First, standard consultation texts and human response texts are combined to form question-and-answer data pairs. Then, a professional language model, fine-tuned for the customer service domain, is invoked to perform content relevance analysis on each customer service knowledge document in the customer service knowledge base. The professional language model is input with the instruction: "Please analyze whether the following question-and-answer content is related to the topic of the provided knowledge document and whether the content is complementary, avoiding duplication: Question-and-answer content {question-and-answer data pair}, knowledge document {customer service knowledge document content}". Based on the overall theme of the document, the coverage of existing knowledge points, and a correct and in-depth understanding of the industry knowledge system, the professional language model determines whether the question-and-answer data pair should be integrated into the document. The professional language model can be flexibly adapted by those skilled in the art.

[0062] When the professional large language model determines that it should be added to an existing customer service knowledge document, it further generates specific knowledge point descriptions that conform to the contextual logic and expression style of the document. The instruction is: "Please rewrite the following Q&A content into knowledge points that are consistent with the document style, integrate them into the existing structure, and maintain the consistency of professional terminology: Q&A content {Q&A data pairs}, document context {relevant chapter content}".

[0063] When the professional large language model determines that there is no suitable existing document to integrate the knowledge point, a new customer service knowledge document is created in the customer service knowledge base to embed the question-and-answer data pair. For example, a knowledge point document is created, and the question-and-answer data pair is converted into the standard format of the knowledge point document, which includes historical standard consultation text (i.e., the standard consultation text of the question-and-answer data pair), 3-5 semantically equivalent similar consultation texts (which can be generated by the professional large language model that has been fine-tuned in the customer service domain based on the standard consultation text), and the corresponding standard response text (i.e., the human response text of the question-and-answer data pair).

[0064] In this embodiment, an automatic supplementation strategy is introduced when relevant content is missing from the customer service knowledge base, establishing the knowledge base's self-improvement capability. Specifically, when content related to standard consultation text cannot be retrieved from the existing customer service knowledge base, high-quality human response text is automatically extracted from the original human customer service conversation and paired with the standard consultation text to form new knowledge material. This not only effectively fills knowledge gaps but also ensures the professionalism and practicality of the newly added knowledge. More importantly, it avoids the high-cost model of excessive reliance on manual maintenance in traditional knowledge base construction, achieving automation and precision in knowledge accumulation. Simultaneously, it synergizes with the "results-oriented correction logic" to jointly maintain the refinement and consistency of the customer service knowledge base, preventing its disorderly expansion.

[0065] In a further embodiment, after determining that the answer judgment result in the answer evaluation text of the response text to be evaluated is an answer error, step S2410 includes the following steps: Step S3400: Embed the standard consultation text, the human response text, and all recalled customer service knowledge materials into the first prompt template to obtain the first prompt text, which guides the large language model to determine whether the human response text can be inferred based on all recalled customer service knowledge materials; The first prompt template is a pre-designed structured text framework used to guide large language models in assessing their knowledge reasoning capabilities. This template includes explicit instructions, contextual input, and output format requirements. The instructions precisely describe the task objective, for example: "You are a customer service knowledge expert. Please rigorously evaluate whether you can logically deduce a given human response based on the user's specific inquiry and the provided customer service knowledge materials. The evaluation must use the user's question as a boundary condition for reasoning, and must be based solely on information explicitly contained in the knowledge materials and reasonable business rules. External knowledge must not be introduced, and the specific context of the problem must not be ignored." The contextual input organizes data into fixed fields, including a "User Inquiry Question" field embedding standard inquiry text, a "Reference Knowledge Materials" field embedding all recalled customer service knowledge materials (preserving the original document structure and content), and a "Target Response Content" field embedding human response text. The output format requirements mandate a specific response structure that the model must adhere to. For example: "Please output a JSON-formatted result, including a 'reasonable_inference' field (Boolean value). 'reasonable_inference' is true only if the knowledge material contains sufficient information and a core solution to the target response can be derived through reasonable reasoning steps; otherwise, 'reasonable_inference' is false." It's understandable that constraining the model output to Boolean values ​​reduces output time, eliminates redundant characters, and improves efficiency.

[0066] Step S3410: When the judgment result output by the large language model indicates that the artificial response text can be inferred, the error in the answer evaluation text is determined to be due to insufficient performance of the response generation model.

[0067] When the Boolean value in the judgment result output by the large language model is true, it indicates that the recalled customer service knowledge material has sufficient information to support the generation of a correct response, and the core content of the human response text can be obtained through reasonable reasoning. In this case, the substantial difference between the response text to be evaluated and the human response text cannot be attributed to missing knowledge material or retrieval failure, but only to defects in the response generation process. Specifically, the large language model that generates the response text to be evaluated fails to fully utilize the recalled knowledge material for accurate reasoning, manifesting as insufficient generation capabilities such as misunderstanding user intent, failing to correctly understand the recalled content, and failing to generate a correct response to solve the problem. This judgment conclusion has a clear logic: if the knowledge material is sufficient to deduce a correct response, but the automatically generated response still contains substantial errors, then the root cause of the error must lie in the reasoning ability of the generation model or defects in the design of the prompt instructions.

[0068] To further address the cause of this error, a repair process is implemented, including any of the following: optimizing the prompt engineering, fine-tuning the model, introducing a thought chain, integrating a multi-model voting mechanism, and adding business rule constraints. Those skilled in the art can flexibly adapt these methods, or they can be implemented according to the subsequent embodiments. In one embodiment, the prompt engineering design is optimized, and the prompt template used when generating the response text to be evaluated is reconstructed. Specifically, the original simple prompt "Answer the user's question based on the following knowledge" is replaced with a structured multi-stage prompt: "Please strictly follow these steps to generate the response: 1. Problem analysis: Clarify the core needs and constraints of the user's inquiry; 2. Knowledge matching: Extract rule clauses directly related to the problem from the recalled materials; 3. Logical deduction: Reason by combining business rules and the problem context; 4. Response construction: Organize the response according to the three-stage structure of 'confirming the problem - providing a solution - explaining the limitations. Special note: Do not ignore the time limits, amount limits, and exception clauses clearly stated in the knowledge materials." This prompt template explicitly specifies the reasoning path, forcing the model to follow business logic and avoiding the original model outputting incorrect conclusions.

[0069] In another embodiment, the large language model for generating responses undergoes domain-adaptive fine-tuning. A refined training dataset is constructed by collecting all standard consultation texts, recalled knowledge materials, and corresponding human responses from historical amendment examples that were deemed "inadequate in response generation model performance." Each training sample consists of three parts: the input part contains a concatenation of the standard consultation text and recalled knowledge materials; the middle part includes thought chain annotations (e.g., "First, this issue involves refunds for shipped orders; second, according to rule X, a logistics interception attempt should be made; finally, if the interception fails, the return process should begin"); and the output part is the correct human response text. The generation model is then supervised and fine-tuned using this dataset, with the training objective being to minimize the differences between generated responses and human responses in three dimensions: compliance with business rules, coverage of key information, and completeness of logical structure. A rule consistency loss function is introduced during the fine-tuning process, imposing additional penalties when the model-generated content violates the business rules explicitly stated in the knowledge materials. For example, if the knowledge materials stipulate that "cross-border goods returns are subject to customs duties," but the model-generated response does not mention this requirement, the loss function value increases significantly, forcing the model to learn the precise application of business rules. After the fine-tuning process is completed, the original generated model is replaced, significantly improving its inference accuracy in specific business scenarios.

[0070] In another embodiment, a layered generation architecture is introduced to replace single-model generation. A two-stage generation mechanism is deployed: In the first stage, a lightweight routing model determines the type of inquiry and classifies the problem into a preset business rule template library (such as refund, logistics, and after-sales). In the second stage, the corresponding dedicated generation model is activated based on the classification result. Each dedicated model only handles problems in a specific business domain and loads the refined prompt template and rule constraint set for that business domain. For example, when an inquiry is determined to be of the "logistics anomaly refund" type, a dedicated refund model is activated. This model has a built-in "logistics status - refund conditions" mapping table, which mandates that the generated response must include three core elements: logistics query guidance, interception operation time limit, and refund trigger conditions. The prompt template of the dedicated model explicitly includes business rule checkpoints: "Before generating a response, the following must be verified: 1. Whether the interception application time limit is clearly stated; 2. Whether different processing procedures for successful / failed interception are distinguished; 3. Whether the party responsible for freight is mentioned." This architecture decomposes complex general generation tasks into specialized sub-tasks, effectively avoiding the reasoning confusion problem of a single model in scenarios with multiple business rules. After knowledge retrieval, the routing model is run first to determine the business type, and then the corresponding dedicated model is called to generate a response, which significantly improves the consistency between the generated content and the manual response.

[0071] This embodiment refines the error cause analysis mechanism, focusing particularly on identifying the specific cause of insufficient performance in the response generation model. Through a designed prompt template, the large language model is guided to determine whether the existing knowledge material is sufficient to deduce the correct human response. When the judgment result is "can be deduced," the root cause of the problem is precisely located in the response generation stage rather than the knowledge content itself. This attribution mechanism avoids the blind "one-size-fits-all" approach of traditional optimization processes, enabling targeted optimization of the generation model rather than unnecessary expansion of the knowledge base. This precise attribution not only improves optimization efficiency but also provides high-quality, targeted samples for model training, accelerating the iterative improvement of response generation capabilities while maintaining a streamlined knowledge base structure, preventing content redundancy and potential conflicts.

[0072] In a further embodiment, after determining that the answer judgment result in the answer evaluation text of the response text to be evaluated is an answer error, step S2410 includes the following steps: Step S4400: Based on the large language model, determine all customer service knowledge materials in the customer service knowledge base that match the standard consultation text and the manual response text; First, a full traversal of the customer service knowledge base is performed, and each knowledge point unit in the knowledge base is evaluated as a candidate material. The knowledge point unit has different forms according to the type of the document from which it originates: for knowledge point type documents, the knowledge point unit is an independent consultation-response pair; for chapter structure type documents, the knowledge point unit is the title and its subordinate body text; for unstructured documents, the knowledge point unit is a collection of paragraphs formed by semantic coherence segmentation. The large language model receives three input components: standard consultation text, human response text, and the complete content of the knowledge point unit currently being evaluated. It guides the model to make precise matching judgments through structured prompts. The prompt template is designed as follows: "Please rigorously evaluate whether the following customer service knowledge materials contain the core information needed to answer the user's consultation: the user's consultation question is {standard consultation text}, the correct answer is {human response text}, and the knowledge material to be evaluated is {knowledge point unit content}. Judgment criteria: when the knowledge material explicitly contains key rules, operational steps, conditional constraints, or direct answers that support the generation of a correct answer, it is considered a match; when the knowledge material is only partially relevant but lacks the core elements necessary for the answer, or the information it contains substantially conflicts with the correct answer, it is considered a mismatch. Please output only the JSON format result, including the 'is_match' field (Boolean value)." Based on this prompt, the large language model outputs the matching results and collects all knowledge point units with an 'is_match' value of true as matched customer service knowledge materials.

[0073] In practice, two specific matching strategies can be adopted. The first is a direct full-database matching strategy, which loads each customer service knowledge document in the customer service knowledge base sequentially, performs the large language model evaluation process described above independently on each knowledge point unit within the document, and retains all units with a matching result of true.

[0074] The second approach is a hierarchical matching strategy. First, a lightweight semantic encoding model is used to establish a preliminary index for the knowledge base. The initial similarity between the standard consultation text and all knowledge point units is calculated, and a set of candidate units with a similarity higher than 0.25 is selected. Then, the aforementioned large language model is applied to this candidate set for fine-tuning. When the knowledge base is large, the hierarchical strategy can significantly reduce the number of calls to the large language model. For example, a knowledge point base of millions can be reduced to hundreds of candidates through similarity filtering before fine-tuning. During the matching process, the large language model identifies two types of matching relationships: explicit matching, where the knowledge point unit directly contains the key content of the human response text; and implicit matching, where the knowledge point unit contains business rules, and the core conclusion of the human response text can be derived through reasonable reasoning. For example, if the human response mentions "returns must be applied for within 7 days of receipt," and the knowledge material contains "the platform's after-sales service period is 7 calendar days after the goods are signed for," although the wording is different, the logic is equivalent, and it should be considered a match. Finally, all matched knowledge point units are collected and used as complete matching customer service knowledge materials for subsequent comparative analysis with the original recall results.

[0075] Step S4410: Embed the standard consultation text, the human response text and all the customer service knowledge materials they match into the first prompt template to obtain the first prompt text, which guides the large language model to determine whether the human response text can be inferred based on all the matched customer service knowledge materials. The first prompt template is a pre-designed structured text framework used to guide large language models in assessing their knowledge reasoning capabilities. This template includes explicit instructions, contextual input, and output format requirements. The instructions precisely describe the task objective, for example: "You are a customer service knowledge expert. Please rigorously evaluate whether you can logically deduce a given human response based on the user's specific inquiry and the provided customer service knowledge materials. The evaluation must use the user's question as a boundary condition for reasoning, and must be based solely on information explicitly contained in the knowledge materials and reasonable business rules. External knowledge must not be introduced, and the specific context of the problem must not be ignored." The contextual input organizes data into fixed fields, including a "User Inquiry Question" field embedding standard inquiry text, a "Reference Knowledge Materials" field embedding all recalled customer service knowledge materials (preserving the original document structure and content), and a "Target Response Content" field embedding human response text. The output format requirements mandate a specific response structure that the model must adhere to. For example: "Please output a JSON-formatted result, including a 'reasonable_inference' field (Boolean value). 'reasonable_inference' is true only if the knowledge material contains sufficient information and a core solution to the target response can be derived through reasonable reasoning steps; otherwise, 'reasonable_inference' is false." It's understandable that constraining the model output to Boolean values ​​reduces output time, eliminates redundant characters, and improves efficiency.

[0076] Step S4420: When the judgment result output by the large language model indicates that the human response text can be inferred, determine whether all customer service knowledge materials matched by the human response text have been used to generate the response text to be evaluated and thus all of them are recalled. This indicates that the customer service knowledge base contains all the knowledge necessary to correctly answer user inquiries. Based on this, it further determines whether all customer service knowledge materials required for reasoning have been retrieved. If all have been retrieved, it means the defect in the final generated result is not due to a flaw in the retrieval and retrieval stage, but rather a flaw in another stage, such as the generation stage. This judgment process is implemented using a set comparison algorithm. All customer service knowledge materials identified in step S4400 that match the standard inquiry text and the human response text form a reference set A, and the customer service knowledge materials actually retrieved in step S1300 form a set B. The set comparison is performed using precise ID matching. In precise ID matching, each knowledge point unit has a unique identifier in the knowledge base. The algorithm directly compares whether all identifiers in set A appear in set B. This judgment process must be rigorous; the absence of any key knowledge point will result in the generated response lacking core elements. Therefore, an "all or nothing" judgment logic is adopted: if any unit in set A does not find a match in set B, it is determined that not all have been retrieved; otherwise, it is determined that all have been retrieved.

[0077] Step S4430: When not all customer service knowledge materials required for reasoning have been recalled, determine that the error in the answer evaluation text is due to insufficient performance of the retrieval and recall algorithm.

[0078] At this point, it was determined that the error in the answer evaluation text was due to the insufficient performance of the retrieval and recall algorithm. This conclusion was based on clear logic: the customer service knowledge base does indeed contain all the knowledge materials needed to answer the question, but the retrieval and recall algorithm failed to effectively identify and recall all relevant materials, resulting in a lack of complete knowledge support in the subsequent response generation stage, thus producing an erroneous response that is substantially different from the human response.

[0079] To address the performance deficiencies of retrieval and recall algorithms, a remediation process is implemented, specifically including optimizing the semantic encoding model, adjusting retrieval algorithm parameters, and enhancing query understanding capabilities. In the embodiment for optimizing the semantic encoding model, a training set is constructed by collecting historical failure cases. Each sample contains standard consultation text, customer service knowledge materials that should have been recalled but were not, and human response text. A contrastive learning objective function is used to fine-tune the encoding model, enabling it to map semantically relevant standard consultation text and customer service knowledge materials to closer positions in the feature space. The fine-tuning process employs a triplet loss function. For each standard consultation text, one positive sample (relevant customer service knowledge material) and one negative sample (irrelevant customer service knowledge material) are selected, and model parameters are adjusted to reduce the distance between positive samples and increase the distance between negative samples.

[0080] In the embodiment of adjusting the retrieval algorithm parameters, the feature weights in the similarity calculation are dynamically optimized, and differentiated weights are assigned to different types of customer service knowledge documents. For example, for knowledge point type documents, the weight of historical standard consultation text is increased to 0.7, and the weight of similar consultation text is decreased to 0.3; for chapter structure type documents, the title similarity weight is set to 0.6, and the body text similarity weight is set to 0.4; for unstructured documents, a semantic vector calculation method of word frequency-inverse document frequency weighting is adopted. Those skilled in the art can further flexibly adapt and implement this method based on the disclosure herein.

[0081] In an embodiment that enhances query understanding, the input standard consultation text is automatically expanded, and a large language model is invoked to generate 3-5 semantically equivalent expanded queries. For example, "how to return goods" is expanded to "the process of applying for a return", "the steps to return goods", and "how to operate a return". Recall is performed independently for each expanded query, and the results are merged and deduplicated based on relevance.

[0082] This embodiment proposes a precise identification scheme to address performance issues in the retrieval and recall process. By systematically verifying the existence of relevant materials in the customer service knowledge base that can derive the correct response, and further analyzing whether these materials are effectively recalled, the scheme accurately determines whether errors stem from limitations in the retrieval algorithm. When it is confirmed that the knowledge materials are sufficient to support the correct response but are not all recalled, the cause can be clearly attributed to a defect in the retrieval mechanism, thereby triggering targeted algorithm optimization. This achieves a refined evaluation of the knowledge retrieval process, avoiding the misjudgment of incomplete retrieval as knowledge loss leading to duplicate knowledge additions, and effectively maintaining the structural conciseness of the knowledge base. Simultaneously, it provides a clear direction and quantitative basis for the continuous optimization of the retrieval and recall algorithm, promoting self-improvement in knowledge utilization efficiency and significantly increasing the value conversion rate of existing knowledge assets.

[0083] In a further embodiment, after determining that the answer judgment result in the answer evaluation text of the response text to be evaluated is an answer error, step S2410 includes the following steps: Step S5400: Based on the large language model, determine all customer service knowledge materials in the customer service knowledge base that match the standard consultation text and the manual response text; Referring to the relevant disclosure of step S4410, this step will not be described in detail.

[0084] Step S5410: Embed the standard consultation text, the human response text and all the customer service knowledge materials they match into the first prompt template to obtain the first prompt text, which guides the large language model to determine whether the human response text can be inferred based on all the matched customer service knowledge materials. Referring to the relevant disclosure of step S4410, this step will not be described in detail.

[0085] Step S5420: When the judgment result output by the large language model indicates that the manual response text cannot be inferred, the error in the answer evaluation text is determined to be due to the lack of necessary customer service knowledge materials in the customer service knowledge base.

[0086] When the large language model outputs a false result after evaluating all matched customer service knowledge materials, it indicates that although the existing knowledge base contains knowledge points related to the inquiry, the combination of these knowledge points still cannot derive the correct answer provided by human customer service. This judgment clearly indicates that the customer service knowledge base lacks the core knowledge content necessary to resolve this type of user inquiry, rather than a technical defect in the retrieval or generation process. This conclusion is based on clear logic: the determination that the customer service knowledge base lacks necessary customer service knowledge materials relies on strict logical deduction boundary conditions. During the evaluation process, the large language model must strictly adhere to the principle of "using only explicit information in the knowledge materials and publicly available business rules of the e-commerce platform for reasoning." When the knowledge materials lack essential customer service knowledge such as key conditions, exception explanations, or new business rules, even if the model has strong generalization capabilities, it cannot derive an answer consistent with that of human customer service without introducing external knowledge.

[0087] To address the deficiency in the customer service knowledge base due to a lack of necessary customer service knowledge materials, a knowledge supplementation process is implemented. In one embodiment, new knowledge units are automatically constructed by combining standard consultation texts with corresponding human responses to form basic question-and-answer pairs. Then, a professional large-scale language model is invoked to expand and generate 3-5 semantically equivalent similar consultation statements. The new knowledge units are tagged with metadata, including business category, timeliness identifier, and association rule references. Then, an automated verification process is implemented: First, the semantic overlap with existing knowledge is checked by using Sentence-BERT to calculate the cosine similarity with similar knowledge points. If the similarity is below 0.6, it is confirmed as new knowledge. Second, the consistency of business rules is verified by performing conflict detection between the new knowledge and relevant business rule documents to ensure there are no logical contradictions.

[0088] In another embodiment, a collaborative verification mechanism for knowledge supplementation is implemented. When a knowledge gap is detected, a knowledge supplementation task is automatically generated and assigned to a professional customer service representative, providing standard consultation text, manual response text, and relevant business rule context as reference materials. The customer service representative completes the knowledge content in a dedicated editing interface. The edited content is compared to previous versions, change details are recorded, and a cross-departmental review process is triggered to ensure the accuracy and compliance of the knowledge content. After approval, an atomic update operation is performed, using a transaction mechanism to ensure the consistency of the knowledge base. An update log is generated, recording the time of knowledge supplementation, content changes, and responsible person information, providing a basis for subsequent knowledge quality traceability. After the update is completed, erroneous responses caused by the same consultation question within the last 24 hours are automatically reprocessed, generating corrected response text to achieve closed-loop repair.

[0089] This embodiment further refines the error attribution system, providing a specific identification mechanism to address the fundamental problem of missing knowledge base content. By verifying whether existing materials in the customer service knowledge base can logically deduce the correct response, it accurately identifies the existence of knowledge gaps when a response is determined to be "unable to deduce." This identification mechanism goes beyond simple keyword matching, delving into the logical completeness of knowledge, and can discover superficially relevant but substantively insufficient knowledge deficiencies. This not only avoids ineffective optimization caused by misjudging knowledge gaps as model or retrieval problems, but also ensures that the customer service knowledge base is expanded only when necessary. This on-demand supplementation strategy effectively controls the size of the knowledge base while significantly improving the completeness and professionalism of the knowledge content, achieving a dynamic balance between knowledge quality and quantity.

[0090] In a further embodiment, after determining that the answer judgment result in the answer evaluation text of the response text to be evaluated is an answer error, step S2410 includes the following steps: Step S6400: Embed the standard consultation text, the manual response text, and all recalled customer service knowledge materials into the second prompt template to obtain the second prompt text. Use this prompt text to guide the large language model to determine whether there is a conflict when the manual response text and all recalled customer service knowledge materials are used to answer the standard consultation text. The second prompt template is a predefined structured text framework specifically designed to guide large language models in detecting knowledge conflicts. This template includes a clear instruction section, a contextual input section, and a standardized output format. The instruction section precisely describes the task objective, stating: "You are a customer service knowledge review expert. Please rigorously evaluate whether the currently recalled customer service knowledge materials and the human response text contain irreconcilable contradictions in key business rules, operational processes, or constraints when answering user inquiries. The evaluation must be based on the timeliness and consistency principles of business rules. When the knowledge materials and human responses express contradictory conclusions or mutually exclusive conditions in the core solution, a conflict is deemed to exist; when differences only exist in details of expression, example methods, or non-critical information, no conflict is deemed to exist." The contextual input section organizes data according to fixed fields: the "User Inquiry Question" field embeds the standard inquiry text, the "Reference Knowledge Materials" field embeds all recalled customer service knowledge materials, and the "Human Response Content" field embeds the corresponding human response text. The output format requirements section mandates: "Please output a JSON format result, including the 'has_conflict' field (Boolean value). 'has_conflict' is set to true only if the content explicitly stated in the knowledge material directly contradicts the human response in terms of business rule execution results, key conditions, or solutions; otherwise, 'has_conflict' is set to false." This prompt template design ensures that the large language model focuses on conflict detection at the business rule level, rather than on superficial differences in expression.

[0091] Step S6410: When there is a conflict in the judgment result representation output by the large language model, determine that the error in the answer evaluation text is due to the customer service knowledge base not being updated.

[0092] When the `has_conflict` field in the large language model's output is true, the error is determined to be due to an outdated customer service knowledge base. This judgment has a clear technical logic: the existence of a conflict proves that the knowledge base stores historical versions of business rules, while human customer service relies on currently valid rules, leading to a contradiction due to version differences. In e-commerce operations, business rules are updated frequently, including adjustments to promotional strategies, upgrades to service commitments, and changes to compensation standards. If the knowledge base update process fails to synchronize with these business changes, inconsistencies arise between the knowledge content and the actual rules being implemented. For example, if the platform extends the return period from "7 days" to "15 days" on a certain date, but the knowledge base fails to update the relevant content in time, the automatically generated response will still follow the old rules, while human customer service will have implemented the new standards, resulting in a rule conflict. This error identification eliminates the possibility of missing knowledge or retrieval failure, because the knowledge material does exist and has been successfully retrieved, but the content is outdated.

[0093] Based on this assessment, a knowledge base update process is triggered, which includes: extracting specific rule items from the conflict details, locating the corresponding outdated content in the knowledge base, replacing the outdated content with the latest rules from the manual responses while preserving the original document context, and simultaneously updating the version marker and effective date in the document metadata. This process ensures that the knowledge base remains synchronized with the actual business rules, eliminating response errors caused by outdated knowledge at the source.

[0094] This embodiment addresses the issue of knowledge base content timeliness by designing a knowledge conflict detection mechanism to identify response errors caused by outdated or unupdated knowledge. It utilizes a large language model to analyze whether there are logical conflicts between human responses and existing knowledge materials. When a conflict is detected, it is accurately attributed to the lack of timely knowledge updates. This mechanism effectively addresses the challenges of knowledge maintenance in dynamically changing scenarios such as business rules and product information, avoiding erroneous responses due to outdated knowledge. Compared to traditional periodic comprehensive review maintenance methods, it achieves precise update triggering based on actual consultation scenarios, significantly reducing knowledge maintenance costs while improving the timeliness and relevance of updates. By continuously eliminating knowledge conflicts, it effectively maintains the consistency and authority of the knowledge base, providing customers with accurate and reliable consultation services, and significantly improving customer experience and brand trust.

[0095] Please see Figure 3 This application provides a customer service response correction device, which is a functional embodiment of the customer service response correction method of this application. On another note, this customer service response correction device, also provided to meet one of the purposes of this application, includes a session preprocessing module 1100, a consultation determination module 1200, a response generation module 1300, and an error repair module 1400. The session preprocessing module 1100 is used to preprocess each human customer service session in a set of human customer service sessions to obtain a target session set. The consultation determination module 1200 is used to determine, using a large language model, each standard consultation text describing the corresponding user's consultation intent in the target session set. The response generation module 1300 is used to recall all customer service knowledge materials related to the standard consultation text in a preset customer service knowledge base, and use this to generate a corresponding response text to be evaluated. The error repair module 1400 is used to execute a corresponding repair process based on the error cause in the response text to be evaluated when the answer evaluation text of the response text to be evaluated indicates an error, so as to regenerate the corresponding correct response text after repair.

[0096] In a further embodiment, after the consultation determination module 1200, there are: a first response determination submodule, used to determine the corresponding human response text from the target human customer service session from which the standard consultation text originates in the target session set using a large language model when there is no customer service knowledge material related to the standard consultation text in the preset customer service knowledge base; and a material addition submodule, used to construct new customer service knowledge material based on the standard consultation text and the human response text, and add the customer service knowledge material to the customer service knowledge base.

[0097] In a further embodiment, before the error repair module 1400, there are: a second response determination submodule, used to determine the corresponding human response text from the target human customer service session from which the standard consultation text in the target session set originates, using a large language model; and an error judgment submodule, used to determine that the answer judgment result in the answer evaluation text of the answer to be evaluated is an answer error when there is a substantial difference between the corresponding answer content when the human response text and the answer text to be evaluated are used to answer the standard consultation text.

[0098] In a further embodiment, after the error determination submodule, there is a following: a response recall submodule, used to embed the standard consultation text, the human response text, and all recalled customer service knowledge materials into a first prompt template to obtain a first prompt text, which guides the large language model to determine whether the human response text can be inferred based on all recalled customer service knowledge materials; and a first factorization module, used to determine that the error in the answer evaluation text is due to insufficient performance of the response generation model when the judgment result output by the large language model indicates that the human response text can be inferred.

[0099] In a further embodiment, after the error determination submodule, the module includes: a first material matching submodule, used to determine all customer service knowledge materials in the customer service knowledge base that match the standard consultation text and the human response text based on a large language model; a first inventory evaluation submodule, used to embed the standard consultation text, the human response text, and all matching customer service knowledge materials into a first prompt template to obtain a first prompt text, which guides the large language model to determine whether the human response text can be inferred based on all the matching customer service knowledge materials; a recall evaluation submodule, used to determine whether all customer service knowledge materials matching the human response text have been used to generate the response text to be evaluated and have been recalled when the judgment result output by the large language model indicates that the human response text can be inferred; and a second factorization module, used to determine that the error in the answer evaluation text is due to insufficient performance of the retrieval and recall algorithm when not all the customer service knowledge materials required for inference have been recalled.

[0100] In a further embodiment, after the error determination submodule, the system includes: a second material matching submodule, used to determine all customer service knowledge materials in the customer service knowledge base that match the standard consultation text and the human response text based on a large language model; a second inventory evaluation submodule, used to embed the standard consultation text, the human response text, and all their matching customer service knowledge materials into a first prompt template to obtain a first prompt text, which guides the large language model to determine whether the human response text can be inferred based on all the matching customer service knowledge materials; and a third factorization module, used to determine that the error in the answer evaluation text is due to the lack of necessary customer service knowledge materials in the customer service knowledge base when the judgment result output by the large language model indicates that the human response text cannot be inferred.

[0101] In a further embodiment, after the error determination submodule, there is a conflict assessment submodule, used to embed the standard consultation text, the manual response text, and all recalled customer service knowledge materials into a second prompt template to obtain a second prompt text, which guides the large language model to determine whether there is a conflict when the manual response text and all recalled customer service knowledge materials are used to answer the standard consultation text; and a fourth factorization module, used to determine that the error in the answer assessment text is due to the customer service knowledge base not being updated when the judgment result output by the large language model indicates a conflict.

[0102] To address the aforementioned technical problems, embodiments of this application also provide computer equipment. For example... Figure 4 The diagram shows the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable storage medium stores an operating system, a database, and computer-readable instructions. The database may store a sequence of control information. When the computer-readable instructions are executed by the processor, the processor can implement a customer service response correction method. The processor of the computer device provides computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the customer service response correction method of this application. The network interface of the computer device is used for communication with a terminal. Those skilled in the art will understand that… Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0103] In this embodiment, the processor is used to execute... Figure 3The system contains the specific functions of each module and its sub-modules. The memory stores the program code and various data required to execute these modules or sub-modules. The network interface is used for data transmission between the user terminal and the server. In this embodiment, the memory stores the program code and data required to execute all modules / sub-modules in the customer service response correction device of this application. The server can call the server's program code and data to execute the functions of all sub-modules.

[0104] This application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the customer service response correction method of any embodiment of this application.

[0105] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0106] In summary, this application can accurately fix response defects to provide high-quality intelligent customer service.

[0107] Those skilled in the art will understand that the steps, measures, and solutions in the various operations, methods, and processes discussed in this application can be alternated, modified, combined, or deleted. Furthermore, other steps, measures, and solutions in the various operations, methods, and processes discussed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and solutions in the prior art that are similar to those in the open-source operations, methods, and processes of this application can also be alternated, modified, rearranged, decomposed, combined, or deleted.

[0108] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A customer service response correction method, characterized in that, Includes the following steps: Preprocess each human customer service session in the human customer service session set to obtain the target session set; A large language model is used to determine the standard consultation texts in the target conversation set that describe the corresponding user's consultation intent; When customer service knowledge materials related to the standard consultation text exist in the preset customer service knowledge base, all customer service knowledge materials are recalled to generate the corresponding response text to be evaluated. When the solution evaluation text of the response text to be evaluated indicates an error, the corresponding repair process is executed according to the reason for the error in the solution evaluation text, so as to regenerate the corresponding correct response text after the repair.

2. The customer service response correction method according to claim 1, characterized in that, After determining the standard consultation texts describing the corresponding user consultation intent in the target conversation set using a large language model, the following steps are included: When there is no customer service knowledge material related to the standard consultation text in the preset customer service knowledge base, the corresponding human response text is determined from the target human customer service conversation from which the standard consultation text originates in the target conversation set using a large language model. New customer service knowledge materials are constructed based on the standard consultation text and the manual response text, and these customer service knowledge materials are added to the customer service knowledge base.

3. The customer service response correction method according to claim 1, characterized in that, When the solution evaluation text of the response text to be evaluated indicates an error, before executing the corresponding repair process based on the error cause in the solution evaluation text, the following steps are included: From the target human customer service conversations from which the standard consultation texts described in the target conversation set originate, the corresponding human response texts are determined using a large language model; When the content of the manual response text and the response text to be evaluated differs substantially when used to answer the standard consultation text, the answer judgment result in the evaluation text of the response text to be evaluated is determined to be an incorrect answer.

4. The customer service response correction method according to claim 3, characterized in that, After determining that the solution evaluation text of the response text to be evaluated is incorrect, the following steps are included: The standard consultation text, the human response text, and all recalled customer service knowledge materials are embedded into the first prompt template to obtain the first prompt text, which guides the large language model to determine whether the human response text can be inferred based on all recalled customer service knowledge materials. When the judgment result output by the large language model indicates that the artificial response text can be inferred, the error in the answer evaluation text is determined to be due to insufficient performance of the response generation model.

5. The customer service response correction method according to claim 3, characterized in that, After determining that the solution evaluation text of the response text to be evaluated is incorrect, the following steps are included: Based on a large language model, all customer service knowledge materials in the customer service knowledge base that match the standard consultation text and human response text are identified. The standard consultation text, the human response text, and all the matching customer service knowledge materials are embedded into the first prompt template to obtain the first prompt text, which guides the large language model to determine whether the human response text can be inferred based on all the matching customer service knowledge materials. When the judgment result output by the large language model indicates that the human response text can be inferred, it is determined whether all customer service knowledge materials matched by the human response text have been used to generate the response text to be evaluated and all of them are recalled. When not all customer service knowledge materials required for reasoning are retrieved, the error in the solution evaluation text is determined to be due to insufficient performance of the retrieval and recall algorithm.

6. The customer service response correction method according to claim 3, characterized in that, After determining that the solution evaluation text of the response text to be evaluated is incorrect, the following steps are included: Based on a large language model, all customer service knowledge materials in the customer service knowledge base that match the standard consultation text and human response text are identified. The standard consultation text, the human response text, and all the matching customer service knowledge materials are embedded into the first prompt template to obtain the first prompt text, which guides the large language model to determine whether the human response text can be inferred based on all the matching customer service knowledge materials. When the judgment result output by the large language model indicates that the manual response text cannot be inferred, the error in the answer evaluation text is determined to be due to the lack of necessary customer service knowledge materials in the customer service knowledge base.

7. The customer service response correction method according to claim 1, characterized in that, After determining that the solution evaluation text of the response text to be evaluated is incorrect, the following steps are included: The standard consultation text, the manual response text, and all recalled customer service knowledge materials are embedded into the second prompt template to obtain the second prompt text. This prompt text is then used to guide the large language model to determine whether there is a conflict when the manual response text and all recalled customer service knowledge materials are used to answer the standard consultation text. When there is a conflict in the judgment result representation output by the large language model, the error in the answer evaluation text is determined to be due to the customer service knowledge base not being updated.

8. A customer service response correction device, characterized in that, include: The session preprocessing module is used to preprocess each human customer service session in the human customer service session set to obtain the target session set; The consultation determination module is used to determine, using a large language model, the standard consultation texts describing the corresponding user's consultation intent in the target conversation set; The response generation module is used to recall all customer service knowledge materials when there are customer service knowledge materials related to the standard consultation text in the preset customer service knowledge base, and to generate the corresponding response text to be evaluated. The error repair module is used to perform a corresponding repair process based on the error cause in the solution evaluation text when the solution evaluation text of the response text to be evaluated indicates an error, so as to regenerate the corresponding correct response text after repair.

9. A computer device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 7, which, when invoked by a computer, executes the steps included in the corresponding method.

Images