Customer service knowledge self-evolution method and system based on artificial feedback and candidate verification

Abstract

The application discloses a customer service knowledge self-evolution method and system based on artificial feedback and candidate verification. The method comprises the following steps: collecting artificial conversation extraction candidates, searching an official knowledge base after value judgment, and solidifying operation types into new addition, supplement, replacement or non-operation according to the relationship between intention and answer; writing effective candidates into a virtual knowledge base and saving types, target identification, written answers, audits and evidence states; jointly searching the two databases in the service stage, and exposing and using the candidates according to types, audit states, evidence states and official hit results; generating evidence events according to user feedback, manual conversion, and the relationship between artificial and candidate answers after the conversation ends; updating the official knowledge base according to the solidified types when the promotion gate is met, and recording the promotion history to support auditing and rollback. Through the method, the official knowledge base is prevented from being directly polluted by wrong knowledge, and the coverage ability of intelligent customer service for long-tail problems, the knowledge reuse efficiency and the auditable nature of knowledge base management are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to a self-evolving method and system for customer service knowledge based on human feedback and candidate verification. Background Technology

[0002] With the widespread application of large language models and retrieval-enhanced generation technologies in customer service systems, intelligent customer service systems typically rely on question-answer pairs, document fragments, rule entries, or business knowledge graphs from formal knowledge bases to generate user-facing answers. When the formal knowledge base is sufficiently comprehensive and the entries are accurate, intelligent customer service can effectively reduce the workload of human customer service representatives. However, when users raise long-tail questions, questions about new rules, questions about changes in entry points, questions about fee conditions, or questions about changes to old rules, intelligent customer service often fails to retrieve effective answers or retrieves incomplete or outdated formal knowledge, leading to a decline in the quality of the answers and ultimately requiring transfer to human customer service for processing.

[0003] In traditional customer service processes, questions that AI-powered customer service cannot answer are typically transferred to human agents. Human agents, in actual conversations, can provide accurate and reusable answers, such as providing details on how to access certain benefits, explaining the conditions for refunding certain points, explaining why a button is invisible, or clarifying the applicable conditions for certain permission rules. These human answers have the potential to become formal knowledge, but in most existing systems, they remain only in conversation logs, work order notes, or quality control materials. Subsequent similar user inquiries require repeated transfers to human agents, resulting in low efficiency in knowledge reuse. While existing improvement solutions attempt to extract candidate questions and answers from customer service conversations using dialogue summarization models and directly write them into the knowledge base or submit them for human review, they easily overlook the authenticity, stability, and reusability of candidate knowledge in formal service scenarios. Human answers may be case-specific responses to individual user states, may contain temporary policies, may conflict with existing answers in the formal knowledge base, and may lack the review and auditing information required for formal publication. If directly written into the formal knowledge base, incorrect answers will be repeatedly used by AI-powered customer service, causing knowledge pollution.

[0004] Furthermore, existing systems are prone to confusing the criteria for determining whether "candidate knowledge is retrieved" with "candidate knowledge solves the problem." During intelligent customer service, while a candidate knowledge might be retrieved and provided to the answer generator, whether the user is ultimately satisfied, whether they continue to ask follow-up questions, whether they are transferred to a human agent, and whether the human agent confirms the answer is correct, can usually only be determined after the session ends. If the system immediately records a candidate as a valid hit during the retrieval phase, weakly related, low-quality, or conflicting candidates will be mistakenly promoted to formal knowledge. Simultaneously, existing solutions often struggle to distinguish between addition, supplementation, and replacement operations when similar knowledge already exists in the formal knowledge base. Merging based solely on similarity can easily lead to the erroneous merging of questions and answers within the same domain but with different intentions. Adding based solely on the existence of similar entries can result in a large number of duplicate or similar entries in the formal knowledge base. For replacement knowledge, the risk is even higher due to changes in the facts, conditions, values, or support levels between the candidate answer and the original formal answer. Existing customer service knowledge base update processes often lack default blocking and review gates for replacement candidates. Existing customer service knowledge update systems typically lack rollbackable promotion histories. Once knowledge is written into the formal knowledge base, it is difficult to trace the source sessions, candidate evidence, promotion decisions, and differences before and after writing to the database when anomalies are subsequently discovered. The lack of snapshots and rollback basis makes it difficult for enterprise knowledge base governance to meet audit requirements and also reduces the enterprise's trust in automatic knowledge updates.

[0005] Therefore, it is necessary to design a new method to achieve structured extraction of candidate knowledge from human service answers, solidification of operation types, controlled trial verification, promotion gating decision-making, and anomaly rollback. This will not only prevent erroneous knowledge from directly polluting the formal knowledge base, but also improve the intelligent customer service's ability to cover long-tail questions, the efficiency of knowledge reuse, and the auditability of knowledge base governance. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a self-evolving method and system for customer service knowledge based on human feedback and candidate verification.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a customer service knowledge self-evolution method based on manual feedback and candidate verification, comprising:

[0008] Collect human customer service conversations between users and human customer service representatives, and extract at least one candidate knowledge from the human customer service conversations; A preliminary value judgment is made on the candidate knowledge; Before the candidate knowledge enters the virtual knowledge base, the formal knowledge base is searched, and the operation type of the candidate knowledge relative to the formal knowledge base is fixed as addition, supplementation, replacement or no operation based on the knowledge intent relationship and answer relationship between the candidate knowledge and the formal knowledge. When the operation type is addition, supplementation or replacement, the candidate knowledge is written into the virtual knowledge base, and the operation type, target formal knowledge identifier, suggested answer, review status and evidence status are saved. During the intelligent customer service process, the formal knowledge base and the virtual knowledge base are jointly retrieved, and the candidate knowledge in the virtual knowledge base is exposed and used in a controlled manner according to the operation type, the review status, the evidence status and the formal knowledge hit result. After the session ends, evidence events for the candidate knowledge are generated based on user feedback, whether the session was transferred to a human agent, and the relationship between the human agent's answer and the candidate answers. When the candidate knowledge meets the promotion gating conditions, the formal knowledge base is updated according to its fixed operation type, and the promotion history is recorded to support auditing and rollback.

[0009] This invention also provides a customer service knowledge self-evolution system based on human feedback and candidate verification, including: The candidate knowledge acquisition unit is used to collect the human service conversation between the user and the human customer service representative, and extract at least one candidate knowledge from the human service conversation. A preliminary value judgment unit is used to make a preliminary value judgment on the candidate knowledge; The operation type solidification unit is used to retrieve the formal knowledge base before the candidate knowledge enters the virtual knowledge base, and solidify the operation type of the candidate knowledge relative to the formal knowledge base as addition, supplementation, replacement or no operation according to the knowledge intent relationship and answer relationship between the candidate knowledge and the formal knowledge. The virtual knowledge base management unit is used to write the candidate knowledge into the virtual knowledge base when the operation type is to add, supplement, or replace, and to save the operation type, target formal knowledge identifier, suggested answer, review status, and evidence status. The controlled service unit is used to jointly retrieve the formal knowledge base and the virtual knowledge base during the intelligent customer service process, and to expose and use candidate knowledge in the virtual knowledge base in a controlled manner according to the operation type, the review status, the evidence status and the formal knowledge hit result. The hit attribution unit is used to generate evidence events for the candidate knowledge after the session ends, based on user feedback, whether the request was transferred to a human agent, and the relationship between the human agent's answer and the candidate answer. The promotion governance unit is used to update the formal knowledge base according to its fixed operation type when the candidate knowledge meets the promotion gating conditions, and to record the promotion history to support auditing and rollback.

[0010] The beneficial effects of this invention compared to existing technologies are as follows: This invention extracts candidate knowledge in a structured manner from human service sessions, transforming scattered human answers into manageable knowledge objects; before candidate knowledge enters the virtual knowledge base, it searches the formal knowledge base, solidifying the operation type as add, supplement, replace, or no operation, thus carrying a clear intention to write to the knowledge base from its creation; subsequently, in the intelligent customer service, candidate knowledge is exposed and used in a controlled manner based on the operation type, review status, evidence status, and formal knowledge hit results, ensuring that long-tail questions receive controlled answers outside the formal knowledge base, and that unverified content cannot cover formal answers; after the session ends, evidence events are generated based on user feedback, whether the request is transferred to a human, and the human answer; the formal knowledge base is updated only when candidate knowledge meets the promotion gating conditions according to the solidified operation type, avoiding direct contamination of the formal knowledge base by erroneous knowledge; finally, the promotion history is recorded to support auditing, and knowledge is deleted or restored according to the operation type when erroneous promotion occurs, thereby improving the coverage of long-tail questions and the efficiency of knowledge reuse while ensuring the auditability of knowledge base governance.

[0011] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating the customer service knowledge self-evolution method based on human feedback and candidate verification provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the overall architecture and dual closed-loop relationship provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the candidate generation process for transferring from human dialogue to a virtual knowledge base, provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of the virtual knowledge base candidate deduplication, merging, and conflict review process provided in an embodiment of the present invention. Figure 5 This is a schematic diagram of the joint retrieval and response process in the AI ​​service stage provided in an embodiment of the present invention; Figure 6 A schematic diagram of the candidate knowledge hit verification process after the end of a session, provided in an embodiment of the present invention; Figure 7 A schematic diagram of the virtual knowledge promotion formal library and rollback process provided in an embodiment of the present invention; Figure 8A schematic block diagram of a customer service knowledge self-evolution system based on human feedback and candidate verification provided in an embodiment of the present invention; Figure 9 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0016] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0017] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0018] Please see Figure 1 , Figure 1This is a flowchart illustrating the customer service knowledge self-evolution method based on human feedback and candidate verification provided in this embodiment of the invention. This method is applied to a server, extracting candidate knowledge from human service conversations, performing preliminary value judgments, and then searching the formal knowledge base. Based on the knowledge intent and answer relationship, the operation type is solidified as addition, supplementation, replacement, or no operation. Valid candidates are written into a virtual knowledge base, and the operation type, target identifier, suggested answer, review status, and evidence status are saved. During the service phase, the formal and virtual knowledge bases are jointly searched. Candidates are exposed and used in a controlled manner according to operation type, review status, evidence status, and formal hit results, avoiding replacement and unreviewed candidates from directly participating in automatic responses. After the conversation ends... Evidence events are generated based on user feedback, referral to human agents, and the relationship between human agents and candidate answers. Candidate verification information is accumulated through multi-dimensional evidence, including effective hits, negative hits, human confirmation, and conflicts. When a candidate meets the promotion threshold conditions such as compliance with operation type, evidence score, number of uses, satisfaction level, and human review approval, the formal knowledge base is updated according to the fixed operation type. The promotion history, including before and after snapshots, promotion decisions, writing results, and rollback status, is recorded. This avoids erroneous knowledge directly polluting the formal knowledge base while improving the intelligent customer service's ability to cover long-tail issues, knowledge reuse efficiency, and the auditability of knowledge base governance.

[0019] like Figure 2 As shown, the system implementing the method of this invention includes a human service feedback collection layer, a candidate judgment layer, a knowledge storage layer, an AI service verification layer, and a promotion governance layer. The human service feedback collection layer is responsible for converting the original conversation between the user and human customer service into candidate knowledge; the candidate judgment layer is responsible for preliminary value judgment, formal knowledge base positioning, operation type solidification, and virtual knowledge base matching; the knowledge storage layer includes a formal knowledge base, a virtual knowledge base, and a promotion history database; the AI ​​service verification layer is responsible for joint retrieval, answer context arrangement, service trajectory recording, and post-conversation hit evaluation; the promotion governance layer is responsible for evidence scoring, promotion gating, writing to the formal knowledge base, and rollback.

[0020] The formal knowledge base refers to the knowledge base that stores verified knowledge as a source of credible facts for intelligent customer service; the virtual knowledge base refers to the knowledge base that stores candidate knowledge to be verified, used for candidate verification and evidence accumulation; the promotion history database refers to the knowledge base that stores audit information and rollback basis for each update of formal knowledge. The three can be physically deployed in separate databases, or they can be distinguished by status fields and table structures within the same database; they can also be deployed separately in relational databases, vector databases, document databases, or key-value databases.

[0021] During system operation, the human service feedback collection layer and the AI ​​service verification layer form two interconnected processes: the first process generates candidate knowledge from human services, and the second process verifies the usability of candidate knowledge in subsequent intelligent customer service. These two processes together form a closed-loop evolution of candidate knowledge, rather than writing human answers into the formal knowledge base all at once. Human service sessions do not directly alter the formal knowledge base; instead, candidate knowledge is first generated. After entering the virtual knowledge base, candidate knowledge participates in subsequent intelligent customer service as a controlled verification object, and evidence is written back through hit attribution. Only when the evidence meets the promotion gating conditions is the candidate knowledge written into the formal knowledge base. The virtual knowledge base is neither a temporary cache nor a copy of the formal knowledge base, but rather an intermediate governance layer with candidate status, evidence status, review status, and promotion results.

[0022] The modules corresponding to each layer can be deployed as independent services or as multiple sub-modules in the same customer service intelligent agent platform. The modules can communicate with each other through database transactions, message queues, event buses or task scheduling, as long as the state changes of candidate knowledge, evidence changes and the results of writing to the formal knowledge base can be traced.

[0023] Figure 1 This is a flowchart illustrating the customer service knowledge self-evolution method based on manual feedback and candidate verification provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110 to S170.

[0024] S110. Collect the human service conversation between the user and the human customer service representative, and extract at least one candidate knowledge from the human service conversation.

[0025] In this embodiment, a human customer service session refers to the service conversation record between the user and a human customer service representative after the user is transferred to human customer service. This human customer service session can originate from a complete transfer to human customer service session, a follow-up answer from a human customer service representative, a work order processing record, a quality inspection confirmation record, or an expert's human response. For example... Figure 3 As shown, the human customer service session consists of multiple rounds, each including a role, content, timestamp, and optional channel identifier. During data collection, empty content, duplicate messages, and obvious small talk are filtered out. The user's question, the human customer service representative's answer, and the context relevant to understanding the answer are combined into candidate inputs for extraction. In the voice customer service scenario, the speech is first transcribed into text before data collection; in the ticket scenario, the ticket question, processing suggestions, and final reply are combined into a human customer service session; in the internal knowledge assistant scenario, expert human responses constitute the human customer service session.

[0026] In this embodiment, candidate knowledge refers to knowledge units that are structurally extracted from human service sessions and require verification before being written into the formal knowledge base. The candidate knowledge includes a core question, candidate answers, a set of knowledge points, a session source identifier, a scenario identifier, and context fields describing the scope of application of the candidate knowledge. Specifically, the core question refers to the question text representing a standardized, reusable user request; the candidate answer refers to a reusable response given by human customer service; the set of knowledge points refers to the set of fields storing rules, conditions, steps, exceptions, restrictions, or risk warnings in the candidate answer; and the context fields are used for review and traceability. The candidate extraction results may also include candidate source fragments, human response fragments, and the rules, conditions, operation steps, time limits, fee descriptions, applicable objects, exception conditions, and risk warnings covered by the candidate answers. These fields are used for subsequent preliminary value judgments, formal knowledge base positioning, and human review, but it is not required that all fields be written into the final formal knowledge.

[0027] When a human service session contains multiple independent knowledge points, the multiple independent knowledge points are split into multiple candidate knowledge points and enter the subsequent processing flow respectively; when extracting the candidate knowledge, the conversational expressions, greetings and user case information such as user name, order number, contact information or address are removed from the human service session, and only the reusable question intent and reusable answer are retained.

[0028] For example, when the same customer service conversation simultaneously explains a function entry point, a fee rule, and the reason why a button is invisible, it is split into three candidate knowledge points. This avoids mixing multiple questions into one formal knowledge point and allows each candidate knowledge point to accumulate evidence independently. When extracting the candidate knowledge, conversational expressions, greetings, repeated confirmations, emotional statements, and user case information such as user names, order numbers, contact information, or addresses are removed from the customer service conversation. Only reusable question intents and reusable answers are retained to prevent individual case information from entering the virtual knowledge base, while preserving the general knowledge value of the customer service responses. When candidate extraction is implemented using a large language model, the model output is limited to structured fields, such as the core question, candidate answers, knowledge point set, knowledge type, and whether human intervention is required. The output results undergo field integrity verification; if the core question or candidate answer is missing, it is not proceeded to the subsequent step of searching the formal knowledge base.

[0029] In one embodiment, candidate knowledge can be anonymized before entering the virtual knowledge base by removing user names, phone numbers, addresses, order numbers, ID card numbers, and account identifiers; specific object names in business rules can be replaced with generic category names. Anonymization strategies may include field identification, regular expression matching, named entity recognition, large language model recognition, or manual review; the anonymized content can be stored in the source session with controlled permissions for necessary auditing, but will not enter the answerable fields of the virtual knowledge base or the formal knowledge base. Anonymization does not change the operation type or evidence scoring of the candidate knowledge.

[0030] S120. Make a preliminary value judgment on the candidate knowledge.

[0031] In this embodiment, the candidate knowledge is assessed based on four dimensions: answer completeness, reusability, knowledge value, and risk of human intervention. The risk of human intervention is used to identify candidate knowledge involving complaints, legal consultations, medical advice, user privacy, order cases, requiring human verification, or unsuitable for automatic answers. The result of the preliminary value assessment is used to determine whether the candidate knowledge is included in the step of searching the formal knowledge base. The preliminary value assessment is not a formal entry assessment; candidate knowledge that passes the preliminary value assessment still needs to be written into the virtual knowledge base for verification.

[0032] Preliminary value judgment refers to the preliminary judgment used to determine whether candidate knowledge is worthwhile for subsequent formal knowledge base positioning and virtual knowledge base verification; it does not determine whether candidate knowledge is formally included in the database. Specifically, the candidate knowledge is judged based on four dimensions: answer completeness, reusability, knowledge value, and risk of human intervention. Answer completeness indicates whether the candidate answer covers the core question, is specific, and can be directly used for subsequent answers; reusability indicates whether the candidate answer is relevant to the specific case and applicable to similar users in the future; knowledge value indicates whether the candidate knowledge contains rules, steps, conditions, or exceptions worth retaining; and the risk of human intervention is used to identify candidate knowledge involving complaints, legal consultations, medical advice, user privacy, order cases, requiring manual verification, or unsuitable for automatic answers.

[0033] In one implementation, preliminary value judgment can identify three categories of content unsuitable for inclusion in the virtual knowledge base: the first category is highly case-specific content, which refers to content applicable only to a specific user, a specific order, or a specific single processing result, such as order anomalies, personal account status, or single compensation plans; the second category is risky content, such as complaints, legal liabilities, medical advice, or matters involving changes in fees, qualifications, compliance, or rights that require manual verification; the third category is low-knowledge-value content, such as small talk, repeated confirmations, and reassuring replies without clear rules. A score normalized to the 0-1 range can be assigned to each of the above four dimensions, and a weighted basic quality score can be calculated. Alternatively, the semantic model or large language model can directly output whether the content should proceed to the next stage and the reasons thereof. The basic quality score is only a component of the subsequent evidence scoring of candidate knowledge and does not constitute the sole implementation method. The results of the preliminary value judgment can save a summary reason, which is used for subsequent manual review and system debugging, and not as a source of facts for the formal knowledge base.

[0034] The preliminary value judgment result is used to determine whether the candidate knowledge is included in the step of retrieving the formal knowledge base. The preliminary value judgment is not a formal entry judgment. Candidate knowledge that passes the preliminary value judgment still needs to be written into the virtual knowledge base for verification. Formal entry is still determined by subsequent evidence events, satisfaction, review status, and promotion threshold conditions. If candidate knowledge is determined not to enter the virtual knowledge base, it is archived as an invalid candidate or a low-value candidate, and minimal audit information is saved. The minimal audit information refers to the candidate identifier, source session identifier, judgment result, and judgment reason. This archived record can be used for subsequent statistical analysis of the proportion of invalid questions in human services, the type of customer service knowledge gap, and the quality of model extraction, but it does not participate in subsequent intelligent customer service responses.

[0035] S130. Before the candidate knowledge enters the virtual knowledge base, the formal knowledge base is searched, and the operation type of the candidate knowledge relative to the formal knowledge base is fixed as addition, supplementation, replacement or no operation based on the knowledge intent relationship and answer relationship between the candidate knowledge and the formal knowledge.

[0036] In this embodiment, knowledge intent refers to the standardized expression of user requests; knowledge intent relationship refers to whether the core question of candidate knowledge and the question of formal knowledge point to the same standardized user request; answer relationship refers to the relationship between candidate answer and formal answer as consistent, inclusive, complementary or conflicting; operation type refers to the status field that represents the writing intent of candidate knowledge relative to the formal knowledge base.

[0037] In one embodiment, the above-described retrieval of a formal knowledge base includes: A multi-path semantic query is constructed, which includes at least a standard question query based on a core question, a key fact query for answers based on a set of knowledge points, an answer summary query based on candidate answers, and an answer reverse query based on the candidate answers to construct rule-based, cause-based, entry-level, or time-sensitive questions. Specifically, the standard question query is used to find formal knowledge with similar question wording and intent; the key fact query for answers is used to find formal knowledge with similar facts through key rules, conditions, and steps in candidate answers; the answer summary query is used to find formal knowledge with similar content through the overall semantics of the answer; and the answer reverse query is suitable for scenarios where the user question and the formal knowledge have significant differences in their surface meaning but are related to underlying business rules. For example, when the user question describes a phenomenon or anomaly while the formal knowledge describes a rule or condition, question retrieval alone may not be sufficient, while the rule facts in the candidate answers can be used to deduce a question expression closer to the formal knowledge. Since user questions in customer service scenarios often manifest as phenomenon descriptions, missing buttons, rule questions, or follow-up questions about reasons, while questions in the formal knowledge base may be expressed using rule names, function names, or management terminology, the positioning of the formal knowledge base should not rely solely on single-path retrieval using the user's original question.

[0038] Subsequently, the formal knowledge retrieved by the multi-way semantic queries is deduplicated and merged according to the formal knowledge identifier, and the query routes retrieved by each piece of formal knowledge, its ranking position in each query route, and similarity information are retained. In this embodiment, a query route refers to the information recorded about which query route retrieves a piece of formal knowledge, including the query type, query text, the ranking of the formal knowledge in that route, and its similarity or confidence level; the ranking position refers to the rank of the formal knowledge in the single-way query retrieval results; similarity information refers to the semantic similarity or confidence level between the formal knowledge and that query route; when the same formal knowledge is retrieved by multiple routes, its query routes are merged, rather than simply discarding duplicate results. Then, based on the ranking position and similarity information in each query route, a fusion ranking method is used to determine the candidate formal knowledge set for the solidification of the operation type. The candidate formal knowledge set refers to a preset number of top-ranked formal knowledge provided to the operation type solidification step for semantic discrimination. The fusion ranking method can be determined based on the ranking contribution of each formal knowledge in multi-way queries, the highest similarity, the number of recall routes, or a combination thereof, such as one or more of the following: descending ranking fusion, highest score fusion, weighted fusion, learning ranking, or large language model re-ranking. In one embodiment, a preset number of top-ranked formal knowledge, along with their questions, answers, and query routes, are provided to the semantic discrimination model. The semantic discrimination model determines whether the candidate knowledge and the formal knowledge share the same intent, are related but different intents, are already covered, lack supplementary information, or conflict. The location results are used for subsequent operation type solidification, rather than directly determining whether candidate knowledge should be supplemented or replaced.

[0039] Specifically, the operation type solidification includes: when the formal knowledge base does not contain the same knowledge intent, solidifying the candidate knowledge as new; when the formal knowledge base contains the same knowledge intent and the formal answer lacks conditions, steps, costs, time, restrictions, or exceptions, solidifying the candidate knowledge as supplementary and binding it to the target formal knowledge; when the formal knowledge base contains the same knowledge intent and the formal answer is incorrect, outdated, or conflicts with the candidate answer, solidifying the candidate knowledge as replacement and binding it to the target formal knowledge; when the formal knowledge base has completely covered the candidate knowledge, solidifying the candidate knowledge as no operation; when the candidate knowledge and formal knowledge belong to the same business domain but are not the same knowledge intent, solidifying the candidate knowledge as new and recording the relevant parent knowledge identifier, where the relevant parent knowledge identifier refers to the identifier used to indicate the business affiliation or retrieval anchor of the candidate knowledge, which does not indicate a supplement to the target formal knowledge, thereby reducing the risk of erroneous merging within the same domain. From the perspective of knowledge slots, adding a new knowledge slot indicates that the formal knowledge base lacks that knowledge slot; supplementing a knowledge slot indicates that the existing knowledge slot exists but the answer content is insufficient; replacing a knowledge slot indicates that the answer for the existing knowledge slot may be incorrect, outdated, or conflicting; and not performing an operation indicates that the formal knowledge base has already covered the candidate knowledge.

[0040] It should be noted that the boundaries of adding, supplementing, and replacing should not be determined solely by similarity. For example, if the core question is highly similar to a piece of formal knowledge but the candidate answer only contains content already included in the formal answer, it should be fixed as no action taken; if the core question and formal knowledge share the same intent but the candidate answer adds missing conditions or paths, it should be fixed as supplementing; if the candidate answer conflicts with the formal answer in terms of rule content, it should be fixed as replacing.

[0041] The operation type is determined before candidate knowledge enters the virtual knowledge base and is saved as a core state throughout the candidate knowledge's lifecycle. Its technical significance lies in ensuring that candidate knowledge carries a clear intention to be written to the database from the creation stage. Subsequent promotion stages do not re-evaluate whether candidate knowledge should be added, supplemented, or replaced; instead, they only determine whether sufficient evidence has been obtained, thus avoiding result drift caused by re-evaluation during promotion stages. For supplementary candidate knowledge, the target formal knowledge identifier, target formal knowledge question, target formal knowledge answer, and suggested answer are saved. The suggested answer can be a synthesis of the candidate answer and the target formal knowledge answer, or only the supplementary candidate content can be saved and merged during promotion. Supplementary types typically do not change the target formal knowledge question. Replacement candidate knowledge is set to require manual review by default. It cannot be used as a source of fact for automatic answers or for formal knowledge base updates before the review status is approved, to prevent unreviewed candidate answers from directly participating in answers or being written to the formal knowledge base, causing error propagation. The review materials for replacement types include the target formal knowledge answer, candidate answer, conflict fields, source session, candidate evidence, manual confirmation count, negative hit count, and conflict reason. Reviewers can approve, reject, or request modifications to candidate answers. For candidate knowledge that is not an operation type, an archive record is saved to prove that it has been determined to be covered by the formal knowledge base, but it is not written into the virtual knowledge base to participate in subsequent evidence accumulation, so as to avoid accumulating a large number of duplicate candidates with no new value in the virtual knowledge base. In one embodiment, the operation type solidification result also includes intent relationship, answer relationship, confidence level, target formal knowledge identifier, related parent knowledge identifier, whether review is required and the reason for judgment. The above fields together form the state baseline when candidate knowledge enters the virtual knowledge base.

[0042] In one embodiment, such as Figure 4As shown, before writing the candidate knowledge into the virtual knowledge base, existing candidate knowledge is first recalled from the virtual knowledge base. The operation scope of the candidate set is first limited by the rule layer, and then the relationship between knowledge slots and answers is determined by the semantic layer. The operation scope refers to the limited range within which candidate knowledge is subject to duplicate judgment in the virtual knowledge base. The rule layer is only responsible for limiting the candidate pool and does not directly determine whether to merge. Specifically, merging or conflict is only determined within candidate sets with the same operation type and the same operation scope. The same operation scope for supplementary or replacement types means the same target formal knowledge identifier; the same operation scope for newly added types means neither is bound to a target formal knowledge identifier. For newly added candidate knowledge, since there is no target formal knowledge identifier, potentially duplicated existing candidate knowledge can be recalled based on question vectors, keywords, business objects, candidate answers, and question variations. The question vector refers to the vector obtained by semantically vectorizing the core question. Subsequently, the semantic layer determines whether the new candidate knowledge and existing candidate knowledge belong to the same knowledge slot. The knowledge slot refers to a knowledge unit with the same user intent, business object, and answer constraints. The judgment focuses on comparing user intent, business object, answer constraints, and applicable scope, rather than just comparing the literal similarity of the questions. When new candidate knowledge and existing candidate knowledge belong to the same knowledge slot and have the same, inclusive, or complementary answers, no new candidate knowledge is created. Instead, the new candidate knowledge is merged into the evidence event of the existing candidate knowledge. The merged content may include question variations, manual confirmation counts, source sessions, knowledge point sets, satisfaction levels, and supplementary fragments of candidate answers. When they belong to the same knowledge slot but their answers conflict in terms of value, support status, applicable conditions, or restrictive rules (e.g., opposite support / resistance, different costs, different time limits, different applicable objects, different paths, or different preconditions), conflict evidence is recorded and the review status is adjusted to require manual review. When they do not belong to the same knowledge slot, new candidate knowledge is created. This mechanism allows the virtual knowledge base to absorb multiple manual confirmations of the same knowledge while avoiding the erroneous merging of different knowledge. It can also detect changes in business rules or inconsistencies in customer service statements before candidate knowledge enters the formal knowledge base. For new or supplementary candidate knowledge with conflict evidence, its review status can be adjusted from no review required to review required, and it can only be allowed to participate in the promotion gate after manual processing.

[0043] In one embodiment, for multiple candidate knowledge extracted from a manual service session, preliminary value judgment, formal knowledge base location, and operation type solidification can be performed in parallel to improve processing efficiency. During the virtual knowledge base matching and writing phase, serial writing, transaction locks, unique constraints, or optimistic locking are employed to prevent candidate knowledge in the same knowledge slot from being repeatedly created. Specifically, mutual exclusion locks can be set for the same target formal knowledge identifier, the same new intent cluster, or the same candidate recall range. The granularity of the lock can be determined according to the target formal knowledge, candidate intent, business scenario, or vector recall cluster. Even if two candidate knowledges are judged to be in the same knowledge slot almost simultaneously, only one candidate knowledge is created, and the other candidate knowledge is merged into an evidence event or marked as conflicting. If there is a complementary relationship between multiple parallel candidate knowledges, they can be merged into different knowledge points of the same candidate knowledge during the serial merging phase. If there is a conflict relationship, all related candidate knowledge is marked as pending review, and the conflict source session is saved.

[0044] S140. When the operation type is addition, supplementation or replacement, the candidate knowledge is written into the virtual knowledge base, and the operation type, target formal knowledge identifier, suggested answer, review status and evidence status are saved.

[0045] In this embodiment, the target formal knowledge identifier refers to the unique identifier of the formal knowledge that the candidate knowledge points to when the operation type is supplementation or replacement; the suggested answer refers to the answer content to be written into the formal knowledge base during promotion; the review status refers to the status field indicating whether the candidate knowledge needs to be manually reviewed and whether the review is passed, including no review required, manual review required, review passed, and review rejected; the evidence status refers to the status field indicating the evidence accumulation of the candidate knowledge during the service verification process, which is jointly characterized by retrieval count, exposure count, usage count, effective hit count, negative hit count, manual confirmation count, conflict count, satisfaction list, and candidate evidence score.

[0046] The candidate knowledge records in the virtual knowledge base include at least the following: candidate identifier, session identifier, scenario identifier, core question, candidate answer, knowledge point set, context, user intent, basic quality score, operation type, target formal knowledge identifier, target formal knowledge question, target formal knowledge answer, suggested answer, operation reason, review status, review reason, question vector, as well as retrieval count, exposure count, usage count, effective hit count, negative hit count, manual confirmation count, conflict count, satisfaction list, hit history, question variants, evidence session list, and candidate evidence score. The candidate evidence score is the total score of the candidate knowledge. Question variants refer to different ways users ask the same knowledge slot in subsequent services. These can improve subsequent retrieval recall and help manual reviewers understand the scope of candidate application, but should not automatically change the question of formal knowledge unless explicitly confirmed by promotion or manual review process. Hit history refers to the historical information recording each hit attribution result. The evidence session list refers to the list recording the session identifiers that generated the evidence event. The above records can be further divided into basic fields, operation fields, evidence fields, and governance fields: Basic fields describe the candidate knowledge itself, including the core question, candidate answer, knowledge point set, business scenario, context summary, source session identifier, and candidate generation time; Operation fields describe the writing intent of candidate knowledge relative to the formal knowledge base, including operation type, target formal knowledge identifier, related parent knowledge identifier, target formal knowledge question, target formal knowledge answer, suggested answer, and operation reason; Evidence fields describe the usage effect of candidate knowledge in the service, including retrieval count, exposure count, usage count, effective hit count, negative hit count, manual confirmation count, conflict count, satisfaction list, hit history, and evidence session list. By distinguishing between retrieval, exposure, and actual use, it is possible to avoid candidate knowledge gaining excessive credibility simply because it has been recalled; Governance fields describe the review, promotion, and rollback status, including review status, review reason, reviewer, review time, whether participation in answering is allowed, whether promotion is allowed, promotion time, promotion result, rollback status, and rollback reason.

[0047] In one implementation, the virtual knowledge base maintains a lifecycle state for each candidate knowledge. This lifecycle state includes at least one or more of the following: generated, pending review, under verification, available for answering, blocked, promoted, rejected, rolled back, and conflict blocked. Specifically, the "available for answering" state indicates that the candidate knowledge can enter the visible context of the answer generator when the controlled exposure conditions described in step S150 are met; the "blocked" state indicates that the candidate knowledge exists but cannot be used for automatic answering; the "promoted" state indicates that the candidate knowledge has been written into the formal knowledge base; and the "rolled back" state indicates that the update to the formal knowledge base corresponding to the candidate knowledge has been revoked. The lifecycle state is independent of the operation type: the operation type represents the knowledge relationship of the candidate knowledge relative to the formal knowledge base, while the lifecycle state represents the stage of the candidate knowledge in the governance process. The state transitions of candidate knowledge are triggered by events, including candidate creation events, review approval events, review rejection events, retrieval exposure events, actual use events, valid hit events, negative hit events, manually confirmed evidence events, conflict evidence events, successful promotion events, and rollback events. Each event records its occurrence time, source session, and triggering reason. State machine management prevents candidate knowledge from entering the formal knowledge base due to insufficient evidence or incomplete review, and also prevents conflicting candidate knowledge from being referenced by the intelligent customer service. In multi-tenant or multi-business-line scenarios, the state transition conditions for candidate knowledge can be set according to business domain: general consultation-related candidate knowledge can enter the answerable state after meeting a preset number of valid hits; candidate knowledge involving fees, rights, compliance, or account security, even if it receives a valid hit, still needs to be manually reviewed and approved before it can participate in answering or be promoted. In one embodiment, candidate knowledge can also be version-marked. Each time candidate knowledge is merged with manual confirmation, a question variant is added, a suggestion is modified and written into the answer, the review status is changed, or it is promoted to the formal knowledge base, a version record is generated. The version record is linked to the promotion history database so that the complete change chain of candidate knowledge from generation to promotion can be explained during auditing.

[0048] S150. During the intelligent customer service process, jointly search the formal knowledge base and the virtual knowledge base, and according to the operation type, the review status, the evidence status and the formal knowledge hit result, perform controlled exposure and controlled use of candidate knowledge in the virtual knowledge base.

[0049] In this embodiment, "formal knowledge hit result" refers to the judgment result of whether there is formal knowledge in the formal knowledge base that matches the user's question after joint retrieval; "controlled exposure" refers to determining whether candidate knowledge enters the visible context of the answer generator according to preset rules; and "controlled use" refers to determining whether candidate knowledge is actually cited by the answer generator as the basis for an answer according to preset rules. Figure 5As shown, this step does not simply mix the search results from the formal knowledge base and the virtual knowledge base and then hand them over to the answer generator. Instead, it first performs a credibility stratification on the search results. The credibility stratification includes at least formal knowledge, available new candidates, available supplementary candidates, weakly relevant knowledge, candidates under review, candidates whose evidence scores have not reached the preset scoring threshold, and conflicting candidates. The weakly relevant knowledge refers to knowledge that has semantic relevance to the user's question but does not meet the reliable hit conditions.

[0050] Specifically, when the formal knowledge base contains reliable hits, the formal knowledge is used as the primary source of the answer. Supplementary candidate knowledge only participates in the answer when the target formal knowledge it is bound to is currently reliable hit knowledge and the candidate evidence score reaches a preset scoring threshold. Newly added candidate knowledge does not participate in the answer, in order to avoid new candidates overwriting the formal answer.

[0051] When no reliable match is found in the formal knowledge base, newly added candidate knowledge with a score reaching a preset threshold and a review status of "no review required" or "approved" is allowed to participate in automatic answers. Candidate knowledge of the replacement type, candidate knowledge under review, rejected candidate knowledge, candidate knowledge with a score below the preset threshold, and candidate knowledge with conflicting evidence are not allowed as sources of fact for automatic answers. Even if replacement candidate knowledge is highly relevant to the user's question, it is marked as unanswerable automatically, and a prompt is given to transfer to a human or use an existing answer from the formal knowledge base to avoid unconfirmed new rules directly replacing formal knowledge. Furthermore, access control also applies to the candidate knowledge status: pending, conflicting, and replacement candidate knowledge, even if retrieved, cannot be read as answer context by ordinary intelligent customer service; only candidate knowledge that has passed review and meets the answerability criteria can be exposed to the answer generator. Different answer strategies can be configured for different business scenarios. For example, in pre-sales consultation scenarios, newly added candidate knowledge with a score reaching a preset threshold can participate in answers, while in account security, refund, complaint, and compliance scenarios, only formal knowledge and supplementary candidate knowledge that has passed human review are allowed to participate in answers.

[0052] The service process records the service trajectory, which includes user questions, intelligent customer service answers, retrieved formal knowledge identifiers, retrieved candidate knowledge identifiers, candidate knowledge identifiers exposed to the answer generator, candidate knowledge identifiers actually used in the answer, answer routes, and conversation segment identifiers. Retrieval, exposure, and actual use in the answer are recorded as different process evidence records, and this process evidence is not directly equivalent to valid hit evidence. A reliable hit refers to the similarity between the retrieved formal knowledge and the user question reaching a preset similarity threshold, and the knowledge meaning... Figure 1 To.

[0053] Specifically, the service trajectory is recorded during the service process. This service trajectory refers to a collection of information recording the entire process of this round of service, including session identifiers, user questions, intelligent customer service answers, retrieved formal knowledge identifiers, retrieved candidate knowledge identifiers, candidate knowledge identifiers exposed to the answer generator, candidate knowledge identifiers actually used in the answer, answer routes, session fragment identifiers, and the current context state. "Retrieved" indicates that candidate knowledge has entered the search results; "exposed" indicates that candidate knowledge has entered the visible context of the answer generator; "actually used in the answer" indicates that candidate knowledge has been actually referenced by the answer generator or used by the deterministic answer logic. These three have different meanings and are recorded as different process evidence records. This process evidence is not directly equivalent to valid hit evidence to avoid mistaking weakly relevant search results for valid knowledge. The answer route refers to the factual source path adopted in this answer, i.e., whether formal knowledge, supplementary candidate knowledge, or newly added candidate knowledge was used in this answer; the session fragment identifier is a unique identifier for each fragment after segmenting multiple rounds of conversation according to user questions. A complete customer service session may contain multiple questions. For example, a user might first inquire about function access, then about fee rules, and finally ask why a button is not visible. If only a single satisfactory or unsatisfactory result is given for the entire session, it's easy to attribute feedback errors to irrelevant candidate knowledge. Therefore, for each segment, the segment identifier, user question, answer, search results, actually used candidate knowledge, and feedback status are independently saved so that the cause can be attributed segment by segment after the session ends. When an answer uses multiple candidate knowledge simultaneously, the contribution type of each candidate knowledge is recorded. For example, one candidate knowledge provides the core answer, another provides constraints, and a third provides the operation path. When the user is satisfied, different weights are assigned to different candidate knowledge based on their contribution type as valid evidence of hit. The candidate knowledge providing the core answer is assigned a greater weight than the candidate knowledge providing constraints or operation paths. The answer generator can be a large language model, a templated answerer, a retrieval enhancement generator, or a multi-agent answering system. Regardless of the generation method used, evidence of whether candidate knowledge actually participated in the answer is saved, rather than just retrieval logs.

[0054] S160. After the session ends, based on user feedback, whether the request is transferred to a human agent, and the relationship between the human agent's answer and the candidate answers, an evidence event for the candidate knowledge is generated.

[0055] In this embodiment, an evidence event refers to a record used to update the evidence status of the candidate knowledge, including valid hit evidence, negative hit evidence, manually confirmed evidence, and conflicting evidence. For example... Figure 6As shown, hit attribution is performed after the session ends or the session state stabilizes, rather than immediately determining the validity of candidate knowledge during retrieval; a stable session state means that no new messages are added within a preset time period. Hit attribution reads service trajectory, user feedback, whether the issue was resolved, whether it was transferred to a human agent, and the human agent's answer after the transfer. Retrieving candidate knowledge only indicates that it has a semantic relevance to the user's question, and being exposed only indicates that it has the opportunity to be used. Actually using it to answer does not necessarily mean that it is effective. Only by combining user feedback, whether the issue was resolved, and whether it was transferred to a human agent can a more credible hit evidence be formed. User feedback can include explicit feedback, implicit feedback, and human service feedback: explicit feedback includes satisfaction ratings, likes, dislikes, or user text evaluations; implicit feedback includes whether the user continues to ask follow-up questions, whether it was transferred to a human agent, and whether the same question was repeated; human service feedback includes the final answer from the human customer service representative and the human processing result.

[0056] In one embodiment, step S160 described above may include steps S161 to S163.

[0057] S161. When user feedback indicates that the issue has been resolved or the satisfaction level has reached a preset satisfaction threshold, valid hit evidence is generated for the candidate knowledge actually used to answer the question.

[0058] In this embodiment, valid hit evidence refers to positive evidence indicating that candidate knowledge actually participated in answering the question and that the user's question was resolved; when generating valid hit evidence, usage counts are added, the satisfaction list and candidate evidence scores are updated, and the current question variant is recorded; when the user indicates that the question has been resolved, is satisfied, or continues to complete the business operation, it is considered to be resolved.

[0059] S162. When the issue remains unresolved after using candidate knowledge or user feedback, negative hit evidence is generated; when the human answer after being transferred to a human is compatible with the candidate answer, human confirmation evidence is generated.

[0060] In this embodiment, negative hit evidence refers to negative evidence indicating that the user's question remains unresolved even after candidate knowledge has been used to answer it. A negative hit does not necessarily result in the immediate deletion of candidate knowledge, but it lowers its candidate evidence score. When the negative hit count reaches a preset negative hit threshold, the candidate knowledge is transferred to a pending review status. When the human answer after being transferred to a human agent is compatible with the candidate answer, human confirmation evidence is generated. This human confirmation evidence indicates that the human service result supports the content of the candidate answer. When generating human confirmation evidence, newly added conditions or supplementary content from the human answer can also be merged into the candidate knowledge.

[0061] It should be noted that negative hit evidence reflects the unresolved service outcome of user-side issues, while human confirmation evidence reflects the support of human answers for candidate answer content. The two have different evaluation dimensions: when the case is transferred to human counsel and the human answer is compatible with the candidate answer, human confirmation evidence is generated instead of negative hit evidence; when the case is transferred to human counsel and the human answer is irrelevant or incompatible with the candidate answer, negative hit evidence is generated; when conflict evidence is generated, human confirmation evidence is no longer generated simultaneously.

[0062] S163. When the human answer after being transferred to a human examiner conflicts with the candidate answer in terms of numerical value, whether it is supported, applicable conditions, or restrictive rules, conflict evidence is generated; for candidate knowledge that is not actually used to answer, no valid hit evidence is generated.

[0063] In this embodiment, conflict evidence refers to evidence indicating that a candidate answer is factually inconsistent with a human answer or formal knowledge. After generating conflict evidence, the candidate knowledge is prevented from continuing to automatically answer and be automatically promoted. For candidate knowledge that is not actually used to answer, even if it is retrieved or exposed to the answer generator, no valid hit evidence is generated to prevent a large number of weakly related candidate knowledge in the virtual knowledge base from being erroneously promoted due to frequent recall.

[0064] Whenever candidate knowledge generates an evidentiary event, its count field, satisfaction list, issue variants, and evidence session list are updated, and the candidate evidence score is recalculated. Among them, process evidence such as being retrieved and being exposed is used for auditing and statistics, while positive evidence such as valid hit evidence and manually confirmed evidence, and negative evidence such as negative hit evidence and conflicting evidence are used to influence promotion gating judgments.

[0065] In one implementation, the candidate evidence score refers to a score that comprehensively evaluates the credibility of candidate knowledge. It is determined by a base quality score, the number of effective uses, average satisfaction, the number of negative hits, and the number of conflicts. The base quality score reflects whether the candidate knowledge itself is complete, reusable, and has knowledge value. The number of effective uses reflects whether the candidate knowledge is triggered by real problems, and its contribution can be expressed using a saturation growth function. This saturation growth function is a function that gradually decreases the incremental contribution of a single effective hit to the candidate evidence score as the number of effective uses increases. This allows the contribution of repeated hits to gradually converge, preventing a small number of high-frequency problems from causing candidate knowledge to rise too quickly and also preventing abnormal traffic. Or test traffic may cause evidence to be inflated; average satisfaction can be obtained by combining explicit ratings, user feedback, whether to continue questioning, whether to transfer to human intervention, and the results of human intervention. Different weights can be configured for different feedback sources. For example, explicit satisfaction ratings have a higher weight than implicit feedback, no further questioning is considered a weak positive signal, and transferring to human intervention is considered a negative signal or a signal to be confirmed. The number of negative hits and the number of conflicts can be penalized using a penalty function. The penalty function is a function that deducts the candidate evidence score based on the number of negative hits or conflicts. The penalty weight for conflicting evidence is greater than the penalty weight for ordinary negative hits because conflicting evidence may represent that the candidate knowledge content itself is wrong.

[0066] S170. When the candidate knowledge meets the promotion gating conditions, the formal knowledge base is updated according to its fixed operation type, and the promotion history is recorded to support auditing and rollback.

[0067] In this embodiment, the promotion gating conditions include: the operation type is addition, supplementation or replacement; the candidate evidence score reaches a preset scoring threshold; the number of effective uses reaches a preset number of uses threshold; the average satisfaction reaches a preset satisfaction threshold; and the candidate knowledge that requires manual review has been approved. When the candidate knowledge meets the promotion gating conditions, the formal knowledge base is updated according to its fixed operation type, and the promotion history is recorded to support auditing and rollback, including: For newly added candidate knowledge, create new formal knowledge questions and formal knowledge answers; for supplementary candidate knowledge, keep the target formal knowledge question unchanged, and update the target formal knowledge answer with the supplementary suggestion written into the answer. For candidate knowledge of the replacement type, the target formal knowledge question remains unchanged after the review is approved; Update the target's formal knowledge answer with the replaced suggestion in the answer; The promotion history includes candidate knowledge snapshots, promotion gating decisions, formal knowledge base write results, promotion time, rollback status, and rollback results. For supplementary or replacement promotions... The promotion history also includes a snapshot of the target formal knowledge before promotion and a snapshot of the target formal knowledge after promotion; when a rollback occurs, for new types of promotions, the formal knowledge created during the promotion is deleted or archived, and for supplementary or replacement types of promotions, the target formal knowledge answers before promotion are restored.

[0068] Specifically, in this embodiment, the promotion gating condition refers to the set of conditions that must be met before candidate knowledge is allowed to be written into the formal knowledge base; the promotion history refers to the audit information recording the entire process of candidate knowledge being written from the virtual knowledge base to the formal knowledge base; and rollback refers to the operation of reversing the executed formal knowledge base update and restoring it to the state before promotion. The promotion gating condition includes: the operation type is addition, supplementation, or replacement; the candidate evidence score reaches a preset scoring threshold; the number of effective uses reaches a preset number of uses threshold; the average satisfaction reaches a preset satisfaction threshold; and the candidate knowledge requiring manual review has been approved. In one embodiment, the promotion gating can also check the most recent evidence time, which refers to the time when the candidate knowledge last generated evidence events, to avoid candidate knowledge that has not had new evidence for a long time being promoted based solely on historical evidence. The promotion gating adopts a method of satisfying multiple conditions simultaneously, rather than a single total score threshold: even if the candidate evidence score reaches the preset scoring threshold, if the number of effective uses is insufficient, there are unresolved conflicts, or the review status is not approved, the candidate knowledge cannot be promoted, thereby avoiding knowledge pollution caused by scoring errors. Gating parameters can be configured by business domain. In this embodiment, low-risk business refers to general consulting business that does not involve fees, rights, compliance, or account security, while high-risk business refers to business involving changes in fees, qualifications, compliance, rights, or account security. For low-risk business, the effective usage threshold can be lowered to improve the knowledge update speed. For high-risk business, the satisfaction threshold can be increased, manual review can be enforced, or candidate knowledge can be required to undergo a preset observation period. Furthermore, all candidate knowledge of replacement types and candidate knowledge involving changes in fees, qualifications, compliance, or rights can be required to undergo manual review. The parameter configuration itself is also recorded to audit the gating strategy used during a promotion. Different promotion gating conditions can also be set for different operation types: the new type requires no reliable hits in the formal knowledge base and multiple effective hits by independent users; the supplement type requires binding to target formal knowledge and reliable hits in the current question; the replacement type requires manual review approval and a conflict count of zero or that the conflict has been reviewed and explained. The above promotion gating strategies for different operation types can be written into a configuration file to adjust the gating conditions without modifying the main system logic.

[0069] In one embodiment, manual forced promotion is supported, but forced promotion should record the human operator, operation time, reason, candidate knowledge snapshot and formal knowledge snapshot. Forced promotion does not change the regular automatic gating process, but provides an auditable channel for emergency rule updates.

[0070] like Figure 7 As shown, when the candidate knowledge meets the promotion gating conditions, the formal knowledge base is updated according to its fixed operation type, including: for newly added candidate knowledge, creating new formal knowledge questions and answers, and saving relevant parent knowledge identifiers as metadata; for supplementary candidate knowledge, keeping the target formal knowledge question unchanged, and updating the target formal knowledge answer to the supplemented suggestion written into the answer; for replacement candidate knowledge, keeping the target formal knowledge question unchanged after approval, and updating the target formal knowledge answer to the replacement suggestion written into the answer. For supplementary and replacement types, keeping the target formal knowledge question unchanged is because the formal question serves as a stable entry point for knowledge slots; keeping it unchanged prevents candidate knowledge question variants from continuously rewriting the formal knowledge question, which helps reduce the disturbance of user question variations to the formal knowledge structure; the core question of the candidate knowledge can be saved as a question variant or retrieved as enhanced information, without replacing the formal question. For replacement types, the target formal knowledge can be required to be in a replaceable state, which means that the target formal knowledge is not locked by other promotion tasks and currently has no unprocessed conflict markers, and a snapshot of the old answer is saved before writing. If the replacement involves multiple formal knowledge items, it can be broken down into multiple promotion tasks, and the state of each target formal knowledge item before and after can be recorded separately, so as to avoid the impact of a single update on too wide a range.

[0071] Formal knowledge base writing includes two aspects: content writing and index updating. For new entries, new formal knowledge entries are created and question vectors, answer vectors, or document indexes are generated. Composite vectors can also be generated, which are vectors obtained by concatenating or weighting question vectors and answer vectors. Answer vectors are vectors obtained by semantically vectorizing the answers to formal knowledge. For supplementary and replacement entries, the answer content of the target formal knowledge is updated, and related vectors and full-text indexes are refreshed. The write operation has transactional semantics: For relational databases, formal knowledge can be updated, promotion history recorded, and candidate knowledge in the virtual knowledge base archived within the same transaction. For vector databases, formal knowledge and metadata can be written first, then the index updated, and finally the promotion history recorded. If the formal knowledge content update is successful but the index update fails, the content update is rolled back or the knowledge is marked as an index pending refresh. If the index update fails, the reason for the failure in the promotion history is recorded, and candidate knowledge in the virtual knowledge base is not deleted. If the promotion history fails, candidate knowledge in the virtual knowledge base is not deleted to avoid changes to the formal knowledge base that cannot be rolled back. Once the writing is complete, a formal knowledge base verification task can be triggered. The verification task includes checking whether the retrieval is recallable, whether the answer can be referenced by the intelligent customer service, whether the metadata is complete, whether the original candidate knowledge has been archived, whether the promotion history is searchable, and whether the rollback entry is available.

[0072] Before writing to the formal knowledge base, a promotion snapshot is generated. This snapshot includes a candidate knowledge snapshot, a snapshot of the target formal knowledge before promotion, suggested content to be written, promotion gating results, review results, and an evidence summary, used for auditing and subsequent rollback. After writing, the promotion history is recorded. This history includes the candidate identifier, operation type, target formal knowledge identifier, candidate knowledge snapshot, promotion gating decision, formal knowledge base writing result, content before writing, content after writing, promotion time, execution result, rollback status, and rollback result. For supplementary or replacement promotions, the promotion history also includes a snapshot of the target formal knowledge before promotion and a snapshot of the target formal knowledge after promotion. Through the promotion history, reviewers can track which candidate knowledge a piece of formal knowledge came from, which human service session, what evidence it went through, why it reached the promotion gating conditions, and what changes occurred before and after writing. When a rollback occurs, for new promotions, the formal knowledge created during the promotion is deleted or archived; for supplementary or replacement promotions, the snapshot of the target formal knowledge before promotion is read, and the answer and metadata of the target formal knowledge before promotion are restored. Rollback is used to handle situations such as incorrect promotion of candidate knowledge, changes in business rules, issues discovered during manual review, or a subsequent increase in negative hit counts reaching a preset threshold. Before a rollback, the scope of impact can be displayed, including the formal knowledge identifier, the current answer, the answer before promotion, the source candidate knowledge, the number of times the knowledge has been recently retrieved and used, related user questions, and whether there are any subsequent dependent candidate knowledge. If the scope of impact reaches a preset threshold, secondary confirmation is required. After a rollback, the status of the original candidate knowledge is set to "rolled back" or "pending review again." If the rollback reason is an error in the candidate knowledge content, the candidate knowledge will no longer be used for answering. If the rollback reason is a change in business rules, the candidate knowledge will be retained as historical evidence and await new candidate knowledge or manual updates. If a failure occurs during formal knowledge base writing, index refresh, promotion history, or candidate knowledge archiving, the incomplete stage is identified through the promotion task status, and compensation or retry is performed based on the completed actions. Each rollback saves the rollback reason, operator, rollback time, content before recovery, content after recovery, and associated promotion records, thus forming a complete audit chain from candidate knowledge generation to formal knowledge base writing to rollback.

[0073] It should be noted that manual review does not only occur upon formal entry into the database, but can be conducted throughout the candidate knowledge lifecycle. Manual review tasks are triggered at various stages, including candidate generation, operation type solidification, conflict detection, negative hit addition, pre-promotion gating, and rollback review. Review tasks can include core questions, candidate answers, source human service sessions, similar formal knowledge, reasons for operation type judgment, virtual knowledge base merging records, hit evidence summaries, conflict evidence, and suggested actions. Reviewers can choose to approve, reject, modify candidate answers, adjust operation types, mark as postponed, or request re-extraction. For candidate knowledge of replacement types, the review materials highlight the differences between the target formal knowledge answer and the candidate answer, such as whether it is supported, applicable conditions, costs, payment time, operable paths, eligibility restrictions, or exception rules. These differences are saved as structured fields to allow reviewers to quickly determine whether outdated or erroneous old knowledge truly exists. Regarding conflicts, we can distinguish between conflicts between candidate knowledge, conflicts between candidate knowledge and formal knowledge, and conflicts between candidate knowledge and human service results: when candidate knowledge conflicts with each other, merging is temporarily suspended; when candidate knowledge conflicts with formal knowledge, it is put into replacement pending review; when candidate knowledge conflicts with human service results, its candidate evidence score is reduced. Candidate knowledge that passes review is not necessarily promoted immediately. Passing review only indicates that the content risk has been eliminated; candidate knowledge still needs to meet promotion threshold conditions such as effective usage frequency, average satisfaction, and candidate evidence score. By separating review and promotion, we avoid turning manual review into a one-time data entry action. Candidate knowledge that is rejected is archived rather than physically deleted. The archived record explains that the candidate knowledge was proposed, why it was rejected, and the source conversation, which helps in subsequent analysis of model mis-extraction, inconsistent customer service statements, or disputes over business rules. Candidate knowledge rejected for promotion does not need to be deleted immediately; its historical evidence can be retained and set to a frozen, rejected, or manually modified status. If new manual confirmation or modification occurs later, it can re-enter the promotion threshold. The frozen status means that the candidate knowledge is suspended from participating in answering and promotion threshold judgment.

[0074] In one embodiment, the system can support different combinations of model capabilities: candidate extraction can use a large language model, a structured information extraction model, or manually configured templates; formal knowledge base positioning can use a vector model, keyword retrieval, hybrid retrieval, or knowledge graph retrieval; operation type judgment, duplicate judgment, and hit attribution can use a large language model, a semantic classification model, a rule model, or a combination thereof. In enterprise deployment, the formal knowledge base and the virtual knowledge base can use the same relational database to store structured fields and use vector indexes to store question vectors or answer vectors, or they can use a relational database to store metadata, a vector database to store retrieval vectors, and object storage to store original conversations and review materials. This invention does not require the use of a specific large language model; as long as the system retains the candidate knowledge structure, the operation type is fixed, the virtual knowledge base is used under control, post-conversation evidence evaluation is implemented, promotion gating is implemented, and promotion history rollback is implemented, it falls within the technical concept of this invention. The system can also be integrated with an enterprise permission system: ordinary customer service personnel can generate manual service answers, knowledge administrators can review candidate knowledge of replacement types, quality inspectors can mark conflicting or confirm candidate knowledge, system administrators can configure gating thresholds and execute rollbacks, and the operations of different roles are recorded in the candidate knowledge metadata or promotion history. In a multi-tenant deployment, the formal knowledge base, virtual knowledge base, and promotion history of different enterprises, business lines, or knowledge domains are logically isolated, and candidate recall, duplicate judgment, and hit attribution must not cross unauthorized tenants or business domains.

[0075] In one embodiment, when the large language model call fails, times out, has an invalid output format, or insufficient confidence, a degradation process is initiated: During the candidate extraction stage, the original human service session is retained and marked as pending; during the operation type solidification stage, only the formal knowledge base retrieval results are saved and await human judgment, without defaulting to new additions; during the intelligent customer service stage, only the formal knowledge base is used for responses. When vector retrieval is unavailable, keyword retrieval, rule matching, or human tagging is used as a temporary recall method; when the formal knowledge base positioning is insufficient, the automatic judgment confidence of operation type solidification is reduced, and human review is required before determining the operation type. When the virtual knowledge base is unavailable, the formal knowledge base response is returned, and the virtual knowledge base unavailability event is recorded. This event is not interpreted as invalid candidate knowledge, nor does it generate negative hit evidence, because the candidate knowledge did not actually participate in the response. Through this degradation strategy, the formal customer service can remain available when the external model, retrieval service, or storage component is unstable, and abnormal states can be avoided from contaminating candidate evidence and the formal knowledge base.

[0076] In one embodiment, this embodiment is not limited to question-and-answer type knowledge bases, but is also applicable to knowledge forms such as document fragments, process descriptions, policy rules, knowledge graph nodes, tool call instructions, and operation manuals. For document-based knowledge bases, supplementation type is represented by adding condition descriptions or operation steps to a document paragraph, replacement type is represented by replacing outdated paragraphs, and addition type is represented by adding new document fragments. For knowledge graph type knowledge bases, candidate knowledge is represented by adding nodes, supplementing node attributes, replacing relationship attributes, or adding edges. During promotion, the formal knowledge base update module maps candidate knowledge to graph update actions. For multi-business line enterprises, different business lines share the same knowledge self-evolution framework, but configure different candidate extraction templates, operation type judgment criteria, review rules, promotion thresholds, and rollback strategies, and control which knowledge bases candidate knowledge is visible in through business domain identifiers.

[0077] In one embodiment, the system provides observability and evaluation capabilities: it statistically analyzes the number of generated candidates, initial screening pass rate, operation type distribution, virtual knowledge base merging rate, conflict rate, and approval rate; during the intelligent customer service phase, it statistically analyzes the formal knowledge base hit rate, virtual knowledge base hit rate, candidate exposure rate, candidate actual usage rate, candidate effective hit rate, negative hit rate, and rate of conversion to manual intervention; during the promotion phase, it statistically analyzes the average promotion cycle, the number of effective hits required to reach the threshold, the formal knowledge usage rate after promotion, the negative feedback rate after promotion, the rollback rate, and the distribution of rollback reasons. The system can also generate a knowledge base maintenance report, including the number of newly created candidate knowledge, the number of merged candidate knowledge, the number of conflicting candidate knowledge, the number of promoted candidate knowledge, the number of rolled-back candidate knowledge, the average promotion cycle, the main reasons for not meeting the standards, and the high-frequency missing knowledge domains; it can also provide a candidate-level tracking page and management console, displaying the complete link of candidate knowledge from source session, extraction results, initial screening results, formal knowledge base positioning, operation type, virtual knowledge base merging, hit evidence, review results to promotion history, as well as candidate evidence scores, usage counts, satisfaction levels, review status, conflict reasons, and suggested actions. Observable data can also be used for model optimization. For example, when a large number of candidate knowledge is mistakenly rejected in the initial screening, the extraction or initial screening prompts can be optimized. When a large number of supplementary types are misjudged as new, the positioning of the formal knowledge base and the judgment of operation types can be optimized. When the conflict rate of the virtual knowledge base reaches a preset level, the consistency of the manual customer service statements can be checked.

[0078] In one embodiment, evidence events of candidate knowledge can be written into an immutable event table, and then a projector can calculate the current state of the candidate knowledge. The immutable event table refers to an event record table that only appends to and writes to, without modifying or deleting events. The projector refers to a processing component that reads evidence events in the immutable event table in chronological order and calculates the current state of the candidate knowledge accordingly. To simplify implementation, a count field and a history field can also be directly maintained in the virtual knowledge base table. Neither method affects the technical concept of evidence attribution and promotion gating in this invention. The system can also support offline replay: historical customer service conversations, human answers, and user feedback are input into the system in chronological order to reconstruct the candidate generation, evidence accumulation, and promotion process, which is used to evaluate the impact of different threshold configurations on the number of promotions, error rate, and conversion rate to human intervention.

[0079] Through the above steps, the method of this embodiment breaks down the self-evolution of intelligent customer service knowledge from a one-time knowledge import action into auditable steps such as candidate generation, candidate trial, evidence attribution, gating promotion, and exception rollback. Knowledge generated after human customer service solves long-tail problems can be continuously utilized by the system, rather than remaining only in work orders or chat logs. Candidate knowledge is verified in a virtual knowledge base before being promoted to the formal knowledge base, which improves the knowledge coverage and reduces the risk of unverified content being directly added to the database. This embodiment can also distinguish between three different types of problems: knowledge gaps, incomplete knowledge, and outdated knowledge. Additions, supplements, and replacements are solidified before candidate knowledge enters the virtual knowledge base, ensuring that subsequent verification and promotion revolve around clear operational goals and avoiding result drift caused by re-judgment during the promotion stage. Through hit attribution after the conversation ends, retrieved or exposed candidate knowledge is not directly considered valid, reducing the risk of weak... The probability of relevant candidate knowledge being incorrectly promoted; traditional solutions rely on manual periodic organization of work orders and knowledge bases, which is costly and lagging. This embodiment transforms human service feedback into candidate knowledge and automatically collects evidence through the service process, making knowledge maintenance change from periodic manual organization to continuous and controlled evolution; the source, judgment, merging, evidence, review, promotion and rollback of each candidate knowledge can be tracked, enabling enterprises to explain why a certain piece of formal knowledge was generated, when it was updated, what service evidence supports it and how to restore it. Through promotion history and rollback mechanism, this embodiment allows the system to actively absorb candidate knowledge and provide a recovery path for incorrect promotion. This is different from the solution of directly writing human customer service answers into the formal knowledge base, and also different from the solution of importing knowledge only once through manual review. Thus, it continuously absorbs new knowledge generated in human services without sacrificing the credibility of the formal knowledge base.

[0080] This embodiment addresses three types of knowledge deficiency scenarios that exist in enterprise customer service intelligent agents during actual service: First, the formal knowledge base does not cover the user's question, meaning that the content of the user's inquiry does not have corresponding knowledge in the formal knowledge base; second, the answers in the formal knowledge base are incomplete, meaning that existing formal answers lack conditions, steps, costs, time, restrictions, or exceptions; and third, the answers in the formal knowledge base are outdated, meaning that formal answers are incorrect or invalid due to changes in business rules. To address these scenarios, the method in this embodiment converts the human answers generated by human customer service representatives after the user is transferred to human service into candidate knowledge. Before the candidate knowledge is formally added to the formal knowledge base, it undergoes controlled trial use, evidence accumulation, review blocking, promotion gate control, promotion recording, and exception rollback through a virtual knowledge base. This ensures that the human answers are not directly written into the formal knowledge base, but are only promoted after being verified through real service.

[0081] In this embodiment, the formal knowledge base refers to the set of knowledge that the intelligent customer service system allows to be directly used as a source of credible facts; the virtual knowledge base refers to the candidate space located before the formal knowledge base, which stores candidate knowledge to be verified and records the status of evidence. It is neither a temporary cache nor a copy of the formal knowledge base, but an intermediate governance layer that undertakes the functions of candidate verification and evidence accumulation; the promotion history database refers to the audit storage that stores snapshots before and after candidate knowledge enters the formal knowledge base from the virtual knowledge base, promotion decisions, write results, and rollback status, which is used to support the tracking and revocation of each formal knowledge update.

[0082] It should be noted that the method in this embodiment is not limited to the large language model used for candidate extraction, semantic judgment, or answer generation, nor is it limited to the physical storage type of the formal knowledge base, virtual knowledge base, or promotion history database. These three can be physically deployed in separate databases, or they can be distinguished within the same database through status fields and table structures. The technical concept of the method in this embodiment emphasizes the lifecycle governance of candidate knowledge, the solidification of operation types, service process verification, and rollbackable promotion. As long as the system retains the above governance processes, it falls within the protection concept of the method in this embodiment, without relying on a specific model, database, or programming framework.

[0083] The aforementioned customer service knowledge self-evolution method based on human feedback and candidate verification extracts candidate knowledge in a structured manner from human service conversations, transforming scattered human answers into governable knowledge objects. Before entering the virtual knowledge base, the formal knowledge base is searched, and the operation type is fixed as add, supplement, replace, or no operation, ensuring a clear intention to write to the knowledge base from its creation. Subsequently, in the intelligent customer service, candidate knowledge is exposed and used in a controlled manner based on operation type, review status, evidence status, and formal knowledge hit results, ensuring that long-tail questions receive controlled answers outside the formal knowledge base, and that unverified content cannot cover formal answers. After the conversation ends, evidence events are generated based on user feedback, whether the request is transferred to a human agent, and the human agent's answer. The formal knowledge base is only updated according to the fixed operation type when candidate knowledge meets the promotion gating conditions, preventing erroneous knowledge from directly polluting the formal knowledge base. Finally, promotion history is recorded to support auditing, and knowledge is deleted or restored according to the operation type when erroneous promotions occur. This improves the coverage of long-tail questions and the efficiency of knowledge reuse while ensuring the auditability of knowledge base governance.

[0084] Figure 8 This is a schematic block diagram of a customer service knowledge self-evolution system 300 based on human feedback and candidate verification, provided in an embodiment of the present invention. Figure 8 As shown, corresponding to the above-described customer service knowledge self-evolution method based on manual feedback and candidate verification, the present invention also provides a customer service knowledge self-evolution system 300 based on manual feedback and candidate verification. This customer service knowledge self-evolution system 300 includes a unit for executing the above-described customer service knowledge self-evolution method based on manual feedback and candidate verification, and the system can be configured in a server. Specifically, please refer to... Figure 8 The customer service knowledge self-evolution system 300 based on human feedback and candidate verification includes a candidate knowledge collection unit 301, a preliminary value judgment unit 302, an operation type solidification unit 303, a virtual knowledge base management unit 304, a controlled service unit 305, a hit attribution unit 306, and a promotion governance unit 307.

[0085] The candidate knowledge acquisition unit 301 is used to collect the human service conversation between the user and the human customer service representative, and extract at least one candidate knowledge from the human service conversation; the preliminary value judgment unit 302 is used to perform a preliminary value judgment on the candidate knowledge; the operation type solidification unit 303 is used to search the formal knowledge base before the candidate knowledge enters the virtual knowledge base, and solidify the operation type of the candidate knowledge relative to the formal knowledge base as addition, supplementation, replacement or no operation according to the knowledge intent relationship and answer relationship between the candidate knowledge and the formal knowledge; the virtual knowledge base management unit 304 is used to write the candidate knowledge into the virtual knowledge base when the operation type is addition, supplementation or replacement, and save the operation type and target. The system includes: a formal knowledge identifier, a suggested answer, an audit status, and an evidence status; a controlled service unit 305, used to jointly retrieve the formal knowledge base and the virtual knowledge base during the intelligent customer service process, and to perform controlled exposure and controlled use of candidate knowledge in the virtual knowledge base based on the operation type, the audit status, the evidence status, and the formal knowledge hit result; a hit attribution unit 306, used to generate evidence events for the candidate knowledge after the session ends based on user feedback, whether the request is transferred to a human agent, and the relationship between the human agent's answer and the candidate answer; and a promotion governance unit 307, used to update the formal knowledge base according to its fixed operation type when the candidate knowledge meets the promotion gating conditions, and to record the promotion history to support auditing and rollback.

[0086] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned customer service knowledge self-evolution system 300 based on human feedback and candidate verification and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0087] The aforementioned customer service knowledge self-evolution system 300 based on human feedback and candidate verification can be implemented as a computer program, which can be used in various ways, such as... Figure 9 It runs on the computer device shown.

[0088] Please see Figure 9 , Figure 9 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0089] See Figure 9 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0090] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a self-evolving customer service knowledge method based on human feedback and candidate verification.

[0091] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0092] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a customer service knowledge self-evolution method based on human feedback and candidate verification.

[0093] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0094] The processor 502 is used to run the computer program 5032 stored in the memory to implement all the steps of the customer service knowledge self-evolution method based on human feedback and candidate verification.

[0095] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0096] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0097] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform all steps of the self-evolving customer service knowledge method based on human feedback and candidate verification.

[0098] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0099] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0100] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of each unit is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0101] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the system of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0102] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0103] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A self-evolving customer service knowledge method based on human feedback and candidate verification, characterized in that, include: Collect human customer service conversations between users and human customer service representatives, and extract at least one candidate knowledge from the human customer service conversations; A preliminary value judgment is made on the candidate knowledge; Before the candidate knowledge enters the virtual knowledge base, the formal knowledge base is searched, and the operation type of the candidate knowledge relative to the formal knowledge base is fixed as addition, supplementation, replacement or no operation based on the knowledge intent relationship and answer relationship between the candidate knowledge and the formal knowledge. When the operation type is addition, supplementation or replacement, the candidate knowledge is written into the virtual knowledge base, and the operation type, target formal knowledge identifier, suggested answer, review status and evidence status are saved. During the intelligent customer service process, the formal knowledge base and the virtual knowledge base are jointly retrieved, and the candidate knowledge in the virtual knowledge base is exposed and used in a controlled manner according to the operation type, the review status, the evidence status and the formal knowledge hit result. After the session ends, evidence events for the candidate knowledge are generated based on user feedback, whether the session was transferred to a human agent, and the relationship between the human agent's answer and the candidate answers. When the candidate knowledge meets the promotion gating conditions, the formal knowledge base is updated according to its fixed operation type, and the promotion history is recorded to support auditing and rollback.

2. The customer service knowledge self-evolution method based on human feedback and candidate verification according to claim 1, characterized in that, The candidate knowledge includes core questions, candidate answers, a set of knowledge points, a conversation source identifier, a scenario identifier, and context fields used to describe the scope of application of the candidate knowledge; When a human service session contains multiple independent knowledge points, the multiple independent knowledge points are split into multiple candidate knowledge points and enter the subsequent processing flow respectively; when extracting the candidate knowledge, the conversational expressions, greetings and user case information such as user name, order number, contact information or address are removed from the human service session, and only the reusable question intent and reusable answer are retained.

3. The customer service knowledge self-evolution method based on human feedback and candidate verification according to claim 1, characterized in that, The preliminary value judgment of the candidate knowledge includes: The candidate knowledge is assessed based on four dimensions: answer completeness, reusability, knowledge value, and risk of human intervention. The risk of human intervention is used to identify candidate knowledge involving complaints, legal consultations, medical advice, user privacy, order cases, requiring human verification, or unsuitable for automatic answers. The result of the preliminary value assessment is used to determine whether the candidate knowledge is included in the step of searching the formal knowledge base. The preliminary value assessment is not a formal entry assessment. Candidate knowledge that passes the preliminary value assessment still needs to be written into the virtual knowledge base for verification.

4. The customer service knowledge self-evolution method based on manual feedback and candidate verification according to claim 1, characterized in that, The formal knowledge base for retrieval includes: Construct a multi-way semantic query, which includes at least a standard question query based on the core question, a key fact query for the answer based on the knowledge point set, an answer summary query based on the candidate answer, and an answer reverse query based on the candidate answer to construct rule-based questions, cause-based questions, entry-level questions, or time-sensitive questions. The formal knowledge retrieved by the multi-way semantic query is deduplicated and merged according to the formal knowledge identifier, and the information of each formal knowledge being retrieved by the query route, its ranking position in each query route, and its similarity information are retained. Then, a fusion sorting method is used to determine the candidate formal knowledge set for the solidification of the operation type based on the ranking position and similarity information in each query route.

5. The customer service knowledge self-evolution method based on human feedback and candidate verification according to claim 1, characterized in that, The operation type solidification includes: When the formal knowledge base does not contain the same knowledge intent, the candidate knowledge is solidified as new; when the formal knowledge base contains the same knowledge intent and the formal answer lacks conditions, steps, costs, time, restrictions or exceptions, the candidate knowledge is solidified as supplementary and bound to the target formal knowledge. When the formal knowledge base has the same knowledge intent and the formal answer is incorrect, outdated, or conflicts with the candidate answer, the candidate knowledge is solidified to replace and bind the target formal knowledge; When the formal knowledge base has fully covered the candidate knowledge, the candidate knowledge will be fixed as no operation. When the candidate knowledge and the formal knowledge belong to the same business domain but are not the same knowledge intent, the candidate knowledge is solidified as a new addition and a related parent knowledge identifier is recorded. The related parent knowledge identifier is used to indicate the business affiliation of the candidate knowledge rather than to indicate a supplement to the target formal knowledge. Among them, candidate knowledge of the replacement type is set by default to require manual review. It cannot be used as the source of facts for automatic answers or to perform formal knowledge base updates before the review status is approved.

6. The customer service knowledge self-evolution method based on human feedback and candidate verification according to claim 1, characterized in that, Before writing the candidate knowledge into the virtual knowledge base, existing candidate knowledge is recalled in the virtual knowledge base, and it is determined whether to merge or conflict only within the candidate set with the same operation type and the same operation scope. The same operation scope for supplementary or replacement types means that the target formal knowledge identifier is the same, and the same operation scope for new types means that neither is bound to the target formal knowledge identifier. The knowledge slot refers to a knowledge unit with the same user intent, business object, and answer constraints; when a new candidate knowledge and an existing candidate knowledge belong to the same knowledge slot and have the same, inclusive, or complementary answers, the new candidate knowledge is merged into the evidence event of the existing candidate knowledge. When answers belong to the same knowledge slot but conflict in terms of numerical value, support status, applicable conditions, or restriction rules, record the evidence of conflict and adjust the review status to require manual review. When knowledge does not belong to the same knowledge slot, create a new candidate knowledge.

7. The customer service knowledge self-evolution method based on human feedback and candidate verification according to claim 1, characterized in that, The controlled exposure and controlled use of candidate knowledge in the virtual knowledge base includes: When the formal knowledge base has a reliable hit, the formal knowledge is used as the main source of the answer. Supplementary candidate knowledge only participates in the answer when the target formal knowledge it is bound to is the currently reliable hit knowledge and the candidate evidence score reaches the preset score threshold. Newly added candidate knowledge does not participate in the answer. When there is no reliable match in the formal knowledge base, newly added candidate knowledge with a candidate evidence score that reaches a preset score threshold and whose review status is "no review required" or "review passed" is allowed to participate in automatic answering; candidate knowledge of replacement type, candidate knowledge under review, candidate knowledge that has been rejected, candidate knowledge with a candidate evidence score that has not reached the preset score threshold, and candidate knowledge with conflicting evidence shall not be used as the source of facts for automatic answering. The service trajectory is recorded during the service process. The service trajectory includes user questions, intelligent customer service answers, retrieved formal knowledge identifiers, retrieved candidate knowledge identifiers, candidate knowledge identifiers exposed to the answer generator, candidate knowledge identifiers actually used to answer, answer routes, and conversation segment identifiers. Among them, being retrieved, being exposed, and actually used to answer are recorded as different process evidence records. The process evidence is not directly equivalent to valid hit evidence. The reliable hit means that the similarity between the retrieved formal knowledge and the user question reaches a preset similarity threshold and the knowledge intent is consistent.

8. The customer service knowledge self-evolution method based on human feedback and candidate verification according to claim 1, characterized in that, After the session ends, based on user feedback, whether a human intervention was initiated, and the relationship between the human answer and the candidate answers, evidence events for the candidate knowledge are generated, including: When a user reports that the issue has been resolved or their satisfaction level has reached a preset satisfaction threshold, valid hit evidence is generated for the candidate knowledge actually used to answer the question. If the issue remains unresolved after using candidate knowledge and is still transferred to a human or user feedback, negative hit evidence is generated; if the human answer after being transferred to a human is compatible with the candidate answer, human confirmation evidence is generated. When the human answer after being transferred to a human examiner conflicts with the candidate answer in terms of numerical value, whether it is supported, applicable conditions, or restrictive rules, conflict evidence is generated; for candidate knowledge that is not actually used to answer, no valid hit evidence is generated.

9. The customer service knowledge self-evolution method based on manual feedback and candidate verification according to claim 1, characterized in that, The promotion gating conditions include: the operation type is addition, supplementation or replacement; the candidate evidence score reaches a preset scoring threshold; the number of effective uses reaches a preset number of uses threshold; the average satisfaction reaches a preset satisfaction threshold; and the candidate knowledge that requires manual review has been approved. When the candidate knowledge meets the promotion gating conditions, the formal knowledge base is updated according to its fixed operation type, and the promotion history is recorded to support auditing and rollback, including: For newly added candidate knowledge, create new formal knowledge questions and formal knowledge answers; for supplementary candidate knowledge, keep the target formal knowledge question unchanged, and update the target formal knowledge answer with the supplementary suggestion written into the answer. For candidate knowledge of the replacement type, the target formal knowledge question remains unchanged after the review is approved; Update the target's formal knowledge answer with the replaced suggestion in the answer; The promotion history includes candidate knowledge snapshots, promotion gating decisions, formal knowledge base write results, promotion time, rollback status, and rollback results. For supplementary or replacement promotions... The promotion history also includes a snapshot of the target formal knowledge before promotion and a snapshot of the target formal knowledge after promotion; when a rollback occurs, for new types of promotions, the formal knowledge created during the promotion is deleted or archived, and for supplementary or replacement types of promotions, the target formal knowledge answers before promotion are restored.

10. A customer service knowledge self-evolution system based on human feedback and candidate verification, characterized in that: include: The candidate knowledge acquisition unit is used to collect the human service conversation between the user and the human customer service representative, and extract at least one candidate knowledge from the human service conversation. A preliminary value judgment unit is used to make a preliminary value judgment on the candidate knowledge; The operation type solidification unit is used to retrieve the formal knowledge base before the candidate knowledge enters the virtual knowledge base, and solidify the operation type of the candidate knowledge relative to the formal knowledge base as addition, supplementation, replacement or no operation according to the knowledge intent relationship and answer relationship between the candidate knowledge and the formal knowledge. The virtual knowledge base management unit is used to write the candidate knowledge into the virtual knowledge base when the operation type is to add, supplement, or replace, and to save the operation type, target formal knowledge identifier, suggested answer, review status, and evidence status. The controlled service unit is used to jointly retrieve the formal knowledge base and the virtual knowledge base during the intelligent customer service process, and to expose and use candidate knowledge in the virtual knowledge base in a controlled manner according to the operation type, the review status, the evidence status and the formal knowledge hit result. The hit attribution unit is used to generate evidence events for the candidate knowledge after the session ends, based on user feedback, whether the request was transferred to a human agent, and the relationship between the human agent's answer and the candidate answer. The promotion governance unit is used to update the formal knowledge base according to its fixed operation type when the candidate knowledge meets the promotion gating conditions, and to record the promotion history to support auditing and rollback.

Images