Heterogeneous knowledge question and answer model evaluation method and system based on ICT customer service

Abstract

The invention discloses a heterogeneous knowledge question-answer model evaluation method and system based on ICT customer service, and relates to the technical field of natural language processing and artificial intelligence, and the method comprises the steps: collecting user input records, and generating a standardized user query text; identifying a context aware entity network based on the standardized user query text; generating a context update knowledge graph according to the context aware entity network; constructing a semantic contrast vector pair based on the context updating knowledge graph; evaluating semantic consistency and accuracy of the answers according to the semantic comparison vector pairs, and confirming final answers; the semantic contrast vector pair is optimized based on the final answer. By collecting the user input record and generating the standardized user query text, unified processing and quality evaluation of user input are realized, data consistency and accuracy of subsequent processing steps are ensured, efficiency and quality of user input processing are improved, and robustness and user experience are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing and artificial intelligence technology, and in particular to an evaluation method and system for a heterogeneous knowledge question-answering model based on ICT customer service. Background Art

[0002] In recent years, the application of information and communication technology (ICT) in the field of customer service has made significant progress. Traditional customer service systems rely on preset rule bases and keyword matching, which perform well in dealing with simple problems, but show limitations when facing increasingly complex and diverse customer needs. To this end, researchers have introduced advanced technologies such as natural language processing (NLP), machine learning (ML), and knowledge graphs to explore more intelligent and flexible solutions. Among them, knowledge graphs, as structured semantic networks, effectively represent entities and their relationships, enhance the understanding of complex problems and the accuracy of answers; while deep learning models such as BERT and its variants have demonstrated excellent performance in named entity recognition and context understanding, promoting the development of intelligent customer service. In addition, multimodal data fusion technology enables the system to handle multiple forms of user interaction such as text, voice, and images, significantly improving the user experience.

[0003] Although the above technologies have achieved remarkable achievements in their respective fields, there are still some key deficiencies in existing technologies. First, most traditional question-answering systems use static knowledge bases or rule sets, which are difficult to adapt to rapidly changing business environments and user needs. When faced with new concepts or situations, the system is often unable to update its internal model in a timely manner, resulting in a significant reduction in the accuracy and timeliness of the answers. Secondly, existing evaluation methods mostly focus on surface-level correctness judgments and lack in-depth consideration of semantic consistency and contextual coherence, which to some extent limits the system's ability to understand complex queries. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a heterogeneous knowledge question-answering model evaluation method based on ICT customer service to solve the problems of delayed knowledge base update and insufficient semantic consistency evaluation in the prior art.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a heterogeneous knowledge question and answer model evaluation method based on ICT customer service, which includes collecting user input records to generate standardized user query texts; identifying context-aware entity networks based on the standardized user query texts; generating context-updated knowledge graphs based on the context-aware entity networks; constructing semantic comparison vector pairs based on the context-updated knowledge graphs; evaluating the semantic consistency and accuracy of answers based on the semantic comparison vector pairs, and confirming the final answers; and optimizing the semantic comparison vector pairs based on the final answers.

[0008] As a preferred solution of the heterogeneous knowledge question-answering model evaluation method based on ICT customer service described in the present invention, wherein: the step of collecting user input records to generate standardized user query texts is as follows:

[0009] Choose HTTP-based RESTful API as a unified interface to receive all types of user input, and convert data from different channels into the internal processing standard JSON format;

[0010] Match and remove extra spaces in the string, replace irregular punctuation marks with standard ones, and convert all characters to lowercase.

[0011] Perform word segmentation on the text, perform part-of-speech tagging on the word segmentation results, and use the BERT model for named entity recognition;

[0012] The comprehensive scoring function S is introduced to generate the score of the standardized user query text. The expression is:

[0013]

[0014] Among them, S is the standardized user query text score, P w is the word segmentation accuracy, that is, the ratio of the number of correct word segmentations to the total number of words, P p is the accuracy of part-of-speech tagging, that is, the ratio of the number of correctly tagged words to the total number of words, P e is the named entity recognition accuracy, that is, the ratio of the number of correctly recognized entities to the total number of entities, t is the time difference from user input to completion of standardization processing, and λ is the attenuation coefficient of the effect of time on the score;

[0015] When receiving user input, immediately record the current timestamp in the ISO 8601 format, and calculate and record the standardized user query text score;

[0016] Define a user query text threshold I based on the normalized user query text score;

[0017] For texts with a standardized user query text score greater than I, the standardized user query text is directly output. For texts not greater than I, feedback is provided to the user and the user is requested to re-enter the text, and a source channel identifier is added to each input record.

[0018] As a preferred solution of the heterogeneous knowledge question-answering model evaluation method based on ICT customer service described in the present invention, wherein: the context-aware entity network is identified based on standardized user query text, and the specific steps are as follows:

[0019] Select CRF as a tool for named entity recognition, preprocess the standardized user query text, use the preprocessed standardized user query text to train the CRF model, apply the trained CRF model to the new user query text, perform named entity recognition, and extract the entities and their associated nodes in the text;

[0020] For each identified entity, calculate its precision P, expressed as:

[0021]

[0022] Where P is the ratio of the number of correctly identified entities to the total number of identified entities, TP is the number of correctly identified entities, and FP is the number of incorrectly identified entities;

[0023] Calculate the recall rate R, the expression is:

[0024]

[0025] Among them, R is the ratio of the number of correctly identified entities to the total number of entities that actually exist, TP is the number of correctly identified entities, and FN is the number of unidentified entities;

[0026] Calculate the F1 score, the expression is:

[0027]

[0028] Among them, F1 is the harmonic mean of the precision and recall of entity recognition, P is the ratio of the number of correctly recognized entities to the total number of recognized entities, and R is the ratio of the number of correctly recognized entities to the total number of entities that actually exist;

[0029] Introduce the comprehensive evaluation function Q, the expression is:

[0030]

[0031] Among them, Q is a scoring indicator reflecting the accuracy of entity recognition and the initial processing speed, F1 is the harmonic mean of the precision and recall of entity recognition, C is the logical consistency score of the entity in the network, λ is the attenuation coefficient of the influence of time on the score, and τ is the time difference from user input to completion of recognition;

[0032] According to the evaluated entities and their associated nodes, a directed weighted graph G is constructed, which is expressed as:

[0033] G = (V, E, w);

[0034] Where G is a directed weighted graph, V is a set of nodes, each node represents an entity, E is a set of edges, and w is the weight of the edge;

[0035] Check the logical consistency between each node and its adjacent nodes. If any unreasonable connections are found during the consistency check, make adjustments, introduce constraints, and optimize the structure of the directed weighted graph G.

[0036] The optimization graph quality threshold A is defined based on the scoring indicators of entity recognition accuracy and initial processing speed;

[0037] Determine whether the optimized directed weighted graph G satisfies the value of the quality assessment function Q and is not less than A. If so, output the context-aware entity network, otherwise further adjust and optimize.

[0038] As a preferred solution of the heterogeneous knowledge question-answering model evaluation method based on ICT customer service described in the present invention, wherein: the context-updated knowledge graph is generated according to the context-aware entity network, and the specific steps are as follows:

[0039] Based on the newly identified entities and relationships in the context-aware entity network, the relevant entities and associated nodes in the knowledge graph that need to be updated are screened out through context consistency checks. For the selected entities and associated nodes, GraphSAGE is selected as the graph embedding model, and the relevant entities and associated nodes are extracted from the existing knowledge graph. The new embedding vector of each node is calculated using the GraphSAGE model, and incremental updates are performed using database transaction management.

[0040] Introduce update quality score function Q ′ , the expression is:

[0041]

[0042] Where Q′ is the update quality score, F1 is the harmonic mean of the precision and recall of entity recognition, C is the logical consistency score of the entity in the network, τ is the time difference from user input to completion of recognition, λ is the attenuation coefficient of the impact of time on the score, α is the weight factor of the embedding vector similarity, and v iis the embedding vector of node i before updating, v′ i is the embedding vector of node i after update, n is the number of nodes involved in the update, cos(v i ,v′ i ) is the cosine similarity between the embedding vectors of node i before and after update;

[0043] Introduce the Git version control system to record the relevant entities, associated nodes and timestamps of each update, and calculate the update quality score after the update is completed;

[0044] defining an update quality threshold H based on the update quality score;

[0045] For updates with an update quality score less than H, analyze the problem and make adjustments, otherwise confirm that the update is successful and generate a contextual update knowledge graph.

[0046] As a preferred solution of the heterogeneous knowledge question-answering model evaluation method based on ICT customer service described in the present invention, wherein: the semantic comparison vector pair is constructed based on the context-based updated knowledge graph, and the specific steps are as follows: the user's query question and the entities, attributes, relationships and context information in the knowledge graph are used as inputs, and a Seq2Seq model is used for training and reasoning to generate multiple candidate answers;

[0047] Select the BERT model for text encoding, and extract the vector representation of the last layer output of BERT for the text snippets and entity descriptions related to each candidate answer in the knowledge base;

[0048] Introducing the semantic similarity value D, the expression is:

[0049]

[0050] Where D is the semantic similarity value, m is the number of vectors of candidate answers and knowledge base content, cos(a j ,b j ) is the cosine similarity between the jth pair of vectors, w j is the weight of the jth pair of vectors, β is the weight of the time decay factor, γ is the time decay coefficient, τ is the time difference from user input to completion of recognition, a j is the jth vector representation of the candidate answer, b j is the jth vector representation of the knowledge base content;

[0051] According to the similarity score D, the semantic comparison vector pair with the highest score is selected, and the optimal semantic comparison vector pair is output.

[0052] As a preferred solution of the heterogeneous knowledge question-answering model evaluation method based on ICT customer service described in the present invention, wherein: the semantic consistency and accuracy of the evaluation answer based on the semantic comparison vector and the confirmation of the final answer are specifically performed as follows:

[0053] Use contextual updated knowledge graph queries, domain expert annotations, historical interaction data, and user feedback to obtain standard answers;

[0054] The semantic similarity value D is used to calculate the similarity between the candidate answer vector and the standard answer vector, and the candidate answer with the highest score is selected as the final answer. An automated assisted manual review mechanism is introduced to output the final answer.

[0055] As a preferred solution of the heterogeneous knowledge question-answering model evaluation method based on ICT customer service described in the present invention, wherein: the semantic comparison vector pair is optimized based on the final answer, and the specific steps are as follows:

[0056] Set up a clear five-star rating mechanism on the user interaction interface, collect users' ratings of the final answer, and record the data of users' click behavior, dwell time and number of repeated queries, and analyze user ratings, user click behavior scores, user dwell time scores and number of repeated queries;

[0057] The expression of user click behavior score B is:

[0058]

[0059] Among them, B is the user's click behavior score, n c is the total number of click events, θ is the weight of the φth click, and φ is the sequence number of the click event;

[0060] User stay time score S d The expression is:

[0061]

[0062] Among them, S d is the user's residence time score, and η is the user's actual residence time;

[0063] Introducing comprehensive user feedback scoring function F u , the expression is:

[0064]

[0065] Among them, F u is the comprehensive user feedback score, r k Rating for the kth user, t k The timestamp of the k-th user rating, β tis the time weight factor, B is the user click behavior score, L is the total number of clicks, S d is the user's residence time score, Z is the number of repeated queries, μ is the behavior data weight factor, and n r is the total number of ratings;

[0066] Based on the overall feedback score F u Dynamically adjust the parameters and feature weights of the semantic similarity value D and introduce the user feedback enhanced semantic similarity value D ′ (q,y), the expression is:

[0067] D ′ (q,y)=D+x·F u (q,y);

[0068] Among them, D ′ (q,y) is the optimized semantic similarity value, D is the semantic similarity value, F u (q,y) is the comprehensive user feedback score of user query q and candidate answers, x is the user feedback influence factor, q is the user query, and y is the candidate answer.

[0069] In the second aspect, the present invention provides a heterogeneous knowledge question and answer model evaluation system based on ICT customer service, including a user input processing module, an entity recognition module, a knowledge graph update module, a semantic comparison module, an answer evaluation module and an optimization configuration module; the user input processing module is used to receive user queries and generate standardized user query texts; the entity recognition module is used to use standardized user query texts to identify entities and their associated nodes, and construct a context-aware entity network; the knowledge graph update module is used to generate a context-updated knowledge graph based on context-aware entities; the semantic comparison module is used to construct semantic comparison vector pairs based on the context-updated knowledge graph; the answer evaluation module is used to evaluate the semantic consistency and accuracy of the answer, and confirm the final answer; the optimization configuration module is used to optimize the configuration based on the feedback of the final answer.

[0070] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the heterogeneous knowledge question and answer model evaluation method based on ICT customer service as described in the first aspect of the present invention is implemented.

[0071] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the heterogeneous knowledge question and answer model evaluation method based on ICT customer service as described in the first aspect of the present invention.

[0072] The beneficial effects of the present invention are as follows: by collecting user input records and generating standardized user query texts, unified processing and quality assessment of user inputs are achieved, data consistency and accuracy of subsequent processing steps are ensured, the efficiency and quality of user input processing are improved, and robustness and user experience are enhanced. By identifying context-aware entity networks based on standardized user query texts, accurate entity recognition and relationship construction are achieved, the semantic understanding and response capabilities of the question-answering system are improved, the accuracy and logical consistency of entity recognition are enhanced, user intent is understood more accurately, and answers that are more in line with actual scenarios are provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0074] Figure 1 This is a flow chart of the heterogeneous knowledge question and answer model evaluation method based on ICT customer service in Example 1.

[0075] Figure 2 This is a schematic diagram of the heterogeneous knowledge question and answer model evaluation system based on ICT customer service in Example 1. DETAILED DESCRIPTION

[0076] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0077] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0078] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0079] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and provides a heterogeneous knowledge question answering model evaluation method based on ICT customer service, comprising the following steps:

[0080] S1: Collect user input records to generate standardized user query text;

[0081] Choose HTTP-based RESTful API as a unified interface to receive all types of user input, and convert data from different channels into the internal processing standard JSON format;

[0082] Match and remove extra spaces in the string, replace irregular punctuation marks with standard ones, and convert all characters to lowercase.

[0083] Perform word segmentation on the text, perform part-of-speech tagging on the word segmentation results, and use the BERT model for named entity recognition;

[0084] The comprehensive scoring function S is introduced to generate the score of the standardized user query text. The expression is:

[0085]

[0086] Among them, S is the standardized user query text score, P w is the word segmentation accuracy, that is, the ratio of the number of correct word segmentations to the total number of words, P p is the accuracy of part-of-speech tagging, that is, the ratio of the number of correctly tagged words to the total number of words, P e is the named entity recognition accuracy, that is, the ratio of the number of correctly recognized entities to the total number of entities, t is the time difference from user input to completion of standardization processing, and λ is the attenuation coefficient of the effect of time on the score;

[0087] When receiving user input, immediately record the current timestamp in the ISO 8601 format, and calculate and record the standardized user query text score;

[0088] Define a user query text threshold I based on the normalized user query text score;

[0089] For texts with a standardized user query text score greater than I, the standardized user query text is directly output. For texts not greater than I, feedback is provided to the user and re-entry is requested. A source channel identifier is added to each input record.

[0090] It should also be noted that: regular expressions are used to remove redundant spaces in the text, including the beginning and end of a line, and multiple consecutive spaces; non-standard punctuation marks (such as full-width symbols and special symbols) are uniformly replaced with standard formats (such as half-width symbols) to ensure the consistency of all punctuation marks; mature word segmentation tools (such as Jieba and spaCy) are used to segment the cleaned text, divide the text into independent vocabulary units, perform part-of-speech tagging on the word segmentation results, and assign a suitable part-of-speech tag (such as nouns, verbs, and adjectives) to each vocabulary; the user query text threshold I is set to 0.8. For application scenarios with extremely high accuracy requirements, the threshold can be increased to 0.9, and for some application scenarios that are more sensitive to speed and response time, the threshold can be appropriately lowered to 0.7.

[0091] S2: Identifying context-aware entity networks based on standardized user query text;

[0092] Select CRF as a tool for named entity recognition, preprocess the standardized user query text, use the preprocessed standardized user query text to train the CRF model, apply the trained CRF model to the new user query text, perform named entity recognition, and extract the entities and their associated nodes in the text;

[0093] For each identified entity, calculate its precision P, expressed as:

[0094]

[0095] Where P is the ratio of the number of correctly identified entities to the total number of identified entities, TP is the number of correctly identified entities, and FP is the number of incorrectly identified entities;

[0096] Calculate the recall rate R, the expression is:

[0097]

[0098] Among them, R is the ratio of the number of correctly identified entities to the total number of entities that actually exist, TP is the number of correctly identified entities, and FN is the number of unidentified entities;

[0099] Calculate the F1 score, the expression is:

[0100]

[0101] Among them, F1 is the harmonic mean of the precision and recall of entity recognition, P is the ratio of the number of correctly recognized entities to the total number of recognized entities, and R is the ratio of the number of correctly recognized entities to the total number of entities that actually exist;

[0102] Introduce the comprehensive evaluation function Q, the expression is:

[0103]

[0104] Among them, Q is a scoring indicator reflecting the accuracy of entity recognition and the initial processing speed, F1 is the harmonic mean of the precision and recall of entity recognition, C is the logical consistency score of the entity in the network, λ is the attenuation coefficient of the influence of time on the score, and τ is the time difference from user input to completion of recognition;

[0105] According to the evaluated entities and their associated nodes, a directed weighted graph G is constructed, which is expressed as:

[0106] G = (V, E, w);

[0107] Where G is a directed weighted graph, V is a set of nodes, each node represents an entity, E is a set of edges, and w is the weight of the edge;

[0108] Check the logical consistency between each node and its adjacent nodes. If any unreasonable connections are found during the consistency check, make adjustments, introduce constraints, and optimize the structure of the directed weighted graph G.

[0109] The optimization graph quality threshold A is defined based on the scoring indicators of entity recognition accuracy and initial processing speed;

[0110] Determine whether the optimized directed weighted graph G satisfies the value of the quality assessment function Q and is not less than A. If so, output the context-aware entity network, otherwise further adjust and optimize.

[0111] It should also be noted that the BERT model is used to preprocess the standardized user query text, convert it into the token sequence format required by BERT, and obtain the context representation vector of each subword unit through the encoding layer of the BERT model. This process ensures the richness and context sensitivity of text features and improves the accuracy of subsequent named entity recognition; additional constraints are introduced, including path length restrictions, node degree restrictions, and context relevance verification; the value of A is set to 0.8. For application scenarios with extremely high accuracy requirements, the threshold can be increased to 0.9, and for some application scenarios that are more sensitive to speed and response time, the threshold can be appropriately reduced to 0.7.

[0112] S3: Generate context-updated knowledge graph based on context-aware entity network;

[0113] Based on the newly identified entities and relationships in the context-aware entity network, the relevant entities and associated nodes in the knowledge graph that need to be updated are screened out through context consistency checks. For the selected entities and associated nodes, GraphSAGE is selected as the graph embedding model, and the relevant entities and associated nodes are extracted from the existing knowledge graph. The new embedding vector of each node is calculated using the GraphSAGE model, and incremental updates are performed using database transaction management.

[0114] Introduce update quality score function Q ′ , the expression is:

[0115]

[0116] Among them, Q ′ is the update quality score, F1 is the harmonic mean of the precision and recall of entity recognition, C is the logical consistency score of the entity in the network, τ is the time difference from user input to completion of recognition, λ is the attenuation coefficient of the impact of time on the score, α is the weight factor of the embedding vector similarity, and v i is the embedding vector of node i before updating, v i ′ is the embedding vector of node i after update, n is the number of nodes involved in the update, cos(v i ,v i ′ ) is the cosine similarity between the embedding vectors of node i before and after update;

[0117] Introduce the Git version control system to record the relevant entities, associated nodes and timestamps of each update, and calculate the update quality score after the update is completed;

[0118] defining an update quality threshold H based on the update quality score;

[0119] For updates with an update quality score less than H, analyze the problem and make adjustments, otherwise confirm that the update is successful and generate a contextual update knowledge graph.

[0120] It should also be noted that: context consistency checks are performed on newly identified entities and their relationships to ensure that these new contents are logically consistent with the information in the existing knowledge graph. This process can filter out inaccurate or illogical information and improve the quality of subsequent updates; for entities and their associated nodes that pass the consistency check, GraphSAGE is selected as the graph embedding model, which can learn the feature representation (i.e., embedding vector) of each node from the existing knowledge graph and calculate the updated embedding vector, and then use the database transaction management mechanism to perform incremental updates to ensure the atomicity and consistency of the update process; the update quality scoring function Q is used ′To evaluate the quality of each update; Use the Git version control system to record the relevant entities and associated nodes of each update, including the attributes, metadata and timestamps of the entities and nodes, which not only helps to track historical changes, but also facilitates rolling back to the previous version when update problems are found; Set the value of H to 0.8. For application scenarios with extremely high accuracy requirements, the threshold can be increased to 0.9, and for some application scenarios that are more sensitive to speed and response time, the threshold can be appropriately lowered to 0.7;

[0121] S4: Construct semantic contrast vector pairs based on context-based updated knowledge graph;

[0122] The user's query question and the entities, attributes, relationships, and context information in the knowledge graph are used as input, and the Seq2Seq model is trained and inferred to generate multiple candidate answers.

[0123] Select the BERT model for text encoding, and extract the vector representation of the last layer output of BERT for the text snippets and entity descriptions related to each candidate answer in the knowledge base;

[0124] Introducing the semantic similarity value D, the expression is:

[0125]

[0126] Where D is the semantic similarity value, m is the number of vectors of candidate answers and knowledge base content, cos(a j ,b j ) is the cosine similarity between the jth pair of vectors, w j is the weight of the jth pair of vectors, β is the weight of the time decay factor, γ is the time decay coefficient, τ is the time difference from user input to completion of recognition, a j is the jth vector representation of the candidate answer, b j is the jth vector representation of the knowledge base content;

[0127] According to the similarity score D, the semantic comparison vector pair with the highest score is selected, and the optimal semantic comparison vector pair is output.

[0128] It should also be noted that for the text fragments and entity descriptions related to each candidate answer in the knowledge base, the BERT model is used for text encoding, and the vector representation of the last layer output of the BERT model is extracted. These vectors can capture the contextual information of the text and provide feature representation for subsequent similarity calculations.

[0129] S5: Evaluate the semantic consistency and accuracy of the answer based on the semantic contrast vector and confirm the final answer;

[0130] Use contextual updated knowledge graph queries, domain expert annotations, historical interaction data, and user feedback to obtain standard answers;

[0131] The semantic similarity value D is used to calculate the similarity between the candidate answer vector and the standard answer vector, and the candidate answer with the highest score is selected as the final answer. An automated assisted manual review mechanism is introduced to output the final answer.

[0132] It should also be noted that: based on user input, standard answers are generated by using contextual updates to the knowledge graph query, domain expert annotations, historical interaction data, and user feedback;

[0133] The semantic similarity value D is used to calculate the similarity between the candidate answer vector and the standard answer vector, and the candidate answer with the highest score is selected as the final answer. For answers with similar scores and complex question types, they are automatically transferred to the manual review process to ensure the accuracy and rationality of the final answer.

[0134] S6: Optimize the semantic contrast vector pair based on the final answer;

[0135] Set up a clear five-star rating mechanism on the user interaction interface, collect users' ratings of the final answer, and record the data of users' click behavior, dwell time and number of repeated queries, and analyze user ratings, user click behavior scores, user dwell time scores and number of repeated queries;

[0136] The expression of user click behavior score B is:

[0137]

[0138] Among them, B is the user's click behavior score, n c is the total number of click events, θ is the weight of the φth click, and φ is the sequence number of the click event;

[0139] User stay time score S d The expression is:

[0140]

[0141] Among them, S d is the user's residence time score, and η is the user's actual residence time;

[0142] Introducing comprehensive user feedback scoring function F u , the expression is:

[0143]

[0144] Among them, F u is the comprehensive user feedback score, r k Rating for the kth user, t kThe timestamp of the k-th user rating, β t is the time weight factor, B is the user click behavior score, L is the total number of clicks, S d is the user's residence time score, Z is the number of repeated queries, μ is the behavior data weight factor, and n r is the total number of ratings;

[0145] Based on the overall feedback score F u Dynamically adjust the parameters and feature weights of the semantic similarity value D and introduce the user feedback enhanced semantic similarity value D ′ (q,y), the expression is:

[0146] D ′ (q,y)=D+x·F u (q,y);

[0147] Among them, D ′ (q,y) is the optimized semantic similarity value, D is the semantic similarity value, F u (q,y) is the comprehensive user feedback score of user query q and candidate answers, x is the user feedback influence factor, q is the user query, and y is the candidate answer.

[0148] It should also be noted that in the subsequent question-answering process, the user feedback enhanced semantic similarity value D ′ (q,y) selects and sorts candidate answers to ensure higher quality answers.

[0149] The present embodiment also provides a heterogeneous knowledge question and answer model evaluation system based on ICT customer service, including: a user input processing module, an entity recognition module, a knowledge graph update module, a semantic comparison module, an answer evaluation module and an optimization configuration module; the user input processing module is used to receive user queries and generate standardized user query texts; the entity recognition module is used to use standardized user query texts to identify entities and their associated nodes, and build a context-aware entity network; the knowledge graph update module is used to generate a context-updated knowledge graph based on context-aware entities; the semantic comparison module is used to build semantic comparison vector pairs based on the context-updated knowledge graph; the answer evaluation module is used to evaluate the semantic consistency and accuracy of the answer and confirm the final answer; the optimization configuration module is used to optimize the configuration based on the feedback of the final answer.

[0150] This embodiment also provides a computer device, which is suitable for the case of a heterogeneous knowledge question and answer model evaluation method based on ICT customer service, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the heterogeneous knowledge question and answer model evaluation method based on ICT customer service proposed in the above embodiment.

[0151] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0152] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the heterogeneous knowledge question and answer model evaluation method based on ICT customer service proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, disk or optical disk.

[0153] In summary, the present invention realizes unified processing and quality assessment of user input by: collecting user input records and generating standardized user query texts, ensuring data consistency and accuracy of subsequent processing steps, improving the efficiency and quality of user input processing, and enhancing robustness and user experience. By identifying context-aware entity networks based on standardized user query texts, accurate entity recognition and relationship construction are achieved, the semantic understanding and response capabilities of the question-answering system are improved, the accuracy and logical consistency of entity recognition are enhanced, user intent is understood more accurately, and answers that are more in line with actual scenarios are provided.

[0154] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A heterogeneous knowledge question answering model evaluation method based on ICT customer service, characterized by: include, Collect user input records to generate standardized user query text; Identify context-aware entity networks based on standardized user query text; Generate a context-updated knowledge graph based on the context-aware entity network; Based on the context-based updated knowledge graph, semantic contrast vector pairs are constructed; Evaluate the semantic consistency and accuracy of the answers based on the semantic contrast vectors and confirm the final answers; Based on the final answer, optimize the semantic contrast vector pair.

2. The heterogeneous knowledge question answering model evaluation method based on ICT customer service as claimed in claim 1, characterized in that: The specific steps of collecting user input records to generate standardized user query text are as follows: Choose HTTP-based RESTful API as a unified interface to receive all types of user input, and convert data from different channels into the internal processing standard JSON format; Match and remove extra spaces in the string, replace irregular punctuation marks with standard ones, and convert all characters to lowercase. Perform word segmentation on the text, perform part-of-speech tagging on the word segmentation results, and use the BERT model for named entity recognition; The comprehensive scoring function S is introduced to generate the score of the standardized user query text. The expression is: Among them, S is the standardized user query text score, P w is the word segmentation accuracy, that is, the ratio of the number of correct word segmentations to the total number of words, P p is the accuracy of part-of-speech tagging, that is, the ratio of the number of correctly tagged words to the total number of words, P e is the named entity recognition accuracy, that is, the ratio of the number of correctly recognized entities to the total number of entities, t is the time difference from user input to completion of standardization processing, and λ is the attenuation coefficient of the effect of time on the score; When receiving user input, immediately record the current timestamp in the ISO 8601 format, and calculate and record the standardized user query text score; Define a user query text threshold I based on the normalized user query text score; For texts with a standardized user query text score greater than I, the standardized user query text is directly output. For texts not greater than I, feedback is provided to the user and the user is requested to re-enter the text, and a source channel identifier is added to each input record.

3. The heterogeneous knowledge question answering model evaluation method based on ICT customer service as claimed in claim 2, characterized in that: The specific steps of identifying context-aware entity networks based on standardized user query text are as follows: Select CRF as a tool for named entity recognition, preprocess the standardized user query text, use the preprocessed standardized user query text to train the CRF model, apply the trained CRF model to the new user query text, perform named entity recognition, and extract the entities and their associated nodes in the text; For each identified entity, calculate its precision P, expressed as: Where P is the ratio of the number of correctly identified entities to the total number of identified entities, TP is the number of correctly identified entities, and FP is the number of incorrectly identified entities; Calculate the recall rate R, the expression is: Where R is the ratio of the number of correctly identified entities to the total number of entities that actually exist, TP is the number of correctly identified entities, and FN is the number of unidentified entities; Calculate the F1 score, the expression is: Among them, F1 is the harmonic mean of the precision and recall of entity recognition, P is the ratio of the number of correctly recognized entities to the total number of recognized entities, and R is the ratio of the number of correctly recognized entities to the total number of entities that actually exist; Introduce the comprehensive evaluation function Q, the expression is: Among them, Q is a scoring indicator reflecting the accuracy of entity recognition and the initial processing speed, F1 is the harmonic mean of the precision and recall of entity recognition, C is the logical consistency score of the entity in the network, λ is the attenuation coefficient of the influence of time on the score, and τ is the time difference from user input to completion of recognition; According to the evaluated entities and their associated nodes, a directed weighted graph G is constructed, which is expressed as: G = (V, E, w); Where G is a directed weighted graph, V is a set of nodes, each node represents an entity, E is a set of edges, and w is the weight of the edge; Check the logical consistency between each node and its adjacent nodes. If any unreasonable connections are found during the consistency check, make adjustments, introduce constraints, and optimize the structure of the directed weighted graph G. The optimization graph quality threshold A is defined based on the scoring indicators of entity recognition accuracy and initial processing speed; Determine whether the optimized directed weighted graph G satisfies the value of the quality assessment function Q and is not less than A. If so, output the context-aware entity network, otherwise further adjust and optimize.

4. The heterogeneous knowledge question answering model evaluation method based on ICT customer service as claimed in claim 3, characterized in that: The specific steps of generating a context-updated knowledge graph based on a context-aware entity network are as follows: Based on the newly identified entities and relationships in the context-aware entity network, the relevant entities and associated nodes in the knowledge graph that need to be updated are screened out through context consistency checks. For the selected entities and associated nodes, GraphSAGE is selected as the graph embedding model, and the relevant entities and associated nodes are extracted from the existing knowledge graph. The new embedding vector of each node is calculated using the GraphSAGE model, and incremental updates are performed using database transaction management. Introduce update quality score function Q ′ , the expression is: Among them, Q ′ is the update quality score, F1 is the harmonic mean of the precision and recall of entity recognition, C is the logical consistency score of the entity in the network, τ is the time difference from user input to completion of recognition, λ is the attenuation coefficient of the impact of time on the score, α is the weight factor of the embedding vector similarity, and v i is the embedding vector of node i before updating, v i ′ is the embedding vector of node i after update, n is the number of nodes involved in the update, cos(v i ,v i ′ ) is the cosine similarity between the embedding vectors of node i before and after update; Introduce the Git version control system to record the relevant entities, associated nodes and timestamps of each update, and calculate the update quality score after the update is completed; defining an update quality threshold H based on the update quality score; For updates with an update quality score less than H, analyze the problem and make adjustments, otherwise confirm that the update is successful and generate a contextual update knowledge graph.

5. The heterogeneous knowledge question answering model evaluation method based on ICT customer service as claimed in claim 4, characterized in that: The specific steps of constructing semantic contrast vector pairs based on context-based updating of knowledge graph are as follows: The user's query question and the entities, attributes, relationships, and context information in the knowledge graph are used as input, and the Seq2Seq model is trained and inferred to generate multiple candidate answers. Select the BERT model for text encoding, and extract the vector representation of the last layer output of BERT for the text snippets and entity descriptions related to each candidate answer in the knowledge base; Introducing the semantic similarity value D, the expression is: Where D is the semantic similarity value, m is the number of vectors of candidate answers and knowledge base content, cos(a j ,b j ) is the cosine similarity between the jth pair of vectors, w j is the weight of the jth pair of vectors, β is the weight of the time decay factor, γ is the time decay coefficient, τ is the time difference from user input to completion of recognition, a j is the jth vector representation of the candidate answer, b j is the jth vector representation of the knowledge base content; According to the similarity score D, the semantic comparison vector pair with the highest score is selected, and the optimal semantic comparison vector pair is output.

6. The heterogeneous knowledge question answering model evaluation method based on ICT customer service as claimed in claim 5, characterized in that: The semantic consistency and accuracy of the evaluation answer is evaluated based on the semantic comparison vector, and the final answer is confirmed. The specific steps are as follows: Use contextual updated knowledge graph queries, domain expert annotations, historical interaction data, and user feedback to obtain standard answers; The semantic similarity value D is used to calculate the similarity between the candidate answer vector and the standard answer vector, and the candidate answer with the highest score is selected as the final answer. An automated assisted manual review mechanism is introduced to output the final answer.

7. The method for evaluating a heterogeneous knowledge question-answering model based on ICT customer service according to claim 6, characterized in that: Based on the final answer, the semantic contrast vector pair is optimized. The specific steps are as follows: Set up a clear five-star rating mechanism on the user interaction interface, collect users' ratings of the final answer, and record the data of users' click behavior, dwell time, and number of repeated queries, and analyze user ratings, user click behavior scores, user dwell time scores, and number of repeated queries; The expression of user click behavior score B is: Among them, B is the user's click behavior score, n c is the total number of click events, θ is the weight of the φth click, and φ is the sequence number of the click event; User stay time score S d The expression is: Among them, S d is the user's residence time score, and η is the user's actual residence time; Introducing comprehensive user feedback scoring function F u , the expression is: Among them, F u is the comprehensive user feedback score, r k Rating for the kth user, t k The timestamp of the k-th user rating, β t is the time weight factor, B is the user click behavior score, L is the total number of clicks, S d is the user's residence time score, Z is the number of repeated queries, μ is the behavior data weight factor, and n r is the total number of ratings; Based on the overall feedback score F u Dynamically adjust the parameters and feature weights of the semantic similarity value D and introduce the user feedback enhanced semantic similarity value D ′ (q,y), the expression is: D ′ (q,y)=D+x·F u (q,y); Among them, D ′ (q,y) is the optimized semantic similarity value, D is the semantic similarity value, F u (q,y) is the comprehensive user feedback score of user query q and candidate answers, x is the user feedback influence factor, q is the user query, and y is the candidate answer.

8. A heterogeneous knowledge question answering model evaluation system based on ICT customer service, based on the heterogeneous knowledge question answering model evaluation method based on ICT customer service according to any one of claims 1 to 7, characterized in that: Including user input processing module, entity recognition module, knowledge graph update module, semantic comparison module, answer evaluation module and optimization configuration module; A user input processing module, used to receive user queries and generate standardized user query text; The entity recognition module is used to identify entities and their associated nodes using standardized user query text and build a context-aware entity network; A knowledge graph updating module, used to generate a context-updated knowledge graph based on context-aware entities; The semantic comparison module is used to update the knowledge graph based on the context and build semantic comparison vector pairs; The answer evaluation module is used to evaluate the semantic consistency and accuracy of the answer and confirm the final answer; The configuration optimization module is used to optimize the configuration based on the feedback of the final answer.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the heterogeneous knowledge question and answer model evaluation method based on ICT customer service described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the heterogeneous knowledge question and answer model evaluation method based on ICT customer service described in any one of claims 1 to 7 are implemented.

Images