Large model training method and large model-based question and answer method

Abstract

The invention provides a training method of a large model and a question answering method based on the large model, relates to the technical field of neural networks, and is used for improving the accuracy of the large model. The method comprises the following steps: determining a target knowledge point in a knowledge graph and a target confidence coefficient corresponding to the target knowledge point; the target knowledge point is used for representing a relationship between two entities in the knowledge graph; determining rewritten knowledge points having the same semantics as the target knowledge points; inputting the rewritten knowledge point and the first prompt information into a large model, and determining a first estimated confidence coefficient of the rewritten knowledge point through the large model; the first prompt information is used for prompting the large model to judge the accuracy degree of rewriting the knowledge points; the first estimated confidence coefficient is used for representing the accuracy degree; calculating a first loss value based on the target confidence and the first estimated confidence; and when the first loss value is greater than or equal to a first preset threshold value, updating the large model.

Description

Technical Field

[0001] This application relates to the field of neural network technology, and in particular to a training method for large models and a question-answering method based on large models. Background Technology

[0002] In the process of digital transformation for telecom operators, intelligent customer service is key to improving service efficiency and reducing costs. Its performance optimization and system testing heavily rely on large-scale, high-quality customer service dialogue data. Big Data Models, with their powerful natural language understanding and generation capabilities, have significantly improved the response efficiency of telecom operators' customer service, becoming a core engine driving the digital transformation of telecom operators' services.

[0003] However, while large models have achieved success in the application of intelligent customer service by operators, some problems still exist. Due to the insufficient ability of large models to generate low-frequency data, they are prone to misjudging low-frequency scenario data, resulting in relatively low accuracy of large models.

[0004] Therefore, improving the accuracy of large models is an urgent problem to be solved. Summary of the Invention

[0005] This application provides a training method for large models and a question-answering method based on large models, which can improve the accuracy of large models.

[0006] In a first aspect, this application provides a training method for a large model, comprising: determining target knowledge points in a knowledge graph and target confidence levels corresponding to the target knowledge points; the target knowledge points are used to represent the relationship between two entities in the knowledge graph; determining rewritten knowledge points that have the same semantics as the target knowledge points; inputting the rewritten knowledge points and first prompt information into the large model, and determining a first estimated confidence level of the rewritten knowledge points through the large model; the first prompt information is used to prompt the large model to judge the accuracy of the rewritten knowledge points; the first estimated confidence level is used to represent the accuracy level; calculating a first loss value based on the target confidence level and the first estimated confidence level; and updating the large model if the first loss value is greater than or equal to a first preset threshold.

[0007] The technical solution provided in this application brings at least the following beneficial effects: As a professional knowledge base for operator business, the knowledge points in the knowledge graph are all verified. Training large models based on these accurate and authoritative knowledge points in the knowledge graph allows large models to learn accurate knowledge that conforms to business rules, thereby avoiding information distortion and model output deviation caused by information bias.

[0008] Using authoritatively verified target knowledge points and their corresponding target confidence levels in the knowledge graph as objective benchmarks, the model generates synonymous rewrites of knowledge points with diverse expressions. This allows the model to fully engage with and learn various questioning and expression methods. Even if users ask the same question with different wording, the large model can accurately identify its semantic essence, improving the robustness of the large model in semantic recognition of knowledge points with different synonymous expressions.

[0009] Furthermore, by guiding the large model to judge the accuracy of rewritten knowledge points through the first prompt information, and then using the loss function to accurately quantify the deviation between the model's first estimated confidence and the target confidence, and finally triggering the model's closed-loop update based on the first preset threshold, the large model's deviation in semantic judgment can be corrected, and its ability to accurately map target knowledge points can be continuously strengthened, thereby improving the accuracy of the large model.

[0010] As one possible implementation, the two entities are adjacent entities; the rewritten knowledge points are determined to be semantically identical to the target knowledge points, including: sentence transformation and / or synonym replacement of the target knowledge points to obtain the rewritten knowledge points.

[0011] As one possible implementation, two entities are connected through at least one other entity. The target knowledge point is used to represent the relationship between all adjacent entities in the connection link between the two entities. Determining rewritten knowledge points with the same semantics as the target knowledge point includes: decomposing the target knowledge point into N sub-knowledge points representing the relationship between adjacent entities; performing sentence transformation and / or synonym replacement on each of the N sub-knowledge points to obtain N rewritten sub-knowledge points; and reorganizing the N rewritten sub-knowledge points into a combined knowledge point with the same semantics as the target knowledge point, which is then used as the rewritten knowledge point.

[0012] As one possible implementation, the method further includes: constructing a corresponding negation form for the target knowledge point to obtain a knowledge point with the opposite semantics to the target knowledge point; inputting the negated knowledge point and the second hint information into the large model, and determining the second estimated confidence of the negated knowledge point through the large model; the second hint information is used to prompt the large model to judge the accuracy of the negated knowledge point; the second estimated confidence is used to represent the accuracy; calculating the second loss value based on the target confidence and the second estimated confidence; and updating the large model if the second loss value is greater than or equal to the second preset threshold.

[0013] As one possible implementation method, the knowledge graph is constructed in the following way: extracting knowledge points from multiple data sources; the data sources include at least one of the following: package data source, fault handling data source, and customer service dialogue data source; the package data source is used to define and describe the products, fees and business rules provided by the operator, the fault handling data source is used to record communication faults and handling processes, and the customer service dialogue data source is used to represent customer inquiries and solutions; the knowledge points from multiple data sources are integrated to obtain the knowledge graph.

[0014] As one possible implementation, entities are extracted from the data source in the following ways: extracting keywords from the data source to obtain a keyword set; merging adjacent keywords if they meet a preset merging rule to obtain a merged keyword set; the two adjacent keywords are used to represent two keywords that are consecutive in the data source and do not have any other words in between; and determining the entity based on the merged keyword set.

[0015] As one possible implementation, knowledge points from multiple data sources are integrated, including: for any two entities from different data sources, determining the semantic similarity between the two entities based on their semantics; determining the structural similarity between the two entities based on their respective association relationships; weighting the semantic similarity and structural similarity to obtain the similarity between the two entities; and merging the two entities into a single entity if the similarity is greater than a preset entity similarity threshold.

[0016] Secondly, this application provides a question-answering method based on a large model, comprising: obtaining a question statement input by a user; determining the knowledge point corresponding to the question statement in a knowledge graph; inputting the question statement, the corresponding knowledge point, and third prompt information into the large model provided in any embodiment of the first aspect, and obtaining an answer to the question statement through the large model; the third prompt information is used to instruct the large model to perform knowledge retrieval and answer generation based on the knowledge point.

[0017] As one possible implementation, the method also includes: determining candidate answers to the question statement generated by the large model; filtering the candidate answers according to their format to obtain a first answer; the first answer represents the candidate answers that meet the format requirements; filtering the first answer based on the content authenticity of the candidate answers, removing candidate answers whose content authenticity is less than a preset authenticity threshold, to obtain a second answer; and filtering the second answer based on the business quality of the candidate answers to obtain the answer to the question statement.

[0018] As one possible implementation, based on the business quality of candidate answers, the second answer is filtered to obtain the answer to the question statement. This includes: determining the similarity between the keywords of the candidate answer and preset keywords; using preset keywords to reflect the scenario of the question statement; filtering the second answer and removing the second answer with a similarity less than a preset similarity threshold to obtain the third answer; using a reward model, obtaining the quality score of the third answer based on the degree of matching between the third answer and the user's historical preferences; removing the third answer with a quality score less than a preset score threshold to obtain the answer to the question statement.

[0019] Thirdly, this application provides a training apparatus for a large model, comprising: a first determining unit, configured to determine a target knowledge point in a knowledge graph and a target confidence level corresponding to the target knowledge point; the target knowledge point is used to represent the relationship between two entities in the knowledge graph; and to determine a rewritten knowledge point that has the same semantics as the target knowledge point; a first processing unit, configured to input the rewritten knowledge point and a first prompting information into the large model, and to determine a first estimated confidence level of the rewritten knowledge point through the large model; the first prompting information is used to prompt the large model to judge the accuracy of the rewritten knowledge point; the first estimated confidence level is used to represent the accuracy level; a first loss value is calculated based on the target confidence level and the first estimated confidence level; and the large model is updated if the first loss value is greater than or equal to a first preset threshold.

[0020] Fourthly, this application provides a question-answering device based on a large model, comprising: an acquisition unit for acquiring a question statement input by a user; a second determination unit for determining the knowledge point corresponding to the question statement in a knowledge graph; a second processing unit for inputting the question statement, the corresponding knowledge point, and third prompt information into the large model provided in any embodiment of the first aspect, and obtaining an answer to the question statement through the large model; the third prompt information is used to instruct the large model to perform knowledge retrieval and answer generation based on the knowledge point.

[0021] Fifthly, this application provides an electronic device comprising: a processor and a memory; the memory storing processor-executable instructions; when the processor is configured to execute the instructions, causing the electronic device to implement the method of the first aspect or the method of the second aspect described above.

[0022] Sixthly, this application provides a computer-readable storage medium comprising: computer software instructions; when the computer software instructions are executed in an electronic device, they cause the electronic device to implement the method of the first aspect or the method of the second aspect described above.

[0023] In a seventh aspect, this application provides a computer program product comprising a computer program; when the computer program is run in an electronic device, it causes the electronic device to implement the method of the first aspect or the method of the second aspect described above.

[0024] The beneficial effects of the second to seventh aspects mentioned above are described in the corresponding description of the first aspect and will not be repeated here. Attached Figure Description

[0025] Figure 1 A flowchart illustrating a training method for a large model provided in this application; Figure 2 A flowchart illustrating a subgraph partitioning method provided in this application; Figure 3 A flowchart illustrating another subgraph partitioning method provided in this application; Figure 4 A flowchart illustrating another large model training method provided in this application; Figure 5 A flowchart illustrating another large model training method provided in this application; Figure 6 A flowchart illustrating a knowledge graph construction method provided in this application; Figure 7 A flowchart illustrating a question-answering method based on a large model provided in this application; Figure 8 A flowchart illustrating yet another question-answering method based on a large model provided in this application; Figure 9 A schematic diagram illustrating the composition of a training device for a large model provided in this application; Figure 10 A schematic diagram illustrating the composition of a question-answering device based on a large model, provided in this application; Figure 11 This is a schematic diagram of the composition of an electronic device provided in this application. Detailed Implementation

[0026] The following is a detailed description of a call detail record (CDR) data recording method provided in this application, with reference to the accompanying drawings.

[0027] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0028] The terms "first" and "second," etc., used in the specification and drawings of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0029] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0030] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0031] To facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with essentially the same function and effect. Those skilled in the art can understand that the terms "first" and "second" are not intended to limit the quantity or execution order.

[0032] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0033] In some embodiments, within the field of intelligent customer service for telecom operators, for various business-related inquiries from users, such as package consultations, troubleshooting, service processing, and bill inquiries, intelligent question-answering technology based on large models quickly generates responses that conform to business specifications, thus providing answers. This large-model-based intelligent question-answering technology, with its powerful natural language understanding and generation capabilities, can accurately capture the core needs in users' natural language, while generating fluent, logically clear response text that aligns with the operator's business rules, adapting to diverse customer service scenarios.

[0034] Intelligent question answering technology based on large models is based on pre-trained large language models. It generates answers to input questions by clarifying business response constraints through prompting engineering and integrating knowledge graphs from the operator's domain. This enables rapid response to user inquiries, generating authoritative and accurate responses, improving the interaction efficiency and problem-solving rate of intelligent customer service, and reducing the operational pressure on human customer service.

[0035] While large-scale model-based intelligent question-answering technology excels in response fluency and natural interaction, it still suffers from drawbacks due to its model architecture. For low-frequency business scenarios of telecom operators, the limited volume of inquiries results in a very low proportion of relevant data in the training dataset; some niche scenarios even lack samples entirely. The model cannot fully learn the business rules, terminology definitions, and response logic of these scenarios, creating a knowledge blind spot when faced with such inquiries. It can only guess responses based on experience from high-frequency scenarios, leading to relatively low accuracy in large-scale model question-answering. It can be seen that while large-scale models offer a high level of interactive experience and response efficiency, they still suffer from incomplete knowledge coverage in low-frequency scenarios, resulting in lower accuracy. Therefore, improving the accuracy of large-scale models is a pressing issue that needs to be addressed.

[0036] Based on this, this application provides a training method for a large-scale model and a question-answering method based on the large-scale model. The method involves determining target knowledge points in a knowledge graph and their corresponding target confidence levels; the target knowledge points represent the relationship between two entities in the knowledge graph; determining rewritten knowledge points with semantically identical meaning to the target knowledge points; inputting the rewritten knowledge points and first hint information into the large-scale model, which then determines the first estimated confidence level of the rewritten knowledge points; the first hint information prompts the large-scale model to judge the accuracy of the rewritten knowledge points; the first estimated confidence level represents the accuracy level; calculating a first loss value based on the target confidence level and the first estimated confidence level; and updating the large-scale model if the first loss value is greater than or equal to a first preset threshold. Thus, the knowledge graph, as a professional knowledge base for operator services, contains verified knowledge points. Training the large-scale model based on these accurate and authoritative knowledge points in the knowledge graph allows the model to learn precise knowledge that conforms to business rules, thereby avoiding information distortion and model output deviation caused by information bias.

[0037] Using authoritatively verified target knowledge points and their corresponding target confidence levels in the knowledge graph as objective benchmarks, the model generates synonymous rewrites of knowledge points with diverse expressions. This allows the model to fully engage with and learn various questioning and expression methods. Even if users ask the same question with different wording, the large model can accurately identify its semantic essence, improving the robustness of the large model in semantic recognition of knowledge points with different synonymous expressions.

[0038] Furthermore, by guiding the large model to judge the accuracy of rewritten knowledge points through the first prompt information, and then using the loss function to accurately quantify the deviation between the model's first estimated confidence and the target confidence, and finally triggering the model's closed-loop update based on the first preset threshold, the large model's deviation in semantic judgment can be corrected, and its ability to accurately map target knowledge points can be continuously strengthened, thereby improving the accuracy of the large model.

[0039] In some embodiments, the large model training method of this application is applied to a large model training device, wherein the large model training device can be a single server or a server cluster consisting of multiple servers. In some implementations, the server cluster can be a distributed cluster server. This application embodiment does not impose any limitations in this regard.

[0040] like Figure 1 As shown, the training method for the large model in this application includes the following steps: S101. Determine the target knowledge points in the knowledge graph and the target confidence level corresponding to the target knowledge points.

[0041] In this context, target knowledge points are used to represent the relationship between two entities in a knowledge graph.

[0042] In a knowledge graph, a target knowledge point is represented as an "entity-relationship-entity" triple. For example, entity 1 = "Package A", relation = "includes", entity 2 = "50G data", and the target knowledge point is that Package A includes 50G data.

[0043] The target confidence score represents the quantification of the reliability of the triple (target knowledge point) in the knowledge graph, ranging from 0 to 1. The closer the value is to 1, the higher the reliability of the knowledge point. In this application, the knowledge graph has been cross-validated, and its target confidence score is usually set to 1.0, indicating that the knowledge point has absolute reliability and can be directly used as training data for large models.

[0044] As one possible implementation, based on the business requirements that the large model needs to achieve, subgraphs highly related to the business requirements are selected from the knowledge graph, and core entities and their relationships are extracted from the subgraphs to determine the target knowledge points.

[0045] It should be understood that knowledge graphs contain a large number of entities covering all business scenarios (such as various 5G packages, voice call packages, etc.), and there are diverse semantic relationships between entities. The entity association logic varies significantly across different business scenarios. Directly traversing and searching the knowledge graph to identify target knowledge points is inefficient. Therefore, the knowledge graph can be divided into several subgraphs. By dividing it into subgraphs, knowledge modules matching the application needs of the large model can be quickly located, allowing for precise selection of target knowledge points and improving both efficiency and relevance.

[0046] Understandably, training large models requires reliable performance evaluation benchmarks. Without clear reference standards, model optimization can easily fall into semantic judgment bias and blind pursuit of output reliability that cannot be quantified. Rigorously validated target knowledge points and their corresponding target confidence levels in knowledge graphs can serve as objective benchmarks for large models in relation recognition and semantic understanding tasks. This provides a quantitative comparison basis for the accuracy of model output, ensuring that large models can anchor themselves to real knowledge during optimization, avoiding semantic shifts caused by training data noise, and ultimately improving the reliability of the model's output.

[0047] S102. Identify the rewritten knowledge points that have the same semantics as the target knowledge points.

[0048] Among them, the rewritten knowledge points represent entities and relationships that remain unchanged, and only the expression is changed to generate equivalent knowledge points. These equivalent knowledge points have the same semantics as the target knowledge points but different forms.

[0049] It is understandable that large models will encounter the same relationship expressed in different ways in practical applications. If only the target knowledge points are used for training, the model may have defects such as low accuracy in recognizing the relationship between synonymous different expressions and insufficient robustness in semantic understanding.

[0050] As one possible approach, the target knowledge points can be rewritten by replacing synonyms / near-synonyms, transforming sentence structures, and supplementing redundant information.

[0051] For example, assuming that target knowledge point package A includes 50GB of domestic general data traffic, the corresponding rewritten knowledge point can be generated as follows: Synonym / Near-synonym replacement: Replace "includes" with the near-synonym "covers". Sentence structure transformation: "50G domestic general data traffic is an included benefit of Package A" (converting active sentence to judgment sentence), "Package A includes 50G of domestic general data traffic".

[0052] It should be noted that each target knowledge point can correspond to multiple rewritten knowledge points. The number of rewrites can be flexibly set according to actual training needs, ultimately forming a set of rewritten knowledge points with diverse expressions, providing comprehensive semantic training samples for large models.

[0053] In one implementation of this application, when two entities are adjacent entities, determining the rewritten knowledge point that has the same semantics as the target knowledge point can be achieved by performing sentence transformation and / or synonym replacement on the target knowledge point to obtain the rewritten knowledge point.

[0054] In another implementation of this application, two entities are connected through at least one other entity. The target knowledge point is used to represent the relationship between all adjacent entities in the connection link between the two entities. Determining the rewritten knowledge point with the same semantics as the target knowledge point can be achieved by decomposing the target knowledge point into N sub-knowledge points representing the relationship between adjacent entities; performing sentence transformation and / or synonym replacement on each of the N sub-knowledge points to obtain N rewritten sub-knowledge points; and reorganizing the N rewritten sub-knowledge points into a combined knowledge point with the same semantics as the target knowledge point, which is then used as the rewritten knowledge point.

[0055] For example, the starting entity = User A, the intermediate entity = Enjoy 5G Package, and the ending entity = 100GB of domestic general data traffic. The connection link is: User A → (Apply) → Enjoy 5G Package → (Include) → 100GB of domestic general data traffic. The sub-knowledge points are: Sub-knowledge point 1: Entity 1 = User A, Relationship = Apply, Entity 2 = Enjoy 5G Package; Sub-knowledge point 2: Entity 1 = Enjoy 5G Package, Relationship = Include, Entity 2 = 100GB of domestic general data traffic.

[0056] It should be noted that if each sub-knowledge point generates multiple rewritten results, when combining knowledge points, they can be cross-recombined to form multiple combined knowledge points with different expressions.

[0057] It is understandable that when two entities form a multi-hop connection link through one or more intermediate entities, the complete link is first broken down into multiple independent adjacent entity relationship sub-knowledge points, then each sub-knowledge point is rewritten separately, and finally reorganized according to the original link logic to ensure that the rewritten link is still semantically consistent with the original target knowledge point.

[0058] S103. Input the rewritten knowledge points and the first prompt information into the large model, and determine the first estimated confidence level of the rewritten knowledge points through the large model.

[0059] The first prompt information is used to indicate the accuracy of the large model's judgment on the rewritten knowledge points; the first prediction confidence level is used to represent the accuracy.

[0060] It should be understood that the first estimate confidence level is used to quantify the quality of the rewritten knowledge points. By guiding the large model to make targeted judgments through the first prompt information, qualitative dimensions such as the accuracy of the rewritten knowledge points are transformed into quantitative indicators in the 0-1 range. This indicator not only directly reflects the quality of the rewritten knowledge points themselves, but also reflects the large model's mastery of the target knowledge points through the judgment results of the rewritten knowledge points.

[0061] As one possible approach, when there are multiple knowledge points to be rewritten, the confidence level of each knowledge point to be rewritten is determined, and then the average is calculated to obtain the first estimated confidence level.

[0062] S104. Calculate the first loss value based on the target confidence level and the first estimated confidence level.

[0063] The first loss value represents the degree of deviation between the target confidence level and the first predicted confidence level. Essentially, it is the gap between the large model's understanding of knowledge points expressed in different ways and objective knowledge.

[0064] As one possible implementation, the cross-entropy loss between the target confidence and the first estimated confidence is calculated to obtain the first loss value.

[0065] Understandably, the cross-entropy loss function reflects the degree of deviation of the model's prediction results by quantifying the information difference between the true probability distribution and the model's predicted probability distribution. Therefore, the cross-entropy loss function can accurately reflect the degree of deviation between the large model's judgment on the accuracy of rewritten knowledge points and the objective benchmark (target confidence) of the knowledge graph.

[0066] S105. If the first loss value is greater than or equal to the first preset threshold, update the large model.

[0067] Understandably, if the loss value is greater than or equal to the first preset threshold, it indicates that the large model's judgment on the rewritten knowledge points deviates significantly from the objective benchmark of the knowledge graph, and the model has not met the preset business requirements.

[0068] As one possible approach, if the first loss value is greater than or equal to the first preset threshold, the large model can be updated by updating model parameters and enhancing training samples to reduce the deviation between the first estimated confidence and the target confidence, thereby improving the robustness of the model.

[0069] Therefore, as a professional knowledge base for operators' businesses, the knowledge points in the knowledge graph have all been verified. Training large models based on these accurate and authoritative knowledge points in the knowledge graph allows the large models to learn accurate knowledge that conforms to business rules, thereby avoiding information distortion and model output deviation caused by information bias.

[0070] Using authoritatively verified target knowledge points and their corresponding target confidence levels in the knowledge graph as objective benchmarks, the model generates synonymous rewrites of knowledge points with diverse expressions. This allows the model to fully engage with and learn various questioning and expression methods. Even if users ask the same question with different wording, the large model can accurately identify its semantic essence, improving the robustness of the large model in semantic recognition of knowledge points with different synonymous expressions.

[0071] Furthermore, by guiding the large model to judge the accuracy of rewritten knowledge points through the first prompt information, and then using the loss function to accurately quantify the deviation between the model's first estimated confidence and the target confidence, and finally triggering the model's closed-loop update based on the first preset threshold, the large model's deviation in semantic judgment can be corrected, and its ability to accurately map target knowledge points can be continuously strengthened, thereby improving the accuracy of the large model.

[0072] In some embodiments, a knowledge graph can be divided into subgraphs by using a K-hop method to extract subgraphs, where the number of hops K represents the range of associations between entities, in order to filter out other entities and relationships that can be reached within K steps of the target entity.

[0073] like Figure 2 As shown, the process for extracting a single K-hop subgraph includes: inputting a graph G, a starting edge R1, and a graph organization strategy S; initializing subgraph G' with R1; and initializing set C with all edges directly connected to the two endpoints of the starting edge. Checking if set C is non-empty. If yes, selecting edge e from set C according to S; otherwise, outputting subgraph G'. After selecting edge e from set C according to S, adding edge e to subgraph G' and removing it from set C; checking if subgraph G' satisfies the hop count constraint; if yes, outputting subgraph G'; otherwise, obtaining the two endpoints of edge e. Obtaining all edges connected to each endpoint in graph G and adding them to set C, then returning to check if set C is non-empty.

[0074] The above Figure 2 The process is an iterative method to accurately extract a subgraph G' from graph G with the starting edge R1 as the core and a depth of no more than K hops. Specifically, the direction of subgraph expansion is controlled by graph organization strategy S, and the size of the subgraph is controlled by hop count constraint, so as to obtain a focused and controllable associated subgraph.

[0075] Figure 2 In the graph G, the original complete large graph contains all entity-relationship-entity edges. The starting edge R1 represents the starting point for subgraph extraction. The K-jump constraint represents the maximum depth of subgraph expansion. The graph organization strategy S represents the rules for selecting edges from the candidate edge set C, such as prioritizing edges that are strongly related to the business, selecting edges by weight, or random selection.

[0076] The following is about Figure 2 The implementation process will be explained.

[0077] Step 1: Initialization. Initialize subgraph G': Add the starting edge R1 to G', at this point G' = {R1} (there is only 1 edge); Initialize set C (candidate edge pool): Find all edges in the large graph G that directly connect the two endpoints of the starting edge R1 and add them to C.

[0078] Step 2: Loop through the candidate pool C to check if it is not empty. If there are candidate edges, continue selecting; otherwise, stop (meaning there are no more expandable edges). If C is empty: directly output the current subgraph G'. At this time, G' only has the starting edge R1, indicating that the endpoint of R1 has no other associated edges. If C is not empty: select one edge e from C according to the graph organization strategy S.

[0079] Step 3: Update subgraph G' and candidate pool C. Add the selected edge e to subgraph G': at this point, G' = {R1, e}. Remove edge e from set C.

[0080] Step 4: Determine if the hop count constraint is met. Check if the maximum hop count of the current subgraph G' has reached the preset K. If it has, terminate; otherwise, continue expanding.

[0081] Jump count calculation rules: The starting edge R1 is "1 jump", the edge extended from the endpoint of R1 is "2 jump", the edge extended from the endpoint of the 2-jump edge is "3 jump", and so on.

[0082] If the hop count constraint is met: output G' directly (process ends). If the hop count constraint is not met: continue to the next step to expand the candidate pool C.

[0083] Step 5: Expand the candidate pool C and return to the loop in Step 2. Obtain the two endpoints of the edge e that was just added to G', find all connected edges between these two endpoints in the large graph G (excluding edges already in G' or C to avoid duplication), and add them to set C. Return to Step 2, continue to check if C is not empty, and repeat the above loop until the hop count constraint is satisfied or C is empty.

[0084] like Figure 3 As shown, the overall process of subgraph partitioning includes: forming a set all_edge to be processed from all edges in graph G; initializing a dictionary subgraphs to store all subgraphs; determining whether all_edge is empty; if so, outputting subgraphs; if not, selecting edge e from all_edge and deleting it from all_edge; then calling a single subgraph extraction algorithm to add the generated subgraph to subgraphs; and finally returning to determine whether all_edge is empty.

[0085] Figure 3 The process involves dividing the complete graph G into multiple independent subgraphs without omission or repetition, based on the principle of clustering associated edges. Ultimately, each subgraph is an independent set composed of associated edges, with all edges of the subgraphs not overlapping and covering all edges of the large graph G (without omission).

[0086] The following is about Figure 3 The implementation process will be explained.

[0087] Step 1: Initialize the set of edges to be processed and the result storage dictionary. The set of edges to be processed is all_edge = all edges of graph G. Initialize the result dictionary subgraphs = {} (an empty dictionary, prepared to store the split subgraphs).

[0088] Step 2: Loop through and check if all_edge is empty. The core of the loop is to clear the set of edges to be processed. As long as there are still unallocated edges, continue splitting; terminate when all edges have been allocated.

[0089] If all_edge is empty: This means that all edges of the large graph G have been split into subgraphs, and the subgraphs are output directly (process ends). If all_edge is not empty: Perform the following 3 operations: Operation A: Select a seed edge e and remove it from all_edge. Specifically, select one edge from all_edge randomly or according to a rule as the seed edge e, and remove e from all_edge (to avoid duplicate assignment).

[0090] Operation B: Call the single subgraph extraction algorithm to generate associated subgraphs. Specifically, the seed edge e is used as... Figure 2 The starting edge R1 in the process is passed to a single subgraph extraction algorithm to generate a complete subgraph with e as the core.

[0091] Operation C: Add the generated subgraph to subgraphs and return to step 2 in a loop. Specifically, store G1' into subgraphs. Return to step 2 and check if all_edge is empty.

[0092] Step 3: Second loop (process the remaining edges), continue to execute operations A, B, and C.

[0093] Step 4: Output the results. Output the subgraphs.

[0094] The first flowchart is the extraction unit of a single subgraph, which is responsible for generating a subgraph that meets the constraints starting from a certain starting edge. The second flowchart is the overall logic, which extracts the subgraphs corresponding to all edges in the knowledge graph by calling the first flowchart in a loop, and finally obtains the subgraph set.

[0095] In some embodiments, during the rewriting of target knowledge points, in addition to knowledge points with the same semantics, knowledge points with opposite semantics can be introduced as negative samples into the training process to enhance the ability of large models to distinguish semantic differences. For example... Figure 4 As shown, the above method also includes: S401. For the target knowledge point, construct the corresponding negation form to obtain the knowledge point with the opposite semantics to the target knowledge point.

[0096] As one possible approach, the entities in the target knowledge point remain unchanged. The semantic logic of the original target knowledge point is reversed by using methods such as negation semantics, adding negation adverbs, or replacing the relationship with antonyms. This ensures that the antonyms and the target knowledge point form a completely opposite semantic relationship.

[0097] For example, add negative adverbs such as "not", "not", or "no" before the original relationship, and replace the original relationship with antonyms that are completely opposite in meaning, such as rewriting "include" as "do not cover", "exclude", or "do not contain".

[0098] S402. Input the negative knowledge points and the second hint information into the large model, and determine the second estimated confidence level of the negative knowledge points through the large model.

[0099] The second hint is used to indicate the accuracy of the large model's judgment on the negative knowledge point; the second predicted confidence level is used to represent the accuracy.

[0100] S403. Calculate the second loss value based on the target confidence level and the second estimated confidence level.

[0101] S404. If the second loss value is greater than or equal to the second preset threshold, update the large model.

[0102] It should be noted that the specific implementation process of S402, S403 and S404 can refer to the implementation process of S103, S104 and S105 in the above embodiments, and will not be repeated here.

[0103] Therefore, by introducing semantically opposite negative knowledge points as negative samples into the large-scale model training process, forming a positive-negative sample co-training mechanism with semantically identical rewritten knowledge points (positive samples), we can not only avoid semantic overgeneralization caused by the model learning only positive samples, such as misclassifying antonyms as synonyms, but also strengthen the model's understanding of entity relationship semantics through negative samples, while supplementing the model's understanding of negative expressions, enabling it to accurately identify semantic tendencies in practical applications. Ultimately, through training with positive and negative samples, the accuracy and robustness of the large-scale model in semantic recognition, differentiation, and judgment tasks are comprehensively improved.

[0104] For example, for the target knowledge point This can be rewritten as a semantically equivalent knowledge point: And the negative forms of these rewritten knowledge points: .

[0105] In one possible implementation, these rewritten knowledge points are output to the large model to obtain the third estimated confidence level. Based on the third estimated confidence level and the target confidence level, the third loss value is calculated. If the third loss value is greater than the third preset threshold, the large model is updated.

[0106] The third estimate confidence level is calculated using the following formula:

[0107] in, This indicates the third estimate confidence level. It is a knowledge point of rewriting with the same input semantics. At that time, the large model predicts the probability that it is true. It is a knowledge point about rewriting the input semantics in the opposite way. At that time, the large model predicts the probability that it is false.

[0108] The third loss value is calculated using the following formula:

[0109] in, This indicates the third estimate confidence level. This represents the third loss value (cross-entropy loss).

[0110] In some embodiments, due to the limitations of a single data source, such as limited knowledge coverage, delayed information updates, heterogeneous representation formats, and partial information conflicts, it is difficult to support the core requirements of comprehensiveness, consistency, and authority for knowledge in scenarios such as large-scale model training. Knowledge graphs need to compensate for the shortcomings of a single data source by integrating multi-source heterogeneous data to build a comprehensive knowledge system. Figure 5 As shown, the knowledge graph is constructed in the following way: S501. Extract knowledge points from multiple data sources.

[0111] The data source includes at least one of the following: package data source, fault handling data source, and customer service dialogue data source; the package data source is used to define and describe the products, fees and business rules provided by the operator, the fault handling data source is used to record communication faults and handling processes, and the customer service dialogue data source is used to characterize customer inquiries and solutions.

[0112] Understandably, data sources from service packages, fault handling, and customer service dialogues can cover the entire business chain of operators, from product definition and service support to customer service. Service package data sources include official knowledge bases on products, fees, and business rules; fault handling data sources include technical support knowledge such as fault types and handling procedures; and customer service dialogue data sources include application knowledge of user inquiry scenarios, actual needs, and communication scripts. These three sources form a complete knowledge coverage. Extracting knowledge points from multiple data sources avoids the one-sidedness of knowledge points caused by a single data source.

[0113] As one possible implementation, entities are extracted from the data source in the following ways: extracting keywords from the data source to obtain a keyword set; merging adjacent keywords if they meet a preset merging rule to obtain a merged keyword set; the two adjacent keywords are used to represent two keywords that are consecutive in the data source and do not have any other words in between; and determining the entity based on the merged keyword set.

[0114] Among them, the preset merging rules include at least one of the following: Number and unit integration rule: If adjacent keywords contain pure numbers and units of measurement, or have been naturally combined into the form of "number + unit", then they are kept or merged into a unified expression and are not split.

[0115] Modifier-Business Terminology Merging Rules: If among adjacent keywords, the first is a business modifier and the second is a business term, and the two are in consecutive positions in the text without any other words separating them, then they are merged into a complete business concept expression.

[0116] Attribute-Business Terminology Merging Rules: If, among adjacent keywords, the first is an attribute qualifier (such as "general" or "local") and the second is a business term, and both refer to a specific business object, then they are merged into an attribute-based business description.

[0117] Rules for merging consecutive business terms: If adjacent keywords are terms in the same business field (such as "processing" and "conditions"), and the merged term forms a standardized term commonly used in the industry, then they are directly merged into a complete term.

[0118] For example, assuming the data source is a carrier's business text "Enjoy 50G package including 20G domestic targeted data", the first step is to extract keywords from the text using a keyword extraction algorithm, resulting in a keyword set: {"Enjoy", "50G", "package", "includes", "20G", "domestic", "targeted data"}. Preset merging rules include: merging rules for numbers and units (e.g., "20G" does not need to be split and is directly retained), and merging rules for modifiers and business terms (e.g., "Enjoy" is adjacent to "50G" and "package"). After merging, the keyword set is obtained: {"Enjoy 50G package", "includes", "20G domestic targeted data"}. Finally, elements with business entity attributes are selected from the merged keyword set, excluding relational words and conjunctions, ultimately determining the entities as: "Enjoy 50G package" and "20G domestic targeted data".

[0119] Understandably, the above entity extraction method ensures the comprehensiveness of entity extraction (without omitting core business elements in the keywords) by extracting keywords and then using preset merging rules, while avoiding the semantic breakage of entities caused by keyword splitting.

[0120] S502. Merge knowledge points from multiple data sources to obtain a knowledge graph.

[0121] It is understandable that different data sources may contain entities with the same semantics but different expressions (such as the 5G Enjoy Package in the package data source and the 5G Enjoy Package in the customer service dialogue data source). If a knowledge graph is directly constructed, it will lead to redundancy and inconsistency of entities in the knowledge graph, which will affect the accuracy of subsequent large model training.

[0122] As one possible implementation, knowledge points from multiple data sources are integrated, including: for any two entities from different data sources, determining the semantic similarity between the two entities based on their semantics; determining the structural similarity between the two entities based on their respective association relationships; weighting the semantic similarity and structural similarity to obtain the similarity between the two entities; and merging the two entities into a single entity if the similarity is greater than a preset entity similarity threshold.

[0123] In one possible implementation, the semantic similarity between two entities is determined based on their semantics. This can be achieved by converting the two entities into semantic vectors and calculating the degree of matching between the two semantic vectors using a vector similarity algorithm (such as cosine similarity or Euclidean distance). The result takes values ​​in the range [0,1] (1 represents complete synonymy, and 0 represents complete irrelevance).

[0124] In one possible implementation, the structural similarity between two entities is determined based on their respective relationships. This can be achieved by extracting the relationship sets for each entity, where each relationship set includes: {(related entity 1, relationship type 1), (related entity 2, relationship type 2), ...}. The matching degree between the two relationship sets is then calculated. This can be achieved by using the proportion of combinations of identical (or synonymous) related entities and identical relationship types as the structural similarity. Alternatively, entities and their relationships can be considered as subgraphs, and the topological similarity of the subgraphs can be calculated and used as the structural similarity.

[0125] It is understandable that semantic similarity is based on the literal meaning and business semantics of the entities themselves to determine whether two entities refer to the same business object. Based on the association relationship of the entities, it is determined whether the roles and associated objects of the two entities in the business logic are consistent (the business association of the same entity is usually fixed). Combining the importance of the two dimensions (adjusted by weights), a comprehensive similarity is calculated to avoid the one-sidedness of judgment based on a single dimension.

[0126] As another possible implementation, the weights of multiple data sources are determined. When knowledge points are fused, if there is a conflict, the knowledge points corresponding to the high-weight data source are retained, and the conflicting knowledge points of the low-weight data source are filtered out, according to the principle that the data source with the higher weight has higher priority.

[0127] For example, the weight of the package data source is greater than that of the customer service dialogue data source. If there is a conflict of knowledge points, the knowledge points extracted from the package data source shall prevail.

[0128] It should be understood that this weight-based conflict resolution mechanism can quickly eliminate noise interference from low-reliability data sources, ensure the accuracy of knowledge points in the knowledge graph, and avoid semantic confusion in large model training due to the coexistence of conflicting knowledge points.

[0129] Therefore, by extracting knowledge points from multi-source data and then integrating them, we can cover all scenarios of products, faults, and customer service, ensuring that no knowledge points are missed. This provides a comprehensive knowledge point benchmark for subsequent large-scale model training, avoids semantic recognition deviations in large-scale models due to knowledge graph quality issues, and lays a core foundation for improving the accuracy of large-scale models.

[0130] The following is combined Figure 6 The process of constructing a knowledge graph is introduced.

[0131] (1) First, structured data such as package parameter library and business processing process library, semi-structured data such as fault handling rule library, and unstructured data such as historical customer service dialogue library are selected as the core data sources for constructing the knowledge graph. Key information is extracted through regular expressions, data is cleaned (special symbols are removed and duplicate data is removed), and word segmentation is performed for data source preprocessing.

[0132] (2) Knowledge extraction extracts <entity, relation, attribute> triples from the preprocessed data source, including entity extraction, relation extraction, and attribute extraction. First, entity extraction is performed by obtaining deep semantic representations of the text through the BERT model, mapping them to the label space through a linear layer, and finally learning the transition probabilities between entity labels using the CRF layer to output the optimal entity label sequence. Then, entities are specially labeled and masked, and BERT is used to learn semantic associations between entities, with the classification layer predicting the relation type to achieve relation extraction. Finally, the BERT model is reused, utilizing its extractive question-answering capabilities to output the start and end positions of attributes, from which attributes are extracted.

[0133] One possible implementation involves obtaining a deep semantic representation of the text using the BERT model, mapping it to a label space via a linear layer, and finally having a CRF layer learn the transition probabilities between entity labels to output the optimal entity label sequence. This can be achieved by first splitting the original text from the data source into a sequence of tokens. A token is the smallest semantic unit that the model can recognize and process after word segmentation (e.g., "Enjoy 50G package including 300 minutes of domestic voice calls" is split into ["Enjoy", "50G", "package", "includes", "300 minutes", "domestic", "voice calls"]). Based on the token sequence, a first input vector, a second input vector, and a third input vector are generated. The first input vector represents the basic semantics of each token, the second input vector is used to distinguish paragraphs in single-sentence inputs (operator business texts are mostly single-sentence expressions), and the third input vector represents the positional information of the tokens in the sequence, ultimately forming a fixed-dimensional input vector that the BERT model can accept.

[0134] A fixed-dimensional input vector is fed into the BERT model. The BERT model captures the contextual semantic relationships of each word through a multi-layer bidirectional Transformer encoder (such as the semantic binding of "Smart Enjoyment," "30G," and "package," and the logical relationship between "signal interruption" and "fault"), outputting a high-dimensional semantic vector for each word. This high-dimensional semantic vector is then input into a linear layer. The linear layer uses the BIO (Begin-Inside-Outside) annotation method, defining a set of labels based on the operator's business entity type, for example: B-XXXX (entity start), I-XXXX (entity middle), O (non-entity). The linear layer projects the high-dimensional semantic vector of each word into a low-dimensional vector (with the same dimensions as the label set) using a learnable weight matrix, with each dimension corresponding to a predicted score for a label.

[0135] Exemplarily, taking the token sequence "handle, intelligent sharing, 30G, package, can, obtain, 100 minutes, domestic, call" as an example, after the linear layer projects the high-dimensional semantic vectors of each token into low-dimensional vectors with the same dimension as the label set (B-Package start of package entity, I-Package middle of package entity, B-Benefit start of benefit entity, I-Benefit middle of benefit entity, O), the predicted scores of each token corresponding to different labels show significant category discrimination characteristics: among them, the predicted score of the B-Package label corresponding to "intelligent sharing" is the highest, and the predicted scores of the I-Package labels corresponding to "30G" and "package" are the highest. These three tokens together constitute the package entity; the predicted score of the B-Benefit label corresponding to "100 minutes" is the highest, and the predicted scores of the I-Benefit labels corresponding to "domestic" and "call" are the highest. The three belong to the same benefit entity. The predicted scores of tokens such as "handle", "can", and "obtain" corresponding to the O label are the highest, indicating that they do not belong to any business entity category. The highest predicted score of each token intuitively reflects the matching degree between its semantic characteristics and the corresponding entity label.

[0136] The CRF layer forms a transition probability matrix by training and learning the reasonable transition rules between labels (such as B-XXXX can be followed by I-XXXX or O, and cannot be directly followed by B-XXXX; O can be followed by any B-XXXX label and cannot be directly followed by I-XXXX label). Combining the label predicted scores output by the linear layer with the transition probabilities of the CRF layer, the label sequence with the highest probability is calculated through the Viterbi algorithm (such as avoiding label combinations that do not conform to the rules like "B-Package→I-Benefit"). According to the optimal label sequence, the tokens labeled as B-XXXX and I-XXXX are selected, and the consecutive entity tokens of the same type are merged.

[0137] One possible approach is to perform special labeling and masking on entities, using BERT to learn semantic relationships between entities and having a classification layer predict the relationship type to achieve relationship extraction. This can be implemented by accurately identifying two target entities from operator business text, then performing structured encoding on the original text, wrapping the two target entities with pre-defined labels to clarify the entity boundaries within the text. Optional masking is applied to non-key connecting words between entities (such as "contains," "needs," "corresponds," etc.) to force the BERT model to focus on the semantic relationships within the entities themselves rather than surface connecting words. The encoded text is then input into the BERT model, which uses a multi-layer bidirectional Transformer encoder to capture the deep semantic dependencies between the two entities in the context, outputting a fused semantic feature vector of the two target entities. This vector integrates entity semantics, contextual information, and inter-entity relationship features. A softmax function maps the vector to a probability distribution of each relationship type, and finally, the relationship type with the highest probability value is selected as the final relationship between the two target entities, completing the relationship extraction.

[0138] For example, suppose the operator's business text is "Enjoy 50G package includes 50G domestic general data traffic, and you need to bind your real-name mobile phone number to apply". After entity extraction, the target entities E1 = "Enjoy 50G package" and E2 = "50G domestic general data traffic" are obtained. The classification layer performs probability prediction on relationship types such as "includes" and "application conditions are". If the probability value of the inclusion relationship is the highest, then the relationship between E1 and E2 is determined to be inclusion, and the knowledge point that the Enjoy 50G package includes 50G domestic general data traffic is successfully extracted.

[0139] As one possible implementation, the BERT model can be reused, leveraging its extractive question-answering capabilities to output the start and end positions of attributes and extract them. This can be achieved by first transforming the attribute extraction task in the operator's business scenario into an extractive question-answering task: for the identified target entity and the type of attribute to be extracted, a standardized question template is constructed, such as "What is the data allowance of [target entity]?". Then, the input data for extractive question-answering is constructed, using the constructed question as the query statement and the original business text containing the target entity as the context, and encoding it according to the input requirements of the BERT model to form a feature vector that the BERT model can process. The feature vector is input into the BERT model, reusing its extractive question-answering capabilities. The model learns the semantic matching relationship between the question and the context, and leverages the attention mechanism of a multi-layer bidirectional Transformer encoder to focus on key information strongly related to the target entity and attribute type within the context. It captures the semantic association between the question's intent and candidate attribute values ​​in the context, outputting two probability distributions: one is the probability of the starting position corresponding to each word in the context, representing the likelihood that the word is the starting position of an attribute value; the other is the probability of the ending position corresponding to each word, representing the likelihood that the word is the ending position of an attribute value. Valid candidate positions are filtered using a pre-set confidence threshold: first, candidate starting points with a starting position probability greater than the pre-set confidence threshold are selected; then, for each candidate starting point, the optimal candidate ending point is matched where the ending position is greater than the starting position and the ending position probability is greater than the pre-set confidence threshold. If multiple valid (start ≤ end) position combinations exist, the combination with the largest product of the starting and ending probabilities is selected as the final position pair. Finally, the word sequence between the starting and ending positions in the context is extracted, and after semantic integration, it becomes the attribute value corresponding to the target entity, completing attribute extraction.

[0140] For example, assuming the target entity is the Smart 30G Package, the attribute to be extracted is the application conditions, and the question is constructed as "What conditions are required to apply for the Smart 30G Package?", the corresponding context text is "The Smart 30G Package is open to users aged 18-65. To apply, you need to bind your real-name mobile phone number and complete facial verification." After the encoded input text is fed into the BERT model, the model predicts the probability of the corresponding start position and the probability of the corresponding end position. The word sequence "bind your real-name mobile phone number and complete facial verification" is extracted, which is the application condition (attribute value) of the Smart 30G Package.

[0141] (3) The extracted knowledge is fused to solve the problem of conflict and duplication of knowledge from different sources. First, entity alignment is performed. Semantic vectors describing entities are generated by BERT. The semantic similarity between two entities in different data sources is calculated. Then, TransE maps entities and relations to a low-dimensional vector space. The structural similarity between the two entities is calculated by distance. Finally, the two similarity scores are fused to achieve entity alignment.

[0142] (4) Further, perform weight calculation and logical consistency judgment. First, assign weights to different data sources. For example, the weight of the package parameter library is greater than the weight of the historical customer service dialogue library. Then, detect logical consistency based on domain rules. When a conflict occurs, remove knowledge with low data source weight.

[0143] (5) Finally, import the processed <entity, relation, attribute> triples into Neo4j in JSON or CSV format for storage.

[0144] In some embodiments, the large-model-based question-answering method of this application can be applied to a large-model-based question-answering device, wherein the large-model-based question-answering device can be a single server or a server cluster consisting of multiple servers. In some implementations, the server cluster can be a distributed cluster server. This application does not impose any limitations on this.

[0145] It should be noted that the question-answering device for the large model in this embodiment and the large model training device in the above embodiments can be deployed on the same server or on different servers.

[0146] like Figure 7 As shown, the question-answering method based on a large model in this application includes the following steps: S701, Obtain the question statement input by the user.

[0147] One possible approach is to obtain user-input question statements via API interfaces. User interaction channels such as official operator software and SMS inquiries can communicate with servers and other electronic devices through API interfaces, enabling real-time collection and transmission of user question statements.

[0148] S702. Determine the knowledge points corresponding to the question statement in the knowledge graph.

[0149] As one possible approach, the data in the knowledge graph is first restated into a coherent customer service response text. Then, based on the entities, relational keywords, and semantic vectors in the restated text, similarity matching is performed with the question statement to determine the corresponding knowledge points.

[0150] Among them, the atomic question answering scenario refers to a scenario where the user's question only involves a single "entity-relationship-entity" triple, and the answer can be generated by directly matching a single knowledge point in the knowledge graph without the need for multi-step logical reasoning.

[0151] For example, the original triple in the knowledge graph: {Enjoy 5G package, includes 100GB of domestic general data}, restated as customer service response text: Enjoy 5G package includes 100GB of domestic general data. The user inputs the question: How much domestic general data does the Enjoy 5G package include? Matching the entities "Enjoy 5G package" and "domestic general data" in the question with the response text, the semantic vector similarity is 0.98, which is greater than the threshold of 0.8. Therefore, the knowledge point is {Enjoy 5G package, includes 100GB of domestic general data}.

[0152] As one possible approach, for multi-hop answer scenarios, entity linking technology can be used to identify the starting entity (the core object that the user is interested in) and the ending entity (the target information that the user wants to query) in the user's question. Then, the multi-hop relationship chain connecting the starting entity and the ending entity can be extracted from the knowledge graph to clarify the reasoning link. The corresponding multi-hop relationship chain is the set of knowledge points corresponding to the question statement.

[0153] Among them, the multi-hop answer scenario refers to a scenario where the question statement involves multiple entity-relation-entity triples, and a multi-hop relation chain needs to be constructed through intermediate entities, and the answer can only be obtained through multiple steps of logical reasoning.

[0154] S703. Input the question statement, the corresponding knowledge points, and the third prompt information into the large model in the above embodiment, and obtain the answer to the question statement through the large model.

[0155] The third prompt is used to instruct the large model to perform knowledge retrieval and answer generation based on knowledge points.

[0156] As one possible implementation, the large model first parses the structured instructions in the third prompt information, then extracts the core intent and entities from the question, and then uses the core intent and entities as retrieval anchors to perform precise matching from the corresponding knowledge points in the input. Through semantic similarity calculation (such as comparing the semantic vectors of entities with those in knowledge points) and key relationship association (such as judging the suitability of the core intent with the "entity-relationship" in knowledge points), it locates the set of knowledge points that can directly answer the question, and then extracts the core elements of the answer that are strongly related to the question from this set of knowledge points. Finally, the large model integrates the extracted core elements of the answer into a natural and coherent text to obtain the answer to the question statement.

[0157] Therefore, since the large model is trained on rewritten knowledge points from the knowledge graph, and precisely quantifies the deviation between the model's first estimated confidence and the target confidence based on the loss function, and finally triggers a closed-loop update of the model based on a first preset threshold, it can correct the large model's bias in semantic judgment and continuously strengthen the large model's ability to accurately map target knowledge points, thereby improving the accuracy of the large model. Furthermore, this application uses the aforementioned more accurate large model to link knowledge points from the knowledge graph with third-party prompts for question-and-answer generation. First, it matches the user's question statement with the corresponding knowledge points in the knowledge graph, and then constrains the model's generation logic through the third-party prompts. This leverages the improved knowledge point recognition and mapping accuracy of the large model itself to ensure that the response content strictly adheres to the authoritative business knowledge in the knowledge graph, thereby improving the accuracy of the question statement's answer.

[0158] In some embodiments, to further ensure the standardization, authenticity, and business adaptability of the large model's output answers, and to avoid response quality issues caused by uncertainties in model generation, it is necessary to perform tiered screening and verification of candidate answers. Therefore, as Figure 8 As shown, the above method also includes: S801. Determine the candidate answers to the question statement generated by the large model.

[0159] Among them, the candidate answer is a preliminary answer generated by the large model based on the question statement, the corresponding knowledge points, and the third prompt information, and may be one or more.

[0160] As one possible approach, the large model could be guided to output multiple candidate answers according to preset requirements by adding an instruction to the third prompt message to "generate multiple candidate answers with different expressions".

[0161] S802. Filter the candidate answers by format to obtain the first answer.

[0162] The first answer is the one that meets the required format among the candidate answers.

[0163] One possible implementation is to first establish a rule base containing prohibited and mandatory rules. Prohibited rules include: no colloquialisms, no emoticons, and no garbled characters or grammatical errors. Mandatory rules include: consistent terminology and standardized punctuation. For each candidate answer, keyword matching is used to check for prohibited items, regular expressions are used to verify terminology compliance, and finally, the structure is checked to ensure it conforms to the format requirements corresponding to the question type. If a candidate answer does not trigger any prohibited items and meets all mandatory requirements, it is considered formatted correctly and included as the first answer.

[0164] Understandably, candidate answers are raw outputs generated by the model following third-party prompts, without any quality checks. They may contain formatting issues such as colloquialisms, inconsistent terminology, punctuation errors, or fragmented structure. These issues reduce the professionalism and readability of the responses, affecting user comprehension. This application addresses this by formatting candidate answers, selecting the first answer with a valid format. This effectively solves the formatting problem, improves the professionalism, consistency, and readability of the responses, and ensures users can quickly and accurately obtain information.

[0165] S803. Based on the authenticity of the candidate answers, the first answer is filtered out, and candidate answers whose authenticity is less than the preset authenticity threshold are removed to obtain the second answer.

[0166] One possible implementation involves inputting the answer content and knowledge point text into the BERT model. The semantic vector similarity between the two is calculated using a cosine similarity algorithm. Entities, attribute values, and relationship types in the answer are extracted and compared one by one with key information in the knowledge points to obtain the key information matching rate. A weighted fusion method is used, setting weights for semantic similarity and key information matching rate, to calculate a truthfulness score. If the truthfulness score is greater than or equal to a preset truthfulness threshold, the answer is retained; if the truthfulness score is less than the preset truthfulness threshold, it is considered information distortion and the answer is discarded.

[0167] Understandably, while large models generate answers based on knowledge points, there may still be issues with the authenticity of the content, such as conflicts between key information and authoritative knowledge points in the knowledge graph. By evaluating the authenticity of candidate answers, we can accurately identify and eliminate such distorted answers, thus ensuring the authority and accuracy of the responses at the core level.

[0168] S804. Based on the business quality of the candidate answers, the second answer is filtered to obtain the answer to the question statement.

[0169] One possible approach is to quantify and score the second response across four dimensions: completeness, conciseness, compliance, and accuracy. Completeness assesses whether the response fully covers the user's needs; conciseness assesses whether the response contains no redundant information; compliance assesses whether the response contains any inappropriate statements; and accuracy assesses the degree to which the response matches the intent of the question. A weighted sum of the scores for each dimension is calculated to obtain a comprehensive score. A business quality threshold is set. If the comprehensive score is greater than or equal to the business quality threshold, it is directly used as the final response. If the second response contains multiple candidates, the one with the highest comprehensive score is selected as the final response. If the comprehensive score is less than the business quality threshold, a standardized fallback response (e.g., "No relevant business information is available; please contact customer service for assistance") is output.

[0170] Understandably, after screening candidate answers for format and content authenticity, it is only possible to ensure that the answer format is compliant and the content is truthful. However, there may still be room for optimization. For example, the answer may not fully cover the user's core needs, the expression may contain redundant information, or the adaptability to specific business scenarios may be insufficient. Therefore, it is necessary to further optimize these "qualified but not high-quality" answers through business quality screening, and finally output high-quality answers that are both compliant with the specifications and accurately adapted to user needs and business logic.

[0171] As another possible implementation, the second answer is filtered based on the business quality of the candidate answers to obtain the answer to the question statement. This can be achieved by: determining the similarity between the keywords of the candidate answers and preset keywords; the preset keywords are used to reflect the scenario of the question statement; filtering the second answers and removing those with a similarity less than a preset similarity threshold to obtain the third answer; using a reward model, obtaining a quality score for the third answer based on the degree of matching between the third answer and the user's historical preferences; and removing the third answers with a quality score less than a preset score threshold to obtain the answer to the question statement.

[0172] In one possible implementation, determining the similarity between the keywords of the candidate answer and the preset keywords can be achieved by calculating the word frequency overlap rate between the keywords of the candidate answer and the preset keywords using a keyword weight calculation method (e.g., the number of intersections between the candidate keywords and the preset keywords / the total number of preset keywords), thus obtaining the similarity between the keywords of the candidate answer and the preset keywords.

[0173] The keywords in the candidate answers reflect the core content of the answers. Preset keywords are used to define the core information that the answers should include in different scenarios.

[0174] In one possible implementation, a quality score for the third answer is derived based on the degree of matching between the third answer and the user's historical preferences. This can be achieved by first extracting historical preference features from the user profile, including preferences for expression style, information detail, and terminology fit. The third answer is then analyzed to extract key features such as its expression form, information density, and terminology usage. The reward model receives the third answer and the user's historical preference features and outputs a quality score (the higher the score, the higher the match between the answer and the user's preferences).

[0175] Understandably, by using a dual filtering logic of keyword similarity matching and user historical preferences, on the one hand, preset keywords directly anchor the core scenario of the question statement, and similarity filtering can accurately eliminate answers that are true in content but irrelevant to the scenario, ensuring that the answers focus on the user's core needs. On the other hand, quantitative scoring based on user historical preference characteristics can ensure that the filtered answers not only comply with business specifications but also fit the individual user's interaction habits. Ultimately, this not only ensures the accuracy of business quality but also improves the personalization and satisfaction of user interaction.

[0176] Therefore, due to uncertainties such as non-standard formatting, questionable content authenticity, and insufficient business adaptability in the candidate answers output by large-scale models, it is difficult to directly meet the requirements of operators' intelligent customer service scenarios for standardized, authoritative, and practical responses. This application, by screening candidate answers for formatting, verifying the authenticity of content, and assessing business quality, can eliminate unqualified answers with chaotic formatting, distorted information, and low business relevance, ultimately outputting high-quality responses with standardized formatting and authentic content. This effectively ensures the accuracy, compliance, and user experience of services in business scenarios, and solves the uncertainty problem of large-scale model-generated question answering output.

[0177] For example, the process of filtering candidate answers includes: (1) Format filtering: 1. For format templates defined for different models, clearly define the required fields and filter out any missing data in the template.

[0178] Different large models (such as Qwen, ChatGLM, etc.) have fixed input / output format templates. Missing key markers will cause the model to be unable to recognize the data type. Taking the Qwen model as an example, it requires that the assistant's answer must include <|im_start|>assistant (start marker) and <|im_end|> (end marker). If the generated candidate answer lacks these core markers, it means that the format is incompatible and it will be filtered directly.

[0179] 2. Set a length threshold based on task type, and filter out data that is too long or too short by counting the number of characters.

[0180] The length threshold should be set in accordance with the information delivery needs and user interaction habits of different tasks. Reasonable upper and lower thresholds can be determined by analyzing the length distribution of data for each task type. For example, in a single-round customer service Q&A scenario, examples with fewer than 10 characters or more than 200 characters can be filtered out.

[0181] (2) Content authenticity assessment: The matching degree between synthetic data and domain knowledge is verified by using natural language reasoning models in order to eliminate erroneous data that contradicts the facts.

[0182] The task of the natural language inference model is defined as determining whether synthetic data is authentic, based on knowledge from the operator's domain. The assumed data content is generated and filtered using a finely tuned T5-11B natural language inference model. Generated data and operator knowledge are input into the model in pairs. The model determines whether the hypothesis can be inferred from the premises and outputs labels such as implication, contradiction, and confidence level. Data with a confidence level greater than 0.7 is considered implication and is retained; the rest is filtered out.

[0183] In one implementation of this application, "premise + hypothesis" are input in pairs into a T5-11B model fine-tuned with operator data. The model outputs three types of labels: implied: the hypothesis is consistent with the premise; contradictory: the hypothesis conflicts with the premise; neutral: the hypothesis is unrelated to the premise. Only data with the "implied" label and a confidence level ≥ 0.7 are retained (the higher the confidence level, the stronger the factual consistency), while contradictory, neutral, and low-confidence data are discarded.

[0184] (3) Business quality assessment: 1. For different business scenarios, a scenario keyword library is preset, and the similarity between the synthesized data text and the keyword library is calculated using the keyword weight calculation method (TF-IDF). Data with a similarity of less than 0.6 is filtered out.

[0185] In one implementation of this application, a scenario keyword library is first preset: classified by business scenario (e.g., "package processing" scenario keywords: processing conditions, binding, real name, age limit). Then, the similarity is calculated: the similarity between the candidate answer and the corresponding scenario keyword library is calculated by TF-IDF (value range [0,1]). If the similarity is <0.6, the data is regarded as "scenario mismatch" and is directly filtered.

[0186] 2. The reward-model-deberta-v3-large-v2 reward model is used to score the quality of the synthetic data from 0 to 1, and the data with a score less than 0.5 is filtered out.

[0187] Among them, the reward-model-deberta-v3-large-v2 reward model is a pre-trained model specifically designed to evaluate text quality and quantify the quality of data representation.

[0188] In one implementation of this application, candidate answers are input into a reward model. The model outputs a quality score of 0-1 (1 being the best and 0 being the worst) based on dimensions such as fluency of expression, completeness of information, standardization of terminology, and readability of users. Only data with a score ≥ 0.5 is retained, and the remaining data is discarded.

[0189] As can be seen, the above mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the embodiments of this application provide corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0190] This application embodiment can divide the key management system into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. Optionally, the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0191] In some embodiments, this application also provides a large-model training apparatus and a large-model-based question-answering apparatus. The two apparatuses described above may include one or more functional modules for implementing the large-model training method and the large-model-based question-answering method of the above method embodiments.

[0192] For example, Figure 9 This is a schematic diagram illustrating the composition of a large model training device provided in an embodiment of this application. Figure 9 As shown, the training device 900 for the large model includes a first determining unit 901 and a first processing unit 902.

[0193] The first determining unit 901 is used to determine the target knowledge point in the knowledge graph and the target confidence level corresponding to the target knowledge point; the target knowledge point is used to represent the relationship between two entities in the knowledge graph; and to determine the rewritten knowledge point with the same semantics as the target knowledge point.

[0194] The first processing unit 902 is used to input the rewritten knowledge points and the first prompt information into the large model, and to determine the first estimated confidence level of the rewritten knowledge points through the large model; the first prompt information is used to prompt the large model to judge the accuracy of the rewritten knowledge points; the first estimated confidence level is used to represent the accuracy level; based on the target confidence level and the first estimated confidence level, the first loss value is calculated; and if the first loss value is greater than or equal to the first preset threshold, the large model is updated.

[0195] As one possible implementation, the two entities are adjacent entities; the first determining unit 901 is specifically used to perform sentence transformation and / or synonym replacement on the target knowledge points to obtain rewritten knowledge points.

[0196] As one possible implementation, two entities are connected through at least one other entity, and the target knowledge point is used to represent the relationship between all adjacent entities in the connection link between the two entities; the first determining unit 901 is specifically used to decompose the target knowledge point into N sub-knowledge points representing the relationship between adjacent entities; perform sentence transformation and / or synonym replacement on each of the N sub-knowledge points to obtain N rewritten sub-knowledge points; and reorganize the N rewritten sub-knowledge points into a combined knowledge point with the same semantics as the target knowledge point, as the rewritten knowledge point.

[0197] As one possible implementation, the first processing unit 902 is further configured to construct a corresponding negation form for the target knowledge point to obtain a knowledge point with the opposite semantics to the target knowledge point; input the negation knowledge point and the second prompt information into the large model, and determine the second estimated confidence level of the negation knowledge point through the large model; the second prompt information is used to prompt the large model to judge the accuracy of the negation knowledge point; the second estimated confidence level is used to represent the accuracy level; calculate the second loss value based on the target confidence level and the second estimated confidence level; and update the large model if the second loss value is greater than or equal to the second preset threshold.

[0198] As one possible implementation, the first processing unit 902 is also used to extract knowledge points from multiple data sources; the data sources include at least one of the following: package data source, fault handling data source, and customer service dialogue data source; the package data source is used to define and describe the products, fees and business rules provided by the operator, the fault handling data source is used to record communication faults and handling processes, and the customer service dialogue data source is used to represent customer inquiries and solutions; the knowledge points from multiple data sources are fused to obtain a knowledge graph.

[0199] As one possible implementation, the first processing unit 902 is specifically used to extract keywords from the data source to obtain a keyword set; if two adjacent keywords meet the preset merging rules, the two adjacent keywords are merged to obtain a merged keyword set; the two adjacent keywords are used to represent two keywords that are consecutive in the data source and do not have other words in between; and the entity is determined based on the merged keyword set.

[0200] As one possible implementation, the first processing unit 902 is specifically used to determine the semantic similarity between any two entities from different data sources based on the semantics of the two entities; determine the structural similarity between the two entities based on the corresponding association relationship between the two entities; weight the semantic similarity and structural similarity to obtain the similarity between the two entities; and merge the two entities into the same entity if the similarity is greater than a preset entity similarity threshold.

[0201] Figure 10 This is a schematic diagram illustrating the composition of a question-answering device based on a large model, provided as an embodiment of this application. Figure 10 As shown, the question-answering device 1000 based on a large model includes: an acquisition unit 1001, a second determination unit 1002, and a second processing unit 1003.

[0202] Unit 1001 is used to obtain the question statement input by the user.

[0203] The second determining unit 1002 is used to determine the knowledge points corresponding to the question statement in the knowledge graph.

[0204] The second processing unit 1003 is used to input the question statement, the corresponding knowledge points, and the third prompt information into the large model provided in any embodiment of the first aspect, and obtain the answer to the question statement through the large model; the third prompt information is used to instruct the large model to perform knowledge retrieval and answer generation based on the knowledge points.

[0205] As one possible implementation, the second processing unit 1003 is also used to determine candidate answers to the question statement generated by the large model; to perform format filtering on the candidate answers to obtain a first answer; the first answer represents the answer with a qualified format among the candidate answers; to filter the first answer based on the content authenticity of the candidate answers, and to remove candidate answers whose content authenticity is less than a preset authenticity threshold to obtain a second answer; to filter the second answer based on the business quality of the candidate answers to obtain the answer to the question statement.

[0206] As one possible implementation, the second processing unit 1003 is specifically used to determine the similarity between the keywords of the candidate answer and the preset keywords; the preset keywords are used to reflect the scenario of the question statement; the second answer is screened, and the second answer with a similarity less than the preset similarity threshold is removed to obtain the third answer; the quality score of the third answer is obtained based on the matching degree between the third answer and the user's historical preferences through the reward model; the third answer with a quality score less than the preset score threshold is removed to obtain the answer to the question statement.

[0207] In the case of implementing the functions of the integrated modules described above in hardware, this embodiment of the invention provides a possible structural schematic diagram of the electronic device involved in the above embodiments. For example... Figure 11 As shown, the electronic device 1100 includes: a processor 1102, a communication interface 1103, and a bus 1104. Optionally, the electronic device 1100 may also include a memory 1101.

[0208] Processor 1102 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 1102 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 1102 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0209] Communication interface 1103 is used to connect with other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0210] The memory 1101 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0211] As one possible implementation, the memory 1101 can exist independently of the processor 1102. The memory 1101 can be connected to the processor 1102 via the bus 1104 and is used to store instructions or program code. When the processor 1102 calls and executes the instructions or program code stored in the memory 1101, it can implement the large model training method or the question-answering method based on the large model provided in the embodiments of the present invention.

[0212] In another possible implementation, the memory 1101 can also be integrated with the processor 1102.

[0213] Bus 1104 can be an extended industry standard architecture (EISA) bus, etc. Bus 1104 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0214] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the service calling device can be divided into different functional modules to complete all or part of the functions described above.

[0215] This application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be executed by computer instructions instructing related hardware. The program can be stored in the aforementioned computer-readable storage medium, and when executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be any of the foregoing embodiments or memory. The aforementioned computer-readable storage medium can also be an external storage device of the aforementioned service invocation device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the aforementioned service invocation device. Further, the aforementioned computer-readable storage medium can include both internal storage units of the aforementioned service invocation device and external storage devices. The aforementioned computer-readable storage medium is used to store the aforementioned computer program and other programs and data required by the aforementioned service invocation device. The aforementioned computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0216] This application also provides a computer program product comprising a computer program that, when run on a computer, causes the computer to execute any of the large model training methods or large model-based question answering methods provided in the above embodiments.

[0217] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training method for a large model, characterized in that, The method includes: Identify target knowledge points in the knowledge graph and the target confidence level corresponding to each target knowledge point; the target knowledge point is used to represent the relationship between two entities in the knowledge graph. Identify rewritten knowledge points that have the same semantics as the target knowledge points; The rewritten knowledge points and the first prompt information are input into a large model, and the large model determines the first estimated confidence level of the rewritten knowledge points; the first prompt information is used to prompt the large model to judge the accuracy of the rewritten knowledge points; the first estimated confidence level is used to represent the accuracy level. Calculate the first loss value based on the target confidence level and the first estimated confidence level; If the first loss value is greater than or equal to the first preset threshold, the large model is updated.

2. The method according to claim 1, characterized in that, The two entities are adjacent entities; determining the rewritten knowledge point that has the same semantics as the target knowledge point includes: The rewritten knowledge points are obtained by transforming the sentence structure and / or replacing synonyms with the target knowledge points.

3. The method according to claim 2, characterized in that, The two entities are connected by at least one other entity, and the target knowledge point is used to represent the relationship between all adjacent entities in the connection link between the two entities. The process of determining the rewritten knowledge points that are semantically identical to the target knowledge point includes: The target knowledge point is decomposed into N sub-knowledge points representing the relationship between adjacent entities; For each of the N sub-knowledge points, perform sentence transformation and / or synonym replacement to obtain N rewritten sub-knowledge points; The N rewritten sub-knowledge points are reorganized into a combined knowledge point with the same semantics as the target knowledge point, which is then used as the rewritten knowledge point.

4. The method according to claim 1, characterized in that, The method further includes: For the target knowledge point, construct the corresponding negation form to obtain the knowledge point with the opposite semantics to the target knowledge point; The negative knowledge point and the second prompt information are input into the large model, and the large model determines the second estimated confidence level of the negative knowledge point; the second prompt information is used to prompt the large model to judge the accuracy of the negative knowledge point; the second estimated confidence level is used to represent the accuracy level. Based on the target confidence level and the second estimated confidence level, calculate the second loss value; If the second loss value is greater than or equal to the second preset threshold, the large model is updated.

5. The method according to claim 1, characterized in that, The knowledge graph is constructed in the following way: The knowledge points are extracted from multiple data sources; the data sources include at least one of the following: package data source, fault handling data source, and customer service dialogue data source; the package data source is used to define and describe the products, fees and business rules provided by the operator, the fault handling data source is used to record communication faults and handling processes, and the customer service dialogue data source is used to characterize customer inquiries and solutions; The knowledge points from the multiple data sources are fused to obtain the knowledge graph.

6. The method according to claim 5, characterized in that, The entity is extracted from the data source in the following ways: Extract keywords from the data source to obtain a keyword set; If two adjacent keywords meet the preset merging rules, then the two adjacent keywords are merged to obtain a merged keyword set; the two adjacent keywords refer to two keywords that are consecutive in the data source and do not contain any other words in between; The entity is determined based on the merged set of keywords.

7. The method according to claim 6, characterized in that, The fusion of knowledge points from the multiple data sources includes: For any two entities from different data sources, determine the semantic similarity between the two entities based on their semantics; Based on the respective association relationships between the two entities, the structural similarity between the two entities is determined; The semantic similarity and the structural similarity are weighted to obtain the similarity between the two entities; If the similarity is greater than a preset entity similarity threshold, the two entities will be merged into a single entity.

8. A question-answering method based on a large model, characterized in that, The method includes: Get the question statement input by the user; Identify the knowledge points corresponding to the question statement in the knowledge graph; The question statement, the corresponding knowledge point, and the third prompt information are input into the large model according to any one of claims 1-7, and the answer to the question statement is obtained through the large model; the third prompt information is used to instruct the large model to perform knowledge retrieval and answer generation based on the knowledge point.

9. The method according to claim 7, characterized in that, The method further includes: Determine the candidate answers to the question statement generated by the large model; The candidate answers are filtered by format to obtain the first answer; the first answer represents the candidate answers that meet the format requirements. Based on the authenticity of the candidate answers, the first answer is filtered, and candidate answers whose authenticity is less than a preset authenticity threshold are removed to obtain the second answer; Based on the business quality of the candidate answers, the second answer is filtered to obtain the answer to the question statement.

10. The method according to claim 9, characterized in that, The process of filtering the second answer based on the service quality of the candidate answers to obtain the answer to the question statement includes: Determine the similarity between the keywords of the candidate answers and preset keywords; the preset keywords are used to reflect the context of the question statement. The second answer is filtered out, and the second answer with a similarity less than a preset similarity threshold is removed to obtain the third answer; The quality score of the third answer is obtained based on the degree of matching between the third answer and the user's historical preferences using a reward model. The third answer, whose quality score is less than a preset score threshold, is removed to obtain the answer to the question statement.

Images