A large-scale professional question-answering generation method and system based on multi-source evidence retrieval and dynamic confidence assessment

By employing multi-source evidence retrieval and dynamic confidence assessment methods, a high-quality evidence set was selected and generation constraints were constructed. This solved the problem of difficulty in controlling the applicability of multi-source evidence in professional question-and-answer generation, and improved the reliability and accuracy of the question-and-answer results.

CN123068567APending Publication Date: 2026-11-10CHINA MOBILE (SUZHOU) SOFTWARE TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611370030.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-04
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

In existing technologies, it is difficult to effectively control the applicability of each retrieval result and its use in the large language model generation process using multi-source evidence, resulting in a lack of reliability and accuracy in professional question answering results.

Method used

By employing a multi-source evidence retrieval and dynamic confidence assessment method, candidate evidence is retrieved from multiple knowledge sources after a user query. The source authority, content timeliness, semantic relevance, and multi-source consistency of the candidate evidence are evaluated separately, and the comprehensive confidence score is obtained by fusion. The target evidence set is then selected, and generation constraints are constructed to limit the generation basis of the large language model.

Benefits of technology

It improves the quality controllability of evidence in the professional question-and-answer generation process, reduces the interference of inappropriate evidence on the generation process, and enhances the reliability and accuracy of the question-and-answer results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123068567A_ABST
    Figure CN123068567A_ABST
Patent Text Reader

Abstract

This disclosure relates to a method and system for generating professional question-and-answer results using a large-scale model based on multi-source evidence retrieval and dynamic confidence assessment. The method includes: acquiring a user query and performing multi-source evidence retrieval based on the user query to obtain a multi-source candidate evidence set; performing dynamic confidence assessment on each candidate piece of evidence in the multi-source candidate evidence set to obtain a comprehensive confidence score for each candidate piece of evidence; determining a target evidence set from the multi-source candidate evidence set based on the comprehensive confidence score of each candidate piece of evidence; constructing model input information containing generation constraints based on the user query and the target evidence set; wherein the generation constraints are used to limit the large-scale language model to use the target evidence set as the basis for generation; and inputting the model input information into the large-scale language model to obtain professional question-and-answer results corresponding to the user query. This disclosure can reduce the interference of low-quality evidence on question-and-answer generation and improve the factual accuracy and reliability of professional question-and-answer results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method and system for generating professional question answers for large models based on multi-source evidence retrieval and dynamic confidence assessment. Background Technology

[0002] With the development of Large Language Model (LLM) technology, LLM is increasingly being applied to professional question-answering scenarios in fields such as medicine, law, and finance. To improve the factual accuracy of professional question-answering results, related technologies typically employ Retrieval-Augmented Generation (RAG) methods. This involves retrieving relevant content from external knowledge sources based on the user's query and using this retrieved content as a reference for the LLM to generate answers. However, professional knowledge is usually distributed across different types of knowledge sources, and the quality and applicability of the content provided by these sources vary. Retrieval results may contain outdated, unreliable, or conflicting information. Directly using these retrieval results for question-answering generation can easily lead to interference from low-quality information in the LLM. Therefore, improving the quality controllability of multi-source evidence during the professional question-answering process is a major challenge currently faced.

[0003] In related technologies, search results obtained from external knowledge sources are typically provided directly or after processing as contextual information to large language models to assist in generating professional question-and-answer results. However, search results from different knowledge sources may vary in content quality and their support for the current user query. Related technologies struggle to effectively control the applicability of each search result as a basis for answer generation and its use in the generation process. This can lead to inappropriate search results still participating in model generation and interfering with the output content. Consequently, professional question-and-answer results may suffer from biased conclusions due to a lack of reliable information support, failing to meet the accuracy and reliability requirements of professional question-and-answer scenarios. Summary of the Invention

[0004] This disclosure provides a method and system for generating professional question answers for large models based on multi-source evidence retrieval and dynamic confidence assessment.

[0005] According to a first aspect of this disclosure, a large-scale professional question-answering generation method based on multi-source evidence retrieval and dynamic confidence assessment is provided, including: Obtain user queries and perform multi-source evidence retrieval based on user queries to obtain a multi-source candidate evidence set; Dynamic confidence assessments were performed on each candidate piece of evidence in the multi-source candidate evidence set to obtain the overall confidence of each candidate piece of evidence. The overall confidence of each candidate piece of evidence was obtained by fusing the assessment results of the source authority, content timeliness, semantic relevance, and multi-source consistency of the corresponding candidate evidence. Based on the overall confidence level of each candidate piece of evidence, the target evidence set is determined from the multi-source candidate evidence set; The model input information, which includes generation constraints, is constructed based on user queries and the target evidence set; wherein, the generation constraints are used to limit the large language model to use the target evidence set as the basis for generation. Input the model's input information into the large language model to obtain professional question-and-answer results corresponding to the user's query.

[0006] Furthermore, multi-source evidence retrieval is performed based on user queries to obtain a multi-source candidate evidence set, including: Standardize the semantic encoding of user queries to obtain the query semantic encoding results; Based on the query semantic encoding results, parallel retrieval is performed from structured knowledge sources, unstructured document libraries and semi-structured resources to obtain structured candidate evidence sets, unstructured candidate evidence sets and semi-structured candidate evidence sets respectively. By aggregating the structured candidate evidence set, the unstructured candidate evidence set, and the semi-structured candidate evidence set, a multi-source candidate evidence set is obtained; wherein, each candidate piece of evidence in the multi-source candidate evidence set has a source identifier used to identify the corresponding knowledge source.

[0007] Furthermore, dynamic confidence assessments are performed on each candidate piece of evidence in the multi-source candidate evidence set to obtain the overall confidence level of each candidate piece of evidence, including: For each candidate piece of evidence, the candidate evidence is evaluated based on source authority, content timeliness, semantic relevance, and multi-source consistency, and the evaluation results for each dimension of the candidate evidence are obtained. Each dimension's evaluation result is assigned a corresponding weight; the sum of all weights is 1. The overall confidence level of candidate evidence is obtained by weighted summation based on the evaluation results of each dimension and their corresponding weights.

[0008] Furthermore, for each candidate piece of evidence, the evaluation results for the source authority and the evaluation results for the timeliness of the content are obtained through the following processing: Based on the professional level and authority of the knowledge source to which the candidate evidence belongs, the candidate evidence is stratified and scored to obtain the assessment result of the source authority of the candidate evidence. Obtain the publication time and current time of the candidate evidence, and determine the time interval based on the publication time and current time; Based on the time interval and decay coefficient, the evaluation result of the content timeliness corresponding to the candidate evidence is calculated using an exponential decay function; among which, the evaluation result of the content timeliness decreases as the time interval increases.

[0009] Furthermore, for each candidate piece of evidence, the evaluation results of the semantic relevance and multi-source consistency are obtained through the following processing: The Sentence-BERT model is used to semantically encode the evidence content of user queries and candidate evidence, respectively, to obtain query semantic vectors and evidence semantic vectors. Calculate the cosine similarity between the query semantic vector and the evidence semantic vector to obtain the evaluation result of the semantic relevance of the candidate evidence; Based on the content consistency between candidate evidence and other candidate evidence in the multi-source candidate evidence set, the evaluation result of the multi-source consistency of the candidate evidence is determined.

[0010] Furthermore, based on the overall confidence level of each candidate piece of evidence, a target evidence set is determined from the multi-source candidate evidence set, including: The candidate evidence is sorted in descending order of overall confidence level to obtain a candidate evidence sequence. Based on a preset confidence threshold, candidate evidence with a comprehensive confidence level greater than the preset confidence threshold is selected from the candidate evidence sequence, resulting in a target evidence set composed of the selected candidate evidence.

[0011] Furthermore, based on the user query and the target evidence set, model input information containing generation constraints is constructed, including: Based on the degree of content overlap between candidate evidence in the target evidence set, the candidate evidence is clustered to remove duplicates, resulting in a deduplicated evidence set. Text summaries are generated for each candidate piece of evidence in the deduplicated evidence set to obtain the evidence summary corresponding to each candidate piece of evidence. For each candidate piece of evidence in the deduplicated evidence set, an evidence item corresponding to the candidate piece of evidence is generated based on the evidence summary corresponding to the candidate piece of evidence and the comprehensive confidence level of the candidate piece of evidence. Construct a structured evidence list based on each evidence item; The user query, the structured evidence list, and the generation constraints are combined to obtain the model input information.

[0012] Furthermore, the generation constraints include evidence use constraints and evidence citation constraints; The user query, the structured evidence list, and the generation constraints are combined to obtain the model input information, including: Assign evidence numbers to each evidence item in the structured evidence list to obtain a numbered evidence list; The user query, the numbered evidence list, the evidence usage constraint, and the evidence citation constraint are combined to obtain the model input information. Among them, the evidence usage constraint is used to limit the large language model to generate professional question and answer results based only on the numbered evidence list; the evidence citation constraint is used to limit the large language model to cite the evidence number corresponding to the conclusion in the professional question and answer results.

[0013] Furthermore, after inputting the model's input information into a large language model to obtain specialized question-and-answer results corresponding to the user's query, the method also includes: Semantic analysis is performed on the professional Q&A results to extract the main answer text and the evidence numbers cited in the main answer text; Identify each conclusion in the main answer text and the evidence number cited by each conclusion, and determine the evidence item corresponding to each conclusion from the list of numbered evidence based on the evidence number cited by each conclusion; Establish a mapping relationship between each conclusion and its corresponding evidence item.

[0014] Furthermore, after establishing the mapping relationship between each conclusion and its corresponding evidence item, the method also includes: Based on the mapping relationship, for each evidence item corresponding to each conclusion, the evidence content and source information of the candidate evidence corresponding to the evidence item are obtained, and the source information corresponding to the evidence item is obtained. For each piece of evidence corresponding to each conclusion, the source information corresponding to the evidence piece is marked with a color corresponding to the overall confidence level, based on the overall confidence level of the evidence piece, thus obtaining the visual source information corresponding to the evidence piece. The main answer text is displayed in the main answer area, and the source information area displays the visual source information of each evidence item corresponding to each conclusion.

[0015] According to a second aspect of this disclosure, a large-scale professional question-answering generation system based on multi-source evidence retrieval and dynamic confidence assessment is provided, comprising: The multi-source evidence retrieval module is used to obtain user queries and perform multi-source evidence retrieval based on user queries to obtain a set of multi-source candidate evidence. The dynamic confidence assessment module is used to dynamically assess the confidence of each candidate piece of evidence in the multi-source candidate evidence set to obtain the comprehensive confidence of each candidate piece of evidence. The comprehensive confidence of each candidate piece of evidence is obtained by fusing the assessment results of the source authority, content timeliness, semantic relevance and multi-source consistency of the corresponding candidate evidence. The target evidence determination module is used to determine the target evidence set from the multi-source candidate evidence set based on the overall confidence level of each candidate evidence. The model input construction module is used to construct model input information containing generation constraints based on user queries and the target evidence set; wherein, the generation constraints are used to limit the large language model to use the target evidence set as the basis for generation; The professional question-and-answer generation module is used to input the model input information into the large language model and obtain professional question-and-answer results corresponding to the user's query.

[0016] According to a third aspect of this disclosure, an electronic device is provided. The electronic device includes a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the program to implement the method described above.

[0017] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the methods described above.

[0018] According to a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes a computer program that, when executed by a processor, implements the methods described above.

[0019] This disclosure provides a method and system for generating professional question-and-answer questions using a large-scale model based on multi-source evidence retrieval and dynamic confidence assessment. In this embodiment, firstly, a user query is obtained, and multi-source evidence retrieval is performed based on the user query to obtain a set of multi-source candidate evidence. Then, dynamic confidence assessment is performed on each candidate piece of evidence in the multi-source candidate evidence set to obtain the comprehensive confidence score of each candidate piece of evidence. The comprehensive confidence score of each candidate piece of evidence is obtained by fusing the assessment results of the corresponding candidate evidence's source authority, content timeliness, semantic relevance, and multi-source consistency. Next, based on the comprehensive confidence score of each candidate piece of evidence, a target evidence set is determined from the multi-source candidate evidence set. Then, model input information containing generation constraints is constructed based on the user query and the target evidence set. The generation constraints are used to limit the large language model to use the target evidence set as the basis for generation. Finally, the model input information is input into the large language model to obtain professional question-and-answer results corresponding to the user query.

[0020] As described above, this embodiment of the disclosure provides richer candidate evidence for professional question-and-answer generation by performing multi-source evidence retrieval based on user queries. It dynamically evaluates the confidence level of each candidate piece of evidence based on source authority, content timeliness, semantic relevance, and multi-source consistency, and then integrates these evaluations to obtain a comprehensive confidence level. This comprehensive confidence level reflects the reliability of the candidate evidence, its support for the current user query, and its corroboration with other candidate evidence. Furthermore, it determines the target evidence set based on the comprehensive confidence level, enabling quality control of the evidence entering the generation process and reducing the interference of inappropriate candidate evidence. Finally, it utilizes generation constraints to limit the large language model to use the target evidence set as the basis for generation, controlling the generation boundary of the large language model and reducing the possibility of the large language model deviating from reliable evidence in generating content. Therefore, this embodiment of the disclosure can solve the technical problem in related technologies where the applicability of multi-source evidence as the basis for answer generation and its use in the large language model generation process are difficult to effectively control, leading to insufficient factual accuracy and reliability of professional question-and-answer results. Attached Figure Description

[0021] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0022] Figure 1 A flowchart of a large-model professional question-answering generation method based on multi-source evidence retrieval and dynamic confidence assessment provided in this embodiment of the disclosure; Figure 2 A flowchart of the multi-source evidence retrieval method provided in the embodiments of this disclosure; Figure 3 A schematic diagram of a multi-source heterogeneous parallel retrieval process provided in an embodiment of this disclosure; Figure 4 A flowchart of the dynamic confidence assessment method provided in the embodiments of this disclosure; Figure 5 A schematic diagram illustrating the candidate evidence evaluation and processing flow provided in the embodiments of this disclosure; Figure 6 A flowchart illustrating a method for determining a target set of evidence provided in this embodiment of the disclosure; Figure 7 A flowchart illustrating the method for inputting model building information provided in this embodiment of the disclosure; Figure 8 A flowchart illustrating a method for establishing a mapping relationship between conclusions and evidence items, provided in this embodiment of the disclosure; Figure 9 A flowchart of a method for generating and displaying traceability information provided in this embodiment of the disclosure; Figure 10 A structural block diagram of a large-scale professional question-answering generation system based on multi-source evidence retrieval and dynamic confidence assessment provided in this embodiment of the disclosure; Figure 11 A structural block diagram of an electronic device provided as an exemplary embodiment of this disclosure; Figure 12 A structural block diagram of a computer system provided as an exemplary embodiment of this disclosure; Figure 13 A structural block diagram of a computer program product provided for an exemplary embodiment of this disclosure. Detailed Implementation

[0023] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0024] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0025] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0026] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0027] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0028] To facilitate understanding of the technical solutions provided in this disclosure, the following uses an application scenario of a professional knowledge service platform as an example to illustrate the applicable application environment and related technical problems. In this application scenario, users can submit queries related to a specific professional field to the professional knowledge service platform via a terminal. The professional knowledge service platform can obtain information related to the query from multiple external knowledge sources and provide the obtained information directly or after simple processing to a large language model to generate corresponding professional question-and-answer results. Since different knowledge sources may differ in terms of information coverage, publication time, applicable conditions, and content description, the information obtained for the same query may have duplicate content, inconsistent versions, different levels of actual support for the query, or conflicting information. If the suitability of the above information for current professional question-and-answer generation and its use in the generation process is not effectively controlled, some unsuitable information may interfere with the generation process of the large language model, causing the generated professional question-and-answer results to lack reliable information support or lead to biased conclusions, making it difficult for users to judge the credibility of the professional question-and-answer results. Based on the above application scenario, the following describes the large-scale model professional question-and-answer generation method and system provided in this disclosure, in conjunction with the accompanying drawings and specific embodiments.

[0029] In one embodiment, such as Figure 1 As shown, a large-scale professional question-answering generation method based on multi-source evidence retrieval and dynamic confidence assessment is provided, including the following steps: Step 101: Obtain the user query and perform multi-source evidence retrieval based on the user query to obtain a multi-source candidate evidence set.

[0030] Here, the entity executing step 101 can be a question-answering system deployed on a cloud-based inference server. This system can be applied to intelligent question-answering scenarios in professional fields such as medicine, law, and finance. User queries can be natural language queries input by the user that relate to the relevant professional field. After obtaining the user query, the question-answering system can trigger multi-source evidence retrieval and search for candidate evidence related to the user query from multiple knowledge sources, obtaining a multi-source candidate evidence set. This set is used for subsequent dynamic confidence assessment of each candidate piece of evidence.

[0031] In one possible embodiment, such as Figure 2 As shown, multi-source evidence retrieval based on user queries yields a multi-source candidate evidence set, including the following steps: Step 201: Perform standardized semantic encoding on the user query to obtain the query semantic encoding result.

[0032] Here, the question-answering system can standardize the semantic encoding of user queries in natural language form, converting the user query into a query semantic encoding result that can represent its query semantics. The query semantic encoding result is used to provide query semantic basis for the retrieval branches corresponding to different knowledge sources.

[0033] In one possible embodiment, the question-answering system can perform semantic parsing on user queries to identify core entities and keywords, and encode the user queries using a text semantic coding model to obtain a query semantic vector. Then, based on the core entities, core keywords, and query semantic vector, a query semantic coding result is formed. The core entities can be used to construct query statements for structured knowledge sources, the query semantic vector can be used for semantic similarity retrieval in unstructured document libraries, and the core keywords and query semantic vector can be used for keyword matching and semantic matching in semi-structured resources. Thus, the query semantic coding result can be used for corresponding retrieval branches according to the knowledge organization forms and retrieval methods of different knowledge sources.

[0034] Step 202: Based on the query semantic encoding results, perform parallel retrieval from structured knowledge sources, unstructured document libraries, and semi-structured resources to obtain structured candidate evidence sets, unstructured candidate evidence sets, and semi-structured candidate evidence sets, respectively.

[0035] Here, the question-answering system can trigger retrieval branches corresponding to structured knowledge sources, unstructured document libraries, and semi-structured resources based on the query semantic encoding results. Each retrieval branch retrieves candidate evidence related to the user's query from its corresponding knowledge source, without relying on the retrieval results of other retrieval branches as a prerequisite for execution.

[0036] Figure 3 The present disclosure provides an exemplary implementation process for a parallel retrieval of three types of knowledge sources, such as... Figure 3 As shown, the multi-source heterogeneous retrieval engine can distribute the semantically encoded query results to retrieval branches corresponding to structured knowledge sources, unstructured document libraries, and semi-structured resources. The retrieval branch corresponding to the structured knowledge source can obtain structured knowledge evidence through graph queries; the retrieval branch corresponding to the unstructured document library can obtain relevant text paragraphs through semantic vector retrieval; and the retrieval branch corresponding to the semi-structured resources can obtain question-answer pairs or standardized entries through a combination of keyword matching and semantic matching. The three retrieval branches output sets of structured candidate evidence, unstructured candidate evidence, and semi-structured candidate evidence, respectively.

[0037] In one possible embodiment, the structured knowledge source may include a knowledge graph of the corresponding professional field, such as a medical knowledge graph or a legal statute graph. The question answering system can determine the core entity in the user query based on the query semantic encoding result, and use graph query language to construct entity association query statements corresponding to the core entity. It can retrieve triple relationships and attribute information related to the core entity from the knowledge graph to obtain a set of structured candidate evidence.

[0038] The unstructured document library can include unstructured professional documents in the corresponding professional field, such as academic papers, drug instructions, or financial research reports. The question answering system can use a sentence vector model (Sentence-BERT) based on the Bidirectional Encoder Representations from Transformers (BERT) model to encode the user query into a query semantic vector, and perform cosine similarity retrieval in a vector database that stores the semantic vectors corresponding to unstructured professional documents. Text paragraphs that meet the similarity requirements are identified as unstructured candidate evidence, resulting in a set of unstructured candidate evidence. For example, the top five text paragraphs with the highest to lowest cosine similarity can be identified as unstructured candidate evidence.

[0039] In one possible embodiment, the Sentence-BERT sentence vector model can employ a pre-trained model version and include a text encoder and pooling layers with shared model parameters. The question answering system can use the text encoder to encode the evidence content of the user query and candidate evidence respectively, and use the pooling layer to convert the encoding results into query semantic vectors and evidence semantic vectors of the same dimension so as to calculate the cosine similarity between the two. The input length, output vector dimension, and pooling method of the text encoder can be preset according to the selected model version.

[0040] Semi-structured resources can include authoritative Q&A resources or industry standard manuals in the relevant professional fields. The Q&A system can determine the core keywords in the user's query based on the query semantic encoding results, and use a combination of keyword matching and lightweight semantic models to retrieve Q&A pairs or standard entries that contain the core keywords and are marked as verified from the semi-structured resources. The retrieved Q&A pairs or standard entries are identified as semi-structured candidate evidence, resulting in a set of semi-structured candidate evidence.

[0041] In one possible embodiment, the three types of knowledge sources mentioned above can be pre-constructed. Specifically, entities, relationships, and attributes can be extracted from professional domain databases, standard documents, or professional documents and stored as knowledge graphs to form structured knowledge sources; text cleaning and segmentation can be performed on professional documents such as academic papers, drug instructions, or financial research reports, and each text segment and its semantic vector can be associated and stored in a vector database to form an unstructured document library; and question-and-answer pairs and standard entries in authoritative question-and-answer resources or industry standard manuals can be organized according to preset fields to form semi-structured resources.

[0042] Step 203: Summarize the structured candidate evidence set, the unstructured candidate evidence set, and the semi-structured candidate evidence set to obtain the multi-source candidate evidence set.

[0043] Here, the question-answering system can summarize the structured candidate evidence set, unstructured candidate evidence set, and semi-structured candidate evidence set output by each retrieval branch, and retain the correspondence between each candidate evidence and its knowledge source to obtain a multi-source candidate evidence set. Each candidate evidence in the multi-source candidate evidence set has a source identifier used to identify the corresponding knowledge source. The source identifier can be used to identify the knowledge source to which the candidate evidence belongs in subsequent processing, and provide a basis for assessing the source authority of the candidate evidence and tracing the source of professional question-answering results.

[0044] In one possible embodiment, the question-answering system can read each candidate piece of evidence from the structured candidate evidence set, the unstructured candidate evidence set, and the semi-structured candidate evidence set, and associate the evidence content of each candidate piece of evidence with its corresponding source identifier to form a corresponding candidate evidence record. Then, the formed candidate evidence records are aggregated into the same set to obtain a multi-source candidate evidence set. For example, candidate evidence from structured knowledge sources, unstructured document libraries, and semi-structured resources can have source identifiers such as "KG", "PDF", and "Q&A", where KG (Knowledge Graph) is used to identify the knowledge graph source, PDF (Portable Document Format) is used to identify the PDF document source, and Q&A (Question and Answer) is used to identify the question and answer resource source. The above source identifiers are only used to distinguish the retrieval source of candidate evidence, and their specific character format is not limited.

[0045] During the aggregation process, candidate evidence from different knowledge sources that has the same or similar content can be retained separately for subsequent multi-source consistency assessment based on the content consistency among different candidate evidence. For candidate evidence with highly overlapping content, clustering and deduplication can be performed after determining the target evidence set based on the comprehensive confidence level.

[0046] Through steps 201 to 203 above, the question-answering system converts user queries in natural language form into query semantic encoding results usable by different retrieval branches. It then employs appropriate retrieval methods for structured knowledge sources, unstructured document databases, and semi-structured resources in parallel retrieval. This leverages the complementarity of different knowledge sources in terms of knowledge organization and information coverage to obtain more comprehensive professional candidate evidence. Furthermore, by retaining the corresponding source identifiers when summarizing each candidate piece of evidence, a correspondence between candidate evidence and its knowledge source can be established, providing a data foundation for subsequent assessments of the source authority and multi-source consistency of each candidate piece of evidence.

[0047] Step 102: Perform dynamic confidence assessment on each candidate piece of evidence in the multi-source candidate evidence set to obtain the comprehensive confidence of each candidate piece of evidence.

[0048] Here, the executing entity of step 102 can be a question-and-answer system. After obtaining the multi-source candidate evidence set, the question-and-answer system can dynamically evaluate the confidence level of each candidate evidence in the multi-source candidate evidence set based on the current user query, the candidate evidence itself, the knowledge source to which the candidate evidence belongs, the publication time of the candidate evidence, and the content relationship between the candidate evidence and other candidate evidence. The comprehensive confidence level of each candidate evidence is used to determine the target evidence set from the multi-source candidate evidence set in subsequent processing.

[0049] The overall confidence level of each candidate piece of evidence is obtained by integrating the evaluation results of the source authority, content timeliness, semantic relevance, and multi-source consistency of the corresponding candidate evidence. The evaluation result of source authority is used to characterize the professional level and authority of the knowledge source to which the candidate evidence belongs. The evaluation result of content timeliness is used to characterize the validity of the candidate evidence relative to the current time. The evaluation result of semantic relevance is used to characterize the semantic relevance between the evidence content of the candidate evidence and the current user query. The evaluation result of multi-source consistency is used to characterize the content consistency between the candidate evidence and other candidate evidence.

[0050] For ease of explanation, a multi-source candidate evidence set can be represented as: , where e i Let represent the i-th candidate evidence in the multi-source candidate evidence set, and n represent the number of candidate evidences in the multi-source candidate evidence set.

[0051] Figure 4 An exemplary implementation of step 102 provided in an embodiment of this disclosure is illustrated. For example... Figure 4As shown, the specific implementation process includes steps 401 to 403. Step 401 is used to evaluate each candidate evidence from four dimensions: source authority, content timeliness, semantic relevance, and multi-source consistency, to obtain the evaluation results of each dimension corresponding to the candidate evidence. Step 402 is used to set the corresponding weights for the evaluation results of each dimension. Step 403 is used to perform a weighted sum based on the evaluation results of each dimension and their corresponding weights to obtain the comprehensive confidence level of the candidate evidence.

[0052] In one possible embodiment, such as Figure 4 As shown, dynamic confidence assessments are performed on each candidate piece of evidence in the multi-source candidate evidence set to obtain the overall confidence level of each candidate piece of evidence. The process includes the following steps: Step 401: For each candidate piece of evidence, evaluate it based on source authority, content timeliness, semantic relevance, and multi-source consistency to obtain the evaluation results for each dimension of the candidate evidence.

[0053] Here, the question-answering system can determine the evaluation results of the candidate evidence in terms of source authority, content timeliness, semantic relevance, and multi-source consistency. The evaluation results of different dimensions reflect the performance of the candidate evidence in the corresponding evaluation dimensions and together serve as the basic data for calculating the overall confidence level of the candidate evidence.

[0054] In one possible embodiment, for each candidate piece of evidence, the evaluation results of the source authority and the evaluation results of the content timeliness are obtained through the following processing: Based on the professional level and authority of the knowledge source to which the candidate evidence belongs, the candidate evidence is stratified and scored to obtain the assessment result of the source authority of the candidate evidence. Obtain the publication time and current time of the candidate evidence, and determine the time interval based on the publication time and current time; Based on the time interval and decay coefficient, the evaluation result of the content timeliness corresponding to the candidate evidence is calculated using an exponential decay function; among which, the evaluation result of the content timeliness decreases as the time interval increases.

[0055] Specifically, the question-answering system can pre-establish a correspondence between knowledge sources and their authority scores based on the professional level and authority of different knowledge sources. It can then query this correspondence based on the knowledge source to which the current candidate evidence belongs to obtain an assessment of the source authority of that candidate evidence. For example, a knowledge graph might have an authority score of 0.9, an academic journal 0.8, and a general question-answering community 0.5. These scores are merely examples and can be set according to the actual authority of different knowledge sources in their respective professional fields.

[0056] In one possible implementation, for content timeliness, the question-answering system can calculate the evaluation result of the content timeliness corresponding to the candidate evidence according to the following exponential decay function:

[0057] in, This represents the assessment result of the timeliness of the content corresponding to the current candidate evidence, where e represents the natural constant. Indicates the attenuation coefficient. This represents the time interval between the release time of the current candidate evidence and the current time. When using years as the unit of time, Use the attenuation coefficient corresponding to this time unit. For example, A value of 0.3 can be used to significantly reduce the timeliness assessment results of candidate evidence whose publication time is more than three years away from the current time. The attenuation coefficient can also be adjusted according to the timeliness requirements of information in different professional fields.

[0058] In one possible embodiment, for each candidate piece of evidence, the evaluation results of the semantic relevance and the multi-source consistency are obtained through the following processing: The Sentence-BERT model is used to semantically encode the evidence content of user queries and candidate evidence, respectively, to obtain query semantic vectors and evidence semantic vectors. Calculate the cosine similarity between the query semantic vector and the evidence semantic vector to obtain the evaluation result of the semantic relevance of the candidate evidence; Based on the content consistency between candidate evidence and other candidate evidence in the multi-source candidate evidence set, the evaluation result of the multi-source consistency of the candidate evidence is determined.

[0059] Specifically, regarding the current candidate evidence e i Question answering systems can use the Sentence-BERT model to combine user query q and current candidate evidence e. i The evidence content is encoded into query semantic vector v. q and the semantic vector of evidence v ei And calculate the cosine similarity between the query semantic vector and the evidence semantic vector according to the following formula:

[0060] in, This represents the evaluation result of the semantic relevance of the current candidate evidence; cosine(·) represents the cosine similarity operation, v q Represents the query semantic vector, v eiThis represents the semantic vector of the current candidate evidence. The evaluation result of semantic relevance can range from [0, 1]. The larger the value, the higher the semantic relevance between the content of the current candidate evidence and the user query.

[0061] For multi-source consistency, the question-answering system can compare the content of the current candidate evidence with the content of other candidate evidence in the multi-source candidate evidence set, and determine the evaluation result of the multi-source consistency of the current candidate evidence based on the consistency between the other candidate evidence and the current candidate evidence in terms of factual content, relational content, or conclusion content.

[0062] For example, a question-answering system can count the number of other candidate evidences that are consistent with the current candidate evidence, and calculate the proportion of this number in the total number of other candidate evidences participating in the consistency comparison. The obtained proportion of consistent evidence is used to determine the evaluation result of multi-source consistency. The evaluation result of multi-source consistency can range from [0, 1]. When the proportion of consistent evidence is greater than 80%, the evaluation result of multi-source consistency can be determined as 1.0. When the proportion of consistent evidence is less than 50%, the evaluation result of multi-source consistency can be determined as 0.3. The evaluation results corresponding to the proportions of other consistent evidence can be determined according to preset scoring rules.

[0063] Step 402: Set the corresponding weights for the evaluation results of each dimension.

[0064] Here, the question-answering system can assign weights w1, w2, w3, and w4 to the evaluation results of source authority, content timeliness, semantic relevance, and multi-source consistency, respectively, where the sum of each weight is 1. .

[0065] In one possible implementation, the question-answering system can determine the weight configuration corresponding to the professional field of the current user's query, and set the weights of the evaluation results for each dimension according to the determined weight configuration. For example, for different professional fields such as medicine, law, and finance, corresponding weights can be configured according to the different requirements of each professional field for source authority, content timeliness, semantic relevance, and multi-source consistency. The specific values ​​of each weight can be pre-stored in the question-answering system and invoked when dynamically evaluating the confidence of candidate evidence corresponding to the current user's query.

[0066] Step 403: Based on the evaluation results of each dimension and their corresponding weights, perform a weighted summation to obtain the overall confidence level of the candidate evidence.

[0067] Here, the question-answering system can perform a weighted sum of the evaluation results of each dimension corresponding to the current candidate evidence and the weights corresponding to the evaluation results of each dimension, and determine the weighted sum as the overall confidence level of the current candidate evidence.

[0068] Candidate evidence e i Overall confidence level The following formula can be used for calculation:

[0069] in, This represents the overall confidence level of candidate evidence ei. , , and Each represents candidate evidence e. i The corresponding evaluation results for source authority, content timeliness, semantic relevance, and multi-source consistency are represented by w1, w2, w3, and w4, which represent the weights of the evaluation results for each dimension.

[0070] In one possible embodiment, the question-answering system can substitute the evaluation results of the source authority, content timeliness, semantic relevance and multi-source consistency of the current candidate evidence, as well as the weights corresponding to the evaluation results of each dimension, into the above comprehensive confidence calculation formula to obtain the comprehensive confidence of the current candidate evidence and establish the correspondence between the current candidate evidence and its comprehensive confidence.

[0071] Figure 5 An exemplary embodiment of the present disclosure provides a candidate evidence evaluation and processing flow, such as... Figure 5 As shown, the question-answering system can use a multi-source candidate evidence set as the evidence set to be processed, sequentially read the candidate evidence, and execute steps 401 to 403 for the currently read candidate evidence. After obtaining the comprehensive confidence score of the current candidate evidence, the question-answering system determines whether the processing of all candidate evidence has been completed. If not, it continues to read the next candidate evidence and calculate the corresponding comprehensive confidence score. If completed, it sorts the candidate evidence according to the comprehensive confidence score of each candidate evidence and filters it based on a confidence score threshold to obtain the target evidence set. Subsequently, the question-answering system can also cluster and deduplicate candidate evidence with overlapping content in the target evidence set, generate text summaries for the deduplicated candidate evidence, and construct a structured evidence list based on the obtained evidence summaries. It should be noted that... Figure 5 The “initial evidence set” shown corresponds to the aforementioned multi-source candidate evidence set, the “overall confidence score” corresponds to the aforementioned overall confidence score, and the “high-quality evidence set” corresponds to the aforementioned target evidence set.

[0072] in, Figure 5The candidate evidence traversal and comprehensive confidence calculation process shown can be considered as a specific implementation of step 102; the comprehensive confidence ranking and threshold screening process can be considered as a specific implementation of subsequent step 103; and the clustering deduplication, text summarization generation, and structured evidence list construction process can be considered as a specific implementation of subsequent step 104. Figure 5 This is used to illustrate the data flow and execution order between the above processes, and does not mean that step 102 must include the processes in steps 103 and 104 at the same time.

[0073] Through steps 401 to 403 above, the question-answering system evaluates each candidate piece of evidence from four dimensions: source authority, content timeliness, semantic relevance, and multi-source consistency. The evaluation results of each dimension are weighted and integrated, which can comprehensively characterize the applicability of each candidate piece of evidence as the basis for generating current professional questions and answers from aspects such as evidence source, information timeliness, query matching, and mutual corroboration between evidence. This provides a quantitative basis for determining the target evidence set from the multi-source candidate evidence set in the future.

[0074] Step 103: Determine the target evidence set from the multi-source candidate evidence set based on the overall confidence level of each candidate evidence.

[0075] Here, the question-answering system can use the comprehensive confidence level of each candidate evidence obtained in step 102 as the evidence admission criterion to sort and filter the candidate evidence in the multi-source candidate evidence set to obtain the target evidence set. The target evidence set is used to provide an evidentiary basis for the input information of the subsequent model construction.

[0076] In one possible embodiment, such as Figure 6 As shown, the target evidence set is determined from the multi-source candidate evidence set based on the overall confidence level of each candidate piece of evidence, including the following steps: Step 601: Sort the candidate evidence in descending order of the overall confidence level of each candidate piece of evidence to obtain the candidate evidence sequence.

[0077] Here, the question-answering system can sort the candidate evidence in the multi-source candidate evidence set in descending order based on the relationship between each candidate piece of evidence and its corresponding overall confidence level, using the overall confidence level as the sorting criterion. The candidate evidence that is ranked earlier in the candidate evidence sequence has a higher overall confidence level.

[0078] In one possible embodiment, the question-answering system can establish corresponding ranking records for each candidate piece of evidence. Each ranking record includes the evidence identifier and overall confidence level of the candidate evidence. The question-answering system sorts the candidate evidence in descending order based on the overall confidence level in each ranking record and arranges the corresponding candidate evidence according to the ranking results to obtain a candidate evidence sequence. During the formation of the candidate evidence sequence, the correlation between the candidate evidence and its evidence content, source information, and overall confidence level can be maintained for subsequent screening processing.

[0079] Step 602: Based on a preset confidence threshold, select each candidate evidence with a comprehensive confidence level greater than the preset confidence threshold from the candidate evidence sequence to obtain the target evidence set composed of the selected candidate evidence.

[0080] Here, the preset confidence threshold is used to characterize the confidence level requirement that candidate evidence must meet to enter the question-answering generation stage. The question-answering system can compare the comprehensive confidence level of each candidate evidence in the candidate evidence sequence with the preset confidence threshold, and retain the candidate evidence with a comprehensive confidence level greater than the preset confidence threshold in the target evidence set.

[0081] In one possible embodiment, a pre-set reliability threshold can be pre-set according to the requirements of evidence reliability in a professional question-and-answer scenario. The question-and-answer system can sequentially read the comprehensive confidence level corresponding to each candidate piece of evidence according to the order of arrangement in the candidate evidence sequence, and determine whether the comprehensive confidence level is greater than the pre-set reliability threshold. If the comprehensive confidence level is greater than the pre-set reliability threshold, the corresponding candidate evidence is added to the target evidence set; if the comprehensive confidence level is less than or equal to the pre-set reliability threshold, the corresponding candidate evidence is not added to the target evidence set. Each candidate piece of evidence in the target evidence set can continue to retain its corresponding evidence content, source information, evaluation results of each dimension, and comprehensive confidence level for use in the construction of subsequent model input information.

[0082] In one possible embodiment, if no candidate evidence is obtained after screening according to a preset reliability threshold, resulting in an empty target evidence set, the question answering system can determine that the current evidence is insufficient and add a rejection instruction when inputting information to the model in subsequent model construction, so that the large language model outputs a "insufficient evidence" prompt message.

[0083] Through steps 601 to 602 above, the question-answering system can first establish the confidence order among the candidate evidence, and then use a pre-set confidence threshold as the evidence admission condition to determine the candidate evidence that meets the confidence requirements from the candidate evidence sequence. This enables the formed target evidence set to provide a confidence-filtered evidence basis for subsequent question-answering generation, reducing the interference of candidate evidence with low overall confidence on the question-answering generation process.

[0084] Step 104: Construct model input information containing generation constraints based on user queries and the target evidence set.

[0085] Here, after obtaining the target evidence set, the question-answering system can perform deduplication, summarization, and structuring processing on each candidate piece of evidence in the target evidence set. The processed evidence information is then combined with the user query and generation constraints to obtain the model input information. This model input information is used to subsequently input into a large language model to generate professional question-answering results corresponding to the user query. The generation constraints are used to limit the large language model to use the target evidence set as the basis for generation. The generation constraints can specify how the large language model uses the target evidence set and how evidence is cited in the professional question-answering results.

[0086] In one possible embodiment, such as Figure 7 As shown, the model input information containing generation constraints is constructed based on user queries and the target evidence set, including the following steps: Step 701: Based on the degree of content overlap between candidate evidences in the target evidence set, cluster the candidate evidences to remove duplicates, and obtain a deduplicated evidence set.

[0087] Here, the question-answering system can compare the evidence content of each candidate piece of evidence in the target evidence set, and classify candidate pieces of the same or highly overlapping content into the same evidence cluster based on the degree of overlap obtained from the comparison. Then, based on the formed evidence clusters, duplicate candidate pieces of evidence are removed to obtain a deduplicated evidence set.

[0088] In one possible embodiment, the question-answering system can traverse each evidence cluster, merge or remove duplicate candidate evidence in the same evidence cluster, and summarize the processed candidate evidence to form a deduplicated evidence set. During the clustering and deduplication process, the correlation between each candidate evidence and its source identifier, evaluation results of each dimension, and overall confidence level can be retained for subsequent processing.

[0089] Step 702: Generate text summaries for each candidate piece of evidence in the deduplicated evidence set to obtain the evidence summary corresponding to each candidate piece of evidence.

[0090] Here, the question-and-answer system can perform summary processing on the evidence content of each candidate evidence in the deduplicated evidence set, extract information that can characterize the main content of the corresponding candidate evidence, and obtain the evidence summary corresponding to each candidate evidence. The evidence summary is used to generate the evidence item corresponding to the candidate evidence in the subsequent process.

[0091] In one possible embodiment, the question-answering system can extract key content related to the user's query from the evidence content of each candidate evidence, and compress repetitive or redundant expressions in the candidate evidence to form a corresponding evidence summary while maintaining the main meaning of the candidate evidence.

[0092] Step 703: For each candidate piece of evidence in the deduplicated evidence set, generate the corresponding evidence item based on the evidence summary and the comprehensive confidence level of the candidate piece of evidence.

[0093] Here, the question-answering system can associate the evidence summary corresponding to each candidate piece of evidence with its overall confidence level, and organize them according to a preset item format to obtain the evidence items corresponding to the candidate pieces of evidence. The evidence items are used to record candidate evidence information in a structured form for use by the large language model.

[0094] In one possible embodiment, each evidence entry may include the evidence content of the corresponding candidate evidence, source identifier, assessment result of source authority, assessment result of content timeliness, assessment result of semantic relevance, assessment result of multi-source consistency, and overall confidence level. The evidence content in the evidence entry may be the evidence summary generated in step 702, and the association between the evidence entry and the complete evidence content of the corresponding candidate evidence may be preserved for subsequent source tracing.

[0095] Step 704: Construct a structured evidence list based on each evidence item.

[0096] Here, the question-and-answer system can organize each piece of evidence according to a unified data structure, so that the evidence content, source identification, evaluation results of each dimension and overall confidence level contained in each piece of evidence are interconnected, resulting in a structured evidence list.

[0097] In one possible embodiment, the question-answering system can arrange the evidence items in descending order of their overall confidence level, so that the evidence items with higher overall confidence levels are placed at the beginning of the structured evidence list, thus obtaining a structured evidence list sorted by overall confidence level.

[0098] Step 705: Combine the user query, the structured evidence list, and the generation constraints to obtain the model input information.

[0099] Here, the question-answering system can use user queries as questions to be answered, a list of structured evidence as evidence to answer user queries, and generation constraints as instructions to control the use and citation of evidence by the large language model. By combining the above, the model input information is obtained.

[0100] In one possible embodiment, the generation constraints include evidence use constraints and evidence citation constraints; The user query, structured evidence list, and generation constraints are combined to obtain the model input information, including the following steps: Assign evidence numbers to each evidence item in the structured evidence list to obtain a numbered evidence list; The user query, the numbered evidence list, the evidence usage constraint, and the evidence citation constraint are combined to obtain the model input information. Among them, the evidence usage constraint is used to limit the large language model to generate professional question and answer results based only on the numbered evidence list; the evidence citation constraint is used to limit the large language model to cite the evidence number corresponding to the conclusion in the professional question and answer results.

[0101] Specifically, in one possible implementation, the model input information can take the form of standardized prompt text, and an example format of the prompt text is shown in the table below:

[0102] It should be noted that the format of the above-mentioned prompt text is only used as an example to illustrate the organization of model input information and does not constitute a limitation. In other possible embodiments, the expression, arrangement order, field names, number of evidence items, and evidence numbering format of each component in the prompt text can be adjusted according to the professional field, user query type, or input requirements of the large language model. Instructions related to the corresponding professional field or answering task can also be added, as long as the adjusted prompt text includes the user query, evidence information for answering the user query, and generation constraints, and can limit the large language model to generate answers based on the corresponding evidence information and establish the association between the generated conclusion and the corresponding evidence.

[0103] In the table above, {Original User Query Question} is used to fill in the user query, {Evidence 1 Content} and {Evidence 2 Content} are used to fill in the evidence summaries of the corresponding candidate evidence, {Source Identifier 1} and {Source Identifier 2} are used to fill in the source identifiers of the corresponding candidate evidence, {C1} and {C2} represent the overall confidence level of the corresponding candidate evidence, [E1] and [E2] are the evidence numbers of the corresponding evidence items, the ellipsis indicates that the numbered evidence list may include other evidence items, and "Answer:" is used to instruct the large language model to generate a professional question-and-answer result at this position.

[0104] In the table, "No information not mentioned in the evidence shall be fabricated" and "If the evidence is insufficient to answer the question, please state directly that the evidence is insufficient" can be used as specific content of the constraints on the use of evidence; "Cite the evidence number [E1] [E2] etc. at the corresponding conclusion" can be used as specific content of the constraints on the citation of evidence.

[0105] For example, in a medical Q&A scenario, the following domain instruction can be added to the above prompt text: "Please answer the user's query based only on the medical domain evidence information in the numbered evidence list, and mark the corresponding evidence number at each conclusion; if the existing evidence cannot support the answer, please output 'Insufficient evidence'." In other professional Q&A scenarios such as law and finance, the above domain instruction can be adjusted according to the evidence type of the corresponding domain, while the organization of user queries, numbered evidence lists, evidence use constraints, and evidence citation constraints can remain unchanged.

[0106] Through steps 701 to 705 above, the question-answering system can perform clustering, deduplication, and summarization processing on the target evidence set, and organize the processed evidence content, source information, and comprehensive confidence into a structured evidence list. This list is then combined with user queries and generation constraints to form model input information, thereby reducing the interference of duplicate evidence on the model input and providing a basis for limiting the scope of evidence use in large language models and establishing the correspondence between generated conclusions and evidence.

[0107] Step 105: Input the model input information into the large language model to obtain professional question and answer results corresponding to the user's query.

[0108] Here, the question-answering system can input the model input information obtained in step 104 into a domain-adapted large language model. The large language model then performs autoregression based on the user query, numbered evidence list, and generation constraints in the model input information to generate professional question-answering results corresponding to the user query. These professional question-answering results can be used to subsequently extract the main answer text and evidence numbers, and to establish the association between the conclusions in the answer and the corresponding evidence.

[0109] In one possible implementation, the large language model can be selected from the Tongyi Qianwen series, GPT series, or Deep Exploration series, which are capable of autoregressive generation based on prompt text, and deployed on a cloud-based inference server. The question-answering system can send model input information to the cloud-based inference server through the model call interface and receive professional question-answering results returned by the cloud-based inference server. During the autoregressive generation process, the large language model can progressively generate professional question-answering results based on the already generated content and the model input information. Since the evidence items in the numbered evidence list can be arranged in descending order of overall confidence, evidence content with higher overall confidence can be placed at the beginning of the model input information, so that the large language model can prioritize the use of corresponding evidence content. The specific model of the large language model can be selected according to the professional field and deployment conditions.

[0110] In the process of generating professional question-and-answer results, evidence usage constraints can limit the large language model to generate results only based on the evidence content in the numbered evidence list, reducing the generation of information not recorded in the numbered evidence list. Evidence citation constraints can limit the large language model to add the evidence number corresponding to the evidence item on which the conclusion is based when generating the corresponding conclusion. When the evidence in the numbered evidence list is insufficient to answer the user's query, the large language model can generate a "insufficient evidence" prompt message according to the constraint instructions in the model input information. Therefore, professional question-and-answer results can include the main answer text and the evidence numbers cited in the main answer text, or include a prompt message indicating that the current evidence is insufficient to answer the user's query.

[0111] Through steps 101 to 105 above, the question-answering system can first obtain candidate evidence from multiple knowledge sources, then determine the target evidence for question-answer generation through dynamic confidence assessment and comprehensive confidence screening, and use generation constraints to limit the way the large language model uses the target evidence, so that the evidence entering the generation stage and its usage process are controlled, thereby reducing the interference of inappropriate candidate evidence on the generation process and the situation where the large language model deviates from the evidence generation content, and improving the factual accuracy and reliability of professional question-answering results.

[0112] In one possible embodiment, such as Figure 8 As shown, after inputting the model input information into the large language model and obtaining the professional question-and-answer results corresponding to the user query, the method also includes: Step 801: Perform semantic analysis on the professional Q&A results to extract the main answer text and the evidence numbers cited in the main answer text.

[0113] Here, the question-and-answer system can perform semantic analysis on the professional question-and-answer results obtained in step 105 through the result presentation module, identify the natural language content used to answer the user's query in the professional question-and-answer results and the evidence citation marks contained therein, obtain the main answer text and the evidence number cited in the main answer text, and use the extracted main answer text and evidence number to subsequently determine each conclusion in the main answer text and its corresponding evidence item.

[0114] In one possible embodiment, the question-answering system can parse the professional question-answering results based on the evidence number format set in the model input information. For example, when the evidence numbers adopt formats such as [E1] and [E2], the question-answering system can identify each evidence number in the professional question-answering results and record the reference position of each evidence number in the professional question-answering results. At the same time, the question-answering system can extract the natural language text used to directly answer the user's query from the professional question-answering results as the main answer text, and retain the corresponding evidence number in the main answer text.

[0115] Step 802: Determine each conclusion in the main answer text and the evidence number cited by each conclusion, and determine the evidence item corresponding to each conclusion from the list of numbered evidence based on the evidence number cited by each conclusion.

[0116] Here, the question-and-answer system can determine the conclusions in the main answer text and the evidence numbers cited by each conclusion based on the semantic content of the main answer text and the reference positions of each evidence number within the main answer text. The system further uses the evidence numbers cited by each conclusion as the query basis to search for evidence entries with the same evidence numbers from the numbered evidence list obtained in step 705, thus obtaining the evidence entries corresponding to each conclusion. Here, a conclusion can be a statement or semantic fragment in the main answer text used to express a relatively independent answer or judgment result. A conclusion can cite one evidence number or multiple evidence numbers simultaneously.

[0117] In one possible embodiment, the question-answering system can semantically segment the main answer text to obtain multiple conclusions that express the corresponding answer content. Based on the position of each evidence number in the main answer text, the system can determine the conclusion corresponding to each evidence number. When a conclusion refers to an evidence number, the question-answering system can determine an evidence entry corresponding to that evidence number from the list of numbered evidence. When a conclusion refers to multiple evidence numbers, the system can determine the evidence entry corresponding to each evidence number from the list of numbered evidence, so that the conclusion corresponds to multiple evidence entries.

[0118] Step 803: Establish the mapping relationship between each conclusion and the corresponding evidence item.

[0119] Here, the question-and-answer system can record the correspondence between each conclusion and its corresponding evidence item based on each conclusion and its corresponding evidence item determined in step 802, and establish a mapping relationship between each conclusion and its corresponding evidence item. This mapping relationship is used to locate and generate candidate evidence on which the conclusion is based based on the conclusion in the main answer text.

[0120] In one possible implementation, the question-answering system can associate each conclusion in the main answer text with the evidence numbers it cites and the corresponding evidence entries. When a conclusion corresponds to multiple evidence entries, the mapping relationship between the conclusion and each evidence entry can be retained separately, enabling the subsequent tracing information to be obtained for each evidence entry.

[0121] Through steps 801 to 803 above, the question-answering system can convert the professional question-answering results generated by the large language model into a structured mapping relationship between each conclusion and the corresponding evidence item, so that each conclusion in the professional question-answering results can be associated with one or more evidence items on which the conclusion is based, providing a basis for subsequent evidence tracing and result verification.

[0122] In one possible embodiment, such as Figure 9 As shown, after establishing the mapping relationship between each conclusion and its corresponding evidence item, the method also includes: Step 901: Based on the mapping relationship, for each evidence item corresponding to each conclusion, obtain the evidence content and source information of the candidate evidence corresponding to the evidence item, and obtain the source information corresponding to the evidence item.

[0123] Here, the question-and-answer system can traverse the evidence items corresponding to each conclusion based on the mapping relationship between each conclusion and the corresponding evidence item. Furthermore, based on the preserved association between the evidence item and candidate evidence, it can obtain the evidence content and source information of the candidate evidence corresponding to each evidence item, forming the source information for that evidence item. The source information is used to indicate the evidence content and source upon which the corresponding conclusion is based.

[0124] In one possible embodiment, the source information may include the source identifier or source of the knowledge source to which the candidate evidence belongs. For multiple evidence items corresponding to the same conclusion, the question-answering system can obtain the evidence content and source information of the candidate evidence corresponding to each evidence item separately, and form the source information corresponding to each evidence item separately, instead of merging the source information of multiple evidence items into one source information.

[0125] Optionally, the source information corresponding to the evidence item may also include the assessment results of the source authority, content timeliness, semantic relevance and multi-source consistency of the candidate evidence, as well as the overall confidence level, so as to display the confidence evaluation of the corresponding candidate evidence while displaying the evidence content and source information.

[0126] Step 902: For each evidence item corresponding to each conclusion, the source information corresponding to the evidence item is marked with a color corresponding to the overall confidence level, based on the overall confidence level of the evidence item, to obtain the visual source information corresponding to the evidence item.

[0127] Here, the question-and-answer system can determine the color to identify the overall confidence level of each evidence item based on the overall confidence level, and use the determined color to mark the source information of the corresponding evidence item, so that the confidence levels of different evidence items can be distinguished by color, and the source information of each evidence item can be visualized.

[0128] In one possible implementation, the question-answering system can determine the corresponding color based on the overall confidence level. For example, green can be used to indicate a high confidence level, yellow to indicate a medium confidence level, and red to indicate a low confidence level. For multiple evidence items corresponding to the same conclusion, the question-answering system can color-code each evidence item according to its overall confidence level, so that different candidate evidence on which the same conclusion is based can have their own corresponding color codes.

[0129] Step 903: Display the main answer text in the main answer area and display the visual traceability information of each evidence item corresponding to each conclusion in the traceability information area.

[0130] Here, the question-and-answer system can present professional question-and-answer results in a segmented display manner. The main answer text extracted in step 801 is displayed in the main answer area, and according to the mapping relationship, the source information area displays the visual source information of each evidence item corresponding to each conclusion in the main answer text, thereby establishing the display association between the conclusions in the main answer text and the corresponding source information.

[0131] In one possible embodiment, the main answer area can display the main answer text in natural language and retain the corresponding evidence number at each conclusion. The source information area can adopt an expandable tree list format, displaying the visualized source information of one or more evidence items corresponding to each conclusion according to the mapping relationship between each conclusion and the evidence item. The visualized source information corresponding to each evidence item can include the evidence content, source information, source authority, content timeliness, semantic relevance, multi-source consistency evaluation results, and overall confidence level of the candidate evidence.

[0132] In one possible embodiment, the question-answering system can also respond to a user's selection operation on the evidence number in the main answer text or the evidence item in the tracing information area, obtain the complete evidence content and detailed information of the corresponding candidate evidence, and display the obtained complete evidence content and detailed information to achieve interactive tracing from the conclusion in the main answer text to the corresponding candidate evidence.

[0133] In another possible embodiment, the question-answering system may also encapsulate the main answer text, the evidence number referenced in the main answer text, and the source information corresponding to each evidence number into application programming interface (API) response data, and send the API response data to the requesting end for display or processing.

[0134] Through steps 901 to 903 above, the question-and-answer system can generate source information for each piece of evidence corresponding to each conclusion based on the mapping relationship between the conclusion and the evidence items. The system can also visualize the information by using colors corresponding to the overall confidence level, allowing users to view the content, source, and confidence level of the evidence on which each conclusion is based, thereby improving the verifiability and interpretability of the professional question-and-answer results.

[0135] As described above, this disclosed technical solution performs multi-source evidence retrieval based on user queries and dynamically evaluates the confidence level of each candidate piece of evidence from four dimensions: source authority, content timeliness, semantic relevance, and multi-source consistency. This allows for the screening of evidence entering the question-and-answer generation stage using comprehensive confidence levels. Furthermore, by clustering, deduplicating, summarizing, and structuring the target evidence set, and by using generation constraints to limit the large language model to use the target evidence set as the basis for generation and to cite evidence numbers at corresponding conclusions, it reduces the interference of low-quality, outdated, or conflicting candidate evidence on the model generation process and lowers the possibility of the large language model deviating from the evidence generation content. After generating professional question-and-answer results, by establishing a mapping relationship between each conclusion and the corresponding evidence item, and visually displaying the evidence content, source information, and comprehensive confidence level of each evidence item, it enables the verification and tracing of professional question-and-answer results. Therefore, this disclosed technical solution can solve the technical problem that related technologies struggle to effectively control the applicability of retrieval results as the basis for answer generation and their use in the generation process, leading to insufficient factual accuracy, reliability, and interpretability of professional question-and-answer results.

[0136] In one embodiment, such as Figure 10 As shown, a large-scale professional question-and-answer generation system 1000 based on multi-source evidence retrieval and dynamic confidence assessment is provided, including a multi-source evidence retrieval module 1001, a dynamic confidence assessment module 1002, a target evidence determination module 1003, a model input construction module 1004, and a professional question-and-answer generation module 1005.

[0137] The multi-source evidence retrieval module 1001 is used to obtain user queries and perform multi-source evidence retrieval based on user queries to obtain a set of multi-source candidate evidence.

[0138] The dynamic confidence assessment module 1002 is used to perform dynamic confidence assessment on each candidate piece of evidence in the multi-source candidate evidence set to obtain the comprehensive confidence of each candidate piece of evidence; wherein, the comprehensive confidence of each candidate piece of evidence is obtained by integrating the assessment results of the source authority, content timeliness, semantic relevance and multi-source consistency of the corresponding candidate evidence.

[0139] The target evidence determination module 1003 is used to determine the target evidence set from the multi-source candidate evidence set based on the comprehensive confidence level of each candidate evidence.

[0140] The model input construction module 1004 is used to construct model input information containing generation constraints based on user queries and target evidence sets; wherein, generation constraints are used to limit the large language model to use the target evidence set as the basis for generation.

[0141] The professional question and answer generation module 1005 is used to input the model input information into the large language model and obtain professional question and answer results corresponding to the user's query.

[0142] In one possible embodiment, the multi-source evidence retrieval module 1001 can be used to execute the aforementioned steps 101 and 201 to 203; the dynamic confidence assessment module 1002 can be used to execute the aforementioned steps 102 and 401 to 403; the target evidence determination module 1003 can be used to execute the aforementioned steps 103 and 601 to 602; the model input construction module 1004 can be used to execute the aforementioned steps 104 and 701 to 705; and the professional question-and-answer generation module 1005 can be used to execute the aforementioned step 105. The specific processing procedures and corresponding beneficial effects of each module can be found in the relevant descriptions in the aforementioned method embodiments, and will not be repeated here.

[0143] It should be noted that, Figure 10 The module division shown is a functional division method and does not constitute a limitation on the specific settings of each module.

[0144] Figure 11 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this disclosure. For example... Figure 11 As shown, the electronic device 1100 includes at least one processor 1101 and a memory 1102 coupled to the processor 1101. The processor 1101 can perform the corresponding steps in the methods disclosed in the embodiments of this disclosure.

[0145] The processor 1101 described above can also be called a central processing unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in this embodiment can be implemented by the integrated logic circuitry in the processor 1101 or by software instructions. The processor 1101 can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this embodiment can be directly implemented by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can be located in the memory 1102, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor 1101 reads information from the memory 1102 and, in conjunction with its hardware, completes the steps of the method described above.

[0146] Furthermore, various operations / processes according to this disclosure, implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, such as... Figure 12 The computer system 1200 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including functions such as those described above. Figure 12 A block diagram of a computer system provided for an exemplary embodiment of this disclosure.

[0147] Computer system 1200 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0148] like Figure 12As shown, the computer system 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. The RAM 1203 may also store various programs and data required for the operation of the computer system 1200. The computing unit 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0149] Multiple components in the computer system 1200 are connected to the I / O interface 1205, including: an input unit 1206, an output unit 1207, a storage unit 1208, and a communication unit 1209. The input unit 1206 can be any type of device capable of inputting information into the computer system 1200. The input unit 1206 can receive input numerical or character information and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 1207 can be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. The storage unit 1208 may include, but is not limited to, a hard disk and an optical disk. The communication unit 1209 allows the computer system 1200 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, a modem, network card, infrared communication device, wireless communication transceiver, and / or chipset, such as Bluetooth™ device, WiFi device, WiMax device, cellular communication device, and / or the like.

[0150] The computing unit 1201 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above. For example, in some embodiments, the methods disclosed in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1100 via ROM 1202 and / or communication unit 1209. In some embodiments, the computing unit 1201 can be configured to perform the methods disclosed in this disclosure by any other suitable means (e.g., by means of firmware).

[0151] This disclosure also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the methods disclosed in this disclosure.

[0152] The computer-readable storage medium in this disclosure can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The aforementioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specifically, the aforementioned computer-readable storage medium may include electrical connections based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0153] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0154] Figure 13 A computer program product 1300 is provided as an exemplary embodiment of the present disclosure. The computer program product 1300 includes a computer program 1301, wherein the computer program 1301, when executed by a processor, implements the methods disclosed in the embodiments of the present disclosure.

[0155] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer.

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0157] The modules, components, or units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0158] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0159] The above description is merely an embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0160] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A large-scale professional question-answering generation method based on multi-source evidence retrieval and dynamic confidence assessment, characterized in that, include: Obtain user queries and perform multi-source evidence retrieval based on the user queries to obtain a multi-source candidate evidence set; Dynamic confidence assessments are performed on each candidate piece of evidence in the multi-source candidate evidence set to obtain the comprehensive confidence level of each candidate piece of evidence; wherein, the comprehensive confidence level of each candidate piece of evidence is obtained by fusing the assessment results of the source authority, content timeliness, semantic relevance and multi-source consistency of the corresponding candidate piece of evidence; Based on the overall confidence level of each candidate piece of evidence, the target evidence set is determined from the multi-source candidate evidence set; Based on the user query and the target evidence set, model input information containing generation constraints is constructed; wherein, the generation constraints are used to limit the large language model to use the target evidence set as the basis for generation; The input information of the model is input into the large language model to obtain professional question and answer results corresponding to the user query.

2. The method according to claim 1, characterized in that, The process of performing multi-source evidence retrieval based on the user query to obtain a multi-source candidate evidence set includes: The user query is subjected to standardized semantic encoding to obtain the query semantic encoding result; Based on the query semantic encoding results, parallel retrieval is performed from structured knowledge sources, unstructured document libraries, and semi-structured resources to obtain structured candidate evidence sets, unstructured candidate evidence sets, and semi-structured candidate evidence sets, respectively. The structured candidate evidence set, the unstructured candidate evidence set, and the semi-structured candidate evidence set are combined to obtain the multi-source candidate evidence set; wherein each candidate piece of evidence in the multi-source candidate evidence set has a source identifier for identifying the corresponding knowledge source.

3. The method according to claim 1, characterized in that, The step of dynamically evaluating the confidence level of each candidate piece of evidence in the multi-source candidate evidence set to obtain the comprehensive confidence level of each candidate piece of evidence includes: For each candidate piece of evidence, the candidate piece of evidence is evaluated based on the source authority, the content timeliness, the semantic relevance, and the multi-source consistency, to obtain the evaluation results for each dimension of the candidate piece of evidence; Each dimension's evaluation result is assigned a corresponding weight; wherein the sum of all weights is 1. The comprehensive confidence level of the candidate evidence is obtained by weighted summation based on the evaluation results of each dimension and its corresponding weight.

4. The method according to claim 3, characterized in that, For each candidate piece of evidence, the assessment results for the source authority and the content timeliness are obtained through the following processing: Based on the professional level and authority of the knowledge source to which the candidate evidence belongs, the candidate evidence is stratified and scored to obtain the evaluation result of the source authority corresponding to the candidate evidence; Obtain the publication time and current time of the candidate evidence, and determine the time interval based on the publication time and the current time; Based on the time interval and attenuation coefficient, an exponential decay function is used to calculate the evaluation result of the timeliness of the content corresponding to the candidate evidence; wherein, the evaluation result of the timeliness of the content decreases as the time interval increases.

5. The method according to claim 3, characterized in that, For each candidate piece of evidence, the evaluation results of the semantic relevance and the multi-source consistency corresponding to the candidate piece of evidence are obtained through the following processing: The Sentence-BERT model is used to semantically encode the evidence content of the user query and the candidate evidence, respectively, to obtain the query semantic vector and the evidence semantic vector. Calculate the cosine similarity between the query semantic vector and the evidence semantic vector to obtain the evaluation result of the semantic relevance corresponding to the candidate evidence; Based on the content consistency between the candidate evidence and other candidate evidence in the multi-source candidate evidence set, the evaluation result of the multi-source consistency corresponding to the candidate evidence is determined.

6. The method according to claim 1, characterized in that, The step of determining the target evidence set from the multi-source candidate evidence set based on the comprehensive confidence level of each candidate piece of evidence includes: The candidate evidence is sorted in descending order of the overall confidence level corresponding to each candidate evidence to obtain a candidate evidence sequence; Based on a preset confidence threshold, candidate evidence with a comprehensive confidence level greater than the preset confidence threshold is selected from the candidate evidence sequence to obtain the target evidence set composed of the selected candidate evidence.

7. The method according to claim 1, characterized in that, The construction of model input information containing generation constraints based on the user query and the target evidence set includes: Based on the degree of content overlap between the candidate evidences in the target evidence set, the candidate evidences are clustered to remove duplicates, resulting in a deduplicated evidence set. Text summaries are generated for each candidate piece of evidence in the deduplicated evidence set to obtain evidence summaries corresponding to each candidate piece of evidence. For each candidate piece of evidence in the deduplicated evidence set, an evidence entry corresponding to the candidate piece of evidence is generated based on the evidence summary corresponding to the candidate piece of evidence and the comprehensive confidence level of the candidate piece of evidence. Construct a structured evidence list based on each of the aforementioned evidence items; The user query, the structured evidence list, and the generation constraints are combined to obtain the model input information.

8. The method according to claim 7, characterized in that, The generation constraints include evidence usage constraints and evidence citation constraints; The step of combining the user query, the structured evidence list, and the generation constraints to obtain the model input information includes: Each of the evidence entries in the structured evidence list is assigned an evidence number to obtain a numbered evidence list; The user query, the numbered evidence list, the evidence usage constraint, and the evidence citation constraint are combined to obtain the model input information; wherein, the evidence usage constraint is used to limit the large language model to generate the professional question-and-answer result based solely on the numbered evidence list; the evidence citation constraint is used to limit the large language model to cite the evidence number corresponding to the conclusion in the professional question-and-answer result.

9. The method according to claim 8, characterized in that, After inputting the model input information into the large language model to obtain the professional question-and-answer result corresponding to the user query, the method further includes: Semantic analysis is performed on the professional Q&A results to extract the main answer text and the evidence number cited in the main answer text; Determine each conclusion in the main answer text and the evidence number cited by each conclusion, and determine the evidence entry corresponding to each conclusion from the list of numbered evidence based on the evidence number cited by each conclusion; Establish a mapping relationship between each conclusion and the corresponding evidence item.

10. The method according to claim 9, characterized in that, After establishing the mapping relationship between each conclusion and the corresponding evidence item, the method further includes: Based on the mapping relationship, for each evidence item corresponding to each conclusion, the evidence content and source information of the candidate evidence corresponding to the evidence item are obtained, and the source information corresponding to the evidence item is obtained. For each piece of evidence corresponding to each of the aforementioned conclusions, the source information corresponding to the piece of evidence is marked with a color corresponding to the overall confidence level, based on the overall confidence level of the piece of evidence, to obtain the visual source information corresponding to the piece of evidence. The main answer text is displayed in the main answer area, and the visual traceability information of each evidence item corresponding to each conclusion is displayed in the traceability information area.

11. A large-scale professional question-answering generation system based on multi-source evidence retrieval and dynamic confidence assessment, characterized in that, include: The multi-source evidence retrieval module is used to obtain user queries and perform multi-source evidence retrieval based on the user queries to obtain a set of multi-source candidate evidence. The dynamic confidence assessment module is used to perform dynamic confidence assessment on each candidate piece of evidence in the multi-source candidate evidence set to obtain the comprehensive confidence of each candidate piece of evidence; wherein, the comprehensive confidence of each candidate piece of evidence is obtained by fusing the assessment results of the source authority, content timeliness, semantic relevance and multi-source consistency of the corresponding candidate piece of evidence; The target evidence determination module is used to determine the target evidence set from the multi-source candidate evidence set based on the comprehensive confidence level of each candidate evidence; The model input construction module is used to construct model input information containing generation constraints based on the user query and the target evidence set; wherein, the generation constraints are used to limit the large language model to use the target evidence set as the basis for generation; The professional question-and-answer generation module is used to input the model input information into the large language model to obtain professional question-and-answer results corresponding to the user query.

12. An electronic device, characterized in that, include: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method as described in any one of claims 1-10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-10.