Evidence traceability method, device, medium and product for medical model reply content

Abstract

Embodiments of the present disclosure disclose a medical model reply content evidence tracing method, device, medium and product. The method comprises: matching the question data with the knowledge text in the knowledge base to obtain a first confidence degree, which is the standard similarity between the question data and the knowledge text; based on the first confidence degree and the first best parameter, filtering out the actual candidate text from the knowledge base, combining the medical model to obtain the actual reply content; after the actual reply content is semantically segmented into multiple actual fine-grained segments, matching the actual fine-grained segments with the knowledge text to obtain a second confidence degree, which is the standard similarity between the actual fine-grained segments and the knowledge text; based on the second confidence degree and the second best parameter, filtering out the tracing evidence of the actual fine-grained segments from the knowledge base, and then the actual reply content tracing result can be displayed. Through RAG retrieval and tracing retrieval, the original source of the reply content can be checked reversely, high-precision tracing is realized, and the explainability is enhanced.

Description

Technical Field

[0001] This disclosure relates to the field of natural speech processing technology, and in particular to a method, apparatus, medium, and product for tracing the source of evidence of medical model responses. Background Technology

[0002] In today's medical field, medical models are widely used in tasks such as clinical decision support, medical record analysis, and health consultation. To improve the accuracy and professionalism of question answering, the industry generally adopts the RAG (Retrieval Augmentation) technical architecture, which combines a local knowledge base with a medical model to enhance its responsiveness to specialized questions. The specific process involves the system performing a vector search in the knowledge base after a user asks a question to obtain preliminary relevant text. The search results are then used as prompts and fed into the medical model, which generates the response and outputs it to the user.

[0003] While the RAG technical architecture effectively improves the medical model's ability to respond to professional questions, the response may mix knowledge base information with the model's own training content, making it difficult to accurately trace specific facts back to the corresponding knowledge base text. This fails to meet the medical field's requirement for high traceability and also affects clinical trust and regulatory compliance. Summary of the Invention

[0004] In view of this, the present disclosure provides a method, apparatus, medium and product for tracing the source of evidence of medical model responses. Through RAG retrieval and source tracing retrieval, it can not only provide information support for medical models before content generation, but also reverse verify the original source of the response content after generation, achieving high-precision source tracing and enhancing interpretability. This solution is particularly suitable for professional medical scenarios such as assisting medical question answering and quality control review, and is of great significance for improving the transparency, security and trustworthiness of medical AI systems.

[0005] In a first aspect, the present disclosure provides a method for tracing the source of evidence for the content of a medical model's response, employing the following technical solution:

[0006] The system obtains user-inputted question data, matches the question data with knowledge text in a knowledge base, and obtains a first confidence score; wherein, the first confidence score is the standard similarity between the question data and the knowledge text.

[0007] Obtain the first optimal parameter, and based on the first confidence level and the first optimal parameter, filter out actual candidate texts from the knowledge base;

[0008] Based on the actual candidate text and the medical model, the actual response content is obtained;

[0009] The actual response content is semantically segmented into multiple fine-grained segments;

[0010] The actual fine-grained fragment is matched with the knowledge text to obtain a second confidence score; wherein, the second confidence score is the standard similarity between the actual fine-grained fragment and the knowledge text;

[0011] Obtain a second optimal parameter, and based on the second confidence level and the second optimal parameter, filter out the source evidence of actual fine-grained fragments from the knowledge base;

[0012] Based on the source tracing evidence of the actual fine-grained fragments, the source tracing results of the actual response content are displayed on the user interface.

[0013] Optionally, obtaining the first optimal parameter includes:

[0014] The first batch of samples is determined by traversing the first validation set;

[0015] The data samples included in the first batch of samples are matched with the knowledge text to obtain a third confidence level; wherein, the third confidence level is the standard similarity between the data samples in the first batch of samples and the knowledge text;

[0016] Based on the third confidence level and the first threshold of each data sample in the first batch of samples, multiple sets of candidate texts are obtained;

[0017] Based on the multiple candidate text sets and the medical model, multiple first experimental response contents are obtained;

[0018] Based on the multiple first experimental response contents and the multiple candidate text sets, a first precision and a first recall rate are obtained;

[0019] The first threshold is increased by a first preset value and the first validation set is traversed again. Based on the new first threshold and the new first batch of samples, a new first precision and a new first recall are obtained.

[0020] After the first validation set is traversed, based on all the first precision and first recall, the first F1 score corresponding to each first threshold is obtained, and the first threshold with the highest first F1 score is selected as the first optimal parameter.

[0021] Optionally, obtaining the second optimal parameter includes:

[0022] The second batch of samples was determined by traversing the second validation set;

[0023] Based on the data samples contained in the second batch of samples, questions are posed to the medical model to obtain multiple second experimental responses.

[0024] Each second experimental response is broken down into multiple fine-grained experimental segments, and experimental source evidence is obtained based on a second threshold; wherein, the experimental source evidence is the source evidence of the fine-grained experimental segments.

[0025] Based on the experimental source tracing evidence and the content of the second experimental response, the genuine source tracing evidence, false source tracing evidence, and missed source tracing evidence are determined.

[0026] Based on the number of genuine source traceability evidence, the number of false source traceability evidence, and the number of missed source traceability evidence, a second precision and a second recall rate are obtained.

[0027] Increase the second threshold by a second preset value and continue to traverse the second validation set. Based on the new second threshold and the new second batch of samples, obtain the new second precision and the new second recall.

[0028] After the second validation set is traversed, based on all the second precision and second recall, the second F1 score corresponding to each second threshold is obtained, and the second threshold with the highest second F1 score is selected as the second optimal parameter.

[0029] Optionally, the evidence tracing method for the medical model's response content further includes:

[0030] Obtain the length of the problem data, the average confidence score of the actual candidate texts, and the coefficient of variation of the actual candidate texts;

[0031] Based on user feedback data, obtain the accuracy of the source tracing results of actual response content;

[0032] The frequency of use of the current values ​​of the first quantity and the second optimal parameter is obtained; wherein, the first quantity is the actual number of fine-grained fragments;

[0033] Based on the preset time series window size, calculate the length of the problem data, average confidence level, coefficient of variation, number of first occurrences, accuracy rate, and frequency of use within the current time series window;

[0034] Construct an input vector based on the problem data length, average confidence, coefficient of variation, first quantity, accuracy, and frequency of use within the current time window;

[0035] The input vector is fed into the dual-threshold optimization model to obtain new first optimal parameters and new second optimal parameters.

[0036] Optionally, constructing the input vector based on the problem data length, average confidence, coefficient of variation, first quantity, accuracy, and frequency of use within the current time window includes:

[0037] Based on the average confidence level and the accuracy rate, a comprehensive retrieval quality score is obtained;

[0038] Obtain the number of sentence segments in the actual response content, and based on the number of sentence segments and the length of the question data, obtain the content generation efficiency;

[0039] The preset time window size is T. Based on the problem data length, average confidence, coefficient of variation, first quantity, accuracy, usage frequency, comprehensive retrieval quality score and content generation efficiency within the current time window, an input vector of shape (T, 8) is constructed.

[0040] Optionally, the step of filtering actual candidate texts from the knowledge base based on the first confidence level and the first optimal parameter includes:

[0041] The knowledge text with the first confidence level greater than or equal to the first optimal parameter is used as the initial text, and a second quantity is counted; wherein, the second quantity is the number of initial texts;

[0042] Determine whether the second quantity is greater than K;

[0043] If the confidence level is greater than the given confidence level, the initial text is sorted in descending order of the first confidence level.

[0044] Select the top K initial texts from the sorted list as actual candidate texts;

[0045] If it is not greater than, then the initial text is taken as the actual candidate text.

[0046] Optionally, the step of semantically segmenting the actual response content into multiple fine-grained segments includes:

[0047] Based on the available GPU memory size, the semantic segmentation model divides the semantic segmentation task of the actual response content into batches to obtain subtasks for each batch.

[0048] Initialize a thread pool, and use each thread in the thread pool to execute subtasks in the batch in parallel;

[0049] Once all subtasks have been executed, multiple actual fine-grained fragments are generated.

[0050] Secondly, this disclosure also provides an evidence tracing system for the content of medical model responses, employing the following technical solution:

[0051] The first matching module is used to obtain question data input by the user, match the question data with knowledge text in the knowledge base, and obtain a first confidence score; wherein, the first confidence score is the standard similarity between the question data and the knowledge text;

[0052] The first filtering module is used to obtain the first optimal parameter and filter out actual candidate texts from the knowledge base based on the first confidence level and the first optimal parameter.

[0053] The response acquisition module is used to acquire the actual response content based on the actual candidate text and the medical model;

[0054] The semantic segmentation module is used to semantically segment the actual response content into multiple fine-grained segments.

[0055] The second matching module is used to match the actual fine-grained fragment with the knowledge text to obtain a second confidence score; wherein, the second confidence score is the standard similarity between the actual fine-grained fragment and the knowledge text;

[0056] The second filtering module is used to obtain the second optimal parameter, and based on the second confidence level and the second optimal parameter, to filter out the source evidence of actual fine-grained fragments from the knowledge base;

[0057] The results display module is used to display the source tracing results of the actual response content on the user interface based on the source tracing evidence of the actual fine-grained fragments.

[0058] Thirdly, this disclosure also provides a computer device, which adopts the following technical solution:

[0059] The computer device includes:

[0060] At least one processor; and,

[0061] A memory communicatively connected to the at least one processor; wherein,

[0062] The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform the evidence tracing method for the medical model response content described above.

[0063] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing computer instructions for causing a computer to execute the evidence tracing method for the medical model response content described above.

[0064] Fifthly, embodiments of this disclosure also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0065] The evidence tracing method for medical model response content provided in this disclosure matches user-input question data with knowledge text in a knowledge base and obtains a first confidence level, providing a quantitative basis for subsequent knowledge screening. This allows for the initial identification of the knowledge scope related to the user's question, facilitating the rapid focus on potentially useful information from a massive knowledge base. Utilizing a first optimal parameter and a first confidence level to filter actual candidate texts from the knowledge base further narrows the knowledge scope based on specific business needs, improving the relevance of the selected texts to the user's question and reducing redundant information in subsequent processing. Based on the actual candidate texts and the medical model, actual response content is generated. Because the candidate texts have been screened and are more closely aligned with the user's question, the medical model can use these high-quality texts to generate more accurate and targeted responses, improving the quality and reliability of the responses. Semantically segmenting the actual response content into multiple fine-grained fragments facilitates more detailed analysis and processing of the response content. This allows for in-depth mining of information from each fragment, providing more precise units for subsequent tracing and evaluation, and preventing the omission of certain key information due to the complexity of the overall response. Matching actual fine-grained fragments with knowledge text and obtaining a second confidence level allows for individual similarity assessment of each fragment, clarifying its source similarity in the knowledge base and providing more detailed information for accurate source tracing. Combining the second optimal parameter and the second confidence level, source tracing evidence for actual fine-grained fragments is filtered from the knowledge base. The optimal parameter further optimizes the filtering of source tracing evidence, ensuring that the found evidence is highly relevant to the actual fine-grained fragments, thus improving the accuracy and reliability of source tracing. Based on the source tracing evidence from actual fine-grained fragments, the source tracing results of the actual response content are displayed on the user interface, allowing users to clearly understand the source and basis of the medical model's response, increasing the credibility and interpretability of the response, and helping users better understand and accept the medical model's response.

[0066] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0067] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 A flowchart illustrating the evidence tracing method for medical model response content provided in this embodiment of the disclosure;

[0069] Figure 2 A flowchart illustrating the first optimal parameter acquisition method provided in this embodiment of the disclosure;

[0070] Figure 3 A schematic flowchart illustrating the actual fine-grained fragment acquisition method provided in this embodiment of the disclosure;

[0071] Figure 4 A flowchart illustrating the second optimal parameter acquisition method provided in this embodiment of the disclosure;

[0072] Figure 5 A flowchart illustrating the first and second optimal parameter optimization methods provided in the embodiments of this disclosure;

[0073] Figure 6 A schematic diagram of the evidence tracing system for medical model response content provided in this embodiment of the disclosure;

[0074] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. Detailed Implementation

[0075] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0076] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0077] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0078] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0079] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0080] Reference Figure 1 This disclosure provides a method for tracing the source of evidence for the content of a medical model response, including the following steps:

[0081] S1: Obtain the question data input by the user, match the question data with the knowledge text in the knowledge base, and obtain the first confidence score;

[0082] S2: Obtain the first optimal parameter, and based on the first confidence level and the first optimal parameter, filter out the actual candidate texts from the knowledge base;

[0083] S3: Obtain the actual response content based on the actual candidate text and medical model;

[0084] S4: Semantically segment the actual response content into multiple fine-grained segments;

[0085] S5: Match the actual fine-grained fragments with the knowledge text to obtain the second confidence level;

[0086] S6: Obtain the second optimal parameter, and based on the second confidence level and the second optimal parameter, filter out the source evidence of the actual fine-grained fragments from the knowledge base;

[0087] S7: Based on the source evidence of actual fine-grained fragments, display the source tracing results of the actual response content on the user interface.

[0088] The first confidence level is the standard similarity between the problem data and the knowledge text; the second confidence level is the standard similarity between the actual fine-grained fragments and the knowledge text; the medical model can adopt large language models such as the medical version of the Transformer model, GPT, and LLaMA.

[0089] The evidence tracing method for medical model responses disclosed herein matches user-input question data with knowledge text in a knowledge base and obtains a first confidence level. This provides a quantitative basis for subsequent knowledge screening, enabling the initial identification of the knowledge scope related to the user's question and facilitating the rapid focus on potentially useful information from a massive knowledge base. Utilizing the first optimal parameter and the first confidence level to filter actual candidate texts from the knowledge base further narrows the knowledge scope based on specific business needs, improving the relevance of the selected texts to the user's question and reducing redundant information in subsequent processing. Based on the actual candidate texts and the medical model, actual response content is generated. Because the candidate texts have been screened and are more closely aligned with the user's question, the medical model can use these high-quality texts to generate more accurate and targeted responses, improving the quality and reliability of the responses.

[0090] Semantically segmenting the actual response content into multiple fine-grained fragments facilitates more detailed analysis and processing. This allows for in-depth information mining of each fragment, providing more precise units for subsequent tracing and evaluation, and preventing the omission of crucial information due to the overall complexity of the response. Matching these fine-grained fragments with knowledge text and obtaining a second confidence level enables individual similarity assessment of each fragment, clarifying its source similarity in the knowledge base and providing more detailed information for accurate tracing. Combining the second optimal parameter and the second confidence level, tracing evidence for the fine-grained fragments is filtered from the knowledge base. This further optimizes the filtering of tracing evidence based on the optimal parameter, ensuring that the found evidence is highly relevant to the fine-grained fragments, improving the accuracy and reliability of tracing. Based on the tracing evidence from the fine-grained fragments, the tracing results of the actual response content are displayed on the user interface, allowing users to clearly understand the source and basis of the medical model's response, increasing the credibility and interpretability of the response, and helping users better understand and accept the medical model's response.

[0091] In S1, when a user asks a question in natural language using a medical model, the system extracts the user's input question data, such as "Can diabetic patients use GLP-1 drugs?". Subsequently, the system uses this question data to conduct the first phase of retrieval in a knowledge base that integrates a massive amount of knowledge text, compiled from structured text (such as tables, medical terminology dictionaries, etc.) and unstructured text (such as medical guidelines, PDF files, electronic medical records, etc.). During this first-phase retrieval, the cosine similarity between the question data and the knowledge text is calculated. This cosine similarity is then standardized to obtain a first confidence level ranging from 0 to 1. The formula for calculating the first confidence level is:

[0092] Z1 = (cos(θ1) + 1) / 2;

[0093] In the formula, Z1 is the first confidence level; θ1 is the cosine similarity between the question data and the knowledge text.

[0094] Furthermore, before calculating the cosine similarity between the question data and the knowledge text, a pre-trained embedding model in the medical field is used to transform the question data and knowledge text into query vectors and knowledge vectors, respectively. Then, the cosine similarity between the query vector and the knowledge vector is calculated. The embedding model is a model specifically pre-trained on large amounts of text data in the medical field, such as BioBERT. Using this embedding model, the question data and knowledge text are processed and converted into high-dimensional vectors, facilitating subsequent similarity calculation and retrieval. Efficient vector index structures are built using libraries such as FAISS (Facebook AI Similarity Search) and Milvus to support fast similarity retrieval. FAISS and Milvus are open-source libraries specifically designed for efficient vector search, providing a series of algorithms and data structures that can quickly find the vector most similar to the query vector in a large-scale vector database. These libraries are used to construct index structures from the previously generated high-dimensional vectors, such as inverted indexes and KD-trees. When a new query vector is input, these index structures can be used to quickly find knowledge vectors with high similarity in the vector library. The corresponding knowledge text is the medical knowledge related to the query question, thus achieving fast information retrieval.

[0095] In S2, a dual-screening strategy is employed to obtain relevant candidate texts: Knowledge texts with a first confidence level greater than or equal to the first optimal parameter are used as initial texts, and a second quantity is counted (the number of initial texts). It is then determined whether the second quantity is greater than K; if so, the initial texts are sorted in descending order of the first confidence level; the top K initial texts are selected as candidate texts. If not, all initial texts are used as candidate texts. This method uses dual-screening conditions to filter knowledge texts. First, the initial texts are screened using the first optimal parameter, quickly eliminating a large number of low-relevance texts and focusing on more likely relevant texts, reducing the number of texts processed subsequently and improving overall retrieval efficiency. When there are many initial texts meeting the first optimal parameter condition, further sorting by the first confidence level and selecting the top K texts accurately identifies the texts most closely matching the question, improving the accuracy of the retrieval results. For different scales of retrieval needs, the K value can be adjusted flexibly. When the number of initial texts is small, they can be directly used as candidate texts, ensuring that potentially useful information is not overlooked.

[0096] The first optimal parameters were obtained during the first experimental phase, referencing... Figure 2 The flowchart illustrating the method for obtaining the first optimal parameter shows that "obtaining the first optimal parameter" includes the following steps:

[0097] S21: Determine the first batch of samples by traversing the first validation set;

[0098] S22: Match the data samples included in the first batch of samples with the knowledge text to obtain the third confidence score; whereby the third confidence score is the standard similarity between the data samples in the first batch of samples and the knowledge text.

[0099] S23: Based on the third confidence level and the first threshold of each data sample in the first batch of samples, obtain multiple sets of candidate texts;

[0100] S24: Based on multiple candidate text sets and medical models, obtain multiple first-experiment response contents;

[0101] S25: Based on multiple first-experiment response contents and multiple candidate text sets, obtain the first precision and the first recall;

[0102] S26: Increase the first threshold by a first preset value and continue to traverse the first validation set. Based on the new first threshold and the new first batch of samples, obtain the new first precision and the new first recall.

[0103] S27: After the first validation set is traversed, based on all the first precision and first recall, obtain the first F1 score corresponding to each first threshold, and select the first threshold with the highest first F1 score as the first optimal parameter.

[0104] Based on the above steps, a large-scale medical guideline database containing 1000 authoritative guideline texts was structured and analyzed to generate multiple data samples. These data samples were divided into two datasets, named the first validation set and the second validation set, respectively. The first validation set was further divided into multiple batches. By traversing the first validation set, the data samples of the current batch were obtained, and the samples in this batch were defined as the first batch of samples. For each data sample contained in the first batch of samples, a RAG (Retrieval Enhancement Generation) retrieval operation was performed to obtain the corresponding third confidence level. On this basis, a dual screening strategy was adopted, combining the third confidence level with a preset first threshold to select multiple candidate text sets. Each candidate text set was obtained by performing a RAG retrieval on a single data sample in the first batch of samples.

[0105] A set of candidate texts is integrated with the original data samples to form a prompt message. This prompt message is then input into a medical model, which outputs a response message based on this prompt. This response message is defined as the first experimental response message. Since each set of candidate texts is combined with a corresponding original data sample, multiple first experimental response messages can ultimately be obtained.

[0106] We manually analyze the candidate text set and the first experimental response content obtained in the current round, counting the number of texts in the candidate text set that match the first experimental response content (here, matching means that the texts in the candidate text set correspond to the content of the first experimental response content in terms of content), and the number of texts that were incorrectly identified as matching the first experimental response content (referring to those texts in the candidate text set that actually have no corresponding relationship to the first experimental response content but were retrieved); at the same time, we identify the number of texts in the first experimental response content that did not find a matching text in the candidate text set (referring to the number of texts in the first experimental response content that should have a corresponding text in the candidate text set but were not actually found). Based on the above analysis results, we can further calculate and obtain the first precision and the first recall, where the formula for calculating the first precision is:

[0107] P a =TP a / (TP) a +FP a );

[0108] In the formula, P a For first-degree accuracy; TP a The number of texts in the candidate text set that match the responses from the first experiment; FP a This represents the number of texts in the candidate text set that were incorrectly identified as matching the responses from the first experiment.

[0109] The formula for calculating the first recall rate is:

[0110] R a =TP a / (TP) a +FN a );

[0111] In the formula, R a The highest recall rate; FN a The number of responses in the first experiment that did not find matching texts in the candidate text set.

[0112] The formula for calculating the first F1 score is:

[0113] F1 a = 2P a ×R a / (P) a +R a );

[0114] In the formula, F1 a This is the first F1 score.

[0115] The first threshold is used to determine multiple candidate text sets during the RAG retrieval and screening process. Different first thresholds will produce different candidate text sets, which will affect the matching of the first experimental response content with the candidate text sets, ultimately resulting in different calculated first precision and first recall. The first F1 score is calculated from the first precision and first recall, so there is a corresponding relationship between the first F1 score and the first threshold.

[0116] The first threshold is increased by a first preset value, for example, the initial first threshold is 0.5 and the first preset value is 0.02. Then, the first validation set is iterated over again. For the new first threshold, a new first batch of samples is obtained. RAG retrieval is performed on each data sample in the new first batch to obtain a new third confidence level. The dual screening strategy is applied again, combining the new third confidence level and the new first threshold to obtain a new set of candidate texts. This new set of candidate texts is combined with the medical model to obtain new first experimental responses. Based on the new set of candidate texts and the new first experimental responses, a new first precision and a new first recall are calculated, and then the first F1 score corresponding to the new first threshold is calculated.

[0117] When the new first threshold is greater than the first preset value (e.g., 0.9), stop traversing the first validation set, count the first F1 scores corresponding to all first thresholds, compare the first F1 scores corresponding to different first thresholds, select the first threshold with the highest first F1 score, and determine it as the first optimal parameter.

[0118] Optionally, during the iteration of the first validation set, the first F1 score can be omitted initially; instead, the first precision and first recall can be recorded to plot a first precision-first recall curve. After the first validation set iteration is complete, the first F1 score for each first threshold is obtained based on the first precision-first recall curve, and the first threshold with the highest first F1 score is selected as the first optimal parameter. Through experimentation, this scheme ultimately selected 0.73 as the initial first optimal parameter.

[0119] In S3, the filtered actual candidate text is concatenated with the user's question data to form an input prompt (Prompt). The prompt vector is then input into the medical model to obtain the response content output by the medical model, which is the actual response content.

[0120] In S4, refer to Figure 3 The flowchart illustrating the actual fine-grained segment acquisition method shows that "semantically segmenting the actual response content into multiple actual fine-grained segments" includes the following steps:

[0121] S41: Based on the available GPU memory size, the semantic segmentation model divides the semantic segmentation task of the actual response content into batches, generating multiple subtasks;

[0122] S42: Initialize the thread pool and use each thread in the thread pool to execute subtasks in parallel;

[0123] S43: After all subtasks have been executed, multiple actual fine-grained fragments are generated.

[0124] In S41, for semantic segmentation scenarios, the performance of multiple open-source models of different sizes was compared to determine the optimal model as the base model. Qwen3-8B was ultimately selected as the base model and fine-tuned. This fine-tuning was conducted on a single server equipped with two NVIDIA H800 GPUs (80GB VRAM) based on a carefully constructed dataset of 500 high-quality medical semantic segmentation annotations. With a maximum sequence length of 32768, a batch size of 8, and a learning rate of 5e-6, after five rounds of training, the base model maintained a performance decrease of less than 2% on general benchmark tests while significantly improving semantic segmentation accuracy for medical texts by 23%, thus obtaining a semantic segmentation model suitable for this scenario.

[0125] Check the available GPU memory size using GPU management tools (such as NVIDIA's nvidia-smi command). Select response content samples of different lengths and perform semantic segmentation tests on them using a semantic segmentation model. Record the GPU memory size occupied by the semantic segmentation model when processing a single response content sample. Add up all the recorded memory sizes and divide by the total number of test tasks to calculate the average memory size occupied by all tasks. Then divide the available memory size by the average memory size to obtain the number of subtasks that can be accommodated in each batch. Based on this number, allocate the actual response content semantic segmentation tasks to each batch in sequence until all tasks are allocated. This divides the semantic segmentation task into multiple batches, with each batch containing a corresponding number of subtasks.

[0126] In S42, the number of threads in the thread pool is set and initialized, taking into account factors such as the number of available GPU cores, thread overhead, and task parallelism. Then, the threads in the thread pool execute subtasks in parallel, with each thread corresponding to one subtask. They simultaneously perform semantic segmentation operations on their respective subtasks, which can improve processing efficiency and reduce overall processing time.

[0127] In S43, after all subtasks are completed, the segmentation results of each subtask are organized and integrated to form multiple actual fine-grained segments. These segments are independent semantic units after semantic segmentation. For example, an answer containing multiple knowledge points is broken down into 5 to 10 independent semantic units, which are stored in the form of an array.

[0128] In the S41-S43 solution, parallel retrieval during the source tracing phase is implemented, using a thread pool + GPU batch processing method to ensure that the latency is controlled within 300ms, without significantly affecting the end-to-end experience.

[0129] In S5, the second-stage retrieval is performed in the knowledge base using the acquired fine-grained fragments. During this second-stage retrieval, the cosine similarity between the fine-grained fragments and the knowledge text is calculated. This cosine similarity is then standardized to obtain a second confidence score ranging from 0 to 1. The formula for calculating the second confidence score is as follows:

[0130] Z2 = (cos(θ2) + 1) / 2;

[0131] In the formula, Z2 is the second confidence level; θ2 is the cosine similarity between the actual fine-grained fragment and the knowledge text.

[0132] In S6, the largest second confidence score is selected from the second confidence scores between the actual fine-grained fragment and each knowledge text. If the largest second confidence score is greater than or equal to the second optimal parameter, the knowledge text corresponding to the largest second confidence score is determined as the source evidence of the actual fine-grained fragment. If the largest second confidence score is less than the second optimal parameter, it is determined that there is no source evidence for the actual fine-grained fragment, and it may be generated by the medical model during training.

[0133] The second optimal parameters were obtained in the second experimental phase, referring to... Figure 4 The flowchart illustrating the method for obtaining the second optimal parameter shows that "obtaining the second optimal parameter" includes the following steps:

[0134] S61: Determine the second batch of samples by traversing the second validation set;

[0135] S62: Based on the data samples contained in the second batch of samples, ask questions to the medical model and obtain multiple second experimental responses;

[0136] S63: Each second experimental response is broken down into multiple fine-grained experimental segments, and experimental source evidence is obtained based on a second threshold; wherein, experimental source evidence is source evidence of the fine-grained experimental segments.

[0137] S64: Based on experimental source tracing evidence and the content of the second experimental response, determine the true source tracing evidence, false source tracing evidence, and missed source tracing evidence;

[0138] S65: Based on the number of genuine source traceability evidence, the number of false source traceability evidence, and the number of missed source traceability evidence, obtain the second precision and the second recall.

[0139] S66: Increase the second threshold by a second preset value and continue to traverse the second validation set. Based on the new second threshold and the new second batch of samples, obtain the new second precision and the new second recall.

[0140] S67: After the second validation set is traversed, based on all the second precision and second recall, obtain the second F1 score corresponding to each second threshold, and select the second threshold with the highest second F1 score as the second optimal parameter.

[0141] Based on the above steps, the second validation set is further divided into multiple batches. During the traversal of the second validation set, data samples from the current batch are obtained, and these samples are defined as the second batch samples. Combining the first optimal parameter, RAG retrieval and medical model inference operations are performed on each data sample in the second batch. The response output by the medical model is the second experimental response, and each second experimental response corresponds to a data sample in the second batch.

[0142] The second experimental response content was segmented using a semantic segmentation model. The segmented fine-grained experimental fragments were then matched with knowledge text to obtain a fourth confidence score, which represents the standard similarity between the fine-grained experimental fragment and the knowledge text. The highest fourth confidence score was selected from the four scores between the fine-grained experimental fragment and each knowledge text. If the highest fourth confidence score was greater than or equal to a second threshold, the knowledge text corresponding to the highest fourth confidence score was considered source evidence for that fine-grained experimental fragment; otherwise, if the highest fourth confidence score was less than the second threshold, the fine-grained experimental fragment was deemed to lack source evidence.

[0143] The experimental source evidence was manually compared with the second experimental response to analyze whether the experimental source evidence was used to generate the second experimental response, and to identify evidence that should have been used in the second experimental response but was not, thereby determining true source evidence, false source evidence, and missed source evidence. True source evidence refers to experimental source evidence used to support the generation of the second experimental response; false source evidence refers to experimental source evidence irrelevant to the generation of the second experimental response; and missed source evidence refers to evidence that should have been used in generating the second experimental response but did not appear in the experimental source evidence. The number of true source evidence, false source evidence, and missed source evidence were counted, and based on these numbers, the second precision and second recall were calculated. The formula for calculating the second precision is:

[0144] P b =TP b / (TP) b +FP b );

[0145] In the formula, P b For second-highest precision; TP b The quantity of authentic source evidence; FP b The number of false tracing evidence.

[0146] The formula for calculating the second recall rate is:

[0147] R b =TP b / (TP) b +FN b );

[0148] In the formula, R b The second highest recall rate; FN b This represents the number of missed tracing evidence.

[0149] The formula for calculating the second F1 score is:

[0150] F1 b = 2Pb ×R b / (P) b +R b );

[0151] In the formula, F1 b This is the second F1 score.

[0152] The second threshold is used to determine experimental source evidence during the retrieval process in the source tracing phase. Different second thresholds will result in different experimental source evidence, ultimately leading to different calculated second precision and second recall. Since the second F1 score is calculated from the second precision and second recall, there is a corresponding relationship between the second F1 score and the second threshold.

[0153] The second threshold is increased by a second preset value, and the second validation set is traversed again. Based on the new second batch of samples, multiple new second experimental responses are obtained. Combined with the new second threshold, new experimental source evidence is obtained. Based on the new multiple second experimental responses and the new experimental source evidence, a new second precision and a new second recall are calculated, and then the second F1 score corresponding to the new second threshold is calculated.

[0154] When the new second threshold is greater than the second preset value, stop traversing the second validation set, count the second F1 scores corresponding to all second thresholds, compare the second F1 scores corresponding to different second thresholds, select the second threshold with the highest second F1 score, and determine it as the second optimal parameter.

[0155] Optionally, during the iteration of the second validation set, the second F1 score can be omitted initially; instead, the second precision and second recall can be recorded to plot a second precision-second recall curve. After the second validation set iteration is complete, the second F1 score for each second threshold is obtained based on the second precision-second recall curve, and the second threshold with the highest second F1 score is selected as the second optimal parameter. Through experiments, this scheme ultimately selected 0.81 as the initial second optimal parameter.

[0156] In S7, the actual response content is broken down into multiple sentence segments; the actual fine-grained segment semantics are broken down into multiple sub-segments, and the actual source evidence is assigned to each sub-segment as the source evidence of the sub-segment; the sub-segment with the highest matching degree with the sentence segment is obtained, and the source evidence of the sub-segment is used as the source evidence of the sentence segment.

[0157] When displaying the source tracing results of actual responses on the user interface, a structured format of "sentence / segment + source evidence / source of source evidence (document title, page number, link) + second confidence level of the source evidence" will be adopted to facilitate efficient and clear viewing for users. For each actual response, it will be broken down into segments, with the corresponding source information following each segment. Segments will be presented in text format using a uniform font and size to ensure comfortable reading. For the source evidence section, document titles will be displayed as hyperlinks, allowing users to access the corresponding document by clicking. If an online version of the document is available, the link will directly lead to the page containing the document; if it is a local document, it can be downloaded. Page numbers will follow the document title and be marked in parentheses to accurately locate the evidence within the document. Links will be displayed on a separate line in a smaller font to ensure the overall interface remains concise. The second confidence level is displayed as a percentage to the right of the source tracing evidence, with different colors assigned to different levels of confidence: high confidence (e.g., 80%-100%) is displayed in green, medium confidence (50%-79%) in yellow, and low confidence (below 50%) in red, allowing users to quickly and intuitively assess the reliability of the evidence. Furthermore, the entire source tracing results display area can be expanded and collapsed, allowing users to choose to view all details or only the response content, optimizing interface space utilization and improving the user experience.

[0158] Furthermore, referring to Figure 5 The flowchart illustrating the optimization methods for the first and second optimal parameters is shown below. During the user adoption phase, the steps for adjusting and optimizing the first and second optimal parameters based on user feedback are as follows:

[0159] S81: Obtain the length of the problem data, the average confidence score of the actual candidate texts, and the coefficient of variation of the actual candidate texts;

[0160] S82: Based on user feedback data, obtain the accuracy of the source tracing results of the actual response content;

[0161] S83: Obtain the usage frequency of the current values ​​of the first quantity and the second optimal parameter; where the first quantity is the actual number of fine-grained fragments;

[0162] S84: Based on the preset time series window size, calculate the length of the problem data, average confidence level, coefficient of variation, number of first items, accuracy rate, and frequency of use within the current time series window;

[0163] S85: Construct an input vector based on the problem data length, average confidence, coefficient of variation, first quantity, accuracy, and frequency of use within the current time window;

[0164] S86: Input the input vector into the dual-threshold optimization model to obtain the new first optimal parameters and the new second optimal parameters.

[0165] Based on the above steps, the following operations will be performed during each question-and-answer session: on the one hand, the average confidence level of all actual candidate texts selected in this question-and-answer session will be calculated, and the coefficient of variation will be calculated based on the first confidence level of these actual candidate texts; on the other hand, the length and first quantity of the question data will be recorded, the current value of the second optimal parameter will be obtained, and the frequency of use of the current value will be counted, thereby recording the frequency of use of each value of the second optimal parameter in real time.

[0166] A user feedback interface is provided to enable user feedback functionality. After each question-and-answer session displays the source tracing results, users can evaluate the second confidence level corresponding to the source tracing evidence and label the source tracing evidence of a sentence segment as "correct," "irrelevant," or "lacking evidence." This feedback data will serve as supervisory data for subsequent system optimization iterations. Simultaneously, based on the label type, the accuracy rate of the source tracing evidence of the sentence segment, i.e., the source tracing matching success rate, is calculated. Among these parameters, the question data length reflects the complexity of the input task; the average confidence level measures the overall relevance of the text recalled by the retrieval module; the coefficient of variation measures the difference in confidence scores of actual candidate texts (e.g., a large coefficient of variation indicates high discriminative power, suggesting the existence of a highly matching document); the number of sentence segments reflects the comprehensiveness of the generated actual response content; the accuracy rate measures the proportion of the actual response content supported by the retrieved text, a key indicator of factual consistency; and the usage frequency reflects how frequently the current value is triggered within a specific historical time period. At each time step (after processing a user request), these six core dynamic parameters are collected, reflecting the system's real-time performance from different dimensions and forming a 6-dimensional feature vector.

[0167] Based on the average confidence level and accuracy, a comprehensive search quality score is obtained to reflect the overall quality level of the search results in terms of both accuracy and reliability. The formula for calculating the comprehensive search quality score is as follows:

[0168] ZF = CP × Q;

[0169] In the formula, ZF is the overall retrieval quality score; CP is the average confidence level; and Q is the accuracy rate.

[0170] The number of sentence segments in the actual response is obtained. Based on this number and the question data length, the content generation efficiency is calculated. This reflects the ability to generate a certain number of sentence segments per unit of question data length, taking into account the impact of question data length. It demonstrates the relationship between the speed of content generation and the complexity of the question. The formula for calculating content generation efficiency is as follows:

[0171] L = S / (1+log(D));

[0172] In the formula, L represents the content generation efficiency; S represents the number of sentence segments; and D represents the length of the question data.

[0173] By comprehensively calculating the retrieval quality score and content generation efficiency, at each time point t, the original 6-dimensional feature vector can be expanded into a more informative 8-dimensional feature vector. Since the system's state changes are time-dependent, a sliding time window of size T (T=5) is used for data acquisition to capture this dynamic. Within the current time window, combining the acquired question data length, average confidence, coefficient of variation, first quantity, accuracy, usage frequency, comprehensive retrieval quality score, and content generation efficiency, an input vector of shape (T, 8) is constructed. This vector contains 8 dimensions of system state information from the past T consecutive time points.

[0174] A lightweight Transformer model (Tiny-Transformer) is used to process the temporal input (T, 8) above. This model is the double threshold optimization model, which includes an input layer, two encoder layers (Encoder Layers × 2), a sequence aggregation layer, and an output prediction head.

[0175] An input vector of shape (T, 8) is fed into the input layer, which provides the initial temporal feature data for the entire dual-threshold optimization model. Each encoder layer contains two core sub-modules: a single-head self-attention layer and a feed-forward network. The input vector passes through two identical encoder layers sequentially. Each encoder layer gradually refines and deepens the understanding of temporal information. The single-head self-attention layer uses a self-attention mechanism to allow the dual-threshold optimization model to focus on the relationships between different time steps in the sequence and learn the contextual information of the sequence. The feed-forward network performs an independent nonlinear transformation on the feature vector at each time step. Specifically, it first passes through a linear layer to increase the feature dimension from 8 to 64, then introduces nonlinearity through the ReLU activation function, and finally passes through another linear layer to reduce the dimension back to 8, enhancing the feature representation ability at each time point. Furthermore, the outputs of both the single-head self-attention and feedforward neural network sub-modules are added to the input of the sub-module through residual connections, and then layer normalization is performed. Residual connections and layer normalization can stabilize the training process and promote the flow of information in deep networks.

[0176] The sequence information aggregation layer receives the (T, 8) tensor output by the two encoder layers as input. It condenses the sequence, which contains rich contextual information at each time step after being processed by the encoder, into a single vector for final prediction. Here, mean pooling is used to calculate the average value of the output vectors at T time steps, so that the (T, 8) tensor is aggregated into an (8)-dimensional summary vector.

[0177] The output prediction head receives an (8)-dimensional summary vector from the sequence information aggregation layer as input, which represents the state of the entire time window. The output prediction head maps this summary vector to the final prediction target, completes the dimensionality transformation through a simple linear layer, and finally outputs a (2)-dimensional output vector, which represents the optimal parameter pair (thres1, thres2) for the next time point of prediction, where thres1 is the new first optimal parameter and thres2 is the new second optimal parameter. In this process, the output prediction head follows key control logic, which includes threshold constraints and an abnormal circuit breaker mechanism. The threshold constraints refer to following thres1∈[0.65, 0.85] and thres2∈[0.75, 0.90] for hard boundary protection; the abnormal circuit breaker mechanism refers to the fallback to the first baseline threshold (the first threshold with the highest first F1 score of 0.73) when the predicted value of thres1 fluctuates by more than 15% for three consecutive times, and the fallback to the second baseline threshold (the second threshold with the highest second F1 score of 0.81) when the predicted value of thres2 fluctuates by more than 15% for three consecutive times.

[0178] The deployment of the dual-threshold optimization model is lightweight, with fewer than 1 million model parameters and a single inference time of less than 100ms, meeting real-time requirements. For data optimization, a closed-loop optimization mechanism is constructed, automatically collecting online prediction errors every 24 hours and triggering incremental training; simultaneously, the source matching success rate is calculated based on user feedback, and model recalibration is forcibly initiated when the source failure rate increases by 2%. In terms of solution characteristics, a dynamic collaborative approach is adopted, modeling dual-threshold correlations through a joint output layer; and it possesses lightweight and high-efficiency characteristics, with Tiny-Transformer saving 40% of inference resources compared to LSTM. In terms of performance, significant performance improvements are achieved, with a 35% increase in source accuracy, as determined by precise matching of response sentences and document paragraphs; coverage is improved by 19%, meaning more response fragments can be traced back to the knowledge base content.

[0179] In summary, this solution addresses the issue of untraceable and unverifiable response content in high-trust scenarios such as medical applications using current retrieval-enhanced technologies. It proposes a "dual RAG verification" method, designing differentiated precision and recall calculation methods based on the objectives of the two different stages: RAG retrieval and source tracing retrieval. Through two experiments, the system accurately obtains the first and second optimal parameters for initial use. The first optimal parameter is determined in the first experiment, and then a second experiment is conducted based on this parameter to obtain a more accurate second optimal parameter. During the user phase, the system can automatically optimize the first and second optimal parameters based on user feedback, continuously improving the accuracy and reliability of source tracing. This evidence-based source tracing method for medical model responses effectively solves the problem of accurately tracing specific facts in response content back to the corresponding knowledge base text under the RAG technology architecture, significantly enhancing the traceability of medical model responses. It ensures that each response can be clearly linked to specific text in the knowledge base, greatly meeting the stringent requirements for high traceability in the medical field, helping to improve clinical trust, ensuring that medical models operate within a regulatory compliance framework, and providing more reliable and trustworthy support for medical decision-making.

[0180] To ensure the overall system response time remains within acceptable limits, this solution employs a comprehensive optimization strategy. In the first retrieval stage, candidate text retrieval utilizes the Approximate Nearest Neighbor (ANN) algorithm, deployed with the Faiss library, to efficiently filter relevant documents. In the second retrieval stage, for retrieval of multiple statements, parallel processing is achieved through GPU parallel computing, significantly accelerating the retrieval speed. Furthermore, token-level compression and pruning of the Prompt input length and result display structure effectively improve I / O efficiency, optimizing system performance from multiple levels to ensure response time meets requirements.

[0181] Reference Figure 6 This disclosure provides an evidence tracing system for medical model response content, including:

[0182] The first matching module 101 is used to obtain the question data input by the user, match the question data with the knowledge text in the knowledge base, and obtain the first confidence score; wherein, the first confidence score is the standard similarity between the question data and the knowledge text;

[0183] The first filtering module 102 is used to obtain the first optimal parameter and filter out actual candidate texts from the knowledge base based on the first confidence level and the first optimal parameter.

[0184] The response acquisition module 103 is used to acquire the actual response content based on the actual candidate text and the medical model;

[0185] Semantic segmentation module 104 is used to semantically segment the actual response content into multiple actual fine-grained segments;

[0186] The second matching module 105 is used to match the actual fine-grained fragments with the knowledge text to obtain a second confidence score; wherein, the second confidence score is the standard similarity between the actual fine-grained fragments and the knowledge text.

[0187] The second filtering module 106 is used to obtain the second optimal parameter and, based on the second confidence level and the second optimal parameter, to filter out the source evidence of actual fine-grained fragments from the knowledge base.

[0188] The results display module 107 is used to display the source tracing results of the actual response content on the user interface based on the source tracing evidence of actual fine-grained fragments.

[0189] The various variations and specific examples of the evidence tracing method for medical model response content provided above are also applicable to the evidence tracing system for medical model response content provided in this disclosure. Through the foregoing detailed description of the evidence tracing method for medical model response content, those skilled in the art can clearly understand the implementation method of the evidence tracing system for medical model response content. For the sake of brevity, it will not be described in detail here.

[0190] A computer device according to embodiments of the present disclosure includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.

[0191] The processor may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the computer device to perform desired functions. In one embodiment of this disclosure, the processor is used to execute computer-readable instructions stored in the memory, causing the computer device to perform all or part of the steps of the evidence tracing method for medical model response content of the foregoing embodiments of this disclosure.

[0192] Those skilled in the art will understand that, in order to solve the technical problem of how to achieve a good user experience, this embodiment may also include well-known structures such as communication buses and interfaces, and these well-known structures should also be included within the protection scope of this disclosure.

[0193] like Figure 7This is a schematic diagram of a computer device provided for an embodiment of the present disclosure. It illustrates a structural schematic diagram suitable for implementing the computer device in the embodiments of the present disclosure. Figure 7 The computer device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0194] like Figure 7 As shown, a computer device may include a processor (such as a central processing unit, graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) or programs loaded from storage devices into random access memory (RAM). The RAM also stores various programs and data required for the operation of the computer device. The processor, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0195] Typically, the following devices can be connected to the I / O interface: input devices, such as sensors or visual information acquisition devices; output devices, such as displays; storage devices, such as magnetic tapes or hard drives; and communication devices. Communication devices allow the computer device to communicate wirelessly or wiredly with other devices (such as edge computing devices) to exchange data. Although Figure 7 A computer apparatus with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or included alternatively.

[0196] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processor, all or part of the steps of the evidence tracing method for medical model response content according to embodiments of this disclosure are performed.

[0197] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0198] A computer-readable storage medium according to embodiments of the present disclosure stores non-transitory computer-readable instructions. When these non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the evidence tracing method for medical model response content described in the foregoing embodiments of the present disclosure are performed.

[0199] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).

[0200] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0201] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0202] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.

[0203] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0204] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0205] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0206] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0207] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for evidence provenance of medical model reply content, characterized in that, The method comprises the following steps: obtaining user input question data, matching the question data with knowledge text in a knowledge base, and obtaining a first confidence degree; wherein the first confidence degree is a standard similarity degree between the question data and the knowledge text; obtaining a first optimal parameter, screening actual candidate text from the knowledge base based on the first confidence degree and the first optimal parameter; obtaining actual reply content based on the actual candidate text and a medical model; semantically segmenting the actual reply content into a plurality of actual fine-grained segments; matching the actual fine-grained segments with the knowledge text, and obtaining a second confidence degree; wherein the second confidence degree is a standard similarity degree between the actual fine-grained segments and the knowledge text; obtaining a second optimal parameter, screening actual fine-grained segment provenance evidence from the knowledge base based on the second confidence degree and the second optimal parameter; based on the actual fine-grained segment provenance evidence, displaying actual reply content provenance results on a user interface; wherein the standard similarity degree is a standardized cosine similarity degree, and the first optimal parameter and the second optimal parameter are preset confidence degree thresholds.

2. The evidence sourcing method of medical model reply content according to claim 1, characterized in that, The method comprises the following steps: determining a first batch of samples by traversing a first verification set; matching data samples contained in the first batch of samples with the knowledge text, and obtaining a third confidence degree; wherein the third confidence degree is a standard similarity degree between the data samples in the first batch of samples and the knowledge text; based on the third confidence degree of each data sample in the first batch of samples and a first threshold value, obtaining a plurality of candidate text sets; based on the plurality of candidate text sets and the medical model, obtaining a plurality of first experimental reply contents; based on the plurality of first experimental reply contents and the plurality of candidate text sets, obtaining a first precision and a first recall rate; increasing the first threshold value by a first preset value and continuing to traverse the first verification set, obtaining a new first precision and a new first recall rate based on a new first threshold value and a new first batch of samples; after the first verification set is traversed, based on all the first precisions and the first recall rates, obtaining a first F1 score corresponding to each first threshold value, and selecting the first threshold value with the highest first F1 score as the first optimal parameter.

3. The evidence sourcing method of medical model reply content according to claim 2, characterized in that, The method comprises the following steps: determining a second batch of samples by traversing a second verification set; asking the medical model based on data samples contained in the second batch of samples, and obtaining a plurality of second experimental reply contents; splitting each second experimental reply content into a plurality of experimental fine-grained segments, and obtaining experimental provenance evidence based on a second threshold value; wherein the experimental provenance evidence is provenance evidence of the experimental fine-grained segments; based on the experimental provenance evidence and the second experimental reply content, determining real provenance evidence, false provenance evidence and missed provenance evidence; based on the number of real provenance evidence, the number of false provenance evidence and the number of missed provenance evidence, obtaining a second precision and a second recall rate; increase the second threshold value by a second preset value and continue traversing the second verification set, based on the new second threshold value and the new second batch of samples, obtain a new second precision and a new second recall rate; After the traversal of the second verification set is completed, based on all the second precisions and the second recall rates, obtain a second F1 score corresponding to each second threshold value, and select the second threshold value with the highest second F1 score as the second optimal parameter.

4. The evidence sourcing method of medical model reply content according to claim 3, characterized in that, Further comprising: obtaining the problem data length, the average confidence of the actual candidate text, and the dispersion coefficient of the actual candidate text; based on the user feedback data, obtaining the correctness of the actual reply content traceability result; obtaining the first quantity and the usage frequency of the current value of the second optimal parameter; wherein the first quantity is the number of actual fine-grained segments; According to a preset time sequence window size, the problem data length, the average confidence, the dispersion coefficient, the first quantity, the correctness and the usage frequency in the current time sequence window are counted; Based on the problem data length, the average confidence, the dispersion coefficient, the first quantity, the correctness and the usage frequency in the current time sequence window, an input vector is constructed; input the input vector into the double threshold optimization model to obtain a new first optimal parameter and a new second optimal parameter.

5. The evidence sourcing method of medical model reply content according to claim 4, characterized in that, The input vector is constructed based on the problem data length, the average confidence, the dispersion coefficient, the first quantity, the correctness and the usage frequency in the current time sequence window, comprising: Based on the average confidence and the correctness, a comprehensive retrieval quality score is obtained; obtaining the number of sentence segments of the actual reply content, and based on the number of sentence segments and the problem data length, obtaining the content generation efficiency; The preset time sequence window size is T, and based on the problem data length, the average confidence, the dispersion coefficient, the first quantity, the correctness, the usage frequency, the comprehensive retrieval quality score and the content generation efficiency in the current time sequence window, an input vector with a shape of (T, 8) is constructed.

6. The method of evidence sourcing for medical model reply content of claim 1, wherein, The first confidence and the first optimal parameter are used to filter the actual candidate text from the knowledge base, comprising: The knowledge text with the first confidence greater than or equal to the first optimal parameter is taken as an initial text, and a second quantity is counted; wherein the second quantity is the number of initial texts; determine whether the second quantity is greater than K; If greater, the initial texts are sorted in descending order of the first confidence; select the top K initial texts as the actual candidate texts; If not greater, the initial texts are taken as the actual candidate texts.

7. The method of evidence sourcing for medical model reply content of claim 1, wherein, The actual reply content is semantically segmented into a plurality of actual fine-grained segments, comprising: based on the available GPU memory size, the semantic segmentation model is used to divide the semantic segmentation task of the actual reply content into batches to obtain subtasks in each batch; initialize a thread pool, and use each thread in the thread pool to execute the subtasks in the batch in parallel; When all the subtasks are executed, a plurality of actual fine-grained segments are generated.

8. A computer apparatus, comprising: The computer device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of evidence provenance of medical model reply content of any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing a computer to perform the method of evidence provenance of medical model reply content of any one of claims 1-7.

10. A computer program product comprising computer instructions, characterized in that, The computer instructions, when executed by the processor, implement the steps of the method of any one of claims 1-7.

Images