Method and system for optimizing large model questions and answers based on multi-scale fine-grained feedback

Abstract

The invention relates to the technical field of natural language processing, in particular to a method and system for optimizing large model questions and answers based on multi-scale fine-grained feedback, a candidate reply set is generated through multi-model collaboration, and then the large model questions and answers are obtained through a fine-grained judgment model of a multi-dimensional evaluation system including loyalty, integrity, conciseness and the like. The method comprises the following steps: performing structured scoring on each reply, generating a detailed scoring report, and finally, dynamically optimizing target model parameters by adopting a direct preference optimization algorithm and combining user-defined dimension weight distribution to enable an output result to adapt to different scene requirements. According to the method, through multi-model collaborative generation, fine-grained feedback learning and dynamic preference adaptation, the accuracy, reliability and scene adaptability of question and answer reply are remarkably improved, and the method can be widely applied to the fields such as intelligent customer service, medical consultation and education assistance which have strict requirements on answer quality.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a method and system for optimizing large model question answering based on multi-scale fine-grained feedback. Background Technology

[0002] Contextual question answering is an important research direction in the field of natural language processing. It refers to a model generating an answer to a user's question based on one or more pieces of relevant background text (i.e., "context"). This task is widely used in high-value scenarios such as open-domain question answering, academic question answering systems, document retrieval, automated legal and regulatory consultation, and medical information systems. With the widespread application of large language models, such as GPT, PaLM, and Claude, the performance of contextual question answering has made significant progress, but it still faces many challenges in practical applications.

[0003] Current mainstream methods often concatenate the context and question and input them into a large language model, relying on its zero-shot or few-shot capability to generate answers. However, this method lacks the ability to uniformly model multiple quality indicators (such as fidelity, completeness, and conciseness). In practical applications, the following problems often occur: (1) severe hallucination phenomenon, with the generated content inconsistent with the original context; (2) one-sided answers, omission of key information, and insufficient information completeness; (3) verbose and redundant language, affecting user experience; and these problems are difficult to effectively alleviate during the training phase using a single loss function.

[0004] To address these issues, researchers have attempted to improve question-answering generation quality from multiple dimensions. For example, Zhang et al. (2020) proposed the FAQA (Feature-Frequency Evaluation) metric for "fidelity," which measures fidelity by calculating the semantic consistency between the answer and the original text. Fabbri et al. (2021) introduced a multidimensional quality evaluation framework in the field of summary generation, including concepts such as consistency, coverage, and conciseness, laying the theoretical foundation for multidimensional feedback mechanisms. Works such as Longformer (Beltagy et al., 2020) focus on extending the context window of the Transformer model to capture information from long documents, indirectly influencing question-answering performance from the dimension of "context length."

[0005] In the area of ​​feedback optimization, recent algorithms such as Direct Preference Optimization (DPO, Rafailov et al., 2023) and Instructive Preference Optimization (IPO, Xu et al., 2023) have begun to explore how to incorporate human preference signals into the model training process. However, these works are often based on binary preference feedback (i.e., "whether it's better or not"), lacking structured, multi-dimensional scoring mechanisms, making it difficult to explain the sources of model bias and to finely control different aspects of the model's output quality.

[0006] Furthermore, existing "reward model + RLHF" processes are costly, training is unstable, and they only optimize for a single quality objective (such as fidelity), failing to simultaneously consider other user-focused dimensions, such as completeness and simplicity. More importantly, most methods remain at a coarse-grained level in their evaluation mechanisms, failing to refine them to various sub-dimensional factors (such as factual consistency, style fit, and knowledge injection).

[0007] Chinese patent CN117076653B discloses a knowledge base question-answering method based on thought chains and visualization to enhance contextual learning. The method includes: retrieving examples similar to the query question from the knowledge base; generating a logical reasoning process for both the similar examples and the query question using thought chains; learning the logical reasoning process using a CodeGeex2 model to generate the logical form of the query question; obtaining the user-corrected logical form; extracting knowledge, binding entities, and binding relationships for the query question in the knowledge base; determining the answer to the query question using a majority vote strategy; and outputting the answer to the user. While this method introduces thought chains before the model generates the logical form and utilizes symbolic memory to improve the multi-hop reasoning ability of complex questions in large language models, it lacks comprehensive consideration of the overall fidelity, completeness, and conciseness of the context. Therefore, it is prone to problems such as one-sided answers and verbose language. Designing a learning framework that can flexibly adjust quality dimensions, possesses structured scoring capabilities, and supports efficient feedback optimization has become an urgent problem to be solved in current technological development. Summary of the Invention

[0008] To address these issues, this invention provides a method and system for optimizing large-scale question answering models based on multi-scale fine-grained feedback, thereby overcoming the problems of unstable response quality, single evaluation dimension, coarse-grained feedback mechanism, low training efficiency, and insufficient scene adaptability of existing large language models in contextual question answering tasks.

[0009] To achieve the above objectives, this invention provides a method for optimizing large-scale question answering models based on multi-scale fine-grained feedback, including multi-model collaborative generation of candidate responses, inputting context text and question text into N large language models to output candidate responses;

[0010] Multi-dimensional fine-grained scoring: Based on multi-dimensional evaluation criteria, a pre-trained discriminative model scores the candidate responses and outputs corresponding scoring reasons. The dimensions include fidelity, completeness, and simplicity.

[0011] Dynamic preference optimization receives the dimensional weight allocation, generates a preference pair dataset based on the scores output by the multi-dimensional evaluation criteria, outputs the optimization parameters of the large language model using a weighted multi-task loss function, and updates the large language model using a direct preference optimization algorithm based on the optimization parameters.

[0012] Where N is greater than or equal to 3.

[0013] Furthermore, the context text and question text input to the N large language models are preprocessed;

[0014] The preprocessing methods include sentence segmentation, noise character removal, and standardized encoding.

[0015] Furthermore, the process of outputting responses from the N large language models includes,

[0016] Set the temperature parameters, Top-p sampling values, and repetition penalties for the large language models. Input the preprocessed context text and the question text into N large language models simultaneously to get N responses.

[0017] Deduplication of responses is performed based on semantic consistency to obtain candidate responses.

[0018] Furthermore, the process of deduplicating responses based on semantic consistency includes,

[0019] An algorithm is used to calculate text similarity, and duplicate responses are removed when the similarity is greater than the preset similarity.

[0020] Furthermore, the scoring criteria for the fidelity dimension include the factual consistency sub-criteria, the adjective matching sub-criteria, the knowledge consistency sub-criteria, and the style consistency sub-criteria;

[0021] The integrity dimension includes the key information coverage sub-standard, the detail sufficiency sub-standard, and the logical coherence sub-standard.

[0022] The simplicity dimension includes sub-standards such as the proportion of redundant information, sentence complexity, and information density.

[0023] Furthermore, the scoring formula for the multi-dimensional fine-grained scoring is as follows:

[0024] Overall score for the loyalty dimension SF1 is the score for the factual consistency subcriterion, SF2 is the score for the adjective matching subcriterion, SF3 is the score for the knowledge consistency subcriterion, and SF4 is the score for the style consistency subcriterion.

[0025] Overall score for simplicity dimension SP1 is the score for the key information coverage sub-criteria, SP2 is the score for the detail sufficiency sub-criteria, and SP3 is the score for the logical coherence sub-criteria.

[0026] Total score for completeness dimension SC1 is the score for the redundancy information ratio, SC2 is the score for the sentence complexity, and SC3 is the score for the information density.

[0027] Furthermore, the process of pre-training the discriminative model includes,

[0028] Supervised training is performed on a question-and-answer dataset with fine-grained annotations, where the fine-grained annotations include the scores for the main dimension and sub-criteria of the corresponding response and the corresponding scoring reasons.

[0029] The parameters of the discriminant model are jointly optimized by using the cross-entropy loss function and the consistency regularization term.

[0030] Furthermore, the dynamic preference optimization process includes,

[0031] The receiving dimension weights are assigned, where α1 is the fidelity dimension weight, α2 is the integrity dimension weight, and α3 is the simplicity dimension weight, and α1+α2+α3=1;

[0032] The scores output by the multi-dimensional evaluation criteria are generated based on size rules to produce preference pairs for each dimension, and a preference dataset for each dimension is constructed.

[0033] The corresponding score difference is calculated based on the dimensional weights, the preference pairs in the preference dataset, and the scores output by the multi-dimensional evaluation criteria for each candidate response;

[0034] A weighted multi-task loss function is used to output the preference loss for each dimension of the large language model. The total loss is calculated based on the dimension weights and the preference losses for each dimension. The large language model is then updated using a direct preference optimization algorithm based on the total loss. The formula for calculating the total loss is as follows:

[0035] L total =α1L faithfulness +α2L completeness +α3L conciseness ,

[0036] Where L is the preference loss of the direct preference optimization algorithm for the corresponding dimension.

[0037] Furthermore, the formula for calculating the preference loss in the direct preference optimization algorithm is as follows:

[0038]

[0039] Where β is the temperature parameter, π θ For the target model, π ref As the reference model, D represents the preference pair dataset, and L... DPO Let x be the question text and y be the preference loss. w For the preference response, y l This is a non-preference response.

[0040] Secondly, the present invention also provides a large-model question-answering system based on multi-scale fine-grained feedback optimization, comprising:

[0041] The user interaction module is used to receive question text, dimension weight allocation, and output the optimized response quality comparison results.

[0042] The data acquisition module, which is connected to the user interaction module, is used to receive context text and question text;

[0043] The preprocessing module, which is connected to the acquisition module, is used to segment the received context text and question text into sentences, remove noise characters, and standardize the encoding.

[0044] The multi-model generation module, which is connected to the preprocessing module, contains several large language models and is used to call the large language models to generate several candidate responses based on the preprocessed context text and question text.

[0045] The fine-grained scoring module, which is connected to the multi-model generation module, is used to call the pre-trained discriminant model to score candidate responses based on multi-dimensional evaluation criteria and output the corresponding scoring reasons.

[0046] The dynamic optimization module, which is connected to the fine-grained scoring module and the user interaction module, is used to generate a preference dataset based on the scores, generate the score difference of each candidate response based on the dimension weights, output the optimization parameters of the large language model based on the weighted multi-task loss function, and update the large language model based on the optimization parameters using the direct preference optimization algorithm.

[0047] Compared with existing technologies, the advantages of this invention lie in its multi-model collaborative generation of candidate responses. This fully leverages the strengths of different large language models to generate diverse answers, avoiding the limitations of a single model. Combining a fine-grained discriminative model with multi-dimensional (fidelity, completeness, conciseness) scoring of candidate responses allows for a comprehensive and accurate evaluation of each response's quality, providing a scientific basis for subsequent optimization. Furthermore, dynamic preference optimization adjusts the target model parameters based on user-defined dimensional weights, enabling the output to flexibly adapt to different scenarios. This significantly improves the accuracy, reliability, and scenario adaptability of question-and-answer responses, effectively solving problems such as unstable response quality, single evaluation dimensions, and coarse-grained feedback mechanisms in existing technologies. It can be widely applied in fields with stringent requirements for response quality, such as intelligent customer service, medical consultation, and educational assistance, demonstrating significant practical application value and broad market prospects.

[0048] Furthermore, segmenting, denoising, and standardizing the input text can improve input quality, optimize model understanding, and enhance generation effectiveness and efficiency. Sentence segmentation helps to grasp the text structure, denoising reduces interference, and standardization ensures cross-platform compatibility, providing a foundation for high-quality response generation and enhancing system stability and reliability.

[0049] Furthermore, setting temperature parameters, Top-p sampling values, and duplication penalties can control the diversity of generated responses, avoiding single and repetitive answers. Deduplication based on semantic consistency can improve candidate response diversity, remove redundant content, save computational resources, improve scoring accuracy and user experience, provide high-quality data for subsequent optimization, and enhance system performance and output quality.

[0050] Furthermore, by subdividing the dimensions of fidelity, completeness, and conciseness into sub-criteria, this method can conduct a refined evaluation of the quality of question-and-answer responses, effectively avoiding issues such as factual illusions, information omissions, and linguistic redundancy, thereby significantly improving the accuracy, comprehensiveness, and conciseness of the answers and enhancing the overall quality of question-and-answer responses and user experience.

[0051] Furthermore, by weighting and summing the scores of each sub-criteria according to certain weights, the impact of different sub-criteria on the overall dimension can be comprehensively considered, resulting in a more accurate and reasonable overall dimension score. This quantitative scoring method not only improves the transparency and interpretability of the scoring, but also facilitates subsequent analysis and processing of the scoring results, providing a more reliable basis for dynamic preference optimization. This further enhances the controllability of the overall system's performance and output quality, ensuring the optimization effect of question-and-answer responses across all dimensions.

[0052] Furthermore, by conducting supervised training on a question-and-answer dataset with fine-grained annotations, and jointly optimizing the discriminant model parameters using the cross-entropy loss function and consistency regularization term, we can ensure a high degree of consistency between the scoring results of the discriminant model and the manually annotated data, thereby improving the accuracy and reliability of the discriminant model. Fine-grained annotations include scores for the corresponding main dimension and sub-criteria of the response, along with the corresponding scoring reasons. This rich annotation information provides ample guidance for the discriminant model's learning, enabling it to better understand and master the scoring rules and standards of different dimensions and sub-criteria. This, in turn, generates more accurate and reasonable scores and scoring reasons for candidate responses, providing stronger support for the entire system's optimization process and further improving the overall performance and reliability of the output quality.

[0053] Furthermore, the dynamic preference optimization process accepts user-defined dimension weight assignments, enabling the system to flexibly adjust the importance of each dimension according to different scenarios and user needs, thereby achieving personalized optimization of the target model. Based on the scores output by multi-dimensional evaluation criteria, preference pairs are generated, and preference datasets for each dimension are constructed. A weighted multi-task loss function is used to output the preference losses for each dimension of the large language model and calculate the total loss. This comprehensively considers the optimization needs of different dimensions, avoiding the limitations of single-dimensional optimization. Finally, the direct preference optimization algorithm is used to update the large language model based on the total loss, realizing dynamic adjustment and optimization of model parameters. This allows the model output to better adapt to the needs of different scenarios, further improving the system's flexibility, adaptability, and output quality optimization, providing users with higher quality and more targeted question-and-answer responses.

[0054] Furthermore, by combining the user-defined dimensional weights and the preference losses of each dimension to calculate the total loss, and then using the direct preference optimization algorithm to update the large language model, the transparency and operability of the model optimization process can be ensured. This further improves the optimization efficiency and controllability of the output quality of the system, enabling the model to more accurately meet the diverse needs of users in different scenarios, and enhancing the overall performance and practicality of the system.

[0055] Furthermore, the multi-scale fine-grained feedback optimization large-model question-answering system described in this invention can achieve fully automated processing from question reception, text acquisition, preprocessing, multi-model generation, fine-grained scoring to dynamic optimization. The user interaction module facilitates user input of question text and dimension weight allocation, and can output optimized response quality comparison results in real time, improving user experience and system interactivity. The collaborative work between the acquisition module, preprocessing module, multi-model generation module, fine-grained scoring module, and dynamic optimization module ensures the efficient operation and high-quality output of the entire system. Through multi-model collaborative generation, fine-grained feedback learning, and dynamic preference adaptation, this system significantly improves the accuracy, reliability, and scenario adaptability of question-answering responses. It can be widely applied in fields with stringent requirements for response quality, such as intelligent customer service, medical consultation, and educational assistance, possessing significant practical application value and broad market prospects, and providing strong technical support for the intelligent development of related fields. Attached Figure Description

[0056] Figure 1 This is a block diagram of the large model question answering system based on multi-scale fine-grained feedback optimization in this invention;

[0057] Figure 2 This is a flowchart of the method for optimizing large model question answering based on multi-scale fine-grained feedback in this invention;

[0058] Figure 3 This is an architecture diagram of the large model question answering system based on multi-scale fine-grained feedback optimization in this invention;

[0059] Figure 4 This is a flowchart of the dynamic preference optimization described in this invention. Detailed Implementation

[0060] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0061] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0062] Please see Figure 1 The diagram shown is a block diagram of the large-scale question-answering system based on multi-scale fine-grained feedback optimization in this invention. The large-scale question-answering system based on multi-scale fine-grained feedback optimization, as described in this invention, includes a user interaction module, a data acquisition module, a preprocessing module, a multi-model generation module, a fine-grained scoring module, and a dynamic optimization module.

[0063] The user interaction module is used to receive question text, receive dimension weight allocation, and output optimized response quality comparison results.

[0064] The acquisition module is connected to the user interaction module and is used to receive context text and question text;

[0065] The preprocessing module, which is connected to the acquisition module, is used to segment the received context text and question text into sentences, remove noise characters, and standardize the encoding.

[0066] The multi-model generation module, which is connected to the preprocessing module, contains several large language models and is used to call the large language models to generate several candidate responses based on the preprocessed context text and question text.

[0067] The fine-grained scoring module, which is connected to the multi-model generation module, is used to call a pre-trained discriminant model to score candidate responses based on multi-dimensional evaluation criteria and output corresponding scoring reasons.

[0068] The dynamic optimization module, which is connected to the fine-grained scoring module and the user interaction module, is used to generate a preference dataset based on the scores, generate the score difference of each candidate response based on the dimension weights, output the large language model optimization parameters based on the weighted multi-task loss function, and update the large language model based on the optimization parameters using the direct preference optimization algorithm.

[0069] Specifically, the user interaction module includes preset scene templates, which include medical mode, customer service mode and education mode;

[0070] In this context, the weight α1 ∈ [0.7, 0.9] is set for the fidelity dimension in the medical model;

[0071] Set the weight α3∈[0.6,0.8] for the simplicity dimension in the customer service mode;

[0072] The weight α2 ∈ [0.5, 0.6] is set for the integrity dimension in the education model;

[0073] The user interaction module provides a weight configuration slider, which allows users to adjust the weights α1 for loyalty, α2 for integrity, and α3 for simplicity in real time.

[0074] The multi-model generation module supports API calls and local model deployment, and provides a model management interface that allows users to dynamically add or remove language models.

[0075] The fine-grained scoring module integrates a fact verifier based on the Wikidata knowledge graph; it constructs a logic parser based on Stanford CoreNLP dependency parsing; and it constructs a style classifier based on Fine-tuned BERT.

[0076] The dynamic optimization module can visualize the training curve and show the trend of loss changes in each dimension.

[0077] Please see Figure 2 The diagram shows a flowchart of the method for optimizing large-scale question answering models based on multi-scale fine-grained feedback in this invention. The steps of the method include:

[0078] S1: Multi-model collaborative generation of candidate responses, inputting the context text and question text into N large language models to output candidate responses;

[0079] S2: Multi-dimensional fine-grained scoring, based on multi-dimensional evaluation criteria, the candidate responses are scored by a pre-trained discriminant model and the corresponding scoring reasons are output. The dimensions include fidelity, completeness and conciseness.

[0080] S3: Dynamic preference optimization, receiving the dimension weight allocation, generating a preference pair dataset based on the scores output by the multi-dimensional evaluation criteria, outputting the large language model optimization parameters using a weighted multi-task loss function, and updating the large language model based on the optimization parameters using a direct preference optimization algorithm.

[0081] The fidelity dimension ensures that the answer is semantically consistent with the context and the question, avoiding the introduction of errors or false information. This is especially crucial in professional fields such as medicine and law, enhancing the system's credibility. The completeness dimension guarantees that the answer covers all key information, providing sufficient detail and logical coherence, avoiding omissions, and allowing users to fully understand the answer. This is particularly important in complex question-and-answer scenarios. The conciseness dimension ensures that the answer is clear and concise, avoiding redundancy and verbosity, quickly conveying core information, saving users time and effort. This is particularly suitable for scenarios such as customer service and news reporting. By fine-grained scoring and optimization across these three dimensions, high-quality, diverse answers can be generated, improving user experience and system reliability, while meeting the specific answer quality requirements of different scenarios.

[0082] A large language model is an artificial intelligence model trained using deep learning techniques, specifically designed to process and understand human language. A large language model can be initialized with different parameters for the same structure, or a combination of pre-trained models with different architectures, including but not limited to any one or more of GPT-4, Claude, PaLM, and DeepSeek, which will not be elaborated further.

[0083] Among the N large language models, N is greater than or equal to 3.

[0084] Furthermore, preprocessing the context text and question text input to N large language models can improve input quality, optimize model understanding, enhance generation performance, and increase efficiency, thereby providing more accurate and efficient input to the large language models and further optimizing question-answering performance.

[0085] The preprocessing methods include sentence segmentation, noise character removal, and standardized encoding.

[0086] The method of sentence segmentation is not limited in principle. Technicians can choose different sentence segmentation methods according to the characteristics of the text. The sentence segmentation methods include, but are not limited to, using NLP libraries for sentence segmentation and using pre-trained models for sentence segmentation, which will not be elaborated here.

[0087] The noise characters refer to unnecessary characters in the text that do not help with text analysis or understanding, or may even interfere with the processing results. These include, but are not limited to, any one or more of the following: punctuation marks, special characters, HTML tags, stop words, emoticons, and spelling errors. The method for removing noise characters is not limited, and technicians can choose the removal method according to the text being processed.

[0088] The standardization encoding refers to converting text into a unified, widely supported character encoding format to ensure that the text can be correctly displayed and processed across different systems, platforms, and applications.

[0089] The steps of the standardized encoding are not limited in principle. The basic steps include determining the target encoding format, detecting the current encoding, using the encoding conversion function in a text editor or programming language to convert the text to the target encoding, deciding whether to retain or remove the byte order mark as needed, verifying whether the text has been correctly encoded after conversion, ensuring that there are no garbled characters or missing characters, saving the encoded text as a new file or overwriting the original file, and ensuring that the correct encoding format is used for saving.

[0090] Furthermore, the process of outputting responses from the N large language models includes,

[0091] Set the temperature parameters, Top-p sampling values, and repetition penalties for the large language models. Input the preprocessed context text and the question text into N large language models simultaneously to get N responses.

[0092] Deduplication of responses is performed based on semantic consistency to obtain candidate responses.

[0093] Furthermore, the process of deduplicating responses based on semantic consistency includes using an algorithm to calculate text similarity, and deduplicating responses when the similarity is greater than a preset similarity.

[0094] Deduplicating responses based on semantic consistency helps improve the diversity of candidate responses, avoids wasting computational resources and interfering with scoring accuracy by repeating content, thereby improving user experience and optimizing the effect of dynamic preference adjustment.

[0095] Specifically, the process of the N large language models outputting candidate responses includes:

[0096] Setting the control temperature parameter β∈[0.5,0.8] helps to increase the diversity of generation;

[0097] Set Top-p∈[0.7, 0.9] to retain the top 70%-90% of words in terms of probability quality;

[0098] The repetition penalty is set to λ∈[1,1.3] to suppress the generation of repetitive segments.

[0099] Controlling the settings for temperature parameters, Top-p, and repetition penalty helps ensure response diversity, thereby providing more valuable samples for candidate responses.

[0100] The input sequence, containing context and question, is fed in parallel into N large language models to generate N candidate response sets {O1, O2, ..., O...}. N};

[0101] The MinHash algorithm is used to calculate text similarity. If the Jaccard similarity G between two responses is greater than the preset similarity G1, the redundant response is removed. After removing the redundant response, candidate responses are obtained. The preset similarity G1 is set to [0.7, 0.9].

[0102] Furthermore, the scoring criteria for the fidelity dimension include the factual consistency sub-criteria, the adjective matching sub-criteria, the knowledge consistency sub-criteria, and the style consistency sub-criteria;

[0103] The integrity dimension includes the key information coverage sub-standard, the detail sufficiency sub-standard, and the logical coherence sub-standard.

[0104] The simplicity dimension includes sub-standards such as the proportion of redundant information, sentence complexity, and information density.

[0105] The fidelity dimension scoring criteria encompass four sub-criteria: factual consistency, adjective matching, knowledge consistency, and style consistency, to comprehensively evaluate the fidelity of the response. These sub-criteria ensure that the response is consistent with the context in terms of factual accuracy, semantic matching, knowledge relevance, and language style, thereby improving the quality and reliability of the response and meeting the needs of diverse scenarios. The completeness dimension covers the sub-criteria of key information coverage, detail sufficiency, and logical coherence to ensure that the response is comprehensive, detailed, and logically clear, improving quality and user experience. The conciseness dimension evaluates the refinement of the response through three sub-criteria: redundancy percentage, sentence complexity, and information density. Redundancy percentage measures the amount of repetitive or irrelevant content in the response, avoiding verbosity; sentence complexity assesses whether the sentence structure of the response is concise and clear, facilitating understanding; and information density measures the amount of effective information conveyed per unit of words, ensuring that the response efficiently conveys the core content. The purpose of these sub-criteria is to ensure that the response is concise and clear, quickly conveys key information, and improves user experience and system efficiency.

[0106] Specifically, the evaluation method for the factual consistency sub-standard is to check the matching degree between entities such as numerical values ​​and proper nouns and the source text, and to score the factual consistency sub-standard based on the matching degree.

[0107] The evaluation process for the adjective matching sub-criteria is to assess the semantic equivalence of the adjectives;

[0108] The evaluation process for the knowledge consistency sub-standard is to detect whether external knowledge not mentioned in the context has been introduced.

[0109] The evaluation process for the style consistency sub-criterion involves analyzing the matching degree between sentence complexity and context.

[0110] The evaluation process for the key information coverage sub-standard is to calculate the proportion of contextual key entities mentioned in the response.

[0111] The evaluation process for the detail sufficiency sub-criterion is to analyze whether the response contains modifiers from the context;

[0112] The evaluation process for the logical coherence sub-criterion is to detect missing or contradictory causal chains through dependency syntax trees.

[0113] The evaluation process for the redundancy information ratio sub-standard is to count the proportion of repeated phrases or irrelevant descriptions.

[0114] The evaluation process for the sentence complexity sub-criteria is to calculate the average sentence length and the number of nested clauses, and deduct points if the average sentence length is higher than the average level of the context.

[0115] The evaluation process for the information density sub-standard is to use TF-IDF weighted evaluation to assess the amount of information conveyed per unit word.

[0116] The scoring range for each sub-criteria is 1-5 points.

[0117] Specifically, the implementation methods of the factual consistency sub-standard, the adjective matching sub-standard, the knowledge consistency sub-standard, the style consistency sub-standard, the key information coverage sub-standard, the detail sufficiency sub-standard, the logical coherence sub-standard, the redundant information ratio sub-standard, the sentence complexity sub-standard, and the information density sub-standard are as follows:

[0118] The implementation of the fact consistency sub-standard is based on precise string matching and named entity recognition.

[0119] The adjective matching sub-criteria is implemented by using a pre-trained Sentence-BERT model to calculate the semantic similarity between the candidate response and the adjectives in the context. The adjective matching similarity threshold is set to τ∈[0.6,0.8], and if it is lower than the threshold, it is marked as a mismatch.

[0120] The knowledge consistency sub-standard is implemented by detecting whether external knowledge is introduced through a rule engine. If the context of the words mentioned in the candidate response is not involved, it is marked as external knowledge injection.

[0121] The implementation of the style consistency sub-standard is to use a pre-trained style classifier (such as RoBERTa) to determine the degree of academic rigor matching between the response and the context.

[0122] The key information coverage sub-standard is implemented by calculating the proportion of contextual key entities mentioned in the response, using the following formula:

[0123] Furthermore, the scoring formula for the multi-dimensional fine-grained scoring is as follows:

[0124] Overall score for the loyalty dimension SF1 is the score for the factual consistency subcriterion, SF2 is the score for the adjective matching subcriterion, SF3 is the score for the knowledge consistency subcriterion, and SF4 is the score for the style consistency subcriterion.

[0125] Overall score for simplicity dimension SP1 is the score for the key information coverage sub-criteria, SP2 is the score for the detail sufficiency sub-criteria, and SP3 is the score for the logical coherence sub-criteria.

[0126] Total score for completeness dimension SC1 is the score for the redundancy information ratio, SC2 is the score for the sentence complexity, and SC3 is the score for the information density.

[0127] Furthermore, the process of pre-training the discriminative model includes,

[0128] Supervised training is performed on a question-and-answer dataset with fine-grained annotations, where the fine-grained annotations include the scores for the main dimension and sub-criteria of the corresponding response and the corresponding scoring reasons.

[0129] Using the cross-entropy loss function and the consistency regularization term to jointly optimize the parameters of the discrimination model helps to ensure the consistency between the scoring results and the manual annotation.

[0130] Furthermore, the process of dynamic preference optimization includes,

[0131] The receiving dimension weights are assigned, where α1 is the fidelity dimension weight, α2 is the integrity dimension weight, and α3 is the simplicity dimension weight, and α1+α2+α3=1;

[0132] The scores output by the multi-dimensional evaluation criteria are used to generate preference pairs based on size rules, and a preference dataset is constructed.

[0133] The corresponding score difference is calculated based on the dimensional weights, the preference pairs in the preference dataset, and the scores output by the multi-dimensional evaluation criteria for each candidate response;

[0134] A weighted multi-task loss function is used to output the preference loss for each dimension of the large language model. The total loss is calculated based on the dimension weights and the preference losses for each dimension. The large language model is then updated using a direct preference optimization algorithm based on the total loss. The formula for calculating the total loss is as follows:

[0135] L total =α1L faithfulness +α2L completeness +α3L conciseness ,

[0136] Where L is the preference loss of the direct preference optimization algorithm for the corresponding dimension.

[0137] Furthermore, the formula for calculating the preference loss in the direct preference optimization algorithm is as follows:

[0138]

[0139] Where β is the temperature parameter, π θ For the target model, π ref As the reference model, D represents the preference pair dataset, and L... DPO Let x be the question text and y be the preference loss. w For the preference response, y l This is a non-preference response.

[0140] Please see Figure 4 As shown, it is a flowchart of the dynamic preference optimization described in this invention. The dynamic preference optimization process includes:

[0141] Obtain the user-defined weights for loyalty (α1), integrity (α2), and simplicity (α3);

[0142] Preference pairs are generated based on a size rule: Reply ratings are iterated, and if reply A's rating > reply B's rating, a preference pair for that dimension is generated: A > B. This is then used to construct a loyalty dimension preference dataset D. faithfulness Integrity Dimension Preference Dataset D completeness Simplicity dimension preference dataset D conciseness ;

[0143] The corresponding score difference is calculated based on the dimensional weights, the preference pairs in the preference dataset, and the scores output by the multi-dimensional evaluation criteria for each candidate response;

[0144] The weighted composite score is calculated based on the dimensional weights and the scores output by the multi-dimensional evaluation criteria for each candidate response. The weighted composite score for each response is Score. i =α1S faithfulness_i +α2S completeness_i +α3S conciseness_i Score i Let i be the i-th reply, where i = 1, 2, 3, ..., n;

[0145] The score difference for the corresponding dimension is then the difference in the weighted composite score of the corresponding preference pair. This is calculated by iterating through the loyalty dimension preference dataset D. faithfulness Integrity Dimension Preference Dataset D completeness Simplicity dimension preference dataset D conciseness The score difference is calculated sequentially, and the score with the highest weight and the best score difference is selected to update the model.

[0146] The loyalty dimension preference dataset D faithfulness Integrity Dimension Preference Dataset D completeness Simplicity dimension preference dataset D conciseness Substituting these values ​​into the preference loss calculation formula of the direct preference optimization algorithm,

[0147]

[0148] And input the temperature parameter β and the target model π θ and reference model π ref Question text x, preferred response y w Non-preference response y l The loyalty preference loss L is obtained. faithfulness Integrity preference loss L completeness And simplicity preference loss L conciseness ;

[0149] The user-defined weights for loyalty (α1), completeness (α2), and simplicity (α3), along with the loyalty preference loss L, are used. faithfulness Integrity preference loss L completeness And simplicity preference loss L conciseness Substitute into the total loss calculation formula L total =α1L faithfulness +α2L completeness +α3L conciseness The total loss was incurred.

[0150] The AdamW optimizer is used to iteratively update the target model parameters until the validation set loss converges.

[0151] Example 1

[0152] The response is generated using the multi-scale fine-grained feedback optimization large model question-answering system described in this invention, with the preset scenario template being medical mode;

[0153] In this embodiment, the context text is: a medical literature document, which includes... liver cancer treatment methods include surgical resection, radiofrequency ablation, and targeted drug therapy...;

[0154] The question is: What treatment methods are mentioned in this literature?

[0155] The workflow of the large-model question-answering system based on multi-scale fine-grained feedback optimization is as follows:

[0156] The user interaction module receives dimension weights of α1 = 0.8, α2 = 0.15, and α3 = 0.05.

[0157] The acquisition module collects the context text "...Liver cancer treatment methods include surgical resection, radiofrequency ablation and targeted drug therapy...";

[0158] The preprocessing module uses an NLP library to segment sentences, then removes HTML tags from the text and converts the text to UTF-8 format;

[0159] The multi-model generation module is set with temperature parameter β = 0.7, Top-p sampling p = 0.9, and repetition penalty λ = 1.2.

[0160] The multi-model generation module simultaneously calls the large language models GPT-4, Claude-2, and PaLM-2 to generate answers, and the output answer is:

[0161] O1 (GPT-4): Treatment options for liver cancer include surgical resection, radiofrequency ablation, and targeted drug therapy.

[0162] O2 (Claude-2): Treatment options include surgical resection, radiofrequency ablation, and targeted drug therapy.

[0163] O3 (PaLM-2): Treatment options for liver cancer include surgical resection and targeted drug therapy.

[0164] Generate a response set {O1, O2, O3};

[0165] The MinHash algorithm is used to remove duplicates from the candidate response set {O1, O2, O3}. The Jaccard similarity is set to be greater than or equal to 0.8. After removing redundant responses, the candidate responses {O1, O3} are output.

[0166] The fine-grained scoring module calls the discriminant model to score the candidate responses {O1, O3};

[0167] Rating for O1's fidelity dimension:

[0168] Factual consistency: 5; Adjective matching: 4; Knowledge consistency: 5; Style consistency: 5

[0169] Total score of the loyalty dimension

[0170] Scoring criteria: Factual consistency: The response is completely consistent with the literature, with no deviation; Adjective matching: The original text uses "including", while the response uses "has", but the scoring criteria require strict matching of modifiers, so 1 point is deducted; Knowledge consistency: The response is consistent with medical knowledge; Style consistency: The formal academic style matches the literature.

[0171] Scoring of O1's integrity dimension:

[0172] Key information coverage: 5; Detail sufficiency: 4; Logical coherence: 5

[0173] Completeness dimension overall score S completeness =4.67;

[0174] Scoring criteria: Coverage of key information: Covers all treatment methods (surgical resection, radiofrequency ablation, targeted drug therapy); Sufficient detail: The response concisely lists the methods, but does not expand on details (such as indications or efficacy), so it is given 4 points; Logical coherence: The list is clear and logically sound.

[0175] Rating for O1's simplicity:

[0176] Redundancy percentage: 5%; Sentence complexity: 5%; Information density: 5%.

[0177] Simplicity dimension overall score S conciseness =5;

[0178] Reasons for the rating: Redundancy percentage: No redundant words or sentences; Sentence complexity: Simple declarative sentences, easy to understand; Information density: High, directly conveying key information.

[0179] Rating for O3's fidelity dimension:

[0180] Factual consistency: 3; Adjective matching: 4; Knowledge consistency: 3; Style consistency: 5; Total score for fidelity dimension: S faithfulness =3.75;

[0181] Scoring criteria: Factual consistency: "radiofrequency ablation" is omitted, which is inconsistent with the literature; Adjective matching: "has" is used instead of "includes", which is a partial match but not precise enough; Knowledge consistency: The knowledge is incomplete, and standard treatment methods are omitted; Style consistency: The style is formal and matches the literature.

[0182] Scoring of O3's integrity dimension:

[0183] Key information coverage: 2; Detail sufficiency: 3; Logical coherence: 5;

[0184] Completeness dimension overall score S completeness =3.33;

[0185] Rating criteria: Coverage of key information: "Radiofrequency ablation" was omitted, indicating incomplete key information; Sufficient detail: Only some details were provided, and not all methods were covered; Logical coherence: The response was logically clear, but missing information affected the overall effect.

[0186] Rating for O3's simplicity:

[0187] Redundancy percentage: 5%; Sentence complexity: 5%; Information density: 4%.

[0188] Simplicity dimension overall score S conciseness =4.67;

[0189] Scoring criteria: Redundancy percentage: No redundancy; Sentence complexity: Simple sentence structure; Information density: Density is reduced due to missing information, so 4 points are awarded.

[0190] The dynamic optimization module generates preference pairs: fidelity: O1>O3; completeness: O1>O3; simplicity: O1>O3;

[0191] The dynamic optimization module generates a loyalty dimension preference dataset D. faithfulness Completeness Dimension Preference Dataset D completeness And the simplicity dimension preference dataset D conciseness ;

[0192] The dynamic optimization module calculates the loyalty preference loss:

[0193]

[0194] Integrity preference loss:

[0195]

[0196] Simplicity preference loss:

[0197]

[0198] Where, π ref For unoptimized GPT-4, π θ GPT-4 to be optimized;

[0199] Total loss: L total =α1L faithfulness +α2L completeness +α3L conciseness

[0200] Where α1 = 0.8, α2 = 0.15, α3 = 0.05, L is obtained. total =0.686;

[0201] Set the temperature coefficient β = 0.1, the learning rate η = 1e-5, and the batch size B = 32;

[0202] The AdamW optimizer is used to iteratively update the GPT-4 parameters to be optimized until the validation set loss converges.

[0203] The response results were scored based on the fidelity, completeness, and conciseness dimensions using a discriminant model, and coherence was assessed manually. The scores are shown in Table 1.

[0204] Example 2

[0205] The response is generated using the multi-scale fine-grained feedback optimization large model question-answering system described in this invention, with the preset template being customer service mode;

[0206] The context text for Example 2 is as follows: ...Xiaomi headphones are available with a 15% discount, and Huawei headphones are eligible for a 15% discount under the national subsidy...

[0207] Question: What is the maximum discount I can get when buying Bluetooth headphones?

[0208] The workflow of the large-model question-answering system based on multi-scale fine-grained feedback optimization is as follows:

[0209] The user interaction module receives dimension weights of α1 = 0.8, α2 = 0.15, and α3 = 0.05.

[0210] The data acquisition module collects the context text "...Xiaomi headphones are available with a 15% discount, Huawei headphones are available with a 15% discount under the national subsidy..."

[0211] The preprocessing module uses an NLP library to segment sentences, then removes HTML tags from the text and converts the text to UTF-8 format;

[0212] The multi-model generation module is set with temperature parameter β = 0.7, Top-p sampling p = 0.9, and repetition penalty λ = 1.2.

[0213] The multi-model generation module simultaneously calls the large language models GPT-4, Claude-2, and PaLM-2 to generate answers, and the output answer is:

[0214] O1(GPT-4): Based on the context, Xiaomi Bluetooth headphones are eligible for a 15% discount, and Huawei Bluetooth headphones enjoy a national subsidy of 15%, which is equivalent to a 15% discount. Therefore, the maximum discount for purchasing Bluetooth headphones is 15%.

[0215] O2 (Claude-2): Xiaomi headphones are 15% off, Huawei headphones are 15% off with national subsidy, and Bluetooth headphones can be purchased with a maximum discount of 15%.

[0216] O3 (PaLM-2): Maximum discount of 15%.

[0217] Generate a response set {O1, O2, O3};

[0218] The MinHash algorithm is used to remove duplicates from the candidate response set {O1, O2, O3}. The Jaccard similarity is set to be greater than or equal to 0.8. After removing redundant responses, the candidate responses {O1, O3} are output.

[0219] The fine-grained scoring module calls the discriminant model to score the candidate responses {O1, O3};

[0220] Rating for O1's fidelity dimension:

[0221] Factual consistency: 5; Adjective matching: 5; Knowledge consistency: 5; Style consistency: 5;

[0222] Total score S of the loyalty dimension faithfulness =5;

[0223] Rating criteria: Factual consistency: Accurately reflects Xiaomi's 15% discount and Huawei's 15% off, with no errors; Adjective matching: No adjectives in the context, therefore inapplicable, full marks; Knowledge consistency: Discount calculation conforms to common sense; Style consistency: Formal and neutral, consistent with customer service mode.

[0224] Scoring of O1's integrity dimension:

[0225] Key information coverage: 5; Detail sufficiency: 5; Logical coherence: 5;

[0226] Completeness dimension overall score S completeness =5;

[0227] Reasons for the rating: Coverage of key information: Complete coverage of Xiaomi and Huawei brands and details of the offers; Sufficient detail: Provides specific figures and offer mechanisms; Logical coherence: Clear logic from cause to effect.

[0228] Rating for O1's simplicity:

[0229] Redundancy percentage: 4%; Sentence complexity: 4%; Information density: 4%.

[0230] Simplicity dimension overall score S conciseness =4;

[0231] Reasons for scoring: Redundancy: Contains words such as "according to the context" and "therefore", indicating slight redundancy; Sentence complexity: The compound sentence is slightly long; Information density: The information is rich but the number of words is relatively large, resulting in a medium density.

[0232] Rating for O3's fidelity dimension:

[0233] Factual consistency: 4; Adjective matching: 5; Knowledge consistency: 5; Style consistency: 4;

[0234] Total score S of the loyalty dimension faithfulness =4.5;

[0235] Rating reasons: Factual consistency: The maximum discount value is correct, but the brand source is not mentioned, which is incomplete; Adjective matching: There are no adjectives in the context, so it is not applicable, and it is given full marks; Knowledge consistency: The common sense of 15% off is correct; Style consistency: Too simple, and the customer service lacks formality.

[0236] Scoring of O3's integrity dimension:

[0237] Key information coverage: 2; Detail sufficiency: 1; Logical coherence: 4;

[0238] Completeness dimension overall score S completeness =2.3;

[0239] Rating criteria: Coverage of key information: Only mentions the discount value, lacking brand and contextual basis; Sufficient detail: No supporting details; Logical coherence: Simple and straightforward, but incomplete information affects the depth of logic.

[0240] Rating for O3's simplicity:

[0241] Redundancy percentage: 5%; Sentence complexity: 5%; Information density: 5%.

[0242] Simplicity dimension overall score S conciseness =5;

[0243] Reasons for the rating: Redundancy: No superfluous words; Sentence complexity: Simple sentences, easy to understand; Information density: Core information is conveyed efficiently, with few words but sufficient information.

[0244] The dynamic optimization module generates preference pairs: fidelity dimension: O1>O3; integrity dimension: O1>O3; simplicity dimension: O3>O1.

[0245] The dynamic optimization module generates a loyalty dimension preference dataset D. faithfulness Completeness Dimension Preference Dataset D completeness And the simplicity dimension preference dataset D conciseness ;

[0246] The dynamic optimization module calculates the loyalty preference loss:

[0247]

[0248] Integrity preference loss:

[0249]

[0250] Simplicity preference loss:

[0251]

[0252] Where, π ref For unoptimized GPT-4, π θ GPT-4 to be optimized;

[0253] Total loss

[0254] Total loss: L total =α1L faithfulness +α2L completeness +α3L conciseness

[0255] Where α1 = 0.8, α2 = 0.15, α3 = 0.05, L is obtained. total =0.693;

[0256] Set the temperature coefficient β = 0.1, the learning rate η = 1e-5, and the batch size B = 32;

[0257] The AdamW optimizer is used to iteratively update the GPT-4 parameters to be optimized until the validation set loss converges.

[0258] The response results were scored based on the fidelity, completeness, and conciseness dimensions using a discriminant model, and coherence was assessed manually. The scores are shown in Table 1.

[0259] Example 3

[0260] The response is generated using the multi-scale fine-grained feedback optimization large model question-answering system described in this invention, with the preset scenario template being the education mode;

[0261] The context text for Example 3 is as follows: ...Irradiance refers to the light radiation power received per unit area, usually expressed in watts per square meter; radiant flux refers to the radiant energy passing through a cross-section per unit time...

[0262] Question: What do irradiance and radiation flux reflect respectively?

[0263] The workflow of the large-model question-answering system based on multi-scale fine-grained feedback optimization is as follows:

[0264] The user interaction module receives dimension weights of α1 = 0.3, α2 = 0.6, and α3 = 0.1.

[0265] The acquisition module collects the context text: "...Irradiance refers to the light radiation power received per unit area, usually expressed in watts per square meter; radiant flux refers to the radiant energy passing through a cross-section per unit time..."

[0266] The preprocessing module uses an NLP library to segment sentences, then removes HTML tags from the text and converts the text to UTF-8 format;

[0267] The multi-model generation module is set with temperature parameter β = 0.7, Top-p sampling p = 0.9, and repetition penalty λ = 1.2.

[0268] The multi-model generation module simultaneously calls the large language models GPT-4, Claude-2, and PaLM-2 to generate answers, and the output answer is:

[0269] O1(GPT-4): Irradiance reflects the light radiation power received per unit area, i.e., the intensity distribution of radiation; radiant flux reflects the radiant energy passing through a cross section per unit time, i.e., the transmission rate of radiant power.

[0270] O2 (Claude-2): Irradiance represents the radiant power received over a given area, reflecting the intensity of light; radiant flux represents the radiant energy flowing through a region per unit time, reflecting the rate of energy flow.

[0271] O3 (PaLM-2): Irradiance is the radiant power received per square meter, which measures the intensity of light; radiant flux is the radiant energy passing through per second, which represents the radiant flow.

[0272] Generate a response set {O1, O2, O3};

[0273] The MinHash algorithm is used to remove duplicates from the candidate response set {O1, O2, O3}. The Jaccard similarity is set to be greater than or equal to 0.8. After removing redundant responses, the candidate responses {O1, O2} are output.

[0274] The fine-grained scoring module calls the discriminant model to score the candidate responses {O1, O2};

[0275] Scoring of O1's fidelity dimensions: Factual consistency: 5; Adjective matching: 5; Knowledge consistency: 5; Style consistency: 5;

[0276] Total score S of the loyalty dimension faithfulness =5;

[0277] Reasons for the rating: The response accurately matches the contextual facts (irradiance is power per unit area, and radiative flux is energy per unit time); the adjectives "intensity distribution" and "transmission rate" are appropriate and consistent with physical concepts; it is consistent with general physics knowledge and contains no errors; the style is formal and consistent, suitable for educational settings.

[0278] Scoring of O1's integrity dimension:

[0279] Key information coverage: 5; Detail sufficiency: 4; Logical coherence: 5;

[0280] Completeness dimension overall score S completeness =4.67;

[0281] Reasons for the rating: It covers the core definitions of irradiance (power per unit area) and radiant flux (energy per unit time); the basic details are complete, but units (such as watts per square meter) could be added to enhance the educational value; the sentences are logically clear, defining the terms before explaining them.

[0282] Rating for O1's simplicity:

[0283] Redundancy percentage: 5%; Sentence complexity: 4%; Information density: 5%.

[0284] Simplicity dimension overall score S conciseness =4.67;

[0285] Reasons for the score: No redundancy, directly focusing on the definition; simple sentence structure (subject-verb-object structure), easy to understand; high density, each sentence conveys the core meaning.

[0286] Rating for O2's fidelity dimension:

[0287] Factual consistency: 4; Adjective matching: 5; Knowledge consistency: 5; Style consistency: 5;

[0288] Total score S of the loyalty dimension faithfulness =4.75;

[0289] Reasons for scoring: The phrase "on a given area" in the irradiance section is inaccurate (the context is "per unit area"), while the radiative flux section is correct; the adjectives "light intensity" and "energy flow rate" are appropriate; the knowledge is correct and there are no conflicts; the style is consistent and formal.

[0290] Scoring of O2's integrity dimension:

[0291] Key information coverage: 5; Detail sufficiency: 3; Logical coherence: 5;

[0292] Completeness dimension overall score S completeness =4.33;

[0293] Reasons for the score: It covers the core definitions (radiated power and energy); it lacks unit details (units are mentioned in the context), and the description of "a certain region" is vague; the logic is smooth and the structure is reasonable.

[0294] Rating for O2's simplicity:

[0295] Redundancy percentage: 4%; Sentence complexity: 5%; Information density: 4%.

[0296] Simplicity dimension overall score S conciseness =4.33;

[0297] Reasons for the rating: Slightly redundant (e.g., "reflects the rate of energy flow" could be simplified); clear sentence structure without complex structures; moderate density, with some expressions being slightly lengthy.

[0298] The dynamic optimization module generates preference pairs: fidelity: O1>O2; completeness: O1>O2; simplicity: O1>O2;

[0299] The dynamic optimization module generates a loyalty dimension preference dataset D. faithfulness Completeness Dimension Preference Dataset D completeness And the simplicity dimension preference dataset D conciseness ;

[0300] The dynamic optimization module calculates the loyalty preference loss:

[0301]

[0302] Integrity preference loss:

[0303]

[0304] Simplicity preference loss:

[0305]

[0306] Where, π ref For unoptimized GPT-4, π θ GPT-4 to be optimized;

[0307] Total loss: L total =α1L faithfulness +α2L completeness +α3L conciseness

[0308] Where α1 = 0.3, α2 = 0.6, α3 = 0.1, we can obtain L. total =0.297;

[0309] Set the temperature coefficient β = 0.1, the learning rate η = 1e-5, and the batch size B = 32;

[0310] The AdamW optimizer is used to iteratively update the GPT-4 parameters to be optimized until the validation set loss converges.

[0311] The response results were scored based on the fidelity, completeness, and conciseness dimensions using a discriminant model, and coherence was assessed manually. The scores are shown in Table 1.

[0312] Comparative Example 1

[0313] Replies are generated using reinforcement learning from human feedback (RLHF);

[0314] The contextual text described in Comparative Example 1 is: a medical literature article, which includes... liver cancer treatment methods include surgical resection, radiofrequency ablation, and targeted drug therapy...;

[0315] The question is: What treatment methods are mentioned in this literature?

[0316] The response results were scored based on the fidelity, completeness, and conciseness dimensions using a discriminant model, and coherence was assessed manually. The scores are shown in Table 1.

[0317] Comparative Example 2

[0318] Responses are generated using traditional DPO methods;

[0319] The contextual text described in Comparative Example 2 is: a medical literature article, which includes... liver cancer treatment methods include surgical resection, radiofrequency ablation, and targeted drug therapy...;

[0320] The question is: What treatment methods are mentioned in this literature?

[0321] The response results were scored based on the fidelity, completeness, and conciseness dimensions using a discriminant model, and coherence was assessed manually. The scores are shown in Table 1.

[0322] Comparative Example 3 uses traditional DPO to generate responses;

[0323] The contextual text in Comparative Example 3 is as follows: ...Xiaomi headphones are available with a 15% discount, and Huawei headphones are eligible for a 15% discount under the national subsidy...

[0324] Question: What is the maximum discount I can get when buying Bluetooth headphones?

[0325] The response results were scored based on the fidelity, completeness, and conciseness dimensions using a discriminant model, and coherence was assessed manually. The scores are shown in Table 1.

[0326] Comparative Example 4 uses traditional DPO to generate responses;

[0327] The contextual text in Comparative Example 4 is as follows: ...Irradiance refers to the power of light radiation received per unit area, usually expressed in watts per square meter; radiant flux refers to the radiant energy passing through a cross-section per unit time...

[0328] Question: What do irradiance and radiation flux reflect respectively?

[0329] The response results were scored based on the fidelity, completeness, and conciseness dimensions using a discriminant model, and coherence was assessed manually. The scores are shown in Table 1.

[0330] Table 1

[0331]

[0332] As can be observed from Table 1, compared with Comparative Examples 1-4, Examples 1-3 generally showed better performance than Comparative Examples 1-4. Specifically, the fidelity dimension score of Example 1 was improved by 43.1% compared with the traditional DPO (Comparative Example 2); while in the e-commerce scenario (Example 2), the response simplicity of the optimized model was improved by 260% (compared with Comparative Example 3), and the average response time was shortened to 2 seconds. In the education mode (Example 3), the integrity dimension score was improved by 219% compared with Comparative Example 4.

[0333] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for optimizing large-scale question-answering models based on multi-scale fine-grained feedback, characterized in that, include, Multi-model collaborative generation of candidate responses involves inputting the context text and question text into N large language models to output candidate responses. Multi-dimensional fine-grained scoring: Based on multi-dimensional evaluation criteria, a pre-trained discriminative model scores the candidate responses and outputs corresponding scoring reasons. The dimensions include fidelity, completeness, and simplicity. Dynamic preference optimization receives the dimensional weight allocation, generates a preference pair dataset based on the scores output by the multi-dimensional evaluation criteria, outputs the optimization parameters of the large language model using a weighted multi-task loss function, and updates the large language model using a direct preference optimization algorithm based on the optimization parameters. Where N is greater than or equal to 3.

2. The method for large-scale question answering based on multi-scale fine-grained feedback optimization according to claim 1, characterized in that, The context text and question text of the N large language models are preprocessed. The preprocessing methods include sentence segmentation, noise character removal, and standardized encoding.

3. The method for large-scale question answering based on multi-scale fine-grained feedback optimization according to claim 2, characterized in that, The process of outputting responses from the N large language models includes... Set the temperature parameters, Top-p sampling values, and repetition penalties for the large language models. Input the preprocessed context text and the question text into N large language models simultaneously to get N responses. Deduplication of responses is performed based on semantic consistency to obtain candidate responses.

4. The method for optimizing large-scale question answering models based on multi-scale fine-grained feedback according to claim 3, characterized in that, The process of deduplicating responses based on semantic consistency includes: An algorithm is used to calculate text similarity, and duplicate responses are removed when the similarity is greater than the preset similarity.

5. The method for large-scale question answering based on multi-scale fine-grained feedback optimization according to claim 1, characterized in that, The scoring criteria for the fidelity dimension include the factual consistency sub-criteria, the adjective matching sub-criteria, the knowledge consistency sub-criteria, and the style consistency sub-criteria; The integrity dimension includes the key information coverage sub-standard, the detail sufficiency sub-standard, and the logical coherence sub-standard. The simplicity dimension includes sub-standards such as the proportion of redundant information, sentence complexity, and information density.

6. The method for large-scale question answering based on multi-scale fine-grained feedback optimization according to claim 1, characterized in that, The scoring formula for the multi-dimensional fine-grained scoring is as follows: Overall score for the loyalty dimension SF1 is the score for the factual consistency subcriterion, SF2 is the score for the adjective matching subcriterion, SF3 is the score for the knowledge consistency subcriterion, and SF4 is the score for the style consistency subcriterion. Overall score for simplicity dimension SP1 is the score for the key information coverage sub-criteria, SP2 is the score for the detail sufficiency sub-criteria, and SP3 is the score for the logical coherence sub-criteria. Total score for completeness dimension SC1 is the score for the redundancy information ratio, SC2 is the score for the sentence complexity, and SC3 is the score for the information density.

7. The method for large-scale question answering based on multi-scale fine-grained feedback optimization according to claim 1, characterized in that, The process of pre-training a discriminative model includes, Supervised training is performed on a question-and-answer dataset with fine-grained annotations, where the fine-grained annotations include the scores for the main dimension and sub-criteria of the corresponding response and the corresponding scoring reasons. The parameters of the discriminant model are jointly optimized by using the cross-entropy loss function and the consistency regularization term.

8. The method for large-scale question answering based on multi-scale fine-grained feedback optimization according to claim 1, characterized in that, The dynamic preference optimization process includes: The receiving dimension weights are assigned, where α1 is the fidelity dimension weight, α2 is the integrity dimension weight, and α3 is the simplicity dimension weight, and α1+α2+α3=1; The scores output by the multi-dimensional evaluation criteria are generated based on size rules to produce preference pairs for each dimension, and a preference dataset for each dimension is constructed. The corresponding score difference is calculated based on the dimensional weights, the preference pairs in the preference dataset, and the scores output by the multi-dimensional evaluation criteria for each candidate response; A weighted multi-task loss function is used to output the preference loss for each dimension of the large language model. The total loss is calculated based on the dimension weights and the preference losses for each dimension. The large language model is then updated using a direct preference optimization algorithm based on the total loss. The formula for calculating the total loss is as follows: L total =α1L faithfulness +α2L completeness +α3L conciseness , Where L is the preference loss of the direct preference optimization algorithm for the corresponding dimension.

9. The method for large-scale question answering based on multi-scale fine-grained feedback optimization according to claim 8, characterized in that, The formula for calculating the preference loss of the direct preference optimization algorithm is as follows: Where β is the temperature parameter, π θ For the target model, π ref As the reference model, D represents the preference pair dataset, and L... DPO Let x be the question text and y be the preference loss. w For the preference response, y l This is a non-preference response.

10. A large-model question-answering system based on multi-scale fine-grained feedback optimization using the method of any one of claims 1-9, characterized in that, include, The user interaction module is used to receive question text, dimension weight allocation, and output the optimized response quality comparison results. The data acquisition module, which is connected to the user interaction module, is used to receive context text and question text; The preprocessing module, which is connected to the acquisition module, is used to segment the received context text and question text into sentences, remove noise characters, and standardize the encoding. The multi-model generation module, which is connected to the preprocessing module, contains several large language models and is used to call the large language models to generate several candidate responses based on the preprocessed context text and question text. The fine-grained scoring module, which is connected to the multi-model generation module, is used to call the pre-trained discriminant model to score candidate responses based on multi-dimensional evaluation criteria and output the corresponding scoring reasons. The dynamic optimization module, which is connected to the fine-grained scoring module and the user interaction module, is used to generate a preference dataset based on the scores, generate the score difference of each candidate response based on the dimension weights, output the optimization parameters of the large language model based on the weighted multi-task loss function, and update the large language model based on the optimization parameters using the direct preference optimization algorithm.

Images