Large language model reasoning enhancement method and device based on multi-source retrieval and summary module

Abstract

The invention relates to a large language model reasoning enhancement method and device based on a multi-source retrieval and summary module, and the method comprises the steps: taking Qwen2.5-32B as a large language model, and carrying out the fine adjustment of the large language model through a preset retrieval data set; multiple rewards are introduced through reinforcement learning to improve the self-cognition ability of the large language model; constructing a vector database and a real-time network retrieval system based on the FAISS, and performing multi-source retrieval on the text data through the vector database and the real-time network retrieval system; text information of the text data is summarized through a preset small language model, and the text information is output to the large language model. According to the method, the self-cognition ability of the big language model is improved by adopting reinforcement learning (GPRO), and the multi-source retrieval module is constructed, so that the data with similar semantics can be quickly queried to assist the big language model in deeply understanding the semantic connotation of the data, and the accuracy and rationality of retrieval are improved.

Description

Technical Field

[0001] The present application relates to the technical field of large language models, and in particular to a large language model reasoning enhancement method and device based on a multi-source retrieval and summarization module. Background Art

[0002] In the field of artificial intelligence, reasoning enhancement methods based on large language models (LLMs) have become one of the core technologies for solving complex problems. However, traditional methods still have some shortcomings in data processing and multi-source information integration, especially in the following areas where improvement is urgently needed:

[0003] For example, existing systems often rely on a single data source or static database (such as a local knowledge base), which limits the scope of searches and makes it difficult to cover dynamically updated online information. Traditional search methods lack compatibility with multi-source heterogeneous data (such as text, tables, and image-derived text) and lack efficient vectorized indexing mechanisms, resulting in high retrieval response latency and low recall rates. For example, keyword-matching-based search techniques cannot effectively capture semantic relevance and are prone to missing critical contextual information.

[0004] Real-time online retrieval often introduces a large amount of irrelevant text (such as advertisements and duplicate content). Traditional filtering algorithms rely on rule engines or simple classification models, making it difficult to achieve context-aware noise elimination. Directly inputting redundant information into large language models will distract attention and significantly reduce the accuracy and coherence of the generated content. Summary of the Invention

[0005] In order to improve tasks that mainly rely on internal knowledge, especially those involving knowledge-intensive problems, private information in local databases, and time-sensitive problems, this reliance on internal knowledge can easily lead to inaccurate results and hallucinations, affecting the output accuracy of large language models. This application provides a large language model reasoning enhancement method and device based on a multi-source retrieval and summarization module.

[0006] In the first aspect, the present application provides a large language model reasoning enhancement method based on a multi-source retrieval and summarization module, which adopts the following technical solutions:

[0007] The Qwen2.5-32B distilled by DeepSeekR1 is used as a large language model, and the large language model is fine-tuned using a preset retrieval dataset;

[0008] By introducing format rewards, correct rewards, and rejection rewards through reinforcement learning, the self-awareness ability of the large language model is improved;

[0009] Building a vector database and a real-time network retrieval system based on FAISS, and multi-source retrieval of text data through the vector database and the real-time network retrieval system;

[0010] The text information of the text data is summarized by a preset small language model, and the text information is output to the large language model.

[0011] Preferably, the Qwen2.5-32B distilled by DeepSeekR1 is used as a large language model, and the large language model is fine-tuned using a preset retrieval dataset, including:

[0012] Load the Qwen2.5-32B model according to the preset structure definition and weights, use Qwen2.5-32B as the large language model, and ensure its compatibility with the DeepSeekR1 distilled version;

[0013] Fine-tuning the large language model based on the retrieval dataset;

[0014] Performing distillation-enhanced training to optimize the large language model using task data;

[0015] The retrieval and reasoning capabilities of the large language model are optimized based on a contrastive training mechanism.

[0016] Preferably, fine-tuning the large language model according to the retrieval dataset includes:

[0017] The parameters are efficiently fine-tuned through the LoRA low-rank adaptation technology. The LoRA low-rank adaptation technology transforms the original weight update method into updating only an approximate transformation. The formula is as follows:

[0018] △W≈AB;

[0019] Where: A∈R d×r ,B∈R r×k , r<<min(d,k), A and B are learnable low-rank matrices, the original weight W remains unchanged, R represents a set of real numbers, r, d, k represent the number of rows and columns of the matrix;

[0020] The cross-entropy loss function is used to measure the difference between the probability distribution of the "next token" predicted by the large language model and the true label. The cross-entropy loss function formula is as follows:

[0021]

[0022] Where: x i is the context input token, y i is the target token, θ represents the model parameters, P represents the probability distribution predicted by the model, and L CE is the cross entropy loss.

[0023] Preferably, performing distillation enhancement training to optimize the large language model using task data includes:

[0024] Perform the distillation enhancement training, the distillation loss function L distill Expressed as:

[0025] L distill =α·L CE (y student ,y teacher )+(1―α)·MSE(f student (x),f teacher (x));

[0026] Among them, L CE is the cross entropy loss, y student and y teacher are the prediction results of the student model and the teacher model respectively, MSE is the mean square error loss, f student (x) and f teacher (x) are the feature outputs of the student model and the teacher model for the input x, and α is the balance parameter;

[0027] At the same time, a weight regularization term is added to the total loss function. The total loss function is the sum of the distillation loss function and the weight regularization term. The formula of the weight regularization term is as follows:

[0028]

[0029] Among them, θ s represents the current parameters of the student model; θ t represents the original parameters of the teacher model; λ is the weight coefficient of the regularization term.

[0030] Preferably, the optimizing the retrieval and reasoning capabilities of the large language model according to the comparative training mechanism includes:

[0031] The contrast training mechanism includes first-stage training and second-stage training;

[0032] The first stage of training learns accurate text representation by comparing the original input;

[0033] The second stage of training is to perform robustness training through the rewritten version;

[0034] Alternate between the first and second phases of training, and continue iterative optimization until the loss converges or the evaluation indicator steadily improves to the preset value.

[0035] The contrast loss L of the first stage training contrast1 Expressed as:

[0036]

[0037] Among them, q is the representation vector of query, d + The document representation of the positive sample is, represents the jth negative sample, sim(.,.) is the similarity function, and τ is the temperature hyperparameter;

[0038] The contrast loss with enhancement in the second stage training is expressed as:

[0039]

[0040] where q aug , Represent the enhanced representations of query and positive samples respectively.

[0041] Preferably, it is characterized in that the introduction of format rewards, correct rewards and rejection rewards through reinforcement learning to improve the cognitive ability of the large language model includes:

[0042] Construct a dataset containing various question types, including questions with clear answers, ambiguous questions, or questions beyond the scope of knowledge, and annotate each sample in the dataset with a reference answer or a rejection template;

[0043] defining a reward function, the reward function including the format reward, the correct reward, and the refusal reward;

[0044] Integrating the reward function into GRPO and training the large language model through GRPO;

[0045] The reward function is as follows:

[0046]

[0047] Among them, R represents the total reward, C represents whether the answer is correct or refused (1 is correct, 0 is wrong, and 2 is refused), and F represents whether the format is completely correct or partially correct (1 is completely correct, 0 is wrong, and 2 is not completely correct).

[0048] Preferably, integrating the reward function into GRPO and training the large language model through GRPO includes:

[0049] The GRPO strategy is optimized by maximizing the objective function:

[0050]

[0051] in, represents the objective function of GRPO, q~P(Q) represents the query sampled from the problem distribution;i ~π θold represents samples generated based on the old strategy; o i,t is the tth token in the ith sample; is the advantage function, which measures the advantage of the current token compared to the average baseline; ε is the clipping threshold, which controls the range of strategy changes; KL[π θ ‖π ref ] is the KL divergence between the current strategy and the reference strategy, which serves as a regular term; β is the KL penalty coefficient, which is used to control the degree of deviation between the model and the reference strategy; r i,t is the sampling strategy ratio;

[0052]

[0053] Among them, π θ (o i,t │q,o i,<t ) represents the probability of generating the tth token by the current strategy, π θold (o i,t │q,o i,<t ) represents the probability of generating the tth token using the old strategy;

[0054] GRPO estimates KL divergence based on unbiased estimation, the formula is as follows:

[0055]

[0056] Among them, π ref (o i,t │q,o i,<t ) represents the probability of generating the tth token using the reference strategy.

[0057] Preferably, it is characterized in that the vector database and real-time network retrieval system based on FAISS is constructed, including:

[0058] Cleaning the local data, converting the local data into vector representation using an embedding model, and storing the vector representation in the vector database;

[0059] Configure the API interface, obtain real-time web page data, and generate a real-time network retrieval system.

[0060] Preferably, before summarizing the text information of the text data by using a preset small language model and outputting the text information to the large language model, the method further includes:

[0061] Acquire required text data through multi-source retrieval of the vector database and the real-time network retrieval system, and generate a sample data set based on the text data;

[0062] Based on the manual annotation of the sample data set, the search content and manual summary are analyzed and cleaned, and the data is organized in an input-output format;

[0063] The lightweight pre-trained model Qwen2.5-7B is selected as the small language model;

[0064] Dividing the sample data set into a training set, a validation set, and a test set according to a preset ratio;

[0065] Inputting the training set and the test set into the small language model, fine-tuning the small language model using the LoRA low-rank adaptation technology, and optimizing the model output using a preset cross-entropy loss function to obtain a text summary;

[0066] The model performance is evaluated using the validation set and the text summary, and the model structure is adjusted or the sample data diversity is increased.

[0067] In a second aspect, the present application provides a large language model reasoning enhancement device based on a multi-source retrieval and summarization module, which adopts the following technical solutions, including:

[0068] A fine-tuning module, configured to use the Qwen2.5-32B distilled by DeepSeekR1 as a large language model and fine-tune the large language model using a preset retrieval dataset;

[0069] A reward module, which is used to improve the self-awareness ability of the large language model by introducing format rewards, correct rewards, and rejection rewards through reinforcement learning;

[0070] A retrieval module is used to construct a vector database and a real-time network retrieval system based on FAISS, and to perform multi-source retrieval of text data through the vector database and the real-time network retrieval system;

[0071] The summarizing module is used to summarize the text information of the text data through a preset small language model and output the text information to the large language model.

[0072] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the large language model reasoning enhancement method based on a multi-source retrieval and summarization module as described in any embodiment of the first aspect.

[0073] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the large language model reasoning enhancement method based on a multi-source retrieval and summarization module as described in any embodiment of the first aspect.

[0074] Beneficial effects:

[0075] The integration of the FAISS-based vector database and a real-time web search system enables cross-modal semantic retrieval of structured and unstructured text (such as web pages, papers, and text derived from charts), supporting millisecond-level responses for billions of data items. Through a dynamic update mechanism and multi-source data alignment algorithm, retrieval coverage has increased by over 60%, effectively avoiding the semantic bias caused by traditional keyword matching.

[0076] By using format rewards (ensuring the standardization of the output structure), correct rewards (a reward function based on fact verification) and refusal rewards (actively refusing to answer low-confidence questions), the model's error rate in open-domain question answering is reduced by 35%, while also reducing the phenomenon of "fictitious facts".

[0077] By pre-summarizing search results using a small language model, filtering redundant information and extracting core logical chains, the text length input to the large language model is reduced by an average of 70%, and inference speed is increased by 2.3 times. Furthermore, the distilled Qwen2.5-32B model reduces GPU memory usage by 45% while maintaining 97% of the original model's performance, making it more suitable for edge computing scenarios.

[0078] The real-time web retrieval system, combined with a semantic update strategy, can capture the latest knowledge (such as scientific and technological advances and emergencies), addressing the timeliness shortcomings of traditional static knowledge bases. In frequently updated data fields such as finance and healthcare, the model's timely answer accuracy has increased to 89.4%, a 41% improvement over the baseline method. Through the combined training of retrieval enhancement and reinforcement learning, the model performs exceptionally well in few-shot and zero-shot scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 This is a flow chart of a large language model reasoning enhancement method based on a multi-source retrieval and summarization module in an embodiment of the present application.

[0080] Figure 2 It is a flowchart of the large language model fine-tuning method in an embodiment of the present application.

[0081] Figure 3 This is an overall block diagram of a large language model reasoning enhancement based on a multi-source retrieval and summarization module in an embodiment of the present application.

[0082] Description of reference numerals: 210, fine-tuning module; 220, reward module; 230, retrieval module; 240, summary module. DETAILED DESCRIPTION

[0083] The following combination Figure 1-Figure 3 This application is described in further detail.

[0084] In the field of natural language processing, traditional retrieval-augmented generation (RAG) methods and current retrieval models have many limitations. Traditional RAG relies on the original question statement to construct retrieval queries. When faced with questions containing potential sub-goals, requiring decomposition and understanding, or with high task ambiguity, it is easy to cause retrieval information deviation or introduce irrelevant content interference, affecting the accuracy and stability of subsequent reasoning paths. Current retrieval models perform initial thinking based on pre-training data. When the user's question semantic space conflicts with the pre-training data, the starting point of thinking is prone to deviation. Even if strong relevant information is subsequently retrieved, the correct answer may not be generated due to the deviation of the initial thinking path.

[0085] This application proposes a series of innovative technical mechanisms to address these issues. The first is RAG-Driven Reasoning Alignment (RDRA), which adds an additional RAG background retrieval phase before triggering initial reasoning. Through this pre-process, the system can pre-perceive the specific context of user questions, effectively identify and resolve potential conflicts between user questions and the pre-trained data context, laying the foundation for accurate reasoning.

[0086] When handling complex reasoning tasks, multiple rounds of "think-retrieve" operations in language models can lead to problems such as duplicate candidate search content, the propagation of erroneous information, and the recurrence of low-confidence content, which impact the reliability and diversity of answers. To address this, we propose the Search-Think Iterative Enhancement (STIE) mechanism, which introduces a "memory-filtering-confidence control" framework to dynamically analyze and strategically regulate candidate search content in each round, significantly improving the quality of multi-round retrieval reasoning.

[0087] In terms of model training, a progressive training strategy is adopted. In the format cold start phase, a retrieval dataset containing 600K data is constructed based on ReSearch-Qwen and ZeroSearch, of which noise and high-quality data account for 50% each, for the format cold start of Qwen3. The initial phase is based on 100% high-quality data. As the number of searches increases, the retrieval difficulty and the proportion of noise data are gradually increased. The improved DAPO+ algorithm is used for post-reinforcement learning training.<short_answer> and<long_answer> Two answer modes are designed for general reasoning and summarization tasks respectively. The reward function includes format rewards, length rewards based on different modes, and answer accuracy rewards. Masked document token retrieval is used to avoid gradient interference.

[0088] Aiming at the problems of searching based entirely on local knowledge base and the high cost of network search, we propose NetworkLocalRouting, a cost-reward function based on reinforcement learning, by modeling<local_search> and<web_search> The reward value is used to realize the automatic selection of the retrieval method.

[0089] Reference Figure 1 , this application at least includes steps S10 to S40.

[0090] S10 uses Qwen2.5-32B distilled by DeepSeekR1 as the large language model and fine-tunes the large language model using the preset retrieval dataset.

[0091] S20, through reinforcement learning, introduces format rewards, correct rewards and rejection rewards to improve the self-awareness ability of large language models.

[0092] S30, builds a vector database and real-time network retrieval system based on FAISS, and uses the vector database and real-time network retrieval system to retrieve text data from multiple sources.

[0093] S40, summarizing the text information of the text data through a preset small language model, and outputting the text information to the large language model.

[0094] Specifically, by fine-tuning Qwen2.5-32B, an initial large language model is generated, avoiding the need to train a new model from scratch. This saves training time and resources and ensures a certain level of model quality from the outset. Furthermore, reinforcement learning (GPRO) enhances the large language model's self-awareness, strengthening its ability to "know what it doesn't know." By pre-setting multiple reward dimensions, the model is provided with clear behavioral guidelines and optimization goals, enabling it to develop towards the desired output under the guidance of different reward values. Furthermore, the system incorporates a multi-source retrieval module, including a vector database and a real-time network retrieval system, enabling rapid querying of semantically similar data. This helps the large language model gain a deeper understanding of the semantic content of the data, improving the accuracy and rationality of retrieval. When faced with knowledge-intensive problems, private information in local databases, and time-sensitive tasks, the system not only relies on local data but also expands the data base through multi-source retrieval. The system's ability to "know what it doesn't know" effectively reduces inaccurate results and hallucinations, thereby improving the output accuracy of the large language model. Furthermore, the system uses a small language model to summarize the retrieved and input text data and passes this summarized text information to the inference module of the large language model, significantly improving the large language model's speed in processing complex information and retrieval accuracy. Finally, all of these modules are integrated according to the inference process to obtain the required large language model.

[0095] In some embodiments, reference Figure 2 , step S10 specifically includes the following steps:

[0096] S101, load the Qwen2.5-32B model according to the preset structure definition and weights, use Qwen2.5-32B as a large language model, and ensure its compatibility with the DeepSeekR1 distilled version.

[0097] S102, fine-tuning the large language model based on the retrieval dataset.

[0098] S103: Perform distillation enhancement training to optimize the large language model using task data.

[0099] S104: Optimize the retrieval and reasoning capabilities of the large language model based on a comparative training mechanism.

[0100] Specifically, by loading Qwen2.5-32B and ensuring its compatibility with the DeepSeekR1 distilled version, it can inherit the general knowledge compression capabilities of the teacher model, avoid knowledge gaps and improve the model's adaptability to new tasks; based on retrieval dataset fine-tuning and distillation-enhanced training, the model can deeply integrate domain data features with the teacher model's reasoning logic, significantly reducing hallucinations and enhancing answer correctness in open domain question answering; through the comparative training mechanism, the retrieval reasoning ability is optimized, and the model's semantic matching and contextual association analysis capabilities for multi-source information are enhanced, ultimately improving the multi-step reasoning efficiency of complex problems and the robustness of long text understanding.

[0101] Furthermore, step S102 specifically includes the following steps: performing efficient parameter fine-tuning through LoRA low-rank adaptation technology. The LoRA low-rank adaptation technology converts the original weight update method into updating only an approximate transformation. The formula is as follows:

[0102] △W≈AB;

[0103] Where: A∈R d×r ,B∈R r×k , r<<min(d,k), A and B are learnable low-rank matrices, the original weight W remains unchanged, R represents a set of real numbers, r, d, k represent the number of rows and columns of the matrix;

[0104] The cross-entropy loss function is used to measure the difference between the probability distribution of the "next token" predicted by the large language model and the true label. The cross-entropy loss function formula is as follows:

[0105] Where: x i is the context input token, y i is the target token, θ represents the model parameters, P represents the probability distribution predicted by the model, and L CE is the cross entropy loss.

[0106] It should be understood that when fine-tuning large models, LoRA only trains a relatively small number of parameters. Compared to traditional fine-tuning methods, this not only significantly reduces the computational resources and memory usage required for training, but also speeds up training. Furthermore, by performing a low-rank decomposition of parameters, it helps the model better capture common features and patterns in the data when learning new tasks, thereby improving the model's generalization ability across different tasks and datasets.

[0107] Specifically, in LoRA fine-tuning, only the low-rank matrices A and B are updated, and the cross-entropy loss function guides the adjustment direction of these low-rank parameters, so that the model can learn the specific patterns of new tasks while retaining the original knowledge (freezing the original weights W). It cooperates with LoRA's low-rank parameter update mechanism to ensure that the model improves its adaptability to retrieval-enhanced reasoning tasks while being efficiently fine-tuned (only 0.1% to 1% of parameters are trained).

[0108] In some embodiments, step S103 specifically includes the following steps: performing distillation enhancement training, distillation loss function L distill Expressed as:

[0109] L distill =α·L CE (y student ,y teacher )+(1―α)·MSE(f student (x),f teacher (x));

[0110] Among them, L CE is the cross entropy loss, y student and y teacher are the prediction results of the student model and the teacher model respectively, MSE is the mean square error loss, f student (x) and f teacher (x) are the feature outputs of the student model and the teacher model for the input x, respectively, and α is the balance parameter. At the same time, a weight regularization term is added to the total loss function. The total loss function is the sum of the distillation loss function and the weight regularization term. The weight regularization term formula is as follows:

[0111]

[0112] Among them, θ s represents the current parameters of the student model; θ t represents the original parameters of the teacher model; λ is the weight coefficient of the regularization term.

[0113] Specifically, the system achieves knowledge transfer through the distillation loss function, which combines cross-entropy loss (which measures the difference between the student model's prediction and the teacher model's soft label) and mean squared error loss (which aligns the intermediate layer features of the student and teacher) to ensure that the model inherits the teacher model's reasoning ability and knowledge compression characteristics during fine-tuning, thereby improving the correctness of answers to open-domain tasks. Furthermore, a weight regularization term is introduced based on the distillation loss. L2 regularization is used to constrain the deviation of the student model parameters from the teacher model, reducing the destruction of important knowledge structures of the original model during fine-tuning and avoiding catastrophic forgetting. The synergy between the two balances knowledge transfer and stability: the distillation loss drives the model to learn new task features, while the regularization term maintains the conservatism of the parameter space. Ultimately, while improving reasoning accuracy, it ensures that the model retains general language understanding capabilities and adapts to multi-task requirements in complex scenarios.

[0114] In some embodiments, step S104 specifically includes the following steps: the comparative training mechanism includes a first-stage training and a second-stage training; the first-stage training learns accurate text representation by comparative training with the original input; the second-stage training performs robustness training with the rewritten version; the first-stage training and the second-stage training are performed alternately, and iterative optimization is continuously performed until the loss converges or the evaluation index is stably improved to a preset value;

[0115] The contrast loss L in the first stage of training contrast1 Expressed as:

[0116]

[0117] Among them, q is the representation vector of query, d + The document representation of the positive sample is, represents the jth negative sample, sim(.,.) is the similarity function, and τ is the temperature hyperparameter;

[0118] The contrast loss with enhancement in the second stage training is expressed as:

[0119]

[0120] where q aug , Represent the enhanced representations of query and positive samples respectively.

[0121] Specifically, a two-stage contrastive learning mechanism (original input and perturbation input) is used to optimize the model's text representation ability and enhance its adaptability to noise and semantic changes. In the first stage, the original input is used for contrast training to learn accurate text representation. In the second stage, the rewritten version (random insertion, cropping, and sequential perturbation) is used for robustness training. The above two-stage models are trained alternately, and iterative optimization is continuously performed until the loss converges or the evaluation indicators are stably improved.

[0122] In some embodiments, step S20 specifically includes the following steps: constructing a dataset containing multiple question types, including questions with clear answers, ambiguous questions, or questions beyond the scope of knowledge, and annotating each sample in the dataset with a reference answer or a rejection template; defining a reward function, including a format reward, a correct reward, and a rejection reward; integrating the reward function into GRPO, and training a large language model through GRPO;

[0123] The reward function is as follows:

[0124]

[0125] Among them, R represents the total reward, C represents whether the answer is correct or refused (1 is correct, 0 is wrong, and 2 is refused), and F represents whether the format is completely correct or partially correct (1 is completely correct, 0 is wrong, and 2 is not completely correct).

[0126] Specifically, by constructing a dataset containing various question types and annotating reference answers or refusal templates, the model can be trained to recognize the response boundaries in different scenarios, improve the ability to "know what you don't know", and reduce the error generation of vague or out-of-scope questions; defining reward functions (format, correctness, refusal reward) to provide quantitative goals for GRPO training, format rewards to ensure that answers comply with specifications (such as structured output), correctness rewards to drive answer accuracy, and refusal rewards to guide the model to actively avoid risks when uncertain; through GRPO, the reward function is integrated into policy optimization, and the model dynamically adjusts the generation strategy based on the relative reward values ​​of samples within the group. For example, the highest reward (1.0 points) is obtained when the answer is completely correct and the format is compliant, the second highest reward (0.7 points) is obtained when the answer is correct but the format is incorrect, a medium reward (0.4 points) is obtained when the answer is refused but the format is compliant, and there is no reward for incorrect answers, thereby balancing quality and safety in reinforcement learning, and ultimately achieving highly accurate and highly controllable reasoning capabilities in open domain scenarios.

[0127] Furthermore, the GRPO strategy is optimized by maximizing the objective function:

[0128]

[0129] in, represents the objective function of GRPO, q~P(Q) represents the query sampled from the problem distribution; i ~π θold represents samples generated based on the old strategy; o i,t is the tth token in the ith sample; is the advantage function, which measures the advantage of the current token compared to the average baseline; ε is the clipping threshold, which controls the range of strategy changes; KL[π θ ‖π ref ] is the KL divergence between the current strategy and the reference strategy, which serves as a regular term; β is the KL penalty coefficient, which is used to control the degree of deviation between the model and the reference strategy; r i,t is the sampling strategy ratio;

[0130]

[0131] Among them, π θ (o i,t │q,o i,<t ) represents the probability of generating the tth token by the current strategy, π θold (o i,t │q,o i,<t ) represents the probability of generating the tth token using the old strategy;

[0132] GRPO estimates KL divergence based on unbiased estimation, the formula is as follows:

[0133]

[0134] Among them, π ref (o i,t │q,o i,<t ) represents the probability of generating the tth token using the reference strategy.

[0135] Specifically, the GRPO objective function maximizes the product of the reward-driven strategy ratio and the advantage function, and combines the clipping threshold to limit the strategy update amplitude to ensure training stability and avoid excessive deviation; the KL divergence regularization term is introduced to constrain the difference between the current strategy and the reference strategy, preventing the model from destroying the original knowledge structure or generating uncontrollable content during the optimization process; the KL divergence calculation based on unbiased estimation ensures the accuracy of strategy updates, so that the model can absorb new task reward signals in the reinforcement learning stage while maintaining the logical coherence and security of the generated results, and ultimately achieve multi-objective balanced optimization of answer quality, format compliance and refusal ability in open domain question answering.

[0136] In some embodiments, step S30 specifically includes the following steps: cleaning local data, converting local data into vector representation using an embedding model, and storing it in a vector database; configuring an API interface, obtaining real-time web page data, and generating a real-time network retrieval system.

[0137] In practical terms, we prepare a local database: collect and organize structured data (documents, tables, knowledge bases), and store it in a local database. We clean local and web data (removing noise and formatting), using an embedding model (BERT) to convert text into vector representations, and store them in a vector database (Faiss). To access web data, we configure an API (Bing API) to obtain real-time web data.

[0138] Multi-source retrieval module design: Receive user-entered queries and preprocess the query text (word segmentation, noise removal, and semantic expansion). Local database retrieval: Use vector similarity search (such as cosine similarity) or keyword matching to retrieve relevant documents from the local vector database. Real-time web page retrieval: Use search engine APIs to retrieve web page content based on the query.

[0139] In some embodiments, considering the problem of training a small language model, the corresponding processing steps are as follows: obtain the required text data through multi-source retrieval of a vector database and an implementation of a network retrieval system, and generate a sample data set based on the text data; based on manual annotation of the sample data set, analyze and clean the retrieval content and manual summary, and organize the data in an input-output format; select the lightweight pre-trained model Qwen2.5-7B as a small language model; divide the sample data set into a training set, a validation set, and a test set according to a preset ratio; input the training set and the test set into the small language model, fine-tune the small language model through the LoRA low-rank adaptation technology, and optimize the model output through a preset cross-entropy loss function to obtain a text summary; evaluate the model performance through the validation set and the text summary, and adjust the model structure or increase the diversity of the sample data.

[0140] Specifically, vector databases and real-time network text data are obtained through a multi-source retrieval system, high-quality sample data sets are constructed, and manual annotations are performed to generate concise summaries to ensure that core information is retained; data is cleaned and segmented, and then organized into input-output pairs to improve the model's ability to parse structured information; the lightweight model Qwen2.5-7B is selected and combined with LoRA technology for efficient parameter fine-tuning, and only low-rank matrix parameters are updated to reduce computational overhead; the training set, validation set, and test set are divided, and the generation quality is optimized through cross-entropy loss; based on the validation set, the accuracy and information completeness of the summary results are evaluated, and the model structure is dynamically adjusted or data enhancement strategies are introduced, ultimately improving the semantic compression ability of small models for long texts, providing contextual support for the main model, and increasing the reasoning efficiency and answer correctness of open domain question answering.

[0141] The implementation principle of a large language model reasoning enhancement method based on a multi-source retrieval and summarization module in the embodiment of the present application is as follows: by fine-tuning Qwen2.5-32B, an initial large language model is obtained, avoiding the need to train a new model from scratch, thereby saving training time and resources and ensuring that the model has a certain generation quality in the initial stage. In addition, the self-awareness ability of the large language model is improved through reinforcement learning (GPRO), which can enhance the judgment of "knowing what you don't know". By pre-setting multiple reward dimensions, the model is provided with clear behavioral guidelines and optimization goals, so that it can develop towards the expected output results under the guidance of different reward values. At the same time, the system constructs a multi-source retrieval module, including a vector database and a real-time network retrieval system, which realizes the function of quickly querying semantically similar data, helping the large language model to deeply understand the semantic connotation of the data, and improving the accuracy and rationality of the retrieval. When the system faces knowledge-intensive problems, private information in the local database, and time-sensitive tasks, it not only relies on local data, but also expands the data range through multi-source retrieval. The system has the ability to "know what it doesn't know", which can effectively reduce inaccurate results and hallucination problems, thereby improving the output accuracy of the large language model. In addition, the system uses a small language model to summarize the retrieved and input text data, and passes the summarized text information to the reasoning module of the large language model, significantly improving the speed of the large language model in processing complex information and the accuracy of retrieval.

[0142] Figure 1 FIG. 1 is a flow chart of a large language model reasoning enhancement method based on a multi-source retrieval and summarization module in one embodiment. It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows; unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps may be executed in other orders; and Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0143] Based on the same technical concept, Figure 3 The embodiment of the present application further provides a large language model reasoning enhancement device based on a multi-source retrieval and summarization module, which adopts the following technical solution. The device includes:

[0144] A fine-tuning module 210 is configured to use the Qwen2.5-32B distilled by DeepSeekR1 as a large language model and fine-tune the large language model using a preset retrieval dataset;

[0145] Reward module 220, used to improve the self-awareness of the large language model by introducing format rewards, correct rewards, and rejection rewards through reinforcement learning;

[0146] Retrieval module 230, for building a vector database and a real-time network retrieval system based on FAISS, and multi-source retrieval of text data through the vector database and the real-time network retrieval system;

[0147] The summarizing module 240 is configured to summarize the text information of the text data using a preset small language model and output the text information to the large language model.

[0148] In some embodiments, the fine-tuning module 210 is specifically used to load the Qwen2.5-32B model according to the preset structure definition and weights, use Qwen2.5-32B as a large language model, and ensure its compatibility with the DeepSeekR1 distilled version;

[0149] Fine-tune the large language model based on the retrieval dataset;

[0150] Perform distillation-enhanced training to optimize large language models using task data;

[0151] Optimize the retrieval and reasoning capabilities of large language models based on a contrastive training mechanism.

[0152] In some embodiments, the fine-tuning module 210 is further configured to perform efficient parameter fine-tuning through LoRA low-rank adaptation technology. The LoRA low-rank adaptation technology converts the original weight update method into updating only an approximate transformation, as shown in the following formula:

[0153] △W≈AB;

[0154] Where: A∈R d×r ,B∈R r×k , r<<min(d,k), A and B are learnable low-rank matrices, the original weight W remains unchanged, R represents a set of real numbers, r, d, k represent the number of rows and columns of the matrix;

[0155] The cross-entropy loss function is used to measure the difference between the probability distribution of the "next token" predicted by the large language model and the true label. The cross-entropy loss function formula is as follows:

[0156]

[0157] Where: x i is the context input token, yi is the target token, θ represents the model parameters, P represents the probability distribution predicted by the model, and L CE is the cross entropy loss.

[0158] In some embodiments, the fine-tuning module 210 is further configured to perform distillation enhancement training, where the distillation loss function L distill Expressed as:

[0159] L distill =α·L CE (y student ,y teacher )+(1―α)·MSE(f student (x),f teacher (x));

[0160] Among them, L CE is the cross entropy loss, y student and y teacher are the prediction results of the student model and the teacher model respectively, MSE is the mean square error loss, f student (x) and f teacher (x) are the feature outputs of the student model and the teacher model for the input x, and α is the balance parameter;

[0161] At the same time, a weight regularization term is added to the total loss function. The total loss function is the sum of the distillation loss function and the weight regularization term. The weight regularization term formula is as follows:

[0162]

[0163] Among them, θ s represents the current parameters of the student model; θ t represents the original parameters of the teacher model; λ is the weight coefficient of the regularization term.

[0164] In some embodiments, the fine-tuning module 210 is further configured to compare training mechanisms including first-stage training and second-stage training;

[0165] In the first stage of training, accurate text representation is learned through contrast training with the original input;

[0166] The second stage of training is to conduct robustness training through the rewritten version;

[0167] Alternate between the first and second phases of training, and continue iterative optimization until the loss converges or the evaluation indicator steadily improves to the preset value.

[0168] The contrast loss L in the first stage of training contrast1 Expressed as:

[0169]

[0170] Among them, q is the representation vector of query, d + The document representation of the positive sample is, represents the jth negative sample, sim(.,.) is the similarity function, and τ is the temperature hyperparameter;

[0171] The contrast loss with enhancement in the second stage training is expressed as:

[0172]

[0173] where q aug , Represent the enhanced representations of query and positive samples respectively.

[0174] In some embodiments, the reward module 220 is specifically used to construct a data set containing multiple question types, including questions with clear answers, ambiguous questions, or questions beyond the scope of knowledge, and to mark each sample in the data set with a reference answer or a rejection template;

[0175] Define the reward function, which includes format reward, correct reward, and rejection reward;

[0176] Integrate the reward function into GRPO and use GRPO to train large language models;

[0177] The reward function is as follows:

[0178]

[0179] Among them, R represents the total reward, C represents whether the answer is correct or refused (1 is correct, 0 is wrong, and 2 is refused), and F represents whether the format is completely correct or partially correct (1 is completely correct, 0 is wrong, and 2 is not completely correct).

[0180] In some embodiments, the reward module 220 is further configured to optimize the GRPO policy by maximizing an objective function:

[0181]

[0182] in, represents the objective function of GRPO, q~P(Q) represents the query sampled from the problem distribution; i ~π θold represents samples generated based on the old strategy; o i,t is the tth token in the ith sample; is the advantage function, which measures the advantage of the current token compared to the average baseline; ε is the clipping threshold, which controls the range of strategy changes; KL[π θ ‖πref ] is the KL divergence between the current strategy and the reference strategy, which serves as a regular term; β is the KL penalty coefficient, which is used to control the degree of deviation between the model and the reference strategy; r i,t is the sampling strategy ratio;

[0183]

[0184] Among them, π θ (o i,t │q,o i,<t ) represents the probability of generating the tth token by the current strategy, π θold (o i,t │q,o i,<t ) represents the probability of generating the tth token using the old strategy;

[0185] GRPO estimates KL divergence based on unbiased estimation, the formula is as follows:

[0186]

[0187] Among them, π ref (o i,t │q,o i,<t ) represents the probability of generating the tth token using the reference strategy.

[0188] In some embodiments, the retrieval module 230 is specifically used to clean the local data, convert the local data into a vector representation using an embedding model, and store it in a vector database;

[0189] Configure the API interface, obtain real-time web page data, and generate a real-time network retrieval system.

[0190] In some embodiments, the summarizing module 240 obtains the required text data through multi-source retrieval of a vector database and a real-time network retrieval system, and generates a sample data set based on the text data;

[0191] Based on manual annotation of sample data sets, the search content and manual summary are segmented and cleaned, and the data is organized in an input-output format;

[0192] The lightweight pre-trained model Qwen2.5-7B is selected as the small language model;

[0193] The sample data set is divided into training set, validation set and test set according to the preset ratio;

[0194] The training set and test set are input into the small language model, the small language model is fine-tuned using the LoRA low-rank adaptation technology, and the model output is optimized using the preset cross-entropy loss function to obtain a text summary;

[0195] Evaluate model performance through validation sets and text summaries, and adjust model structure or increase sample data diversity.

[0196] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0197] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0198] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0199] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0200] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided in this application may be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, etc., but are not limited to these.

[0201] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0202] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A large language model reasoning enhancement method based on a multi-source retrieval and summarization module, characterized in that: include: The Qwen2.5-32B distilled by DeepSeekR1 is used as a large language model, and the large language model is fine-tuned using a preset retrieval dataset; By introducing format rewards, correct rewards, and rejection rewards through reinforcement learning, the self-awareness ability of the large language model is improved; Building a vector database and a real-time network retrieval system based on FAISS, and multi-source retrieval of text data through the vector database and the real-time network retrieval system; The text information of the text data is summarized by a preset small language model, and the text information is output to the large language model.

2. The method according to claim 1, characterized in that The Qwen2.5-32B distilled by DeepSeekR1 is used as a large language model, and the large language model is fine-tuned using a preset retrieval dataset, including: Load the Qwen2.5-32B model according to the preset structure definition and weights, use Qwen2.5-32B as the large language model, and ensure its compatibility with the DeepSeekR1 distilled version; Fine-tuning the large language model based on the retrieval dataset; Performing distillation-enhanced training to optimize the large language model using task data; The retrieval and reasoning capabilities of the large language model are optimized based on a contrastive training mechanism.

3. The method according to claim 2, characterized in that The fine-tuning of the large language model according to the retrieval dataset includes: Efficient parameter fine-tuning through LoRA low-rank adaptation technology, which transforms the original weight update method into updating only an approximate transformation; The cross entropy loss function is used to measure the difference between the probability distribution of the "next token" predicted by the large language model and the actual label.

4. The method according to claim 3, characterized in that The performing distillation enhancement training to optimize the large language model using task data includes: The distillation enhancement training is performed, and a weight regularization term is added to the total loss function, where the total loss function is the sum of the distillation loss function and the weight regularization term.

5. The method according to claim 4, characterized in that The optimization of the retrieval and reasoning capabilities of the large language model based on the comparative training mechanism includes: The contrast training mechanism includes first-stage training and second-stage training; The first stage of training learns accurate text representation by comparing the original input; The second stage of training is to perform robustness training through the rewritten version; The first stage training and the second stage training are performed alternately, and the iterative optimization is continued until the loss converges or the evaluation index is stably improved to the preset value.

6. The method according to claim 1, characterized in that The method of introducing format rewards, correct rewards, and rejection rewards through reinforcement learning to improve the cognitive ability of the large language model includes: Construct a dataset containing various question types, including questions with clear answers, ambiguous questions, or questions beyond the scope of knowledge, and annotate each sample in the dataset with a reference answer or a rejection template; defining a reward function, the reward function including the format reward, the correct reward, and the refusal reward; The reward function is integrated into GRPO, and the large language model is trained by GRPO.

7. The large language model reasoning enhancement method based on the multi-source retrieval and summarization module according to claim 1 is characterized in that: The vector database and real-time network retrieval system based on FAISS includes: Cleaning the local data, converting the local data into vector representation using an embedding model, and storing the vector representation in the vector database; Configure the API interface, obtain real-time web page data, and generate a real-time network retrieval system.

8. The large language model reasoning enhancement method based on the multi-source retrieval and summarization module according to claim 1 is characterized in that: Before summarizing the text information of the text data by using a preset small language model and outputting the text information to the large language model, the method further includes: Acquire required text data through multi-source retrieval of the vector database and the real-time network retrieval system, and generate a sample data set based on the text data; Based on the manual annotation of the sample data set, the search content and manual summary are analyzed and cleaned, and the data is organized in an input-output format; The lightweight pre-trained model Qwen2.5-7B is selected as the small language model; Dividing the sample data set into a training set, a validation set, and a test set according to a preset ratio; Inputting the training set and the test set into the small language model, fine-tuning the small language model using the LoRA low-rank adaptation technology, and optimizing the model output using a preset cross-entropy loss function to obtain a text summary; The model performance is evaluated using the validation set and the text summary, and the model structure is adjusted or the sample data diversity is increased.

9. A large language model reasoning enhancement device based on a multi-source retrieval and summarization module, characterized in that: The device is used to perform the method according to any one of claims 1 to 8, and the device comprises: A fine-tuning module, configured to use the Qwen2.5-32B distilled by DeepSeekR1 as a large language model and fine-tune the large language model using a preset retrieval dataset; A reward module, which is used to improve the self-awareness ability of the large language model by introducing format rewards, correct rewards, and rejection rewards through reinforcement learning; A retrieval module is used to construct a vector database and a real-time network retrieval system based on FAISS, and to perform multi-source retrieval of text data through the vector database and the real-time network retrieval system; The summarizing module is used to summarize the text information of the text data through a preset small language model and output the text information to the large language model.

Images