Method and device for improving long-range deep search reasoning capability of large language model

Abstract

The invention provides a method and a device for improving the long-range deep search reasoning capability of a large language model. According to the method for improving the long-range deep search reasoning ability of the large language model, the long-range search reasoning ability of the large language model in cross-domain complex problems is remarkably improved, and the large language model is improved through high-difficulty data automatic generation driven by the knowledge graph and a three-layer difficulty screening mechanism. And by combining a two-stage training strategy of supervised fine tuning and GRPO reinforcement learning, the multi-hop reasoning depth of the model and the collaboration of external tool calling are effectively enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence-large language model training, and in particular to a method and device for improving long-range deep search reasoning capability of a large language model. BACKGROUND

[0002] Large language models (LLMs) are important technical carriers in the field of artificial intelligence and are widely used in open-domain question answering, intelligent assistants, knowledge reasoning, and other scenarios.

[0003] With the continuous expansion of model size, LLMs exhibit significant reasoning capabilities in tasks such as mathematics, logic, and code generation. In related technologies, a full-process reasoning system from problem understanding to answer generation is constructed through the collaborative work of chain-of-thought (CoT) and reinforcement learning (RL), including key links such as single-step reasoning, multi-hop retrieval, and tool invocation. Specifically, existing systems such as OpenAI o1 and DeepSeek-R1 optimize answer correctness by introducing reinforcement learning mechanisms, improving the model's autonomous error correction and long-chain planning capabilities in complex tasks. However, in existing LLM training methods, shallow multi-hop datasets (such as HotpotQA) are directly used for training, which does not fully simulate the complex reasoning process of cross-domain and multi-round interaction in real scenarios. This may lead to problems such as short-sighted search and ineffective evidence fusion when the model faces fuzzification, high difficulty, and long-range retrieval tasks, or it may be difficult to stabilize convergence due to sparse training signals, thereby affecting its actual application effect in industries such as finance and medicine that require high knowledge timeliness and reasoning depth. SUMMARY

[0004] The present application aims to at least partially solve one of the technical problems in the related art.

[0005] To this end, the first object of the present application is to propose a method for improving the long-range deep search reasoning capability of a large language model.

[0006] The second object of the present application is to propose a device for improving the long-range deep search reasoning capability of a large language model.

[0007] To achieve the above purpose, the first aspect of the embodiment of the present application proposes a method for improving the long-range deep search reasoning capability of a large language model, comprising: performing semantic-guided random walk based on a knowledge graph to generate structured data containing multi-hop entity paths, wherein each node carries attribute information, and key entities or time information in the path are blurred by an information masking strategy; performing three-level difficulty screening on the generated structured data, first using a model without search capability to perform zero retrieval judgment to filter simple problems, second performing multi-round attempts by a model with search capability to screen medium difficulty problems, and finally filtering problems that are not dependent on external retrieval according to the average search call number; using a two-stage training strategy, in the first stage, generating high-quality samples containing "thinking-searching-observing" interaction tracks by a teacher model, and performing supervised fine-tuning on the samples to transfer the basic search strategy, in the second stage, using GRPO reinforcement learning algorithm in the real search environment, optimizing the multi-round search and reasoning process of the model based on the normalized advantage function, and giving positive reward only when the model completes normal interaction and outputs the correct answer.

[0008] In an embodiment of the present application, the semantic-guided random walk based on a knowledge graph to generate structured data containing multi-hop entity paths further comprises: using a large language model to evaluate the semantic relevance of candidate next-hop entities, and only retaining entities with strong semantic connection to the current node as path expansion nodes; by adjusting the path length k and the node out-degree threshold of the random walk, the complexity of the generated problem is controlled, wherein the default value of k is 5, and the default value of the out-degree threshold is ≥ 3.

[0009] In an embodiment of the present application, the three-level difficulty screening of the generated structured data further comprises: using a GPT-4o model without browsing capability to perform zero retrieval judgment on the problem, if the model directly gives the correct answer without external search, the problem is excluded; using a GLM-4-Z1-Rumination-32B model with browsing function to perform 8 times of search-based reasoning attempts, and only retaining the problems with correct rate between 0.2 and 0.8 as medium difficulty samples.

[0010] In an embodiment of the present application, the two-stage training strategy further comprises: in the supervised fine-tuning stage, using the "thinking-searching-observing" interaction tracks generated by the teacher model as training samples, only retaining the tracks with correct final answers for training of the student model; in the reinforcement learning stage, using Group Relative Policy Optimization (GRPO) algorithm, by calculating the normalized advantage function and maximizing the objective function, the multi-round search and reasoning strategy of the model is optimized, wherein the default value of the importance sampling truncation width ε is 0.2.

[0011] In an embodiment of the present application, further comprising: dynamically adjusting the reward of the interaction trajectory of the model in the reinforcement learning stage, wherein in each round of interaction, if the matching degree of the search query generated by the model and the path entity in the knowledge graph exceeds the preset threshold, a part of positive reward is given to enhance the intermediate step guiding ability of the model in the search process.

[0012] To achieve the above purpose, the second aspect of the present application proposes a device for improving the long-range deep search reasoning ability of a large language model, comprising: a knowledge graph guiding module for performing semantic-guided random walk based on a knowledge graph to generate structured data containing multi-hop entity paths, wherein each node carries attribute information, and the key entities or time information in the path are blurred by an information masking strategy; a difficulty screening module for screening the generated structured data in three layers, first using a model without search capability to perform zero retrieval judgment to filter simple problems, secondly using a model with search capability to perform multiple attempts to screen medium difficulty problems, and finally filtering problems that are not dependent on external retrieval according to the average search call number; a two-stage training module for adopting a two-stage training strategy, in the first stage, a teacher model generates high-quality samples containing "thinking-search-observation" interaction trajectories, and the samples are supervised and fine-tuned to transfer the basic search strategy, in the second stage, a GRPO reinforcement learning algorithm is used in a real search environment, and the multi-round search and reasoning process of the model is optimized based on a normalized advantage function, and only when the model completes normal interaction and outputs the correct answer, a positive reward is given; a dynamic reward adjustment module for dynamically adjusting the reward of the interaction trajectory of the model in the reinforcement learning stage, wherein in each round of interaction, if the matching degree of the search query generated by the model and the path entity in the knowledge graph exceeds the preset threshold, a part of positive reward is given to enhance the intermediate step guiding ability of the model in the search process.

[0013] In an embodiment of the present application, the knowledge graph guiding module is further used for: using a large language model to evaluate the semantic relevance of candidate next-hop entities, and only retaining entities with strong semantic connection with the current node as path expansion nodes; by adjusting the path length k and the node out-degree threshold of the random walk, the complexity of the generated problem is controlled, wherein the default value of k is 5, and the default value of the out-degree threshold is ≥ 3.

[0014] In an embodiment of the present application, the difficulty screening module is further used for: using a GPT-4o model without browsing capability to perform zero retrieval judgment on the problem, if the model directly gives the correct answer without external search, the problem is excluded; using a GLM-4-Z1-Rumination-32B model with browsing function to perform 8 inference attempts with search, and only retaining the problems with correct rate between 0.2 and 0.8 as medium difficulty samples.

[0015] In an embodiment of the present application, the two-stage training module is further configured to: in the supervised fine-tuning stage, using the "thinking-searching-observing" interaction trajectory generated by the teacher model as the training sample, only retaining the trajectory with correct final answer for training the student model; in the reinforcement learning stage, using the Group Relative Policy Optimization (GRPO) algorithm, optimizing the multi-round search and reasoning strategy of the model by calculating the normalized advantage function and maximizing the objective function, wherein the default value of the importance sampling truncation width epsilon is 0.2.

[0016] In an embodiment of the present application, further comprising a dynamic reward adjustment module configured to dynamically adjust the interaction trajectory of the model in the reinforcement learning stage, wherein in each round of interaction, if the matching degree of the search query generated by the model and the path entity in the knowledge graph exceeds a preset threshold, a partial positive reward is given to enhance the intermediate step guiding ability of the model in the search process.

[0017] The method and device of the embodiments of the present application significantly improve the long-range retrieval reasoning ability of the large language model in cross-domain complex problems, enhance the tool calling strategy and evidence fusion effect of the model under the multi-hop path, and effectively overcome the problems of outdated knowledge and short-sighted search.

[0018] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description, taken in conjunction with the accompanying drawings, in which:

[0020] Figure 1 A flowchart of a method for improving the long-range deep search reasoning ability of a large language model according to an embodiment of the present application;

[0021] Figure 2 A BrowseDev framework diagram according to an embodiment of the present application;

[0022] Figure 3 An automatic multi-hop data construction graph based on a knowledge graph according to an embodiment of the present application;

[0023] Figure 4 A performance evaluation diagram of the model after two-stage training of the BrowseDev framework and other models according to an embodiment of the present application;

[0024] Figure 5 A flowchart of another method for improving the long-range deep search reasoning ability of a large language model according to an embodiment of the present application;

[0025] Figure 6 FIG. 1 is a structural diagram of an apparatus for improving long-range deep search reasoning capability of a large language model according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings and in combination with the embodiments. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0028] A method and apparatus for improving long-range deep search reasoning capability of a large language model according to an embodiment of the present application will be described below with reference to the accompanying drawings.

[0029] Figure 1 FIG. 1 is a flowchart of a method for improving long-range deep search reasoning capability of a large language model according to an embodiment of the present application, as shown in FIG. 1, comprising: Figure 1

[0030] S1, performing semantic-guided random walk based on a knowledge graph to generate structured data containing multi-hop entity paths, wherein each node carries attribute information, and key entities or time information in the path are blurred by an information masking strategy.

[0031] Specifically, the "performing semantic-guided random walk based on a knowledge graph to generate structured data containing multi-hop entity paths, and blurring key entities or time information in the path by an information masking strategy" in the present application is the core technical step of the data automatic generation module, and the purpose is to construct training samples with high search difficulty and long-range reasoning requirements, so as to improve the performance of the model in complex search tasks.

[0032] ​At the technical implementation level, this step first performs a restricted random walk in a structured knowledge graph (such as KILT, Aminer, etc.). The walk process is assisted by a large language model (LLM) to ensure the semantic coherence and cross-domain nature of the path. The initial node is determined by random sampling, and the selection of subsequent nodes combines rule filtering and semantic guidance strategies. Rule filtering includes limiting entity out-degree (the default threshold is 5) to avoid overly divergent paths; semantic guidance scores candidate entities for semantic relevance using LLM, and only keeps entities with strong semantic connections to the current node as the next hop node. The walk length k can be adjusted, and in the experiment it is set to 5 hops to generate paths of moderate complexity.

[0033] Each node in the generated path carries attribute information (such as time, location, and character identity), and one attribute of the end node is selected as the answer. Subsequently, an information masking strategy is used to obscure key entities or time information in the path, such as replacing "1975" with "1970s" or "New York" with "a major international city". This strategy is implemented through a predefined obfuscation rule library (such as time granularity degradation and entity name generalization) to enhance the model's reliance on context reasoning rather than memory.

[0034] At the parameter index level, this step involves key parameters such as path length k (default 5), entity out-degree threshold (default 5), and obfuscation granularity (such as time obfuscation to ten-year level, entity obfuscation to category level). By adjusting these parameters, the difficulty distribution of the generated data can be flexibly controlled to adapt to the needs of different training stages.

[0035] In application scenarios, this step is mainly used to generate multi-hop retrieval training data, suitable for long-range reasoning tasks (such as BrowseComp, Frames, etc.), especially in complex problems across domains and time. Through obfuscation, the model is forced to gradually reason in multiple rounds of search, thereby improving its adaptability in real search environments.

[0036] The technical effect of this step is that, through the combination of structured path generation and information masking strategies, the retrieval complexity and reasoning depth of the training data are effectively improved, providing a high-quality and high-difficulty sample basis for subsequent supervised fine-tuning and reinforcement learning, significantly enhancing the model's long-range search and decision-making capabilities.

[0037] S2, the generated structured data is subjected to three levels of difficulty screening, first using a model without search capability to filter simple problems, then using a model with search capability to attempt multiple times to filter medium difficulty problems, and finally filtering problems that are not dependent on external retrieval based on the average number of search calls

[0038] Specifically, the application adopts a three-layer difficulty screening mechanism in the data screening stage to ensure that the training data achieves an optimal balance between complexity and retrieval dependency. The technical implementation principle of this step is based on hierarchical evaluation of model capability and quantitative analysis of behavior patterns, and the specific operation mode is as follows:

[0039] In the first layer screening, a model without search capability (such as GPT-4o) is used for zero retrieval judgment. The model performs single inference on the generated structured question without accessing any external retrieval interface, and if its output is consistent with the reference answer, it is determined as an "easy question" and is rejected. This process aims to filter out questions that can be answered only by relying on internal knowledge of the model, ensuring that the subsequent training data is challenging to the model's retrieval capability.

[0040] The second layer screening is performed by a model with search capability (such as GLM-4-Z1-Rumination-32B) for multiple attempts, a total of 8 independent inferences and retrieval interactions. In each interaction, the model can call the Jina search engine for information acquisition and generate answers based on the returned results. The application counts the number of correct answers c i for each question and sets the screening condition c i , that is, only medium difficulty questions are retained. This parameter range ensures that the questions are neither completely unsolvable nor too simple, thereby providing effective training signals for the reinforcement learning stage.

[0041] The third layer screening is based on the average search call number ˉ{n}_i, if ˉ{n}_i<;3, it is considered that the problem is not dependent on external retrieval, and is also rejected. This threshold is set based on the average behavior pattern of the model in the multi-hop task in the experiment, ensuring that the retained questions require the model to perform sufficient multi-round retrieval and information integration.

[0042] This three-layer screening mechanism plays a key role in the entire technical solution. Through automated filtering strategies, it effectively improves the rationality of the difficulty distribution of the training data and the diversity of the retrieval behavior, providing a high-quality, high-challenge sample basis for the subsequent two-stage training (SFT+RL), thereby significantly enhancing the model's reasoning and decision-making capabilities in long-range deep search tasks.

[0043] S3, adopts a two-stage training strategy, the first stage generates high-quality samples containing "thinking-search-observation" interaction trajectories through a teacher model, and performs supervised fine-tuning on the samples to transfer the basic search strategy, the second stage uses GRPO reinforcement learning algorithm in the real search environment, based on the normalized advantage function to optimize the multi-round search and inference process of the model, only when the model completes the normal interaction and outputs the correct answer, give positive reward.

[0044] Specifically, the two-stage training strategy of the present application is the core technical path to realize the improvement of model inference ability in long-range deep search tasks. In the first stage, the teacher model generates high-quality samples containing "thinking-search-observation" interaction trajectories, and through supervised fine-tuning (Supervised Fine-Tuning, SFT), the basic search strategy is migrated to the student model. Specifically, the teacher model is GLM-4-Z1-Rumination-32B, which has strong search inference ability and can simulate the decision-making process of humans in multi-hop retrieval tasks. In this stage, the teacher model generates multiple interaction trajectories for each question, each trajectory consisting of several "thinking-search-observation" cycles. The "thinking" stage generates inference instructions, the "search" stage calls the Jina search engine to obtain web results, and the "observation" stage parses the search results and updates the context state. To ensure sample quality, only the trajectory with the correct final answer is retained as a supervision signal to minimize the cross-entropy loss function of the student model, and the objective function is:

[0045] pi θ a t s t h t-1

[0046] where pi θ is the student strategy network, a t is the search action at step t, s t is the current state, and h t-1 is the historical context. The learning rate of the SFT stage is set to 1e-5, and the training is performed for 3 rounds to ensure that the student model can stably learn the search behavior pattern of the teacher model.

[0047] In the second stage, the Group Relative Policy Optimization (GRPO) reinforcement learning algorithm is used to optimize in the real search environment. GRPO guides policy updates through normalized advantage function s t a t , and its advantage function is calculated as:

[0048] s t a t s t a t

[0049] where s t a tThe normalization process can enhance the stability of policy update for the original advantage function. In this application, only when the model completes the normal interaction process and outputs the final answer consistent with the reference answer, a positive reward r = 1 is given, otherwise r = 0. The maximum number of interactions is set to 50, the sampling temperature τ = 0.8, the learning rate 3e-5, and the SGlang inference framework is used for trajectory parallel sampling in the reinforcement learning stage. The long-term decision-making ability of the model is optimized through multiple rounds of interaction in this stage, so that it has stronger search path planning and evidence fusion ability when facing ambiguous and cross-domain problems.

[0050] The two-stage training strategy effectively solves the problem of unstable model search behavior and short-sighted reasoning path in traditional single-stage training. By establishing a basic behavior pattern through SFT and optimizing the strategy in a real environment through RL, the performance of the model in long-range search tasks is significantly improved. Experimental results show that this method is better than GPT-4o and DeepSeek-R1 on BrowseComp-266 and Frames datasets, demonstrating its practical value and innovation in complex reasoning scenarios.

[0051] Further, as shown in Figure 2 The overall architecture of the present application is composed of three main modules: data automatic generation, difficulty screening, and two-stage training, aiming to systematically enhance the long-range reasoning ability of large models in complex search and reasoning tasks.

[0052] In the data automatic generation module, the present application uses a knowledge graph to perform a restricted random walk to extract long-path entity sequences, and uses a large language model to construct multi-hop and fuzzy questions. These questions have higher uncertainty and retrieval ambiguity than the deterministic entities and shallow connections involved in HotpotQA, which can significantly stimulate the deep reasoning potential of the model.

[0053] In the data selection and difficulty screening stage, the present application proposes a three-layer filtering mechanism: first, exclude easy-to-answer questions through a static large language model; second, remove overly difficult samples through dynamic browser interaction evaluation; finally, further filter out samples with insufficient search usage. Through this series of screening, the present application finally constructs a high-quality training set that is both challenging and practical.

[0054] In the two-stage training process, the present application first uses an expert model with search capability to generate high-quality trajectories, and distills them to the student model through supervised fine-tuning to establish an initial strategy; then introduces GRPO reinforcement learning to train in a real search environment through multiple rounds of interaction, and optimizes the strategy update combined with the reward of answer correctness.

[0055] Step 1: High-difficulty data automatic generation module (complex question construction based on knowledge graph).

[0056] Knowledge graph path sampling: using LLM-assisted random walk method to construct multi-hop entity paths P = [v1, …, v k ] in the knowledge graph, where each node is attached with attribute information A = [a1, …, a k ].

[0057] Question and answer construction: based on path and entity attributes, automatically generate multi-hop reasoning questions and corresponding answers. Some attribute of the end entity as the answer.

[0058] Information masking strategy: fuzzy processing of key information (such as specific time, name, place name, etc.) in question text, to improve the model's reasoning ability for context and structured information.

[0059] Difficulty control mechanism: by adjusting the path length k and entity connection degree d min ,d max , control the complexity of generated questions, and realize difficulty-adjustable data synthesis.

[0060] Step 2: Data selection and difficulty screening (multi-stage screening to ensure high quality).

[0061] If the automatically constructed questions are too simple or too difficult, the questions will hinder the update speed of the model in the reinforcement learning process. Therefore, the present application designs a comprehensive screening mechanism.

[0062] Easy question filtering: use GPT-4o without browsing ability to perform batch evaluation, and remove questions that can be directly answered by parameter knowledge.

[0063] Difficulty screening: use GLM-4-Z1-Rumination-32B with browsing function to perform 8 attempts, and calculate the correct rate for each question, and filter out difficult and easy samples.

[0064] Tool call filtering: count the average search call times of each question, if it is less than the threshold T min , it means that the question is not dependent on external search, and it is removed.

[0065] Step 3: Two-stage training (SFT+RL).

[0066] Phase one: supervised fine-tuning (SFT).

[0067] Teacher-student framework: the teacher model is GLM-4-Z1-Rumination-32B, and the student model is QwQ-32B.

[0068] Trajectory Sampling and Filtering: The teacher model solves the problem multiple times, and rejection sampling is used to retain only the high-quality problem-solving process that generates the correct answer, which is then used as training samples for the student model.

[0069] Skills transfer: Transferring and establishing basic search tool usage skills by imitating search patterns.

[0070] Phase Two: Multi-round Reinforcement Learning (RL) Based on a Search Framework.

[0071] Interactive modeling: Each round consists of a "thinking → searching → observing" cycle. The model needs to dynamically determine the search and click actions based on environmental feedback, and finally generate the answer.

[0072] Reward function design: If the final answer is correct and the interaction process is completed, the reward is 1; otherwise, it is 0.

[0073] Optimization method: The Group Relative Policy Optimization (GRPO) strategy is adopted, which drives the policy network update through normalized advantage function to promote the learning of deep inference and tool calling strategies.

[0074] The BrowseDev framework of this invention revolves around the core objective of "optimizing the long-range search capability of large language models," and improves the model's capabilities in this regard from three aspects: automated data synthesis, data filtering, and multi-stage model training.

[0075] This data construction algorithm is one of the core innovations of this application. Existing multi-hop question-answering datasets (such as HotpotQA) often explicitly provide all entity clues in the question, allowing the model to assemble the answer with only a few direct searches, without truly requiring "thinking and searching simultaneously." To encourage the model to continuously call external search tools during the problem-solving process, such as... Figure 3 As shown, this application designs a method for automated data synthesis based on knowledge graphs:

[0076] First, using a knowledge graph as the framework, this application employs Large Language Model (LLM)-assisted random walks to collect logically coherent and cross-domain multi-hop paths:

[0077] P = [v1, ..., v] k ]

[0078] Each node is accompanied by attribute information A = [a1, ..., a2]. k The initial node is randomly selected. Regarding the selection of the next-hop entity, this application implements two policy constraints: one is rule filtering, where each selection of the next-hop entity satisfies the following:

[0079] d min ≤d(v)≤dmax

[0080] Another aspect is to prompt the large language model to remove candidates that are weakly semantically connected to the current node, in order to maintain the logical consistency of the path. Starting from the starting node v0, the path P is gradually expanded according to the above strategy, and a certain explicit attribute of the end node is randomly selected as the answer.

[0081] Secondly, given the path and the selected answer, the large language model deliberately obscures the key information (for example, replacing 1975 with the 1970s, or hiding the proper name and the place), forcing the model to rely on external retrieval rather than direct memory.

[0082] By adjusting the walk length k and the out-degree threshold d min ,d max , the difficulty of the question can be flexibly controlled. In the experiment, k ∈ [5, 8], d min = 3, d max = 9, based on two kinds of knowledge graphs KILT and Aminer, the automatic process generates 6808 high-difficulty samples as original data.

[0083] Further, after obtaining 6808 high-difficulty candidate question and answer pairs in the automatic generation stage, there may still be two types of samples mixed in: one type is a simple question that can be answered only by relying on the internal knowledge of the model, and the other type is a difficult question that cannot be solved no matter how sampling is performed. To ensure that the remaining questions require external retrieval and can produce positive rewards within a limited interaction, the application designs an algorithm to perform three filters in turn. The entire process does not introduce any manual labeling and is completely automated by the program.

[0084] First, use GPT-4o with search function turned off to sample all samples once without retrieval. Any question that the model directly hits the answer is excluded; this step cuts off shallow questions that can be completed by relying on internal memory.

[0085] Next, give the remaining samples to GLM-4-Z-Rumination-32B with browsing ability to perform N = 8 times of search reasoning, and record the number of times q that each question is answered correctly. Only when the inequality 0 < f1 < q < f2 < N holds, the question will be kept, thus excluding "too difficult" samples that are never answered correctly (q = 0), and also excluding most "too easy" samples that can be easily answered by the model simply calling the tool.

[0086] Finally, even after the first two data filtering, some questions may still be easily solved by the tool model with a small number of web interactions, so the average number of times n tools the search tool is called for each question is counted. If n toolsIf the value is less than α, then the question is considered to have insufficient reliance on external retrieval and will be deleted accordingly.

[0087] In the experiment, f1=1, f2=6, α=3. After three rounds of screening, the data size was reduced from 6808 to 2205 (retention rate 32%), of which 1034 were used for supervised fine-tuning and the remaining 1171 were used for reinforcement learning.

[0088] After obtaining 2,205 high-quality question-answer pairs, this application adopts a two-stage strategy of supervised fine-tuning (SFT) and multi-round reinforcement learning (RL). First, the model learns the multi-round tool call format, and then the long-range deep search capability of the model is further improved through the reward signal of reinforcement learning.

[0089] First, supervised fine-tuning is performed on the current model. GLM-4-Z1-Rumination-32B is selected as the teacher model, and DeepSeek-R1-Distill-Qwen-14B as the student model. For each question-and-answer data point, the teacher model generates multiple "think-search-observe" interaction trajectories, which are only retained when the entire trajectory's answer is correct. These high-quality trajectories are used to minimize the cross-entropy loss of the student model.

[0090]

[0091] Where s t It is the historical context of step t, π θ This is a student policy network. The training phase consists of 3 rounds with a learning rate of lr = 1e-5.

[0092] Subsequently, multiple rounds of reinforcement learning were conducted to optimize the decision-making quality of the model's "thinking and searching" process. The interaction trajectory formed by sampling each interaction between the model and the environment is formalized as follows:

[0093] T = (prompt, q, (c, a, o)) m ,y,a eos ),m≤n

[0094] Where state s t =(c <t ,a t-1 ,o t-1 ), a∈{search,click}, a eos =finish, the search API is executed by the Jina engine, and the inference temperature is set to T=0.9 for each step. When the trajectory is at a eos The final answer y given by the model and the reference answer y * Consistency is required to receive a reward:

[0095]

[0096] The above reward indicates that a positive reward can be obtained only when the model stops normally and gives the correct answer. During the training process, the loss function masks the text given by the environment (i.e., o in each round of dialogue).

[0097] The present application adopts the GRPO (Group Relative Policy Optimization) reinforcement learning algorithm for optimization. The algorithm uses the SGlang inference framework for sampling, and for the same prompt, a higher sampling temperature T is set to extract G search interaction trajectories in parallel and calculate the normalized advantage:

[0098]

[0099] The subsequent maximization of the objective function is used for training:

[0100]

[0101] Where ε controls the truncation width of the importance sampling ratio, and β is used to punish the deviation distance between the current policy and the reference policy π ref , ensuring that the gradient update is both exploratory and stable.

[0102] In the reinforcement learning experiment, the maximum interaction round number of the model and the search environment is set to 50, the sampling temperature T = 0.9, ε = 0.2, β = 0.02, and the learning rate lr = 2e-6.

[0103] The present application is mainly aimed at multi-step search tasks and verifies the effectiveness of the present application. The data and algorithms obtained by using the present application are used to test the open source model QwQ-32B. The test data set includes four public search benchmark data sets: BrowseComp-266, BrowseComp-ZH randomly selected from BrowseComp, and HotpotQA-512 and Frames randomly selected from the HotpotQA development evaluation set, and comparative analysis is performed with the current mainstream large model.

[0104] For example, Figure 4As shown, the experimental results show that, through supervised fine-tuning (SFT), the accuracy of the model on the BrowseComp dataset reaches the current top inference model such as QwQ-32B and Deepseek-R1; further, after reinforcement learning (RL), the evaluation effect of the model on all datasets is significantly improved compared with supervised fine-tuning. Specifically, on the BrowseComp dataset, the performance is improved to 6.0 points; on the Frames dataset, the performance is improved to 71.4 points. On these two datasets, the performance of the model completely surpasses GPT-4o (access function call) and DeepSeek-R1. This series of experiments fully prove that combining high-difficulty automatically synthesized data with the "two-stage SFT + RL" strategy can significantly enhance the ability of the model in the long-range search task, thereby endowing the model with the real "thinking while searching" long-range reasoning and search ability.

[0105] The method for improving the long-range deep search reasoning ability of the large language model of the embodiment of the application significantly improves the long-range reasoning and search ability of the large language model in the cross-domain multi-hop retrieval task. Through the knowledge graph driven data automatic generation and the two-stage training strategy, the performance of the model under the complex search path is better than that of the existing mainstream large model.

[0106] Embodiment 2

[0107] Figure 5 is a flowchart of the method for improving the long-range deep search reasoning ability of the large language model according to an embodiment of the application.

[0108] As Figure 5 shown, the method for improving the long-range deep search reasoning ability of the large language model includes the following steps:

[0109] S1, performing semantic guided random walk based on a knowledge graph to generate structured data containing a multi-hop entity path, wherein each node carries attribute information, and the key entities or time information in the path are blurred through an information masking strategy.

[0110] The technical effect of this step is that, through the combination of structured path generation and the information masking strategy, the retrieval complexity and reasoning depth of the training data are effectively improved, providing a high-quality and high-difficulty sample basis for subsequent supervised fine-tuning and reinforcement learning, and significantly enhancing the long-range search and decision-making ability of the model.

[0111] Further, S1 includes:

[0112] S11, using a large language model to perform semantic relevance evaluation on the candidate next-hop entity, and only retaining the entity with strong semantic connection with the current node as the path expansion node.

[0113] Specifically, the "semantic relevance evaluation of candidate next-hop entities using large language models, and only keeping entities with strong semantic connection to the current node as path expansion nodes" in this application is a key step in the knowledge graph-driven multi-hop path generation process. The core goal is to ensure that the generated path has logical coherence at the semantic level, thereby improving the complexity and reasoning value of the subsequent synthesized questions.

[0114] At the technical implementation level, this step is based on the semantic understanding ability of large language models (LLM) to evaluate the semantic relevance of candidate next-hop entities of the current node in the knowledge graph. Specifically, the model first receives the entity name and its attribute information (such as category, description, and associated relationships) of the current node, and combines the context semantics of the current path to calculate the semantic matching degree of the candidate entities. In some implementations, this process can use the cosine similarity (Cosine Similarity) between entity embedding (Entity Embedding) and contextual embedding (Contextual Embedding) as an evaluation indicator, and the similarity threshold is set to 0.65 by default to filter out entities with weak semantic relevance. Further, the model can also distinguish between strongly related entities and weakly related entities through a contrastive learning (Contrastive Learning) mechanism, thereby improving the robustness of the screening.

[0115] At the parameter indicator level, the LLM model used in this application is GLM-4-Z1-Rumination-32B, which has strong semantic reasoning and entity relationship modeling capabilities. In the knowledge graph, the out-degree of each node (i.e., the number of candidate next-hop entities) is usually controlled between 5 and 10 to avoid overly divergent paths. At the same time, the model dynamically adjusts the screening intensity based on the current path length k (default k = 4-6) when generating the path, and the longer the path, the higher the semantic relevance requirement to maintain the incremental difficulty of reasoning.

[0116] At the application scenario level, this step is widely used in the automatic generation process of multi-hop question and answer data, especially in constructing cross-domain, fuzzed, and high-difficulty retrieval questions, to ensure that the model must rely on external information rather than internal memory in the search path. For example, in synthesizing a question about "a scientist won the Nobel Prize in the 1970s", the model needs to gradually reason from the "scientist" node to the "Nobel Prize" node in the knowledge graph, and the entity selection at each step needs to be evaluated by semantic relevance to ensure the reasonableness of the path and the depth of reasoning.

[0117] The technical effect of this step is that, through entity screening at the semantic level, the logical consistency and reasoning complexity of the path are effectively improved, thereby generating more challenging training samples. This not only enhances the model's reasoning ability in long-range search tasks, but also provides a high-quality training data foundation for subsequent supervised fine-tuning and reinforcement learning, significantly improving the model's performance on datasets such as BrowseComp-266 and Frames.

[0118] S12, by adjusting the path length k of random walk and the node out-degree threshold, the complexity of the generated problem is controlled, where the default value of k is 5 and the default value of the out-degree threshold is ≥ 3.

[0119] Specifically, in the data automatic generation module, the application adjusts the path length k of random walk and the node out-degree threshold d to realize dynamic control of the complexity of the generated problem. This step is based on structured knowledge graph (such as KILT, Aminer, etc.) for entity path sampling, and its core is to construct cross-domain, multi-hop entity sequences through LLM-assisted semantic guided random walk, thereby generating high-difficulty retrieval problems with ambiguous clues. The path length k determines the number of reasoning hops required for the problem, and the larger k is, the more complex the problem is, and the model needs to perform more retrievals and information integration. In this application, the default setting is k = 5, i.e. each path contains 5 entity nodes, ensuring that the problem has sufficient reasoning depth, while avoiding semantic breakage or generation efficiency due to too long path.

[0120] The node out-degree threshold d is used to control the number of next-hop entities that can be selected at each step of random walk. The application sets the default out-degree threshold d ≥ 3, i.e. each node contains at least 3 out-edges, to increase the diversity and uncertainty of the path, thereby improving the retrieval difficulty and ambiguity of the generated problem. In specific implementation, the random walk process combines rule filtering and semantic filtering: rule filtering ensures that the candidate nodes meet the structured constraints (such as relationship type matching), and semantic filtering evaluates the semantic consistency of the candidate nodes and the current path through LLM, and eliminates jumps with inconsistent semantics.

[0121] This step plays a key role in the entire training framework, and its output directly affects the training quality of subsequent supervised fine-tuning and reinforcement learning. By parameterizing k and d, the application realizes the adjustability and automation of data complexity, providing challenging training samples for the model, thereby effectively stimulating its long-range reasoning and multi-hop retrieval ability.

[0122] S2, the generated structured data is subjected to three-level difficulty screening, first, a model without search capability is used to judge zero retrieval to filter simple problems, second, a model with search capability is used to attempt multiple times to filter medium difficulty problems, and finally, problems that are not dependent on external retrieval are filtered according to the average number of search calls.

[0123] Further, S2 comprises:

[0124] S21, using a GPT-4o model without browsing capability to make zero retrieval judgment on the question, if the model directly gives the correct answer without external search, the question is eliminated.

[0125] Specifically, in the "easy question filtering" step of the data screening stage, the application uses a GPT-4o model without browsing capability to make zero retrieval judgment on the automatically generated fuzzy retrieval question. The core technical principle of this step is to use a model that is not connected to an external search interface to evaluate whether the question can be answered directly through its internal parameter knowledge, thereby identifying and eliminating low difficulty samples that do not require external retrieval. This process aims to ensure that only questions that truly rely on multiple rounds of search and reasoning are retained in the subsequent training data, improving the challenge and training efficiency of the training set.

[0126] The specific implementation is: input the automatically generated 6808 fuzzy retrieval questions into the GPT-4o model, which performs reasoning based on its pre-trained knowledge base without external search support and outputs the answer. If the model can give a result consistent with the reference answer in a single reasoning, it is determined that the question is an "easy question" and is removed from the training set. This process uses batch processing to automatically evaluate through API interface or locally deployed reasoning engine, ensuring the scalability and consistency of data screening.

[0127] In terms of parameter settings, the reasoning temperature (temperature) of the GPT-4o model is set to 0.7 by default to balance the diversity and accuracy of the generated answers. At the same time, the application sets the answer matching criterion to exact string match, that is, the output answer must be exactly the same as the reference answer, and does not accept approximate or semantically similar answers. In addition, this step is executed in a one-time batch evaluation without iteration or multiple attempts, ensuring screening efficiency.

[0128] This step is mainly used to build high-quality training data sets in practical applications, especially in long-range search tasks, to ensure that the model does not produce strategy degradation due to the presence of a large number of "easy questions" in the training data. By eliminating questions that can be answered only by relying on internal knowledge, the model is forced to rely on external retrieval and multiple rounds of reasoning in subsequent training, thereby improving its search planning and information integration capabilities on complex and fuzzy questions.

[0129] In terms of technical effects, this step effectively filters about 68% of low-difficulty problems, significantly improving the difficulty distribution quality of the training data. Experimental results show that after three screenings, of the 2205 problems finally retained, only 1034 are used for supervised fine-tuning, and the rest are used for reinforcement learning, ensuring that the model can obtain challenging sample inputs in different training stages, thereby enhancing its long-range reasoning and search capabilities.

[0130] S22, using the GLM-4-Z1-Rumination-32B model with browsing function, 8 inference attempts with search are performed, the accuracy is counted, and only the problems with accuracy between 0.2 and 0.8 are retained as medium difficulty samples.

[0131] Specifically, in the "difficulty screening" step of the data screening phase, the GLM-4-Z1-Rumination-32B model with browsing function is used to perform 8 inference attempts with search on the 6808 candidate problems generated automatically, to count their accuracy, and to screen medium difficulty samples accordingly. The core technical principle of this step is to evaluate the reasoning ability and stability of the model in the face of fuzzification, cross-domain, multi-hop retrieval problems through multiple rounds of interaction with external search tools, thereby constructing a challenging but learnable training dataset.

[0132] In specific implementation, each question is independently input to the GLM-4-Z1-Rumination-32B model, and the model generates a "thinking-search-observation" interaction trajectory in each round of inference, where the search operation is performed through the Jina engine. In each round of inference, the model decides whether to call the search API based on the current context state, and parses the returned web page content to assist subsequent reasoning. This process is performed 8 times, each time using an independent random seed to ensure sampling diversity. Finally, the number of correct answers of the model for each question is counted, and its accuracy is calculated, and the questions with accuracy between 0.2 and 0.8 are retained as medium difficulty samples.

[0133] The key parameters involved in this step include: number of inference attempts (default 8), accuracy threshold range (0.2 to 0.8), search API response parsing strategy (entity extraction and context matching based on HTML structure). By setting this range, it avoids the problem of the model lacking training value due to too easy questions, and prevents the problem of gradient disappearance or strategy collapse in the training process due to too difficult questions.

[0134] This step is mainly used to build high-quality training data sets in practical applications, especially for model pre-training and fine-tuning of long-range reasoning and multi-round search tasks. Its technical effect lies in effectively filtering out samples with low training value, ensuring that the sample distribution in the subsequent supervised fine-tuning and reinforcement learning stage is in a reasonable difficulty interval, thereby improving the reasoning robustness and generalization ability of the model in complex search tasks.

[0135] S3 adopts a two-stage training strategy. In the first stage, a teacher model generates high-quality samples containing "thinking-search-observation" interaction trajectories, and the samples are supervised fine-tuned to transfer the basic search strategy. In the second stage, the GRPO reinforcement learning algorithm is used in the real search environment to optimize the multi-round search and reasoning process of the model based on the normalized advantage function, and only when the model completes normal interaction and outputs the correct answer is a positive reward given.

[0136] Further, S3 includes:

[0137] S31, in the supervised fine-tuning stage, the "thinking-search-observation" interaction trajectory generated by the teacher model is used as the training sample, and only the trajectory with the correct final answer is retained for the training of the student model.

[0138] Specifically, in the supervised fine-tuning stage, the present application adopts a "teacher-student" framework, uses the "thinking-search-observation" interaction trajectory generated by the teacher model as the training sample, and only retains the trajectory with the correct final answer for the training of the student model. The core of this step is to guide the student model to learn effective multi-round search and reasoning strategies through high-quality trajectory data, thereby realizing the cold start optimization of long-range deep search tasks.

[0139] In terms of technical implementation, the teacher model is GLM-4-Z1-Rumination-32B, which has strong search and reasoning capabilities. For each input question, the teacher model performs multiple rounds of "thinking-search-observation" interaction to generate a complete reasoning process. In each step, the model first generates reasoning text based on the current context (thinking), then calls the search API (such as Jina engine) to obtain relevant information (search), and finally observes and updates the context based on the search results. This process can be repeated multiple times until the model outputs the final answer or reaches the maximum number of interactions (such as 50 rounds).

[0140] To ensure the quality of training data, the application introduces a reject-sampling mechanism, which only retains trajectories with final answers consistent with reference answers. Specifically, if a trajectory ends with a "stop" state and the output answer matches the standard answer, the trajectory is retained; otherwise, it is discarded. This process models each step in the trajectory through a cross-entropy loss function, enabling the student model QwQ-32B to learn the decision logic and search behavior of the teacher model.

[0141] In terms of parameter settings, the supervised fine-tuning phase conducts 3 rounds of training, with a learning rate of 1e-5, a batch size of 8, and a maximum trajectory length of 50. The context state of each step in the trajectory is encoded as the model input, ensuring that the model can understand the contextual dependencies of multiple rounds of interaction.

[0142] This step plays a key role in the overall framework as a cold start, providing a structured and imitable search strategy for the subsequent reinforcement learning phase. By retaining only correct trajectories, it effectively avoids the propagation of incorrect behavior and improves the initial reasoning ability and search call accuracy of the student model in complex search tasks.

[0143] S32, in the reinforcement learning phase, the Group Relative Policy Optimization (GRPO) algorithm is used to optimize the multi-round search and reasoning strategy of the model by calculating the normalized advantage function and maximizing the objective function, where the default value of the importance sampling truncation width ε is 0.2.

[0144] Specifically, in the reinforcement learning phase, the application uses the Group Relative Policy Optimization (GRPO) algorithm to optimize the multi-round reasoning and search strategy of the model in long-range deep search tasks by calculating the normalized advantage function and maximizing the objective function. The GRPO algorithm introduces a relative advantage estimation mechanism based on policy groups on the basis of the traditional PPO (Proximal Policy Optimization), thereby improving the stability and exploration efficiency of policy updates in complex search environments with multiple policy cooperation.

[0145] At the technical implementation level, GRPO generates multiple policy groups of interaction trajectories through parallel sampling, and each group of trajectories corresponds to a different search behavior sequence. Specifically, for the same prompt, the model generates N trajectories (N is usually set to 8-16) in parallel at a higher sampling temperature (such as 0.8), each trajectory containing multiple rounds of “thinking-searching-observing” cycles, where the search API is executed by the Jina engine. In each round of interaction, the model generates an action (i.e., a search query or a click operation) based on the current state (state), and receives environmental feedback (observation) to update the state. The trajectory is terminated with a “STOP” marker, and if the final answer is consistent with the reference answer, a positive reward (+1) is given, otherwise 0.

[0146] The GRPO algorithm evaluates the relative value of each trajectory by calculating the normalized advantage function. The advantage function is defined as the difference between the trajectory reward and the expected reward of the reference policy, and is normalized by Z-score to eliminate the reward scale difference between different tasks. In the policy updating process, importance sampling truncation technology is used, where the default value of the truncation width ε is 0.2, to limit the ratio deviation between the current policy and the reference policy, preventing high variance problems in the policy updating process. The objective function is the weighted maximization of the policy gradient.

[0147] In practical applications, this step runs in a training environment with multiple rounds of search interaction capability, and the model needs to complete problem solving within a maximum of 50 rounds of interaction. During training, the inference temperature of each round of interaction is set to 0.7, and the learning rate is set to 5e-5 to balance the exploration and convergence speed. This reinforcement learning strategy optimizes the model's performance in fuzzy retrieval, cross-domain reasoning, and multi-hop path planning, and is a key link for the BrowseDev framework to realize the “thinking while searching” capability.

[0148] The method for improving the long-range deep search reasoning capability of a large language model according to the embodiments of the present application significantly improves the long-range reasoning and search capability of a large language model in a cross-domain multi-hop retrieval task. Through knowledge graph driven data automatic generation and a two-stage training strategy, the model performs better than existing mainstream large models in complex search paths.

[0149] S4, dynamically adjusting the rewards of the interaction trajectories of the model in the reinforcement learning stage, wherein in each round of interaction, if the search query generated by the model matches the path entity in the knowledge graph by more than a preset threshold, a partial positive reward is given to enhance the intermediate step guiding capability of the model in the search process.

[0150] Specifically, the step of "dynamically adjusting the reward of the interaction trajectory of the model in the reinforcement learning phase" in this application is a key link to improve the reasoning ability of the model in long-range deep search tasks. The core technical principle of this step is that the matching degree of the search query generated by the model and the path entity in the knowledge graph is evaluated in real time in each round of interaction, and partial positive rewards are given according to the matching degree, so as to enhance the guiding ability of the model to the intermediate reasoning steps in the reinforcement learning process, and avoid falling into local optimum or producing search short-sighted behavior.

[0151] In terms of technical implementation, the semantic matching model (such as BERTScore, Sentence-BERT or entity similarity calculation based on graph embedding) is used to evaluate the matching degree of the search query generated by the model and the current path entity in the knowledge graph. Specifically, the model generates a search query in each reasoning step, and the system calculates the semantic similarity between the query and the entity and its attributes that should be accessed in the current path. If the matching degree exceeds the preset threshold (for example, 0.75), partial positive rewards (for example, 0.2-0.5) are given. This reward mechanism encourages the model to gradually approach the target path during the search process, rather than only focusing on the correctness of the final answer.

[0152] In terms of parameter indicators, the matching degree threshold can be dynamically adjusted according to the task complexity, for example, a higher threshold is set in a multi-hop task to improve search accuracy. The setting of reward value needs to balance exploration and utilization. Too high may lead the model to rely too much on reward signals and ignore the actual answer quality, and too low may not effectively guide policy update. In the experiment, the matching degree threshold is 0.75 and the reward value is 0.4, which achieves good training effect.

[0153] In practical application scenarios, this step is suitable for complex search tasks such as cross-domain, multi-hop and fuzzy clues, such as BrowseComp-266, Frames and other datasets. The model needs to generate search queries in combination with the context in each round of interaction, and obtain immediate feedback through matching degree evaluation with the path of the knowledge graph, so as to gradually optimize its search strategy.

[0154] The technical effect of this step is that by introducing the reward mechanism of intermediate steps, the search guiding ability and path exploration efficiency of the model in long-range reasoning tasks are significantly improved. Compared with the traditional method of giving rewards only when the final answer is correct, the dynamic reward adjustment strategy of this application can more finely depict the behavior of the model in multiple rounds of interaction, enhance its decision-making ability and robustness in complex search paths, and thus realize higher quality of policy optimization.

[0155] Embodiment 3

[0156] In order to realize the above-mentioned embodiments, as Figure 6As shown, the device 10 for improving the long-range deep search reasoning capability of a large language model is also provided in this embodiment, which comprises:

[0157] The knowledge graph guiding module 100 is configured to perform semantic-guided random walk based on a knowledge graph to generate structured data containing multi-hop entity paths, wherein each node carries attribute information, and key entities or time information in the path are blurred through an information masking strategy.

[0158] The difficulty screening module 200 is configured to perform three-layer difficulty screening on the generated structured data, first, a zero retrieval judgment is performed by a model without search capability to filter simple problems, second, a multi-round attempt is performed by a model with search capability to screen medium difficulty problems, and finally, problems that are not dependent on external retrieval are filtered according to the average search call number.

[0159] The two-stage training module 300 is configured to adopt a two-stage training strategy, in the first stage, a teacher model is used to generate high-quality samples containing "thinking-searching-observing" interaction tracks, and the samples are supervised and fine-tuned to transfer the basic search strategy, in the second stage, a GRPO reinforcement learning algorithm is used in a real search environment, and the multi-round search and reasoning process of the model is optimized based on a normalized advantage function, and only when the model completes normal interaction and outputs a correct answer, a positive reward is given.

[0160] The dynamic reward adjustment module 400 is configured to dynamically adjust the interaction track of the model in the reinforcement learning stage, wherein in each round of interaction, if the matching degree between the search query generated by the model and the path entity in the knowledge graph exceeds a preset threshold, a partial positive reward is given to enhance the intermediate step guiding ability of the model in the search process.

[0161] Further, the knowledge graph guiding module is further configured to:

[0162] Use a large language model to evaluate the semantic relevance of candidate next-hop entities, and only keep entities with strong semantic connection with the current node as path expansion nodes;

[0163] By adjusting the path length k and the node out-degree threshold of the random walk, the complexity of the generated problem is controlled, wherein the default value of k is 5, and the default value of the out-degree threshold is ≥ 3.

[0164] Further, the difficulty screening module is further configured to:

[0165] Use a GPT-4o model without browsing capability to perform zero retrieval judgment on the problem, if the model directly gives a correct answer without external search, the problem is excluded;

[0166] The GLM-4-Z1-Rumination-32B model with a browsing function is used for 8 reasoning attempts with search, the correct rate is counted, and only the problems with the correct rate between 0.2 and 0.8 are reserved as medium-difficulty samples.

[0167] Further, the two-stage training module is further used for:

[0168] In the supervised fine-tuning stage, the "thinking-search-observation" interaction trajectory generated by the teacher model is used as a training sample, and only the trajectory with the correct final answer is reserved for the training of the student model;

[0169] In the reinforcement learning stage, the Group Relative Policy Optimization (GRPO) algorithm is adopted, the normalized advantage function is calculated, and the target function is maximized to optimize the multi-round search and reasoning strategy of the model, wherein the default value of the importance sampling truncation width epsilon is 0.2.

[0170] Further, it further comprises a dynamic reward adjustment module for dynamically adjusting the interaction trajectory of the model in the reinforcement learning stage, wherein in each round of interaction, if the search query generated by the model matches the path entity in the knowledge graph by more than a preset threshold, a part of the positive reward is given to enhance the intermediate step guiding ability of the model in the search process.

[0171] The device for improving the long-range deep search reasoning capability of the large language model according to the embodiment of the application further enhances the intermediate step guiding ability of the model in the long-range deep search task by introducing the dynamic reward adjustment mechanism based on the knowledge graph entity matching degree in the reinforcement learning stage, effectively improves the accuracy and stability of the search path, and thus realizes better end-to-end reasoning performance in the complex multi-hop retrieval scene.

[0172] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0173] Furthermore, the terms "first", "second", "third", "fourth", "fifth" and "sixth" are used herein for descriptive purposes only and are not to be construed as indicating or implying relative importance or a significant nature of so described technical features. It is to be understood that a technical feature described with the "first", "second", "third", "fourth", "fifth" or "sixth" can implicitly or explicitly include at least one of the technical features described with the "first", "second", "third", "fourth", "fifth" or "sixth". In the description of the present application, the meaning of "a plurality" is at least two, for example, two, three, etc., unless otherwise specifically defined.

Claims

1. A method for improving the long-range deep search reasoning ability of large language models, characterized in that, include: Based on the knowledge graph, a semantically guided random walk is performed to generate structured data containing multi-hop entity paths, where each node carries attribute information, and key entity or time information in the path is obfuscated through an information masking strategy. The generated structured data is filtered in three layers of difficulty. First, a model without search capabilities is used to make a zero-search judgment to filter out simple questions. Second, a model with search capabilities is used to make multiple attempts to filter out medium-difficulty questions. Finally, questions with insufficient reliance on external retrieval are filtered based on the average number of search calls. A two-stage training strategy is adopted. In the first stage, a teacher model generates high-quality samples containing interaction trajectories of thinking, searching and observing, and supervises the fine-tuning of the samples to transfer the basic search strategy. In the second stage, the GRPO reinforcement learning algorithm is used in a real search environment to optimize the multi-round search and reasoning process of the model based on the normalized advantage function, and positive rewards are given only when the model completes normal interaction and outputs the correct answer.

2. The method as described in claim 1, characterized in that, The step of performing semantically guided random walks based on knowledge graphs to generate structured data containing multi-hop entity paths also includes: A large language model is used to evaluate the semantic relevance of candidate next-hop entities, and only entities with strong semantic connections to the current node are retained as path expansion nodes. The complexity of the generated problem can be controlled by adjusting the path length k of the random walk and the node out-degree threshold, where the default value of k is 5 and the default value of the out-degree threshold is ≥3.

3. The method as described in claim 1, characterized in that, The three-layer difficulty screening of the generated structured data also includes: The GPT-4o model without browsing capabilities is used to perform zero-search judgment on the question. If the model directly gives the correct answer without external search, the question is removed. Eight inference attempts with search were performed using the GLM-4-Z1-Rumination-32B model with browsing capabilities. The accuracy was calculated, and only questions with an accuracy between 0.2 and 0.8 were retained as medium-difficulty samples.

4. The method as described in claim 1, characterized in that, The two-stage training strategy also includes: During the supervised fine-tuning phase, the thinking-search-observation interaction trajectory generated by the teacher model was used as training samples, and only the trajectory with the correct final answer was retained for training the student model. During the reinforcement learning phase, the GRPO algorithm is adopted. By calculating the normalized advantage function and maximizing the objective function, the multi-round search and inference strategy of the model is optimized. The default value of the importance sampling cutoff width ε is 0.

2.

5. The method as described in claim 1, characterized in that, Also includes: The model's interaction trajectory during the reinforcement learning phase is dynamically adjusted in terms of rewards. In each round of interaction, if the search query generated by the model matches the path entity in the knowledge graph more than a preset threshold, a partial positive reward is given to enhance the model's ability to guide intermediate steps in the search process.

6. An apparatus for enhancing the long-range deep search reasoning capability of large language models, characterized in that, include: The knowledge graph guidance module is used to perform semantically guided random walks based on knowledge graphs, generating structured data containing multi-hop entity paths, where each node carries attribute information, and key entity or time information in the path is obfuscated through information masking strategies. The difficulty filtering module is used to perform three-level difficulty filtering on the generated structured data. First, it uses a model without search capabilities to perform zero-search judgment to filter simple questions. Second, it uses a model with search capabilities to perform multiple rounds of attempts to filter medium-difficulty questions. Finally, it filters questions that do not rely enough on external retrieval based on the average number of search calls. The two-stage training module is used to adopt a two-stage training strategy. In the first stage, a high-quality sample containing the interaction trajectory of thinking-searching-observation is generated by the teacher model, and the sample is supervised and fine-tuned to transfer the basic search strategy. In the second stage, the GRPO reinforcement learning algorithm is used in the real search environment to optimize the multi-round search and reasoning process of the model based on the normalized advantage function, and positive rewards are given only when the model completes normal interaction and outputs the correct answer. The dynamic reward adjustment module is used to dynamically adjust the rewards for the model's interaction trajectory during the reinforcement learning phase. In each round of interaction, if the matching degree between the search query generated by the model and the path entity in the knowledge graph exceeds a preset threshold, a partial positive reward is given to enhance the model's ability to guide intermediate steps in the search process.

7. The apparatus as claimed in claim 6, characterized in that, The knowledge graph guidance module is also used for: A large language model is used to evaluate the semantic relevance of candidate next-hop entities, and only entities with strong semantic connections to the current node are retained as path expansion nodes. The complexity of the generated problem can be controlled by adjusting the path length k of the random walk and the node out-degree threshold, where the default value of k is 5 and the default value of the out-degree threshold is ≥3.

8. The apparatus as claimed in claim 6, characterized in that, The difficulty filtering module is also used for: The GPT-4o model without browsing capabilities is used to perform zero-search judgment on the question. If the model directly gives the correct answer without external search, the question is removed. Eight inference attempts with search were performed using the GLM-4-Z1-Rumination-32B model with browsing capabilities. The accuracy was calculated, and only questions with an accuracy between 0.2 and 0.8 were retained as medium-difficulty samples.

9. The apparatus as claimed in claim 6, characterized in that, The two-stage training module is also used for: During the supervised fine-tuning phase, the "think-search-observe" interaction trajectory generated by the teacher model was used as training samples, and only the trajectory with the correct final answer was retained for training the student model. During the reinforcement learning phase, the GRPO algorithm is adopted. By calculating the normalized advantage function and maximizing the objective function, the multi-round search and inference strategy of the model is optimized. The default value of the importance sampling cutoff width ε is 0.

2.

10. The apparatus as claimed in claim 6, characterized in that, Also includes: The dynamic reward adjustment module is used to dynamically adjust the rewards for the model's interaction trajectory during the reinforcement learning phase. In each round of interaction, if the matching degree between the search query generated by the model and the path entity in the knowledge graph exceeds a preset threshold, a partial positive reward is given to enhance the model's ability to guide intermediate steps in the search process.

Images