Complex query rewriting method based on reinforcement learning

Abstract

The application discloses a complex query rewriting method based on reinforcement learning. The method comprises the following steps: constructing a policy network, a reference model and a reward model; obtaining historical query texts and solution documents; inputting the query texts into the policy network to generate a plurality of reasoning contents, and outputting corresponding comprehensive rewards according to each reasoning content and the solution documents by the reward model; normalizing all the comprehensive rewards to obtain intra-group relative advantages; updating the parameters of the policy network according to the intra-group relative advantages, and simultaneously constraining the parameter update amplitude of the policy network through the reference model; repeatedly obtaining new intra-group relative advantages and updating the parameters of the policy network for a plurality of times until a trained policy network is obtained; obtaining a query text to be rewritten, inputting the query text to be rewritten into the trained policy network, and outputting a rewritten query of the query text to be rewritten. The method can reduce the cost of manual query optimization, and provide an efficient, interpretable and reliable query optimization scheme for an intelligent retrieval system.

Description

Technical Field

[0001] This invention belongs to the field of educational natural language processing technology, specifically involving a method for rewriting complex queries based on reinforcement learning. Background Technology

[0002] Information retrieval (IR) is a core technology for locating content relevant to a user's information needs from large-scale datasets (such as web pages, document repositories, knowledge bases, and various structured or unstructured corpora). It is widely used in search engines, intelligent question answering, and knowledge service systems. Typically, it matches user-input queries with document content based on sparse representation models (such as BM25), dense vector retrieval models (Dense Retriever), or neural ranking models, thereby returning relevant ranking results. Under ideal conditions where queries are clearly expressed and semantically complete, modern retrieval models have achieved good results in standard evaluation environments. However, in real-world applications, user queries are often arbitrary, semantically incomplete, or contain implicit intentions, resulting in significant discrepancies between the query format and the input assumptions of the retrieval system.

[0003] Due to the aforementioned biases, retrieval models often struggle to accurately identify users' true information needs in real-world environments. Some queries involve relationships between multiple facts, with the actual retrieval target relying on implicit reasoning processes, while the original expression doesn't provide complete clues. Other queries lack sufficient information and necessary limiting conditions, leading to an overly broad search scope. Still others indirectly express needs by describing phenomena or scenarios, with the core intent hidden within the natural language narrative. Existing retrieval models primarily rely on surface semantic matching or vector similarity calculations, lacking the ability to model complex semantic structures and implicit intents. This easily results in search results deviating from the actual needs, relevant documents being ranked low, or even being completely unsearched.

[0004] To alleviate these problems, existing technologies have attempted to employ query rewriting or query expansion methods, adjusting the original query format through keyword supplementation, synonym replacement, or template conversion. Some solutions also utilize large language models for natural language rewriting. However, existing methods mostly involve shallow semantic transformations, lacking systematic analysis of the deep semantics and reasoning structure of queries. Furthermore, the rewriting process is often not directly related to actual retrieval results, making it difficult to optimize for the input preferences of specific retrieval models. This results in a disconnect between the rewriting results and improved retrieval performance, making it difficult to consistently improve overall retrieval performance in complex query scenarios. Summary of the Invention

[0005] To address the problems existing in the background technology, this invention provides a complex query rewriting method based on reinforcement learning. This method solves the technical problems of insufficient query rewriting capability of existing models, lack of effective correlation between query rewriting and retrieval feedback, and limited performance improvement in complex query scenarios. The purpose of this invention is to generate rewritten queries that are semantically clear, have well-defined objectives, and conform to the processing characteristics of retrieval models by performing semantic analysis and structured reconstruction on the original query. Combined with a feedback optimization mechanism based on retrieval results, this allows the query rewriting strategy to adapt to different retrieval models, enabling retrieval models to better identify the document content corresponding to complex queries from the document library, thereby improving overall retrieval performance in complex query scenarios.

[0006] The technical solution adopted in this invention is: I. A complex query rewriting method based on reinforcement learning: S1. Construct a reinforcement learning model that includes a policy network, a reference model, and a reward model; the reference model is a copy of the policy network before training begins, and its parameters remain fixed during training.

[0007] S2, Retrieve historical query text and corresponding solution documentation .

[0008] S3, query text The same query text is generated by inputting it into the policy network. The corresponding several include intermediate inference representations and rewrite query Reasoning content The reward model is based on each inference content. Rewrite query and solution documents Matching relationships and reasoning content According to the format specifications, output the corresponding reasoning content. Rewrite query Comprehensive rewards .

[0009] S4. For all reasoning content Rewrite query Comprehensive rewards Perform normalization to obtain each rewrite query within the group relative advantage .

[0010] S5. Based on all rewritten queries Intra-group relative advantage Policy network parameters of reinforcement learning models Update the parameters of the policy network by referencing the model, and simultaneously constrain the parameter update magnitude of the policy network.

[0011] S6. Repeat steps S2-S5 several times, based on the new query text obtained each time. and corresponding solution documentation The policy network is updated until it meets the preset convergence condition or reaches the preset number of iterations, resulting in a trained policy network.

[0012] S7. Obtain the query text to be rewritten. The query text to be rewritten The input is fed into the trained policy network, which then outputs the query text to be rewritten according to the structured reasoning and analysis-rewriting policy. intermediate inference representation and rewrite query .

[0013] The reinforcement learning model uses the GRPO algorithm, the policy network uses a pre-trained large language model, and the reward model is used to score the output of the policy network based on the comprehensive reward policy.

[0014] The specific processing procedure of the policy network is as follows: query text The input is fed into the policy network, and structured reasoning and analysis-rewriting policy guides the policy network to generate several intermediate reasoning representations. and rewrite query Reasoning content .

[0015] Structured Reasoning and Analysis - Rewriting Policy Guides the Policy Network to Generate Each Including Intermediate Reasoning Representation and rewrite query Reasoning content The process is as follows: the guiding strategy network determines the target based on the query text. Perform semantic analysis to output intermediate inference representations with inference chains. Then guide the policy network based on the intermediate inference representation. Generate query text The corresponding rewrite query Thus, we obtain intermediate inference representations. and rewrite query Reasoning content .

[0016] The comprehensive reward acquisition strategy is implemented according to the following steps to acquire comprehensive rewards. : D1. Calculate each inference content generated by the policy network. Rewrite query in Match each document in the preset document library, sort the documents in the document library in descending order according to the matching value, and obtain the rewritten query. With solution document Matching value Solving the document Rank of documents in the document library after sorting in descending order And non-solution documents in the preset document library Other than rewrite query Highest matching value .

[0017] D2. According to ranking Get search ranking rewards ; regarding the content of the reasoning Formatting review results in a formatting reward. ; Based on rewrite query With solution document The matching value, non-resolved documents in the preset document library Other than rewrite query The highest matching value will receive a search discrimination reward. .

[0018] D3. Rewards for ranking searches Format rewards and search distinguishability reward The reasoning content is obtained by fusion. Rewrite query Comprehensive rewards.

[0019] The comprehensive reward The result is obtained by processing it using the following formula: ; ; ; ; in, As a comprehensive reward; and All are weighting coefficients; , and The content of the reasoning The search ranking bonus, format bonus, and search distinguishability bonus; To solve the document Rank of documents in the document library after sorting in descending order; and The solution documents are as follows With rewrite query The matching value, non-resolved documents in the preset document library Other than rewrite query The highest matching value; and This is the ranking reward coefficient; The decay parameter is a power function. For exponential decay parameters; To prevent smoothing constants with a denominator of zero.

[0020] Policy network parameters of reinforcement learning models Update using the following formula: ; in, and These are the policy network parameters before and after the update, respectively. The learning rate; and The policy network and the reference model are respectively used for a given query text. The generated rewrite query The probability distribution; To adjust the policy network parameters Find the gradient; For all rewritten queries A set; To rewrite the query The corresponding relative advantage within the group; Generate rewritten queries for the reference model The probability distribution; This represents the KL divergence between the probability distribution output by the current policy network and the probability distribution output by the reference model. is the KL divergence constraint coefficient.

[0021] II. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0022] 3. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0023] The beneficial effects of this invention are: This invention significantly improves the rewriting quality and retrieval performance of complex queries in information retrieval by combining structured reasoning with reinforcement learning. The structured reasoning and analysis-rewriting strategy performs phased semantic analysis and structured rewriting of the original query, accurately uncovering the query's implicit intent and potential reasoning relationships, generating rewritten queries that are semantically clear, goal-oriented, and conform to the processing characteristics of the retrieval model. The reward model combines matching relationships and format specifications for comprehensive evaluation, providing multi-dimensional optimization signals to the strategy network. By normalizing the calculation of intra-group relative advantages, the reward scale is balanced, ensuring the stability and fairness of the strategy. The strategy network is continuously updated and optimized based on intra-group relative advantages, gradually adapting to the preferences of different retrieval machines.

[0024] Through the synergistic effect of the above mechanisms, this invention can generate high-quality rewritten queries, improve the matching effect of complex queries in the target retrieval system, and enhance the model's ability to understand diverse, fuzzy, or implicit intent queries, thereby significantly improving overall retrieval performance in complex query scenarios. The method of this invention can reduce the cost of manual query optimization and provide an efficient, interpretable, and reliable query optimization solution for intelligent retrieval systems. Attached Figure Description

[0025] Figure 1 This is a flowchart of the method of the present invention.

[0026] Figure 2 This is an example diagram illustrating the application of the method of the present invention in Example 2.

[0027] Figure 3 This is a KL divergence loss diagram between the policy network and the reference model obtained from the biological dataset in Example 1.

[0028] Figure 4 Update the KL divergence plot for the policy obtained in the biological dataset in Example 1.

[0029] Figure 5 This is the policy gradient loss map obtained from the biological dataset in Example 1.

[0030] Figure 6 This is the average comprehensive reward curve obtained from the biological dataset in Example 1. Detailed Implementation

[0031] The present invention will now be described in more detail with reference to the accompanying drawings and embodiments. However, the present invention is not limited thereto. For those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention. Contents not described in detail in this specification are prior art known to those skilled in the art.

[0032] like Figure 1 As shown, the complex query rewriting method in this embodiment includes the following steps: S1. Construct a reinforcement learning model that includes a policy network and a reward model. In this implementation, the reinforcement learning model adopts the GRPO algorithm, the policy network adopts a pre-trained large language model, and the reward model is used to score the output of the policy network based on the comprehensive reward policy.

[0033] Specifically, the pre-trained large language model uses Qwen2.5-7B-Instruct or Qwen2.5-14B-Instruct.

[0034] S2. Retrieve complex query text from historical GitHub / Stack Overflow software development domains. and corresponding solution documentation (Query text) (The expected best search results).

[0035] In this embodiment: Query text The question is: "How should I add a library search path for a custom plugin? I'm running a gz-sim (Garden version) simulation, and the model works fine if I copy the .so file to any default search path... but I'd prefer to set an environment variable, like in Gazebo-Classic... I tried exporting GAZEBO_PLUGIN_PATH, but that doesn't seem to work in gz-sim garden. Is there another way to add it to the gz-sim library search path?"

[0036] Corresponding solution document For: Includes official documentation or community Q&A content on setting custom plugin paths in gz-sim Garden.

[0037] S3, query text The same query text is generated by inputting it into the policy network. The corresponding several include intermediate inference representations and rewrite query Reasoning content That is, to form a set of reasoning content The reward model is based on each inference content. Rewrite query and solution documents Matching relationships and reasoning content According to the format specifications, output the corresponding reasoning content. Rewrite query Comprehensive rewards .

[0038] The specific processing procedure of the policy network is as follows: The query text is... The input is fed into the policy network, and structured reasoning and analysis-rewriting policy guides the policy network to generate several intermediate reasoning representations. and rewrite query Reasoning content .

[0039] Structured Reasoning and Analysis - Rewriting Policy Guides the Policy Network to Generate Each Including Intermediate Reasoning Representation and rewrite query Reasoning content The process is as follows: the guiding strategy network determines the target based on the query text. Perform semantic analysis to output intermediate inference representations with inference chains. Then guide the policy network based on the intermediate inference representation. Generate query text The corresponding rewrite query Thus, we obtain intermediate inference representations. and rewrite query Reasoning content .

[0040] Specifically, the structured reasoning and analysis-rewriting strategy is essentially based on cue words, which are combined with the query text. They are input together into the policy network, thereby guiding the output of the policy network.

[0041] Structured Reasoning and Analysis - Rewriting the Reasoning Content Output of the Policy-Guided Policy Network It must conform to the following format: <think> Intermediate reasoning representation with reasoning chain .

[0042] < / think > < rewrite > Rewrite query .

[0043] < / rewrite > The above is the output reasoning content. Required formatting: <think> and< / think > The output between them is an intermediate inference representation with an inference chain. <rewrite> and< / rewrite > The output between them is based on the intermediate inference representation. Generated rewrite query .

[0044] In this embodiment, the generated reasoning content as follows: <think> 1. Users want to add custom plugin paths in gz-sim Garden.

[0045] 2. The user tried using the environment variable $GAZEBO_PLUGIN_PATH$, but this method was ineffective.

[0046] 3. Users hope to find an alternative method in gz-sim Garden to achieve the same functionality.

[0047] 4. The implicit intent of the query is "how to correctly configure the plugin path so that it can be recognized by the emulator".

[0048] < / think > < rewrite > In Gz-sim (Garden version), what environment variable or method can be used to add a custom plugin path to the plugin's library search path? < / rewrite > The comprehensive reward acquisition strategy is implemented according to the following steps to acquire comprehensive rewards. : D1. Use the target retrieval device to process each inference content generated by the policy network. Rewrite query in Match each document in the preset document library, sort the documents in the document library in descending order according to the matching value, and obtain the solution document. Rank of documents in the document library after sorting in descending order In this embodiment, the obtained solution document Rank of documents in the document library after sorting in descending order .

[0049] The target retrieval matching employs a retrieval method based on keyword matching, semantic vector matching, neural network semantic modeling, or a combination thereof. In this embodiment, semantic vector matching is used.

[0050] D2. According to ranking Get search ranking rewards ; regarding the content of the reasoning Formatting review results in a formatting reward. ; Based on rewrite query With solution document The matching value, non-resolved documents in the preset document library Other than rewrite query The highest matching value will receive a search discrimination reward. .

[0051] D3. Rewards for ranking searches Format rewards and search distinguishability reward The reasoning content is obtained by fusion. Rewrite query Comprehensive rewards.

[0052] Specifically, regarding the content of the reasoning Format review is equivalent to reviewing the reasoning content of the output. Determine if it conforms to the prescribed format. For example: if the reasoning content... It does not have <think>< / think > <rewrite> and< / rewrite > None of the tags are formatted correctly; if the reasoning content... Adding any extra label is incorrect; if there is an intermediate inference representation with an inference chain. Not in <think> and< / think > The relationship between them is incorrect; if the generated rewrite query is incorrect. Not in <rewrite> and< / rewrite > The relationship between them is incorrect; <think>< / think > <rewrite> and< / rewrite > The order is also incorrect.

[0053] Comprehensive Rewards The result is obtained by processing it using the following formula: ; ; ; ; in, As a comprehensive reward; and All are weighting coefficients; , and The content of the reasoning The search ranking bonus, format bonus, and search distinguishability bonus; To solve the document Rank of documents in the document library after sorting in descending order; and The solution documents are as follows With rewrite query The matching value, non-resolved documents in the preset document library Other than rewrite query The highest matching value; and This is the ranking reward coefficient; The decay parameter is a power function. This is the exponential decay parameter, used to control the decay rate of the ranking reward; To prevent smoothing constants with a denominator of zero.

[0054] In this embodiment, The values ​​are 0.9 and 0.1 respectively; The value is 0.27; The values ​​are 0.9, 0.1, and 0.5 respectively. The value is 5; Values .

[0055] exist Based on this, we can obtain Reasoning content Formatting rewards were awarded based on formatting review. =1 (correct format), according to and Calculate the retrieval discrimination reward =0.30 finally obtained 0.96.

[0056] This invention employs a comprehensive reward acquisition strategy to guide the output of the strategy network. This comprehensive reward integrates search ranking rewards, format rewards, and search discrimination rewards, using actual search performance as the optimization objective. This allows the strategy network to improve the ranking position of the target document while also considering the relevance differences between the target document and competing documents, and ensuring that the output content conforms to a preset format. This generates rewritten queries that better meet real search needs, enhancing the ability to rewrite complex search-based queries and improving overall search performance.

[0057] S4. For all reasoning content Rewrite query Comprehensive rewards Perform normalization to obtain each rewrite query Intra-group relative advantage .

[0058] Each rewrite query Intra-group relative advantage Obtained by processing using the following formula: ; in, To rewrite the query The relative advantage within the group; and Each of the total rewards The mean and standard deviation; This is to smooth out the terms and prevent the denominator from being zero.

[0059] By normalizing within groups, the model can balance the policy update signal under different query difficulties, thereby avoiding a few high-reward or low-reward queries dominating the training process and improving the stability and generalization ability of the query rewriting policy.

[0060] S5. Based on relative advantages within the group Policy network parameters of reinforcement learning models Update the model to allow it to learn and generate rewritten queries that align with the target searcher's preferences.

[0061] Policy network parameters of reinforcement learning models Update using the following formula: ; in, and These are the policy network parameters before and after the update, respectively. The learning rate; and The policy network and the reference model are respectively used for a given query text. The generated rewrite query The probability distribution; To adjust the policy network parameters Find the gradient; For all rewritten queries A set; To rewrite the query The corresponding relative advantage within the group; Generate rewritten queries for the reference model The probability distribution; This represents the KL divergence between the probability distribution output by the current policy network and the probability distribution output by the reference model. is the KL divergence constraint coefficient.

[0062] Specifically, through the relative advantages of the groups By applying weighted gradients, reinforcement learning strategy models can simultaneously consider different query difficulties and retrieval needs, thereby improving the overall effectiveness of query rewriting.

[0063] S6. Repeat steps S2-S5 several times, based on the new query text obtained each time. and corresponding solution documentation The policy network is updated until it meets the preset convergence condition or reaches the preset number of iterations, resulting in a trained policy network.

[0064] Specifically, the preset convergence condition is set as: the comprehensive reward in N consecutive training rounds. The average reward no longer increases significantly, meaning the difference between the average comprehensive rewards of two adjacent rounds is less than the preset convergence threshold; or the comprehensive reward of all reasoning content... The standard deviation is lower than the preset stability threshold, indicating that the quality of the inference content generated by the policy network tends to be stable.

[0065] In this embodiment, the convergence condition is set as follows: if the change in the average comprehensive reward over three consecutive training rounds is less than 0.01, the policy network is considered to have converged, and training is stopped. The preset number of iterations is set to 1000 rounds.

[0066] The complete output example of the above process is as follows: Original query (query text) "How should I customize plugins..."

[0067] Solve the document "Includes information about setting up custom settings in gz-sim Garden...".

[0068] Rewrite query "In Gz-sim (Garden version), you can use...".

[0069] Solve the document Rank in the document library after descending sort: 2.

[0070] Comprehensive Rewards : 0.83.

[0071] Iterative update notes: Based on relative strength within the group The strategy model is updated so that it gradually learns to generate query expressions that conform to the preferences of the target retrieval machine, while maintaining the interpretability of intermediate inference.

[0072] S7. Obtain the query text to be rewritten. The query text to be rewritten The input is fed into the trained policy network, which then outputs the query text to be rewritten according to the structured reasoning and analysis-rewriting policy. intermediate inference representation and rewrite query .

[0073] The query text to be rewritten in this embodiment for: How can I add a library search path for a custom plugin? I'm running a gz-sim (Garden version) simulation launched from PX4 and want to load a custom plugin into the model. Currently, if I copy the plugin's corresponding .so file to any default plugin search path, the model runs correctly, for example: / home / anton / .gz / sim / plugins / ; / usr / lib / x86_64-linux-gnu / gz-sim-7 / plugins / ; / home / anton / .ignition / gazebo / plugins / ; However, I would prefer to be able to set environment variables to tell Gazebo to search for plugins in other directories, just like Gazebo-Classic does.

[0074] I tried to execute it: export GAZEBO_PLUGIN_PATH="$PWD:$GAZEBO_PLUGIN_PATH"; However, this method seems to be ineffective in gz-sim Garden. Is there another way to add a library search path to gz-sim? The obtained intermediate inference representation and rewrite query as follows: <think> The user wants to resolve an issue where custom plugins cannot be loaded via environment variables in gz-sim (Garden version). The query includes contextual information such as the runtime environment, previous attempts (copying .so files, setting GAZEBO_PLUGIN_PATH), and the failure symptoms, but this is background information and not the ultimate goal. The user's true intention is to find the correct method for configuring the search path for custom plugin libraries in gz-simGarden, including supported environment variables, configuration methods, or officially recommended solutions.

[0075] < / think> <rewrite> This document explains how to configure a custom plugin library search path in gz-sim Garden, including the environment variables corresponding to the custom plugin loading path and the official configuration method.

[0076] <rewrite> Specifically, the query text to be rewritten Rewrite query This is the rewritten query text that we ultimately need. Rewrite the query. It can more accurately reflect the original query (query text to be rewritten) It can capture the implicit intent and explicit semantic features of documents and achieve higher retrieval performance in the document library.

[0077] To further demonstrate the beneficial effects of the method of the present invention: The retrieval performance of the method in this embodiment and the conventional method after query rewriting on the complex query dataset BRIGHT is compared (the index is nDCG@10 for both). Table 1 is shown below.

[0078] Table 1: Note: (1) Biology, Economics, Psychology, Code and Theoretical Question Answering are all subsets of the complex query dataset BRIGHT.

[0079] (2) nDCG (Normalized Discounted Cumulative Gain) is a core indicator used in the field of information retrieval to evaluate the quality of search result lists. It considers both relevance level (higher-relevant documents rank higher and score higher) and ranking position (the higher the result ranks, the greater its weight), and compares it with the ideal ranking result through normalization to finally obtain a score between 0 and 1. The higher the value, the better the quality of the search result list and the more the ranking meets the user's expectations.

[0080] (3) In this embodiment, nDCG@10, which is most frequently used in similar works, is selected. The top 10 results are considered. Not only is the relevance of the results considered, but also higher weight is given to the high-ranking results, so that the evaluation is closer to the actual search experience of users.

[0081] (4) In this embodiment, the method of the present invention significantly outperforms other conventional methods on all datasets. On the biological dataset, the nDCG@10 of the method of the present invention reaches 20.95, while the original query is only 3.50, an improvement of nearly 6 times; on the economics, psychology and theoretical question answering datasets, the method of the present invention achieves 15.47, 19.19 and 15.88 respectively, all of which are significantly better than CoT (11.92, 14.17, 12.25), Query2doc (9.00, 10.00, 9.07) and QA-Expand (7.92, 9.86, 8.21); on the code dataset, the method of the present invention reaches 24.71, which is also higher than other baseline methods.

[0082] (5) Overall, the experimental results show that the method of the present invention, by combining the ability of the reinforcement learning strategy model to rewrite the original query, enables the model to effectively capture the semantic information and potential intent in the query during the training process, and significantly improves the retrieval relevance of complex queries. It performs well in different fields, further proving that the method of the present invention has robustness and wide applicability in complex query rewriting tasks.

[0083] like Figure 3 , Figure 4 , Figure 5 and Figure 6 The figures show the KL divergence loss diagram between the policy network and the reference model, the policy update KL divergence diagram, the policy gradient loss diagram, and the average comprehensive reward curve obtained from the biological dataset in the complex query dataset. The KL divergence loss between the policy network and the reference model reflects the degree to which the policy network deviates from the initial model; the policy update KL divergence measures the magnitude of the update of the current policy compared to the old policy, avoiding excessive policy changes; the policy gradient loss represents the loss value of the policy network optimized based on the comprehensive reward, reflecting the training and convergence of the policy network; and the average comprehensive reward is used to evaluate the overall quality and retrieval performance of the rewritten queries generated by the current policy network.

[0084] This embodiment also provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0085] This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0086] Example 2: This embodiment is implemented following the same steps as Embodiment 1, and one of the query texts during the training of the strategy model... and the inference content output by the corresponding strategy model like Figure 2 As shown: Query text He asked, "Why do I only breathe through one nostril? I was sitting there with my hand next to my nose just now and realized that air was only coming out of my right nostril... Have I always only been breathing through my right nostril?"

[0087] The reasoning content output by the strategy model for: " <think>The user asked about... This is a normal physiological phenomenon called the nasal cycle, where airflow alternates between the two nostrils to optimize air conditioning and odor perception.

[0088] < / think> <rewrite>Why do humans primarily breathe through alternating nostrils on one side of the body? Please explain the mechanism of the nasal cycle, and the role and significance of airflow alternation.

[0089] < / rewrite> " The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.< / rewrite> < / rewrite>

Claims

1. A method for rewriting complex queries based on reinforcement learning, characterized in that, Includes the following steps: S1. Construct a reinforcement learning model including a policy network, a reference model, and a reward model; the reference model is a copy of the policy network before training begins, and its parameters remain fixed during training. S2, Retrieve historical query text and corresponding solution documentation ; S3, query text The same query text is generated by inputting it into the policy network. The corresponding several include intermediate inference representations and rewrite query Reasoning content The reward model is based on each inference content. Rewrite query and solution documents Matching relationships and reasoning content According to the format specifications, output the corresponding reasoning content. Rewrite query Comprehensive rewards ; S4. For all reasoning content Rewrite query Comprehensive rewards Perform normalization to obtain each rewrite query within the group relative advantage ; S5. Based on all rewritten queries within the group relative advantage Policy network parameters of reinforcement learning models Update the parameters, while simultaneously constraining the parameter update magnitude of the policy network through the reference model; S6. Repeat steps S2-S5 several times, based on the new query text obtained each time. and corresponding solution documentation The policy network is updated until it meets the preset convergence condition or reaches the preset number of iterations, resulting in a trained policy network. S7. Obtain the query text to be rewritten. The query text to be rewritten The input is fed into the trained policy network, which then outputs the query text to be rewritten according to the structured reasoning and analysis-rewriting policy. intermediate inference representation and rewrite query .

2. The complex query rewriting method according to claim 1, characterized in that: The reinforcement learning model uses the GRPO algorithm, the policy network uses a pre-trained large language model, and the reward model is used to score the output of the policy network based on the comprehensive reward policy.

3. The complex query rewriting method according to claim 1, characterized in that, The specific processing procedure of the policy network is as follows: query text The input is fed into the policy network, and structured reasoning and analysis-rewriting policy guides the policy network to generate several intermediate reasoning representations. and rewrite query Reasoning content ; Structured Reasoning and Analysis - Rewriting Policy Guides the Policy Network to Generate Each Including Intermediate Reasoning Representation and rewrite query Reasoning content The process is as follows: the guiding strategy network determines the target based on the query text. Perform semantic analysis to output intermediate inference representations with inference chains. Then guide the policy network based on the intermediate inference representation. Generate query text The corresponding rewrite query Thus, we obtain intermediate inference representations. and rewrite query Reasoning content .

4. The complex query rewriting method according to claim 3, characterized in that, The comprehensive reward acquisition strategy is implemented according to the following steps to acquire comprehensive rewards. : D1. Calculate each inference content generated by the policy network. Rewrite query in Match each document in the preset document library, sort the documents in the document library in descending order according to the matching value, and obtain the rewritten query. With solution document Matching value Solving the document Rank of documents in the document library after sorting in descending order And non-solution documents in the preset document library Other than rewrite query Highest matching value ; D2. According to ranking Get search ranking rewards ; regarding the content of the reasoning Formatting review results in a formatting reward. ; Based on rewrite query With solution document The matching value, non-resolved documents in the preset document library Other than rewrite query The highest matching value will receive a search discrimination reward. ; D3. Rewards for ranking searches Format rewards and search discrimination reward The reasoning content is obtained by fusion. Rewrite query Comprehensive rewards.

5. The complex query rewriting method according to claim 4, characterized in that: The comprehensive reward The result is obtained by processing it using the following formula: ； ； ； ； in, As a comprehensive reward; and All are weighting coefficients; , and The content of the reasoning The search ranking bonus, format bonus, and search distinguishability bonus; To solve the document Rank of documents in the document library after sorting in descending order; and The solution documents are as follows Rewrite query The matching value, non-resolved documents in the preset document library Other than rewrite query The highest matching value; and This is the ranking reward coefficient; The decay parameter is a power function. For exponential decay parameters; To prevent smoothing constants with denominators of zero.

6. The complex query rewriting method according to claim 1, characterized in that: Policy network parameters of reinforcement learning models Update using the following formula: ； in, and These are the policy network parameters before and after the update, respectively. The learning rate; and The policy network and the reference model are respectively used for a given query text. The generated rewrite query The probability distribution; To adjust the policy network parameters Find the gradient; For all rewritten queries A set; To rewrite the query The corresponding relative advantage within the group; Generate rewritten queries for the reference model The probability distribution; This represents the KL divergence between the probability distribution output by the current policy network and the probability distribution output by the reference model. is the KL divergence constraint coefficient.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images