Method, device, equipment, medium and product for task retrieval optimization of intelligent agent

Abstract

Embodiments of the present application provide an agent task retrieval optimization method, which can be applied to the field of artificial intelligence technology. The agent task retrieval optimization method comprises: generating a query rewriting task according to original problem data of preset training data; obtaining a target loss function based on a semi-rule reward function corresponding to the query rewriting task; optimizing a preset language model according to the target loss function to generate a target retrieval model; and performing task retrieval optimization on an updated input problem through the target retrieval model. Embodiments of the present application also provide an agent task retrieval optimization device, equipment, storage medium and program product.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to the field of data processing technology, and more specifically to a task retrieval optimization method, apparatus, device, medium, and product for intelligent agents. Background Technology

[0002] Artificial intelligence (AI) is a key driving force of the new round of technological revolution and industrial transformation. It is a new and crucial technological science that studies and develops theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence. As an important component of intelligent science, AI attempts to understand the essence of intelligence and produce a new type of intelligent machine (i.e., intelligent agent) that can react in a way similar to human intelligence.

[0003] Information retrieval (IR) is the process of finding information related to a specific query. Traditionally, information retrieval relies primarily on keyword matching and Boolean logic, but the introduction of artificial intelligence (AI) has brought new possibilities. Combining information retrieval with AI to achieve intelligent search is an important aspect of AI, potentially involving technologies such as natural language processing, machine learning, and deep learning to significantly improve the efficiency and accuracy of information retrieval. For information retrieval in the AI ​​field, existing retrieval systems are still mainly based on semantic or text matching methods, which cannot effectively solve higher-order retrieval problems. Although theoretically, query rewriting using language models can solve related problems, the lack of real-world rewriting standard data prevents direct supervised model training, hindering the model's ability to achieve efficient and accurate higher-order retrieval. Summary of the Invention

[0004] In view of at least one of the above problems, embodiments of the present invention aim to provide a task retrieval optimization method, apparatus, device, medium and product that enables intelligent agents to achieve more intelligent, efficient, accurate and high-order information retrieval, thereby providing a method that can efficiently train small-sized open-source language models and achieve or exceed the performance of large commercial language models in inferential query rewriting tasks.

[0005] One aspect of the present invention provides a task retrieval optimization method for an intelligent agent, comprising: generating a query rewriting task based on original question data from preset training data; obtaining a target loss function based on a semi-rule reward function corresponding to the query rewriting task; optimizing a preset language model based on the target loss function to generate a target retrieval model; and performing task retrieval optimization for updating the input question using the target retrieval model.

[0006] According to an embodiment of the present invention, the process of generating a query rewriting task based on the original question data of preset training data includes: obtaining the original question data of preset training data and the corresponding question-related documents; and generating a query rewriting task corresponding to the question-related documents based on a preset language model.

[0007] According to one embodiment of the present invention, before obtaining the target loss function based on the semi-rule reward function corresponding to the query rewriting task, the method further includes: generating a semi-rule reward function based on the query rewriting task.

[0008] According to an embodiment of the present invention, the process of generating a semi-rule reward function based on a query rewriting task includes: obtaining a first relevance function generated by the query rewriting task and a second relevance function generated by the original question data; and generating a semi-rule reward function based on the question-related documents corresponding to the original question data, the first relevance function, and the second relevance function.

[0009] According to one embodiment of the present invention, the process of obtaining the target loss function based on the semi-rule reward function corresponding to the query rewriting task includes: generating a reward value corresponding to the query rewriting task through the semi-rule reward function; and estimating the relative advantage information of the preset language model through the reward value.

[0010] According to one embodiment of the present invention, in obtaining the target loss function based on the semi-rule reward function corresponding to the query rewriting task, the method further includes: generating a target loss function that conforms to the preset group relative strategy optimization rules through relative advantage information.

[0011] Another aspect of the present invention provides a task retrieval optimization device for an intelligent agent, comprising a task generation module, a function acquisition module, a model generation module, and a retrieval optimization module. The task generation module generates a query rewriting task based on original problem data from preset training data; the function acquisition module obtains a target loss function based on the semi-rule reward function corresponding to the query rewriting task; the model generation module optimizes a preset language model based on the target loss function to generate a target retrieval model; and the retrieval optimization module performs task retrieval optimization for updating the input problem using the target retrieval model.

[0012] Another aspect of the present invention provides an electronic device including one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the above-described task retrieval optimization method for an intelligent agent.

[0013] Another aspect of the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the above-described task retrieval optimization method for an intelligent agent.

[0014] Another aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the above-described task retrieval optimization method for an intelligent agent.

[0015] The task retrieval optimization method for intelligent agents provided in this invention can at least partially solve the problem that existing retrieval systems in related technologies cannot better solve high-order retrieval problems, and therefore can achieve at least one of the following technical effects:

[0016] Compared to traditional information retrieval schemes, the agent-based task retrieval optimization method described in this embodiment of the invention enables the reasoning query rewriting task for high-order information retrieval to be trained using reinforcement learning methods based on relevant documents through a relevance semi-rule reward function based on mature semantic models. This solves the problem of the lack of real rewriting standard data for this task, which prevents direct supervised model training. Furthermore, using the aforementioned semi-rule reward method, reinforcement learning can be used to efficiently train parameters of small-sized open-source language models (e.g., Qwen2.5-1.5B) with low computational resource requirements. The performance of the trained model can approach that of large-sized commercial language models (e.g., GPT-4o), achieving high-efficiency information retrieval optimization based on small-sized language models. Moreover, through reasoning query rewriting, the high-order retrieval task based on causal reasoning is transformed into a traditional task based on text or semantic relevance. The rewritten query can be directly used for semantic or text retrieval, achieving better recall or ranking results than the original query, thus realizing the effect of converting a high-order retrieval task into a conventional retrieval task.

[0017] Therefore, the task retrieval optimization method of the intelligent agent described in the embodiments of the present invention can efficiently train a small-sized open-source language model and achieve or exceed the effect of a large commercial language model in the inferential query rewriting task, so as to ultimately achieve efficient and highly accurate high-order retrieval.

[0018] It should be understood that the above general description and the following specific embodiments are merely exemplary and illustrative, and do not limit the scope of the invention. Attached Figure Description

[0019] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0020] Figure 1 The illustration schematically depicts an application scenario of a task retrieval optimization method, apparatus, device, medium, and program product for intelligent agents according to embodiments of the present invention.

[0021] Figure 2A flowchart illustrating a task retrieval optimization method for an intelligent agent according to an embodiment of the present invention is shown schematically.

[0022] Figure 3 This schematically illustrates a flowchart of an application scenario for the task retrieval optimization method for intelligent agents according to an embodiment of the present invention.

[0023] Figure 4 A schematic diagram illustrating the structure of a task retrieval optimization apparatus for an intelligent agent according to an embodiment of the present invention is shown; and

[0024] Figure 5 A block diagram of an electronic device suitable for implementing a task retrieval optimization method for intelligent agents according to an embodiment of the present invention is shown schematically.

[0025] The accompanying drawings mentioned above are part of the specification of embodiments of the present invention, illustrating exemplary embodiments of the invention. The drawings, together with the description in the specification, serve to illustrate the principles of the embodiments of the present invention. It should be understood that the above general description with reference to the drawings and the following detailed description are merely exemplary and illustrative, and do not limit the scope of the invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the spirit of the contents disclosed in the present invention will be clearly explained below with reference to the accompanying drawings and detailed description. After understanding the embodiments of the present invention, any person skilled in the art can make changes and modifications based on the technology taught in the present invention without departing from the spirit and scope of the present invention.

[0027] The illustrative embodiments and descriptions of the present invention are used to explain the invention, but are not intended to limit the invention. Furthermore, elements / components using the same or similar reference numerals in the drawings and embodiments are used to represent the same or similar parts.

[0028] The terms "first," "second," etc., used in this invention do not specifically refer to any order or sequence, nor are they intended to limit the invention; they are merely used to distinguish elements or operations described using the same technical terms.

[0029] The directional terms used in this invention, such as up, down, left, right, front, or back, are merely for reference to the accompanying drawings. Therefore, the directional terms used are for illustrative purposes and not for limiting the scope of this invention.

[0030] The terms “comprising,” “including,” “having,” “containing,” etc., used in this invention are all open-ended terms, meaning that they include but are not limited to.

[0031] The term "and / or" as used in this invention includes any or all combinations of the things mentioned.

[0032] In this invention, "multiple" includes "two" and "more than two"; in this invention, "multiple groups" includes "two groups" and "more than two groups".

[0033] The terms "approximately," "about," etc., used in this invention are intended to modify any quantity or error that may vary slightly, but these slight variations or errors do not change the essence of the quantity or error. Generally, the range of slight variations or errors modified by such terms may be 20% in some embodiments, 10% in some embodiments, 5% in some embodiments, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0034] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0035] When expressions such as "at least one of A, B, and C" are used, they should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). When expressions such as "at least one of A, B, or C" are used, they should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). A person skilled in the art should also understand that any conjunction and / or phrase that substantially arbitrarily indicates two or more optional items, whether in the specification, claims, or drawings, should be understood to indicate the possibility of including one of these items, either of these items, or both items. For example, the phrase “A or B” should be understood as including the possibility of “A” or “B”, or “A and B”.

[0036] Relevance is a core issue in information retrieval. Given a query, the goal of a retrieval system is to retrieve documents relevant to that query from a large pool of documents, sort them by relevance, and return them to the user. Traditional information retrieval systems primarily focus on textual or semantic relevance. However, in recent years, with the development and application of new products such as language models and deep research (e.g., DeepResearch), information retrieval systems need to retrieve effective answers to given questions based on causal relevance—where there may not be a significant textual or semantic correlation between the question and the answer. Such problems can be called "higher-order retrieval problems," and existing retrieval systems, mainly based on semantic or text matching methods, cannot adequately solve these higher-order retrieval problems.

[0037] Specifically, level 1 retrieval tasks are typically implemented using keyword-based retrieval. For example, if the query is "What is the widest highway in North America?", keyword matching will yield a positive document: "The part of highway 401 that passes through Toronto is North America's busiest highway, and one of the widest." Furthermore, level 2 retrieval tasks are typically implemented using semantic-based retrieval. For example, if the query is "How human activities influence climatesystem?", semantic matching will yield a positive document: "Deforestation and urbanization result in increased emissions, urban heat island effects and changes in natural watercycle."

[0038] For higher-order retrieval problems (level 3), reasoning-based retrieval can be used. Retrieval problems can include sustainable living-post, code-issue, and math-question questions. For example, a sustainable living-post question might be "At home, after I water my plants, the water goes to plates below the pots. Can I rouse it for my plants next time?"; a code-issue question might be "I have this table and need to transform it to... I don't like UNPIVOT. Is there a better function in snowflake for this?"; and a math-question question might be "Let k = 2008^2 + 2^2008. What is the unit digit of k*2 + 2^k?". For the Sustainable living-post relevance "Risk of using recycled plant water", the corresponding positive document is "Soluble salts are commonly found in soils. When they build up, they destroy the soil structure and cause direct damage to roots". For the Code-issue relevance "Alternative function", the corresponding positive document is "The function FLATTEN flattens (explodes) compound values ​​into multiple rows... FLATTEN INPUT→ <expr>...”; For the Math-question relevance "Uses the same theorem", the corresponding positive sample document is "Determine all positive integers relatively prime to allthe terms of the infinite sequence a_n=2^n+3^n+6^n -1...". Here, the positive sample document can be understood as the correct answer to the retrieval question.

[0039] One feasible solution is to leverage the causal reasoning capabilities of language models to rewrite the input question using reasoning, generating a new, rewritten query. This new query is typically closer to relevant documents at the textual or semantic level. Therefore, using the rewritten query, relevant documents can be directly retrieved based on existing text or semantic retrieval systems, achieving better search results without modifying the entire retrieval system framework. However, this type of query rewriting method relies on language models with a large number of parameters (such as GPT-4o or Llama3-70B), resulting in high inference costs and making practical deployment difficult. A more ideal approach is to train a query rewriting model with a small number of parameters based on a small amount of labeled data. However, since manual data annotation can only label the relevance between the question and the document, it cannot directly generate a rewritten query. Therefore, traditional supervised learning methods cannot be directly applied to this task.

[0040] Therefore, this results in at least the following technical problems in task retrieval solutions for high-order retrieval problems:

[0041] (1) High reasoning cost: Existing causal reasoning query rewriting methods rely directly on high-performance language models, resulting in high reasoning cost and difficulty in improving rewriting effect.

[0042] (2) Inability to effectively utilize labeled data: For retrieval tasks, labeled data is in the form of relevance labels between the original query and the document (e.g., relevant or irrelevant). Such labeled data can be used to train discriminative models (e.g., recall or ranking models), but cannot be used to train generative models. Therefore, even if labeled data and training resources are available, the current method cannot be used for model training.

[0043] In view of at least one of the above problems, embodiments of the present invention aim to provide a task retrieval optimization method, apparatus, device, medium and product that enables intelligent agents to achieve more intelligent, efficient, accurate and high-order information retrieval, thereby providing a method that can efficiently train small-sized open-source language models and achieve or exceed the performance of large commercial language models in inferential query rewriting tasks.

[0044] One aspect of the present invention provides a task retrieval optimization method for an intelligent agent, comprising: generating a query rewriting task based on original question data from preset training data; obtaining a target loss function based on a semi-rule reward function corresponding to the query rewriting task; optimizing a preset language model based on the target loss function to generate a target retrieval model; and performing task retrieval optimization for updating the input question using the target retrieval model.

[0045] Figure 1 The illustration schematically depicts an application scenario of a task retrieval optimization method, apparatus, device, medium, and program product for intelligent agents according to embodiments of the present invention.

[0046] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, network 104, and server 105. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0047] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0048] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0049] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0050] It should be noted that the task retrieval optimization method for intelligent agents provided in this embodiment of the invention can generally be executed by server 105. Correspondingly, the task retrieval optimization device for intelligent agents provided in this embodiment of the invention can generally be located in server 105. The task retrieval optimization method for intelligent agents provided in this embodiment of the invention can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the task retrieval optimization device for intelligent agents provided in this embodiment of the invention can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0051] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0052] The following will be based on Figure 1 The described scene, through Figures 2-3 The task retrieval optimization method for the intelligent agent according to the disclosed embodiments is described in detail.

[0053] like Figure 2 As shown, one aspect of the present invention provides a task retrieval optimization method for an intelligent agent, which includes operations S201 to S204.

[0054] In operation S201, a query rewriting task is generated based on the original problem data of the preset training data.

[0055] In operation S202, the target loss function is obtained based on the semi-rule reward function corresponding to the query rewrite task;

[0056] In operation S203, the preset language model is optimized based on the target loss function to generate the target retrieval model; and

[0057] In operation S204, task retrieval optimization is performed on the updated input problem using the target retrieval model.

[0058] The intelligent agent can be the executing subject of the task retrieval optimization method described in the embodiments of the present invention, or it can be the executor controlled by the task retrieval optimization method. Specifically, it can be a humanoid intelligent robot or other AI device, which usually has its own actuators to complete specific action tasks.

[0059] For high-order retrieval tasks, it is necessary to construct a targeted retrieval model for the high-order retrieval problem to achieve the expected efficient and accurate high-order retrieval results. The preset training data is used to train this retrieval model, enabling it to possess the task retrieval optimization capabilities of this invention's embodiments. Specifically, this data can be obtained in advance through historical literature data or historical experimental data (e.g., data crawling and preprocessing), or it can be manually set in advance. The high-order retrieval task can be understood as an intelligent retrieval task targeting a high-order retrieval problem. This high-order retrieval task is specifically executed by the retrieval model generated after training and optimization using the preset training data, ultimately obtaining the corresponding retrieval results.

[0060] Pre-set training data typically includes original question data and question-related documents corresponding to the original question data. The original question data can be understood as multimodal data such as search text (e.g., words, phrases, sentences, and paragraphs), images, audio, and video, targeting the question-related documents. It can be understood as the retrieval question of a higher-order retrieval task, i.e., a question. Furthermore, the question-related documents can correspond to the standard retrieval answers to the original question data, i.e., answers. These question-related documents can also be represented by various multimodal data such as text (e.g., words, phrases, and paragraphs), images, audio, and video. It should be noted that the pre-set training data consisting of the original question data and question-related documents can be ordinary retrieval training data used in conventional information retrieval techniques.

[0061] Query rewriting tasks can be query tasks that rewrite the original question based on semantic reasoning or even image reasoning, i.e., Querry. Specifically, they can be obtained by rewriting the original question data from a pre-set training dataset. The rewriting process typically involves using a pre-set language model to extract the semantics of the original question data through semantic reasoning, thereby generating the query rewriting task. After query rewriting, the task corresponds to the relevant documents of the original question data; in other words, the query rewriting task generated from the original question data has a correspondence with its corresponding relevant documents.

[0062] A semi-rule reward function can be a reward function that predicts retrieval results for both the query rewriting task and the original question data from the pre-set training data, and is correlated with the relevance of the retrieval results. Specifically, this retrieval result prediction can be implemented using a mature semantic model. The target loss function can be a loss function optimized for training a pre-set language model.

[0063] The preset language model can be a small-sized open-source language model (such as Qwen2.5-1.5B) implemented through reinforcement learning. This preset language model requires relatively few computational resources, and its data processing performance is typically incomparable to language models such as GPT-4o before the optimization process described in step S203 of this embodiment. By optimizing this preset language model using the aforementioned target loss function, a target retrieval model can be finally generated. Specifically, this target retrieval model can be a retrieval model provided by the task retrieval optimization method in this embodiment to achieve higher-order retrieval, and can be obtained by optimizing the preset language model using the target loss function. Therefore, it is possible to efficiently train a small-sized open-source language model (such as Qwen2.5-1.5B) using ordinary retrieval question data, ensuring that the target retrieval model can achieve or exceed the retrieval performance of large commercial language models (such as GPT-4o or Llama3-70B) on inference query rewriting tasks.

[0064] The updated input question can be one or more original question data from the original question dataset of the pre-set training data (which can also serve as validation data for the target retrieval model), or it can be other retrieval question data different from the pre-set training data. By processing the updated input question through the target retrieval model, the retrieval capability of at least a large language model can be achieved on the inference query rewriting task based on the updated input question, obtaining the best retrieval results and achieving the goal of task retrieval optimization.

[0065] Therefore, the task retrieval optimization method for intelligent agents described in this embodiment of the invention can break free from the dependence on high-performance, high-cost language models (such as GPT-4o or Llama3-70B), resulting in lower inference costs, significantly improved query rewriting performance, and easier deployment in practical application scenarios. Furthermore, this target retrieval model can perform efficient model training based on query rewriting of the original question data, realizing a reinforcement learning process based on semi-rule rewards, thereby solving the problem in traditional technical solutions where labeled data cannot be used for model training in inference-based rewriting tasks. Finally, it can efficiently train a small-sized open-source language model, achieving or exceeding the retrieval performance of large commercial language models in inference-based query rewriting tasks, ultimately achieving efficient and highly accurate high-order retrieval and completing the optimization of the high-order retrieval process.

[0066] To enable those skilled in the art to have a clearer understanding of the task retrieval optimization method for intelligent agents described in the embodiments of the present invention, the following further provisions are provided. Figures 3-4 Explanation.

[0067] like Figure 2 and Figure 3 As shown, according to an embodiment of the present invention, in operation S201, the query rewriting task is generated based on the original problem data of the preset training data, including:

[0068] Obtain the original problem data and corresponding problem-related documents for the preset training data;

[0069] Generate query rewriting tasks based on a preset language model and corresponding documents related to the question.

[0070] like Figure 3 As shown, for a high-order retrieval task, there can typically be a set of pre-set training data in the form of "original question data and corresponding question-related documents". Each corresponding "original question data 311 and corresponding question-related document 312" can be used as a pre-set training data 301. The original question data 311 can be text, image, audio, or even video data, and the corresponding question-related document 312 can also be text, image, audio, or even video data.

[0071] The original question data 311 can be a retrieval question (i.e., Question), while the related documents 312 can be the standard answers (i.e., Answers) corresponding to the original question data, specifically represented as follows: ,in It is a positive integer greater than or equal to 1. For example, for a level 1 routine retrieval task, the original question data could be "What is the widest highway in North America?", and the corresponding related document could be "The part of highway 401 that passes through Toronto is North America's busiest highway, and one of the widest". For a level 3 advanced retrieval task, the corresponding original question data and related documents can be designed based on the aforementioned Sustainable living-post, Code-issue, and Math-question, which will not be elaborated on in detail.

[0072] A pre-defined language model can be used as the language model for query rewriting of the original question data (e.g., Qwen2.5-1.5B). Specifically, this query rewriting task can be represented by the following formula 1:

[0073]

[0074] Where query is the rewritten query, rewrite task 302 (e.g., ...). Figure 3 As shown), LLM is the preset language model used for query rewriting, and question is the original question data 311 (as shown). Figure 3 (As shown).

[0075] After rewriting using the aforementioned pre-defined language model (LLM), a mapping relationship can be established between the query rewriting task and the corresponding document (doc). This allows the query rewriting task to directly retrieve the corresponding document through an existing text or semantic retrieval / ranking system. Furthermore, the recall effect is significantly better than the retrieval method that directly uses the original question data for recall.

[0076] Furthermore, based on the above query rewriting task generation process, for each batch of preset training data, the above-mentioned preset language model to be trained is used to generate multiple rounds of data on the original question data in the input preset training data, which can repeatedly generate multiple different query rewriting tasks.

[0077] Therefore, by pre-setting a language model, efficient query rewriting for routine original problem data can be achieved, enabling the generation of higher-order retrieval tasks. Specifically, through inferential query rewriting, higher-order retrieval tasks based on causal reasoning are transformed into traditional tasks based on text or semantic relevance. The rewritten queries can be directly used for semantic or text retrieval, achieving recall or ranking effects superior to the original queries.

[0078] like Figure 2 and Figure 3 As shown, according to an embodiment of the present invention, before operating S202 to obtain the target loss function based on the semi-rule reward function corresponding to the query rewriting task, the method further includes:

[0079] The task is rewritten based on the query to generate a semi-rule reward function.

[0080] In this embodiment of the invention, the semi-rule reward can be understood as the calculation of the relevance score for the query rewriting task in the reward function. Specifically, it can rely on a relevance model for implementation and is not purely a rule-based reward function. Based on the aforementioned semi-rule reward function, a small-sized pre-defined open-source language model can be trained using reinforcement learning methods to obtain a high-performance inferential query rewriting model, i.e., a target retrieval model.

[0081] like Figure 2 and Figure 3 As shown, according to an embodiment of the present invention, the process of generating a semi-rule reward function based on a query rewrite task includes:

[0082] Obtain the first relevance function generated by the query rewrite task and the second relevance function generated from the original problem data;

[0083] A semi-rule reward function is generated based on the relevant documents corresponding to the original problem data, the first correlation function, and the second correlation function.

[0084] In one embodiment of the present invention, each input original problem data can be processed using the query rewriting method described in Formula 1 above to generate a corresponding query rewriting task. For example... Figure 3 As shown, further, for the original problem data 311, the corresponding second relevance function 304 can be obtained through a mature semantic model (such as the BGE series model) or a heuristic relevance algorithm (such as BM25); correspondingly, for the query rewriting task 302, the corresponding first relevance function 303 can be generated through the same semantic model or heuristic algorithm.

[0085] Therefore, this involves combining the collection of all positive sample documents related to the query rewriting task that cover the real user intent. For each input of preset training data, the corresponding semi-rule reward function 305 (e.g., ...) can be applied to the original problem data. Figure 3 (As shown) can be expressed as Formula 2 below:

[0086]

[0087] in, This can be understood as a reward score. It can be a collection of all positive sample documents that cover the real user intent corresponding to the query rewriting task query. This can be a relevance calculation function. This relevance calculation function can be implemented using a mature semantic model (such as the BGE series models) or a heuristic relevance algorithm (such as BM25). Therefore, a semi-rule reward construction method based on semantic relevance can be implemented for reinforcement learning training of language models, injecting positive sample information into the model using an unsupervised learning method.

[0088] The calculation logic of the semi-rule reward function in Formula 2 above can be expressed as: the query rewriting task generated through inferential rewriting. At the textual and semantic levels, it should possess more prominent intent information, which is more easily identified by relevance models (relevance calculation functions). Capture – For each positive sample, relevant documentation Query rewrite task The relevance score should be as high as possible compared to the original problem data. Therefore, for the relevance score of documents related to the positive sample problem, the query rewriting task... Compared to the original problem data The increase in value is the result of this query rewrite task. The reward score obtained. Specifically, the correlation score can be between 0 and 1.

[0089] Here, "semi-rule-based reward" refers to the relevance score calculation in the reward function of Formula 2 above, which can rely on another relevance model and is not purely rule-based. It should be noted that, compared to the process reward model commonly used in language model reinforcement learning training, the mature retrieval relevance model used in this embodiment of the invention also possesses the advantages of low computational cost and good robustness inherent in rule-based reward functions. Furthermore, the rewritten results (i.e., query rewriting tasks) used to calculate the reward are... The reward function is the final output of the model, without involving intermediate processes (such as problem-solving steps in mathematical problems), and is immune to reward hacking, just like a regular reward function. Therefore, it can be defined here as a "semi-regular reward". To facilitate computation and improve robustness, a semantic-based relevance model can be introduced. If the calculation of the relevance score relies on a heuristic algorithm (such as BM25), then the reward function can also be a fully rule-based reward function.

[0090] Therefore, by constructing a semi-rule reward method based on semantic relevance (i.e., the relevance semi-rule reward function of mature semantic models), the reasoning query rewriting task for high-order information retrieval can be trained on the basis of relevant documents through reinforcement learning. This can solve the problem that the task lacks real rewriting labeled data and cannot be directly trained in a supervised manner.

[0091] like Figure 2 and Figure 3 As shown, according to an embodiment of the present invention, in operation S202, the target loss function is obtained based on the semi-rule reward function corresponding to the query rewriting task, including:

[0092] The reward value for the query rewrite task is generated using a semi-rule reward function.

[0093] The relative advantage information of the pre-set language model is estimated by the reward value.

[0094] In one embodiment of the present invention, the semi-rule reward function provided by Formula 2 above can be used to calculate the reward value for each query rewriting task corresponding to the preset training data. The relative advantage information of the preset language model in calculating the retrieval action corresponding to the query rewriting task can be estimated by using the reward values ​​of all query rewriting tasks. Here, the relative advantage information can be understood as the reward score corresponding to the semi-reward rule function above.

[0095] like Figure 2 and Figure 3 As shown, according to an embodiment of the present invention, in operation S202, the target loss function is obtained based on the semi-rule reward function corresponding to the query rewriting task, and the following further steps are taken:

[0096] A target loss function that conforms to the preset relative strategy optimization rules is generated using relative advantage information.

[0097] Furthermore, such as Figure 3 As shown, the target loss function 306 based on the preset group relative strategy optimization rules can be calculated through the above relative advantage information, thereby optimizing the preset language model, and finally generating the target retrieval model through the target loss function.

[0098] Specifically, the preset group relative policy optimization rule can be a reinforcement learning algorithm rule based on Group Relative Policy Optimization (GRPO), which can be used to enhance the reasoning ability of the preset language model.

[0099] Therefore, by using reinforcement learning methods based on preset group relative policy optimization rules and semi-rule reward functions, a preset language model used for query rewriting can be trained and optimized to generate a target retrieval model.

[0100] This further ensures the implementation of a semi-rule-based reward construction method based on semantic relevance, which is used to train the language model for reinforcement learning, injecting positive sample information into the model in an unsupervised learning manner.

[0101] In summary, based on the aforementioned semi-rule reward function, training a small-sized open-source language model using a GRPO-based reinforcement learning method yields a high-performance target model for inferential query rewriting. Specifically, this semi-rule reward function enables efficient parameter training of small-sized open-source language models (e.g., Qwen2.5-1.5B) through reinforcement learning, requiring fewer computational resources, and achieving performance close to that of large-sized commercial language models (e.g., GPT-4o).

[0102] Compared to traditional information retrieval schemes, the agent-based task retrieval optimization method described in this embodiment of the invention enables the reasoning query rewriting task for high-order information retrieval to be trained using reinforcement learning methods based on relevant documents through a relevance semi-rule reward function based on mature semantic models. This solves the problem of the lack of real rewriting standard data for this task, which prevents direct supervised model training. Furthermore, using the aforementioned semi-rule reward method, reinforcement learning can be used to efficiently train parameters of small-sized open-source language models (e.g., Qwen2.5-1.5B) with low computational resource requirements. The performance of the trained model can approach that of large-sized commercial language models (e.g., GPT-4o), achieving high-efficiency information retrieval optimization based on small-sized language models. Moreover, through reasoning query rewriting, the high-order retrieval task based on causal reasoning is transformed into a traditional task based on text or semantic relevance. The rewritten query can be directly used for semantic or text retrieval, achieving better recall or ranking results than the original query, thus realizing the effect of converting a high-order retrieval task into a conventional retrieval task.

[0103] Therefore, the task retrieval optimization method of the intelligent agent described in the embodiments of the present invention can efficiently train a small-sized open-source language model and achieve or exceed the effect of a large commercial language model in the inferential query rewriting task, so as to ultimately achieve efficient and highly accurate high-order retrieval.

[0104] In the above-described intelligent agent task retrieval optimization method of this invention, the core technical points lie in the construction and application of the semi-rule reward function and the model training process for inferential retrieval. Specifically, a semi-rule reward construction method based on semantic relevance is used to train the language model through reinforcement learning, injecting positive sample information into the model using an unsupervised learning method. Furthermore, a model training process that enables the model to possess inferential query rewriting capabilities is implemented through reinforcement learning. Therefore, it is possible to efficiently train a small-sized open-source language model, achieving or exceeding the performance of large commercial language models in inferential query rewriting tasks.

[0105] Based on the above-described task retrieval optimization method for intelligent agents, this invention also provides a task retrieval optimization device for intelligent agents. The following will be combined with... Figure 4 The device 400 is described in detail.

[0106] Figure 4 The diagram illustrates the structure of a task retrieval optimization device 400 for an intelligent agent according to an embodiment of the present invention.

[0107] like Figure 4 As shown, the task retrieval optimization device 400 for intelligent agents in this embodiment includes a task generation module 410, a function acquisition module 420, a model generation module 430, and a retrieval optimization module 440.

[0108] The task generation module 410 is used to generate query rewriting tasks based on the original problem data of the preset training data. In one embodiment, the task generation module 410 can be used to perform the operation S201 described above, which will not be repeated here.

[0109] The function acquisition module 420 is used to obtain the target loss function based on the semi-rule reward function corresponding to the query rewrite task. In one embodiment, the function acquisition module 420 can be used to perform the operation S202 described above, which will not be repeated here.

[0110] The model generation module 430 is used to optimize a preset language model based on a target loss function to generate a target retrieval model. In one embodiment, the model generation module 430 can be used to perform the operation S203 described above, which will not be repeated here.

[0111] The retrieval optimization module 440 is used to perform task retrieval optimization for updating the input problem using the target retrieval model. In one embodiment, the retrieval optimization module 440 can be used to perform the operation S204 described above, which will not be repeated here.

[0112] According to embodiments of the present invention, any plurality of modules among the task generation module 410, function acquisition module 420, model generation module 430, and retrieval optimization module 440 may be merged into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the task generation module 410, function acquisition module 420, model generation module 430, and retrieval optimization module 440 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in hardware or firmware, or in any one of software, hardware, and firmware implementations, or in a suitable combination of any of these. Alternatively, at least one of the task generation module 410, function acquisition module 420, model generation module 430, and retrieval optimization module 440 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0113] Figure 5 A block diagram of an electronic device suitable for implementing a task retrieval optimization method for intelligent agents according to an embodiment of the present invention is shown schematically.

[0114] The electronic device provided in the embodiments of the present invention includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the task retrieval optimization method of the intelligent agent.

[0115] like Figure 5 As shown, an electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0116] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 502 and / or RAM 503. It should be noted that the programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0117] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0118] The present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the task retrieval optimization method of the aforementioned intelligent agent.

[0119] The computer-readable storage medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0120] According to embodiments of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, the computer-readable storage medium may include ROM 502 and / or RAM 503 and / or one or more memories other than ROM 502 and RAM 503 described above.

[0121] Embodiments of the present invention also include a computer program product comprising a computer program that, when executed by a processor, implements the above-described task retrieval optimization method for intelligent agents.

[0122] The computer program includes program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the methods provided in the embodiments of the present invention.

[0123] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0124] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0125] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0126] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0128] Furthermore, all actions involving the acquisition of information, signals, or data in this invention are carried out in compliance with the relevant data protection laws, regulations, and policies of the country where the invention is located, and with the authorization granted by the owner of the corresponding device.

[0129] Those skilled in the art will understand that the features described in the various embodiments and / or claims of the present invention can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention can be combined or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0130] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of the invention is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.< / expr>

Claims

1. A method for task retrieval optimization of an agent, characterized in that, The method comprises: generating a query rewriting task according to original question data of preset training data; wherein the preset training data comprises multimodal original question data and question-related documents corresponding to the original question data; the question-related documents correspond to standard search answers of the original question data; the query rewriting task is a query task after rewriting the original question based on semantic reasoning or even image reasoning, and has a corresponding relationship with the question-related documents corresponding to the original question data; obtain a first correlation function generated by the query rewriting task and a second correlation function generated by the original question data; generate a semi-regular reward function based on a question-related document corresponding to the original question data, the first correlation function and the second correlation function, the semi-regular reward function is expressed as follows: ， wherein, for reflecting the reward score; is a query rewriting task generated by the inference rewriting; is original question data; is a positive sample question-related document corresponding to the original question data; is a set of positive sample documents covering all real user intents corresponding to the query rewriting task; is a relevance calculation function, respectively used to define a first relevance function based on the query rewriting task and a second relevance function based on the original question data ; wherein, for the relevance score of the positive sample question-related document, the increase value of the query rewriting task relative to the original question data is the reward score of the query rewriting task; obtaining a target loss function conforming to a preset group relative strategy optimization rule based on a semi-rule reward function corresponding to the query rewriting task; wherein the preset group relative strategy optimization rule is a reinforcement learning algorithm rule based on group relative strategy optimization, and is used to enhance the reasoning ability of a preset language model; optimizing the preset language model according to the target loss function to generate a target search model; and performing task search optimization on an updated input question through the target search model.

2. The method of claim 1, wherein, In the step of generating a query rewriting task according to original question data of preset training data, the method comprises: obtaining original question data of the preset training data and corresponding question-related documents; generating a query rewriting task corresponding to the question-related documents according to the preset language model.

3. The method of claim 1, wherein, In the step of obtaining a target loss function based on a semi-rule reward function corresponding to the query rewriting task, the method comprises: generating a reward value corresponding to the query rewriting task through the semi-rule reward function; estimating relative advantage information of the preset language model through the reward value.

4. The method of claim 3, wherein, In the step of obtaining a target loss function based on a semi-rule reward function corresponding to the query rewriting task, the method further comprises: generating a target loss function conforming to a preset group relative strategy optimization rule through the relative advantage information.

5. An apparatus for task retrieval optimization of an agent for implementing the method of any one of claims 1-4, characterized in that, The method comprises: a task generation module configured to generate a query rewriting task according to original question data of preset training data; a function obtaining module configured to obtain a target loss function based on a semi-rule reward function corresponding to the query rewriting task; a model generation module configured to optimize a preset language model according to the target loss function to generate a target search model; and a search optimization module configured to perform task search optimization on an updated input question through the target search model. 6.An electronic device, comprising: one or more processors; memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, enable the one or more processors to perform the method of any one of claims 1-4. 7.A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1-4. 8.A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-4.

Images