Parallel query optimization method and device based on reinforcement learning, equipment and readable storage medium

Abstract

The invention provides a reinforcement learning-based parallel query optimization method, apparatus and device, a readable storage medium and a computer program product. The reinforcement learning-based parallel query optimization method comprises the following steps of: receiving a query request input by a user; generating the query request into a parallel search instruction comprising a plurality of sub-queries, wherein each sub-query is separated by a predefined separator and packaged in a single instruction block; analyzing the instruction block to extract all the sub-queries, and concurrently executing all the sub-queries to obtain a plurality of retrieval results; and aggregating the plurality of retrieval results, and inputting the aggregated retrieval results into a large language model to generate a final answer. The method and the device can realize concurrent execution of a plurality of independent sub-queries, are suitable for scenes such as intelligent questions and answers, knowledge retrieval, decision support and the like which require efficient utilization of external information, and improve user experience.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and natural language processing, and in particular to a parallel query optimization method, apparatus, device, readable storage medium, and computer program product based on reinforcement learning. Background Technology

[0002] With the widespread application of large language models in complex interactive tasks, reasoning-enhanced search agents based on large language models (LLMs) (such as Search-R1, ReAct, and Toolformer) have become the mainstream technical solution for expanding the knowledge boundaries of models. These solutions mainly achieve the solution of complex problems through the following technical paths:

[0003] First, the LLM is trained to master search timing and strategies through reinforcement learning fine-tuning (such as PPO and GRPO algorithms), enabling it to autonomously decide when and how to initiate search queries to acquire external knowledge. Second, a carefully designed prompting engineering template (such as a "think-search-observe" loop) guides the model through multi-step reasoning. Finally, a strict multi-round serial interaction architecture is established, requiring the model to generate only a single search query in each reasoning step and wait for the query result to be returned and analyzed before initiating subsequent queries.

[0004] However, this technology has significant limitations. In terms of efficiency, the serial processing architecture forces all queries to execute sequentially, failing to leverage parallelism even with logically independent, parallelizable subqueries (such as multi-entity attribute comparisons), leading to an unnecessary increase in LLM calls and accumulated search latency. Regarding cost control, the additional inference steps and API requests significantly increase cloud resource consumption. In terms of architectural design, existing systems lack the ability to identify the potential for query parallelization, resulting in wasted computing resources. More importantly, existing reinforcement learning reward mechanisms focus only on answer accuracy, lacking explicit optimization guidance for execution efficiency and query decomposition quality, preventing the model from learning efficient parallelization execution strategies.

[0005] These shortcomings severely restrict the performance and economic benefits of intelligent search systems in real-time interactive scenarios, and there is an urgent need for a new generation of query processing framework that can identify parallelization opportunities, support concurrent execution, and is efficiency optimization oriented. Summary of the Invention

[0006] This disclosure is made to solve the above-mentioned problems. Its purpose is to provide a parallel query optimization method, system, device and storage medium based on reinforcement learning. It designs a multi-dimensional and verifiable reward function to train LLM to autonomously identify the parallelization potential of queries and generate and execute multiple independent subqueries in a single inference step.

[0007] This disclosure provides a summary portion of the invention to briefly introduce the concepts, which will be described in detail in the detailed description portion that follows. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0008] To address the aforementioned technical problems, this disclosure provides a parallel query optimization method based on reinforcement learning, employing the following technical solution:

[0009] Receive query requests from users;

[0010] The query request is generated into a parallel search instruction containing multiple subqueries, wherein each subquery is separated by a predefined delimiter and encapsulated in a single instruction block;

[0011] The instruction block is parsed to extract all subqueries, and all subqueries are executed concurrently to obtain multiple search results;

[0012] The multiple search results are aggregated and input into a large language model to generate the final answer.

[0013] To address the aforementioned technical problems, this disclosure also provides a parallel query optimization apparatus based on reinforcement learning, used to implement the steps of the method described above, and employs the following technical solution, including:

[0014] The request receiving module is used to receive query requests input by the user.

[0015] The instruction generation module is used to generate the query request into a parallel search instruction containing multiple subqueries, wherein each subquery is separated by a predefined delimiter and encapsulated in a single instruction block;

[0016] The concurrent execution module is used to parse the instruction block to extract all subqueries and execute all subqueries concurrently to obtain multiple search results;

[0017] The answer generation module is used to aggregate the multiple search results and input them into a large language model to generate the final answer.

[0018] To address the aforementioned technical problems, this disclosure also provides a computer device that employs the following technical solution, including:

[0019] A memory, a processor, and a computer program stored on the memory, characterized in that the processor executes the computer program to implement the steps of the method as described in any of the preceding claims.

[0020] To address the aforementioned technical problems, this disclosure also provides a computer-readable storage medium, employing the technical solutions described below, including:

[0021] It stores a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps of the method as described in any of the preceding claims.

[0022] To address the aforementioned technical problems, this disclosure also provides a computer program product, including a computer program, employing the following technical solution:

[0023] When the computer program is executed by a processor, it implements the steps of the method as described in any of the preceding methods.

[0024] According to the technical solution disclosed in this disclosure, compared with the prior art, this disclosure significantly improves execution efficiency. By executing independent subqueries in parallel, it reduces the average number of LLM calls, greatly reducing inference latency and system response time; it reduces deployment and operation costs by reducing LLM inference steps and search API calls, directly reducing cloud computing resource consumption and API fees; it improves task accuracy, achieving higher accuracy while maintaining high efficiency; it has a flexible and versatile architecture, representing an architecture-level optimization that does not depend on a specific LLM or search engine and can be widely applied to various intelligent agent systems requiring information retrieval; and it has a clear training objective, transforming the engineering goal of "execution efficiency" into a learnable RL signal through a carefully designed reward function, enabling the model to stably and effectively learn parallelization strategies and improving user experience. Attached Figure Description

[0025] Figure 1 This is an exemplary system architecture diagram to which this disclosure can be applied;

[0026] Figure 2 This is a flowchart of an embodiment of the method according to the present disclosure;

[0027] Figure 3 This is a flowchart of an embodiment of the method according to the present disclosure;

[0028] Figure 4 This is a flowchart of a model training embodiment based on the method of this disclosure;

[0029] Figure 5 This is a schematic diagram of one embodiment of the apparatus according to the present disclosure;

[0030] Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of the present disclosure.

[0031] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale. Detailed Implementation

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this disclosure are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this disclosure are used to distinguish different objects and not to describe a particular order.

[0033] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this disclosure. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0034] To enable those skilled in the art to better understand the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0035] System Structure

[0036] First, the structure of a system according to an embodiment of this disclosure will be described. For example... Figure 1 As shown, system architecture 100 may include, for example, terminal devices 101, 102, 103, and 104, network 105, and server 106. Network 105 is used as a medium to provide a communication link between terminal devices 101, 102, 103, and 104 and server 106.

[0037] In this embodiment, the templated heterogeneous database ETL method runs on electronic devices (e.g., Figure 1The terminal devices 101, 102, 103, or 104 shown can transmit various types of information via network 105. Network 105 can include various connection types, such as wired, wireless communication links, or fiber optic cables. It should be noted that the aforementioned wireless connection methods can include, but are not limited to, 3G / 4G / 5G connections, Wi-Fi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB connections, local area networks (“LAN”), wide area networks (“WAN”), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad-hoc end-to-end networks), as well as other currently known or future-developed network connection methods. Network 105 can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium.

[0038] Users can interact with server 106 via network 105 using terminal devices 101, 102, 103, and 104 to receive or send messages. Various client applications can be installed on terminal devices 101, 102, 103, or 104, such as video streaming and playback applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platforms.

[0039] Terminal devices 101, 102, 103, or 104 may be various electronic devices with touch screens and / or web browsing capabilities, including but not limited to smartphones, tablets, e-book readers, MP3 (Motion Picture Experts Compression Standard Audio Layer 3) players, MP4 (Motion Picture Experts Compression Standard Audio Layer 4) players, head-mounted displays, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and mobile terminals such as digital TVs and desktop computers.

[0040] Server 106 may be, for example, a server that provides various services, such as a backend server that supports the pages displayed or the data transmitted on terminal devices 101, 102, 103 or 104.

[0041] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0042] Here, the terminal device may implement the methods of the embodiments disclosed herein, either independently or in conjunction with other electronic terminal devices, by running applications in various operating systems.

[0043] Parallel query optimization methods based on reinforcement learning

[0044] refer to Figure 2 The diagram illustrates a flowchart of an embodiment of the reinforcement learning-based parallel query optimization method according to this disclosure. The reinforcement learning-based parallel query optimization method includes the following steps:

[0045] S21, Receive the query request input by the user. For example... Figure 3 As shown, for example, a user enters a query request to "compare the payout ratios and waiting periods for minor illnesses of three critical illness insurance policies, A, B, and C."

[0046] S22 generates a parallel search instruction containing multiple subqueries, where each subquery is separated by a predefined delimiter and encapsulated in a single instruction block. For example... Figure 3 As shown, for example, LLM determines whether the decomposition can be done in parallel based on the learned strategy. If it can be done in parallel, it generates multiple subqueries separated by ##, such as the following output:

[0047] <think> I need to check the payout ratio and waiting period for minor illnesses for products A, B, and C respectively.< / think>

[0048] <search> Critical Illness Insurance Plan A: Payout Ratio and Waiting Period for Minor Illnesses ##Critical Illness Insurance Plan B: Payout Ratio and Waiting Period for Minor Illnesses ##Critical Illness Insurance Plan C: Payout Ratio and Waiting Period for Minor Illnesses< / search> .

[0049] If the LLM determines that parallel decomposition is not possible, it generates a single search query or directly provides an answer, and then performs a serial search or direct reasoning.

[0050] In one or more embodiments, the ## delimiter mechanism provides a simple and standardized interface that allows the LLM to clearly express its parallel intent, facilitating system parsing and execution. Of course, the delimiter here is not limited to ##; other delimiters can be used as needed.

[0051] S23, the instruction block is parsed to extract all subqueries, and the search engine is invoked concurrently to execute all subqueries concurrently to obtain multiple search results in parallel. In one or more embodiments, for example, the system parses three subqueries and simultaneously sends requests to an insurance product database or a search engine.

[0052] In one or more embodiments, concurrent execution is achieved by simultaneously initiating multiple independent retrieval requests to one or more knowledge sources.

[0053] S24, the system aggregates multiple search results and inputs them into a large language model (LLM). The LLM synthesizes the information, generates a final answer, and outputs it. In one or more embodiments, for example, the system returns search results for three products to the LLM. The LLM analyzes all information and generates a final comparison report.

[0054] <think> According to the search results, Option A offers the highest payout ratio but has the longest waiting period, Option B offers the most balanced compensation, and Option C has the shortest waiting period...< / think>

[0055] <answer> In summary, if you..., we recommend option B; if you..., we recommend option C.< / answer>

[0056] This disclosure employs a parallel query decomposition and execution mechanism. By introducing delimiters (e.g., ##) in the prompt template, it guides the LLM to decompose complex queries into multiple independent subqueries and encapsulate them in a single... <search>Within the tag. Then the system parses it. <search>The content within the tags is split into multiple subqueries by delimiters, and the search engine is called concurrently. All results are returned to the LLM for comprehensive inference. This fundamentally changes the serial architecture of the search agent, achieving true parallel search and significantly reducing the number of LLM calls and the overall response time.

[0057] In addition, the method disclosed herein has adaptive parallel and serial processing capabilities. During inference, the model can autonomously determine whether the current query is suitable for parallel processing or must be processed serially based on the learned patterns, and take corresponding strategies. This ensures the versatility of the method, not only improving the efficiency of parallel queries, but also outperforming the baseline model when processing multi-hop queries that must be processed serially, thus optimizing the overall search strategy.

[0058] In one or more embodiments, the method of this disclosure further includes, for example, model training of the method, including:

[0059] First, prepare a mixed dataset containing both parallelizable and non-parallelizable queries. For example... Figure 4 As shown, for example, when training begins, a batch of training problems are sampled, and LLM generates response trajectories (which may contain parallel search instructions).

[0060] Secondly, configure a reward function that includes at least one reward component and set the weight of the reward component; for example, the query decomposition reward weight is 0.15 (decomposition reward weight), and the execution efficiency reward or search count reward weight is 0.35, etc.

[0061] Finally, a reward function is used to train the model for reinforcement learning. The reward function includes at least one of the following:

[0062] The answer accuracy reward is used to assess the degree of matching between the final answer and the standard answer, and to assess whether the final answer is correct (exact string match). For example, if the final answer matches the standard answer string exactly, the reward is 1; otherwise, it is 0.

[0063] The query decomposition reward is used to optimize the decomposition strategy based on the semantic independence of subqueries. For example, if a query is parallelizable (e.g., multi-entity comparison) and the model decomposes it correctly, the reward is +0.15; if it is not parallelizable (e.g., multi-hop inference) but is incorrectly decomposed, the penalty is -0.15. In one or more embodiments, the query decomposition reward is calculated based on whether there is a lack of logical or informational dependencies between subqueries. A positive reward is given for correctly decomposing parallelizable queries, and a negative penalty is given for incorrectly decomposing non-parallelizable queries. This teaches the model to determine whether a problem is suitable for parallel processing, ensuring that the model only decomposes semantically independent queries and avoids information loss caused by incorrect parallelization. In one or more embodiments, for example, the dependencies between subqueries are also analyzed. By constructing a query semantic dependency graph, the logical relationships between entities are analyzed and a parallelization weight factor α (0≤α≤1) is calculated based on the dependency strength. Then, the parallel execution strategy is dynamically adjusted according to the α value. For example, if α>0.8, it is fully parallel; if -0.3<α≤0.8, it is partially parallel with cache sharing; and if α≤0.3, it is serially executed. The α value is incorporated into the reward function as an evaluation index of the query decomposition quality.

[0064] Execution efficiency is rewarded with a Search Count Reward, for example, to incentivize the completion of all subqueries in a single instruction, enabling parallelizable problems to retrieve all information in a "single search." Penalties are imposed if multiple searches are performed, such as reducing the time all subqueries need to be executed in a single search. <search>If the search is completed in a single parallel search instruction, the reward is +0.35, for example, the maximum reward value; if the search is divided into multiple rounds, the penalty is -0.35 for each additional round. In one or more embodiments, the execution efficiency reward is configured to grant a preset reward value, for example, the maximum reward value, when all parallelizable subqueries are completed in a single parallel search instruction, while the reward value is reduced by a preset proportion, for example, 10%, for each additional serial search round. This reward function directly drives the model to pursue the goal of "minimum number of search rounds" and is the core driving force for reducing latency and cost.

[0065] The Format Reward ensures that the output conforms to a preset structured format, such as following a predefined reasoning-search-observation format, or that the output strictly adheres to... <think> ... <search> ... <answer>If the format is correct, the reward is +0.1; otherwise, it is 0.

[0066] The reward function system disclosed above not only ensures the accuracy of the answer, but also explicitly teaches the model "when to decompose" and "how to execute efficiently", thereby internalizing parallelization capabilities during RL training.

[0067] In one or more embodiments, reinforcement learning training is implemented using a policy optimization algorithm, which enables the model to simultaneously learn answer generation and efficient policy execution through a reward function. For example, the GRPO algorithm is used to fine-tune the LLM under the guidance of prompt templates. The model attempts to generate responses containing parallel subqueries. <search>Actions, the system calculates the comprehensive reward according to the correctness of the decomposition, the rationality of the search times and the accuracy of the final answer, and updates the model parameters. As shown in Figure 4 The LLM determines whether the parallel search instruction contains a label, and if it does, extracts the subquery and performs a search concurrently, and inserts all search results into the track, and if it does not contain a label, directly evaluates the answer. According to the above results, the reward under the composite reward function system is calculated, that is, four reward components are calculated: answer correctness, decomposition rationality, search round efficiency, and format specification, the total reward = the sum of the four parts, and the LLM parameters are updated using the RL algorithm. Determine whether the updated LLM parameters are trained? If the training is completed, the parallel decomposition query model is obtained and applied.

[0068] The present disclosure is suitable for intelligent question answering, knowledge retrieval, decision support and other scenarios that require efficient use of external information, for example by training the LLM to autonomously identify parallelization structures in queries and concurrently execute multiple independent subqueries.

[0069] It should be understood that although each step in the flowchart of the accompanying drawings is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other orders. Moreover, at least some of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be alternately executed with at least part of other steps or sub-steps or stages of other steps.

[0070] Parallel query optimization device based on reinforcement learning

[0071] To implement the technical solutions in the embodiments of the present disclosure, one embodiment of the present disclosure provides a parallel query optimization device based on reinforcement learning, which is used to implement the above-mentioned parallel query optimization method based on reinforcement learning. The device can be applied to various electronic terminal devices, such as Figure 5 As shown in the figure, it includes a request receiving module 501, an instruction generating module 502, a concurrent execution module 503, and an answer generating module 504.

[0072] The request receiving module 501 is used to receive the query request input by the user. Here, the request receiving module 501 is used to implement each step and function in S21 of the above-mentioned method, which will not be described here.

[0073] The instruction generation module 502 is configured to generate the query request into a parallel search instruction containing multiple sub-queries, wherein each sub-query is separated by a predefined delimiter and encapsulated in a single instruction block. Here, the instruction generation module 502, for example, implements each step and function in S22 in the method, which will not be repeated here.

[0074] The concurrent execution module 503 is configured to parse the instruction block to extract all sub-queries, and concurrently execute all sub-queries to obtain multiple search results; here, the concurrent execution module 503, for example, implements each step and function in S23 in the method, which will not be repeated here.

[0075] The answer generation module 504 is configured to input the multiple search results after aggregation into a large language model to generate a final answer. Here, the answer generation module 504, for example, implements each step and function in S24 in the method, which will not be repeated here.

[0076] It should be understood that although each block in the block diagram of the accompanying drawings can represent a module, part of which contains one or more executable instructions for implementing the specified logic function, these modules are not necessarily executed sequentially in order. The modules and functional units in the device embodiments in the present disclosure can be integrated in one processing module, or each unit can be physically present separately, or two or more modules or functional units can be integrated in one module. The above integrated modules can be realized in the form of hardware or in the form of software functional modules. The integrated modules, if realized in the form of software functional modules and sold or used as independent products, can also be stored in a computer readable storage medium. The above-mentioned storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0077] Computer device

[0078] The following refers to Figure 6 , which shows a structural schematic diagram of a computer device or electronic device (for example Figure 1 terminal device or server in the system) 600 suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure can be various terminal devices in the above-mentioned system. The electronic device shown in the figure is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0079] As shown in Figure 6 , the electronic device 600 can include a processing device (for example, a central processor, a graphics processor, etc.) 601 for controlling the overall operation of the electronic device. The processing device can include one or more processors to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing device 601 can also include one or more modules for processing and interaction between other devices.

[0080] The storage 602 is used to store various types of data and can be a computer readable storage medium or a combination of the same, such as an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this disclosure, a computer readable storage medium can be any tangible medium that contains or stores a program used or work with or by an instruction execution system, apparatus, or device.

[0081] The sensor device 603 is used to sense information of a specified measured quantity and convert it into a usable output signal according to a certain rule, and can include one or more sensors. For example, it can include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor, etc. to detect changes in the opening / closing state, relative positioning, acceleration / deceleration, temperature, humidity, and light of the electronic device.

[0082] The processing device 601, the storage 602, and the sensor device 603 are connected to each other through the bus 604. The input / output (I / O) interface 605 is also connected to the bus 604.

[0083] The multimedia device 606 can include input devices such as a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc. to receive input signals from a user, and various input devices can cooperate with various sensors of the sensor device 603 to complete gesture operation input, image recognition input, distance detection input, etc. The multimedia device 606 can also include output devices such as a liquid crystal display (LCD), a speaker, a vibrator, etc.

[0084] The power device 607 is used to provide power to various devices in the electronic device and can include a power management system, one or more power sources, and components for distributing power to other devices.

[0085] The communication device 608 can allow the electronic device 600 to communicate with other devices wirelessly or through a wire to exchange data.

[0086] Each of the above devices can also be connected to the I / O interface 605 to implement applications of the electronic device 600.

[0087] Although the figures show electronic devices with various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or included alternatively.

[0088] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device. When the computer program is executed by a processing device, it performs the functions defined in the methods of embodiments of this disclosure.

[0089] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0090] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0091] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0092] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer via any type of network, or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0093] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0094] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0095] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0096] According to one or more embodiments of this disclosure, a parallel query optimization method based on reinforcement learning is provided, which adopts the following technical solution, including:

[0097] Receive query requests from users;

[0098] The query request is generated into a parallel search instruction containing multiple subqueries, wherein each subquery is separated by a predefined delimiter and encapsulated in a single instruction block;

[0099] The instruction block is parsed to extract all subqueries, and all subqueries are executed concurrently to obtain multiple search results;

[0100] The multiple search results are aggregated and input into a large language model to generate the final answer.

[0101] According to one or more embodiments of this disclosure, a parallel query optimization method based on reinforcement learning is provided, employing the technical solution described below, and further including model training for the method, including:

[0102] Prepare a mixed dataset containing both parallelizable and non-parallelizable queries;

[0103] Configure a reward function that includes at least one reward component and set the weight of the reward component;

[0104] The model is trained using the reward function, wherein the reward function includes at least one of the following:

[0105] Answer accuracy rewards are used to assess how well the final answer matches the standard answer;

[0106] The query decomposition reward is used to optimize the decomposition strategy based on the semantic independence of the subquery.

[0107] An execution efficiency reward is used to incentivize the completion of all subqueries in a single instruction.

[0108] Instruction format reward, used to ensure that the output conforms to the preset structured format.

[0109] According to one or more embodiments of this disclosure, a parallel query optimization method based on reinforcement learning is provided, which adopts the following technical solution, including:

[0110] The calculation of the query decomposition reward is based on whether there is a lack of logical or informational dependencies between the subqueries. Positive rewards are given for correct decompositions that can be queried in parallel, and negative penalties are given for incorrect decompositions that cannot be queried in parallel.

[0111] According to one or more embodiments of this disclosure, a parallel query optimization method based on reinforcement learning is provided, which adopts the following technical solution, including:

[0112] The execution efficiency reward configuration is as follows: when all parallelizable subqueries are completed in a single parallel search instruction, a preset reward value, such as the maximum reward value, is granted, while the reward value is reduced by a preset ratio for each additional serial search round.

[0113] According to one or more embodiments of this disclosure, a parallel query optimization method based on reinforcement learning is provided, which adopts the following technical solution, including:

[0114] The reinforcement learning training is implemented using a policy optimization algorithm, and the reward function enables the model to simultaneously learn answer generation and efficient policy execution.

[0115] According to one or more embodiments of this disclosure, a parallel query optimization method based on reinforcement learning is provided, which adopts the following technical solution, including: the concurrent execution is achieved by simultaneously initiating multiple independent retrieval requests to one or more knowledge sources.

[0116] According to one or more embodiments of this disclosure, a parallel query optimization apparatus based on reinforcement learning is provided for implementing the method described in any of the preceding claims, and employs the following technical solutions, including:

[0117] The request receiving module is used to receive query requests input by the user.

[0118] The instruction generation module is used to generate the query request into a parallel search instruction containing multiple subqueries, wherein each subquery is separated by a predefined delimiter and encapsulated in a single instruction block;

[0119] The concurrent execution module is used to parse the instruction block to extract all subqueries and execute all subqueries concurrently to obtain multiple search results;

[0120] The answer generation module is used to aggregate the multiple search results and input them into a large language model to generate the final answer.

[0121] According to one or more embodiments of this disclosure, a computer device is provided, employing the technical solutions described below, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method as described in any of the preceding claims.

[0122] According to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, which employs the technical solution described below. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the method as described in any of the preceding claims.

[0123] According to one or more embodiments of this disclosure, a computer program product is provided, including a computer program that employs the technical solutions described below, including the steps of implementing the method as described in any of the preceding claims when the computer program is executed by a processor.

[0124] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0125] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0126] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.< / search> < / answer> < / search> < / think> < / search> < / search> < / search>

Claims

1. A parallel query optimization method based on reinforcement learning, characterized in that, include: Receive query requests from users; The query request is generated into a parallel search instruction containing multiple subqueries, wherein each subquery is separated by a predefined delimiter and encapsulated in a single instruction block; The instruction block is parsed to extract all subqueries, and all subqueries are executed concurrently to obtain multiple search results; The multiple search results are aggregated and input into a large language model to generate the final answer.

2. The method as described in claim 1, characterized in that, It also includes model training for the method, including: Prepare a mixed dataset containing both parallelizable and non-parallelizable queries; Configure a reward function that includes at least one reward component and set the weight of the reward component; The model is trained using the reward function, wherein the reward function includes at least one of the following: Answer accuracy rewards are used to assess how well the final answer matches the standard answer; The query decomposition reward is used to optimize the decomposition strategy based on the semantic independence of the subquery. An execution efficiency reward is used to incentivize the completion of all subqueries in a single instruction. Instruction format reward, used to ensure that the output conforms to the preset structured format.

3. The method as described in claim 2, characterized in that, The calculation of the query decomposition reward is based on whether there is a lack of logical or informational dependencies between the subqueries. Positive rewards are given for correct decompositions that can be queried in parallel, and negative penalties are given for incorrect decompositions that cannot be queried in parallel.

4. The method as described in claim 2, characterized in that, The execution efficiency reward configuration is to grant a preset reward value when all parallelizable subqueries are completed in a single parallel search instruction, and to reduce the reward value by a preset ratio for each additional serial search round.

5. The method as described in claim 2, characterized in that, The reinforcement learning training is implemented using a policy optimization algorithm, and the reward function enables the model to simultaneously learn answer generation and efficient policy execution.

6. The method as described in claim 1, characterized in that, The concurrent execution is achieved by simultaneously initiating multiple independent retrieval requests to one or more knowledge sources.

7. A parallel query optimization apparatus based on reinforcement learning, used to implement the steps of the method as described in any one of claims 1-6, characterized in that, include: The request receiving module is used to receive query requests input by the user. The instruction generation module is used to generate the query request into a parallel search instruction containing multiple subqueries, wherein each subquery is separated by a predefined delimiter and encapsulated in a single instruction block; The concurrent execution module is used to parse the instruction block to extract all subqueries and execute all subqueries concurrently to obtain multiple search results; The answer generation module is used to aggregate the multiple search results and input them into a large language model to generate the final answer.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method as claimed in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-6.

Images