Data processing method of large language model, training method and device of decision network, equipment, storage medium and program product

Abstract

This application provides a data processing method for a large language model, a training method for a decision network, an apparatus, a device, a storage medium, and a program product. The method includes: constructing initial context data based on the input information to be responded to by the large language model; performing multiple iterative processing, each iteration including: encoding state features into the context data to be processed to obtain state features; performing feature mapping on the state features to obtain evaluation values ​​of multiple types of candidate actions in the action space, and determining the target action from the multiple types of candidate actions based on the evaluation values; stopping the iterative processing when the target action completes the generation of response data, and extracting the response data generated by the input information from the context data to be processed. This application enables adaptive selection of the target action through iterative evaluation of the current state, improving the flexibility of data processing and the accuracy of response.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method for a large language model, a training method for a decision network, an apparatus, a device, a storage medium, and a program product. Background Technology

[0002] With the development of artificial intelligence technology, large language models are widely used in various intelligent question answering and information processing scenarios. In related technologies, to generate responses based on input information using large language models, Retrieval-Augmented Generation (RAG) technology is employed. This involves retrieving content from an external knowledge base based on the input information to construct a contextual environment, and then interacting with the large language model to obtain the response result. However, this context construction and interaction mode, based on a fixed retrieval strategy, makes it difficult to flexibly adjust the processing flow according to the complexity of the input information or the generation status, thus affecting the quality of the generated response data. Summary of the Invention

[0003] This application provides a data processing method for a large language model, a training method for a decision network, an apparatus, a device, a storage medium, and a program product, which can adaptively select target actions by iteratively evaluating the current state, thereby improving the flexibility and response accuracy of data processing.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a data processing method for a large language model, the method comprising: Initialized context data is constructed based on the input information to be responded to from the large language model; Multiple iterations are performed, wherein in the first iteration, the initialized context data is used as the context data to be processed, and in subsequent iterations, the updated context data based on the previous iteration is used as the context data to be processed; each iteration includes: The context data to be processed is encoded with state features to obtain state features; the state features are then mapped to obtain evaluation values ​​of multiple types of candidate actions in the action space; and the target action is determined from the multiple types of candidate actions based on the evaluation values. If the target action is to complete the generation of response data, the iterative process is stopped, and the response data of the input information generated by the large language model is extracted from the context data to be processed.

[0005] This application provides a method for training a decision network, the method comprising: Initialized context data samples are constructed based on the input information samples to be responded to from the large language model; Multiple iterative processes are performed. In the first iteration, the initialized context data sample is used as the context data sample to be processed. In subsequent iterations, the updated context data sample based on the previous iteration is used as the context data sample to be processed. Each iteration includes: The context data sample to be processed is encoded with state features to obtain state feature samples; The state feature samples are feature mapped through the decision network to be trained to obtain evaluation value samples of multiple types of candidate actions in the action space, and the target action sample is determined from the multiple types of candidate actions based on the evaluation value samples. If the target action is to complete the generation of response data, the iterative process is stopped, and response data samples of the input information generated by the large language model are extracted from the context data to be processed. The reward value is determined based on the response data sample, and the parameters of the decision network to be trained are updated based on the reward value.

[0006] This application provides a data processing apparatus for a large language model, including: The first data processing module is used to construct initial context data based on the input information to be responded to from the large language model; The second data processing module is used to perform multiple iterative processes. In the first iterative process, the initialized context data is used as the context data to be processed. In subsequent iterative processes, the updated context data based on the previous iterative process is used as the context data to be processed. Each iterative process includes: The context data to be processed is encoded with state features to obtain state features; the state features are then mapped to obtain evaluation values ​​of multiple types of candidate actions in the action space; and the target action is determined from the multiple types of candidate actions based on the evaluation values. If the target action is to complete the generation of response data, the iterative process is stopped, and the response data of the input information generated by the large language model is extracted from the context data to be processed.

[0007] This application provides a training apparatus for a decision network, comprising: The data processing module is used to construct an initial context data sample based on the input information sample to be responded to from the large language model; The data processing module is further configured to perform multiple iterative processes, wherein in the first iterative process, the initialized context data sample is used as the context data sample to be processed, and in subsequent iterative processes, the context data sample updated based on the previous iterative process is used as the context data sample to be processed; each iterative process includes: The context data sample to be processed is encoded with state features to obtain state feature samples; The state feature samples are feature mapped through the decision network to be trained to obtain evaluation value samples of multiple types of candidate actions in the action space, and the target action sample is determined from the multiple types of candidate actions based on the evaluation value samples. If the target action is to complete the generation of response data, the iterative process is stopped, and response data samples of the input information generated by the large language model are extracted from the context data to be processed. The training module is used to determine the reward value based on the response data sample and to update the parameters of the decision network to be trained based on the reward value.

[0008] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the data processing method for the large language model provided in the embodiments of this application, or implements the training method for the decision network provided in the embodiments of this application.

[0009] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implement the data processing method for a large language model provided in this application, or the training method for a decision network provided in this application.

[0010] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the data processing method for a large language model provided in this application, or the training method for a decision network provided in this application.

[0011] The embodiments of this application have the following beneficial effects: By performing multiple iterations, the context data to be processed is encoded and mapped in each iteration, and the target action is determined from the action space based on the obtained evaluation value. This enables dynamic control of the processing behavior of the large language model based on the current context state, and allows for adaptive selection of the target action according to real-time needs during processing, improving the flexibility and targeting of input information processing. At the same time, by using the context data updated based on the previous iteration to drive subsequent iterations until the target action indicates the completion of response data generation, it is ensured that the response data extracted from the context data to be processed is based on the progressively improved information state and dynamically optimized action path, effectively improving the accuracy and generation quality of the response data. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the data processing system architecture for a large language model provided in an embodiment of this application; Figure 2A This is a first structural schematic diagram of the electronic device provided in an embodiment of this application; Figure 2B This is a schematic diagram of the second structure of the electronic device provided in the embodiments of this application; Figure 3A This is a first flowchart illustrating the data processing method for a large language model provided in an embodiment of this application; Figure 3B This is a second flowchart illustrating the data processing method for a large language model provided in an embodiment of this application; Figure 3C This is a schematic diagram of the third process of the data processing method for a large language model provided in the embodiments of this application; Figure 4 This is a flowchart illustrating the training method for the decision network provided in an embodiment of this application; Figure 5 This is a schematic diagram of the architecture of the data processing method for a large language model provided in an embodiment of this application; Figure 6 This is a schematic diagram illustrating the principle of the data processing method for a large language model provided in the embodiments of this application; Figure 7 This is a schematic diagram illustrating the first principle of the number threshold provided in the embodiments of this application; Figure 8 This is a schematic diagram illustrating the second principle of the number threshold provided in the embodiments of this application; Figure 9 This is a schematic diagram illustrating the principle of obtaining state features provided in the embodiments of this application; Figure 10 This is a schematic diagram illustrating the principle of decision network action evaluation and reward value calculation provided in the embodiments of this application.

[0013] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0016] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0017] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0018] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0019] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0020] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0021] 1) Large Language Models (LLMs), also known as large models, are large-scale language models designed to understand and generate human language. They are trained on massive amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more. The defining characteristic of large language models is their sheer size, containing billions of parameters that help them learn complex patterns in language data. They are typically based on deep learning architectures. Large language models refer to deep learning models trained on vast amounts of text data, containing billions or even more parameters. They can be used to generate natural language text and understand its meaning. Through training, the models learn the statistical regularities and semantic relationships of language to build a vast language knowledge base, thereby simulating human language comprehension and generation capabilities.

[0022] 2) Retrieval-Enhanced Generation (RAG) is a technical architecture that combines a pre-trained large language model (as the generator) with an external knowledge retrieval mechanism (as the retrieval unit). Under the RAG framework, when the model receives input, it first retrieves relevant background information or factual data from external knowledge bases or document collections, and then provides these retrieval results as supplementary contextual information along with the original input to the large language model. This approach effectively alleviates the "illusion" problem of large language models, enabling the model to utilize up-to-date information or private domain knowledge beyond the training data, thereby improving the accuracy, timeliness, and reliability of the generated content.

[0023] 3) Reinforcement Learning (RL) is an important branch of machine learning that focuses on how an agent takes actions within an environment to maximize a cumulative reward. In reinforcement learning, the agent observes the current state of the environment, selects an action based on a policy, and the environment transitions to a new state after the action is executed, providing the agent with a reward signal. Through continuous interaction with the environment, trial and error, and iteration, the agent learns the optimal policy to achieve the goal of a specific task. In the embodiments of this application, the decoding and generation process of a large language model or the tool invocation process can both be modeled as reinforcement learning processes.

[0024] 4) Q-learning is a value-based reinforcement learning algorithm. Its core idea is to maintain a Q-table to record the "quality" or "value" (Q-value) of taking a specific action in a given state. The Q-value represents the expected cumulative reward obtained by performing the action and following the optimal policy from the current state. Q-learning uses the Bellman Equation to iteratively update the Q-value, allowing the agent to select the optimal action based on the Q-value (e.g., the action with the highest Q-value), thus learning the optimal policy without requiring an environmental model.

[0025] 5) Deep Q-Network (DQN) is a reinforcement learning algorithm that combines deep learning with Q-learning. DQN introduces a deep neural network (i.e., a Q-network) to approximate the Q-value function, which is the input state features. Through the calculation and feature mapping of the neural network, it outputs the Q-value (i.e., the evaluation value) corresponding to each action.

[0026] With the development of artificial intelligence technology, large language models are widely used in various intelligent question answering and information processing scenarios. In related technologies, to generate responses based on input information using large language models, retrieval-enhanced generation (RAG) technology is employed. This involves retrieving content from an external knowledge base based on the input information to construct a contextual environment, and then interacting with the large language model to obtain the response result. However, this context construction and interaction mode, based on a fixed retrieval strategy, makes it difficult to flexibly adjust the processing flow according to the complexity of the input information or the generation status, thus affecting the quality of the generated response data.

[0027] This application provides a data processing method for a large language model, a training method for a decision network, an apparatus, a device, a computer-readable storage medium, and a computer program product. These methods can adaptively select target actions by iteratively evaluating the current state, improving the flexibility and accuracy of data processing. The following describes exemplary applications of the electronic devices provided in this application. These electronic devices can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and in-vehicle terminals, or as servers.

[0028] See Figure 1 , Figure 1 This is a schematic diagram of the data processing system architecture for a large language model provided in an embodiment of this application. Figure 1The system involves server 100, terminal device 200, and network 300. Terminal device 200 is connected to server 100 through network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0029] In some embodiments, the data processing method for the large language model provided in this application can be implemented collaboratively by a server and a terminal device. For example, the terminal device 200 sends input information to the server 100, the server 100 receives the input information, obtains the corresponding data corresponding to the input information through the data processing method for the large language model provided in this application, and sends the response data to the terminal device 200.

[0030] In other embodiments, the data processing method for the large language model provided in this application can also be implemented independently by a terminal device or a server. The terminal device 200 or the server 100 generates response data corresponding to the input data using the data processing method for the large language model provided in this application.

[0031] In some embodiments, the training method for the decision network provided in this application can be implemented collaboratively by a server and a terminal device. For example, the terminal device 200 sends input information samples to the server 100, the server 100 receives the input information samples, and trains the decision network using the training method for the decision network provided in this application.

[0032] Here, the trained decision network is used to determine the target action from the action space based on state features during the data processing of the large language model.

[0033] In other embodiments, the training method for the decision network provided in this application can also be implemented by a terminal device or a server alone. The terminal device 200 or the server 100 trains the decision network using the training method provided in this application.

[0034] Here, server 100 can be a single server. In this case, the data processing method for the large language model and the training method for the decision network provided in this application embodiment can be implemented by the same server. Server 100 can also be a cluster of servers. In the case that server 100 is a server cluster, the data processing method for the large language model and the training method for the decision network provided in this application embodiment can be implemented by different servers. This application embodiment does not impose any limitations on this.

[0035] The data processing method for large language models provided in this application can be applied to various complex scenarios such as intelligent customer service, professional domain question answering, and copywriting assistance. Examples are given below.

[0036] 1) Intelligent customer service scenario, for example, users send consultation instructions (i.e. input information) to the server through voice (e.g., "How to apply for an order refund and check the progress?") or text input on terminal devices (such as smartphones or web pages); the server generates accurate customer service response data through the data processing method of the large language model provided in the embodiments of this application, so as to achieve efficient and accurate customer service.

[0037] 2) In professional question-and-answer scenarios, for example, users send professional consultation instructions (such as "What are the dietary restrictions when taking XX drug?") to the intelligent question-and-answer system via text input on terminal devices (such as medical tablets or office computers); the server generates answer data with professional basis through the data processing method of the large language model provided in the embodiments of this application, so as to realize accurate query and suggestion of medical information.

[0038] 3) In copywriting assistance scenarios, for example, users send writing instructions to the large language model via text input on their terminal devices (such as "Please write an annual work summary report based on the annual meeting minutes in the local knowledge base"); the server generates logically coherent and detailed summary report data through the data processing method of the large language model provided in this application embodiment, thereby achieving high-quality assisted writing for various documents.

[0039] 4) Hot topic information query scenario, for example, a user sends an information query command to the hot topic information assistant through text input on the terminal device (such as "summarize the major developments in the technology field in the past week and analyze their impact"); the server generates information brief data with both timeliness and depth through the data processing method of the large language model provided in the embodiments of this application, so as to realize the rapid tracing and multi-dimensional interpretation of hot topics.

[0040] Taking electronic devices (such as servers and terminal devices) for data processing of large language models as an example, see [link to relevant documentation]. Figure 2A , Figure 2A This is a first structural schematic diagram of the electronic device provided in an embodiment of this application. Figure 2A The illustrated electronic device 100-1 includes at least one processor 110-1, a memory 130-1, and at least one network interface 120-1. The various components in the electronic device 100-1 are coupled together via a bus system 140-1. It is understood that the bus system 140-1 is used to implement communication between these components. In addition to a data bus, the bus system 140-1 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2A The general designated all buses as Bus System 140-1.

[0041] The processor 110-1 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0042] The memory 130-1 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 130-1 may optionally include one or more storage devices physically located away from the processor 110-1.

[0043] The memory 130-1 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 130-1 described in this application embodiment is intended to include any suitable type of memory.

[0044] In some embodiments, memory 130-1 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0045] Operating system 131-1 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 132-1 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 120-1, such as Bluetooth, WiFi, and Universal Serial Bus (USB). In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2A A data processing apparatus 133 for a large language model stored in memory 130-1 is shown. This apparatus can be software in the form of programs and plug-ins, and includes the following software modules: a first data processing module 1331 and a second data processing module 1332. These modules are logically related and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0046] Taking the use of electronic devices (such as servers and terminal devices) for training decision networks as an example, see [link to relevant documentation]. Figure 2B , Figure 2B This is a schematic diagram of the second structure of the electronic device provided in the embodiments of this application. Figure 2B The illustrated electronic device 100-2 includes at least one processor 110-2, a memory 130-2, and at least one network interface 120-2. The various components in server 100-2 are coupled together via a bus system 140-2. It is understood that the bus system 140-2 is used to implement communication between these components. In addition to a data bus, the bus system 140-2 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2B All buses are labeled as Bus System 140-2. For detailed descriptions of Processor 110-2 and Memory 130-2, please refer to the above text; they will not be repeated here.

[0047] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2B A training device 134 for a decision network stored in memory 130-2 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a data processing module 1341 and a training module 1342. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of the modules will be described below.

[0048] In some embodiments, the terminal device or server can implement the data processing method for the large language model and the training method for the decision network provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0049] The following will describe the data processing method for the large language model provided in the embodiments of this application, with the server as the execution subject, by combining the exemplary application and implementation of the server provided in the embodiments of this application.

[0050] See Figure 3A , Figure 3A This is a first flowchart illustrating the data processing method for a large language model provided in this application embodiment, which will be combined with... Figure 3A The steps shown are explained.

[0051] In step 101, initial context data is constructed based on the input information to be responded to by the large language model.

[0052] In some embodiments, context data is a multi-dimensional collection of information, including input information to be responded to, dialogue history (i.e., historical interaction round pairs), and generated response data fragments (used to maintain autoregressive properties during the generation process; if it is the first iteration, the response data fragment is empty or has a specific identifier).

[0053] Here, the input information to be responded to refers to the sequence of natural language requests received at the current moment that has not yet been processed by the large language model. For example, the input information can be a query question sent by the user terminal, such as "Please analyze the sales trend of new energy vehicles in 2024"; it can also be the latest instruction entered by the user in a continuous multi-turn dialogue scenario, such as "Based on the above analysis, give a purchase suggestion", in which case the input information is short but rich in referential dependence on the preceding information; or it can include a text paragraph with a clear task objective, such as "Summarize this meeting record".

[0054] For example, the input information to be responded to can be a discrete sequence of tokens processed by a tokenizer, denoted as... Lexical sequences carry semantic instructions or questions.

[0055] It should be noted that in scenarios involving Multimodal Large Language Models (MLLM), the input information to be responded to can also be a combination of multimodal data. For example, the input information can be a feature sequence of "image + text", where the image data is encoded as a visual feature vector (Visual Embedding).

[0056] Here, dialogue history refers to one or more ordered pairs of "input information-response data" (QA Pairs) generated before the current request (i.e., the request corresponding to the input information to be responded to). For example, it can be represented as... ,in, Represents the rounds of dialogue. This represents the input information for the (k-1)th round of dialogue (such as the question entered by the user). This represents the response data for the (k-1)th round of dialogue (such as the answer output by a large language model).

[0057] For example, if the context window length of a large language model is limited (i.e., the length of the input data is restricted), a sliding window algorithm can be used to prioritize retaining the data closest to the current time step. Rounds of dialogue, as a history of dialogue.

[0058] Here, the generated response data fragments refer to the known output prefixes used as conditions for autoregressive decoding during the process of generating response data. These prefixes instruct the large language model to continue generating response data based on the "preceding context" (the generated response data, such as the generated answer fragments).

[0059] For example, in the first iteration, the large language model has not yet generated response data. At this time, the generated response data fragments are initialized to an empty set, or assigned a preset start and end character (such as...). <bos>or <start>In subsequent iterations, the response data fragments include the accumulated response data up to the current iteration.

[0060] For example, suppose the input to be responded to is "Please explain machine learning", and the dialogue history includes a round of question and answer {(Q: {Hello}, A: {Hello! How can I help you?}}.

[0061] The initialized context data can be represented as: [Role_User]: Hello [SEP][Role_Bot]: Hello! How can I help you? [SEP][Role_User]: Please explain machine learning. [SEP][Role_Bot]: <bos>.

[0062] Among them, [Role_User] and [Role_Bot] are role identifiers used to indicate to the large language model whether the current processing context belongs to the input part (such as the questioner) or the response part (such as the answerer), and [SEP] is the separator. <bos>This is the starting bit used to initialize the response data fragment.

[0063] By constructing multi-dimensional contextual data that includes input information to be responded to, historical dialogues, and generated fragments, and using role identifiers and delimiters for structured representation, ambiguity between different information sources can be effectively eliminated, clear semantic boundaries can be established, and large language models can accurately capture long-range semantic dependencies and multi-turn dialogue logic, thereby ensuring a high degree of consistency and coherence between the content of subsequently generated response data and the contextual environment.

[0064] In step 102, multiple iterative processes are performed. In the first iteration, the initialized context data is used as the context data to be processed. In subsequent iterations, the updated context data based on the previous iteration is used as the context data to be processed. Each iteration includes: encoding the state features of the context data to be processed to obtain state features; performing feature mapping on the state features to obtain evaluation values ​​of multiple types of candidate actions in the action space, and determining the target action from the multiple types of candidate actions based on the evaluation values; stopping the iterative process when the target action is to complete the generation of response data, and extracting the response data generated by the large language model from the context data to be processed.

[0065] In some embodiments, in the first iteration, the initialized context data is used as the context data to be processed, and in subsequent iterations, the context data updated based on the target action determined in the previous iteration is used as the context data to be processed. Each iteration process includes: encoding the state features of the context data to be processed to obtain state features; performing feature mapping on the state features to obtain evaluation values ​​of multiple types of candidate actions in the action space; and determining the target action from the multiple types of candidate actions based on the evaluation values. If the target action is to generate complete response data for the input information, stop the iterative processing and extract the response data of the input information generated by the large language model from the context data to be processed corresponding to the current round of iterative processing. If the target action is not to generate complete response data, execute the data processing instructions corresponding to the target action to obtain the processing result and update the context data to be processed based on the processing result.

[0066] For example, the action space includes multiple types of candidate actions, such as performing retrieval, generating response data, and completing response data generation. See [link to action space]. Figure 6 , Figure 6 This is a schematic diagram illustrating the principle of the data processing method for a large language model provided in the embodiments of this application, as shown below. Figure 6 As shown, the process first acquires input information and establishes initial context data to be processed before entering the iterative processing stage. In each iteration, the context data is processed by state feature encoding and a decision network to determine the target action. If the target action is to perform a retrieval, external information is obtained from an external knowledge base through the retrieval module, and the context is updated. If the target action is to generate response data, the generated response data is obtained using a large language model, and the context is updated. In both cases, the process returns to the next iteration. If the target action is to complete the generation of response data, the iteration loop is exited, and the final response data is output.

[0067] In some embodiments, the context data to be processed includes input information, dialogue history, and generated response data fragments; multiple types of candidate actions include performing a retrieval; see also Figure 3B The state features are encoded by the context data to be processed, which can be achieved through the following steps 201 to 203, as explained in detail below.

[0068] In step 201, the input information is feature-encoded to obtain input information features, the dialogue history is feature-encoded to obtain dialogue history features, and the response data fragments are feature-encoded to obtain response features.

[0069] In some embodiments, the process of feature encoding the input information, dialogue history, and generated response data fragments can share the same feature extractor, or it can use separate feature extractors with independent parameters. The feature extractor can be a pre-trained language model based on a Transformer architecture.

[0070] For example, feature encoding of input information can be achieved as follows: First, a tokenizer is used to convert the input information into a sequence of tokens, and each token is mapped to an embedding vector. Positional encoding is then superimposed on the embedding vector. Next, a multi-head self-attention mechanism is used to process the superimposed vector. In this process, the dot product of the query vector and the key vector is calculated to obtain the attention weights, and the value vector is weighted and aggregated according to the attention weights to generate a hidden state sequence that includes contextual semantics. Finally, to obtain a fixed-dimensional feature representation, pooling operations (such as average pooling or taking the first [CLS] token vector) can be performed on the hidden state sequence to output the final input information features.

[0071] For example, for input information: the input information is fed into the feature extractor to extract semantic representation, denoted as... .

[0072] Regarding dialogue history: to balance the richness of contextual information with computational overhead, the most recent dialogue history is pre-selected. Wheel (e.g.) The dialogue content is concatenated into a single text sequence, which is then input into a feature extractor to obtain an encoded vector. .

[0073] For the generated response data fragments: if the large language model has already generated a partial response (such as an answer fragment) at the current time, then the generated response data fragments are input into the feature extractor to obtain the encoded vector. If the current state is the initial moment of generation (i.e., no content has been generated yet), then directly... Set it to a zero vector that matches the feature dimension.

[0074] For example, suppose the output dimension of the feature extractor is... .

[0075] For the input information, after encoding and pooling, the dimension is obtained as follows: vector .

[0076] For the dialogue history, the content of the most recent 3 rounds is concatenated and encoded to obtain the dimension. vector .

[0077] For the generated response data fragment, if content exists, it is encoded as follows: vector If empty, then construct a value that is all 0. Vector as .

[0078] In step 202, the number of times the target action determined from multiple types of candidate actions before the current iteration is retrieved is obtained, and the number is feature-encoded to obtain the retrieval count feature.

[0079] Here, the number of retrievals is a scalar value that represents the frequency with which the large language model has referenced external knowledge in the current generation task.

[0080] For example, during the multi-round iterative processing, a counter variable is maintained in real time. This counter variable is initialized to 0. Each time "execute retrieval" is selected as the target action in a previous iteration and executed, the counter variable is incremented by 1. The current value of the counter variable is used to determine the number of retrievals performed so far. .

[0081] In some embodiments, the method of feature encoding the number of times may include: directly using the scalar value as the retrieval number feature; or, normalizing the number of times (e.g., dividing it by a preset maximum retrieval number limit) and using the normalized value as the retrieval number feature; or, mapping the scalar value to the retrieval number feature through a preset embedding layer.

[0082] In step 203, the input information features, dialogue history features, response features, and retrieval frequency features are fused to obtain the state features.

[0083] In some embodiments, input information features, dialogue history features, response features, and retrieval frequency features are concatenated to obtain a state vector.

[0084] For example, see Figure 9 , Figure 9 This is a schematic diagram illustrating the principle of acquiring state features provided in an embodiment of this application, such as... Figure 9 As shown, Figure 9 The diagram illustrates the data sources for state features and their corresponding feature extraction mappings. The input data for constructing state features encompasses four dimensions: input information (e.g., "comparing product A and product B"), dialogue history as context (e.g., "the last 3 rounds of dialogue"), response data fragments generated at the current moment, and the number of retrievals (i.e., numerical features) representing the frequency of external knowledge citations. In the mapping relationship, the data from the first three text modalities are respectively connected to feature extractors to transform unstructured text into high-dimensional semantic vectors; while the retrieval counts, representing the numerical modality, are connected to the normalization / embedding layer. The outputs of these four dimensions converge at the bottom, collectively triggering the final state feature (fusion vector).

[0085] For example, state characteristics can be represented as: By sequentially splicing the above four components along the feature dimension, this "hard connection" method preserves the independence of each semantic channel.

[0086] For example, based on the aforementioned numerical settings, the dimensions of each component are as follows: , , , (here) As a scalar value directly participating in the concatenation (or considered as a vector of dimension 1), the final state feature obtained after the concatenation operation is... Total dimensions The calculation is as follows: .

[0087] In other embodiments, if the number of searches If the state features are mapped to high-dimensional vectors (e.g., 32-dimensional) through the embedding layer, the dimensions of the state features will be adjusted accordingly (e.g., ...). (Dimension), this application embodiment does not limit this, as long as the state vector is guaranteed. Information encompassing the four dimensions mentioned above is sufficient.

[0088] By independently extracting features from input information, dialogue history, and generated fragments, and integrating retrieval frequency features, a comprehensive state representation including semantic content and operational status is constructed. This enables the decision network to not only deeply understand the current contextual dependence, but also accurately perceive the frequency and progress of external knowledge acquisition, thereby effectively avoiding redundant retrieval or information scarcity, and ensuring a dynamic balance and precise control between content richness and generation efficiency in subsequent action decisions.

[0089] In other embodiments, if the number of times equals a threshold number, the execution retrieval is removed from the action space. The threshold number is determined by: evaluating the complexity of the input information to obtain a complexity parameter; querying the threshold number range corresponding to the complexity parameter from a preset first mapping table; and determining the threshold number based on the threshold number range corresponding to the complexity parameter. The first mapping table includes the correspondence between multiple different complexity parameters and different threshold number ranges, and the threshold number is a positive integer greater than or equal to zero.

[0090] In some embodiments, the complexity of the input information is evaluated to obtain a complexity parameter, which can be achieved as follows: The input information is input into a pre-trained evaluation network (e.g., a classifier based on a network architecture such as Transformer or Multilayer Perceptron (MLP)), the semantic representation vector of the input information is extracted, and the semantic representation vector of the input information is mapped to a scalar value through a fully connected layer. The scalar value is then normalized to an interval after being processed by a sigmoid activation function. As a complexity parameter .

[0091] Here, the complexity parameter of the input information represents a quantitative indicator of the difficulty of answering the input question. The higher the parameter value, the broader the knowledge involved in the question, the longer the logical chain, or the more precise facts required, thus necessitating more external searches to assist in the generation process.

[0092] In some embodiments, the complexity of the input information can be evaluated to obtain the complexity parameter of the input information, which can also be achieved in the following ways: First, the input information is segmented into words to obtain a sequence of word units; then, part-of-speech tagging is performed on each word unit, and the weight category of each word unit is determined based on the part-of-speech tagging results; at the same time, based on a pre-built large-scale corpus statistical table, the inverse document frequency (IDF) of each word unit is queried. The inverse document frequency represents the scarcity or information content of the word unit in the general context; finally, the complexity parameter is determined based on the part-of-speech weight and inverse document frequency of each word unit.

[0093] For example, determining the weight category of each word based on part-of-speech tagging results can be achieved in the following way: pre-construct a mapping table between part-of-speech tags and weight coefficients, determine the semantic role of the word based on the part-of-speech tagging results, classify content words (such as noun, verb, and adjective tags) that carry core semantic information into high-weight categories, and classify function words (such as prepositions, conjunctions, and auxiliary words tags) that mainly play a grammatical connection or auxiliary role into low-weight categories, thereby assigning differentiated weight coefficients to word elements with different information densities.

[0094] For example, the weight of noun entities can be set to 1.5, and the weight of stop words can be set to 0.1.

[0095] For example, the complexity parameter can be determined based on the part-of-speech weight and inverse document frequency of each word. This can be achieved by calculating the product of the part-of-speech weight and the inverse document frequency of each word, summing the products of all words in the input information, and dividing the sum by the total number of words to obtain the complexity parameter.

[0096] For example, suppose the input information contains The word element, for the first word element each word element Its inverse document frequency (IDF) is The weighted average information content of the input information is calculated as a complexity parameter. The calculation formula can be expressed by formula (1): (1) in, This is the part-of-speech weight (e.g., the weight of noun entities is set to 1.5, and the weight of stop words is set to 0.1). If the calculated... A value of 0.75 indicates that the input information contains many low-frequency entity words and has high complexity.

[0097] In other embodiments, the complexity parameter may also be a function of the text length of the input information, the number of Named Entity Recognition (NER) results, or the perplexity (PPL) score generated by the large language model, and this application does not impose any limitations.

[0098] By performing multi-dimensional and complex quantitative evaluations of input information (such as extracting semantic representations using neural networks or combining inverse document frequency statistics with part-of-speech weights), and adaptively adjusting the retrieval threshold accordingly, the on-demand dynamic allocation of external retrieval resources is achieved. This mechanism ensures that when faced with highly complex queries, the model can acquire sufficient external knowledge to guarantee the integrity of the logical chain and the accuracy of the answer. When dealing with low-complexity queries, it can promptly truncate the retrieval action, effectively avoiding the waste of computing resources and response delays caused by invalid and redundant retrievals. This significantly optimizes the reasoning efficiency and interactive experience of the large language model in all scenarios.

[0099] In some embodiments, the number of times threshold is determined based on the range of times threshold values ​​corresponding to the complexity parameter queried from a preset first mapping table. This can be achieved in the following ways: if the complexity parameter is less than or equal to the first complexity threshold, then the preset initial value is used as the number of times threshold; if the complexity parameter is greater than the first complexity threshold and less than the second complexity threshold, then the complexity parameter is mapped to the number of times threshold through a preset linear growth function, wherein the number of times threshold is greater than the initial value and less than the preset target value; if the complexity parameter is greater than or equal to the second complexity threshold, then the target value is used as the number of times threshold.

[0100] Example, Figure 7 This is a schematic diagram illustrating the first principle of the number threshold provided in the embodiments of this application, such as... Figure 7 As shown, the process first involves evaluating the complexity of the input information to obtain quantified complexity parameters. Next, in the stage of determining the number of iterations threshold based on the first mapping table, the mapping curve is horizontal when the complexity parameter is less than or equal to the first complexity threshold, indicating that the initial value is used as the number of iterations threshold. When the complexity parameter is between the first and second complexity thresholds, the mapping curve is a straight line with a positive slope, indicating that a linear growth strategy is executed, such as... Figure 7 As shown by the dashed arrow, the specific complexity parameter is projected onto the diagonal line to determine the corresponding number of searches thresholds. When the complexity parameter is greater than or equal to the second complexity threshold, the mapping curve becomes horizontal again but remains at a high position, indicating that the target value is used as the number of searches threshold at this time. Finally, based on the above mapping relationship, the specific numerical value is obtained, and the number of searches threshold is output. The number of searches threshold will be used to limit the maximum number of searches performed in subsequent iterations.

[0101] For example, suppose the complexity parameter of the current input information is... And satisfy ,in, This represents the first complexity threshold. Let the second complexity threshold be represented, then the preset linear growth function can be expressed by formula (2): (2) Where K represents the threshold number of times, This represents the normalized position proportion of the current complexity within the growth range. This indicates that the calculation result is rounded (e.g., rounded to the nearest integer or rounded up) to ensure that the number of times the threshold is an integer.

[0102] Assuming a preset first complexity threshold Second complexity threshold Initial value of the number of times Target value If the complexity parameter of the input information is calculated through the preceding steps... First, calculate the normalization ratio as follows: Then calculate the number of thresholds. After rounding, the final threshold for the number of iterations was determined to be 4. This process dynamically and smoothly increases the number of processing iterations or the retrieval depth as the complexity of the information increases.

[0103] By establishing a segmented dynamic mapping mechanism between the complexity of input information and the threshold number of attempts, an adaptive matching between the allocation of computing resources and the task difficulty requirements is achieved. This not only locks a low threshold for low-complexity scenarios to significantly reduce response latency and computational overhead and avoid resource waste, but also ensures sufficient retrieval and reasoning attempts in high-complexity scenarios to improve the accuracy and richness of generated content. Furthermore, it utilizes a linear growth function in the intermediate interval to achieve a smooth transition and fine-grained control from a low-energy consumption mode to a high-precision mode, effectively avoiding the risk of threshold mutation. Ultimately, the optimal balance between system throughput efficiency, response speed, and response quality is achieved through dynamic adaptation.

[0104] In other embodiments, the process of mapping the complexity parameter to the number of iterations threshold can also be implemented using a non-linear growth function, such as a variant of the sigmoid function or exponential function instead of the linear growth function described above. For example, between the first and second complexity thresholds, a logarithmic growth curve can be used, so that the number of iterations threshold increases slowly when the complexity is low, while rapidly approaching the target value when the complexity is close to the second threshold, thus more proactively handling highly complex input information. The embodiments of this application do not limit the specific type of mapping function.

[0105] By employing a dynamic segmentation mapping strategy based on complexity intervals (such as low-order preservation, linear smooth growth, and high-order truncation), it is possible to achieve a precise match between the retrieval threshold and the difficulty of the input information. This mechanism avoids invalid retrieval and resource waste by limiting the number of retrievals to a low threshold when dealing with simple tasks. On the other hand, it linearly increases the retrieval depth as needed to ensure knowledge supply when facing medium-difficulty tasks. At the same time, it prevents retrieval divergence or long waiting times by truncating the upper limit for high-difficulty tasks. Thus, it maximizes the utilization efficiency of computing resources while ensuring the quality and depth of the response.

[0106] In other embodiments, the number of searches threshold can also be determined by: evaluating the complexity of the input information to obtain a complexity parameter; classifying the input information by domain to determine the target domain to which the input information belongs, and querying the fault tolerance parameter corresponding to the target domain from a preset second mapping table, wherein the second mapping table includes the correspondence between multiple different target domains and different fault tolerance parameters; obtaining the service priority corresponding to the input information, and determining the baseline search count corresponding to the service priority; determining an adjustment coefficient based on the fault tolerance parameter and the complexity parameter, and adjusting the baseline search count based on the adjustment coefficient to obtain the number of searches threshold.

[0107] Here, the implementation method for evaluating the complexity of the input information and obtaining the complexity parameter of the input information can be found in the description above, and will not be repeated here.

[0108] Here, the fault tolerance parameter refers to the tolerance of the domain corresponding to the input information for the accuracy of the search results. The lower the fault tolerance parameter, the lower the tolerance of the domain for errors (i.e., the higher the accuracy requirement), and more searches are needed to ensure recall quality.

[0109] For example, see Figure 8 , Figure 8 This is a schematic diagram illustrating the second principle of the number threshold provided in the embodiments of this application, such as... Figure 8 As shown, the input information is first acquired and processed in three branches: the left branch evaluates the complexity of the input information to obtain complexity parameters; the middle branch classifies the input information into a domain and determines the target domain, and obtains the corresponding fault tolerance parameters by querying a second mapping table; the right branch determines the baseline retrieval count by obtaining service priority. Next, the complexity and fault tolerance parameters obtained from the left and middle branches are combined to determine the adjustment coefficient. Finally, the baseline retrieval count determined by the right branch is adjusted based on the adjustment coefficient, thus obtaining and outputting the final count threshold.

[0110] In some embodiments, domain classification of input information can be achieved by inputting the semantic vector of the input information into a classifier consisting of a fully connected layer and a softmax activation function, outputting the probability distribution in various preset domains (such as medical, legal, daily conversation, programming, etc.), and selecting the category with the highest probability as the target domain.

[0111] For example, the pre-defined second mapping table is stored in a local database or cloud configuration center in the form of key-value pairs. Assuming the identified target domain is "medicine," the corresponding fault tolerance parameter can be found by looking up the table. (Low fault tolerance, high precision requirements); If the target domain is "daily dialogue", the corresponding fault tolerance parameter can be found by looking up the table. (High fault tolerance, low precision requirement).

[0112] In some embodiments, determining the adjustment coefficient and the number of searches based on the fault tolerance parameter and the complexity parameter can be achieved in the following way: First, determine the baseline number of searches according to the service priority (e.g., the first priority for scenarios with high real-time requirements). (For example, (times). Subsequently, the adjustment coefficients were constructed. The computational model, which is related to the complexity parameter It is positively correlated with the fault tolerance parameter. It shows a negative correlation.

[0113] Example, adjustment coefficient The calculation can be performed using the following formula (3): (3) in, and This is a preset weighting factor. The final threshold for the number of occurrences. pass Calculated. Assuming the baseline retrieval count , , .

[0114] Example 1: The input is of high complexity ( ), medical field ( The query for ) will then Number of times threshold Second-rate.

[0115] Example 2: Input is of low complexity ( ), casual conversation area ( The query for ) will then Number of times threshold Second-rate.

[0116] This demonstrates that by introducing a combined adjustment of fault tolerance and complexity, the retrieval depth can be significantly improved for challenging tasks in rigorous fields.

[0117] In other embodiments, the adjustment coefficients can also be determined by looking up a preset multidimensional correlation matrix. For example, the complexity parameter can be discretized into several levels (such as high, medium, and low), and the fault tolerance parameter can be discretized into several levels. A two-dimensional lookup table can be constructed to directly obtain the corresponding adjustment coefficients, thereby reducing the overhead of real-time floating-point operations. The embodiments of this application do not limit the specific calculation method.

[0118] By integrating complexity assessment of input information, domain classification (and its corresponding fault tolerance parameters), and service priority, a multi-dimensional dynamic adjustment mechanism for retrieval thresholds is constructed. This mechanism can automatically increase retrieval depth for vertical domains with low fault tolerance and high accuracy requirements, such as medical and legal fields, to ensure the rigor of nursing care and reasoning. At the same time, it can reasonably reduce retrieval frequency in high fault tolerance scenarios, such as casual conversations. This achieves an optimal balance between computing resource allocation and response quality while meeting the differentiated accuracy requirements of different business scenarios.

[0119] In some embodiments, see Figure 3C The state features are mapped to obtain the evaluation values ​​of multiple types of candidate actions in the action space. This can be achieved through the following steps 301 to 303, which are explained in detail below.

[0120] In step 301, the state features are input into a pre-trained decision network, wherein the decision network includes a cascaded feature extraction layer and an output layer.

[0121] For example, the decision network can be implemented as a deep Q-network (DQN), a policy gradient network, or other network structures, and the embodiments in this application are not limited thereto.

[0122] Here, cascading refers to connecting multiple network layers sequentially in a feedforward manner, so that the output of the previous layer directly serves as the input of the next layer, thereby achieving layer-by-layer abstraction of features and extraction of higher-order semantics.

[0123] In step 302, the state features are mapped by a first feature mapping through a cascaded feature extraction layer to obtain the mapped features.

[0124] Here, the feature extraction layer may include a convolutional neural network module consisting of a convolutional layer, a pooling layer, and an activation function, or a Transformer encoder module consisting of a multi-head self-attention mechanism. This application does not limit the specific structure of the feature extraction layer.

[0125] For example, the first feature mapping of the state features is performed through cascaded feature extraction layers to obtain the mapped features. This can be achieved as follows: the state features are used as the initial input, and the forward propagation calculation is performed through each feature extraction layer in cascaded order. After the state features are abstracted and semantically extracted layer by layer, the feature vector output by the last feature extraction layer is used as the mapped features.

[0126] In step 303, the mapping features are subjected to a second feature mapping through the output layer to obtain the action evaluation vector. The number of elements in the action evaluation vector is consistent with the number of action types in the action space, and the value of each element in the action evaluation vector represents the evaluation value of each candidate action.

[0127] In some embodiments, the output layer may consist of fully connected layers to project abstracted mapped features onto the action space. The second feature mapping can be understood as a linear transformation process designed to compute the value or unnormalized logits of each candidate action in the current state. The number of action types in the action space is predefined. The number of neurons in the output layer is configured to be the same as... Consistent.

[0128] In other embodiments, the context data to be processed includes dialogue history; before inputting the state features into the pre-trained decision network, the following processing may also be performed: calculating the semantic relevance between the dialogue history and the input information, and detecting historical retrieval records in the dialogue history that have performed retrieval operations to obtain detection results; determining the confidence level of the dialogue history based on the semantic relevance and the detection results; fusing the dialogue history confidence level with the state features to obtain fused state features, and then proceeding to the processing of inputting the state features into the pre-trained decision network based on the fused state features.

[0129] Here, dialogue history confidence refers to a quantitative indicator used to characterize the reference value or effectiveness of contextual information in the dialogue history for the current input information. Specifically, dialogue history confidence comprehensively reflects the semantic relevance of past dialogue content to the current question, as well as the completeness of external retrieval information acquired in historical interactions. Dialogue history confidence aims to indicate to what extent the decision network should rely on or maintain the current dialogue context when determining the target action (e.g., whether to directly answer using historical retrieval results), or to determine whether the current input information constitutes a significant topic shift, thus requiring a reduction in the weighting of historical information and the initiation of new retrieval exploration.

[0130] For example, the semantic relevance between dialogue history and input information can be calculated as follows: Semantic vector representations of the input information and dialogue history are extracted from a pre-trained semantic encoding model (such as a bidirectional encoder representations from transformers (BERT), a transformer encoder, etc.), the cosine similarity between the two semantic vectors is calculated, and the calculated similarity value is used as the semantic relevance representing the degree of semantic content correlation between the two.

[0131] For example, to detect historical retrieval records in the dialogue history where retrieval operations have been performed and obtain the detection results, the following methods can be used: traverse the metadata or operation logs corresponding to each round of dialogue in the dialogue history, identify whether there is an action tag indicating "perform retrieval", count the number of retrieval actions that occurred within a preset historical window period, or generate a binary status identifier representing whether there is a valid retrieval history, as the detection result.

[0132] For example, the confidence level of dialogue history can be determined based on semantic relevance and detection results in the following ways: normalize the semantic relevance and detection results, and then perform a weighted sum of the two normalized results based on preset weight coefficients, or input the two results into a preset activation function (such as the Sigmoid function) for non-linear mapping to obtain a scalar value between 0 and 1, which serves as the confidence level of dialogue history representing the reliability of the current dialogue history content.

[0133] For example, fusing dialogue history confidence with state features to obtain fused state features can be achieved as follows: the scalar form of dialogue history confidence is mapped to a confidence feature vector that matches the state feature dimension through a linear projection layer or embedding layer, and the confidence feature vector and state features are concatenated or element-wise added along the feature channel dimension to obtain the fused state features.

[0134] For example, the process of inputting the fused state features into a pre-trained decision network can be achieved as follows: the fused state features are fed as an input tensor into the input layer of the decision network, instructing the decision network to perform forward propagation calculations on the fused state features using its internal cascaded feature extraction and output layers, thereby initiating the evaluation process for each candidate action in the action space.

[0135] In other embodiments, the candidate actions of multiple types in the action space include performing a retrieval; after performing a second feature mapping on the mapping features through the output layer to obtain the action evaluation vector, the following processing can also be performed: determining the action weighting coefficient for performing the retrieval based on the dialogue history confidence, adjusting the element values ​​corresponding to performing the retrieval in the action evaluation vector through the action weighting coefficient, and determining the target action based on the adjusted action evaluation vector, wherein the dialogue history confidence is negatively correlated with the evaluation value of performing the retrieval.

[0136] For example, determining the action weighting coefficient for performing a retrieval based on the confidence level of the dialogue history can be achieved by setting a decay function based on the confidence level of the dialogue history, and then calculating the action weighting coefficient corresponding to the confidence level of the dialogue history using the decay function.

[0137] For example, the confidence level of the dialogue history is... The preset adjustment intensity hyperparameter is: (For example Action weighting coefficients It can be represented as That is, the higher the confidence level of the dialogue history (i.e., the more relevant the historical information is and the more it contains retrieval records), the smaller the calculated action weighting coefficient will be.

[0138] For example, adjusting the values ​​of elements corresponding to the execution retrieval in the action evaluation vector using action weighting coefficients can be achieved as follows: Weight the action evaluation vector using action weighting coefficients, for example, if the action evaluation vector is... ,in, This corresponds to the original evaluation value (such as Q-value or Logits) for the "perform retrieval" action. The adjustment process performs element-wise multiplication operations. For example, it calculates the adjusted evaluation value. .

[0139] Continuing with the previous example, the adjustment process can also be represented as a subtraction operation of the penalty term, for example, as... ,in, This is a penalty factor.

[0140] The above adjustments reduce the score for the "execute search" action when the historical confidence level is high, thereby suppressing unnecessary duplicate searches.

[0141] For example, determining the target action based on the adjusted action evaluation vector can be achieved by taking the candidate action corresponding to the largest element value in the adjusted action evaluation vector as the target action.

[0142] Continuing with the previous example, after obtaining the adjusted motion evaluation vector... Then, the motion evaluation vector Perform a maximum value index lookup operation (Argmax). If the adjusted value is... The score is lower than the evaluation value of other actions due to inhibition by high confidence (e.g., If the decision network selects the candidate action with the highest evaluation value (such as "generate response data") as the target action, then the decision network will select the candidate action with the highest evaluation value (such as "generate response data") as the target action.

[0143] By utilizing decision networks to perform deep mapping of state features and introducing a dialogue history confidence mechanism to dynamically negatively adjust the evaluation value of the "perform retrieval" action, unnecessary repeated retrievals are automatically suppressed when it is confirmed that the dialogue history has high semantic relevance and contains valid retrieval records. This effectively avoids the waste of computing resources and response delays while ensuring contextual coherence and answer accuracy, and significantly improves the reasoning efficiency and interactive experience of large language models in multi-turn dialogue scenarios.

[0144] In some embodiments, multiple types of candidate actions in the action space include: generating response data, which instructs the large language model to generate response data fragments based on the context data to be processed; performing retrieval, which instructs a pre-set retrieval module to perform retrieval based on the context data to be processed to obtain external information; and completing response data generation, which instructs to stop iterative processing and calls a pre-set output module to extract response data of the input information generated by the large language model from the context data to be processed.

[0145] Here, the action of "generating response data" is used to invoke the generative capabilities of the large language model for single-step or multi-step autoregressive prediction. For example, when the decision network selects "generating response data" as the target action, it inputs the current context data to be processed as the input sequence into the large language model. The large language model captures long-distance semantic dependencies in the input sequence through a multi-layer self-attention mechanism and outputs the probability distribution of the next token or a sequence of tokens.

[0146] For example, large language models transform contextual data into dimensions of... The embedding matrix, where, The sequence length of the context data. This represents the hidden layer dimension (e.g., 4096). The embedding matrix is ​​processed by the Transformer decoder layer and mapped to the vocabulary dimension through a linear head layer. (e.g., 50000 dimensions) yields the Logits vector. The Logits vector is then processed using the Softmax function to obtain the word probability distribution. The tokens in the generated response data fragments are determined based on the token probability distribution. For example, the token with the highest probability in the token probability distribution is selected.

[0147] Here, the "perform retrieval" action aims to compensate for the limitations of implicit knowledge in large language models. When the decision network determines that the current context requires factual verification or the addition of new knowledge, it selects "perform retrieval" as the target action.

[0148] For example, the process of performing a "retrieval" includes: first, using a query generator to generate one or more natural language queries based on the context data to be processed, or directly extracting key entities from the context; then, inputting the queries into a pre-set retrieval module (such as a dense retrieval module based on a vector database). The retrieval module calculates the similarity index between the query vector and the candidate document vectors, and returns the top-ranked documents (e.g., Top-K) based on similarity. Document fragments are used as external information.

[0149] This application does not limit the specific implementation of the retrieval module. Retrieval can also be performed by calling an external search engine API (such as keyword search). In this case, the processing logic includes extracting keywords, sending an HTTP request, parsing the returned HTML or JSON data to extract a snippet, and integrating the snippet as external information into the context data.

[0150] Here, the action of "completing response data generation" serves as the termination signal for the entire iterative decision-making process. When the decision network determines that the currently generated response data fragment has completely answered the input information (such as the user's question), or that the current context data already includes a clear end marker (EOS Token), the decision network will select "completing response data generation" as the target action. The control flow will then exit the iteration loop and no longer perform new response data generation or retrieval operations. Subsequently, the output module is called to parse the accumulated context data, stripping away content such as retrieved external information and dialogue history, and extracting only the response portion related to the input information as the final response data.

[0151] For example, extracting response data based on the context data to be processed involves string manipulation of the buffer. Assume the context data is stored in memory as a list: [System_Prompt, User_Input, Thought_Step1, Retrieved_Doc1, Thought_Step2, Answer_Part]. The output module locates the Answer_Part field by matching regular expressions or using predefined separator tokens. If the Answer_Part is scattered across multiple generation steps, the output module performs string concatenation, merging all fragments marked "FinalAnswer," and performs format cleaning (such as removing extra spaces and Markdown formatting) before finally outputting it to the interactive interface.

[0152] In other embodiments, the response data generation process also includes a confidence verification step. Before extracting the response data, the perplexity of the entire generation path is calculated, or the content's standardization (such as formatting) and relevance are scored using a reward model. If the score is below a preset threshold, the extraction operation is not performed; instead, a fallback mechanism is triggered, re-entering the generation process or outputting a default fallback response.

[0153] In some embodiments, after determining the target action from multiple types of candidate actions based on the evaluation value, the following processing can also be performed: executing the data processing instruction corresponding to the target action to obtain the processing result, wherein if the target action is to retrieve or generate response data, the context data to be processed is updated based on the processing result, and the updated context data is used as the context data to be processed in the next round of iteration; if the target action is to complete the generation of response data, the process is to stop the iteration and extract the response data generated by the large language model from the context data to be processed.

[0154] In some embodiments, executing data processing instructions corresponding to the target action to obtain processing results can be achieved in the following way: when the target action is to generate response data, a preset first prompt word and the context data to be processed are input into the large language model to obtain response data fragments, and the response data fragments are used as processing results; wherein, the first prompt word is used to instruct the large language model to generate response data fragments if it detects that external information needs to be obtained for answer generation during the process of generating response data fragments, and then generate response data fragments carrying the information to be retrieved tags, the information to be retrieved tags being used to characterize the content of the external information to be retrieved.

[0155] For example, executing data processing instructions corresponding to the target action and obtaining the processing result can be achieved as follows: When the target action is to generate response data, the large language model is configured in an "on-demand retrieval" generation mode. Preset first prompt words and context data to be processed are input into the large language model, where the first prompt words include format constraint instructions. Format constraint instructions require the large language model to determine its own knowledge boundaries in real time during the generation of response data fragments. If the large language model detects that the currently generated semantic content touches a knowledge blind spot or requires external data support, it generates a structured request containing predefined tags, i.e., "retrievable information tags." "Retrievable information tags" include "retrieval information summary" tags (e.g., represented as "[NEED_INFO]") and "retrieval keywords" tags (e.g., represented as "[QUERY]"). The "retrieval information summary" tag describes a summary (or key points) of the external information to be retrieved, and the "retrieval keywords" tag represents the query terms.

[0156] For example, the first prompt word can be configured as the following prompt word template: Please strictly use the following format only when external information is required: [NEED_INFO]: <Briefly describe the information you need> [QUERY]: <Suggested search keywords> Note: Each answer is only allowed to contain one such tag; the rest will be generated directly as the answer text.

[0157] In this way, the response data fragments generated by the large language model not only include the response to the input information (such as the reply to the user's question), but also the control signals (i.e. the information to be retrieved) used to trigger subsequent retrieval actions.

[0158] For example, suppose the context data to be processed involves a query (input information) about the recent market performance of a certain "technology company A", such as "What big moves has technology company A made recently?".

[0159] When processing the context data to be processed, the large language model first generates a partial answer: "Technology company A recently released its latest flagship product, the X series, and has increased its investment in the field of artificial intelligence." Subsequently, the large language model recognized that because the training data was limited to a certain point in the past, it was impossible to obtain the company's latest financial data. At this point, based on the constraints of the first prompt word, the large language model output the following response data fragment: "In addition, regarding its latest financial situation, [NEED_INFO]: Revenue data for the latest quarter of Technology Company A [QUERY]: Financial report for the latest quarter of Technology Company A."

[0160] The response data fragment "Technology Company A recently released its latest flagship product, the X series, and has increased its investment in the field of artificial intelligence. In addition, regarding its latest financial situation, [NEED_INFO]: Technology Company A's latest quarterly revenue data [QUERY]: Technology Company A's latest quarterly financial report" is the processing result.

[0161] In some embodiments, inputting a preset first prompt word and context data to be processed into a large language model to obtain a response data fragment can be achieved in the following way: calling the large language model to generate response text based on the first prompt word and context data to be processed, and detecting the number of lexical units generated by the large language model; when the number of lexical units reaches a preset threshold, instructing the large language model to stop generating response text, and combining the currently generated lexical units into a response data fragment.

[0162] In some embodiments, inputting a preset first prompt word and context data to be processed into a large language model to obtain a response data fragment can be achieved in the following way: calling the large language model to generate response text based on the first prompt word and context data to be processed, and detecting the number of tokens generated by the large language model in real time; when the number of tokens reaches a preset token number threshold (e.g., 64 or 128 tokens), or when a specific end character (such as a newline character or a marker end character) is detected, instructing the large language model to stop generating response text, and combining the currently generated tokens into a response data fragment.

[0163] In some embodiments, when the response data fragment is used as the processing result, updating the context data based on the processing result can be achieved by appending the response data fragment to the end of the context data to be processed to form the updated context data.

[0164] For example, if the total length of the context data exceeds the preset window limit of the large language model, a sliding window strategy is adopted to discard the earliest historical data of the dialogue history in the context data.

[0165] By constructing a closed-loop iterative context update mechanism and a prompt-word-based "on-demand retrieval" mode, the large language model can perceive its own knowledge boundaries in real time during the generation process and actively generate structured instructions carrying the information to be retrieved to make up for information gaps. This mechanism effectively avoids the waste of resources caused by model illusion and invalid retrieval, and achieves fine-grained control over the generation rhythm through the threshold of the number of lexical units, thereby significantly improving the accuracy, coherence and controllability of complex task processing.

[0166] In some embodiments, executing data processing instructions corresponding to the target action to obtain processing results can also be achieved in the following ways: when the target action is to perform a search, a pre-set search module is invoked to search an external knowledge base based on the context data to be processed, external information is obtained, and the external information is used as the processing result.

[0167] Continuing with the previous example, executing data processing instructions corresponding to the target action and obtaining the processing result can also be achieved in the following way: When the target action is to perform a retrieval, the response data fragments generated by the large language model in the previous iteration are first parsed. For example, the text content following the tag of the information to be retrieved in the response data fragment is identified and extracted, and used as the retrieval content. Subsequently, a pre-set retrieval module (such as an internal enterprise knowledge base interface or an internet search engine) is called to perform a retrieval in the external knowledge base based on the search keywords. The retrieval module returns relevant document fragments or webpage summaries based on the keyword matching degree, and uses this external information as the processing result of this retrieval action.

[0168] For example, if the response data fragment output by the previous large language model includes "[QUERY]: Date of new product launch of a certain technology company A in 2024", the search module will directly initiate a query with "Date of new product launch of a certain technology company A in 2024" as the keyword and obtain a list of search results including the specific date.

[0169] In some embodiments, when external information is used as the processing result, updating the context data based on the processing result can be achieved in the following way: by using a large language model, the external information is fused into the context data to be processed to obtain the updated context data, and the updated context data is used as the context data to be processed in the next round of iteration.

[0170] For example, the retrieved external information is structured and encapsulated using a large language model or pre-defined template processing logic, and then integrated into the context data to be processed to obtain updated context data.

[0171] For example, to ensure that the large language model can accurately distinguish between "dialogue history", "currently retrieved external information" and "generated response data fragments", the updated context data can be constructed following a pre-defined structured template.

[0172] A structured template can include the following information fields: System Prompt, Dialogue History, Retrieval Observation, User Query, and Generated Response.

[0173] For example, a structured template can be represented as: [System Prompt] (Prompt Field) You are a smart hot topic information assistant.

[0174] [Dialogue History] User: ... Assistant: ... [Retrieval Observation] (Retrieval Observation Domain) --- Search Result 1 --- {Fill in the content of external information fragment 1 here} --- Search Result 2 --- {Fill in the content of external information fragment 2 here} ... [User Query] (Current Question Domain) User: {question} [Generated Response] (Content field already generated) Assistant: {Generated answer fragment}.

[0175] When updating the context data, the retrieved external information is populated into the [Retrieval Observation] area. It's important to note that the large language model's generation strategy is configured in "incremental generation" mode. That is, in the next iteration, the large language model will continue generating subsequent lexical units from the end of the [Generated Response] field based on the updated context data, rather than regenerating existing response data fragments. This approach maintains the semantic coherence of the response data, avoids repeatedly generating existing content, and significantly reduces computational overhead.

[0176] By constructing an execution mechanism capable of parsing retrieval intent and calling external knowledge bases, and combining structured context templates with incremental generation strategies, the efficient integration of external information and seamless continuation of generation tasks are achieved. While breaking the knowledge closure of large language models, it also avoids the repeated calculation of already generated content, significantly reduces inference overhead, and ensures the semantic coherence of long text generation.

[0177] Through steps 101 and 102, multiple iterative processes are performed. In each iteration, the context data to be processed is encoded with state features and mapped with features. Based on the obtained evaluation value, the target action is determined from the action space. This achieves dynamic control of the processing behavior of the large language model according to the current context state. It can adaptively select the target action according to the real-time needs during the processing, improving the flexibility and targeting of the input information processing. At the same time, by using the context data updated based on the previous iteration to drive the subsequent iteration processing until the target action indicates the completion of response data generation, it is ensured that the response data finally extracted from the context data to be processed is based on the progressively improved information state and dynamically optimized action path, effectively improving the accuracy and generation quality of the response data.

[0178] To more clearly illustrate the application of the large language model data processing method provided in this application embodiment in real-world scenarios, the following explanation is based on a complete process example.

[0179] For example, suppose the user input (input information) is: "Compare the market performance of electronic product A and electronic product B this year."

[0180] 1. Initial State Construction and Encoding Upon receiving user input, initial context data is constructed. The components of the context data are then encoded using a pre-defined feature extractor (e.g., a pre-trained language model based on the Transformer architecture).

[0181] Input information encoding: Taking "Compare the market performance of electronic product A and electronic product B this year" as input information, it is encoded as a 768-dimensional vector. .

[0182] Dialogue history encoding: Since this is the first round of dialogue, the dialogue history is empty. It is encoded as an all-zero vector or a specific empty sequence vector. (The dimension is also 768).

[0183] Answer encoding generated: Generation has not yet begun; the encoding is a 768-dimensional vector of all zeros. .

[0184] Search frequency characteristics: Initial search frequency is 0, encoded as a scalar or one-dimensional vector. .

[0185] State feature synthesis: The above vectors are concatenated on the feature channels to obtain a 2305-dimensional vector. initial state characteristics .

[0186] 2. Initial Decision and Search Execution Initial state characteristics Input a pre-trained decision network (e.g., a DQN network).

[0187] The decision network outputs the evaluation value (Q-value) of each candidate action in the action space based on the current state. An example output is shown below: ; ; .

[0188] Based on the maximum value strategy (Argmax), the current target action is determined to be "execute retrieval".

[0189] Next, the "Execute Search" action is performed, and the number of searches becomes 1.

[0190] Since no intermediate response results have been generated yet, keywords can be extracted directly based on the input information, or the input information itself can be used as the query terms: "Electronic Product A, Electronic Product B, Market Performance This Year".

[0191] The system invokes a pre-defined search module to retrieve the K most relevant document fragments (e.g., 5) from an external knowledge base. Simultaneously, it updates the search count counter to 1.

[0192] After formatting these 5 document fragments, populate them into the [Retrieval Observation] field of the context data.

[0193] 3. Model generation and dynamic retrieval triggering The updated contextual data (containing the initial retrieval results) is re-encoded and fed into the decision network. At this point, with the addition of external information, the decision network selects "generating response data" as the target action.

[0194] The large language model begins generating a response (response data fragment) based on contextual data populated with external information retrieved from the search: "Based on market data from the first half of this year, electronic product A and electronic product B each exhibited distinct characteristics in their performance:" 1. Market share: Electronic product A maintains a leading position in the global market, with a market share of approximately XX%... Electronic product B is performing strongly in a specific regional market. At this point, the large language model performs self-knowledge boundary detection based on the instructions in the preset first prompt. The model detects a lack of precise data regarding the specific sales volume of "electronic product B," therefore it automatically pauses regular text generation and instead generates the following tags: "[NEED_INFO]: Specific sales data for electronic product B in a specific region [QUERY]: Latest sales statistics for electronic product B series".

[0195] The context data is updated again based on the generated answer and is also re-encoded. At this point, the state features change (the retrieval count is 1, and answer features (i.e., response features) have been generated). This includes the information in the aforementioned information to be retrieved markers. The decision network re-evaluates based on the updated state features, and since a clear retrieval intent marker is detected, it determines with high probability that the target action is "execute retrieval".

[0196] Next, the search action is executed again, and the search module detects the specific [NEED_INFO] and [QUERY] tags. The content after the [QUERY] tag, "Latest Sales Statistics of Electronic Product B Series," is extracted as the search keyword for precise retrieval, obtaining document fragments containing specific sales data.

[0197] The retrieval module returns new external information, which is then appended to or updated in the [RetrievalObservation] field of the context data. The retrieval count counter is updated to 2.

[0198] 4. Generation, fusion, and completion The context data was updated again, now including the latest evidence regarding sales of "Electronic Product B".

[0199] The large language model is invoked to continue generation from the point of last interruption (in-context continuation) and integrates newly retrieved information: "...Since its release, the latest series of electronic products B has achieved cumulative sales of over XX million units in this region (data is supplemented here based on newly retrieved information)."

[0200] 2. Product strategy: Electronic product A focuses on a high-end ecosystem... Electronic product B, on the other hand, made a comeback through technological innovation... 3. Future Outlook: ...”.

[0201] During this process, the large language model dynamically integrates the information retrieved step by step, ensuring the accuracy and coherence of the content.

[0202] Suppose the large language model completes the generation of all content and outputs the end-of-content marker (EOS). Alternatively, after evaluating the current state (the generated content is semantically complete and there are no new retrieval tags), the decision network outputs "Response data generation complete" with the highest Q-value (e.g., 0.9), then stops iterative processing, calls the output module to extract the final generated complete text, and returns it to the user interface.

[0203] Through the above processing, the embodiments of this application realize the dynamic switching from "macro-level problem retrieval" to "on-demand retrieval of micro-details during the generation process", which significantly improves the logical coherence and factual accuracy in complex information environments.

[0204] The following will describe the training method of the decision network provided in the embodiments of this application, with the server as the execution subject, by combining the exemplary application and implementation of the server provided in the embodiments of this application.

[0205] See Figure 4 , Figure 4 This is a flowchart illustrating the training method for the decision network provided in this application embodiment, which will be combined with... Figure 4 The steps shown are explained.

[0206] In step 401, an initial context data sample is constructed based on the input information sample to be responded to by the large language model.

[0207] In some embodiments, the input information sample may be a user query extracted from historical logs or a question from a pre-built training dataset. The construction of the initial context data sample is similar to that described in step 101 above, and will not be repeated here.

[0208] In step 402, multiple iterative processes are performed. In the first iteration, the initialized context data sample is used as the context data sample to be processed. In subsequent iterations, the updated context data sample based on the previous iteration is used as the context data sample to be processed. Each iteration includes: encoding the state features of the context data sample to be processed to obtain state feature samples; performing feature mapping on the state feature samples through the decision network to be trained to obtain evaluation value samples of multiple types of candidate actions in the action space, and determining the target action sample from the multiple types of candidate actions based on the evaluation value samples; stopping the iterative process when the target action is to complete the generation of response data, and extracting the response data sample of the input information generated by the large language model from the context data to be processed.

[0209] In some embodiments, in the first iteration, the initialized context data sample is used as the context data sample to be processed, and in subsequent iterations, the updated context data sample based on the target action sample determined in the previous iteration is used as the context data sample to be processed. Each iteration process includes: encoding the state features of the context data sample to be processed to obtain state feature samples; performing feature mapping on the state feature samples to obtain evaluation value samples of multiple types of candidate actions in the action space; and determining the target action sample from the multiple types of candidate actions based on the evaluation value samples. If the target action sample is a completed response data generation for the input information sample, the iterative processing stops, and the response data sample of the input information sample generated by the large language model is extracted from the context data sample to be processed corresponding to the current round of iterative processing. If the target action sample is not a completed response data generation, the data processing instruction corresponding to the target action sample is executed to obtain the processing result sample, and the context data sample to be processed is updated based on the processing result sample.

[0210] In some embodiments, the context data sample to be processed includes an input information sample, a dialogue history sample, and a generated response data fragment sample; multiple types of candidate actions include performing a retrieval; the context data sample to be processed is encoded with state features to obtain a state feature sample, which can be achieved by: encoding the input information sample with features to obtain an input information feature sample, encoding the dialogue history sample with features to obtain a dialogue history feature sample, and encoding the response data fragment sample with features to obtain a response feature sample (see the description of step 201 above); obtaining the number of times the target action sample determined from multiple types of candidate actions is to perform a retrieval before the current iteration processing, and encoding the number of times with features to obtain a retrieval count feature sample (see the description of step 202 above); fusing the input information feature sample, the dialogue history feature sample, the response feature sample, and the retrieval count feature sample to obtain a state feature sample (see the description of step 203 above).

[0211] In some embodiments, determining a target action sample from multiple types of candidate actions based on evaluation value samples can be achieved in the following way: determining a first value based on the cumulative training steps of the decision network, wherein the first value is configured to decay according to a preset decay coefficient and a preset step period as the cumulative training steps increase, until it decays to a preset value and remains unchanged; generating a second value located in a preset value range; if the second value is less than the first value, selecting a candidate action from multiple types of candidate actions as the target action sample; if the second value is greater than or equal to the first value, selecting the candidate action corresponding to the evaluation value sample with the largest value as the target action sample.

[0212] For example, the process of determining the target action sample described above can be achieved through... -greedy - Greedy strategy implementation. The first value is the exploration rate. The second value is a generated random number. It can be expressed by formula (4): (4) in, This represents the current state feature sample. This indicates the output of the decision network for the state. Take candidate actions The evaluation value (Q-value). In the early stages of training, Setting it to a large value (e.g., 1.0) makes the decision network tend to randomly explore different action sequences to cover more of the state space; as the number of training steps increases, The value ε is gradually decayed to a smaller value (e.g., 0.1), causing the decision network to gradually tend to utilize the learned strategy and select the action with the largest Q-value as the optimal solution. For example, during the training phase, ε starts at 1.0 and is multiplied by a decay factor of 0.995 every 1000 steps, with a minimum value of 0.05. During the inference phase (i.e., the application phase after the decision network has been trained), ε is set to 0, and the action with the largest Q-value is always selected as the target action.

[0213] By forcing the decision network to extensively traverse the state space in the early stages of training to effectively avoid local optima traps, and then focusing on the in-depth mining and solidification of high-value actions in the later stages, the convergence efficiency and global optimization ability of the decision network in complex environments are significantly improved.

[0214] In step 403, a reward value is determined based on the response data sample, and the parameters of the decision network to be trained are updated based on the reward value.

[0215] In some embodiments, multiple types of candidate actions include performing a retrieval; determining a reward value based on response data samples can be achieved by: evaluating the quality of the response data samples to obtain quality parameter values; obtaining the total response time consumed in generating the response data samples, and obtaining the total number of times the target action sample performs a retrieval in multiple iterations; determining a first penalty value based on the ratio of the total response time to a preset time threshold, and determining a second penalty value based on the ratio of the total number of times to a preset number of times threshold; and fusing the quality parameter value, the first penalty value, and the second penalty value to obtain a reward value.

[0216] For example, the reward value can be calculated using formula (5): (5) in, Indicates the reward value. Indicates the value of the quality parameter. This represents the first penalty value. This represents the second penalty value. Indicates the total response time. Indicates the time threshold. This indicates the total number of searches. Indicates the threshold number of times. , For example, preset weights Set to 0.2, Setting it to 0.1 is used to adjust the model's trade-off strategy among response quality, response speed, and retrieval cost.

[0217] For example, see Figure 10 , Figure 10 This is a schematic diagram illustrating the principle of action evaluation and reward calculation in the decision network provided in this application embodiment, including two functional areas: action decision logic and reward calculation logic. In the action decision logic area, state features are fed as input vectors to the decision network (Q network). The Q network is responsible for mapping the high-dimensional state features to the action space and outputting the estimated evaluation value (Q value) for different candidate actions (including generating response data, performing retrieval, and completing response data generation). These evaluation values ​​then flow into the decision module, based on Argmax or... - Greedy strategy ( -Greedy) selects the target action for the current moment from candidate actions. In the reward calculation logic area, a multi-objective value evaluation system is constructed to guide the optimization of the decision network, aggregating three dimensions of evaluation indicators: the quality parameter value as a positive gain, and the total response time penalty (first penalty value) and the total number of retrievals penalty (second penalty value) as negative constraints. These three are integrated in the reward value calculation unit ( Figure 10 The symbols shown are After weighted aggregation is completed, the final reward value generated is applied to the decision network through the reverse link, triggering the parameter update process, thereby strengthening the network's decision-making ability in subsequent iterations.

[0218] Through the design of the above reward function, the embodiments of this application guide the decision network to learn the following strategy during training: while ensuring the quality of the answer (maximizing...). Under the premise of minimizing unnecessary search counts, and minimizing inference time This enables efficient and accurate data processing.

[0219] In some embodiments, updating the parameters of the decision network to be trained based on the reward value can be achieved by: calculating the loss value based on the reward value and a pre-set loss function (e.g., mean squared error loss, MSE Loss); calculating the gradient of the loss value with respect to the decision network parameters using the back propagation algorithm; and updating the weight parameters of the decision network using a gradient descent optimizer (e.g., Adam, SGD) to minimize the loss function until the decision network converges.

[0220] Taking DQN (denoted as Q network) as an example, the training of the decision network can be achieved in the following way.

[0221] The current Q-network (Online Q-Network, with the same configuration structure and parameters) has the same parameters. ) and TargetQ-Network (parameters denoted as) ).

[0222] Here, the current Q-network refers to the network that outputs a predicted evaluation value (Q-value) of an action based on state feature samples, thereby guiding action selection. During the training phase, the parameters... The main neural network iteratively updates in real time based on the calculated gradients; the target Q-network is used to assist in calculating the target Q-value (the sum of the immediate reward and the maximum expected reward in the next state) when calculating the loss function, and in order to maintain the stability of the training objective, the parameters... An auxiliary neural network that remains fixed for a certain period of time and synchronizes by copying parameters from the current Q-network at a preset frequency.

[0223] Before training begins, initialize the parameters of the current Q-network. and the parameters of the target Q network Initialize to Same as above. Simultaneously, initialize the Experience Replay Buffer. It is used to store historical data generated by the interaction between large language models and the environment, including an experience replay buffer. The capacity is set to 100,000 data entries. If the buffer is full, the earliest data is overwritten using the first-in-first-out principle.

[0224] During training, for each training round (Episode), the initial state is first obtained. (Corresponding to the state feature samples mentioned above). Subsequently, in each iteration, based on -greedy( The -greedy strategy starts from the current state. Select action from the corresponding candidate actions (Corresponding to the target action sample above). Here, The exploration rate is not fixed, but is configured to be dynamically adjusted as the training progresses: The initial value was set to 1.0 (i.e., completely random exploration), and as the number of training steps increased, The system exhibits exponential decay according to a preset decay coefficient (e.g., 0.995) until it decays to a preset minimum value of 0.05 and then remains constant, thus preserving a certain level of exploratory capability in the later stages of training.

[0225] In performing the action Afterwards, observe the new status and feedback. and instant rewards (award For the specific calculation method, please refer to the aforementioned explanation of formula (5). The current state will be used to calculate the result. Actions performed Rewards received and new status Quadruple As a piece of empirical data, it is stored in the empirical replay buffer. middle.

[0226] When the experience replay buffer When the amount of data stored in the buffer meets the preset conditions, it is retrieved from the buffer. A batch of data is randomly sampled for training in the current step. For example, the batch size is set to 64, meaning 64 historical interaction records are randomly selected from the buffer each time. This random sampling mechanism effectively breaks the temporal correlation between data, improving training stability.

[0227] For the sampled batch data, calculate the error between the target Q-value and the current predicted Q-value, and construct the loss function. The loss function can be constructed based on the mean squared error (MSE), and the calculation formula is shown in formula (6): (6) in, For the current Q network based on the current state and actions Output of the predicted evaluation value; The target Q-value (Target Label) is calculated based on the target Q-network. This is the discount factor, used to balance the weight of immediate rewards and future rewards; Indicates a new state The action that enables the target Q-network to output the maximum evaluation value.

[0228] Based on the above loss function, calculate the parameters of the current Q-network. The gradient is calculated, and gradient descent is performed using a preset optimizer (such as the Adam optimizer with a learning rate of 0.001) to update the parameters of the current Q-network. .

[0229] For example, to maintain the stability of the training objective, the parameters of the target Q-network are... Instead of updating in real-time with the current Q-network, a periodic synchronization mechanism can be used. For example, the target network can be set to update every 500 training steps. That is, every time the current Q-network completes 500 parameter updates, the parameters of the current Q-network can be synchronized. Copy to the target Q network and execute. The decision network learns, through the above process, how to select the optimal action in different contexts, thereby maximizing long-term cumulative rewards.

[0230] The hyperparameters used to train the Q-network can be set as shown in Table 1: Table 1

[0231] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0232] Retrieval-enhanced generation (RAG) is a significant advancement in natural language processing in recent years, aiming to overcome the inherent limitations of large language models (LLMs), such as poor knowledge timeliness, insufficient domain coverage, and severe illusion problems. The basic principle of RAG is to combine an external knowledge base with the language model, retrieving relevant information from the knowledge base and integrating it into the generation process to improve the accuracy and reliability of the responses.

[0233] With the widespread application of large language models, RAG technology has become the mainstream solution for improving the performance of large language models in information-intensive tasks, and is widely used in various scenarios such as enterprise knowledge management, customer service, and professional domain question answering.

[0234] In related technologies, RAG technology can be implemented as the following technical solutions: 1. Single fixed search mode It comprises two core components: a Retriever and a Generator. The workflow follows a linear "retrieval-generation" paradigm. For example, after receiving a user's input question, a retrieval query is generated directly based on the question. A single retrieval operation is performed to obtain relevant documents. The retrieval results are then concatenated with the original question and passed to the generator, which finally outputs the answer.

[0235] The limitation of this approach lies in its fixed "pre-retrieval" strategy: the retrieval process is only executed once at the initial stage of the question input, regardless of the complexity of the question or whether the generative model discovers new missing information during subsequent reasoning, and no subsequent remedial retrieval is triggered. This over-reliance on the initial retrieval results in poor robustness; if the initial retrieval is inaccurate or incomplete, the quality of the generated answer will significantly decrease.

[0236] 2. Multi-stage enhanced search mode To improve retrieval quality, the relevant technologies introduce a multi-stage processing mechanism, including a query rewriting module, an initial screening retrieval unit, a re-ranker, and a generator. The workflow includes: first, the original query is rewritten by understanding and expanding its intent; then, a broad initial search is performed to obtain candidate documents; subsequently, a refined re-ranking model is used to select the most relevant documents; and finally, the query is generated.

[0237] While such methods improve document relevance by optimizing pre-retrieval processing steps, they are essentially still within the "pre-generation retrieval" paradigm. That is, all retrieval actions (regardless of the number of internal stages) must be completed before the generator begins working, lacking the ability to dynamically identify new information needs and trigger additional retrievals based on contextual changes during the generation process.

[0238] In summary, the data processing methods for large language models in related technologies have the following limitations: 1. Lack of flexibility in retrieval decision-making (Static Decision-making). Related technologies often employ a "one-size-fits-all" strategy, executing a predetermined retrieval process for all input questions. This results in redundant and delayed responses for simple, common-sense questions; while for questions requiring complex reasoning, a single retrieval is often insufficient to provide a complete answer. Although some existing technologies attempt to use classifiers to predict questions (such as Adaptive-RAG), this classification is typically a one-time, static judgment that cannot adapt to the dynamic needs that emerge during question processing.

[0239] 2. Inability to dynamically supplement information during generation (Decoupling of Retrieval and Generation). In the existing architecture, the retrieval and generation phases are strictly decoupled and temporally separated. During the generation of long texts or multi-step inference by a large language model, if the model finds that a specific detail (such as a particular data metric or date) is missing, it cannot "go back" to perform the retrieval because it has already entered the generation phase. This forces the model to forcibly generate information based on existing incomplete context, resulting in uncertain or vague answers, and even hallucinations.

[0240] 3. Inefficient resource allocation and difficulty in continuously optimizing retrieval strategies. In technologies supporting multi-hop retrieval, the number of retrieval rounds is usually preset and fixed (e.g., a fixed 3 retrieval iterations). This rigid setting makes it difficult to balance computational cost and response quality: some questions can be solved with just one retrieval, and fixing it to multiple times wastes computational resources; while other questions require more explorations. Furthermore, rule-based or simple classifier-based retrieval strategies mainly rely on human experience for adjustment, lacking feedback mechanisms based on generated results (such as reward signals in reinforcement learning) to automatically optimize their decision-making strategies, making it difficult to achieve continuous evolution with the accumulation of data.

[0241] In view of this, this application provides a data processing method for a large language model, which is described in detail below.

[0242] The data processing method for large language models provided in this application can be used in various scenarios such as intelligent question answering, intelligent assistants, and enterprise knowledge base retrieval, and is especially suitable for complex information service scenarios that require both response speed and answer depth.

[0243] For example, the data processing method for large language models provided in this application can be applied to hot topic question-and-answer and hot topic information assistant products. In such applications, users will ask various questions related to hot topics, and the question types exhibit a high degree of diversity: Simple fact-based questions: These questions point to a single, clear fact, such as "What are the big news stories today?" or "Who is this public figure?" These questions usually do not require complex reasoning and can be answered with a single retrieval or by the model's built-in knowledge.

[0244] Explanatory and analytical questions: These questions involve causal relationships or in-depth interpretations, such as "Why did this happen?" or "What long-term impact will this policy have?" These types of questions often require continuous supplementation with background information during the question generation process.

[0245] Comprehensive comparison questions: These questions span multiple entities or timelines, such as "What are the important developments in the technology field in the past week?" or "Compare the similarities and differences between two events." These questions are the most complex, requiring multiple jumps to retrieve different information sources during the generation process.

[0246] Due to the diverse types and varying degrees of complexity of questions, a fixed retrieval strategy in related technologies (i.e., performing a uniform retrieval process regardless of the difficulty of the question) can lead to problems such as "slow response time for simple questions" (waste of resources) and "incomplete answers to complex questions" (missing information). The dynamic retrieval mechanism of this application embodiment can automatically adjust the strategy according to the characteristics of the question and the real-time status during the generation process, realizing "retrieval when it should be retrieved, and generation when it should be generated," thereby significantly improving the overall user experience.

[0247] For example, see Figure 5 , Figure 5 This is a schematic diagram of the architecture of the data processing method for a large language model provided in the embodiments of this application, such as... Figure 5 As shown, a data processing system based on a large language model is presented, including the following functional modules, the specific functions of each module are described below: Input processing module: Responsible for receiving the question text input by the user, preprocessing the text, including but not limited to word segmentation, stop word removal, named entity recognition (NER) and other operations, so as to extract the basic feature information of the question and provide raw materials for subsequent state construction.

[0248] The state encoding module is responsible for converting various discrete information at the current moment into a vector representation that can be processed by a computer. For example, it jointly encodes the "current user question", "dialogue history up to the present moment", and "response fragments generated in the current round (if any)" to construct a state vector, which will serve as the input to the subsequent decision network.

[0249] A Q-Network is a neural network model based on deep reinforcement learning (such as the DQN architecture). It receives a state vector output by a state encoding module and evaluates it in the action space to determine the target action. For example, the Q-Network outputs a Q-value (quality value) corresponding to three candidate actions (perform retrieval, continue generation (corresponding to generating response data above), and complete the answer (corresponding to completing response data generation above)). The target action is determined based on the Q-value of each candidate action (e.g., selecting the candidate action with the largest Q-value as the target action).

[0250] Generation Module: The generation module is activated when the decision network selects the "Continue Generation" action. It invokes a Large Language Model (LLM) to generate the next segment of the answer based on the current context. Simultaneously, the generation module also has a knowledge gap detection function, capable of identifying whether there is insufficient information during the generation process and generating corresponding markers (such as "Knowledge Gap Mark" or "No Knowledge Gap Mark," where "Knowledge Gap Mark" corresponds to the "Information to be Retrieved Mark" mentioned above). This feedback is then sent to the state coding module to influence the next round of decision-making.

[0251] Retrieval Module: When the decision network selects the "Execute Retrieval" action, the retrieval module is activated. It will construct a specific retrieval query statement based on the currently identified knowledge gaps or key information, and retrieve relevant information from a pre-set external knowledge base or the Internet to obtain external documents or data fragments.

[0252] Context Update Module: Responsible for the fusion and updating of data streams. When the retrieval module obtains new information, or the generation module produces new text, the context update module integrates this new content (retrieval results or generation results) into the current context, updating the context data for use in the next iteration.

[0253] Output module: When the decision network selects the "Complete Answer" action, it indicates that the currently generated response content (answer) is complete and accurate. At this time, the output module is activated, stops iterative processing, formats and organizes the final generated response data, and returns it to the user.

[0254] It should be noted that the above Figure 5 The functional module division shown is mainly to clearly illustrate the data processing logic and information flow of the embodiments of this application, that is, the division at the logical architecture level, and does not constitute a limitation on specific hardware deployment or software code entities. Each module (such as the input processing module, state encoding module, generation module, context update module, etc.) can be an independent component based on different neural network models, algorithms, or interfaces (for example, calling a dedicated BERT model for state encoding, or calling a third-party search engine interface to implement a retrieval module), or it can be an externalization of the functions of the same large language model at different processing stages. In other words, by reusing the powerful general semantic understanding and generation capabilities of the large language model, and through different prompt word engineering or internal attention mechanisms, multiple functions such as input parsing, state representation, content generation, and context maintenance can be completed serially or in parallel within a single model, while the decision network acts as an external "controller," operating independently of the large language model, and scheduling these internal functions of the large language model by outputting control signals (action instructions) according to the current state.

[0255] The following section explains the specific implementation details of each of the above modules.

[0256] 1. Status coding module The state coding module is used to transform various discrete information in the current interactive environment into a continuous vector representation that can be processed by a computer, serving as the input basis for the decision network to evaluate subsequent actions.

[0257] Specifically, the state encoding module first acquires three dimensions of text information at the current moment: the first dimension is the question text currently input by the user (corresponding to the input information mentioned above); the second dimension is the dialogue history text up to the current moment (corresponding to the dialogue history mentioned above), for example, extracting the "user-system" dialogue content from the most recent N rounds (e.g., 3 rounds); the third dimension is the response fragments already generated by the large language model in the current round (corresponding to the generated response data fragments mentioned above), which is empty if the generation is in the initial stage. Then, a pre-trained long text encoding model (e.g., the BGE-large model) is used to encode the above texts respectively. Simultaneously, the number of retrievals performed is obtained, normalized, and used as the numerical feature of the fourth dimension. Finally, the question encoding vector, dialogue history encoding vector, generated fragment encoding vector, and retrieval count feature are concatenated to construct a high-dimensional state vector (corresponding to the state features mentioned above). The state vector can comprehensively represent the information demand level and contextual completeness faced by the current system.

[0258] For example, in this embodiment of the application, the BGE-large model can be used as the text encoder. The output vector of the BGE-large model has a dimension of 768. (State vector) Can be constructed as In the form of. Among them, This is a 768-dimensional vector obtained by text encoding the current problem. This is a 768-dimensional vector obtained by concatenating the history of the most recent 3 rounds of dialogue into a long text and then encoding it (if the history is less than 3 rounds, empty strings are added). This is a 768-dimensional vector encoded from the currently generated response fragment (initialized to a 768-dimensional zero vector if generation has not yet begun). This represents the numerical feature (1-dimensional) of the number of searches performed so far. By concatenating the above four components, a total dimension of 2305 is obtained (i.e., ...). The state feature vectors are directly input into the decision network to calculate the Q-values ​​of each candidate action.

[0259] 2. Decision-making network A decision network (Q-Network) is used to evaluate and select the best action policy based on the input state vector.

[0260] In some embodiments, the decision network can be constructed based on a deep Q-network (DQN) architecture. The internal structure of the decision network includes an input layer, several hidden layers (e.g., two fully connected layers with a ReLU activation function in between), and an output layer. The input layer receives the state vector output by the state encoding module. After feature extraction and nonlinear transformation by the hidden layers, the output layer outputs the Q-value (action value evaluation value) corresponding to each candidate action in the action space.

[0261] In some embodiments, the action space includes at least three discrete actions: Continue generating: Instructs the generation of the next text segment directly using the existing context; Execute Search: Instructs to pause generation and initiate external knowledge retrieval; Complete Answer: Indicates the end of the current answering process and outputs the result.

[0262] During the inference phase, the decision network employs a greedy strategy, selecting the action with the largest Q-value as the current action to be executed. Furthermore, to prevent infinite loops in the retrieval process, the decision network incorporates a rule masking mechanism: when the number of retrievals executed reaches a preset limit (e.g., 3 times), the Q-value corresponding to the "execute retrieval" action is forcibly set to negative infinity or masked before action selection, forcing the decision network to choose only from the remaining actions.

[0263] For example, the decision network can adopt a multilayer perceptron (MLP) structure, see Table 2: Table 2

[0264] The three neurons in the output layer correspond to the Q-value estimates of three actions: Q(s, continue generating), Q(s, perform retrieval), and Q(s, complete the answer).

[0265] For details on the specific training of the decision network, please refer to Formula (6) above and the relevant explanation in Table 1. They will not be repeated here.

[0266] 3. Generation Module The generation module is responsible for calling a large language model (LLM) to generate answers based on the current context. Its core lies in the integration of a knowledge gap identification mechanism, and the specific implementation process is as follows: When the generation module invokes LLM, it not only receives the context input but also injects specific instructions through system prompts to guide the model to self-examine during the generation process. When the model encounters nodes with missing information, uncertainty, or that require the latest data support during generation, the model is trained or prompted to stop forcibly generating ambiguous content and instead output specific knowledge gap markers.

[0267] For example, the generation module can pre-configure the following prompt templates to construct input instructions, thereby guiding the large language model to perform knowledge gap labeling: "You are a trending information assistant, answering users' questions. During the answering process, if you find yourself unsure about specific information or need the latest data, please use the following format to mark it: [NEED_INFO]: <Briefly describe the information you need> and [QUERY]: <Suggested search keywords>."

[0268] Note: Use this marker only when you truly need external information; for content you are confident about, simply answer directly; mark a maximum of one knowledge gap at a time.

[0269] Based on this, the "current dialogue context" and "user question" are filled into the template and input into the large language model. Based on the above instructions, when the large language model recognizes that external information is needed, it will output a text stream containing specific identifiers, such as a special text fragment in the form of "[NEED_INFO]:B brand's latest flagship model sales and [QUERY]:B brand flagship model sales 2024".

[0270] For example, the generation module can detect the output stream in real time. Once the above knowledge gap marker is detected, it determines that a knowledge gap exists and sends a signal to the large language model to pause the generation process and trigger a retrieval action. If the marker is not detected, it is considered normal generation, and the text fragments generated by the large language model are cached and used for subsequent updates.

[0271] 4. Search Module The retrieval module is activated in response to the "execute retrieval" command from the decision network or the "knowledge gap" signal detected by the generation module. Its main function is to construct query statements and retrieve supporting documents from external knowledge bases. In practice, the retrieval module first determines the search keywords (Query). If triggered by a knowledge gap, it directly extracts the content of the [QUERY] field output by the generation module; if triggered by insufficient initial state information determined by the decision network, it extracts keywords or rewrites the query based on the user's original question. Subsequently, the retrieval module uses the constructed Query to perform a search in a pre-built knowledge base (such as a vector database, inverted index, or internet search engine interface).

[0272] For example, the retrieval module is configured to support multi-source heterogeneous retrieval strategies to accommodate different types of information needs, such as: Vector database retrieval: The query is vectorized using embedding models such as BGE-large, and semantically similar documents are retrieved from vector libraries such as Faiss to capture deep semantic relationships; Full-text search: Use search engines such as Elasticsearch for precise keyword matching; Real-time Search API: When dealing with highly timely news or data, this API calls an internet search engine to retrieve the latest information. By default, the system prioritizes vector database retrieval and returns the top-N (e.g., 5) documents sorted by semantic relevance score from highest to lowest as a candidate set.

[0273] For example, after obtaining the list of candidate documents, the retrieval module performs the following processing: (1) Deduplication: Remove content that is highly repetitive with existing context; (2) Truncation: Keep the first 500 characters of each document; (3) Sorting: Sort by relevance score; (4) Concatenation: Concatenate the processed documents into the search result text.

[0274] In some embodiments, based on the data processing method for the large language model provided in this application, the decision network can adaptively plan different action execution sequences according to the complexity parameters of the input information, the state characteristics of the current context, and the preset action evaluation logic. Based on the frequency and triggering timing of the "perform retrieval" action included in the action sequence, the data processing flow of this application can be implemented in three retrieval modes: pure generation mode, single retrieval mode, and iterative retrieval mode. The following detailed description is provided in conjunction with specific implementation methods. (1) Pure generation mode In some embodiments, when the decision network evaluates the complexity of the input information to be low, or determines that the internal parameter knowledge of the large language model is sufficient to cover the knowledge points required for the input information, the data processing flow presents a pure generation mode.

[0275] In the initial iteration, the decision network evaluates the initial context data. If the input information is a common-sense question (e.g., "Hello, please introduce yourself"), or belongs to a high-confidence domain covered by the large language model's training data, the evaluation value (Q-score) of the "generate response data" action output by the decision network will be significantly higher than the evaluation value of the "perform retrieval" action. In this case, the target action is determined to be "generate response data". Subsequently, the large language model directly generates complete response data based on the input information, and no knowledge gap marker ([NEED_INFO]) is triggered during the generation process. In the next iteration, the decision network detects that the response is complete and determines "complete response data generation" as the target action. In this mode, the system does not trigger the external retrieval module, achieving a fast response with zero retrieval overhead, suitable for simple casual conversation or creative tasks.

[0276] (2) Single search mode In some embodiments, when the input information involves a specific fact query and the information requirement is relatively focused and does not require deep reasoning, the data processing flow presents a single retrieval mode.

[0277] In the initial iteration, the decision network identifies a knowledge gap in the large language model regarding the input information (e.g., regarding the latest weather, such as "What's the weather like in City A today?"). At this point, the "Execute Retrieval" action receives the highest evaluation value, and "Execute Retrieval" becomes the target action. Performing a retrieval update the context with the acquired external information. In subsequent iterations, the decision network evaluates the input based on the updated context (which now includes the required facts), and the "Generate Response Data" action receives the highest evaluation value. The large language model generates a complete answer based on the retrieval results. Finally, the decision network selects "Complete Response Data Generation." In this mode, the task is completed through a sequence of "one retrieval + one generation," supplementing necessary information while avoiding redundant queries, making it suitable for factual question-answering scenarios.

[0278] (3) Iterative retrieval mode In some embodiments, when the input information is highly complex, involving multi-hop reasoning, many-to-many comparisons, or long text generation, the data processing flow takes the form of an iterative retrieval mode.

[0279] For example, the input information is "Compare the financial highlights and future strategies of Company A and Company B in the second quarter of this year".

[0280] Phase 1 (Initial Retrieval): In the first iteration, the decision network determines that the problem is highly complex and selects "perform retrieval" as the target action to obtain the basic financial statement documents of companies A and B.

[0281] Phase Two (Attempt to Generate and Disrupt): After the context update, the decision network selects "Generate Response Data". The large language model begins generating comparative analysis. During the generation process, the large language model discovers that although there is financial data, it lacks specific details (knowledge gap) regarding "Company B's future strategy", and therefore generates an intermediate response fragment marked with [NEED_INFO].

[0282] Phase Three (Dynamic Remedial Retrieval): In the next iteration, the state feature encoding module captures the aforementioned knowledge gap markers, and the decision network accordingly increases the evaluation value of "Execute Retrieval" again, selecting "Execute Retrieval" as the target action. The knowledge gap is analyzed, and a second round of precise retrieval is initiated targeting "Company B's future strategy."

[0283] Phase Four (Generation, Fusion, and Completion): After acquiring new information and updating the context, the decision network again selects "Generate Response Data," and the large language model continues to complete the remaining report generation. The above "retrieval-generation-evaluation" cycle may be executed multiple times until sufficient information is available or a preset retrieval threshold is reached. In this mode, through multiple rounds of dynamic interaction, the problem of unclear information requirements or excessive information volume in complex problems is solved.

[0284] By adaptively switching between the three retrieval modes described above, the embodiments of this application can dynamically allocate computing resources according to the difficulty of the actual problem: minimizing response latency in pure generation mode, balancing efficiency and accuracy in single retrieval mode, and maximizing the depth and breadth of the answer in iterative retrieval mode, thereby significantly improving the overall performance of the system and the user experience.

[0285] 5. Context Update Module The context update module is responsible for maintaining and updating the data context input to the large language model, and is a key component for realizing the dynamic loop of "generating, retrieving, and updating simultaneously". Whenever the retrieval module returns new retrieval results, or the generation module produces new text fragments, the context update module will concatenate these new contents into the existing context according to a specific time sequence and structure to build an updated input prompt.

[0286] For example, this application embodiment constructs a multi-level contextual prompt structure, which may include five parts: [system instructions], [historical dialogues], [accumulated search information], [current question], and [generated answer fragments]. The specific organization is as follows: [System Commands]: Located at the forefront, these commands define the model's roles (such as "Hot News Assistant") and behavioral guidelines. [Historical Dialogue]: Contains records of past interactions between the user and the system; [Accumulated Search Information]: Used for dynamically storing external knowledge. After a search is performed, the returned document content is formatted (e.g., marked as --- Search Result N ---) and appended to this area, ensuring the context contains the latest external evidence; [Current Issue]: Users' immediate questions; [Generated Response Fragment]: Records the text content that the large language model has already output but has not yet finished before this retrieval was triggered.

[0287] For example, it can be represented as: [System Prompt]: You are a trending news assistant... [Dialogue with History]: User: ... Assistant: ... [Search Information]: Search result 1: {document content} Search result 2: {document content} ... [Current Issue]: User: {question} [Generated content]: {Generated answer fragment}}.

[0288] For example, based on the above structure, not only is dynamic information injection achieved, but also "seamless continuation" of the generation process is realized. When a retrieval is triggered due to a knowledge gap and a result is obtained, the context update module writes the new result into the [accumulated retrieval information] area and retains the content before the interruption in the [generated answer fragment] area. Subsequently, when the large language model is called again, the model is guided to continue predicting the next token from the end of the [generated answer fragment], rather than regenerating the entire answer.

[0289] In this way, when generating the next round of data, the large language model can both "see" the newly added external knowledge and maintain the logical connection with the previous text, thus continuing to generate subsequent content based on more complete information, ensuring the coherence, accuracy, and efficiency of the final answer.

[0290] 6. Output Module The output module is used to encapsulate the final result after the decision network selects the "complete answer" action. This includes, but is not limited to, the following processing: stopping the inference iteration of the large language model, splicing and merging all generated text fragments; formatting the answer according to preset format requirements (such as Markdown, JSON, etc.); and finally returning the processed complete answer to the client or downstream application.

[0291] The following section demonstrates the complete process of the data processing method of this application embodiment in a practical application through a "hot topic information Q&A" scenario.

[0292] Suppose a user asks: "Compare the market performance of brand A mobile phones and brand B mobile phones this year."

[0293] 1) Initial Status Assessment and First Round Decision The input processing module receives the question text input by the user. The state encoding module takes the question text, an empty dialogue history, an empty generated fragment, and the current retrieval count (0 times) as input to encode and generate an initial state vector. The decision network receives this state vector and calculates the Q-value of each action. Since the question involves specific market data comparison and the current context is empty, the decision network outputs the action with the highest Q-value, "Execute Retrieval".

[0294] 2) First-round retrieval and context update In response to the "execute search" action command, the search module extracts the keywords "Market performance of Brand A and Brand B mobile phones in 2024" based on the question, performs a search in the pre-built knowledge base, and obtains document fragments including a portion of the market share of both brands in a certain quarter. Subsequently, the context update module integrates the newly obtained document fragments into the current context, forming an updated context that includes the information from the first round of searches.

[0295] 3) Generating action decisions and trial generation The state encoding module reacquires the current update context and encodes it into a new state vector. The decision network evaluates the new state again, determines that there is now some supporting information, and therefore outputs the action with the highest Q value, "Continue generating".

[0296] In response to the "Continue generating" instruction, the generation module begins generating an answer based on the context, including the information from the first round of retrieval: "Based on data from the first half of 2024, Brand A maintains its leading position in the high-end market, with a market share of approximately... while Brand B is experiencing rapid growth in shipments..."

[0297] 4) Knowledge Gap Identification and Status Update During the generation module's operation, when the large language model generates the specific sales data for brand B, it identifies a lack of sales data for brand B's latest flagship model in the existing context. This triggers the knowledge gap identification mechanism, which outputs the following tags: [NEED_INFO]: Sales data for brand B's latest flagship model [QUERY]: Sales data for brand B's flagship model in 2024.

[0298] At this point, the generation module pauses generation and feeds back the knowledge gap marker to the state encoding module. The state encoding module encodes the feature "a knowledge gap exists" into the current state vector.

[0299] 5) Dynamic remedial retrieval decision-making and execution The decision network receives a state vector containing "knowledge gap" features, and after evaluation, outputs the action with the highest Q value, "execute retrieval".

[0300] In response to the "Execute Search" command, the search module parses the [Query] "B brand flagship sales 2024" in the notch mark and initiates a second round of precise search, obtaining the latest news data from external data sources.

[0301] 6) Secondary context update and decision-making for continued generation The context update module integrates the newly retrieved news data into the current context. The state encoding module updates the state vector again (this time the retrieval count is updated to 2, and the context information is richer). After evaluation, the decision network outputs the action "Continue generating".

[0302] The generation module resumes generating text from where it left off: "...especially its newly released flagship model, which achieved sales of over XX0,000 units in its first month, significantly boosting overall performance. In contrast, Brand A..."

[0303] 7) Complete the answer decision The generation module completed the writing of the answer and no new knowledge gaps were detected. The state encoding module updated the final state, and the decision network evaluated it, determining that the answer was complete, logically sound, and required no further information, and output the action "Answer Completed".

[0304] In response to the "Complete Answer" command, the output module is activated, which organizes and formats the final generated response text and sends it back to the user.

[0305] 8) Reward Calculation and Strategy Optimization (Training Phase) In addition, during the training phase of the decision network, in order to continuously optimize the network's strategy (i.e., to retrieve at the appropriate time and generate at the appropriate time), a reward calculation step is performed after a complete question-and-answer interaction is completed.

[0306] For example, the overall reward value is calculated based on the final performance of this interaction. The specific process is as follows: Response quality score acquisition: The accuracy and fluency of the final response are scored through manual evaluation or a higher-order reward model. Assuming the quality score is obtained... It is 0.85; Efficiency metrics statistics: Record the total response time of this interaction. (e.g., 3.2 seconds) and total number of searches (For example, twice); Comprehensive reward calculation: The final reward is calculated based on the preset reward function (formula (5)).

[0307] Experience storage: storing the state sequence, action sequence, and calculated final reward during this interaction. The samples are encapsulated as experience samples and stored in the experience replay pool. They are then used to update the parameters of the decision network through reinforcement learning algorithms, enabling it to make better decisions in future interactions.

[0308] The data processing method for large language models provided in this application has the following beneficial effects: 1. It has enabled dynamic and intelligent retrieval decisions, improving resource utilization efficiency.

[0309] Unlike the rigid, fixed retrieval patterns in related technologies, the embodiments of this application utilize a decision network trained by reinforcement learning, which can intelligently determine whether a retrieval operation needs to be performed based on the complexity of the problem and the current generation state. For simple problems, it can choose to generate directly, avoiding unnecessary retrieval overhead and response latency; for complex problems, it can choose to perform multiple retrievals, realizing on-demand allocation of computing resources and achieving an optimal balance between response speed and resource consumption.

[0310] 2. A "knowledge replenishment" mechanism was established during the generation process, which significantly improved the accuracy and completeness of the answers.

[0311] By identifying knowledge gaps and using a dynamic retrieval loop, this application breaks the limitation of "separation of retrieval and generation time sequence" in related technologies. It can perform retrieval operations in real time for missing specific details during the process of generating answers, effectively solving the illusion problem caused by missing information in the model. Especially in complex question-and-answer scenarios that require long-chain reasoning or comparison of multi-source information, it can provide more accurate and detailed answers.

[0312] 3. A self-evolving decision optimization closed loop has been constructed.

[0313] Based on a reinforcement learning framework, this embodiment constructs a comprehensive reward function that incorporates answer quality, response time, and retrieval cost. Through continuous accumulation of interactive data, the parameters of the decision network are continuously updated using gradient descent, automatically optimizing the retrieval strategy. This means that as usage increases, the corresponding decision logic becomes increasingly sophisticated, adaptively determining the appropriate level of retrieval support for different types of questions, without requiring frequent manual adjustments to complex rules and strategies.

[0314] To verify the technical effectiveness of the large language model data processing method provided in this application embodiment in a real-world application scenario, online testing was conducted in a hot topic question-and-answer scenario. The testing period was [duration missing]. The study covered approximately 15,000 question-and-answer samples. The method described in this application was compared with methods in related technologies employing a fixed single-response strategy (hereinafter referred to as the "benchmark method"). The experimental results are as follows: 1. In terms of response efficiency Experimental data show that the data processing method for large language models provided in the application embodiment can significantly reduce the overall average response time. Specifically, the overall average response time is reduced from 2.1 seconds to 1.4 seconds, a reduction of approximately 33%. The performance is as follows for problems of different complexities: Simple questions (approximately 45%): Since the decision network makes intelligent judgments without the need for retrieval, the average response time has been significantly reduced from 2.1 seconds to 0.8 seconds, a reduction of approximately 62%, greatly improving the user experience; Medium-sized problems (accounting for approximately 40%): The average response time is basically the same as the benchmark method (1.9 seconds for the method in this application embodiment, and 2.0 seconds for the benchmark method). Complex questions (approximately 15%): Although the average response time increased from 2.3 seconds to 3.1 seconds, this time consumption was due to the execution of multiple rounds of iterative retrieval to obtain more comprehensive information, resulting in a significant improvement in the completeness of the answer.

[0315] 2. Regarding the quality of the answers The experiment used a combination of manual evaluation (random sampling of 500 entries) and model evaluation to evaluate the generated results. The data shows that the data processing method of the large language model provided in the application embodiment is superior to the benchmark method in many indicators: Satisfaction with responses: Satisfaction with manual assessments increased from 72% to 84% compared to the baseline method; Fact accuracy: improved from 81% to 89% of the baseline method; Information completeness: Especially when dealing with complex problems that require support from multiple aspects of information, thanks to the iterative retrieval mechanism, the information completeness score increased from 58% to 79%, indicating that the data processing method of the large language model provided in the application embodiment can obtain more comprehensive information and effectively solve the defect of incomplete answers to complex problems.

[0316] 3. In terms of resource optimization The data processing method for large language models provided in this application achieves on-demand allocation of computing resources through intelligent decision-making, avoiding unnecessary retrieval overhead. Total number of retrieval calls: Since simple questions no longer trigger retrieval actions, the overall number of retrieval calls is reduced by approximately 38% compared to the baseline method; Average number of searches: The average number of searches per question and answer session is reduced from 1 to 0.72 by the method in this embodiment of the application; Search relevance: Thanks to the knowledge gap identification mechanism, the generated query terms are more targeted, and the matching degree between search results and user questions (search relevance) is improved by about 15%.

[0317] In summary, the experimental data fully validates the significant technical effects of the large language model data processing method provided in this application embodiment in improving response speed, enhancing answer quality, and optimizing resource allocation.

[0318] The following description continues to illustrate the exemplary structure of the data processing device 133 for the large language model provided in this application embodiment as a software module. In some embodiments, such as Figure 2A As shown, the software modules in the data processing device 133 storing the large language model in memory 130-1 may include: The first data processing module 1331 is used to construct initial context data based on the input information to be responded to by the large language model.

[0319] The second data processing module 1332 is used to perform multiple iterative processes. In the first iterative process, the initialized context data is used as the context data to be processed. In subsequent iterative processes, the updated context data based on the previous iterative process is used as the context data to be processed. Each iterative process includes: encoding the context data to be processed with state features to obtain state features; performing feature mapping on the state features to obtain evaluation values ​​of multiple types of candidate actions in the action space, and determining the target action from the multiple types of candidate actions based on the evaluation values; if the target action is to complete the generation of response data, stopping the iterative process, and extracting the response data of the input information generated by the large language model from the context data to be processed.

[0320] In some embodiments, the context data to be processed includes the input information, dialogue history, and generated response data fragments; the multiple types of candidate actions include performing a retrieval; the second data processing module 1332 is further configured to perform feature encoding on the input information to obtain input information features, perform feature encoding on the dialogue history to obtain dialogue history features, and perform feature encoding on the response data fragments to obtain response features; obtain the number of times the target action determined from the multiple types of candidate actions before the current iteration processing is the performance of the retrieval, and perform feature encoding on the number to obtain retrieval count features; and fuse the input information features, the dialogue history features, the response features, and the retrieval count features to obtain the state features.

[0321] In some embodiments, the second data processing module 1332 is further configured to remove the execution retrieval from the action space if the number of times equals a number of times threshold; wherein the number of times threshold is determined by: evaluating the complexity of the input information to obtain a complexity parameter of the input information; querying the number of times threshold range corresponding to the complexity parameter from a preset first mapping table; and determining the number of times threshold based on the number of times threshold range corresponding to the complexity parameter, wherein the first mapping table includes multiple different correspondences between the complexity parameters and different number of times threshold ranges.

[0322] In some embodiments, the second data processing module 1332 is further configured to: if the complexity parameter is less than or equal to a first complexity threshold, use a preset initial value as the number of iterations threshold; if the complexity parameter is greater than the first complexity threshold and less than a second complexity threshold, map the complexity parameter to the number of iterations threshold using a preset linear growth function, wherein the number of iterations threshold is greater than the initial value and less than a preset target value; if the complexity parameter is greater than or equal to the second complexity threshold, use the target value as the number of iterations threshold.

[0323] In some embodiments, the second data processing module 1332 is further configured to remove the execution retrieval from the action space if the number of times equals a number of times threshold; wherein the number of times threshold is determined by: evaluating the complexity of the input information to obtain a complexity parameter of the input information; classifying the input information into a domain to determine the target domain to which the input information belongs, and querying a fault tolerance parameter corresponding to the target domain from a preset second mapping table, wherein the second mapping table includes a correspondence between multiple different target domains and different fault tolerance parameters; obtaining the service priority corresponding to the input information, and determining the baseline retrieval number corresponding to the service priority; determining an adjustment coefficient based on the fault tolerance parameter and the complexity parameter, and adjusting the baseline retrieval number based on the adjustment coefficient to obtain the number of times threshold.

[0324] In some embodiments, the second data processing module 1332 is further configured to input the state features into a pre-trained decision network, wherein the decision network includes a cascaded feature extraction layer and an output layer; the state features are subjected to a first feature mapping through the cascaded feature extraction layer to obtain a mapped feature; the mapped feature is subjected to a second feature mapping through the output layer to obtain an action evaluation vector, wherein the number of elements in the action evaluation vector is consistent with the number of action types in the action space, and the values ​​of the elements in the action evaluation vector respectively represent the evaluation value of each candidate action.

[0325] In some embodiments, the context data to be processed includes dialogue history; the second data processing module 1332 is further configured to calculate the semantic correlation between the dialogue history and the input information, and detect historical retrieval records in the dialogue history that have performed retrieval operations to obtain detection results; based on the semantic correlation and the detection results, determine the confidence level of the dialogue history; before inputting the state features into the pre-trained decision network, the method further includes: fusing the confidence level of the dialogue history with the state features to obtain fused state features, and transferring the processing of inputting the state features into the pre-trained decision network based on the fused state features.

[0326] In some embodiments, the candidate actions of multiple types in the action space include execution retrieval; the second data processing module 1332 is further configured to determine the action weighting coefficient for the execution retrieval based on the dialogue history confidence, adjust the element value corresponding to the execution retrieval in the action evaluation vector through the action weighting coefficient, and determine the target action based on the adjusted action evaluation vector, wherein the dialogue history confidence is negatively correlated with the evaluation value of the execution retrieval.

[0327] In some embodiments, the second data processing module 1332 is further configured to execute a data processing instruction corresponding to the target action to obtain a processing result. If the target action is to perform a retrieval or to generate response data, the context data to be processed is updated based on the processing result, and the updated context data is used as the context data to be processed in the next round of iteration. If the target action is to complete the generation of response data, the process proceeds to stopping the iteration and extracting the response data of the input information generated by the large language model from the context data to be processed.

[0328] In some embodiments, the second data processing module 1332 is further configured to, when the target action is to generate response data, input a preset first prompt word and the context data to be processed into the large language model to obtain a response data fragment, and use the response data fragment as the processing result; wherein, the first prompt word is used to instruct the large language model, during the process of generating the response data fragment, if it detects that external information needs to be obtained for answer generation, then generate a response data fragment carrying a tag for the information to be retrieved, and the tag for the information to be retrieved is used to characterize the content of the external information to be retrieved.

[0329] In some embodiments, the second data processing module 1332 is further configured to invoke the large language model to generate answer text based on the first prompt word and the context data to be processed, and to detect the number of lexical units generated by the large language model; if the number of lexical units reaches a preset lexical unit number threshold, instruct the large language model to stop generating the answer text and combine the currently generated lexical units into the response data fragment.

[0330] In some embodiments, the second data processing module 1332 is further configured to, when the target action is to perform retrieval, perform a retrieval in an external knowledge base based on the context data to be processed to obtain external information, and use the external information as the processing result; through the large language model, integrate the external information into the context data to be processed to obtain updated context data, and use the updated context data as the context data to be processed in the next round of iteration processing.

[0331] The following description continues to illustrate the exemplary structure of the training device 134 for the decision network provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2B As shown, the software modules in the training device 134 of the decision network stored in memory 130-2 may include: The data processing module 1341 is used to construct an initial context data sample based on the input information sample to be responded to from the large language model.

[0332] In some embodiments, the data processing module 1341 is further configured to perform multiple iterative processes, wherein in the first iterative process, the initialized context data sample is used as the context data sample to be processed, and in subsequent iterative processes, the context data sample updated based on the previous iterative process is used as the context data sample to be processed; each iterative process includes: encoding state features of the context data sample to be processed to obtain state feature samples; performing feature mapping on the state feature samples through a decision network to be trained to obtain evaluation value samples of multiple types of candidate actions in the action space, and determining a target action sample from the multiple types of candidate actions based on the evaluation value samples; if the target action is to complete the generation of response data, stopping the iterative process, and extracting the response data sample of the input information generated by the large language model from the context data to be processed; Training module 1342 is used to determine a reward value based on the response data sample and to update the parameters of the decision network to be trained based on the reward value.

[0333] In some embodiments, the data processing module 1341 is further configured to determine a first value based on the cumulative training steps of the decision network, wherein the first value is configured to decay according to a preset decay coefficient and a preset step period as the cumulative training steps increase, until it decays to a preset value and remains unchanged; generate a second value located in a preset value range; if the second value is less than the first value, select a candidate action from the plurality of candidate actions as the target action sample; if the second value is greater than or equal to the first value, select the candidate action corresponding to the evaluation value sample with the largest value as the target action sample.

[0334] In some embodiments, the plurality of candidate actions include performing a retrieval; the training module 1342 is further configured to evaluate the quality of the response data sample to obtain a quality parameter value; obtain the total response time consumed in generating the response data sample, and obtain the total number of times the target action sample is the retrieval performed in the multiple iterations; determine a first penalty value based on the ratio of the total response time to a preset time threshold, and determine a second penalty value based on the ratio of the total number of times to a preset number of times threshold; and fuse the quality parameter value, the first penalty value, and the second penalty value to obtain the reward value.

[0335] This application provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the large language model data processing method described in this application embodiment, or the decision network training method described in this application embodiment.

[0336] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the data processing method for the large language model provided in this application, or the training method for the decision network provided in this application, for example... Figure 3A The data processing method of the large language model shown Figure 4 The training method for the decision network is shown.

[0337] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0338] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0339] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0340] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0341] In summary, through the embodiments of this application, by performing multiple iterative processes, encoding and mapping the state features of the context data to be processed in each iteration, and determining the target action from the action space based on the obtained evaluation value, dynamic control of the processing behavior of the large language model based on the current context state is achieved. The target action can be adaptively selected according to the real-time needs during the processing, improving the flexibility and targeting of the input information processing. At the same time, by using the context data updated based on the previous iteration to drive subsequent iterations until the target action indicates the completion of response data generation, it is ensured that the response data finally extracted from the context data to be processed is based on the progressively improved information state and dynamically optimized action path, effectively improving the accuracy and generation quality of the response data.

[0342] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.< / bos> < / bos> < / start> < / bos>

Claims

1. A data processing method for a large language model, characterized in that, The method includes: Initialized context data is constructed based on the input information to be responded to from the large language model; Multiple iterations are performed, wherein in the first iteration, the initialized context data is used as the context data to be processed, and in subsequent iterations, the updated context data based on the previous iteration is used as the context data to be processed; each iteration includes: The context data to be processed is encoded with state features to obtain state features; the state features are then mapped to obtain evaluation values ​​of multiple types of candidate actions in the action space; and the target action is determined from the multiple types of candidate actions based on the evaluation values. If the target action is to complete the generation of response data, the iterative process is stopped, and the response data of the input information generated by the large language model is extracted from the context data to be processed.

2. The method according to claim 1, characterized in that, The context data to be processed includes the input information, dialogue history, and generated response data fragments; the multiple types of candidate actions include performing a retrieval. The process of encoding the context data to be processed to obtain state features includes: The input information is feature-encoded to obtain input information features, the dialogue history is feature-encoded to obtain dialogue history features, and the response data fragment is feature-encoded to obtain response features; Prior to the current iteration, the target action determined from the multiple types of candidate actions is the number of times the retrieval is performed, and the number is feature-encoded to obtain the retrieval count feature; The input information features, the dialogue history features, the response features, and the retrieval frequency features are fused to obtain the state features.

3. The method according to claim 2, characterized in that, The method further includes: If the number of times equals the number of times threshold, the execution retrieval is removed from the action space; The number threshold is determined in the following way: The complexity of the input information is evaluated to obtain the complexity parameter of the input information; From a preset first mapping table, query the frequency threshold range corresponding to the complexity parameter, and determine the frequency threshold based on the frequency threshold range corresponding to the complexity parameter. The first mapping table includes multiple correspondences between the complexity parameter and different frequency threshold ranges.

4. The method according to claim 3, characterized in that, Determining the number threshold based on the range of the number threshold corresponding to the complexity parameter includes: If the complexity parameter is less than or equal to the first complexity threshold, then the preset initial value is used as the number of times threshold. If the complexity parameter is greater than the first complexity threshold and less than the second complexity threshold, then the complexity parameter is mapped to the number threshold through a preset linear growth function, wherein the number threshold is greater than the initial value and less than the preset target value; If the complexity parameter is greater than or equal to the second complexity threshold, then the target value is used as the number of times threshold.

5. The method according to claim 2, characterized in that, The method further includes: If the number of times equals the number of times threshold, the execution retrieval is removed from the action space; The number threshold is determined in the following way: The complexity of the input information is evaluated to obtain the complexity parameter of the input information; The input information is classified into different domains to determine the target domain to which the input information belongs. Then, the fault tolerance parameter corresponding to the target domain is queried from a preset second mapping table. The second mapping table includes the correspondence between multiple different target domains and different fault tolerance parameters. Obtain the service priority corresponding to the input information, and determine the baseline retrieval count corresponding to the service priority; An adjustment coefficient is determined based on the fault tolerance parameter and the complexity parameter, and the number of benchmark searches is adjusted based on the adjustment coefficient to obtain the number threshold.

6. The method according to claim 1, characterized in that, The step of performing feature mapping on the state features to obtain evaluation values ​​for multiple types of candidate actions in the action space includes: The state features are input into a pre-trained decision network, wherein the decision network includes a cascaded feature extraction layer and an output layer; The cascaded feature extraction layers perform a first feature mapping on the state features to obtain the mapped features. The output layer performs a second feature mapping on the mapped features to obtain an action evaluation vector. The number of elements in the action evaluation vector is consistent with the number of action types in the action space, and the value of each element in the action evaluation vector represents the evaluation value of each candidate action.

7. The method according to claim 6, characterized in that, The context data to be processed includes the dialogue history; The method further includes: Calculate the semantic correlation between the dialogue history and the input information, and detect historical retrieval records in the dialogue history that have performed retrieval operations to obtain the detection results; Based on the semantic relevance and the detection results, the confidence level of the dialogue history is determined; Before inputting the state features into the pre-trained decision network, the method further includes: The dialogue history confidence is fused with the state features to obtain fused state features, and the process is then transferred to the pre-trained decision network based on the fused state features.

8. The method according to claim 7, characterized in that, The action space includes multiple types of candidate actions, including execution retrieval; After performing a second feature mapping on the mapped features through the output layer to obtain the action evaluation vector, the method further includes: Based on the confidence level of the dialogue history, a weighting coefficient for the action to be performed is determined. The element values ​​corresponding to the action to be performed in the action evaluation vector are adjusted by the action weighting coefficient. The target action is determined based on the adjusted action evaluation vector. The confidence level of the dialogue history is negatively correlated with the evaluation value of the action to be performed.

9. The method according to claim 1, characterized in that, The action space contains multiple types of candidate actions, including: Generate response data to instruct the large language model to generate response data fragments based on the context data to be processed; Execute a search, which instructs a pre-set search module to perform a search based on the context data to be processed in order to obtain external information; The completion of response data generation is used to indicate the cessation of the iterative process and to call a pre-set output module to extract response data of the input information generated by the large language model from the context data to be processed.

10. The method according to claim 9, characterized in that, After determining the target action from the plurality of candidate actions based on the evaluation value, the method further includes: Execute the data processing instruction corresponding to the target action to obtain the processing result. If the target action is to perform retrieval or generate response data, then update the context data to be processed based on the processing result and use the updated context data as the context data to be processed in the next round of iteration. If the target action is to complete the generation of response data, then proceed to stop the iteration and extract the response data of the input information generated by the large language model from the context data to be processed.

11. The method according to claim 10, characterized in that, The execution of the data processing instructions corresponding to the target action to obtain the processing result includes: When the target action is to generate response data, the preset first prompt word and the context data to be processed are input into the large language model to obtain response data fragments, and the response data fragments are used as the processing result; Wherein, the first prompt word is used to instruct the large language model, during the process of generating the response data fragment, if it detects that external information needs to be obtained for answer generation, then generate a response data fragment carrying a tag for the information to be retrieved, wherein the tag for the information to be retrieved is used to characterize the content of the external information to be retrieved.

12. The method according to claim 11, characterized in that, The process involves inputting a preset first prompt word and the context data to be processed into the large language model to obtain response data fragments, including: The large language model is invoked to generate the answer text based on the first prompt word and the context data to be processed, and the number of lexical units generated by the large language model is detected. When the number of lexical units reaches a preset threshold, the large language model is instructed to stop generating the response text and combine the currently generated lexical units into the response data fragment.

13. The method according to claim 10, characterized in that, The execution of the data processing instructions corresponding to the target action to obtain the processing result includes: When the target action is to perform a retrieval, the external knowledge base is searched based on the context data to be processed to obtain external information, and the external information is used as the processing result. Updating the context data to be processed based on the processing result includes: The external information is fused into the context data to be processed using the large language model to obtain updated context data, which is then used as the context data to be processed in the next iteration.

14. A method for training a decision network, characterized in that, The method includes: Initialized context data samples are constructed based on the input information samples to be responded to from the large language model; Multiple iterative processes are performed. In the first iteration, the initialized context data sample is used as the context data sample to be processed. In subsequent iterations, the updated context data sample based on the previous iteration is used as the context data sample to be processed. Each iteration includes: The context data sample to be processed is encoded with state features to obtain state feature samples; The state feature samples are feature mapped through the decision network to be trained to obtain evaluation value samples of multiple types of candidate actions in the action space, and the target action sample is determined from the multiple types of candidate actions based on the evaluation value samples. If the target action is to complete the generation of response data, the iterative process is stopped, and response data samples of the input information generated by the large language model are extracted from the context data to be processed. The reward value is determined based on the response data sample, and the parameters of the decision network to be trained are updated based on the reward value.

15. The method according to claim 14, characterized in that, The step of determining the target action sample from the multiple types of candidate actions based on the evaluation value sample includes: Based on the cumulative training steps of the decision network, a first value is determined, wherein the first value is configured to decrease according to a preset decay coefficient and a preset step period as the cumulative training steps increase, until it decreases to a preset value and remains unchanged. A second value located within a preset numerical range is generated. If the second value is less than the first value, a candidate action is selected from the multiple types of candidate actions as the target action sample. If the second value is greater than or equal to the first value, then the candidate action corresponding to the evaluation value sample with the largest value is taken as the target action sample.

16. The method according to claim 14, characterized in that, The multiple types of candidate actions include performing a retrieval; The process of determining the reward value based on the response data sample includes: The quality of the response data samples is evaluated to obtain quality parameter values; Obtain the total response time consumed in generating the response data sample, and obtain the total number of times the target action sample is executed in the multiple iterations; A first penalty value is determined based on the ratio of the total response time to a preset time threshold, and a second penalty value is determined based on the ratio of the total number of times to a preset number of times threshold. The reward value is obtained by combining the quality parameter value, the first penalty value, and the second penalty value.

17. A data processing device for a large language model, characterized in that, The device includes: The first data processing module is used to construct initial context data based on the input information to be responded to from the large language model; The second data processing module is used to perform multiple iterative processes. In the first iterative process, the initialized context data is used as the context data to be processed. In subsequent iterative processes, the updated context data based on the previous iterative process is used as the context data to be processed. Each iterative process includes: The context data to be processed is encoded with state features to obtain state features; the state features are then mapped to obtain evaluation values ​​of multiple types of candidate actions in the action space; and the target action is determined from the multiple types of candidate actions based on the evaluation values. If the target action is to complete the generation of response data, the iterative process is stopped, and the response data of the input information generated by the large language model is extracted from the context data to be processed.

18. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 13, or implements the method according to any one of claims 14 to 16.

19. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method according to any one of claims 1 to 13, or the method according to any one of claims 14 to 16.

20. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method according to any one of claims 1 to 13, or the method according to any one of claims 14 to 16.

Images