A large language model reinforcement learning system, a reinforcement learning method, and related devices

Abstract

The application provides a large language model reinforcement learning system, a reinforcement learning method and related equipment, the system comprises: a management module for unified management and calling of multiple category value functions, specifically comprising: an environment registration unit for establishing a global registration table, storing environment functions and corresponding meta information and mapping relationship with value functions; an environment running unit for locating environment functions according to unique identification information and instantiating running environment, calling environment functions to obtain running results, and inputting the answers generated by the large language model into the value function for evaluation, obtaining reward results and result categories; an integration module for generating reward signals suitable for the reinforcement learning process based on the reward results of the value function and the reward results of the value model; a training module for updating the policy parameters of the large language model based on the reward signals. The application realizes unified calling of multiple heterogeneous running environments and improves the generalization ability of the model across tasks.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to a large language model reinforcement learning system, reinforcement learning method and related equipment. Background Technology

[0002] With the rapid development of Large Language Models (LLMs), Reinforcement Learning with Human Feedback (RLHF) has become an important technique for improving model output quality and enhancing alignment capabilities. In this type of method, reward signals typically originate from two sources: first, a reward model trained on human-labeled data, reflecting human subjective preferences; and second, a reward function defined by specific task or environmental rules, providing an objective and verifiable evaluation of the correctness of the task results.

[0003] However, existing technologies still have the following shortcomings in practical applications:

[0004] 1. Lack of an effective way to integrate value models and value functions. Due to differences in numerical scale, definition standards, and applicable scenarios, direct use may lead to incomparable reward signals, thereby affecting the stability and effectiveness of the reinforcement learning process.

[0005] 2. Lack of a unified management and joint training mechanism for multiple categories of value functions. Different task scenarios often correspond to different categories of value functions, such as logic verification, code execution, and fact-finding. Existing technologies typically use a distributed invocation approach, lacking a unified interface and management framework, resulting in difficulty in system expansion and low training efficiency.

[0006] Based on this, existing solutions are usually designed for a single category or a single task, lacking architectural support for the overall coordination of multiple value functions and value models, making it difficult to meet the requirements for stability and versatility in large-scale model training. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this application provides a large language model reinforcement learning system, reinforcement learning method, and related equipment, which at least solves the problems of scattered reward signals and lack of unified architecture support in existing technologies, resulting in insufficient model training stability and versatility.

[0008] To achieve the above objectives and other advantages, some embodiments of this application provide the following aspects:

[0009] In a first aspect, some embodiments of this application provide a large language model reinforcement learning system, including:

[0010] The management module is used for unified management and invocation of value functions of multiple categories. The management module includes:

[0011] An environment registration unit is used to establish a global registry, which stores environment functions and their corresponding metadata, as well as the mapping relationship between each environment function and its corresponding value function.

[0012] The environment operation unit is used to locate the environment function and instantiate the corresponding operation environment according to the unique identification information in the global registry, call the environment function to obtain the operation result, and input the operation result and the model answer generated by the large language model based on the input question into the value function mapped to the environment function for evaluation, so as to obtain the reward result and result category of the value function;

[0013] An integration module is used to perform numerical scale alignment processing on the reward result of the value function at least, and to generate a reward signal adapted to the reinforcement learning process based on the reward result of the value function and the reward result of the value model.

[0014] The training module is used to update the policy parameters of the large language model based on the reward signal to complete the reinforcement learning process.

[0015] Secondly, some embodiments of this application provide a large language model reinforcement learning method, including:

[0016] A global registry is established to store multiple environment functions and their corresponding metadata, and a mapping relationship is established between each environment function and its corresponding value function.

[0017] The environment function is located and the runtime environment is instantiated based on the unique identifier information in the global registry. The environment function is called to obtain the running result. The running result and the model answer generated by the large language model based on the input question are input together into the value function mapped to the environment function for evaluation to obtain the reward result and result category of the value function.

[0018] The reward result of the value function is numerically scaled and aligned, and a reward signal adapted to the reinforcement learning process is generated based on the reward result of the value function and the reward result of the value model.

[0019] The policy parameters of the large language model are updated based on the reward signal to complete the reinforcement learning process.

[0020] Thirdly, some embodiments of this application also provide an electronic device, the electronic device comprising:

[0021] One or more processors; and a memory storing computer program instructions that, when executed, cause the processors to perform a large language model reinforcement learning method as described above.

[0022] Fourthly, some embodiments of this application also provide a computer-readable storage medium having a computer program and / or instructions stored thereon, which, when executed by a processor, implement the large language model reinforcement learning method as described above.

[0023] Fifthly, some embodiments of this application also provide a computer program product, including a computer program and / or instructions that, when executed by a processor, implement the large language model reinforcement learning method as described above.

[0024] Compared with related technologies, the solution provided in this application achieves unified management and invocation of multiple categories of value functions through a management module, overcoming the problems of scattered and difficult-to-uniformly schedule various value functions in the prior art, thereby improving the system's scalability and maintainability. A global registry is established through an environment registration unit, and unique identifiers are used to locate and instantiate environment functions, enabling the system to support unified invocation of multiple heterogeneous operating environments such as function-based, external, and storage-based environments, significantly enhancing the system's adaptability in complex application scenarios. Furthermore, the integration module performs numerical scale alignment on reward results from different sources and generates reward signals based on the joint results of value functions and value models, thus solving the problem of incomparability between different signals and ensuring the rationality and stability of the reward signals. Finally, the training module updates the policy parameters of the large language model based on the reward signals, enabling the large language model to gradually optimize its task execution strategy during iterative training, thereby improving the model's learning efficiency, training stability, and cross-task generalization ability. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other implementation methods can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the structure of a large language model reinforcement learning system provided in an embodiment of this application;

[0027] Figure 2This is a flowchart illustrating a large language model reinforcement learning method provided in an embodiment of this application;

[0028] Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] The following terms are used in this document.

[0031] Reinforcement Learning with Human Feedback (RLHF) is a training technique that combines reinforcement learning algorithms from machine learning with human subjective judgment. This approach aims to leverage human intuition and judgment to guide artificial intelligence systems in learning more complex and human-like behavioral patterns, particularly in fields such as Natural Language Processing (NLP) and decision-making. During training, it relies not only on the statistical patterns of large-scale corpora but also on human-provided preference feedback. The basic idea of ​​RLHF is to transform human evaluations into reward signals through reinforcement learning, guiding model optimization and making the model's output more aligned with human values ​​and usage needs.

[0032] A reward model (RM) is a predictive model trained using machine learning to evaluate the quality of the output of a large language model. Its typical training process includes collecting data on human preferences or comparative scoring of the model's output, and training a machine learning model based on this data, enabling the model to score new candidate outputs. The reward model reflects human subjective preferences or value orientations and is often used as a reward signal source in reinforcement learning to guide the language model's behavior to gradually align with human values.

[0033] A reward function (RF) is a reward function directly defined by the external environment, rules, or task truth value, used to objectively evaluate the correctness or effectiveness of the model's output. Typical forms include the correctness of a math problem answer, the test results of code execution, and the unique solution to a logic puzzle. The reward function has a definite truth standard, enabling direct judgment and scoring of the model's output. In reinforcement learning, the reward function typically serves as a reward signal with objective and verifiable standards, providing the model with an information source different from human annotations.

[0034] Example 1

[0035] In existing technologies, various value functions are typically implemented in a decentralized manner. For example, mathematical correctness evaluation functions, code executability verification functions, and text fluency judgment functions are each called and return results through different interfaces. Due to the lack of a unified management mechanism, developers need to adapt the input and output formats separately when calling different value functions, resulting in complex system structures, high call overhead, and often requiring modifications to multiple call logics when extending to new value functions, leading to high maintenance costs. Therefore, referring to… Figure 1 As shown, this embodiment provides a large language model reinforcement learning system, including a management module for unified management and invocation of value functions of multiple categories. By establishing a unified input / output interface specification, the invocation process of different value functions is standardized, ensuring that all value function outputs are returned in a unified data structure containing reward results (numerical scores, such as confidence scores in the range of 0 to 1), result categories (including correct, incorrect, rejected, system error, etc.), and additional meta-information (identification information of environment functions representing the running context of value functions, task category information, running parameters and status information, and intermediate evaluation indicators). In this way, the system can maintain the consistency of output format in cross-task scenarios, simplifying the processing logic of subsequent integration modules.

[0036] The management module includes:

[0037] The environment registration unit is used to establish a global registry. The global registry stores environment functions and their corresponding metadata, as well as the mapping relationship between each environment function and its corresponding value function.

[0038] In some embodiments, the environment registration unit is used to establish and maintain a global registry. The global registry can be implemented using a database, key-value store, or distributed directory service, and is used to centrally store and manage the calling methods of different runtime environments and their correspondence with value functions. With the help of this global registry, the system can support unified management of various heterogeneous environments, including function-based, external, and stored environments. Whether it's a fast verification function that needs to be executed locally, an external call that depends on a remote service, or a stored task that needs to load a local executable file, all can be registered and invoked in the same registry, thus ensuring consistent access and unified scheduling for different types of environments, significantly improving the system's scalability and adaptability in complex application scenarios.

[0039] An environment function is an executable logical unit that can perform external verification or auxiliary testing on the answers generated by a large language model. Examples include mathematical calculation functions, code execution functions, or external knowledge verification functions. The output of an environment function is usually an intermediate verification signal such as a Boolean flag, execution result, or verification information, rather than a final reward value.

[0040] In practical applications, when a specific environment function needs to be called, its location can be quickly determined using the unique identifier in the registry. Then, based on the metadata recorded in the registry, dependencies are loaded, input / output specifications are established, and the runtime environment is instantiated as a process, container, or remote service. This allows the environment function to be called and the execution result to be returned. For example, for a mathematical reasoning problem, the registry not only stores the call path of the mathematical calculation function but also records metadata such as input / output format definitions, required dependencies, timeout settings, and function tags. This metadata guides the runtime environment unit to correctly load and execute the function.

[0041] Furthermore, the registry explicitly defines the mapping relationship between environment functions and value functions. For example, the result of a mathematical calculation function is automatically passed to the mathematical correctness evaluation value function for scoring; the result of a code execution function is processed by the executability evaluation value function. Through this mapping mechanism, the system can directly connect to the corresponding value evaluation logic after calling an environment function, avoiding the tedious operation of developers repeatedly adapting input and output formats between different tasks.

[0042] The environment execution unit is used to locate the environment function based on the unique identifier information in the global registry and instantiate the corresponding execution environment. It calls the environment function to obtain the execution result and inputs the execution result and the model answer generated by the large language model based on the input question into the value function mapped to the environment function for evaluation, so as to obtain the reward result and result category of the value function.

[0043] In some embodiments, the target environment function is quickly located from the data index layer based on unique identifier information in the global registry, and an instantiated runtime environment for that function is generated. During this stage, metadata from the registry is extracted to guide the formatting and transformation of data input and output, such as parameter type checking, serialization of input data, and loading of dependencies required by the runtime environment.

[0044] When an environment function is invoked, the environment runtime unit encapsulates the input question-related data (including candidate answers generated by the large language model, question feature vectors, etc.) according to the interface specification, forming a standardized call request data packet. This data packet enters the corresponding runtime environment, undergoes function logic processing, and produces runtime result data. For example, for mathematical calculation functions, the output data might be Boolean symbols, specific numerical results, and explanatory text; for code execution functions, the output data might be the standard output stream, error logs, or runtime status information.

[0045] Subsequently, the environment execution unit aligns and binds the execution result with the model answer generated by the large language model, forming structured joint input data, which is then passed to the value function mapped to the environment function. When performing evaluation, the value function's input data includes not only the semantic representation of the answer generated by the large language model but also the verification signal returned by the execution environment. The semantic representation can be obtained by embedding encoding, extracting key features, or vectorizing the output text of the large language model, and is used to characterize the semantic content and logical structure of the answer; the verification signal originates from the execution result of the external execution environment and is obtained through parsing, such as Boolean tokens, numerical similarity, error codes, or confidence values.

[0046] In practical applications, the value function can comprehensively calculate semantic representations and validation signals. On the one hand, it uses semantic representations to judge the linguistic rationality of the model's response; on the other hand, it uses validation signals to constrain and verify the objective correctness of the response. Through this comprehensive calculation, the value function can output a numerical reward result to quantify the quality of the response and further generate result categories, such as labels like "correct," "incorrect," "partially correct," or "uncertain."

[0047] By combining semantic representation and verification signals as dual information sources, the system can provide more refined evaluations in complex tasks such as mathematical reasoning and code execution, thereby improving the effectiveness of reward signals in reinforcement learning and promoting the convergence and robustness of large language model training.

[0048] The integration module is used to perform numerical scale alignment on the reward result of the value function at least, and to generate a reward signal adapted to the reinforcement learning process based on the reward result of the value function and the reward result of the value model.

[0049] Because the reward distribution varies significantly across different task categories—for example, the results of mathematical calculation tasks are more dispersed, while those of text generation tasks are more concentrated—numerical scale alignment ensures a unified evaluation benchmark for different task categories during fusion, preventing disproportionate impacts on training from tasks with excessively large numerical scales. The integration module can employ various normalization methods when aligning the reward results of the value function. These include:

[0050] a) Normalization based on nonlinear activation functions: For example, using sigmoid or tanh functions to compress the original reward value to a specific range (e.g., 0 to 1). This method can avoid optimization instability caused by excessively large numerical ranges. For example, when using the PPO (Proximal Policy Optimization) algorithm, sudden increases in gradient norm or reward value may lead to training crashes.

[0051] b) Dynamic normalization during training: The normalization coefficients are continuously updated during training, for example, by tracking changes in the numerical distribution during training using a moving average method. Its advantage is that the normalization ratio can be automatically adjusted as training progresses, thereby enhancing the dynamic adaptability of the reward signal and avoiding distortion caused by a fixed normalization range in long-term training.

[0052] c) Task-Category-Based Grouping Normalization: Independent normalization coefficients are used for different task categories (such as mathematical reasoning, creative writing, code execution, and knowledge-based question answering). Its advantage lies in its adaptability to the varying difficulty levels of different tasks. For example, mathematical tasks often have more dispersed results, while creative tasks tend to have more concentrated results. Grouping normalization can improve the discriminative effectiveness of reward signals and enhance cross-task consistency.

[0053] After numerical scaling of the reward results of the value function is completed, the output range of the value function is usually limited to 0 to 1, while the output of the value model may be distributed in a wider range such as -10 to 10, and there is no direct comparability between the two. Moreover, even after the initial normalization operation, the absolute values ​​of the two may not be semantically consistent. For example, a high score of 0.9 given by the value function is not necessarily equivalent to a +8 rating given by the value model.

[0054] To address the above problems, this embodiment proposes several combination solutions:

[0055] In some scenarios, the reward signal can be generated using only the result of the value function. This is particularly suitable when the value model has limited discriminative ability, contains biases, or is not yet fully trained. In such cases, the evaluation result of the value function can ensure the reliability and stability of the reward signal. However, in scenarios where the value model is well-trained and can cover a wider range of evaluation dimensions, a fusion strategy can be adopted to improve the comprehensiveness and generalization ability of the reward signal. The integration module can normalize the numerical range of the value model output and then use a weighted average to synthesize the two results into a unified reward signal, achieving a balanced evaluation.

[0056] In another scenario, a fusion approach using the maximum value operation can be adopted. This means that when the evaluation result of either the value function or the value model indicates a high quality answer from the model, the system generates a positive reward signal. For example, in mathematical reasoning tasks, if the value function (function validator) confirms the correctness of the answer calculation, or the value model (large language model discriminator) determines the answer is logically sound, either condition is sufficient to determine that the answer is acceptable, thereby improving the system's robustness and fault tolerance.

[0057] By employing the above approach, the system can ensure the stability of reward signals while taking into account the advantages of different evaluation sources, thereby improving training efficiency and generalization ability in the reinforcement learning process.

[0058] The training module is used to update the policy parameters of the large language model based on the reward signal to complete the reinforcement learning process.

[0059] The training module can employ policy gradient algorithms (such as the PPO algorithm) to iteratively optimize the model's parameters: when the reward signal is positive, the policy is reinforced by increasing the probability of the model generating the current answer; when the reward signal is negative, the policy is suppressed by decreasing the probability of that answer. Furthermore, the training module can combine supervised learning signals with reinforcement learning reward signals for joint optimization, enabling the model to maintain accurate memorization of known knowledge while continuously improving its policy exploration ability under complex tasks. Through these mechanisms, the training module allows the large language model to gradually optimize its task execution strategy during iterative training, thereby significantly improving the model's learning efficiency, training stability, and cross-task generalization ability.

[0060] In some embodiments, the management module further includes a computing unit, which is used to parse multiple environment functions corresponding to the batch request and dynamically load them when receiving a batch request input by the user, and to perform parallel scheduling of the calling tasks of the multiple environment functions.

[0061] Specifically, when a user submits a batch request containing multiple tasks at once, the processing unit first parses the batch request to determine the types of environment functions involved and their corresponding unique identifiers. After parsing, the processing unit can dynamically load the required environment functions based on the global registry, avoiding redundant loading of irrelevant modules and thus reducing system resource consumption and initialization overhead. For example, when the batch request only contains mathematical verification and code execution tasks, the system only loads modules related to mathematical verification and code execution functions, and does not load function modules related to text fluency or external interface calls.

[0062] Furthermore, the computing unit possesses parallel scheduling capabilities, enabling it to allocate and execute multiple environment function calls in parallel, significantly reducing overall computation time. Specifically, mechanisms such as thread pools or asynchronous task queues can be used to distribute different tasks to multiple execution channels, allowing various tasks to run simultaneously. For example, when a batch request simultaneously includes verification tasks for 10 mathematical reasoning problems and 5 code snippets, the computing unit will split these tasks and call the mathematical verification function and code execution function in parallel within a multi-threaded environment, thus avoiding the latency caused by serial processing.

[0063] Through the above mechanism, the computing unit not only achieves efficient parallel execution under batch requests, but also supports dynamic module loading and invocation, ensuring the system's scalability and resource utilization. When new environment function categories are added subsequently, only the relevant information needs to be registered in the global registry, and the computing unit can automatically complete the parsing, loading, and scheduling without requiring significant modifications to the existing calling logic, thereby greatly reducing the system's maintenance costs.

[0064] In some embodiments, the arithmetic unit is further configured to distinguish between loading heavyweight runtime environments and lightweight runtime environments when dynamically loading multiple environment functions;

[0065] When dynamically loading heavy-duty operating environments, multiple server instances are pre-configured and started based on concurrency levels to form a normalized service pool.

[0066] When dynamically loading a lightweight runtime environment, the main program manages and schedules it uniformly, so that it can be loaded and released immediately when a request is triggered, thereby achieving elastic utilization of resources.

[0067] Specifically, for heavyweight runtime environments, which rely on numerous external components and have long initialization times, such as tasks that require calling remote inference services or depend on complex library functions, to avoid response delays caused by temporary loading when batch requests are triggered, the computing unit can pre-configure and start multiple server instances based on the estimated concurrency during system startup or peak task periods, forming a normalized service pool. During actual execution, when batch requests need to call such environment functions, the computing unit can directly allocate available instances from the service pool for processing, thereby significantly shortening loading time and improving concurrent response capabilities. For example, environment functions related to code compilation and execution typically require loading a runtime environment and a security sandbox, making them suitable for pre-provisioning in the service pool for reuse.

[0068] In contrast, lightweight runtime environments have lower initialization overhead and fewer execution dependencies, such as mathematical expression solving functions or text regular expression matching functions. In such scenarios, the computation units can be uniformly scheduled and managed by the main program, and processed using a method of immediate loading and immediate release after execution. This allows for rapid completion of calls when requests are triggered and resource reclamation after completion, thereby achieving elastic resource utilization and avoiding the maintenance of too many redundant resident processes.

[0069] Through the above mechanism, the system can establish differentiated scheduling strategies between heavyweight and lightweight operating environments. The heavyweight environment relies on a constant service pool to ensure low latency and high throughput, while the lightweight environment relies on real-time loading and releasing to reduce resident overhead. This solution not only improves overall operating efficiency but also enhances the system's scalability in high-concurrency and multi-tasking scenarios.

[0070] In some embodiments, the management module further includes an interaction unit, which is used to receive a question, an environment function, and an environment identifier that identifies the environment function as input to the large language model. The environment identifier corresponds to a unique identifier in the global registry, and the environment function and environment identifier are bound to the question.

[0071] Specifically, for example, when a user poses a mathematical reasoning question, the interaction unit can bind the question to a mathematical verification function and its identifier; when a user poses a programming execution question, the interaction unit binds the question to a code execution function and its identifier. Through this binding method, the system can ensure that in subsequent environment execution units, the target environment function is quickly located and the execution environment is instantiated based on the identifier, avoiding call errors and resource waste, thereby improving the accuracy of task scheduling and execution efficiency.

[0072] In some embodiments, the environment runtime unit establishes a unified input / output interface specification, encapsulating the questions, environment functions, and environment identifiers input to the large language model into standardized call requests to support the invocation of various heterogeneous environment functions. In this way, different types of environment functions follow a consistent input / output format when called, thereby avoiding the additional adaptation overhead caused by inconsistent interfaces in existing technologies.

[0073] Heterogeneous environment functions include: functional environment functions, external environment functions, and stored environment functions;

[0074] For functional environment functions, the environment execution unit directly matches the preset logic based on the model answer and environment identifier and returns the corresponding result;

[0075] For external environment functions, the environment execution unit generates remote call requests through network communication protocols, and performs exception handling and result parsing after the call returns a result;

[0076] For stored environment functions, the environment execution unit calls local files and / or executable programs, parses the execution output and converts it into a data format that conforms to a unified interface;

[0077] Furthermore, the environment operation unit adopts a unified resource scheduling and management mechanism to determine the calling order, isolate concurrent requests, and manage exceptions and caches when calling functions of multiple heterogeneous environments.

[0078] Specifically, when a function-based environment function is invoked, the environment runtime unit can directly match the preset logic with the model's answer and environment identifier. For example, for mathematical calculation problems, the environment runtime unit can directly invoke the locally preset calculation logic function, return the calculation result, and provide necessary numerical interpretations.

[0079] When an external environment function is invoked, the environment runtime unit generates a remote call request based on a unified interface (such as HTTP or gRPC protocols) and sends standardized input data to the external service. For example, in a code execution scenario, the system can send code snippets to a remote security sandbox service and, upon receiving the return result, automatically perform error log parsing, runtime status monitoring, and timeout exception handling.

[0080] When a stored environment function is called, the environment execution unit can load the corresponding executable file or script program from the local file system, execute it with the model's response as input parameters, and parse and format the generated results, such as converting the raw log output into Boolean values, numerical scores, or status labels defined by a unified interface.

[0081] Furthermore, during the process of calling the aforementioned heterogeneous environment functions, the environment operation unit can also determine the calling order, isolate concurrent requests, and manage exceptions and caches based on a unified resource scheduling and management mechanism.

[0082] Specifically, the system has a unified resource scheduling and management mechanism for global coordination of calls to environment functions. This mechanism can use metadata stored in the global registry (including function type, dependencies, expected latency, resource consumption, priority, and dependency topology) to execute a unified scheduling strategy during task enqueuing, distribution, and recycling.

[0083] The scheduler first determines the calling order based on priority rules and dependency topology. For example, for environment functions whose output will be used as input for other functions, they are executed first according to their dependencies to ensure the correctness of the calling process. At the same time, it takes into account the expected latency and resource overhead, and prioritizes the allocation of tasks with "low computational cost but high contribution". When it detects that a certain type of environment function is congested or remote service is unavailable, it dynamically adjusts its priority or switches to a backup implementation of the same type.

[0084] In concurrent request scenarios, the scheduler achieves isolation through service pooling, quotas, and rate limiting mechanisms. For example, environment functions for heavyweight runtime environments can maintain pooled instances of containers or remote services and dynamically scale them up or down based on concurrency; environment functions for lightweight runtime environments are loaded and released in real-time through a unified scheduling thread. Different requests can run in independent threads, processes, or sandbox environments to achieve resource isolation and security control, thereby avoiding resource contention or the spread of anomalies.

[0085] In terms of fault tolerance, this mechanism configures call timeout thresholds for different environment functions. When the threshold is reached, the call is immediately interrupted and an error code is returned. For idempotent operations, an exponential backoff strategy can be used for retries, while for non-idempotent operations, shadow calls or rollbacks are used to prevent error propagation. When the failure rate of a certain environment function exceeds the threshold within a specific time window, the system triggers circuit breaker logic, rejecting new call requests for a short period and switching to a backup implementation. At the same time, when unavailable or with excessively high latency, degradation processing can be performed using approximate evaluation based solely on the value function or cached results to ensure the continuity of the upper-layer process.

[0086] The scheduler can dynamically optimize resource scheduling strategies based on metrics collected during operation, such as latency distribution, success rate, retry rate, circuit breaker count, cache hit rate, and instance utilization. For example, it can adjust the cache update frequency and lifetime based on the cache hit rate, and readjust priority allocation and call routing based on failure modes, thereby achieving highly stable and scalable environment function scheduling in complex application scenarios.

[0087] Example 2

[0088] Reference Figure 2 As shown, this embodiment provides a reinforcement learning method for large language models, including:

[0089] Step S1: Establish a global registry, store multiple environment functions and their corresponding metadata in the global registry, and establish a mapping relationship between each environment function and its corresponding value function;

[0090] Step S2: Locate the environment function and instantiate the runtime environment based on the unique identifier information in the global registry, call the environment function to obtain the running result, and input the running result and the model answer generated by the large language model based on the input question into the value function mapped to the environment function for evaluation, so as to obtain the reward result and result category of the value function;

[0091] Step S3: Perform numerical scale alignment on the reward result of the value function, and generate a reward signal that is adapted to the reinforcement learning process based on the reward result of the value function and the reward result of the value model.

[0092] Step S4: Update the policy parameters of the large language model based on the reward signal to complete the reinforcement learning process.

[0093] Specifically, a global registry is established to store multiple environment functions and their corresponding metadata in a unified manner. A mapping relationship is established between each environment function and its corresponding value function in the registry, thereby forming a centralized function management and invocation foundation.

[0094] Based on this, when a reinforcement learning process needs to be executed, the target environment function can be quickly located using the unique identifier information in the global registry, and the corresponding runtime environment can be instantiated so that the runtime function can be called to obtain the runtime result. The obtained runtime result, along with the model answer generated by the large language model based on the input question, will be input into the value function mapped to the environment function for evaluation, thereby obtaining a numerical reward result and the corresponding result category.

[0095] Subsequently, the system performs numerical scaling on the reward results of the value function to address the issue of inconsistent reward distribution across different task categories. The scale-aligned reward results are then fused with the output of the value model to generate a comprehensive reward signal adapted to the reinforcement learning process.

[0096] Finally, the training module updates the policy parameters of the large language model based on the aforementioned reward signals to complete the reinforcement learning process. During this update process, the system can iteratively optimize the model using policy gradient algorithms (such as the Proximal Policy Optimization (PPO) algorithm), thereby significantly improving the learning efficiency, stability, and cross-task generalization ability of the large language model in multi-task scenarios.

[0097] Compared with related technologies, the solution provided in this application establishes a global registry to uniformly store and manage multiple environment functions and their corresponding metadata, and establishes a mapping relationship between environment functions and value functions in the registry. This enables the system to quickly locate and instantiate different types of environment functions when calling them, avoiding the problems of repeated adaptation and high maintenance costs caused by scattered interfaces and inconsistent input / output formats in the prior art.

[0098] Furthermore, through centralized management, it can not only support unified access to various heterogeneous environments such as function-based, external, and storage-based environments, but also automatically connect to the corresponding value function for evaluation after the call, ensuring consistent processing of different task categories under the same framework.

[0099] It is not difficult to see that this embodiment is a method embodiment corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the first embodiment.

[0100] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0101] Furthermore, some embodiments of this application also provide an electronic device. The electronic device can be various forms of digital computer, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device can also be various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0102] The electronic device includes: one or more processors; and a memory storing computer program instructions, which, when executed, cause the processor to perform a large language model reinforcement learning method as provided in the above embodiments. Figure 3 An exemplary structural diagram of the electronic device is disclosed. The electronic device includes one or more processors 1101, a memory 1102, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations. The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0103] The electronic device may further include an input device 1103 and an output device 1104. The processor 1101, memory 1102, input device 1103, and output device 1104 may be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0104] Input device 1103 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 1104 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display, a light-emitting diode display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0105] To provide interaction with the user, the electronic device can be a computer. The computer has: a display device (e.g., a cathode ray tube or LCD monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback); and input from the user can be received in any form (e.g., voice input or tactile input).

[0106] In this embodiment, a computer-readable medium stores a computer program / instruction, which, when executed by a processor, implements a large language model reinforcement learning method provided in the above embodiments. This computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into that device. The computer-readable medium carries one or more computer-readable instructions.

[0107] The memory 1102 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 1102, thereby implementing the program instructions / modules corresponding to the methods provided in any one or more of the embodiments described above in this application.

[0108] The memory 1102 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 1102 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1102 may optionally include memory remotely located relative to the processor 1101, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0109] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0110] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technologies, read-only optical discs, digital versatile optical discs or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0111] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0112] In the above embodiments, all or part of the implementation can be achieved through software, hardware, firmware, or any combination thereof. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the above steps or functions. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. In addition, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0113] The computer program product provided in this application includes one or more computer programs / instructions. When executed by a processor, these computer programs / instructions generate, in whole or in part, the processes or functions described in this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0114] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0115] The scope of this application is defined by the appended claims rather than the foregoing description, and is therefore intended to encompass all variations falling within the meaning and scope of equivalents of the claims. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device in software or hardware. Terms such as "first" and "second" are used only to distinguish descriptions and do not indicate any particular order, nor should they be construed as indicating or implying relative importance.

[0116] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily made by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims, and the above embodiments should be regarded as exemplary and non-limiting.

Claims

1. A large language model reinforcement learning system, characterized in that, include: The management module is used for unified management and invocation of value functions of multiple categories. The management module includes: An environment registration unit is used to establish a global registry, which stores environment functions and their corresponding meta-information, as well as the mapping relationship between each environment function and its corresponding value function. The environment function is an executable logic unit that can perform external verification or auxiliary detection on the answers generated by the large language model. An interaction unit is used to receive a question, an environment function, and an environment identifier that identifies the environment function as input to the large language model. The environment identifier corresponds to a unique identifier in the global registry, and the environment function and the environment identifier are bound to the question. The environment operation unit is used for: The question is input into a large language model to generate candidate answers for the large language model; The question-related data is encapsulated according to the interface specification to form a standardized call request data packet, wherein the question-related data includes the candidate answers; The environment function is located and the corresponding runtime environment is instantiated based on the unique identifier information in the global registry. After the call request data packet enters the corresponding runtime environment, the environment function is called to obtain the running result. The running result and the model answer generated by the large language model based on the input question are input together into the value function mapped to the environment function for evaluation to obtain the reward result and result category of the value function. An integration module is used to perform numerical scale alignment processing on the reward result of the value function, and generate a reward signal adapted to the reinforcement learning process based on the reward result of the value function and the reward result of the value model. The value model is a trained prediction model used to evaluate the quality of the model's response to the large language model and output the reward result. The training module is used to update the policy parameters of the large language model based on the reward signal to complete the reinforcement learning process.

2. The large language model reinforcement learning system according to claim 1, characterized in that, The management module also includes a computing unit, which is used to parse multiple environment functions corresponding to the batch request and dynamically load them when receiving batch requests input by the user, and to perform parallel scheduling of the calling tasks of the multiple environment functions.

3. The large language model reinforcement learning system according to claim 2, characterized in that, The computing unit is also used to distinguish between loading heavyweight runtime environments and lightweight runtime environments when dynamically loading the plurality of environment functions; When dynamically loading the heavyweight operating environment, multiple server instances are pre-configured and started according to the concurrency level to form a normalized service pool. When the lightweight operating environment is dynamically loaded, it is uniformly scheduled and managed by the main program so that it can be loaded and released immediately when a request is triggered, thereby realizing the elastic utilization of resources.

4. The large language model reinforcement learning system according to claim 1, characterized in that, The environment operation unit establishes a unified input / output interface specification, encapsulating the questions, environment functions, and environment identifiers input to the large language model into standardized call requests to support the invocation of various heterogeneous environment functions.

5. The large language model reinforcement learning system according to claim 4, characterized in that, The heterogeneous environment functions include: functional environment functions, external environment functions, and stored environment functions; For the functional environment function, the environment execution unit directly matches the preset logic based on the model answer and environment identifier and returns the corresponding result; For the external environment function, the environment execution unit generates a remote call request through a network communication protocol, and performs exception handling and result parsing after the call returns a result; For the stored environment function, the environment runtime unit calls a local file and / or an executable program, parses the runtime output and converts it into a data format that conforms to a unified interface; Furthermore, the environment operation unit adopts a unified resource scheduling and management mechanism to determine the calling order, isolate concurrent requests, and manage exceptions and caches when calling multiple heterogeneous environment functions.

6. A reinforcement learning method for large language models, characterized in that, include: A global registry is established to store multiple environment functions and their corresponding meta-information in the global registry, and a mapping relationship is established between each environment function and its corresponding value function. The environment function is an executable logic unit that can perform external verification or auxiliary detection on the answers generated by the large language model. The system receives a question, an environment function, and an environment identifier that identifies the environment function as input to the large language model. The environment identifier corresponds to a unique identifier in the global registry. The system then binds the environment function and the environment identifier to the question. The question is input into a large language model to generate candidate answers for the large language model; The question-related data is encapsulated according to the interface specification to form a standardized call request data packet, wherein the question-related data includes the candidate answers; The environment function is located and the runtime environment is instantiated based on the unique identifier information in the global registry. After the call request data packet enters the corresponding runtime environment, the environment function is called to obtain the running result. The running result and the model answer generated by the large language model based on the input question are input together into the value function mapped to the environment function for evaluation to obtain the reward result and result category of the value function. The reward result of the value function is numerically scale aligned, and a reward signal adapted to the reinforcement learning process is generated based on the reward result of the value function and the reward result of the value model. The value model is a trained prediction model used to evaluate the quality of the model's response to the large language model and output the reward result. The policy parameters of the large language model are updated based on the reward signal to complete the reinforcement learning process.

7. An electronic device, characterized in that, The electronic device includes: One or more processors; and a memory storing computer program instructions that, when executed, cause the processors to perform the large language model reinforcement learning method as described in claim 6.

8. A computer-readable storage medium having a computer program and / or instructions stored thereon, characterized in that, When the computer program and / or instructions are executed by the processor, they implement the large language model reinforcement learning method as described in claim 6.

9. A computer program product, comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by the processor, they implement the large language model reinforcement learning method as described in claim 6.

Images