A model reinforcement learning optimization system, method, and electronic device

Abstract

This application discloses a model reinforcement learning optimization system, method, and electronic device, which relates to the field of artificial intelligence technology. Since each computing device is assigned with the weight of the model to be optimized, the weight of the reference model, and the weight of the reward model, that is, the computing device is equipped with all types of intelligent agents, it can simultaneously execute multiple computing tasks in the pre-trained language model optimization process. This avoids the problem of idle and wasted computing resources caused by the serial execution of tasks at different stages on different devices in related technologies, and improves the optimization efficiency of the pre-trained language model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model reinforcement learning optimization system, method and electronic device. Background Technology

[0002] Currently, reinforcement learning that enhances pre-trained language models has demonstrated significant application value in multiple fields, such as robotics, autonomous driving, energy management, and healthcare. In the field of robotics applications, the diversity, contextual dependence, and potential ambiguity of language demand stronger language understanding capabilities from the models. Therefore, further optimization of existing pre-trained language models has become a key research focus.

[0003] In related technologies, due to the significant hardware requirements for training pre-trained language models, multiple agents involved in the pre-trained language model optimization process are typically deployed on different computing devices. These devices then transfer computational results to each other in a continuous loop until the training task is completed. However, because different types of agents run different stages of computational tasks on different computing devices, the computing resources of some devices are often left idle and wasted, which is detrimental to ensuring the optimization efficiency of the pre-trained language model. Summary of the Invention

[0004] This application provides a model reinforcement learning optimization system, method, and electronic device to at least solve the problem in related technologies that leads to idle and wasted computing resources of computing devices, which is not conducive to ensuring the optimization efficiency of pre-trained language models.

[0005] This application provides a model reinforcement learning optimization system, including: multiple computing devices, wherein the weights of the model to be optimized, the reference model, and the reward model are respectively distributed on the multiple computing devices;

[0006] Multiple computing devices perform inference calculations based on the assigned weights of the model to be optimized, and obtain the current inference result;

[0007] Multiple computing devices perform inference calculations based on the assigned reference model weights to obtain reference results;

[0008] Multiple computing devices determine the current loss value based on the assigned reward model weights, the current inference result, and the reference result.

[0009] Each computing device optimizes the weights of the model to be optimized based on the current loss value to obtain the target model;

[0010] The target model includes the optimized weights of the model to be optimized for each computing device.

[0011] This application also provides a model reinforcement learning optimization method, applied to the model reinforcement learning optimization system as described above, the method comprising:

[0012] In the control model reinforcement learning optimization system, multiple computing devices perform inference calculations based on the assigned weights of the model to be optimized, and obtain the current inference result.

[0013] Multiple computing devices are controlled to perform inference calculations based on assigned reference model weights to obtain reference results;

[0014] Control multiple computing devices based on the assigned reward model weights, and determine the current loss value according to the current inference result and the reference result;

[0015] Based on the current loss value, the weights of the models to be optimized for each computing device are optimized to obtain the target model;

[0016] The target model includes the optimized weights of the model to be optimized for each computing device.

[0017] This application also provides a model reinforcement learning optimization apparatus for use in the model reinforcement learning optimization system described above, the apparatus comprising:

[0018] The first control module is used to control multiple computing devices in the model reinforcement learning optimization system to perform inference calculations based on the assigned weights of the model to be optimized, and to obtain the current inference result.

[0019] The second control module is used to control multiple computing devices to perform inference calculations based on the assigned reference model weights to obtain reference results;

[0020] The third control module is used to control multiple computing devices to determine the current loss value based on the assigned reward model weights and the current inference results and reference results.

[0021] The optimization module is used to optimize the weights of the models to be optimized on each computing device based on the current loss value in order to obtain the target model;

[0022] The target model includes the optimized weights of the model to be optimized for each computing device.

[0023] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described model reinforcement learning optimization methods.

[0024] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described model reinforcement learning optimization methods.

[0025] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described model reinforcement learning optimization methods.

[0026] Through this application, since each computing device is assigned with the weight of the model to be optimized, the weight of the reference model, and the weight of the reward model, that is, the computing device is deployed with all types of intelligent agents, it can simultaneously execute multiple computing tasks in the pre-trained language model optimization process, avoiding the problem of idle and wasted device computing resources caused by the serial execution of tasks at different stages on different devices in related technologies, and improving the optimization efficiency of the pre-trained language model. Attached Figure Description

[0027] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A schematic diagram of the structure of a model reinforcement learning optimization system provided in an embodiment of this application;

[0029] Figure 2 A schematic diagram of another model reinforcement learning optimization system provided in this application embodiment;

[0030] Figure 3 A schematic diagram of the structure of an exemplary model reinforcement learning optimization system provided in the embodiments of this application;

[0031] Figure 4 A flowchart illustrating the model reinforcement learning optimization method provided in this application embodiment;

[0032] Figure 5 This is a schematic diagram of the structure of the model reinforcement learning optimization device provided in the embodiments of this application;

[0033] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0035] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0036] Currently, the combination of pre-trained language models and reinforcement learning has become a research hotspot in the field of artificial intelligence, achieving significant progress while also facing numerous challenges. In recent years, reinforcement learning-enhanced pre-trained language models have emerged continuously, and research progress shows that reinforcement learning techniques have great potential in optimizing the performance and alignment effects of pre-trained language models.

[0037] At the technical level, advancements in techniques such as Human Feedback Reinforcement Learning (RLHF) and Artificial Feedback Reinforcement Learning (RLAIF) have enabled pre-trained language models to better align with human values, preferences, and expectations. Simultaneously, Direct Preference Optimization (DPO) methods are being explored, further enhancing the model's alignment by directly utilizing human preference data while bypassing reward models. These technological optimizations provide a more solid foundation for the application of pre-trained language models in complex tasks. Furthermore, reinforcement learning enhanced by pre-trained language models has demonstrated significant application value in multiple fields, such as robotics, autonomous driving, energy management, and healthcare. For example, in robotics, it can improve human-computer interaction efficiency, helping robots better understand human needs and behavioral logic.

[0038] However, research in this field still faces many challenges. The diversity, context dependence, and potential ambiguity of human language require models to have stronger language understanding capabilities to improve performance. At the same time, pre-trained language models have high computational costs and slow interaction efficiency, which can affect the speed of agent-environment interaction in online reinforcement learning, limiting the application of models in scenarios with high real-time requirements.

[0039] Therefore, improving the training speed of pre-trained language models for reinforcement learning can significantly shorten the time from training to deployment, improve R&D efficiency, and reduce computational resource consumption and costs. At the same time, faster training speeds facilitate rapid model iteration, better adapting to dynamically changing environments and task requirements, and enhancing the model's practicality and competitiveness.

[0040] In related technologies, traditional frameworks typically use agents as the basic unit. Different agents are deployed on different computing devices, and the devices transfer computational results to each other in a continuous loop until the training task is completed. In actual training, this framework results in idle and wasted computing resources because different stages of computation are executed on different devices. Furthermore, maintaining multiple copies of the same weights leads to wasted memory.

[0041] To address the aforementioned technical problems, this application provides a model reinforcement learning optimization system. The system includes: multiple computing devices, with weights for the model to be optimized, a reference model, and a reward model respectively allocated across these devices; multiple computing devices perform inference calculations based on the allocated weights of the model to be optimized to obtain a current inference result; multiple computing devices perform inference calculations based on the allocated weights of the reference model to obtain a reference result; multiple computing devices determine a current loss value based on the allocated weights of the reward model, according to the current inference result and the reference result; each computing device optimizes the weights of the model to be optimized on its current computing device based on the current loss value to obtain a target model; wherein the target model includes the optimized weights of the model to be optimized from each computing device. The system provided by the above solution, because each computing device is allocated weights for the model to be optimized, reference model, and reward model—meaning the computing devices are equipped with all types of intelligent agents—can simultaneously execute multiple computational tasks during the pre-trained language model optimization process. This avoids the problem of idle and wasted computing resources caused by the serial execution of tasks at different stages on different devices in related technologies, thus improving the optimization efficiency of the pre-trained language model.

[0042] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] This application provides a model reinforcement learning optimization system for optimizing and training a pre-trained language model.

[0044] like Figure 1 The diagram shown is a structural schematic of a model reinforcement learning optimization system provided in an embodiment of this application. The system includes multiple computing devices, and the weights of the model to be optimized, the reference model, the reward model, and the evaluation model are distributed across the multiple computing devices.

[0045] In this process, multiple computing devices perform inference calculations based on the assigned weights of the model to be optimized, and obtain the current inference result; multiple computing devices perform inference calculations based on the assigned weights of the reference model, and obtain the reference result; multiple computing devices determine the current loss value based on the assigned weights of the reward model, according to the current inference result and the reference result; each computing device optimizes the weights of the model to be optimized of the current computing device according to the current loss value, so as to obtain the target model.

[0046] The target model includes the optimized weights of the model to be optimized for each computing device.

[0047] It should be noted that the model reinforcement learning optimization system provided in this application establishes a cross-device storage pool that stores the mapping between physical addresses and virtual addresses. This pool is used to uniformly manage and maintain the weight storage of each agent (the model to be optimized, the reference model, and the reward model). During training, the agent weights are read and updated through this system.

[0048] Specifically, after each computing device obtains the prompt word, it performs inference calculations based on the locally allocated agent weights, and sequentially based on the weights of the model to be optimized and the reference model weights to obtain the current inference result and the reference result. It then determines the current loss value of the model to be optimized and optimizes the weights of the current model to be optimized based on the current loss values ​​determined by all computing devices. This enables parallel processing of model inference, calculation and training, ensuring that the device is continuously in a computing state and improving the resource utilization of the computing device.

[0049] Based on the above embodiments, such as Figure 2 The diagram shown is a structural schematic of another model reinforcement learning optimization system provided in this application embodiment. Taking four computing devices, device 1 to device 4, as an example, as an implementable approach, in one embodiment, the system further includes a distributed weight management module.

[0050] The distributed weight management module is used to allocate the weights of the model to be optimized, the reference model weights, and the reward model weights in the weight storage space of each computing device.

[0051] It should be noted that the reference model can be the inference model before optimization, such as a pre-trained language model or an older version of the inference model. Before reinforcement learning training, the distributed weight management module first distributes the weights of each agent (policy model, reference model, and reward model, etc.) to each computing device using weight distribution methods such as uniform distribution, layer-by-layer distribution, or tensor parallel distribution. Each model is allocated specific computing resources, and these computing devices are responsible for the storage, management, and computation of the agents. During training, each computing device executes the computation of each model simultaneously and sequentially. This distributed loading method can fully utilize the computing power of multiple devices and improve training efficiency. At the same time, since each computing device only needs to maintain the model parameters it requires, the transmission of all parameters is avoided, saving communication time.

[0052] Specifically, after receiving the prompt words, the model to be optimized generates a current inference result through the inference process. The current inference result can be a textual answer or a decision suggestion. The reference model simultaneously receives the same prompt words and generates a reference result through the forward computation process. This reference result is used to evaluate the inference quality of the model to be optimized. The reward model compares the current inference result with the reference result, determines the difference between the two, and generates the current loss value.

[0053] Specifically, in one embodiment, the reward model is used to determine the current empirical value of the current model to be optimized based on the current inference result; determine the reference value based on the reference result; and determine the current loss value of the model to be optimized based on the gap between the current inference result and the reference result represented by the current empirical value and the reference value.

[0054] Specifically, after the model to be optimized generates its current inference result based on the prompt words, the reward model extracts its performance features in task execution based on the current inference result. These performance features include the accuracy, rationality, and relevance of the inference, and are quantified as current empirical values. After the reference model generates a reference result based on the same prompt words, the reward model uses this reference result as a benchmark to extract its corresponding ideal features and quantifies these ideal features as reference values. The reward model compares the current empirical value with the reference value to quantify the difference between them, such as the difference or ratio; this difference is the current loss value. The magnitude of the loss value reflects the degree of deviation between the current model to be optimized and the reference model, providing a basis for subsequent weight optimization of the strategy model (the model to be optimized).

[0055] Based on the above embodiments, as an implementable approach, in one embodiment, multiple computing devices are further assigned evaluation model weights; different model reinforcement learning optimization systems share the current loss value; based on the assigned evaluation model weights, the multiple computing devices determine the global loss value of the model to be optimized according to the current loss value shared among the various model reinforcement learning optimization systems.

[0056] Accordingly, in one embodiment, each computing device optimizes the weights of the model to be optimized for the current computing device based on the global loss value.

[0057] The prompts obtained by different model reinforcement learning optimization systems can be from different domains. For example, if the system includes two model reinforcement learning optimization systems, one model reinforcement learning optimization system can receive prompts from the mathematical domain, such as mathematical problems, while the other model reinforcement learning optimization system can receive prompts from the computer domain, such as computer problems.

[0058] Specifically, the evaluation model is used to summarize the current loss values ​​generated by all model reinforcement learning optimization systems and calculate the global loss value to avoid the data bias of a single model reinforcement learning optimization system affecting the optimization results of the model.

[0059] Specifically, in one embodiment, the computing device includes a process pool and a process manager.

[0060] The process pool includes various model optimization processes, including a model inference process, a reference model calculation process, a reward model calculation process, a policy model training process, and an evaluation model calculation process. The model inference process calls the weights of the model to be optimized, performs inference on the prompt words, and generates the current inference result. The reference model calculation process calls the reference model weights and generates a reference result corresponding to the prompt words. The reward model calculation process calls the reward model weights and calculates the current loss value of the model to be optimized based on the current inference result and the reference result. The policy model training process calls the model to be optimized, determines the gradient value of the model to be optimized based on the global loss value, and optimizes the weights of the model to be optimized based on the gradient value. The evaluation model calculation process calls the evaluation model weights and determines the global loss value of the model to be optimized based on the current loss value determined by all computing devices.

[0061] The process manager is used to synchronize the model inference process, reference model computation process, reward model computation process, policy model training process, and evaluation model computation process, so that multiple computing devices in the same model reinforcement learning optimization system run the same model optimization process at the same time. During training, the process manager sends instructions to each computing device to ensure that different computing devices run the same process and maintains communication between computing devices, executing the policy model-reference model-reward model computation process in sequence.

[0062] It should be noted that by using a process manager to process multiple processes within a single computing device in sequence, and by having multiple computing devices run the same model optimization process at the same time, the idle problem caused by the serial execution of different devices in the traditional framework is further avoided. This keeps the computing device in a computing state as much as possible and further improves the resource utilization of the computing device.

[0063] It should be further explained that the global loss value can be determined by weighting the current loss values ​​determined by the reinforcement learning optimization system of each model. The global loss value reflects the overall deviation of the global model to be optimized from the reference in the current training stage. When the optimization model uses the backpropagation algorithm, it calculates the degree of influence of each parameter on the loss based on the global loss value, i.e., the gradient value. The magnitude and direction of the gradient value determine the magnitude and direction of the parameter adjustment.

[0064] Specifically, each computing device builds and maintains a process pool containing multiple specialized processes, each responsible for different tasks. These processes work together during training to ensure the efficient operation of the entire reinforcement learning training process. In this way, each computing device not only stores and manages the model but also achieves efficient parallel processing of model inference, computation, and training through multiple processes in the process pool. This architectural design enables reinforcement learning training to be performed efficiently in a large-scale distributed environment, significantly accelerating the model's learning speed and improving system scalability.

[0065] Specifically, when reinforcement learning training begins, the model inference process starts first. At this point, the system receives the necessary prompts for inference and broadcasts these prompts to the corresponding computing devices based on the number of parallel data paths. Each computing device, upon receiving the prompts, performs model inference using the weights of the policy model (the weights of the model to be optimized), according to a predefined tensor parallel or pipelined parallel mode. The main task of this stage is to generate the model's inference results based on the input prompts, providing foundational data for subsequent training steps.

[0066] After the model inference is complete, the reference model computation process begins. The reference model performs forward computation using its own weights. The purpose of this computation is to generate a reference result for subsequent reward calculation. Once completed, the reference model's computation result (reference result) and the previous model's inference result (current inference result) are used as inputs to the reward model computation process. The reward model uses its own weights to compute on these inputs, providing direct feedback and evaluation of the quality of the inference result. Based on the current inference result and the reference result, the reward model calculates the current loss value for the inference model to be optimized.

[0067] The model reinforcement learning optimization system provided in this application embodiment pre-allocates weight storage space in each computing device based on a distributed weight management module, and then loads each model weight into this storage space. After the weights are loaded, each computing device is responsible for managing and maintaining its own weights, without needing to manage the weights of other parts. During training, each computing device sequentially calls the relevant weight space (weight storage space) through the process pool manager to execute each training process, and then updates its own weights, reducing the time spent on multiple loading operations and accelerating training efficiency.

[0068] Specifically, in one embodiment, the evaluation model is further used to determine whether the model to be optimized meets the preset model optimization termination condition based on the global loss value of the model to be optimized; when the model to be optimized meets the preset model optimization termination condition, the weights of multiple models to be optimized in the model reinforcement learning optimization system are merged to obtain the target model.

[0069] The preset termination conditions for model optimization include the global loss value falling below a preset threshold, the number of iterations reaching the upper limit, and the global loss value no longer decreasing after multiple consecutive rounds.

[0070] Specifically, the evaluation model continuously monitors the training progress based on the global loss value. When the termination condition is met, the current model to be optimized is determined as the target inference model. That is, the target model is obtained by merging the weights of multiple models to be optimized in the model reinforcement learning optimization system.

[0071] Specifically, the training process of the model to be optimized begins, and the weights of the model are updated during training. The goal of this stage is to adjust the weights of the policy model based on the empirical and loss values ​​provided by the reward model to optimize the model's behavior policy. After obtaining the empirical and loss values, the policy model calculates the gradient of the loss function with respect to the parameters through backpropagation using a custom loss function, and updates the parameters, thus completing a full training process. The entire system will continuously repeat the above process until the model training reaches the predetermined termination condition. Through this iterative approach, the policy model gradually learns the optimal behavior policy to maximize long-term rewards. Throughout the entire training process, information transmission relies entirely on high-speed interconnection between devices, with a communication volume comparable to that of large-scale model pre-training phases. Each device is only responsible for maintaining its own parameters, thereby significantly improving device utilization efficiency. Furthermore, by adopting asynchronous parameter updates, not only is overall performance further improved, but the scalability of the application is also enhanced.

[0072] Specifically, in one embodiment, the computing device is further configured to, after obtaining the target model, acquire the inference instruction input by the user; and, in response to the inference instruction, perform inference based on the target model to obtain the target inference result corresponding to the inference instruction.

[0073] Specifically, the computing device obtains the reasoning instructions input by the user, such as domain-specific knowledge queries or computing task instructions, calls the target model to perform reasoning and outputs the results, thus obtaining the target reasoning result of the reasoning instructions.

[0074] For example, such as Figure 3The diagram illustrates the structure of an exemplary model reinforcement learning optimization system provided in this application embodiment. Prompts are broadcast to multiple computing devices (e.g., device 1, device 2) to provide a unified input for parallel computing. Taking device 1 as an example, the system calls the weights of the model to be optimized, performs inference on the prompts, and generates the current inference result. Using the reference model weights, a reference result corresponding to the prompts is generated, providing a benchmark for subsequent loss calculation. Combining the inference result and the reference result, and using the reward model weights / function, the current loss value of the model to be optimized is calculated to reflect the model's current performance. The current loss values ​​of multiple model reinforcement learning optimization systems are aggregated to determine the global loss value. Based on the global loss value, the gradient is calculated via backpropagation to update the weights of the model to be optimized, achieving model optimization. After parallel updates across multiple devices, the model weight parameters are synchronized to ensure model consistency. This supports unified optimization decisions across devices.

[0075] Specifically, in one embodiment, a version recording module can be added to each computing device to generate a unique version identifier for each weight update, such as after parameter updates during the policy model training process. This identifier is associated with metadata such as the global loss value, update time, involved computing device, and prompt word batch, and stored in an independent partition of the cross-device storage pool. When the evaluation model detects abnormal fluctuations in the global loss value, such as a sudden increase exceeding a preset threshold or multiple consecutive rounds of optimization failing to meet expectations, a rollback mechanism is automatically triggered. Based on the version record, the historical best version is called, such as the weight version with the lowest global loss value, to overwrite the current weights, and the training process is restarted to further improve the reliability of the optimization results of the pre-trained language model.

[0076] The model reinforcement learning optimization system provided in this application includes: multiple computing devices, with weights for the model to be optimized, a reference model, and a reward model respectively allocated across the multiple computing devices; the multiple computing devices perform inference calculations based on the allocated weights of the model to be optimized to obtain a current inference result; the multiple computing devices perform inference calculations based on the allocated weights of the reference model to obtain a reference result; the multiple computing devices determine a current loss value based on the allocated weights of the reward model, according to the current inference result and the reference result; each computing device optimizes the weights of the model to be optimized on the current computing device according to the current loss value to obtain a target model; wherein, the target model includes the optimized weights of the model to be optimized from each computing device. The system provided by the above scheme, because each computing device is allocated weights for the model to be optimized, reference model, and reward model, i.e., the computing devices are deployed with all types of intelligent agents, can simultaneously execute multiple computational tasks in the pre-trained language model optimization process, avoiding the problem of idle and wasted computing resources caused by the serial execution of tasks at different stages on different devices in related technologies, thus improving the optimization efficiency of the pre-trained language model. Furthermore, because each model weight is allocated on a different computing device, rather than copying the weights of the entire model to each device. In this way, the same weight only needs to occupy one memory location, thus avoiding duplicate memory usage and waste. Furthermore, this application's embodiments simplify the training process, optimizing it into a process switching operation. Under this new deployment method, different processes can simultaneously maintain the same set of weight data in memory without frequently transferring weights between different devices. This not only reduces data transmission overhead but also improves training efficiency, reduces system complexity and latency, thereby effectively enhancing the overall performance and resource utilization efficiency of the pre-trained language model reinforcement learning framework.

[0077] Through the above description of the embodiments, those skilled in the art can clearly understand that the system according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0078] This application provides a model reinforcement learning optimization method, which is applied to the model reinforcement learning optimization system provided in the above embodiments.

[0079] like Figure 4 The diagram shown is a flowchart of a model reinforcement learning optimization method provided in an embodiment of this application. The method includes:

[0080] Step 401: Control multiple computing devices in the model reinforcement learning optimization system to perform inference calculations based on the assigned weights of the model to be optimized, and obtain the current inference result;

[0081] Step 402: Control multiple computing devices to perform inference calculations based on the assigned reference model weights to obtain reference results;

[0082] Step 403: Control multiple computing devices to determine the current loss value based on the assigned reward model weights, the current inference result, and the reference result;

[0083] Step 404: Optimize the weights of the models to be optimized for each computing device based on the current loss value to obtain the target model.

[0084] The target model includes the optimized weights of the model to be optimized for each computing device.

[0085] For a description of the features in the embodiment corresponding to the model reinforcement learning optimization method, please refer to the relevant description of the embodiment corresponding to the model reinforcement learning optimization system, which will not be repeated here.

[0086] Embodiments of this application also provide a model reinforcement learning optimization apparatus for executing the model reinforcement learning optimization method provided in the above embodiments.

[0087] like Figure 5 The diagram shown is a structural schematic of the model reinforcement learning optimization device provided in an embodiment of this application. The model reinforcement learning optimization device 50 includes: a first control module 501, a second control module 502, a third control module 503, and an optimization module 504.

[0088] The system comprises the following modules: a first control module, which controls multiple computing devices in the model reinforcement learning optimization system to perform inference calculations based on the assigned weights of the model to be optimized, and obtain the current inference result; a second control module, which controls multiple computing devices to perform inference calculations based on the assigned weights of the reference model, and obtain the reference result; a third control module, which controls multiple computing devices to determine the current loss value based on the assigned weights of the reward model, the current inference result, and the reference result; and an optimization module, which optimizes the weights of the model to be optimized on each computing device based on the current loss value, in order to obtain the target model.

[0089] The target model includes the optimized weights of the model to be optimized for each computing device.

[0090] For a description of the features in the embodiment corresponding to the model reinforcement learning optimization device, please refer to the relevant description of the embodiment corresponding to the model reinforcement learning optimization method, which will not be repeated here.

[0091] Embodiments of this application also provide an electronic device, such as... Figure 6The diagram shown is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, including a processor 10 and a memory 20. The memory 20 stores a computer program, and the processor 10 is configured to run the computer program to execute the steps in any of the above-described model reinforcement learning optimization method embodiments.

[0092] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the model reinforcement learning optimization method at runtime.

[0093] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0094] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the model reinforcement learning optimization method.

[0095] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described model reinforcement learning optimization method embodiments.

[0096] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0097] The foregoing has provided a detailed description of the model reinforcement learning optimization system, method, and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A model reinforcement learning optimization system, characterized in that, include: Multiple computing devices are used, and the weights of the model to be optimized, the reference model, and the reward model are respectively allocated to the multiple computing devices; The multiple computing devices perform inference calculations based on the assigned weights of the model to be optimized, and obtain the current inference result; The multiple computing devices perform inference calculations based on the assigned reference model weights to obtain reference results; The multiple computing devices determine the current loss value based on the assigned reward model weights, the current inference result, and the reference result. Each computing device optimizes the weights of the model to be optimized on the current computing device according to the current loss value to obtain the target model; The target model includes the optimized model weights of each of the computing devices; The computing device includes: a process pool; the process pool includes multiple model optimization processes, the model optimization processes include a model inference process, a reference model calculation process, and a reward model calculation process; The model inference process is used to call the weights of the model to be optimized, infer the prompt words, and generate the current inference result. The reference model calculation process is used to call the reference model weights and generate reference results corresponding to the prompt words; The reward model calculation process is used to call the reward model weights and calculate the current loss value based on the current inference result and the reference result; the magnitude of the current loss value reflects the degree of deviation between the current model to be optimized and the reference model.

2. The model reinforcement learning optimization system according to claim 1, characterized in that, The system also includes: a distributed weight management module; The distributed weight management module is used to allocate the weights of the model to be optimized, the reference model, and the reward model in the weight storage space of each computing device.

3. The model reinforcement learning optimization system according to claim 1, characterized in that, The reward model is used for: Based on the current inference results, determine the current empirical value of the model to be optimized; Based on the aforementioned reference results, a reference value is determined; The current loss value is determined based on the gap between the current inference result and the reference result, as represented by the current empirical value and the reference value.

4. The model reinforcement learning optimization system according to claim 1, characterized in that, The multiple computing devices are also assigned evaluation model weights; The current loss value is shared among different model reinforcement learning optimization systems; The plurality of computing devices determine the global loss value of the model to be optimized based on the assigned evaluation model weights and the current loss value shared among the reinforcement learning optimization systems of each model.

5. The model reinforcement learning optimization system according to claim 4, characterized in that, Each computing device optimizes the weights of the model to be optimized for the current computing device based on the global loss value.

6. The model reinforcement learning optimization system according to claim 5, characterized in that, The computing device includes: a process manager; The model optimization process also includes a policy model training process and an evaluation model computation process; The strategy model training process is used to call the weights of the model to be optimized, determine the gradient value of the model to be optimized based on the global loss value, and optimize the weights of the model to be optimized based on the gradient value of the model to be optimized. The evaluation model calculation process is used to call the evaluation model weights and determine the global loss value of the model to be optimized based on the current loss values ​​determined by all computing devices. The process manager is used to synchronize the model inference process, reference model computation process, reward model computation process, policy model training process, and evaluation model computation process, so that multiple computing devices in the same model reinforcement learning optimization system run the same model optimization process at the same time.

7. The model reinforcement learning optimization system according to claim 6, characterized in that, The evaluation model is also used for: Based on the global loss value of the model to be optimized, determine whether the model to be optimized meets the preset model optimization termination condition; When the model to be optimized meets the preset model optimization termination condition, the weights of multiple models to be optimized in the model reinforcement learning optimization system are merged to obtain the target model.

8. The model reinforcement learning optimization system according to claim 1, characterized in that, The computing device is also used for: After obtaining the target model, obtain the user's input instructions for inference; In response to the instruction to be reasoned, reasoning is performed based on the target model to obtain the target reasoning result corresponding to the instruction to be reasoned.

9. A model reinforcement learning optimization method, applied to the model reinforcement learning optimization system as described in any one of claims 1 to 8, characterized in that, The method includes: The system controls multiple computing devices in the model reinforcement learning optimization system to perform inference calculations based on the assigned weights of the model to be optimized, and obtain the current inference result. The multiple computing devices are controlled to perform inference calculations based on the assigned reference model weights to obtain reference results; The multiple computing devices are controlled to determine the current loss value based on the assigned reward model weights and the current inference result and reference result. Based on the current loss value, the weights of the models to be optimized for each of the computing devices are optimized to obtain the target model; The target model includes the optimized model weights of each of the computing devices; The computing device includes: a process pool; the process pool includes multiple model optimization processes, the model optimization processes include a model inference process, a reference model calculation process, and a reward model calculation process; The system controls multiple computing devices in the model reinforcement learning optimization system to perform inference calculations based on the assigned weights of the model to be optimized, and obtain the current inference result, including: The multiple computing devices are controlled to call the weights of the model to be optimized based on the model inference process, infer the prompt words, and generate the current inference result. The control of the multiple computing devices to perform inference calculations based on the assigned reference model weights, and to obtain reference results, includes: The multiple computing devices are controlled to call the reference model weights based on the reference model calculation process to generate reference results corresponding to the prompt words; The control of the plurality of computing devices, based on the assigned reward model weights and according to the current inference result and reference result, determines the current loss value, including: The multiple computing devices are controlled to call the reward model weights based on the reward model calculation process, and calculate the current loss value according to the current inference result and the reference result; the magnitude of the current loss value reflects the degree of deviation between the current model to be optimized and the reference model.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the model reinforcement learning optimization method as described in claim 9 when executing the computer program.

Images