Model reinforcement learning optimization system and method and electronic equipment

Abstract

The invention discloses a model reinforcement learning optimization system and method and an electronic device, and relates to the technical field of artificial intelligence. Each computing device is allocated with a to-be-optimized model weight, a reference model weight and a reward model weight, that is, the computing device is deployed with all types of agents; according to the method, multiple computing tasks in the pre-training language model optimization process can be executed at the same time, the problems of idle and waste of device computing resources caused by serial execution of tasks in different stages in different devices in related technologies are avoided, and the optimization efficiency of the pre-training language model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a model reinforcement learning optimization system and method and electronic device. BACKGROUND

[0002] At present, pre-training language model enhanced reinforcement learning has important application value in many fields, such as robot technology, autonomous driving, energy management, and medical health. In the field of robot technology application, due to the diversity, context dependence and potential ambiguity of language, the model needs to have stronger language understanding ability, therefore, how to further optimize the existing pre-training language model has become a key research content.

[0003] In related technologies, since the hardware requirement of pre-training language model training is large, the pre-training language model optimization process usually involves multiple agents deployed on different computing devices, and the computing results are transmitted between different computing devices, and the cycle is repeated until the training task is completed. However, since different types of agents run different stage computing tasks, and different types of agents are executed in different computing devices, there is often idle and waste of computing resources of part of the computing devices, which is not conducive to ensuring the optimization efficiency of the pre-training language model. SUMMARY

[0004] The present application provides a model reinforcement learning optimization system and method and electronic device to at least solve the problem of idle and waste of computing resources of computing devices in related technologies, which is not conducive to ensuring the optimization efficiency of the pre-training language model.

[0005] The present application provides a model reinforcement learning optimization system, comprising: a plurality of computing devices, the weights of a to-be-optimized model, a reference model and a reward model are respectively distributed in the plurality of computing devices; The plurality of computing devices perform inference calculation based on the distributed to-be-optimized model weights to obtain a current inference result; The plurality of computing devices perform inference calculation based on the distributed reference model weights to obtain a reference result; The plurality of computing devices determine a current loss value based on the current inference result and the reference result based on the distributed reward model weights; Each computing device optimizes the to-be-optimized model weights of the current computing device according to the current loss value to obtain a target model; The target model comprises the optimized to-be-optimized model weights of each computing device.

[0006] The present application also provides a model reinforcement learning optimization method applied to the above model reinforcement learning optimization system, which comprises: The plurality of computing devices in the model reinforcement learning optimization system perform inference calculation based on the assigned to-be-optimized model weights to obtain a current inference result. The plurality of computing devices perform inference calculation based on the assigned reference model weights to obtain a reference result. The plurality of computing devices determine a current loss value based on the current inference result and the reference result based on the assigned reward model weights. According to the current loss value, the to-be-optimized model weights of each computing device are optimized to obtain a target model. The target model includes the to-be-optimized model weights of each computing device after optimization.

[0007] The application also provides a model reinforcement learning optimization device applied to the model reinforcement learning optimization system, and the device comprises: The first control module is configured to control the plurality of computing devices in the model reinforcement learning optimization system to perform inference calculation based on the assigned to-be-optimized model weights to obtain a current inference result. The second control module is configured to control the plurality of computing devices to perform inference calculation based on the assigned reference model weights to obtain a reference result. The third control module is configured to control the plurality of computing devices to determine a current loss value based on the current inference result and the reference result based on the assigned reward model weights. The optimization module is configured to optimize the to-be-optimized model weights of each computing device according to the current loss value to obtain a target model. The target model includes the to-be-optimized model weights of each computing device after optimization.

[0008] The application also provides an electronic device comprising a memory for storing a computer program and a processor for executing the computer program to implement the steps of any of the model reinforcement learning optimization methods.

[0009] The application also provides a computer readable storage medium, and the computer readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement the steps of any of the model reinforcement learning optimization methods.

[0010] The application also provides a computer program product comprising a computer program, and the computer program is executed by a processor to implement the steps of any of the model reinforcement learning optimization methods.

[0011] Through the present application, since each computing device is allocated with the model weight to be optimized, the reference model weight and the reward model weight, that is, the computing device is deployed with all types of agents, various computing tasks in the pre-training language model optimization process can be executed simultaneously, the problems of idle and waste of computing resources of devices caused by serial execution of different stage tasks in related technologies are avoided, and the optimization efficiency of the pre-training language model is improved. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0013] Figure 1 A structural schematic diagram of a model reinforcement learning optimization system provided by an embodiment of the present application is shown in the figure. Figure 2 A structural schematic diagram of another model reinforcement learning optimization system provided by an embodiment of the present application is shown in the figure. Figure 3 A structural schematic diagram of an exemplary model reinforcement learning optimization system provided by an embodiment of the present application is shown in the figure. Figure 4 A flowchart of a model reinforcement learning optimization method provided by an embodiment of the present application is shown in the figure. Figure 5 A structural schematic diagram of a model reinforcement learning optimization device provided by an embodiment of the present application is shown in the figure. Figure 6 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0015] It should be noted that, in the description of the present application, the terms “include”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0016] Currently, the combination of pre-trained language models and reinforcement learning has become a research hotspot in the field of artificial intelligence, making significant progress while facing many challenges. In recent years, reinforcement learning-enhanced pre-trained language models have emerged, and research progress shows that reinforcement learning techniques have great potential in optimizing the performance and alignment of pre-trained language models.

[0017] At the technical level, human feedback reinforcement learning (RLHF) and artificial intelligence feedback reinforcement learning (RLAIF) and other technologies are constantly developing, enabling pre-trained language models to better align with human values, preferences, and expectations. At the same time, direct preference optimization (DPO) methods are also being explored, bypassing reward models and directly utilizing human preference data to further improve model alignment. These optimizations provide a more solid foundation for the application of pre-trained language models in complex tasks. In addition, reinforcement learning enhanced by pre-trained language models has important application value in multiple fields such as robotics, autonomous driving, energy management, and healthcare. For example, in the field of robotics, it can improve human-robot interaction efficiency and help robots better understand human needs and behavior logic.

[0018] However, research in this field still faces many difficulties. The diversity, context dependence, and potential ambiguity of human language require models to have stronger language understanding capabilities to improve performance. At the same time, the computational overhead of pre-trained language models is large, and the interaction efficiency is slow, which will affect the interaction speed of agents and the environment in online reinforcement learning, limiting the application of models in real-time scenarios.

[0019] Therefore, improving the training speed of pre-trained language model reinforcement learning can significantly shorten the time from model training to deployment, improve research and development efficiency, and reduce computing resource consumption and cost. At the same time, faster training speed helps to quickly iterate models, better adapt to dynamically changing environments and task requirements, and enhance the practicality and competitiveness of models.

[0020] In related technologies, traditional frameworks usually take agents as the basic unit, deploy different agents on different computing devices, and transmit computing results between devices, constantly cycling until the training task is completed. In the actual training process, different stages of computing tasks are executed in different devices, causing idle and waste of computing resources. At the same time, the same weight needs to be maintained in multiple copies, causing waste of memory.

[0021] To solve the above technical problems, an embodiment of the present application provides a model reinforcement learning optimization system, which includes: multiple computing devices, wherein the weights of the model to be optimized, the reference model, and the reward model are respectively distributed on the multiple computing devices; the multiple computing devices perform reasoning calculations based on the allocated weights of the model to be optimized to obtain the current reasoning result; the multiple computing devices perform reasoning calculations based on the allocated weights of the reference model to obtain the reference result; the multiple computing devices determine the current loss value based on the allocated reward model weights according to the current reasoning result and the reference result; each computing device optimizes the weights of the model to be optimized of the current computing device according to the current loss value to obtain the target model; wherein the target model includes the weights of the model to be optimized after optimization by each computing device. The system provided by the above scheme, since each computing device is allocated with the weights of the model to be optimized, the weights of the reference model, and the weights of the reward model, that is, the computing devices are deployed with all types of intelligent agents, can simultaneously execute multiple computing tasks in the process of optimizing the pre-trained language model, avoid the problem of idle and wasted computing resources of the device caused by the serial execution of tasks at different stages on different devices in the related technology, and improve the optimization efficiency of the pre-trained language model.

[0022] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0023] An embodiment of the present application provides a model reinforcement learning optimization system for optimizing and training a pre-trained language model.

[0024] like Figure 1 As shown, it is a structural diagram of a model reinforcement learning optimization system provided in an embodiment of the present application. The system includes: multiple computing devices, and the weights of the model to be optimized, the reference model, the reward model and the evaluation model are distributed on multiple computing devices.

[0025] Among them, multiple computing devices perform inference calculations based on the assigned weights of the model to be optimized to obtain the current inference results; multiple computing devices perform inference calculations based on the assigned weights of the reference model to obtain the reference results; multiple computing devices determine the current loss value based on the assigned reward model weights according to the current inference results and the reference results; each computing device optimizes the weights of the model to be optimized of the current computing device according to the current loss value to obtain the target model.

[0026] The target model includes the weights of the models to be optimized after optimization of each computing device.

[0027] It should be noted that the model reinforcement learning optimization system provided by the embodiment of the application establishes a cross-device storage pool that saves the mapping of physical addresses and virtual addresses, and uniformly manages and maintains the weight storage of each agent (to-be-optimized model, reference model and reward model) through the system.

[0028] Specifically, after each computing device obtains the prompt word, the agent weight allocated locally is used to perform inference calculation based on the to-be-optimized model weight and the reference model weight in turn, to obtain the current inference result and the reference result, and to determine the current loss value of the to-be-optimized model, and then the current to-be-optimized model is optimized according to the current loss value determined by all computing devices, to realize parallel processing of model inference, calculation and training, to ensure that the device is continuously in a computing state, and to improve the resource utilization rate of the computing device.

[0029] Based on the above embodiment, as shown in Figure 2 The structure diagram of another model reinforcement learning optimization system provided by the embodiment of the application is shown, taking four computing devices, device 1 to device 4, as an example, as a kind of implementable mode, in an embodiment, the system also includes a distributed weight management module.

[0030] The distributed weight management module is used to allocate the to-be-optimized model weight, the reference model weight and the reward model weight in the weight storage space of each computing device.

[0031] It should be noted that the reference model can be an inference model before optimization, such as a pre-trained pre-trained language model and an old version of an inference model. Before reinforcement learning training, the distributed weight management module first allocates the weights of each agent (policy model, reference model and reward model, etc.) to each computing device according to weight allocation methods such as uniform allocation, layer allocation or tensor parallel allocation. Each model will be allocated to a specific computing resource, and these computing devices will be responsible for the storage, management and operation of the agent. When performing training, each computing device will simultaneously perform the calculation of each model in turn. This distributed loading method can fully utilize the computing power of multiple devices and improve the training efficiency. At the same time, since each computing device only needs to maintain part of the model parameters required by itself, the transmission of all parameters is avoided, and the communication time is saved.

[0032] Specifically, the to-be-optimized model receives the prompt word and generates the current inference result through the inference process. The current inference result can be a text answer or a decision suggestion. The reference model synchronously receives the same prompt word and generates the reference result through the forward calculation process. The reference is used to evaluate the inference quality of the to-be-optimized model. The reward model compares the current inference result with the reference result, determines the gap between the two, and generates the current loss value.

[0033] Specifically, in an embodiment, the reward model is specifically configured to determine a current experience value of the current to-be-optimized model according to the current inference result, determine a reference value according to the reference result, and determine a current loss value of the to-be-optimized model according to a gap between the current inference result and the reference result represented by the current experience value and the reference value.

[0034] Specifically, after the to-be-optimized model generates the current inference result according to the prompt word, the reward model extracts performance features of the to-be-optimized model in task execution based on the current inference result, the performance features including accuracy, rationality and relevance of inference, etc., and quantifies the performance features into the current experience value. After the reference model generates the reference result based on the same prompt word, the reward model extracts ideal features corresponding to the reference result based on the reference result, and quantifies the ideal features into the reference value. The reward model quantifies the gap between the current experience value and the reference value, such as a difference value and a ratio, by comparing the current experience value with the reference value, and the gap is the current loss value. The size of the loss value reflects the deviation degree of the current to-be-optimized model from the reference model, and provides a basis for subsequent weight optimization of the strategy model (to-be-optimized model).

[0035] On the basis of the above embodiment, as a kind of implementable mode, in an embodiment, the plurality of computing devices are also assigned with evaluation model weights;Different model reinforcement learning optimization systems share the current loss value between them;The plurality of computing devices determine the global loss value of the to-be-optimized model based on the evaluation model weights assigned and the current loss value shared between each model reinforcement learning optimization system.

[0036] Correspondingly, in an embodiment, each computing device optimizes the to-be-optimized model weight of the current computing device according to the global loss value.

[0037] Wherein, the prompt words obtained by different model reinforcement learning optimization systems can be prompt words in different fields. Taking a system including two model reinforcement learning optimization systems as an example, one model reinforcement learning optimization system can receive prompt words in the field of mathematics, such as mathematical problems, and the other model reinforcement learning optimization system can receive prompt words in the field of computer, such as computer problems.

[0038] Specifically, the evaluation model is used to aggregate the current loss values generated by all model reinforcement learning optimization systems to calculate the global loss value, so as to avoid the influence of data deviation of a single model reinforcement learning optimization system on the optimization result of the model.

[0039] Specifically, in an embodiment, the computing device includes a process pool and a process manager.

[0040] The process pool includes multiple model optimization processes, and the model optimization processes include a model inference process, a reference model calculation process, a reward model calculation process, a policy model training process, and an evaluation model calculation process; the model inference process is used to call the to-be-optimized model weight, infer the prompt word, and generate a current inference result; the reference model calculation process is used to call the reference model weight and generate a reference result corresponding to the prompt word; the reward model calculation process is used to call the reward model weight, calculate the current loss value of the to-be-optimized model according to the current inference result and the reference result; the policy model training process is used to call the to-be-optimized model, determine the gradient value of the to-be-optimized model according to the global loss value, and optimize the weight of the to-be-optimized model according to the gradient value of the to-be-optimized model; and the evaluation model calculation process is used to call the evaluation model weight and determine the global loss value of the to-be-optimized model according to the current loss value determined by all the computing devices.

[0041] The process manager is used to synchronously control the model inference process, the reference model calculation process, the reward model calculation process, the policy model training process, and the evaluation model calculation process, so that multiple computing devices in the same model reinforcement learning optimization system run the same model optimization process at the same time. During the training process, the process manager sends instructions to each computing device to ensure that different computing devices run the same process and maintain communication between the computing devices, and sequentially executes the policy model-reference model-reward model calculation process.

[0042] It should be noted that the multiple processes in a single computing device are processed in sequence by the process manager, and multiple computing devices run the same model optimization process at the same time, which further avoids the idle problem caused by the serial execution of different devices in the traditional framework, and makes the computing devices always in a computing state as much as possible, thereby further improving the resource utilization rate of the computing devices.

[0043] It should be further noted that the global loss value can be determined by weighted calculation on the current loss values determined by each model reinforcement learning optimization system, and the global loss value reflects the overall deviation of the global to-be-optimized model from the reference in the current training stage. When the optimization model calculates the influence degree of each parameter on the loss value based on the global loss value by the back propagation algorithm, the gradient value is obtained. The size and direction of the gradient value determine the amplitude and direction of the parameter adjustment.

[0044] Specifically, each computing device constructs and maintains a process pool containing multiple specialized processes responsible for different tasks, which work together during the training process to ensure efficient operation of the entire reinforcement learning training process. In this way, each computing device not only stores and manages the model, but also implements efficient parallel processing of model inference, computation, and training through multiple processes in the process pool. This architecture design enables efficient reinforcement learning training in a large-scale distributed environment, greatly accelerating the learning speed of the model and improving the scalability of the system.

[0045] Specifically, when the reinforcement learning training begins, the model inference process is started first. At this time, the system receives the prompt words required for inference and broadcasts these prompt words to the corresponding computing devices according to the number of data parallelism. After each computing device receives the prompt words, it performs model inference using the weights of the policy model (to-be-optimized model weights) according to the set tensor parallel or pipeline parallel mode. The main task of this stage is to generate inference results of the model according to the input prompt words to provide basic data for subsequent training steps.

[0046] After the model inference is completed, the reference model calculation process is started. The reference model performs forward calculation using its own weights. The purpose of this calculation process is to generate a reference result for subsequent reward calculation. After the calculation is completed, the calculation result of the reference model (reference result) and the result of the previous model inference (current inference result) are transmitted as inputs to the reward model calculation process. The reward model calculates the current loss value of the to-be-optimized inference model according to the current inference result and the reference result.

[0047] Among them, the model reinforcement learning optimization system provided by the embodiments of the application pre-allocates weight storage space in each computing device based on the distributed weight management module, and then loads each model weight into the storage space. After completing the weight loading, each computing device is responsible for managing and maintaining its own part of the weight, without the need to manage other parts of the weight. During the training process, each computing device calls the relevant weight space (weight storage space) in turn through the process pool manager, executes each training process, and then updates the respective weights, reducing the time for multiple loadings and accelerating the training efficiency.

[0048] Specifically, in an embodiment, the evaluation model is further configured to determine whether the to-be-optimized model satisfies a preset model optimization end condition according to the global loss value of the to-be-optimized model; and when the to-be-optimized model satisfies the preset model optimization end condition, the plurality of to-be-optimized model weights of the model reinforcement learning optimization system are merged to obtain a target model.

[0049] The preset model optimization end condition includes that the global loss value is lower than a preset threshold, the number of iterations reaches an upper limit, and the global loss value does not decrease continuously for multiple rounds.

[0050] Specifically, the evaluation model continuously monitors the training progress based on the global loss value, and when the end condition is met, the current to-be-optimized model is determined as the target inference model, that is, the weights of multiple to-be-optimized models of the model reinforcement learning optimization system are combined to obtain the target model.

[0051] Specifically, the training process of the to-be-optimized model is started, and the weights of the to-be-optimized model are updated. The goal of this stage is to adjust the weights of the policy model according to the experience value and loss value provided by the reward model, so as to optimize the behavior strategy of the model. After obtaining the experience value and loss value, the policy model calculates the gradient value of the loss function with respect to the parameters through the self-defined loss function through back propagation, and updates the parameters, thereby completing a complete training process. The entire system will repeatedly perform the above process until the model training reaches the predetermined end condition. Through this cyclic iteration, the policy model gradually learns the optimal behavior strategy to maximize the long-term return. In the entire training process, information transmission completely depends on the high-speed interconnection between devices, and the communication volume is comparable to that in the large-scale model pre-training stage. Each device is only responsible for maintaining its own parameters, thereby significantly improving the utilization efficiency of the device. In addition, by adopting the asynchronous update parameter mode, the overall performance is further improved, and the scalability of the application is enhanced.

[0052] Specifically, in an embodiment, the computing device is further configured to, after obtaining the target model, obtain a to-be-inferred instruction input by a user; and in response to the to-be-inferred instruction, perform inference based on the target model to obtain a target inference result corresponding to the to-be-inferred instruction.

[0053] Specifically, the computing device obtains a to-be-inferred instruction input by a user, such as a specific field knowledge query or a calculation task instruction, calls the target model to perform inference and output a result, and obtains a target inference result of the to-be-inferred instruction.

[0054] For example, the target inference result of the to-be-inferred instruction is a result of a specific field knowledge query or a calculation result. Figure 3As shown, the structural schematic diagram of the exemplary model reinforcement learning optimization system provided by the embodiment of the present application provides unified input for parallel computing by distributing prompt words to multiple computing devices (such as device 1 and device 2) through broadcasting. For the in-device process, taking device 1 as an example, the model weight to be optimized is called, the prompt word is reasoned, and the current reasoning result is generated. The reference model weight is used to generate the reference result corresponding to the prompt word, which provides a benchmark for subsequent loss calculation. In combination with the reasoning result and the reference result, the current loss value of the model to be optimized is calculated by means of the reward model weight / function, so as to feed back the current performance of the model. The current loss values of multiple model reinforcement learning optimization systems are summarized to determine the global loss value, the gradient is calculated based on the global loss value, the model weight to be optimized is updated, and the model optimization is realized; after the parallel update of multiple devices, the model weight parameters are synchronized to ensure the consistency of the model. The unified optimization decision across devices is supported.

[0055] Specifically, in an embodiment, a version recording function module can also be added in each computing device, which is used to generate a unique version identifier for each weight update, such as the parameter update after the policy model training process, associate the global loss value, the update time, the involved computing devices and the prompt word batch corresponding to the version, and store them in the independent partition of the cross-device storage pool. When the evaluation model monitors abnormal fluctuations in the global loss value, such as sudden increase exceeding the preset threshold or continuous multiple rounds of optimization not meeting the expectation, the rollback mechanism is automatically triggered, the historical optimal version is called based on the version record, such as the weight version when the global loss value is the lowest, the current weight is covered, and the training process is restarted, so as to further improve the reliability of the optimization result of the pre-trained language model.

[0056] The model reinforcement learning optimization system provided by the embodiments of the present application comprises: a plurality of computing devices, the weights of a to-be-optimized model, a reference model and a reward model being respectively distributed in the plurality of computing devices; the plurality of computing devices performing inference calculation based on the distributed to-be-optimized model weights to obtain a current inference result; the plurality of computing devices performing inference calculation based on the distributed reference model weights to obtain a reference result; the plurality of computing devices determining a current loss value based on the distributed reward model weights and the current inference result and the reference result; each computing device optimizing the to-be-optimized model weights of the current computing device according to the current loss value to obtain a target model; wherein the target model comprises the to-be-optimized model weights optimized by each computing device. The system provided by the above scheme can avoid the problems of idle and waste of computing resources of devices caused by the serial execution of different stage tasks in different devices in the related art, and improve the optimization efficiency of the pre-trained language model, because each computing device is distributed with the to-be-optimized model weights, the reference model weights and the reward model weights, that is, the computing device is deployed with all types of agents, and can simultaneously execute multiple computing tasks in the pre-trained language model optimization process. Moreover, each model weight is distributed on different computing devices instead of being copied to each device as a whole. In this way, only one memory is occupied for the same weight, thereby avoiding repeated occupation and waste of memory. Meanwhile, the training process is also simplified to a process switching operation in the embodiments of the present application. In this new deployment manner, different processes can simultaneously maintain the weight data in the same memory without frequent transmission of the weight between different devices. This not only reduces the overhead of data transmission, but also improves the training efficiency, reduces the complexity and delay of the system, and thus effectively improves the overall performance and resource utilization efficiency of the pre-trained language model reinforcement learning framework.

[0057] From the above description of the embodiments, those skilled in the art can clearly understand that the system according to the above embodiments can be implemented by means of software and the necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is a better implementation.

[0058] The embodiments of the present application provide a model reinforcement learning optimization method, applied to the model reinforcement learning optimization system provided by the above embodiments.

[0059] As Figure 4 shown, a flowchart of the model reinforcement learning optimization method provided by the embodiments of the present application is shown, the method comprises: Step 401, control the plurality of computing devices in the model reinforcement learning optimization system to perform inference calculation based on the distributed to-be-optimized model weights to obtain a current inference result; Step 402, control the plurality of computing devices to perform inference calculation based on the distributed reference model weights to obtain a reference result; Step 403 , controlling the plurality of computing devices to determine a current loss value based on the assigned reward model weights, the current inference result, and the reference result; Step 404: Optimize the weights of the models to be optimized of each computing device according to the current loss value to obtain the target model.

[0060] The target model includes the weights of the models to be optimized after optimization of each computing device.

[0061] For the description of the features in the embodiment corresponding to the model reinforcement learning optimization method, please refer to the relevant description of the embodiment corresponding to the model reinforcement learning optimization system, and will not be repeated here.

[0062] An embodiment of the present application also provides a model reinforcement learning optimization device for executing the model reinforcement learning optimization method provided in the above embodiment.

[0063] like Figure 5 FIG. 5 is a schematic diagram of the structure of a model reinforcement learning optimization device according to an embodiment of the present application. The model reinforcement learning optimization device 50 includes: a first control module 501 , a second control module 502 , a third control module 503 and an optimization module 504 .

[0064] Among them, the first control module is used to control multiple computing devices in the model reinforcement learning optimization system to perform inference calculations based on the assigned weights of the model to be optimized to obtain the current inference results; the second control module is used to control multiple computing devices to perform inference calculations based on the assigned reference model weights to obtain reference results; the third control module is used to control multiple computing devices to determine the current loss value based on the assigned reward model weights according to the current inference results and reference results; the optimization module is used to optimize the weights of the model to be optimized of each computing device according to the current loss value to obtain the target model.

[0065] The target model includes the weights of the models to be optimized after optimization of each computing device.

[0066] For the description of the features in the embodiment corresponding to the model reinforcement learning optimization device, please refer to the relevant description of the embodiment corresponding to the model reinforcement learning optimization method, and will not be repeated here.

[0067] The embodiment of the present application also provides an electronic device, such as Figure 6 As shown, it is a structural diagram of an electronic device provided in an embodiment of the present application, including a processor 10 and a memory 20, in which a computer program is stored. The processor 10 is configured to run the computer program to execute the steps in any one of the above-mentioned model reinforcement learning optimization method embodiments.

[0068] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, wherein the computer program is arranged to execute the steps in any of the model reinforcement learning optimization method embodiments when running.

[0069] In an example embodiment, the computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media capable of storing computer programs.

[0070] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in any of the model reinforcement learning optimization method embodiments.

[0071] The embodiment of the present application further provides another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in any of the model reinforcement learning optimization method embodiments.

[0072] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0073] The above provides a detailed description of the model reinforcement learning optimization system, method and electronic device provided by the present application. The principles and implementation modes of the present application are described by applying specific examples in this paper, and the above example descriptions are only used to help understand the method and core idea of the present application. It should be pointed out that, for ordinary skilled person in the technical field, some improvements and modifications can be made to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A model reinforcement learning optimization system, characterized in that: include: A plurality of computing devices, wherein weights of the model to be optimized, the reference model, and the reward model are respectively distributed to the plurality of computing devices; The multiple computing devices perform reasoning calculations based on the assigned weights of the models to be optimized to obtain current reasoning results; The plurality of computing devices perform reasoning calculations based on the assigned reference model weights to obtain reference results; The plurality of computing devices determine a current loss value according to the current inference result and a reference result based on the assigned reward model weights; Each of the computing devices optimizes the weight of the model to be optimized of the current computing device according to the current loss value to obtain a target model; The target model includes the weights of the models to be optimized after optimization by each of the computing devices.

2. The model reinforcement learning optimization system according to claim 1, characterized in that: The system further includes: a distributed weight management module; The distributed weight management module is used to allocate the weight of the model to be optimized, the reference model weight and the reward model weight in the weight storage space of each computing device.

3. The model reinforcement learning optimization system according to claim 1, characterized in that: The reward model is used to: Determining a current empirical value of the model to be optimized based on the current reasoning result; determining a reference value according to the reference result; The current loss value is determined according to the gap between the current reasoning result and the reference result represented by the current experience value and the reference value.

4. The model reinforcement learning optimization system according to claim 1, characterized in that: The plurality of computing devices are further assigned evaluation model weights; Sharing the current loss value between different model reinforcement learning optimization systems; The multiple computing devices determine the global loss value of the model to be optimized based on the assigned evaluation model weights and the current loss value shared between the model reinforcement learning optimization systems.

5. The model reinforcement learning optimization system according to claim 4, characterized in that: Each of the computing devices optimizes the weight of the model to be optimized of the current computing device according to the global loss value.

6. The model reinforcement learning optimization system according to claim 5, characterized in that: The computing device includes: a process pool and a process manager; The process pool includes multiple model optimization processes, including model inference process, reference model calculation process, reward model calculation process, strategy model training process and evaluation model calculation process; The model inference process is used to call the weight of the model to be optimized, infer the prompt word, and generate the current inference result; The reference model calculation process is used to call the reference model weight to generate a reference result corresponding to the prompt word; The reward model calculation process is used to call the reward model weight and calculate the current loss value according to the current inference result and the reference result; The strategy model training process is used to call the weight of the model to be optimized, determine the gradient value of the model to be optimized according to the global loss value, and optimize the weight of the model to be optimized according to the gradient value of the model to be optimized; The evaluation model calculation process is used to call the evaluation model weight and determine the global loss value of the model to be optimized based on the current loss value determined by all computing devices; The process manager is used to synchronously control the model inference process, reference model calculation process, reward model calculation process, strategy model training process and evaluation model calculation process, so that multiple computing devices in the same model reinforcement learning optimization system run the same model optimization process at the same time.

7. The model reinforcement learning optimization system according to claim 6, characterized in that: The evaluation model is also used to: According to the global loss value of the model to be optimized, determining whether the model to be optimized meets the preset model optimization end condition; When the model to be optimized meets the preset model optimization end condition, the weights of multiple models to be optimized in the model reinforcement learning optimization system are merged to obtain the target model.

8. The model reinforcement learning optimization system according to claim 1, characterized in that: The computing device is further configured to: After obtaining the target model, obtain the instructions to be inferred from the user; In response to the instruction to be inferred, reasoning is performed based on the target model to obtain a target reasoning result corresponding to the instruction to be inferred.

9. A model reinforcement learning optimization method, applied to the model reinforcement learning optimization system according to any one of claims 1 to 8, characterized in that: The method comprises: Controlling multiple computing devices in the model reinforcement learning optimization system to perform inference calculations based on the assigned weights of the model to be optimized to obtain current inference results; Controlling the plurality of computing devices to perform inference calculations based on the assigned reference model weights to obtain reference results; controlling the plurality of computing devices to determine a current loss value according to the current inference result and a reference result based on the assigned reward model weights; Optimizing the weight of the model to be optimized of each computing device according to the current loss value to obtain a target model; The target model includes the weights of the models to be optimized after optimization by each of the computing devices.

10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the model reinforcement learning optimization method as claimed in claim 9 when executing the computer program.

Images