Reinforcement learning model training method, device, electronic device and storage medium

Abstract

The present invention discloses a reinforcement learning model training method, device, electronic device and storage medium. The method comprises: obtaining a pre-trained model obtained by reinforcement learning training based on first scene sample data, wherein the pre-trained model comprises a state perception network and an action decision network; reusing the action decision network in the pre-trained model; obtaining second scene sample data, and training the state perception network in the pre-trained model based on the second scene sample data to obtain a target reinforcement learning model. The above technical scheme realizes the transfer learning of the reinforcement learning model and effectively improves the prediction accuracy of the reinforcement learning model in the new scene.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning technology, and in particular to a reinforcement learning model training method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of reinforcement learning technology, reinforcement learning technology has been widely used in various scenarios.

[0003] In the process of implementing the present invention, it is found that there are at least the following technical problems in the prior art: the existing reinforcement learning technical solution has the problem of low prediction accuracy when the reinforcement learning model is migrated to a new scenario. Summary of the invention

[0004] The present invention provides a reinforcement learning model training method, device, electronic device and storage medium to improve the prediction accuracy of the reinforcement learning model.

[0005] According to one aspect of the present invention, a reinforcement learning model training method is provided, comprising:

[0006] Acquire a pre-trained model obtained by performing reinforcement learning training according to the first scene sample data, wherein the pre-trained model includes a state perception network and an action decision network;

[0007] Reusing the action decision network in the pre-trained model;

[0008] Acquire second scenario sample data, and train the state perception network in the pre-trained model based on the second scenario sample data to obtain a target reinforcement learning model.

[0009] According to another aspect of the present invention, a reinforcement learning model training device is provided, comprising:

[0010] A pre-training model acquisition module, used to acquire a pre-training model obtained by performing reinforcement learning training according to the first scene sample data, wherein the pre-training model includes a state perception network and an action decision network;

[0011] An action decision network multiplexing module, used for multiplexing the action decision network in the pre-trained model;

[0012] The target reinforcement learning model determination module is used to obtain second scene sample data, train the state perception network in the pre-trained model based on the second scene sample data, and obtain the target reinforcement learning model.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] at least one processor;

[0015] and a memory communicatively coupled to the at least one processor;

[0016] In which, the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the reinforcement learning model training method described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the reinforcement learning model training method described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention is to obtain a pre-trained model obtained by reinforcement learning training based on sample data of the first scene, wherein the pre-trained model includes a state perception network and an action decision network; reuse the action decision network in the pre-trained model; obtain sample data of the second scene, and train the state perception network in the pre-trained model based on the sample data of the second scene to obtain a target reinforcement learning model, thereby realizing transfer learning of the reinforcement learning model and effectively improving the prediction accuracy of the reinforcement learning model in the new scene.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flowchart of a reinforcement learning model training method provided according to Embodiment 1 of the present invention;

[0022] Figure 2 is a flowchart of a reinforcement learning model training method provided according to Embodiment 2 of the present invention;

[0023] Figure 3 is a schematic diagram of a reinforcement learning model training provided according to an embodiment of the present invention;

[0024] Figure 4 is a structural diagram of a reinforcement learning model training device provided according to Embodiment 3 of the present invention;

[0025] Figure 5 It is a structural schematic diagram of an electronic device for implementing the reinforcement learning model training method of an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The acquisition, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0028] Embodiment 1

[0029] Figure 1 This is a flowchart of a reinforcement learning model training method provided in the first embodiment of the present invention. This embodiment is applicable to the case of reinforcement learning model transfer training. The method can be executed by a reinforcement learning model training device. The reinforcement learning model training device can be implemented in the form of hardware and / or software. The reinforcement learning model training device can be configured in a terminal and / or a server. Figure 1 As shown, the method includes:

[0030] S110. Obtain a pre-trained model obtained by performing reinforcement learning training based on the first scene sample data, wherein the pre-trained model includes a state perception network and an action decision network.

[0031] S120: Reusing the action decision network in the pre-trained model.

[0032] S130. Obtain second scenario sample data, and train the state perception network in the pre-trained model based on the second scenario sample data to obtain a target reinforcement learning model.

[0033] In the disclosed embodiments, the first scene sample data or the second scene sample data refers to the model training data used by the reinforcement learning model in a specific application scenario, wherein the first scene sample data and the second scene sample data are model training data in different application scenarios. Exemplarily, the application scenario may be a game confrontation, an industrial site, a medical treatment, and the like. For example, the first scene sample data may be data that can be used for model training in the A, B, and C map scenes in the game, and the second scene sample data may be data that can be used for model training in the D map scene in the game. The model training data may include data such as character health, character attack power, and character position.

[0034] Exemplarily, the first scenario sample data or the second scenario sample data can be read from a preset storage path of the electronic device, or can be retrieved from other devices or the cloud that are communicatively connected to the electronic device, without limitation here.

[0035] In an embodiment of the present disclosure, a pre-trained model can be obtained by reinforcement learning training based on sample data of the first scene. The pre-trained model is not limited to a state perception network and an action decision network. The state perception network can be used to learn environmental state information, and the action decision network is used to output a state value function value.

[0036] It should be noted that the pre-trained action decision module is used for training in the new scenario, and the network parameters of the action decision module are fixed, so that the model training in the new scenario only needs to focus on training the state perception network, which reduces the model training time and thus improves the model training efficiency of reinforcement learning.

[0037] Optionally, the first scene sample data includes first state information corresponding to the first intelligent agent in the first map scene and first action information corresponding to the first intelligent agent in the first map scene; accordingly, the training step of the pre-trained model includes: inputting the first state information corresponding to the first intelligent agent in the first map scene and the first action information corresponding to the first intelligent agent in the first map scene into the initial reinforcement learning model to obtain a first state value function value; updating the model parameters of the initial reinforcement learning model based on the first state value function value until the training stop condition of the initial reinforcement learning model is met to obtain a pre-trained model.

[0038] In the disclosed embodiment, the first map scene may be a scene corresponding to a map in a game, etc., the first agent may be a character in a game, etc., the number of the first agents may be one or more, the first state information may be information such as the health and attack power of the character, and the first action information may be information such as the displacement or attack action of the character. The first state value function value refers to the action value function value corresponding to the current state information output by the reinforcement learning model.

[0039] It should be noted that the state information and the action information are used together as the input of the model for training, so that the action decision module in the reinforcement learning model can be reused.

[0040] Optionally, the first state value function value includes state value function values ​​corresponding to multiple tasks; accordingly, the model parameters of the initial reinforcement learning model are updated based on the first state value function value, including: aggregating the state value function values ​​corresponding to multiple tasks to obtain a model prediction value; inputting the model prediction value and the model target value into a preset loss function to obtain a model loss; and updating the network parameters corresponding to the state perception network in the pre-trained model based on the model loss.

[0041] In the disclosed embodiment, each task has corresponding state information and action information, and a state-value function value corresponding to the state information and the action information.

[0042] Specifically, the Mixer module can be used to aggregate the state value function values ​​corresponding to multiple tasks to obtain the model prediction value, and then determine the model loss based on the model prediction value and the model target value. Then, the network parameters corresponding to the state-aware network in the pre-trained model are updated according to the model loss until the model training is completed, where the model target value refers to the label of the model training.

[0043] Optionally, the preset loss function includes:

[0044]

[0045]

[0046]

[0047] Among them, ω i represents the weight of the i-th task, loss i represents the loss function of the i-th task; An evaluation index representing the degree of convergence of the i-th task; represents the model prediction value corresponding to the jth training sample, represents the model target value corresponding to the jth training sample, N represents the number of tasks, and M represents the number of training samples.

[0048] In the disclosed embodiment, the evaluation index of the task convergence degree may be the probability of task failure under the current state information, etc. It should be noted that the present embodiment ensures the training accuracy of the model in the case of multiple tasks by dynamically adjusting the weight of each task.

[0049] Optionally, the state-aware network includes a fully connected layer and a gated recurrent unit, and the fully connected layer is connected to the gated recurrent unit; the action decision network includes a fully connected layer.

[0050] Exemplarily, the state information and action information can be input together into the fully connected layer of the state-aware network, so that the input dimension is transformed into the dimension of the hidden layer, and then the dimension-transformed features are input into the gated recurrent unit, and then the gated recurrent unit outputs the state-aware features, and then the state-aware features are input into the fully connected layer of the action decision network to obtain a one-dimensional state value function value.

[0051] The technical solution of the embodiment of the present invention is to obtain a pre-trained model obtained by reinforcement learning training based on sample data of the first scene, wherein the pre-trained model includes a state perception network and an action decision network; reuse the action decision network in the pre-trained model; obtain sample data of the second scene, and train the state perception network in the pre-trained model based on the sample data of the second scene to obtain a target reinforcement learning model, thereby realizing transfer learning of the reinforcement learning model and effectively improving the prediction accuracy of the reinforcement learning model in the new scene.

[0052] Embodiment 2

[0053] Figure 2 A flowchart of a reinforcement learning model training method provided in the second embodiment of the present invention, the method of this embodiment can be combined with the various optional schemes in the reinforcement learning model training method provided in the above embodiments. The reinforcement learning model training method provided in this embodiment has been further optimized. Optionally, the second scene sample data includes the second state information corresponding to the second agent in the second map scene and the second action information corresponding to the second agent in the second map scene; accordingly, the state perception network in the pre-trained model is trained based on the second scene sample data to obtain the target reinforcement learning model, including: inputting the second state information corresponding to the second agent in the second map scene and the second action information corresponding to the second agent in the second map scene into the pre-trained model to obtain the second state value function value; based on the second state value function value, the network parameters corresponding to the state perception network in the pre-trained model are updated until the model training stop condition is met to obtain the target reinforcement learning model.

[0054] like Figure 2 As shown, the method includes:

[0055] S210. Obtain a pre-trained model obtained by performing reinforcement learning training based on the first scene sample data, wherein the pre-trained model includes a state perception network and an action decision network.

[0056] S220: Reuse the action decision network in the pre-trained model.

[0057] S230. Obtain second scene sample data, where the second scene sample data includes second state information corresponding to the second agent in the second map scene and second action information corresponding to the second agent in the second map scene.

[0058] S240. Input the second state information corresponding to the second agent in the second map scene and the second action information corresponding to the second agent in the second map scene into the pre-trained model to obtain a second state value function value.

[0059] S250. Update the network parameters corresponding to the state-aware network in the pre-trained model based on the second state-value function value until the model training stop condition is met to obtain the target reinforcement learning model.

[0060] In the disclosed embodiment, the second map scene may be a scene corresponding to a map in a game, etc., the second agent may be a character in a game, etc., the number of the second agents may be one or more, the state information may be information such as the health and attack power of the character, and the action information may be information such as the displacement or attack action of the character. The second state value function value refers to the action value function value corresponding to the current state information output by the reinforcement learning model.

[0061] For example, Figure 3 is a schematic diagram of a reinforcement learning model training provided according to an embodiment of the present invention. Specifically, in the model pre-training stage, the initial reinforcement learning model can be pre-trained using the state information (s) and action information (a) of N tasks to obtain a pre-trained model. N (s,a) represents the state value function value corresponding to the Nth task. In the model migration stage, the action decision network in the pre-trained model can be reused, and the state perception network can be trained based on the state information and action information corresponding to the new task (task new) to obtain the target reinforcement learning model. It should be noted that in the process of training the new task model, only the state perception network needs to be trained, thereby reducing the model training time and improving the model training efficiency of reinforcement learning.

[0062] The technical solution of the embodiment of the present invention obtains a second state value function value by inputting the second state information corresponding to the second intelligent agent in the second map scene and the second action information corresponding to the second intelligent agent in the second map scene into the pre-trained model, and then updates the network parameters corresponding to the state perception network in the pre-trained model based on the second state value function value until the model training stop condition is met, thereby obtaining the target reinforcement learning model, realizing the transfer learning of the reinforcement learning model, and effectively improving the prediction accuracy of the reinforcement learning model in the new scenario.

[0063] Embodiment 3

[0064] Figure 4 This is a schematic diagram of the structure of a reinforcement learning model training device provided in Embodiment 3 of the present invention. Figure 4 As shown, the device comprises:

[0065] A pre-training model acquisition module 310 is used to acquire a pre-training model obtained by performing reinforcement learning training according to the first scene sample data, wherein the pre-training model includes a state perception network and an action decision network;

[0066] An action decision network multiplexing module 320, used for multiplexing the action decision network in the pre-trained model;

[0067] The target reinforcement learning model determination module 330 is used to obtain second scene sample data, and train the state perception network in the pre-trained model based on the second scene sample data to obtain a target reinforcement learning model.

[0068] The technical solution of the embodiment of the present invention is to obtain a pre-trained model obtained by reinforcement learning training based on sample data of the first scene, wherein the pre-trained model includes a state perception network and an action decision network; reuse the action decision network in the pre-trained model; obtain sample data of the second scene, and train the state perception network in the pre-trained model based on the sample data of the second scene to obtain a target reinforcement learning model, thereby realizing transfer learning of the reinforcement learning model and effectively improving the prediction accuracy of the reinforcement learning model in the new scene.

[0069] In some optional embodiments, the second scene sample data includes second state information corresponding to the second agent in the second map scene and second action information corresponding to the second agent in the second map scene;

[0070] Accordingly, the target reinforcement learning model determination module 330 includes:

[0071] A second state value function value determining unit, configured to input second state information corresponding to the second agent in the second map scene and second action information corresponding to the second agent in the second map scene into the pre-trained model to obtain a second state value function value;

[0072] The state-aware network updating unit is used to update the network parameters corresponding to the state-aware network in the pre-trained model based on the second state value function value until the model training stop condition is met to obtain the target reinforcement learning model.

[0073] In some optional embodiments, the first scene sample data includes first state information corresponding to the first agent in the first map scene and first action information corresponding to the first agent in the first map scene;

[0074] Accordingly, the training process of the pre-trained model includes:

[0075] A first state value function value determination module, used to input first state information corresponding to the first agent in the first map scene and first action information corresponding to the first agent in the first map scene into an initial reinforcement learning model to obtain a first state value function value;

[0076] The pre-trained model parameter updating module is used to update the model parameters of the initial reinforcement learning model based on the first state value function value until the training stop condition of the initial reinforcement learning model is met to obtain the pre-trained model.

[0077] In some optional implementations, the first state-value function value includes state-value function values ​​corresponding to a plurality of tasks; accordingly, the pre-trained model parameter updating module is further used to:

[0078] Aggregate the state value function values ​​corresponding to multiple tasks to obtain the model prediction value;

[0079] Inputting the model prediction value and the model target value into a preset loss function to obtain the model loss;

[0080] The network parameters corresponding to the state-aware network in the pre-trained model are updated based on the model loss.

[0081] In some optional implementations, the preset loss function includes:

[0082]

[0083]

[0084]

[0085] Among them, ω i represents the weight of the i-th task, loss i represents the loss function of the i-th task; An evaluation index representing the degree of convergence of the i-th task; represents the model prediction value corresponding to the jth training sample, represents the model target value corresponding to the jth training sample, N represents the number of tasks, and M represents the number of training samples.

[0086] In some optional embodiments, the state-aware network includes a fully connected layer and a gated recurrent unit, and the fully connected layer is connected to the gated recurrent unit; the action decision network includes a fully connected layer.

[0087] The reinforcement learning model training device provided in the embodiment of the present invention can execute the reinforcement learning model training method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0088] Embodiment 4

[0089] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0090] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The I / O interface 15 is also connected to the bus 14.

[0091] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0092] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 performs the various methods and processes described above, such as a reinforcement learning model training method, which includes:

[0093] Acquire a pre-trained model obtained by performing reinforcement learning training according to the first scene sample data, wherein the pre-trained model includes a state perception network and an action decision network;

[0094] Reusing the action decision network in the pre-trained model;

[0095] Acquire second scenario sample data, and train the state perception network in the pre-trained model based on the second scenario sample data to obtain a target reinforcement learning model.

[0096] In some embodiments, the reinforcement learning model training method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the reinforcement learning model training method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the reinforcement learning model training method in any other appropriate manner (e.g., by means of firmware).

[0097] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0098] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0099] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0100] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0101] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0102] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0103] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0104] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A reinforcement learning model training method, characterized in that: include: Obtain a pre-trained model obtained by performing reinforcement learning training based on the first scene sample data, wherein the pre-trained model includes a state perception network and an action decision network; the first scene sample data includes first state information corresponding to the first agent in the first map scene and first action information corresponding to the first agent in the first map scene; the first state information includes the health and attack power of the game character, and the first action information includes the displacement or attack action of the game character; Reusing the action decision network in the pre-trained model; Acquire second scene sample data, and train the state perception network in the pre-trained model based on the second scene sample data to obtain a target reinforcement learning model; the second scene sample data includes second state information corresponding to the second agent in the second map scene and second action information corresponding to the second agent in the second map scene; the second state information includes the health and attack power of the game character, and the second action information includes the displacement or attack action of the game character; The training steps of the pre-trained model include: Inputting first state information corresponding to the first agent in the first map scene and first action information corresponding to the first agent in the first map scene into the initial reinforcement learning model to obtain a first state value function value; Based on the first state-value function value, the model parameters of the initial reinforcement learning model are updated until a training stop condition of the initial reinforcement learning model is met, thereby obtaining a pre-trained model; The first state-value function value includes state-value function values ​​corresponding to a plurality of tasks; Accordingly, updating the model parameters of the initial reinforcement learning model based on the first state-value function value includes: Aggregate the state value function values ​​corresponding to multiple tasks to obtain the model prediction value; Inputting the model prediction value and the model target value into a preset loss function to obtain the model loss; Based on the model loss, the network parameters corresponding to the state perception network in the pre-trained model are updated; The preset loss function includes: Among them, ω i represents the weight of the i-th task, loss i represents the loss function of the i-th task; The evaluation index of the convergence degree of the i-th task is the probability of task failure under the current state information. represents the model prediction value corresponding to the jth training sample, represents the model target value corresponding to the jth training sample, N represents the number of tasks, and M represents the number of training samples.

2. The method according to claim 1, characterized in that: The step of training the state perception network in the pre-trained model based on the second scene sample data to obtain a target reinforcement learning model includes: Inputting second state information corresponding to the second agent in the second map scene and second action information corresponding to the second agent in the second map scene into the pre-trained model to obtain a second state value function value; Based on the second state value function value, the network parameters corresponding to the state perception network in the pre-trained model are updated until the model training stop condition is met to obtain the target reinforcement learning model.

3. The method according to claim 1, characterized in that: The state perception network includes a fully connected layer and a gated recurrent unit, and the fully connected layer is connected to the gated recurrent unit; the action decision network includes a fully connected layer.

4. A reinforcement learning model training device, characterized in that: include: A pre-trained model acquisition module is used to acquire a pre-trained model obtained by reinforcement learning training based on the first scene sample data, wherein the pre-trained model includes a state perception network and an action decision network; the first scene sample data includes first state information corresponding to the first intelligent agent in the first map scene and first action information corresponding to the first intelligent agent in the first map scene; the first state information includes the health and attack power of the game character, and the first action information includes the displacement or attack action of the game character; An action decision network multiplexing module, used for multiplexing the action decision network in the pre-trained model; a target reinforcement learning model determination module, used to obtain second scene sample data, and train the state perception network in the pre-trained model based on the second scene sample data to obtain a target reinforcement learning model; the second scene sample data includes second state information corresponding to the second agent in the second map scene and second action information corresponding to the second agent in the second map scene; the second state information includes the health and attack power of the game character, and the second action information includes the displacement or attack action of the game character; The training steps of the pre-trained model include: A first state value function value determination module, used to input first state information corresponding to the first agent in the first map scene and first action information corresponding to the first agent in the first map scene into an initial reinforcement learning model to obtain a first state value function value; A pre-trained model parameter updating module, used to update the model parameters of the initial reinforcement learning model based on the first state value function value until the training stop condition of the initial reinforcement learning model is met, thereby obtaining a pre-trained model; The first state-value function value includes state-value function values ​​corresponding to a plurality of tasks; Correspondingly, the pre-trained model parameter updating module is also used for: Aggregate the state value function values ​​corresponding to multiple tasks to obtain the model prediction value; Inputting the model prediction value and the model target value into a preset loss function to obtain the model loss; Based on the model loss, the network parameters corresponding to the state perception network in the pre-trained model are updated; The preset loss function includes: Among them, ω i represents the weight of the i-th task, loss i represents the loss function of the i-th task; The evaluation index of the convergence degree of the i-th task is the probability of task failure under the current state information. represents the model prediction value corresponding to the jth training sample, represents the model target value corresponding to the jth training sample, N represents the number of tasks, and M represents the number of training samples.

5. The device according to claim 4, characterized in that The target reinforcement learning model determination module includes: A second state value function value determining unit, configured to input second state information corresponding to the second agent in the second map scene and second action information corresponding to the second agent in the second map scene into the pre-trained model to obtain a second state value function value; The state-aware network updating unit is used to update the network parameters corresponding to the state-aware network in the pre-trained model based on the second state value function value until the model training stop condition is met to obtain the target reinforcement learning model.

6. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein, the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the reinforcement learning model training method described in any one of claims 1-3.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the reinforcement learning model training method described in any one of claims 1 to 3 when executed.

Images