Learning program, learning method, and information processing device
The learning program efficiently initializes the learning model using weighted parameters, addressing inefficiencies in transfer learning for deep reinforcement learning by enhancing speed and reducing time without increasing model size.
Patent Information
- Application Number
- JP2024014734
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-02-02
- Publication Date
- 2025-08-15
AI Technical Summary
Transfer learning in deep reinforcement learning (DRL) is inefficient due to factors like parameter reuse, leading to increased model complexity and learning time, which is not effectively addressed by existing methods.
A learning program that determines a first parameter by weighting parameters of a trained model and sets a second parameter based on the first parameter, using an information processing device to initialize the learning model efficiently.
Transfer learning is efficiently utilized in deep reinforcement learning without increasing model size, thereby improving learning speed and reducing time.
Smart Images
Figure 2025119754000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a learning program, a learning method, and an information processing device. [Background technology]
[0002] Transfer learning is a common learning method in deep learning, which can shorten the learning time and enable efficient learning. Therefore, it is expected that transfer learning can also be used to efficiently learn in deep reinforcement learning. [Prior art documents] [Non-patent literature]
[0003] [Non-Patent Document 1] Haobin Shi, Jingchen Li, Jiahui Mao, and Kao-Shing Hwang, Lateral Transfer Learning for Multiagent Reinforcement Learning IEEE Transactions on Cybernetics, Volume53, Issue3, MARCH 2023 Summary of the Invention [Problem to be solved by the invention]
[0004] When using transfer learning in deep reinforcement learning (DRL), such as multi-agent DRL, the learning time may not be shortened due to factors such as parameter reuse. Therefore, there is a need for a method to efficiently use transfer learning in DRL.
[0005] Furthermore, there is a method described in Non-Patent Document 1 as a method for shortening learning time. In Non-Patent Document 1, agents are clustered from a model from which the agent is transferred (hereinafter may be referred to as the source), and a cluster that is close to the task of the model to which the agent is transferred (hereinafter may be referred to as the target) is assigned to the target agent. However, with this method, the model becomes more complex with each iteration of transfer learning, and the model size may increase. As a result, the enlargement of the model size may have a negative impact on the learning time, etc. Therefore, it cannot be said that transfer learning can be used efficiently for deep reinforcement learning.
[0006] The disclosed technology has been made in consideration of the above, and aims to efficiently use transfer learning in deep reinforcement learning. [Means for solving the problem]
[0007] In one aspect, a learning program is provided that performs learning using a learning model that utilizes a trained model executed by an information processing device, characterized by causing the information processing device to execute a process that determines a first parameter by weighting the parameters of the trained model in a predetermined manner, and sets a second parameter, which is the value of the initial parameter of the learning model, based on the first parameter. [Effects of the Invention]
[0008] Transfer learning can be used efficiently in deep reinforcement learning. [Brief explanation of the drawings]
[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of an information processing device according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of an output process of a learning model according to an embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of an information processing device that executes a learning program according to an embodiment. [Figure 4] FIG. 4 is a schematic diagram of an application environment in a specific example 1 to which the embodiment is applied. [Figure 5] FIG. 5 is a diagram showing the results of reinforcement learning in specific example 1 to which the embodiment is applied. [Figure 6] FIG. 6 is a diagram showing an application environment in a specific example 2 to which the embodiment is applied. [Figure 7] FIG. 7 is a diagram showing the results of reinforcement learning in specific example 2 to which the embodiment is applied. DETAILED DESCRIPTION OF THE INVENTION
[0010] The present embodiment will be described in detail below with reference to the drawings. The problems and embodiments in this specification are merely examples and do not limit the scope of the rights of the present application. In particular, even if the expressions used are different, the technology of the present application can be applied as long as they are technically equivalent, and do not limit the scope of the rights. Furthermore, each embodiment can be combined as appropriate within the scope of not causing any contradiction in the processing content.
[0011] Furthermore, the terms used and the technical contents described in this specification may be those described in specifications and contributions as standards, or the technical contents of papers, etc., as appropriate.
[0012] Hereinafter, embodiments of a learning program, a learning method, and an information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the disclosed technology is not limited to the following embodiments.
[0013] [Embodiment Mode] FIG. 1 shows an example of a functional configuration diagram of an information processing device according to an embodiment. The information processing device 100 performs, for example, a learning process of multi-agent deep reinforcement learning. The information processing device 100 also includes a control unit 110 and a storage unit 120. The control unit 110 can be described as being divided into an initial setting unit 111 and a learning unit 112. These functional configurations may not be completed by a single information processing device 100 alone, but may be configured to function in combination with multiple information processing devices. These components may also be connected via a network in any unit and function.
[0014] The initial setting unit 111 uses a learned model of the transfer source (hereinafter sometimes referred to as the source domain) in transfer learning to perform initial settings on the parameters (e.g., values of each agent in multi-agent deep reinforcement learning) of the learning model of the application target (hereinafter sometimes referred to as the target domain).
[0015] The learning unit 112 performs deep reinforcement learning using the learning model for which the parameters have been set, and outputs a learned model.
[0016] The storage unit 120 can store various data used in deep reinforcement learning, such as data related to the source domain and a function for adding noise, for example.
[0017] First, two methods by which the initial setting unit 111 of the information processing device 100 determines the initial values of the parameters of the learning model of the target domain will be described in detail.
[0018] First, the first method will be described.
[0019] In the learning process in the source domain, the observed values of each agent in the learning model are saved in advance. Then, the observed values are stored in the storage unit 120 of the information processing device 100 together with the learned model of the source domain. Note that the learning process in the source domain may be performed by another information processing device.
[0020] Furthermore, the initial setting unit 111 of the information processing device 100 acquires the observed values of each agent using predetermined control in the target domain (for example, control performed in the target domain), and stores the values in the storage unit 120 of the information processing device 100. Note that the process of acquiring the observed values may be performed by another information processing device.
[0021] Next, the initial setting unit 111 of the information processing device 100 calculates the correlation between the observation value of each agent in the source domain and the observation value of each agent in the target domain. Then, for each agent in the target domain, a weighted average value is calculated based on the observation value of each agent in the source domain and the correlation between the observation value of the agent in the target domain and the observation value of each agent in the source domain. The weighted average value is an example of a first parameter. The weighted average value can be calculated, for example, by assigning a weight based on the correlation to each of the observation values of the agents in the source domain (e.g., a value such that the sum of the weights for each agent in the target domain is 1), and then using the weight and the observation value of each agent in the source domain to calculate the weighted average value for each agent in the target domain.
[0022] The initial setting unit 111 of the information processing device 100 initializes the value corresponding to the bias term of the output layer, among the above-mentioned weighted average values, to 0, and sets the result as the initial parameter of the learning model of the target domain. Note that a vector (noise) using a predetermined function (e.g., a pseudorandom number generation function) may be added to the weighted average value, and then the value corresponding to the bias term of the output layer may be initialized to 0, and set as the initial parameter. Note that the processes of acquiring and storing the observation values of the source domain and the target domain may be performed in different orders between the source domain and the target domain. Note that the value of the initial parameter is an example of a second parameter.
[0023] Next, a second method will be described. This method can be used when applying a learning model to, for example, the control of a base station in a wireless communication system.
[0024] First, for each agent in the source domain, the features of each agent, such as its relative coordinates and type, are converted into an embedded representation using a means such as a Graph Auto-Encoder. Also, the communication volume data for each predetermined range (grid) in the source domain is converted into an embedded representation using a means such as a time-series Auto-Encoder.
[0025] Then, for each agent in the source domain, an embedded representation (feature) of the source domain is generated by combining the embedded representation of the agent and the embedded representation of the grid in which the agent is included.
[0026] In addition, the initial setting unit 111 of the information processing device 100 generates an embedded representation (features) of the target domain based on features such as the relative coordinates and type of each agent, and data on communication volume for each specified range (grid) in the target domain, using the same means in the target domain as in the source domain.
[0027] The initial setting unit 111 of the information processing device 100 then calculates the correlation between each of the feature amounts of the source domain processed as described above and each of the feature amounts of the target domain. Then, for each agent in the target domain, a weighted average value is calculated based on the feature amounts of each agent in the source domain and the correlation between the feature amounts of the agent in the target domain and each agent in the source domain. The weighted average value is an example of a first parameter. A method for calculating the weighted average value is, for example, to perform weighting in the same manner as the first method, and calculate the weighted average value for each agent in the target domain using the weights and the feature amounts of each agent in the source domain.
[0028] The initial setting unit 111 of the information processing device 100 initializes the value corresponding to the bias term of the output layer, among the above-mentioned weighted average values, to 0, and sets the result as the initial parameter of the learning model of the target domain. Note that the weighted average value may be given a vector (noise) using a predetermined function (e.g., a pseudorandom number generation function), and then the value corresponding to the bias term of the output layer may be initialized to 0, and set as the initial parameter. Note that the processes of generating and storing the embedded representations of the source domain and the target domain may be performed in different orders between the source domain and the target domain. Note that the value of the initial parameter is an example of a second parameter.
[0029] The initial setting unit 111 of the information processing device 100 can set the initial values of the parameters of the learning model of the target domain using either of these two methods. Note that the initial values of the parameters of the learning model of the target domain may be set by performing similar processing using multiple trained models of the source domain. Furthermore, the initial setting unit 111 of the information processing device 100 may use the trained model of the target domain trained by the training unit 112 of the information processing device 100 as the trained model of the source domain to set the initial values of the parameters of other training models of the target domain by performing similar processing.
[0030] Next, the reinforcement learning performed by the learning unit 112 of the information processing device 100 using a learning model of a target domain for which parameters have been set by the initial setting unit 111 of the information processing device 100 will be described in detail with reference to Fig. 2. Note that Fig. 2 is a diagram showing an example of a process up to output of a learning model in the embodiment.
[0031] The learning unit 112 of the information processing device 100 uses the learning model of the target domain to perform reinforcement learning corresponding to the target domain using, for example, states, actions, reward systems, etc., and updates the learning model of the target domain (step S20).
[0032] The learning unit 112 of the information processing device 100 determines whether the update of the learning model of the target domain performed in step S20 above is the first update (step S21).
[0033] If the learning model of the target domain is being updated for the first time (step S21: Yes), the learning unit 112 of the information processing device 100 initializes neurons (dead neurons) that are not activated at all during the initial learning in the learning model of the target domain (step S22).
[0034] If the learning model of the target domain is not updated for the first time (step S21: No), the learning unit 112 of the information processing device 100 does not perform step S22, but performs the process of step S23.
[0035] The learning unit 112 of the information processing device 100 determines whether the learning model of the target domain is learning on-policy (step S23).
[0036] If the learning model of the target domain is learning on-policy (step S23: Yes), the learning unit 112 of the information processing device 100 adds a vector (noise) using a predetermined function (e.g., a pseudo-random number generation function) to the parameters of the learning model of the target domain (step S24). Note that the learning unit 112 of the information processing device 100 may perform a process of gradually decreasing the variance of the predetermined function every time noise is added to the parameters of the learning model of the target domain.
[0037] If the learning model of the target domain is not learned on-policy (step S23: No), the learning unit 112 of the information processing device 100 does not perform step S24 and ends the process.
[0038] By performing steps S20 to S23 or steps S20 to S24, the learning unit 112 of the information processing device 100 can perform reinforcement learning of the target domain (hereinafter, this series of processes may be referred to as a learning step).
[0039] The learning step may be repeated a predetermined number of times (for example, until the reward function becomes asymptotic). If the agent in the learning model of the target domain is set to be fixed at On-Policy or Off-Policy, the determination in step S23 is performed only at the first update, and in the second and subsequent learning steps, the determination may be omitted and the same processing as in the first update in step S24 may or may not be performed.
[0040] In addition, the learning unit 112 of the information processing device 100 can output a learned model of the target domain as appropriate, for example, each time processing of a learning step is completed, or each time processing of a predetermined number of learning steps is completed.
[0041] As described above, through the processing performed by the initial setting unit 111 and the learning unit 112 of the information processing device 100, the information processing device 100 can generate a trained model of the target domain using a trained model of the source domain.
[0042] Note that in the process of generating a trained model for the target domain, the trained model for the source domain simply uses its parameters, and therefore does not affect the size of the trained model for the target domain. Therefore, using the trained model for the source domain does not increase the size of the trained model for the target domain. In other words, transfer learning does not increase the size of the trained model for deep reinforcement learning, and does not have any adverse effects on the learning time, etc. Therefore, the information processing device 100 can efficiently use transfer learning in deep reinforcement learning. Furthermore, for example, even when multiple trained models for the source domain are used or when a trained model trained in the target domain is used as the trained model for the source domain (transfer learning is repeated), the size of the trained model for the target domain similarly does not increase, and the information processing device 100 can efficiently use transfer learning in deep reinforcement learning.
[0043] The processes described above can also be realized by executing a prepared program on an information processing device such as a personal computer. An example of an information processing device that executes a learning program that achieves the same functions as the information processing device 100 shown in Fig. 1 will be described. Fig. 3 is a diagram showing an example of an information processing device 200 that executes a learning program.
[0044] 3, the information processing device 200 includes a CPU (Central Processing Unit) 210 that executes various arithmetic processes, an input device 220 that accepts data input, an output device 230 that outputs data, and a storage device 240. The information processing device 200 also includes a drive device 250 that reads programs and the like, a network I / F (Interface) 260 that communicates data via a network, and a memory 270 that temporarily stores data. The CPU 210, the input device 220, the output device 230, the storage device 240, the drive device 250, the network I / F 260, and the memory 270 are connected by a bus. Note that, when executing the learning program, some of these functions may be executed by another device and received by the information processing device 200 via the network.
[0045] The CPU 210 reads out the learning program, expands it in the memory 270, and executes the same process as the information processing device 100. The learning program may be stored in the storage device 240, or may be stored in an external storage device such as a USB memory or on the cloud, for example.
[0046] As a result, the information processing device 200 can execute the learning program and obtain the same output results as the information processing device 100.
[0047] As described above, the information processing device 100 can output a trained model of the target domain through the above processing. Two specific examples of reinforcement learning using the information processing device 100 will be described below. Note that these specific examples are merely examples and do not limit the applications of the information processing device 100.
[0048] The processes described in the following two specific examples are executed by a learning program on an information processing device.
[0049] (Example 1) FIG. 4 shows a schematic diagram of an environment to which the information processing device 100 in Specific Example 1 is applied (reinforcement learning task). Note that FIG. 4 is a diagram showing an example of a schematic diagram of the application environment in Specific Example 1. In Specific Example 1, the application environment includes N agents and L landmarks (N and L are natural numbers). Each agent moves so as to be close to one of the landmarks. Note that in Specific Example 1, the number of agents and the number of landmarks are the same.
[0050] Next, the settings used in reinforcement learning performed by the information processing device 100 will be described.
[0051] The action space of each agent can be set in five ways: up, down, left, right, or no movement. The amount of movement per movement is a predetermined fixed value. The state space (observation value) of each agent can be set to the agent's own position, the agent's own speed, the relative position of landmarks, and the positions of other agents.
[0052] In addition, two types of reward systems are set up for reinforcement learning. The first is a setting that gives a penalty equal to the sum of the distances between each landmark and the agent located closest to it. The second is a setting that gives a penalty of -1 when agents collide (appear to overlap).
[0053] Next, an agent model used in reinforcement learning performed by the information processing device 100 will be described.
[0054] The model used is DQN (Deep Q-network), in which each agent has a neural network model. The input to the Q-network is the value of the state space of each agent, and the output is the action space of each agent.
[0055] The results of reinforcement learning performed on the information processing device 100 using the above settings will be described with reference to Fig. 5. In this learning, an environment with 6 agents and 6 landmarks was used as the source domain. Also, an environment with 10 agents and 10 landmarks was used as the target domain.
[0056] Figure 5(A) shows a comparison of the results of learning in the information processing device 100 using transfer learning with those in another information processing device that performed reinforcement learning without using transfer learning, with the vertical axis representing reward and the horizontal axis representing episode.
[0057] FIG. 5(B) shows the results of reinforcement learning using the information processing device 100 using three indices. jp (jump start) indicates the percentage value at which reinforcement learning using the information processing device 100 starts learning, assuming that the reward range from the minimum to the maximum value during reinforcement learning without transfer learning is 100%. ap (asymptotic performance) indicates the percentage value to which reinforcement learning using the information processing device 100 ultimately progresses, assuming that the reward range from the minimum to the maximum value during reinforcement learning without transfer learning is 100%. ls (learning speed) indicates how many times faster the learning speed is in reinforcement learning with the information processing device 100 than in reinforcement learning without transfer learning, until the reward reaches a predetermined percentage of the maximum value (90%, 95%, 99%, and 100% in parentheses in FIG. 5(B)).
[0058] To summarize the above results, the learning of the information processing device 100 using transfer learning starts to improve rewards earlier than learning without transfer learning, and the learning speed improves by approximately 3.2 times (learning time reduced by approximately 68%).
[0059] As described above, the reinforcement learning by the information processing device 100 using transfer learning, which has been described in specific example 1, can improve the learning speed compared to reinforcement learning without transfer learning.
[0060] (Example 2) FIG. 6 shows a schematic diagram of an environment to which the information processing device 100 in Specific Example 2 is applied (reinforcement learning task). In Specific Example 2, the information processing device 100 is made to learn efficient control of base stations used in wireless communications, etc. Note that the base station may also be referred to as a communication device, etc. In explaining Specific Example 2, the learning performed by the information processing device 100 was performed using Milan, Italy, where traffic (communication volume) data is publicly available on the Internet (Website: https: / / opencellid.org / , https: / / www.nature.com / articles / sdata201555). Furthermore, the locations (approximately 705 m square) shown in FIG. 6(A) were used as the source domain and target domain, respectively, for the learning. Furthermore, FIG. 6(B) is a diagram showing the actual locations of base stations in the source domain and target domain, where the locations of the base stations in the source domain are indicated by circles and the locations of the base stations in the target domain are indicated by diamonds.
[0061] Next, the settings used in reinforcement learning performed by the information processing device 100 will be described.
[0062] The action space of each base station (agent) is set to two types: a state in which the base station communicates and a state in which it does not communicate. The state space (observation value) of each agent is set to the time, the traffic around the agent, and its own action at the previous time. The value for the base station's action is set to 1 for a state in which it communicates and 0 for a state in which it does not communicate.
[0063] In addition, the reward system given in reinforcement learning was set so that the lower the total power consumption of all agents in the domain, the higher the reward.
[0064] In addition, we set it so that if the load on each agent exceeds a predetermined value, it will be a constraint violation (cost will be incurred).
[0065] Next, an agent model used in reinforcement learning performed by the information processing device 100 will be described.
[0066] The model used is CPO (Constrained Policy Optimization), in which each agent has a neural network model.
[0067] In the model used, the policy function inputs three types of values: time, traffic around the agent, and its own action at the previous time, and outputs the agent's action probability.Furthermore, the reward value function and cost value function input three types of values: time, traffic around the agent, and its own action at the previous time, and output the reward value and cost value, respectively.
[0068] The results of reinforcement learning performed by the information processing device 100 using the above settings will be described with reference to Fig. 7. Note that the same learning model as above was used in learning in the source domain. Weighted average 1 is the result when the weighted average is calculated by the first method in the initial setting unit 111 of the information processing device 100, and weighted average 2 is the result when the weighted average is calculated by the second method in the initial setting unit 111 of the information processing device 100.
[0069] Figure 7(A) shows the results of learning in the information processing device 100 using transfer learning, with the vertical axis representing reward and the horizontal axis representing n_updates (number of model updates), and compares the results of learning in another information processing device that performed reinforcement learning without using transfer learning.
[0070] 7(B) shows three types of indices as the results of reinforcement learning using the information processing device 100. Each indices is the same as in the first specific example.
[0071] To summarize the above results, learning by the information processing device 100 using transfer learning begins to improve rewards earlier, improves learning speed, and shortens learning time compared to learning without transfer learning (with weighted average 1, learning speed improved by approximately 1.7 times and learning time decreased by approximately 41.2%, while with weighted average 2, learning speed increased by approximately 2.3 times and learning time decreased by approximately 56.8%). Furthermore, as the jp value indicates, it can be seen that high rewards can be obtained even in the early stages of learning.
[0072] From the above, it has been shown that the information processing device 100 that performs deep reinforcement learning using transfer learning in specific examples 1 and 2 can improve the learning speed and shorten the learning time compared to a case where conventional transfer learning is not used. In other words, the information processing device 100 can efficiently use transfer learning in deep reinforcement learning. [Explanation of symbols]
[0073] 100 Information processing device 110 control section 111 Initial setting section 112 Learning Department 120 Storage section 200 Information processing device 210 CPU 220 Input Device 230 Output Device 240 Storage device 250 Drive Unit 260 Network Interface 270 memory
Claims
1. In a learning program that performs learning using a learning model that utilizes a trained model executed by an information processing device, the information processing device Determine a first parameter by weighting the parameters of the trained model in a predetermined manner; setting a second parameter, which is a value of an initial parameter of the learning model, based on the first parameter; A learning program characterized by: causing a process to be executed.
2. the first parameter is a weighted average value using parameters of the agent of the trained model based on the weighting; The learning program of claim 1,
3. The weighting is performed based on a correlation between information about the agent in the trained model and information about the agent in the learning model; The learning program of claim 1,
4. the information about the agent of the trained model is information about a state space of the agent of the trained model during learning of the trained model, and the information about the agent of the learning model is information about a state space of the agent of the learning model before learning using a predetermined control in an environment in which the learning program is executed; 4. The learning program of claim 3.
5. the second parameter is a parameter obtained by adding a predetermined vector to the first parameter and setting a bias term of the learning model to 0; The learning program of claim 1,
6. The learning model initializes neurons that are not activated at all when the learning model is updated for the first time; The learning program of claim 1,
7. the learning model reduces the variance of the predetermined vector assigned to the parameters of the learning model every time the learning model is updated during learning; The learning program of claim 1,
8. A storage unit that stores a learning program that performs learning using a learning model that uses a trained model; a control unit that executes the learning program, When the control unit executes the learning program, the control unit Determine a first parameter by weighting the parameters of the trained model in a predetermined manner; setting a second parameter, which is a value of an initial parameter of the learning model, based on the first parameter; An information processing device characterized by:
9. A learning method for performing learning using a learning model that utilizes a trained model executed by an information processing device, Determine a first parameter by weighting the parameters of the trained model in a predetermined manner; setting a second parameter, which is a value of an initial parameter of the learning model, based on the first parameter; A learning method characterized by: