Optional learning method and device
By introducing a variational inference framework and a maximum entropy strategy, combined with low-cost option embedding representation, and optimizing the option learning algorithm, the problems of instability, insufficient exploration, and high computational cost in the optimization process of SMDP are solved, achieving stability and efficient learning in complex environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2026-03-10
Smart Images
Figure CN121638366A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to methods and apparatus for learning options. Background Technology
[0002] Over the past few years, Deep Reinforcement Learning (DRL) has achieved remarkable success in many complex domains, reaching human-level performance in games such as Atari games, Go, poker, and StarCraft II. These achievements demonstrate the potential of combining reinforcement learning with powerful function approximators (such as neural networks) to effectively solve complex tasks requiring fine-grained control. However, despite these breakthroughs, deep reinforcement learning still faces significant challenges, including insufficient exploration in dynamic environments, inefficient learning of time-dependent extended actions, and the enormous number of samples required to train effective behaviors.
[0003] To address these challenges, Hierarchical Reinforcement Learning (HRL) has been proposed, a set of strategies that decompose complex tasks into more manageable hierarchical structures. Among them, Semi-Markov Decision Processes (SMDPs) based on the Options Framework are particularly effective in segmenting non-stationary task phases into temporally extended actions (i.e., options). Options, as temporally abstracted actions, are executed at different time steps, with a master policy determining when each option begins and terminates. Options are typically learned using maximum likelihood methods, aiming to maximize the expected reward over the entire trajectory. This structure not only simplifies the management of complex environments but also enables the systematic discovery and execution of temporal abstractions in long-running tasks. Summary of the Invention
[0004] Embodiments of this disclosure present an option learning method and apparatus.
[0005] In a first aspect, embodiments of this disclosure provide an option learning method, comprising: inputting options, actions, and states into an evaluation network and a target network to obtain evaluation values and target values, respectively; calculating a loss value based on the evaluation value and the target value; adjusting the parameters of the evaluation network based on the loss value; adjusting the parameters of an option policy network based on the evaluation value; adjusting the parameters of an action policy network based on the evaluation value; and adjusting the parameters of the target network based on the parameters of the evaluation network.
[0006] In some embodiments, the method further includes: obtaining experience data by interacting with the environment through randomly selected options, actions, and states, and storing the experience data in an experience pool; uniformly sampling options, actions, and states from the experience pool as inputs to the evaluation network and the target network.
[0007] In some embodiments, calculating the loss value based on the evaluation value and the target value includes: calculating the loss value based on the evaluation value and the target value by minimizing the Bellman residual.
[0008] In some embodiments, the parameters of the option policy network are represented as an embedding vector matrix.
[0009] In some embodiments, adjusting the parameters of the option policy network based on the evaluation value includes: adding the entropy of the option to the evaluation value; and adjusting the parameters of the option policy network based on the evaluation value by maximizing the entropy.
[0010] In some embodiments, adjusting the parameters of the action policy network based on the evaluation value includes: adding the entropy of the options to the evaluation value; and adjusting the parameters of the action policy network based on the evaluation value by maximizing the entropy.
[0011] In some embodiments, the entropy of the option includes a temperature parameter; and the method further includes adjusting the temperature parameter using a gradient descent method.
[0012] In some embodiments, adjusting the parameters of the target network based on the parameters of the evaluation network includes: calculating a moving average of the parameters of the evaluation network as the parameters of the target network.
[0013] Secondly, embodiments of this disclosure provide a method for predicting options, comprising: obtaining a current state; inputting the current state into an option policy network trained according to the method described in any one of the first aspects, and outputting a predicted option; inputting the option and the current state into an action policy network trained according to the method described in any one of the first aspects, and outputting a predicted action.
[0014] Thirdly, embodiments of this disclosure provide an option learning apparatus, comprising: a value prediction unit configured to input options, actions, and states into an evaluation network and a target network, respectively obtaining an evaluation value and a target value; a loss calculation unit configured to calculate a loss value based on the evaluation value and the target value; an evaluation network adjustment unit configured to adjust the parameters of the evaluation network based on the loss value; an option parameter adjustment unit configured to adjust the parameters of an option policy network based on the evaluation value; an action parameter adjustment unit configured to adjust the parameters of an action policy network based on the evaluation value; and a target network adjustment unit configured to adjust the parameters of the target network based on the parameters of the evaluation network.
[0015] In some embodiments, the apparatus further includes an offline unit configured to: obtain experience data by interacting with the environment through randomly selected options, actions, and states, and store the experience data in an experience pool; and uniformly sample options, actions, and states from the experience pool as inputs to the evaluation network and the target network.
[0016] In some embodiments, the loss calculation unit is further configured to calculate a loss value based on the evaluation value and the target value by minimizing the Bellman residual.
[0017] In some embodiments, the parameters of the option policy network are represented as an embedding vector matrix.
[0018] In some embodiments, the option parameter adjustment unit is further configured to: incorporate the entropy of the option into the evaluation value; and adjust the parameters of the option policy network based on the evaluation value by maximizing the entropy.
[0019] In some embodiments, the action parameter adjustment unit is further configured to: add the entropy of the option to the evaluation value; and adjust the parameters of the action policy network based on the evaluation value by maximizing the entropy.
[0020] In some embodiments, the entropy includes a temperature parameter; and the device further includes a temperature parameter adjustment unit configured to adjust the temperature parameter using a gradient descent method.
[0021] In some embodiments, the target network adjustment unit is further configured to: calculate a moving average of the parameters of the evaluation network as parameters of the target network.
[0022] Fourthly, embodiments of this disclosure provide an apparatus for predicting options, comprising: an acquisition unit configured to acquire a current state; an option prediction unit configured to input the current state into an option policy network trained according to the method described in any one of the first aspects, and output a predicted option; and an action prediction unit configured to input the option and the current state into an action policy network trained according to the method described in any one of the first aspects, and output a predicted action.
[0023] Fifthly, embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more computer programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform the method as described in any one of the first or second aspects.
[0024] In a sixth aspect, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of the first or second aspects.
[0025] In a seventh aspect, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, implements the method as described in any one of the first or second aspects.
[0026] In existing technologies, semi-Markov decision processes (SMDPs) based on an option framework suffer from three main problems: instability of the optimization process, insufficient exploration and option degradation, and high computational cost. This invention aims to solve these problems, specifically including:
[0027] Instability in the optimization process: Due to the characteristics of the SMDP framework, existing methods are prone to non-stationary learning environments when simultaneously updating high-level master policies and low-level action policies, leading to instability in the optimization process. This application improves the policy update method by introducing a variational inference framework, ensuring stability in complex continuous domains.
[0028] Insufficient exploration and option degradation: Existing technologies often encounter the problem of option degradation during the exploration process, where a certain option dominates for a long period during learning, thus limiting the diversity of strategies and making the system prone to getting trapped in local optima. This application enhances the option exploration capability by introducing a maximum entropy strategy, encourages the generation of diverse strategies, and avoids the problem of option degradation.
[0029] High computational cost: Traditional option definition methods typically require computationally expensive neural network models, increasing computational costs. This application proposes a low-cost option embedding representation method that represents options as learnable embedding matrices, greatly simplifying the training process while enhancing the model's expressive power.
[0030] Through the above improvements, this application not only improves sample efficiency and computational efficiency, but also ensures stability and robustness in complex environments, providing a more effective and scalable solution for practical applications.
[0031] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0032] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0033] Figure 1 This is an exemplary system architecture diagram to which one embodiment of this disclosure can be applied;
[0034] Figure 2 This is a flowchart of one embodiment of the option learning method according to the present disclosure;
[0035] Figure 3 This is a schematic diagram of an application scenario of the option learning method according to this disclosure;
[0036] Figure 4 This is a flowchart of one embodiment of the method for predicting options according to this disclosure;
[0037] Figure 5 This is a schematic diagram of the structure of an embodiment of the option learning device according to the present disclosure;
[0038] Figure 6 This is a schematic diagram of the structure of an embodiment of a device for predicting options according to the present disclosure;
[0039] Figure 7 This is a schematic diagram of the structure of a computer system suitable for implementing embodiments of the present disclosure. Detailed Implementation
[0040] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.
[0041] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.
[0042] Figure 1 An exemplary system architecture 100 is shown that can be applied to the option learning method, option learning apparatus, option prediction method, or option prediction apparatus according to embodiments of the present disclosure.
[0043] like Figure 1 As shown, system architecture 100 may include terminals 101 and 102, network 103, database server 104, and server 105. Network 103 serves as the medium for providing communication links between terminals 101 and 102, database server 104, and server 105. Network 103 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.
[0044] User 110 can use terminals 101 and 102 to interact with server 105 via network 103 to receive or send messages, etc. Various client applications can be installed on terminals 101 and 102, such as model training applications, game applications, shopping applications, payment applications, web browsers, and instant messaging tools.
[0045] The terminals 101 and 102 here can be either hardware or software. When terminals 101 and 102 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), laptops, and desktop computers. When terminals 101 and 102 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module. No specific limitations are set here.
[0046] Database server 104 can be a database server that provides various services. For example, the database server can store a sample set. The sample set contains a large number of samples. The method of this application can be applied to the field of games; therefore, the samples can include experiential data such as game images, user-selected actions, states, rewards, and states at the next moment. In this way, user 110 can also select samples from the sample set stored in database server 104 through terminals 101 and 102.
[0047] Server 105 can also be a server providing various services, such as a backend server supporting various applications displayed on terminals 101 and 102. The backend server can use samples from the sample set sent by database server 104 to train the initial model and can send the training results (such as the generated option policy network and action policy network) to terminals 101 and 102. In this way, users can apply the generated option policy network and action policy network to predict game strategies, for example, planning which piece to move to which position (i.e., option) for one side based on the current situation (i.e., state) in Go.
[0048] The database servers 104 and 105 here can be either hardware or software. When they are hardware, they can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When they are software, they can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. No specific limitations are made here. Database servers 104 and 105 can also be servers in a distributed system, or servers integrated with blockchain. Database servers 104 and 105 can also be cloud servers, or intelligent cloud computing servers or intelligent cloud hosts with artificial intelligence technology.
[0049] It should be noted that the option learning method or option prediction method provided in this embodiment is generally executed by server 105. Accordingly, the option learning device or option prediction device is also generally provided in server 105.
[0050] It should be noted that if server 105 can perform the relevant functions of database server 104, database server 104 may not be set up in system architecture 100.
[0051] It should be understood that Figure 1 The number of terminals, networks, database servers, and servers shown is merely illustrative. Depending on implementation needs, any number of terminals, networks, database servers, and servers can be included.
[0052] The core of this application lies in introducing a variational inference framework, combining a maximum entropy strategy with low-cost option embedding representations, to implement a novel option learning algorithm—the Variational Markovian Option Critic (VMOC). This algorithm can effectively learn and optimize option policies in complex continuous domains, improving exploration efficiency and reducing computational overhead.
[0053] By transforming the option learning problem into a probabilistic inference problem, this application utilizes a variational inference framework for optimization. Specifically, a variational distribution q(τ) is defined, and the true posterior distribution of the option trajectory is approximated by maximizing the Evidence Lower Bound (ELBO). The trajectory of the options, s0-s T Representing the state, a0-a T Let ε represent the action, and t = 0 - T represent time. ε ∈ {0, 1} is an observable binary optimal random variable. The probability of optimal reward for a given state-action pair (s, a) is expressed as P(ε = 1 | s, a) = exp(r(s, a)), which is an exponential function of the traditional reward function r(s, a), where r(s, a) measures the expectation of the action in a given state. ε = 1 represents the event that the optimal reward occurs. and The reward is optimal at time t. For simplicity, ε will be used instead of ε = 1 in the following text.
[0054] The joint probability distribution on trajectory τ when the reward is optimal is expressed as:
[0055]
[0056] in, Let P(s0, o0) represent the reward-optimal events for action (A) and option (O) at time 1-T, respectively. Let P(s0, o0) represent the initial state and the distribution of the options, respectively. t+1 |s t a t ) is a dynamic model. The states and actions are represented by s. t a t The probability of receiving the optimal reward at that time. The states and actions are represented by s. t a t Current option o t The previous option was O. t-1 The probability of receiving the optimal reward at that time, P(o t )P(a t Let t represent the distribution of options and the distribution of actions at time t, respectively, which can be assumed to be uniform distributions.
[0057] P(s0) represents the initial state distribution. O0-O t This represents the options from time 0 to time t. The final expression is the product of environmental dynamics and maximum likelihood. Directly optimizing Equation (1) would lead to a risky strategy. Therefore, this application approximates it by maximizing the lower bound of evidence (ELBO).
[0058]
[0059] Wherein, the action policy q(a) t |s t ) and option strategy q(O) t |s t O t-1 ) Strategies for approximation and The ELBO of the log-likelihood optimal trajectory in formula (1) can be expressed as follows:
[0060]
[0061] The second line substitutes formulas (1) and (2) into ELBO. f(·) is an arbitrary non-positive function used to measure the preference for choosing an option given a state-action sequence and previously executed options. π A ,π O These represent action strategies and option strategies, respectively. Entropy represents the action. This represents the entropy of the options. Therefore, optimizing ELBO not only yields high-reward options but also maximizes cross-entropy, thereby increasing strategy diversity and exploration capabilities.
[0062] The above demonstrates the flow of the variational inference framework. By optimizing the variational distribution q(τ) of the trajectory τ, an approximate posterior distribution can be obtained to guide the learning of the option policy.
[0063] Continue to refer to Figure 2 The diagram illustrates a flow 200 of an embodiment of an option learning method according to the present disclosure. The option learning method includes the following steps:
[0064] Step 201: Input the options, actions, and states into the evaluation network and the target network to obtain the evaluation value and the target value, respectively.
[0065] In this embodiment, the execution subject of the option learning method (e.g.) Figure 1 The server shown can first initialize the Q-functions of the evaluation network and the target network, as well as the parameters of the option policy network and the action policy network. The Q-function, also known as the action-value function, represents the expected reward of taking a certain action in a given state.
[0066] Options, actions, and states are obtained from the experience pool and input into the evaluation network and the target network. The evaluation network and the target network are neural networks with the same structure (which may include convolutional networks and fully connected networks), and their outputs are both values.
[0067] The algorithm in this application establishes a neural network similar to DQN (Deep Q-network), taking the state, options, and actions as inputs and the corresponding Q-values as outputs. Through training the neural network, the Q-values for each state, option, and action are obtained, eliminating the need to construct a Q-table. After training, the resulting network contains the true values of all states, options, and actions. When the computer makes decisions, the pre-trained neural network automatically outputs the values of each state, option, and action.
[0068] The evaluation network in this application is equivalent to the Q network in DQN, and is the training objective, aiming to learn to generate optimal state-option-action values.
[0069] The target network predicts the "target Q value," which is the optimal Q value among all options and actions that can be taken from the next state of each sample.
[0070] Step 202: Calculate the loss value based on the assessed value and the target value.
[0071] In this embodiment, the loss value can be calculated based on the difference between the evaluated value and the target value. The loss function can be the mean squared error or other loss functions.
[0072] In some optional implementations of this embodiment, calculating the loss value based on the evaluation value and the target value includes: calculating the loss value based on the evaluation value and the target value by minimizing the Bellman residual. This allows the calculated loss value to converge quickly, improving training speed.
[0073] Step 203: Adjust the parameters of the evaluation network based on the loss value.
[0074] In this embodiment, the gradient of the loss value with respect to the parameters of the evaluation network is calculated, and the parameters of the evaluation network are adjusted according to the gradient descent method.
[0075] Step 204: Adjust the parameters of the option strategy network based on the evaluation values.
[0076] In this embodiment, the present application uses two policy networks: an option policy network and an action policy network. These two policy networks are neural networks with the same structure, which may include: convolutional networks, fully connected networks, and softmax activation functions.
[0077] The gradient of the evaluation value with respect to the parameters of the option policy network can be calculated, and the parameters of the option policy network can be adjusted according to the gradient ascent method.
[0078] In some optional implementations of this embodiment, the option policy network is represented as an embedding vector matrix. Traditional option representation methods require complex neural network models, resulting in high computational overhead. This application proposes a low-cost option embedding representation method that represents options as a learnable embedding matrix. This representation method not only simplifies the training process but also enhances the expressive power of the model. Figure 3 The structure of the option embedding representation is shown. By learning the embedding matrix, options can be efficiently represented as low-dimensional embedding vectors, reducing computational costs. Figure 3 The x-axis represents the index of the embedded dimension, and the y-axis represents the option.
[0079] In some optional implementations of this embodiment, the entropy of the option is added to the evaluation value; the parameters of the option policy network are adjusted based on the evaluation value by maximizing the entropy. The evaluation value plus the entropy of the option can be used as the objective function of the option policy network for gradient calculation. The parameters of the option policy network are adjusted to maximize the entropy of the option.
[0080] In some optional implementations of this embodiment, the entropy of the option includes a temperature parameter; and the method further includes adjusting the temperature parameter using gradient descent. The temperature parameter is a weight of the entropy. A first target entropy related to the entropy of the option can be set. By calculating the gradient of the first target entropy with respect to the temperature parameter, the temperature parameters of the option's entropy and the action's entropy are adjusted using gradient descent to maximize the first target entropy. Conventional methods use fixed temperature parameters, while this application introduces an adaptive adjustment mechanism for the temperature parameter to ensure stability and performance at different learning stages. The temperature parameter determines the entropy weight of the strategy. By adaptively adjusting the temperature parameter, the algorithm can achieve a balance between exploration and exploitation, ensuring appropriate exploration intensity and stable performance at different learning stages.
[0081] Step 205: Adjust the parameters of the action policy network based on the evaluation values.
[0082] In this embodiment, similar to step 204, the parameters of the action policy network can be adjusted according to the gradient ascent method by calculating the gradient of the evaluation value with respect to the parameters of the action policy network.
[0083] In some optional implementations of this embodiment, adjusting the parameters of the action policy network based on the evaluation value includes: adding the entropy of the action to the evaluation value; and adjusting the parameters of the action policy network based on the evaluation value by maximizing the entropy. The evaluation value plus the entropy of the action can be used as the objective function of the action policy network for gradient calculation. The parameters of the action policy network are adjusted to maximize the entropy.
[0084] In some optional implementations of this embodiment, the entropy of the action includes a temperature parameter; and the method further includes adjusting the temperature parameter using gradient descent. The temperature parameter is a weight of the entropy. A second target entropy related to the entropy of the action can be set. By calculating the gradient of the second target entropy with respect to the temperature parameter, the temperature parameters of the option entropy and the action entropy are adjusted using gradient descent to maximize the second target entropy. Conventional methods use fixed temperature parameters, while this application introduces an adaptive adjustment mechanism for the temperature parameter to ensure stability and performance at different learning stages. The temperature parameter determines the entropy weight of the strategy. By adaptively adjusting the temperature parameter, the algorithm can achieve a balance between exploration and exploitation, ensuring appropriate exploration intensity and stable performance at different learning stages.
[0085] Step 206: Adjust the parameters of the target network based on the parameters of the evaluation network.
[0086] In this embodiment, after a predetermined number of training iterations, the parameters of the evaluation network can be directly assigned to the parameters of the target network.
[0087] In some optional implementations of this embodiment, adjusting the parameters of the target network based on the parameters of the evaluation network includes: calculating a moving average of the parameters of the evaluation network as the parameters of the target network. This allows for timely updates to the parameters of the target network, accelerating the convergence speed of the evaluation network.
[0088] In some optional implementations of this embodiment, the method further includes: obtaining experience data through interaction with the environment using randomly selected options, actions, and states, and storing the experience data in an experience pool; uniformly sampling options, actions, and states from the experience pool as inputs to the evaluation network and the target network. Through offline learning methods, the algorithm can utilize experience replay buffer technology to fully leverage historical samples, reduce sample waste, and improve learning efficiency. Combined with a maximum entropy strategy, the algorithm can maintain high entropy while achieving high rewards, thus promoting exploration.
[0089] The method provided in the above embodiments of this disclosure transforms the option learning problem into a probabilistic reasoning problem, approximating the true posterior distribution by maximizing the lower bound of evidence (ELBO). Employing a maximum entropy strategy enhances the option exploration capability, encouraging the strategy to select options with high entropy, thereby improving strategy diversity and exploration ability.
[0090] Continue to refer to Figure 4 The diagram illustrates a flow 400 of an embodiment of a method for predicting options according to the present disclosure. The method for predicting options includes the following steps:
[0091] Step 401: Obtain the current state.
[0092] In this embodiment, during the game, the system records the current state of the target object (e.g., a chess piece in a board game), such as the position of each player's pieces and the score of each player. Alternatively, the current state of the target object can be determined by capturing a screenshot of the game screen and performing image recognition.
[0093] Step 402: Input the current state into the option policy network and output the predicted option.
[0094] In this embodiment, the option policy network can be a policy network trained according to the method described in process 200. It is used to determine the probability of each option when the current state outputs the maximum value, and then select the option with the highest probability as the predicted option. For example, if the currently movable pieces on the chessboard are options, and one side assumes there are 3 movable pieces, then there are 3 options. The option policy network determines the option with the highest probability from these 3 options as the target piece for subsequent action prediction.
[0095] Step 403: Input the options and current state into the action policy network, and output the predicted action.
[0096] In this embodiment, the action policy network can be a policy network trained according to the method described in process 200. It is used to output the probability of each action with the highest value based on the current state and options, and selects the action with the highest probability as the predicted action. For example, the action can be the specific movement direction of a piece, and the value is the game score. The target piece has four possible actions: "move up one square," "move down one square," "move left one square," and "move right one square." The action policy network predicts the probability of each action with the highest value from these four actions, and selects the action with the highest probability as the predicted action. If the predicted scores for "move up one square" and "move down one square" are 0, and "move left one square" results in the opponent capturing the target piece and losing points (i.e., a negative score), only "move right one square" scores points. Therefore, the action with the highest value is "move right one square." The method provided by the above embodiments of this disclosure can select the best strategy for the game and obtain the highest reward.
[0097] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of an option learning device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.
[0098] like Figure 5As shown, the option learning device 500 of this embodiment includes: a value prediction unit 501, a loss calculation unit 502, an evaluation network adjustment unit 503, an option parameter adjustment unit 504, an action parameter adjustment unit 505, and a target network adjustment unit 506. The value prediction unit 501 is configured to input options, actions, and states into an evaluation network and a target network, respectively, to obtain an evaluation value and a target value; the loss calculation unit 502 is configured to calculate a loss value based on the evaluation value and the target value; the evaluation network adjustment unit 503 is configured to adjust the parameters of the evaluation network based on the loss value; the option parameter adjustment unit 504 is configured to adjust the parameters of the option strategy network based on the evaluation value; the action parameter adjustment unit 505 is configured to adjust the parameters of the action strategy network based on the evaluation value; and the target network adjustment unit 506 is configured to adjust the parameters of the target network based on the parameters of the evaluation network.
[0099] In this embodiment, the specific processing of the value prediction unit 501, loss calculation unit 502, evaluation network adjustment unit 503, option parameter adjustment unit 504, action parameter adjustment unit 505, and target network adjustment unit 506 of the option learning device 500 can be referred to Figure 2 Steps 201-206 in the corresponding embodiment.
[0100] In some optional implementations of this embodiment, the device further includes an offline unit (not shown in the figures), configured to: obtain experience data by interacting with the environment through randomly selected options, actions, and states, and store the experience data in an experience pool; uniformly sample options, actions, and states from the experience pool as inputs to the evaluation network and the target network.
[0101] In some optional implementations of this embodiment, the loss calculation unit 502 is further configured to calculate the loss value based on the evaluation value and the target value by minimizing the Bellman residual.
[0102] In some optional implementations of this embodiment, the parameters of the option policy network are represented as an embedding vector matrix.
[0103] In some optional implementations of this embodiment, the option parameter adjustment unit 504 is further configured to: add the entropy of the option to the evaluation value; and adjust the parameters of the option policy network based on the evaluation value by maximizing the entropy.
[0104] In some optional implementations of this embodiment, the action parameter adjustment unit 505 is further configured to: add the entropy of the option to the evaluation value; and adjust the parameters of the action policy network based on the evaluation value by maximizing the entropy.
[0105] In some optional implementations of this embodiment, the entropy of the option includes a temperature parameter; and the device further includes a temperature parameter adjustment unit (not shown in the figures) configured to adjust the temperature parameter by gradient descent.
[0106] In some optional implementations of this embodiment, the target network adjustment unit 506 is further configured to: calculate the moving average of the parameters of the evaluation network as the parameters of the target network.
[0107] Further reference Figure 6 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a device for predicting options, which is similar to... Figure 4 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.
[0108] like Figure 6 As shown, the device 600 for predicting options in this embodiment includes: an acquisition unit 601, an option prediction unit 602, and an action prediction unit 603. The acquisition unit 601 is configured to acquire the current state; the option prediction unit 602 is configured to input the current state into an option policy network trained by the device 500 and output a predicted option; the action prediction unit 603 is configured to input the option and the current state into an action policy network trained by the device 500 and output a predicted action.
[0109] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.
[0110] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.
[0111] An electronic device includes: one or more processors; and a storage device having one or more computer programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to implement the method described in processes 200 and 400.
[0112] A computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the methods described in processes 200 and 400.
[0113] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0114] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.
[0115] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0116] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as road planning methods. For example, in some embodiments, the road planning method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the road planning method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform road planning methods by any other suitable means (e.g., by means of firmware).
[0117] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0118] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0119] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0120] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0121] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0122] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be servers in distributed systems or servers incorporating blockchain technology. Servers can also be cloud servers, or intelligent cloud computing servers or intelligent cloud hosts with artificial intelligence technology.
[0123] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0124] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. An option learning method, comprising: inputting an option, an action and a state into an evaluation network and a target network to obtain an evaluation value and a target value, respectively; calculating a loss value according to the evaluation value and the target value; adjusting a parameter of the evaluation network based on the loss value; adjusting a parameter of an option policy network based on the evaluation value; adjusting a parameter of an action policy network based on the evaluation value; adjusting a parameter of the target network based on the parameter of the evaluation network.
2. The method of claim 1, wherein, The method further comprises: obtaining experience data by interacting with an environment through randomly selected options, actions and states, and storing the experience data into an experience pool; uniformly sampling options, actions and states from the experience pool as inputs of the evaluation network and the target network.
3. The method of claim 1, wherein, The calculating a loss value according to the evaluation value and the target value comprises: calculating a loss value according to the evaluation value and the target value by minimizing Bellman residual.
4. The method of claim 1, wherein, The option policy network is represented in the form of an embedding vector matrix.
5. The method of claim 1, wherein, The adjusting a parameter of an option policy network based on the evaluation value comprises: adding an entropy of the option in the evaluation value; adjusting a parameter of an option policy network based on the evaluation value by maximizing the entropy.
6. The method of claim 1, wherein, The adjusting a parameter of an action policy network based on the evaluation value comprises: adding an entropy of the action in the evaluation value; adjusting a parameter of an action policy network based on the evaluation value by maximizing the entropy.
7. The method of claim 5 or 6, wherein, The entropy comprises a temperature parameter; and the method further comprises: adjusting the temperature parameter by gradient descent method.
8. The method of claim 1, wherein, The adjusting a parameter of the target network based on the parameter of the evaluation network comprises: calculating a moving average of the parameter of the evaluation network as the parameter of the target network.
9. A method for predicting an option, comprising: obtaining a current state; inputting the current state into an option policy network trained according to the method of any one of claims 1-8 to output a predicted option; inputting the option and the current state into an action policy network trained according to the method of any one of claims 1-8 to output a predicted action.
10. An option learning device, comprising: a value prediction unit configured to input an option, an action and a state into an evaluation network and a target network to obtain an evaluation value and a target value, respectively; a loss calculation unit configured to calculate a loss value according to the evaluation value and the target value; an evaluation network adjustment unit configured to adjust a parameter of the evaluation network based on the loss value; an option parameter adjustment unit configured to adjust a parameter of an option policy network based on the evaluation value; an action parameter adjustment unit configured to adjust a parameter of an action policy network based on the evaluation value; a target network adjustment unit configured to adjust a parameter of the target network based on the parameter of the evaluation network.
11. A device for predicting an option, comprising: an obtaining unit configured to obtain a current state; an option prediction unit configured to input the current state into an option policy network trained according to the method of any one of claims 1-8 to output a predicted option; An action prediction unit configured to input the options and the current state into an action policy network trained according to the method of any of claims 1-8, outputting a predicted action.
12. An electronic device, comprising: one or more processors; storage having stored thereon one or more computer programs, when the one or more computer programs are executed by the one or more processors, cause the one or more processors to carry out the method of any of claims 1-9.
13. A computer readable medium having stored thereon a computer program, wherein, The computer program, which when executed by a processor, carries out the method of any of claims 1-9.
14. A computer program product comprising a computer program which, when executed by a processor, carries out the method of any of claims 1-9.