Strategy exploration model training method and device, computer equipment and storage medium

Abstract

The embodiment of the invention discloses a strategy exploration model training method and device, computer equipment and a storage medium. Wherein human preference rewards and human non-preference rewards are determined according to the annotation data set, and a reward model is constructed according to the human preference rewards and the human non-preference rewards; determining a preset state of the target non-annotated data and a corresponding first exploration strategy in the non-annotated data set; obtaining an action state data set and an execution result output by the intelligent agent according to the first exploration strategy and the preset state and environment interaction, inputting the execution result into a reward model to output a reward value, and setting the reward value in the action state data set; determining an action value and a label value corresponding to each action state group in the action state data set; and determining an evaluation value corresponding to each action state group according to the action value and the label value, updating the first exploration strategy according to a preset exploration strategy and the evaluation value, and realizing iterative training of the strategy exploration model until training is completed.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a strategy exploration model training method, device, computer equipment and storage medium. Background Art

[0002] With the development of artificial intelligence technology, reinforcement learning from human feedback (RLHF) has become a new way to train artificial intelligence models. That is, the artificial intelligence model can be further trained based on human feedback data, so that the artificial intelligence model can learn human preferences.

[0003] In related technologies, it is necessary to collect annotated data sets, which are data sets annotated by humans, and then use the annotated data sets to train the reward model. It is also necessary to use exploration strategies to obtain unlabeled data to continue to strengthen the reward model. Finally, the trained reward model provides reward signals to help strengthen the learning of the artificial intelligence model.

[0004] However, the related technologies are limited by the exploration strategy and cannot effectively explore complex state spaces or respond to changes in the environment, and thus cannot generate new data to continue to optimize the reward model, which ultimately results in the reward model being unable to generate reward signals based on new data to reinforce the learning of the artificial intelligence model. This results in the artificial intelligence model being unable to adapt to the new environment. Summary of the invention

[0005] The embodiments of the present application provide a strategy exploration model training method, apparatus, computer equipment and storage medium, which can enable the trained strategy exploration model to generate new exploration strategies based on the original exploration strategies, thereby improving the richness of the exploration strategies.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, an embodiment of the present application provides a strategy exploration model training method, including:

[0007] Obtain a labeled data set and an unlabeled data set, determine a human preference reward and a human non-preference reward according to the labeled data set, and construct a reward model according to the human preference reward and the human non-preference reward;

[0008] Determining a preset state of target unlabeled data in the unlabeled data set, and determining a first exploration strategy corresponding to the preset state;

[0009] Obtaining an action state data set and an execution result output by the agent in interaction with the environment according to the first exploration strategy and the preset state, and inputting the execution result into the reward model to output a reward value, and setting the reward value in the action state data set;

[0010] Determine the action value corresponding to each action state group in the action state data set, and determine the label value corresponding to each action state group;

[0011] An evaluation value corresponding to each action state group is determined according to the action value and the label value, and the first exploration strategy is updated according to the preset exploration strategy and the evaluation value to obtain an updated first exploration strategy, so as to realize iterative training of the strategy exploration model, and repeatedly determine the preset state of the target unlabeled data in the unlabeled data set until the number of iterations of the strategy exploration model meets the preset number of iterations. Then, the training of the strategy exploration model is completed, and a trained strategy exploration model is obtained.

[0012] In order to achieve the above-mentioned purpose, according to one aspect of the present application, an embodiment of the present application provides a strategy exploration model training device, including:

[0013] A first acquisition module is used to acquire a labeled data set and an unlabeled data set, determine a human preference reward and a human non-preference reward according to the labeled data set, and construct a reward model according to the human preference reward and the human non-preference reward;

[0014] A first determination module, configured to determine a preset state of target unlabeled data in the unlabeled data set, and determine a first exploration strategy corresponding to the preset state;

[0015] A second acquisition module is used to obtain an action state data set and an execution result output by the agent according to the first exploration strategy and the preset state and the interaction with the environment, and input the execution result into the reward model to output a reward value, and set the reward value in the action state data set;

[0016] A second determination module is used to determine the action value corresponding to each action state group in the action state data set, and determine the label value corresponding to each action state group;

[0017] An iteration module is used to determine the evaluation value corresponding to each action state group according to the action value and the label value, and to update the first exploration strategy according to the preset exploration strategy and the evaluation value to obtain the updated first exploration strategy, so as to realize iterative training of the strategy exploration model, and repeatedly determine the preset state of the target non-labeled data in the non-labeled data set until the number of iterations of the strategy exploration model meets the preset number of iterations, then the strategy exploration model training is completed, and the trained strategy exploration model is obtained.

[0018] In some implementations, the first acquisition module is used to:

[0019] Determine first labeled data preferred by humans according to the labeled data set, and determine a total reward value corresponding to each first labeled data;

[0020] Determine a human preference reward according to the total reward value corresponding to each of the first labeled data;

[0021] Determine second labeled data that is not preferred by humans according to the labeled data set, and determine a total reward value corresponding to each second labeled data;

[0022] A human non-preferred reward is determined according to the total reward value corresponding to each of the second labeled data.

[0023] In some implementations, the first acquisition module is used to:

[0024] Construct a human preference model based on human preference rewards and human non-preference rewards;

[0025] A reward model is constructed based on the human preference model.

[0026] In some embodiments, the second acquisition module is used to:

[0027] Obtaining an execution action determined by the agent according to the first exploration strategy in the preset state;

[0028] Obtain an action state data set and an execution result generated by the agent in the preset state according to the interaction between the execution action and the environment.

[0029] In some implementations, the second determining module is configured to:

[0030] Determine the action, state and reward value corresponding to each action state group in the action state data set;

[0031] The advantage value and the state value are determined according to the action and the state, and the action value corresponding to each action state group is determined according to the reward value, the advantage value and the state value.

[0032] In some embodiments, an iteration module is used to:

[0033] Determine the square value of the difference between the action value and the label value corresponding to each action state group;

[0034] Accumulate the square values ​​corresponding to each action state group to obtain the sum of the square values;

[0035] An evaluation value corresponding to each action state group is determined according to the sum of the square values ​​and the total number of the action state groups.

[0036] In some embodiments, an iteration module is used to:

[0037] Determine the action probability distribution corresponding to each action state group;

[0038] Determine a first KL divergence between the action probability distribution and the action probability distribution corresponding to a preset exploration strategy;

[0039] Determining a second KL divergence between the action probability distribution and the action probability distribution corresponding to the first exploration strategy;

[0040] The first exploration strategy is updated according to the first KL divergence, the second KL divergence, and the evaluation value to obtain an updated first exploration strategy.

[0041] In some embodiments, an iteration module is used to:

[0042] Determine a preset regularization parameter corresponding to the first KL divergence, and multiply the regularization parameter by the first KL divergence to obtain a first result;

[0043] Determine a preset learning rate parameter corresponding to the second KL divergence, and divide the second KL divergence by the preset learning rate parameter to obtain a second result;

[0044] The first exploration strategy is updated according to the first result, the second result and the evaluation value to obtain an updated first exploration strategy.

[0045] In some embodiments, the strategy exploration model further includes an update module, which is used to:

[0046] Generate a target first result according to the preset exploration strategy, the preset learning rate parameter and the preset regularization parameter;

[0047] Generate a target second result according to the current exploration strategy, the preset learning rate parameter and the preset regularization parameter;

[0048] Determine a target evaluation value corresponding to the state-action group input into the trained strategy exploration model, and generate a target third result according to the target evaluation value, the preset learning rate parameter and the preset regularization parameter;

[0049] The first result, the second result and the third result are multiplied together to determine a target exploration strategy after updating the current exploration strategy.

[0050] In some embodiments, the strategy exploration model further includes an input module, which is used, after multiplying the first result, the second result, and the third result to determine the target exploration strategy after updating the current exploration strategy:

[0051] Updating the agent's exploration strategy to the target exploration strategy;

[0052] The agent is guided to interact with the environment according to the target exploration strategy to perform corresponding target actions.

[0053] In order to achieve the above-mentioned purpose, according to one aspect of the present application, an embodiment of the present application provides a computer-readable storage medium, which stores multiple instructions, and the instructions are suitable for a processor to load to execute the strategy exploration model training method provided by the embodiment of the present application.

[0054] In order to achieve the above-mentioned purpose, according to one aspect of the present application, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the strategy exploration model training method provided in the embodiment of the present application is implemented when the processor executes the computer program.

[0055] In an embodiment of the present application, a labeled data set and an unlabeled data set are obtained, human preference rewards and human non-preference rewards are determined according to the labeled data set, and a reward model is constructed according to the human preference rewards and human non-preference rewards; a preset state of the target non-labeled data is determined in the non-labeled data set, and a first exploration strategy corresponding to the preset state is determined; an action state data set and an execution result output by the intelligent agent in interaction with the environment according to the first exploration strategy and the preset state are obtained, and the execution result is input into the reward model to output a reward value, and the reward value is set in the action state data set; the action value corresponding to each action state group in the action state data set is determined, and the label value corresponding to each action state group is determined; the evaluation value corresponding to each action state group is determined according to the action value and the label value, and the first exploration strategy is updated according to the preset exploration strategy and the evaluation value to obtain the updated first exploration strategy, so as to realize the iterative training of the strategy exploration model, and the preset state of the target non-labeled data is determined in the non-labeled data set repeatedly until the number of iterations of the strategy exploration model meets the preset number of iterations, then the strategy exploration model training is completed, and the trained strategy exploration model is obtained. In this way, through the trained strategy exploration model, the exploration strategy can be optimized and enriched, thereby solving the problem of single exploration strategy in related technologies. In this application, the reinforcement learning of the intelligent agent is realized through more exploration strategies, which can improve the efficiency of the reinforcement learning of the intelligent agent.

[0056] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 It is a schematic diagram of the system framework corresponding to the strategy exploration model training method provided in the embodiment of the present application;

[0059] Figure 2 It is a scenario diagram of the strategy exploration model training method provided in an embodiment of the present application;

[0060] Figure 3 It is a flowchart of the strategy exploration model training method provided in the embodiment of the present application;

[0061] Figure 4is another flow chart of the strategy exploration model training method provided in an embodiment of the present application;

[0062] Figure 5 It is a structural diagram of a strategy exploration model training device provided in an embodiment of the present application;

[0063] Figure 6 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0065] It is understandable that in the specific implementation of the present application, related data such as training data corresponding to the file is involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards.

[0066] It should be noted that in some processes described in the specification, claims and the above-mentioned drawings, multiple steps appearing in a specific order are included, but it should be clearly understood that these steps may not be executed in the order in which they appear in this document or may be executed in parallel. The step numbers are only used to distinguish different steps, and the numbers themselves do not represent any execution order. In addition, descriptions such as "first", "second" or "target" in this document are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0067] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0068] The strategy exploration model training method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The strategy exploration model training method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. Cloud servers; software can be applications that implement strategy exploration model training methods, etc., but are not limited to the above forms.

[0069] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0070] Reinforcement Learning from Human Feedback (RLHF) is a machine learning technique that uses direct human feedback to train a "reward model" and then uses this model to optimize the performance of an AI agent through reinforcement learning. This approach is particularly suitable for tasks with complex, ill-defined, or difficult to accurately express goals, such as making large language models (LLMs) more reliable, accurate, and ethical. The main steps of human feedback reinforcement learning are as follows:

[0071] Generate preliminary results: AI generates a solution or answer based on current knowledge.

[0072] Get human feedback: Human users evaluate the AI’s responses, which can be as simple as “good” or “bad” or as more complex as ratings or recommendations.

[0073] Reinforcement learning: AI adjusts its internal model based on human feedback so that the results it generates next time are more in line with human expectations.

[0074] Iterative process: This process will be repeated over and over again, and the AI ​​will gradually improve its performance through trial and error and adjustment.

[0075] Deep-Q-Network (DQN) is a reinforcement learning algorithm that uses deep neural networks to estimate the Q-value function, so that it can handle problems with high-dimensional state space. DQN avoids the problem of Q-table storage and updating in traditional Q learning by using deep neural networks to approximate the Q function. It obtains an accurate Q function by training deep neural networks to minimize the gap between predicted Q values ​​and true Q values.

[0076] Policy subnetwork: The policy subnetwork is the learner of the policy, and its main task is to select actions to maximize the long-term expected return. The policy subnetwork usually uses a parameterized policy function, such as a neural network, which takes the state of the environment as input and outputs an action or action probability distribution. The goal of the policy subnetwork is to learn a good policy to select actions in different states to maximize the total reward.

[0077] Value subnetwork: The value subnetwork is a learner of the value function, and its task is to estimate the expected cumulative reward of taking an action in a given state. The value subnetwork usually uses a parameterized value function, such as a neural network, which takes the state and action as input and outputs an estimate of the expected return. The goal of the value subnetwork is to learn an accurate value function to provide feedback to the actor's strategy and help the policy subnetwork improve the strategy.

[0078] A Markov decision process (MDP) is a mathematical model used to describe decision-making problems in a fully observable environment, where each decision may affect future states and rewards. It consists of the following core parts: States: various situations that the system may be in. Actions: the actions that can be chosen in each state. Transition Probabilities: the probability of transitioning to another state after performing an action. Reward Function: the immediate reward obtained after performing an action. Decision Rule: The decision rule or policy defines which action should be chosen in a given state. The goal is to find a policy that maximizes the long-term reward calculated in some way.

[0079] First, let’s describe the technical problems existing in the relevant technologies:

[0080] With the development of artificial intelligence technology, reinforcement learning from human feedback (RLHF) has become a new way to train artificial intelligence models. That is, the artificial intelligence model can be further trained based on human feedback data, so that the artificial intelligence model can learn human preferences.

[0081] In related technologies, it is necessary to collect annotated data sets, which are data sets annotated by humans, and then use the annotated data sets to train the reward model. It is also necessary to use exploration strategies to obtain unlabeled data to continue to strengthen the reward model. Finally, the trained reward model provides reward signals to help strengthen the learning of the artificial intelligence model.

[0082] However, the related technologies are limited by the exploration strategy and cannot effectively explore complex state spaces or respond to changes in the environment, and thus cannot generate new data to continue to optimize the reward model, which ultimately results in the reward model being unable to generate reward signals based on new data to reinforce the learning of the artificial intelligence model. This results in the artificial intelligence model being unable to adapt to the new environment.

[0083] In other words, in related technologies, there is a problem of a single exploration strategy, which makes it impossible to guide the intelligent agent and the environment to interact effectively, which will lead to inefficient reinforcement learning of the intelligent agent.

[0084] In order to solve the above technical problems, the embodiments of the present application provide a strategy exploration model training method, device, computer equipment and storage medium. Through the trained strategy exploration model, the exploration strategy can be optimized and enriched, thereby solving the problem of single exploration strategy in related technologies. In this application, more exploration strategies are used to realize the reinforcement learning of the intelligent agent, which can improve the efficiency of the reinforcement learning of the intelligent agent.

[0085] The following is a detailed description of the strategy exploration model training method, device, computer equipment and storage medium provided in the embodiments of the present application.

[0086] See also Figure 1 , Figure 1 Schematic diagram of the system framework corresponding to the strategy exploration model training method provided in the embodiment of the present application. The strategy exploration model training method provided in the embodiment of the present application can be applied to the system framework.

[0087] Please refer to Figure 1 , Figure 1 1 is a system architecture diagram used by the strategy exploration model training method provided in the embodiment of the present application, which includes a terminal 140, the Internet 130, a gateway 120, a server 110, and the like.

[0088] The terminal 140 or the server 110 may be a device for executing the policy exploration model training method.

[0089] The terminal 140 includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. The embodiments of the present application can be applied to various scenarios, including but not limited to cloud office, enterprise management, etc. In addition, it can be a single device or a collection of multiple devices. For example, multiple desktop computers are interconnected through a local area network, share a display, etc. to work together, and together constitute a terminal 140. The terminal 140 can communicate with the Internet 130 in a wired or wireless manner to exchange data.

[0090] The server 110 refers to a computer system that can provide certain services to the terminal 140. Compared with the ordinary terminal 140, the server 110 has higher requirements in terms of stability, security, performance, etc. The server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0091] The gateway 120 is also called an internetwork connector or a protocol converter. The gateway realizes network interconnection at the transport layer and is a computer system or device that acts as a converter. The gateway is a translator between two systems that use different communication protocols, data formats or languages, or even completely different architectures. At the same time, the gateway can also provide filtering and security functions. The message sent by the terminal 140 to the server 110 must be sent to the corresponding server 110 through the gateway 120. The message sent by the server 110 to the terminal 140 must also be sent to the corresponding terminal 140 through the gateway 120.

[0092] The strategy exploration model training method in the embodiment of the present application can be applied to a variety of scenarios, such as games, large language model fine-tuning, etc. The scenario to which the strategy exploration model training method in the present application is applied is not limited here.

[0093] See also Figure 2 , Figure 2 It is a scenario diagram of the strategy exploration model training method provided in an embodiment of the present application.

[0094] exist Figure 2 In the scenario shown, the trained policy exploration model can optimize the existing policy and provide better and more exploration strategies to the agent. The agent can interact with the environment according to the exploration strategy and the state of the environment to achieve the target action.

[0095] Then, after the agent interacts with the environment, there is an interaction result. The quality of the interaction result can be judged manually or through a reward model, and the evaluation data can be fed back to the agent for learning, so that the agent's reinforcement learning can be achieved, making the agent more intelligent.

[0096] For example, taking the intelligent agent as a large language model, the large language model can realize knowledge question and answer. For example, when the intelligent agent inputs "What is an atom?", the intelligent agent can find the corresponding knowledge from the knowledge base to answer, and the answer content is "An atom is the smallest indivisible particle of matter in a chemical reaction. It consists of an atomic nucleus and several electrons orbiting the nucleus." In this way, the action execution of the intelligent agent is realized.

[0097] It is understandable that only when the exploration strategy is better and richer can the intelligent agent interact with the environment to learn more accurate knowledge, thereby enabling the intelligent agent to strengthen learning and subsequently achieve more accurate intelligent services.

[0098] In an embodiment of the present application, the trained strategy exploration model can output better and richer exploration strategies, which can enable the intelligent agent to achieve more effective reinforcement learning based on the exploration strategy, thereby improving the efficiency of reinforcement learning.

[0099] See also Figure 3 , Figure 3 : is a flow chart of a strategy exploration model training method provided in an embodiment of the present application. The strategy exploration model training method may include the following steps:

[0100] Step 210: obtaining a labeled data set and an unlabeled data set, determining a human preference reward and a human non-preference reward according to the labeled data set, and constructing a reward model according to the human preference reward and the human non-preference reward;

[0101] Step 220: determining a preset state of target unlabeled data in the unlabeled data set, and determining a first exploration strategy corresponding to the preset state;

[0102] Step 230: Obtain an action state data set and an execution result output by the agent in interaction with the environment according to the first exploration strategy and the preset state, input the execution result into the reward model to output a reward value, and set the reward value in the action state data set;

[0103] Step 240: determine the action value corresponding to each action state group in the action state data set, and determine the label value corresponding to each action state group;

[0104] Step 250: Determine the evaluation value corresponding to each action state group according to the action value and the label value, and update the first exploration strategy according to the preset exploration strategy and the evaluation value to obtain the updated first exploration strategy, so as to realize the iterative training of the strategy exploration model, and repeatedly determine the preset state of the target unlabeled data in the unlabeled data set until the number of iterations of the strategy exploration model meets the preset number of iterations. Then the strategy exploration model training is completed, and the trained strategy exploration model is obtained.

[0105] Steps 210 to 250 will be described in detail below.

[0106] In step 210, a labeled data set and an unlabeled data set are obtained, a human preference reward and a human non-preference reward are determined according to the labeled data set, and a reward model is constructed according to the human preference reward and the human non-preference reward.

[0107] Among them, the labeled data set can be understood as a data set labeled by humans. The labeled data set includes multiple labeled data, some of which are preferred by humans, and some of which are not preferred by humans. For example, the labeled data set corresponds to the data set corresponding to the correct answer text and the incorrect answer text, among which the labeled data corresponding to the correct answer text is the labeled data preferred by humans, and the labeled data corresponding to the incorrect answer text is the labeled data not preferred by humans.

[0108] The labeled dataset can be used Indicates that To label the dataset, Labeled data that humans do not prefer, is the labeled data preferred by humans, m is the number of multiple labeled data, o m is a parameter describing human preferences in the data, and M is the total number of labeled data.

[0109] The unlabeled data contained in the unlabeled dataset contains the preset state corresponding to each data. The unlabeled dataset can be represented as

[0110] In some implementations, determining a human-preferred reward and a human-non-preferred reward based on a labeled data set includes:

[0111] (1.1) Determine the first labeled data preferred by humans based on the labeled data set, and determine the total reward value corresponding to each first labeled data;

[0112] (1.2) determining a human preference reward according to the total reward value corresponding to each first labeled data;

[0113] (1.3) Determine the second labeled data that is not preferred by humans based on the labeled data set, and determine the total reward value corresponding to each second labeled data;

[0114] (1.4) Determine the human non-preference reward based on the total reward value corresponding to each second labeled data.

[0115] Among them, the first labeled data of human preference is determined in the labeled data set, such as The first annotation data contains multiple action state groups, which can be expressed as Where h is the number of each action state group, H is the total number of action state groups, and an action state group contains a state and the corresponding action.

[0116] Then get the total reward value corresponding to the first labeled data, which can be specifically expressed as The total reward can be understood as the cumulative value of the rewards corresponding to each action state group in the first labeled data.

[0117] Similarly, in the labeled data set, determine the second labeled data that is not preferred by humans, such as The second annotation data includes multiple action state groups, which can be expressed as Where h is the number of each action state group, H is the total number of action state groups, and an action state group contains a state and the corresponding action.

[0118] Then obtain the total reward value corresponding to the second labeled data, which can be specifically expressed as The total reward can be understood as the cumulative value of the reward corresponding to each action state group in the second labeled data.

[0119] In some embodiments, constructing a reward model based on human preference rewards and human non-preference rewards includes:

[0120] (2.1) Construct a human preference model based on human preference rewards and human non-preference rewards;

[0121] (2.2) Build a reward model based on the human preference model.

[0122] Among them, a human preference model can be constructed based on human preference rewards and human non-preference rewards, which can be specifically expressed as:

[0123] in, It can be understood as a human preference model, r ★ (τ 1 ) can be the total reward preferred by humans, r ★ (τ 0 ) is the total non-preference reward of humans, Φ can be a link function, specifically a monotonically increasing link function.

[0124] Specifically, it can be constructed according to a basic statistical model. For example, if the basic statistical model is a BTL (Bradley-Terry-Luce) model, the sigmoid function can be used as a link function to combine the BTL model to construct a human preference model.

[0125] Then, a reward model is constructed based on the human preference model. Specifically, the reward model can be learned by labeling the data set and maximum likelihood estimation. The reward model can be expressed as:

[0126]

[0127] in, For the reward model, is the human preference model, and r is the reward value.

[0128] From the above content, it can be seen that in the embodiment of the present application, the reward model constructed through human preferences can simulate the reward signal generated after the human judgment result, so as to realize the subsequent human feedback reinforcement learning.

[0129] In step 220, a preset state of target unlabeled data is determined in the unlabeled data set, and a first exploration strategy corresponding to the preset state is determined.

[0130] In the unlabeled data set, the target unlabeled data can be randomly sampled and the preset state of the target unlabeled data can be determined. Then the first exploration strategy corresponding to the preset state is determined. The first exploration strategy can be expressed as π t express.

[0131] In step 230, the action state data set and execution result output by the agent in interaction with the environment according to the first exploration strategy and the preset state are obtained, and the execution result is input into the reward model to output the reward value, and the reward value is set in the action state data set.

[0132] Among them, the intelligent agent can determine at least one action according to the first exploration strategy in a preset state, and interact with the environment according to the action, so as to output an action state data set and an execution result.

[0133] In some embodiments, obtaining an action state data set and an execution result output by the agent in interaction with the environment according to the first exploration strategy and the preset state includes:

[0134] (1.1) Obtaining an execution action determined by the agent in a preset state according to the first exploration strategy;

[0135] (1.2) Obtain the action state data set and execution results generated by the intelligent agent in a preset state based on the execution action and the interaction with the environment.

[0136] Specifically, the intelligent agent can determine a series of actions and environmental interactions according to the first exploration strategy under a preset state, thereby generating an action state data set and execution results.

[0137] For example, by controlling the interaction between the agent and the environment through rollout under a preset state and a first exploration strategy, the agent performs a series of interaction steps in the environment according to the first exploration strategy to simulate and collect action state data. In each rollout, the agent observes the current state from the environment and then takes an action according to the selected strategy. Next, the agent interacts with the environment, performs the action, and observes the next state and the reward obtained. Thereafter, the agent updates the first exploration strategy according to the new state and selects the next action in the next step. This process continues until a certain stopping condition is met, such as reaching the maximum number of steps or reaching the terminal state. The length of the rollout can be adjusted according to the specific task, and can be a fixed length or a variable length. In some continuous control tasks, the rollout may last for hundreds or thousands of time steps. However, in some discrete decision tasks, the rollout may be completed within a limited number of steps.

[0138] The purpose of Rollout is to generate action state data through interaction with the environment, thereby generating an action state dataset, which is used for subsequent training of the strategy exploration model.

[0139] In some implementations, the execution result can be input into a reward model to output a reward value, and the reward value can be set in the action state data set. The reward model can imitate human preference behavior, and the reward value output by the reward model is more in line with human preference. The execution result of each action in the action state data set can be input into the reward model, so as to obtain the reward value corresponding to each action.

[0140] Among them, the action state dataset can be expressed as It includes a piece of data as an example, which includes the corresponding state, reward value and advantage value of each action, which can be expressed as follows: where a h can be expressed as the action at the current time step, a h′ represents the action of the next time step, S h Represents the state of the current time step, s h′ represents the state of the next time step, represents the reward value corresponding to the next time step action, ln(π t (a h′ ∣s h′ ) / π SFT (a h′ ∣sh′ )) represents the advantage value corresponding to the next time step.

[0141] In step 240, the action value corresponding to each action state group in the action state data set is determined, and the label value corresponding to each action state group is determined.

[0142] In this application, the strategy exploration model may include a strategy sub-network and a value sub-network. The goal of the strategy sub-network is to learn a good strategy to select actions in different states to maximize the total reward. The value sub-network can obtain the expected cumulative reward after executing the action under a given action state group, which is the evaluation value. In addition, each action state group corresponds to a label value, which can be understood as the real cumulative reward after executing the action state group, that is, the real value. Among them, the action state group can be expressed as (s1, a1), (s2, a2), (s3, a3)... That is, the corresponding actions and states at different time steps.

[0143] In some implementations, determining the action value corresponding to each action state group in the action state data set includes:

[0144] (1.1) Determine the action, state and reward value corresponding to each action state group in the action state data set;

[0145] (1.2) The advantage value and state value are determined based on the action and state, and the action value corresponding to each action-state group is determined based on the reward value, advantage value and state value.

[0146] Among them, the action value can be specifically determined by the Q function, where the action value is Q, the advantage value is A, and the state value is V. The advantage value and the state value can be added to get the action value, and the action value can measure the value corresponding to each action. By using the advantage value, the advantage of each action in the action state set relative to other actions can be determined to avoid overestimating the action value of a certain action. Reward value, advantage value and state value can improve the accuracy of action value.

[0147] In step 250, the evaluation value corresponding to each action state group is determined according to the action value and the label value, and the first exploration strategy is updated according to the preset exploration strategy and the evaluation value to obtain the updated first exploration strategy, so as to realize the iterative training of the strategy exploration model, and repeatedly determine the preset state of the target unlabeled data in the unlabeled data set until the number of iterations of the strategy exploration model meets the preset number of iterations. Then, the strategy exploration model training is completed, and the trained strategy exploration model is obtained.

[0148] The evaluation value corresponding to each action state group can be determined by the action value and the label value, and the evaluation value can be understood as the evaluation value output by the value sub-network. The evaluation value can evaluate the value of each action state group.

[0149] In some implementations, determining the evaluation value corresponding to each action state group according to the action value and the label value includes:

[0150] (1.1) Determine the square value of the difference between the action value and the label value corresponding to each action state group;

[0151] (1.2) Accumulate the square values ​​corresponding to each action state group to obtain the sum of the square values;

[0152] (1.3) According to the sum of the square values ​​and the total number of action state groups, the evaluation value corresponding to each action state group is determined.

[0153] The relevant calculation formula is as follows:

[0154]

[0155] Where Q is the evaluation value, (f(s,a)-y) 2 The square value of the difference between the action value and the label value corresponding to each action state group, f(s,a) is the action value, y is the label value, and argmin is to minimize the evaluation value. This minimizes the difference between the evaluation value and the label value, so that the value sub-network can more accurately evaluate the action state group, adjust the parameters of the value sub-network, and make the value sub-network converge as soon as possible to complete the training of the value sub-network.

[0156] In some implementations, updating the first exploration strategy according to the preset exploration strategy and the evaluation value to obtain an updated first exploration strategy includes:

[0157] (2.1) Determine the action probability distribution corresponding to each action state group;

[0158] (2.2) determining the first KL divergence between the action probability distribution and the action probability distribution corresponding to the preset exploration strategy;

[0159] (2.3) determining the second KL divergence between the action probability distribution and the action probability distribution corresponding to the first exploration strategy;

[0160] (2.4) The first exploration strategy is updated according to the first KL divergence, the second KL divergence and the evaluation value to obtain an updated first exploration strategy.

[0161] Among them, the strategy subnetwork can output the action probability distribution corresponding to each action state group, that is, the probability of executing each action. The greater the probability, the higher the possibility of the action being executed.

[0162] Then determine the first KL divergence between the action probability distribution and the action probability distribution corresponding to the preset exploration strategy. KL divergence (Kullback-Leibler Divergence), also known as relative entropy, is a method of measuring the difference between two probability distributions. In this way, the difference between the action probability distribution under the first exploration strategy and the action probability distribution corresponding to the preset exploration strategy can be measured, so that the subsequent estimated exploration strategy will not deviate too far from the preset exploration strategy.

[0163] For example, the default exploration strategy is π SFT , the first exploration strategy after subsequent update is π t+1 The preset exploration strategy is actually the exploration strategy determined by the labeled data set. It can also be understood as the exploration strategy after fine-tuning the current exploration strategy of the intelligent agent. The preset exploration strategy fits the current exploration strategy of the intelligent agent. In this way, when the strategy exploration model is iterating during training, it can be combined with the preset exploration strategy learned through the labeled data set, avoiding the strategy exploration model from scratch. The training can reduce the number of iterations of the strategy exploration model and improve the training efficiency of the strategy exploration model.

[0164] Then determine the second KL divergence between the action probability distribution and the action probability distribution corresponding to the first exploration strategy. The action probability distribution corresponding to the first exploration strategy can be understood as the action probability distribution corresponding to the current agent after execution, and the action probability distribution corresponding to each action state group is the action probability distribution after the strategy subnetwork in the strategy exploration model executes the action. By calculating the second KL divergence between the two, it is possible to avoid the first exploration strategy after subsequent updates deviating too far from the original first exploration strategy.

[0165] Finally, the first exploration strategy is updated according to the first KL divergence, the second KL divergence and the evaluation value to obtain the updated first exploration strategy. In this way, the updated first exploration strategy, i.e., strategy π t+1 The original first exploration strategy π t and the preset exploration strategy π SFT This makes the updated first exploration strategy better, the iteration efficiency of the strategy sub-network is higher, and it will not deviate from the original exploration strategy.

[0166] In some implementations, updating the first exploration strategy according to the first KL divergence, the second KL divergence, and the evaluation value to obtain an updated first exploration strategy includes:

[0167] (2.4.1) determining a preset regularization parameter corresponding to the first KL divergence, and multiplying the regularization parameter by the first KL divergence to obtain a first result;

[0168] (2.4.2) determining a preset learning rate parameter corresponding to the second KL divergence, and dividing the second KL divergence by the preset learning rate parameter to obtain a second result;

[0169] (2.4.3) The first exploration strategy is updated according to the first result, the second result and the evaluation value to obtain an updated first exploration strategy.

[0170] Among them, the calculation formula of the updated first exploration strategy is as follows:

[0171]

[0172] Among them, π t+1 (s) is the updated first exploration strategy, Q(s,·) is the evaluation value, λ is the preset regularization parameter, η is the preset learning rate parameter, and p is the action probability distribution corresponding to each action state group. KL is the KL divergence. The first result is λKL(p∥π SFT (s)), the second result is

[0173] In this way, the strategy subnetwork can realize the first exploration strategy π t Update to obtain the updated first exploration strategy π t+1 In this way, an iteration of the first exploration strategy is realized, and then the parameters of the strategy exploration model are updated according to the iteration of the updated first exploration strategy, thereby realizing iterative training of the strategy exploration model.

[0174] In the present application, the strategy exploration model can be iteratively trained for multiple times until the number of iterations of the strategy exploration model reaches a preset number of iterations, and the strategy exploration model training is considered completed, and a trained strategy exploration model is obtained. In each iterative training, the updated first exploration strategy can be used as the first exploration strategy, and then the first exploration strategy is continuously optimized, thereby achieving continuous optimization of the first exploration strategy.

[0175] In some implementations, after the strategy exploration model training is completed, the method further includes:

[0176] (3.1) generating a target first result according to a preset exploration strategy, a preset learning rate parameter, and a preset regularization parameter;

[0177] (3.2) generating a target second result according to the current exploration strategy, a preset learning rate parameter, and a preset regularization parameter;

[0178] (3.3) determining a target evaluation value corresponding to the state-action group input into the trained strategy exploration model, and generating a target third result according to the target evaluation value, a preset learning rate parameter, and a preset regularization parameter;

[0179] (3.4) The first result, the second result and the third result are multiplied to determine the target exploration strategy after updating the current exploration strategy.

[0180] Among them, the strategy update expression formula of the trained strategy exploration model is:

[0181]

[0182] Among them, π SFT (a|s) is the preset exploration strategy, and the target first result is π t (a|s) is the current exploration strategy, which can be the current exploration strategy of the agent. The second target result is Determine the target evaluation value corresponding to the state-action group input in the trained strategy exploration model. Specifically, it can be the evaluation value output by the value subnetwork of the trained strategy exploration model, i.e., Q(s, a). The target third result is

[0183] In this way, the first result, the second result and the third result are multiplied to determine the target exploration strategy after updating the current exploration strategy. The target exploration strategy is π t+1 (a|s). The target exploration strategy can be understood as the exploration strategy adopted by the subsequent agent when interacting with the environment.

[0184] In some implementations, after multiplying the first result, the second result, and the third result to determine a target exploration strategy after updating the current exploration strategy, the method further includes:

[0185] (4.1) Update the agent's exploration strategy to the target exploration strategy;

[0186] (4.2) Guide the agent to interact with the environment according to the target exploration strategy to perform the corresponding target action.

[0187] Among them, the exploration strategy of the agent can be replaced, and the target exploration strategy output by the trained strategy exploration model can be used. Then, based on the target exploration strategy, the interaction between the agent and the environment can be controlled, so that the agent can interact with the environment to generate more data. Then, the reward model outputs a reward signal according to the execution results of the agent, and the reward signal is used to guide the agent to perform reinforcement learning, so that the agent can iteratively train and learn more knowledge.

[0188] In an embodiment of the present application, the trained strategy exploration model can update the exploration strategy of the agent by combining the exploration strategy corresponding to the human preference data and the interaction data between the agent and the environment. This can enrich the exploration strategy of the agent and enable the agent to learn more knowledge by interacting with the environment, thereby improving the efficiency of the agent's reinforcement learning.

[0189] It can be seen from the above content that in an embodiment of the present application, by obtaining a labeled data set and an unlabeled data set, human preference rewards and human non-preference rewards are determined according to the labeled data set, and a reward model is constructed according to the human preference rewards and human non-preference rewards; the preset state of the target non-labeled data is determined in the non-labeled data set, and the first exploration strategy corresponding to the preset state is determined; the action state data set and the execution result output by the intelligent agent in interaction with the environment according to the first exploration strategy and the preset state are obtained, and the execution result is input into the reward model to output the reward value, and the reward value is set in the action state data set; the action value corresponding to each action state group in the action state data set is determined, and the label value corresponding to each action state group is determined; the evaluation value corresponding to each action state group is determined according to the action value and the label value, and the first exploration strategy is updated according to the preset exploration strategy and the evaluation value to obtain the updated first exploration strategy, so as to realize the iterative training of the strategy exploration model, and repeatedly determine the preset state of the target non-labeled data in the non-labeled data set until the number of iterations of the strategy exploration model meets the preset number of iterations, then the strategy exploration model training is completed, and the trained strategy exploration model is obtained. In this way, through the trained strategy exploration model, the exploration strategy can be optimized and enriched, thereby solving the problem of single exploration strategy in related technologies. In this application, the reinforcement learning of the intelligent agent is realized through more exploration strategies, which can improve the efficiency of the reinforcement learning of the intelligent agent.

[0190] See also Figure 4 , Figure 4 1 is another flow chart of the strategy exploration model training method provided in the embodiment of the present application. The strategy model training method may include the following steps:

[0191] Step 301: Obtain a labeled data set and an unlabeled data set, and determine a human preference reward and a human non-preference reward according to the labeled data set;

[0192] Step 302: construct a human preference model according to the human preference reward and the human non-preference reward, and construct a reward model based on the human preference model;

[0193] Step 303: determining a preset state of target unlabeled data in the unlabeled data set, and determining a first exploration strategy corresponding to the preset state;

[0194] Step 304: Obtaining an action state data set and an execution result output by the agent in interaction with the environment according to the first exploration strategy and the preset state;

[0195] Step 305: determine the square value of the difference between the action value and the label value corresponding to each action state group;

[0196] Step 306: Accumulate the square values ​​corresponding to each action state group to obtain the sum of the square values;

[0197] Step 307: Determine the evaluation value corresponding to each action state group according to the sum of the square values ​​and the total number of action state groups;

[0198] Step 308: determine the action probability distribution corresponding to each action state group, and determine the first KL divergence between the action probability distribution and the action probability distribution corresponding to the preset exploration strategy;

[0199] Step 309: determine a second KL divergence between the action probability distribution and the action probability distribution corresponding to the first exploration strategy;

[0200] Step 310: updating the first exploration strategy according to the first KL divergence, the second KL divergence and the evaluation value to obtain an updated first exploration strategy, so as to implement iterative training of the strategy exploration model;

[0201] Step 311, repeatedly determine the preset state of the target unlabeled data in the unlabeled data set until the number of iterations of the strategy exploration model meets the preset number of iterations, then the strategy exploration model training is completed, and the trained strategy exploration model is obtained.

[0202] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the above strategy exploration model training method, which will not be repeated here.

[0203] See also Figure 5 , Figure 5 It is a structural diagram of a strategy exploration model training device provided in an embodiment of the present application, and the strategy exploration model training device can execute the above-mentioned strategy exploration model training method.

[0204] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0205] Figure 5 The strategy exploration model training device 400 shown includes:

[0206] A first acquisition module 410 is used to acquire a labeled data set and an unlabeled data set, determine a human preference reward and a human non-preference reward according to the labeled data set, and construct a reward model according to the human preference reward and the human non-preference reward;

[0207] A first determination module 420, configured to determine a preset state of target unlabeled data in the unlabeled data set, and determine a first exploration strategy corresponding to the preset state;

[0208] The second acquisition module 430 is used to obtain the action state data set and the execution result output by the agent according to the first exploration strategy and the preset state and the environment interaction, and input the execution result into the reward model to output the reward value, and set the reward value in the action state data set;

[0209] A second determination module 440 is used to determine the action value corresponding to each action state group in the action state data set, and determine the label value corresponding to each action state group;

[0210] The iteration module 450 is used to determine the evaluation value corresponding to each action state group according to the action value and the label value, and to update the first exploration strategy according to the preset exploration strategy and the evaluation value to obtain the updated first exploration strategy, so as to realize the iterative training of the strategy exploration model, and repeatedly determine the preset state of the target unlabeled data in the unlabeled data set until the number of iterations of the strategy exploration model meets the preset number of iterations. Then, the strategy exploration model training is completed, and the trained strategy exploration model is obtained.

[0211] In some implementations, the first acquisition module 410 is configured to:

[0212] Determine first labeled data preferred by humans based on the labeled data set, and determine a total reward value corresponding to each first labeled data;

[0213] Determine a human preference reward according to a total reward value corresponding to each first labeled data;

[0214] Determine second labeled data that is not preferred by humans based on the labeled data set, and determine a total reward value corresponding to each second labeled data;

[0215] A human non-preferred reward is determined according to the total reward value corresponding to each second labeled data.

[0216] In some implementations, the first acquisition module 410 is configured to:

[0217] Construct a human preference model based on human preference rewards and human non-preference rewards;

[0218] Build reward models based on human preference models.

[0219] In some implementations, the second acquisition module 430 is configured to:

[0220] Obtaining an execution action determined by the agent in a preset state according to the first exploration strategy;

[0221] Obtain the action state data set and execution results generated by the agent in the preset state based on the execution action and the interaction with the environment.

[0222] In some implementations, the second determining module 440 is configured to:

[0223] Determine the action, state and reward value corresponding to each action-state group in the action-state data set;

[0224] The advantage value and the state value are determined according to the action and the state, and the action value corresponding to each action-state group is determined according to the reward value, the advantage value and the state value.

[0225] In some implementations, the iteration module 450 is configured to:

[0226] Determine the square of the difference between the action value and the label value corresponding to each action-state group;

[0227] Accumulate the square values ​​corresponding to each action state group to get the sum of the square values;

[0228] According to the sum of the square values ​​and the total number of action state groups, the evaluation value corresponding to each action state group is determined.

[0229] In some implementations, the iteration module 450 is configured to:

[0230] Determine the action probability distribution corresponding to each action state group;

[0231] Determine the first KL divergence between the action probability distribution and the action probability distribution corresponding to the preset exploration strategy;

[0232] Determine a second KL divergence between the action probability distribution and the action probability distribution corresponding to the first exploration strategy;

[0233] The first exploration strategy is updated according to the first KL divergence, the second KL divergence and the evaluation value to obtain an updated first exploration strategy.

[0234] In some implementations, the iteration module 450 is configured to:

[0235] Determine a preset regularization parameter corresponding to the first KL divergence, and multiply the regularization parameter by the first KL divergence to obtain a first result;

[0236] Determine a preset learning rate parameter corresponding to the second KL divergence, and divide the second KL divergence by the preset learning rate parameter to obtain a second result;

[0237] The first exploration strategy is updated according to the first result, the second result and the evaluation value to obtain an updated first exploration strategy.

[0238] In some embodiments, the strategy exploration model further includes an update module, which is used to:

[0239] Generate a target first result according to a preset exploration strategy, a preset learning rate parameter, and a preset regularization parameter;

[0240] Generate a target second result according to the current exploration strategy, the preset learning rate parameter and the preset regularization parameter;

[0241] Determine the target evaluation value corresponding to the state-action group input in the trained strategy exploration model, and generate the target third result according to the target evaluation value, the preset learning rate parameter and the preset regularization parameter;

[0242] The first result, the second result, and the third result are multiplied together to determine a target exploration strategy after updating the current exploration strategy.

[0243] In some embodiments, the strategy exploration model further includes an input module, which, after multiplying the first result, the second result, and the third result to determine a target exploration strategy after updating the current exploration strategy, is used to:

[0244] Update the agent's exploration strategy to the target exploration strategy;

[0245] The target exploration strategy guides the agent to interact with the environment to perform the corresponding target actions.

[0246] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the above strategy exploration model training method, which will not be repeated here.

[0247] It can be seen from the above content that in the embodiment of the present application, the first acquisition module 410 acquires the labeled data set and the unlabeled data set, determines the human preference reward and the human non-preference reward according to the labeled data set, and constructs a reward model according to the human preference reward and the human non-preference reward; the first determination module 420 determines the preset state of the target non-labeled data in the non-labeled data set, and determines the first exploration strategy corresponding to the preset state; the second acquisition module 430 acquires the action state data set and the execution result output by the intelligent agent according to the first exploration strategy and the preset state and the interaction with the environment, and inputs the execution result into the reward model to output the reward value, and sets the reward value in the action state. state data set; the second determination module 440 determines the action value corresponding to each action state group in the action state data set, and determines the label value corresponding to each action state group; the iteration module 450 determines the evaluation value corresponding to each action state group according to the action value and the label value, and updates the first exploration strategy according to the preset exploration strategy and the evaluation value to obtain the updated first exploration strategy, so as to realize the iterative training of the strategy exploration model, and repeatedly executes the preset state of the target non-labeled data in the non-labeled data set until the number of iterations of the strategy exploration model meets the preset number of iterations, then the strategy exploration model training is completed, and the trained strategy exploration model is obtained. In this way, through the trained strategy exploration model, the exploration strategy can be optimized and enriched, thereby solving the problem of single exploration strategy in the related technology. In this application, more exploration strategies are used to realize the reinforcement learning of the intelligent agent, which can improve the efficiency of the reinforcement learning of the intelligent agent.

[0248] The present application also provides a computer device, which may be a terminal. Figure 6 As shown, it shows a schematic diagram of the structure of the terminal involved in the embodiment of the present application, specifically:

[0249] The computer device may include a radio frequency (RF) circuit 501, a memory 502 including one or more computer-readable storage media, an input unit 503, a display unit 504, a sensor 505, an audio circuit 506, a wireless fidelity (WiFi) module 507, a processor 508 including one or more processing cores, and a power supply 509. Those skilled in the art will appreciate that Figure 6 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0250] The RF circuit 501 can be used for receiving and sending signals during the process of sending and receiving information or making calls. In particular, after receiving the downlink information of the base station, it is handed over to one or more processors 508 for processing; in addition, the data related to the uplink is sent to the base station. Usually, the RF circuit 501 includes but is not limited to an antenna, at least one amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 501 can also communicate with the network and other devices through wireless communication. Wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0251] The memory 502 can be used to store software programs and modules. The processor 508 executes various functional applications and information retrieval by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the terminal (such as audio data, a phone book, etc.), etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 502 may also include a memory controller to provide the processor 508 and the input unit 503 with access to the memory 502.

[0252] The input unit 503 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to object setting and function control. Specifically, in a specific embodiment, the input unit 503 may include a touch-sensitive surface and other input devices. The touch-sensitive surface, also known as a touch display screen or a touch pad, can collect touch operations of an object on or near it (such as operations of an object using any suitable object or accessory such as a finger, a stylus, etc. on or near the touch-sensitive surface), and drive the corresponding connection device according to a pre-set program. Optionally, the touch-sensitive surface may include a touch detection device and a touch controller. Among them, the touch detection device detects the touch orientation of the object, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 508, and can receive and execute commands sent by the processor 508. In addition, the touch-sensitive surface can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic waves. In addition to the touch-sensitive surface, the input unit 503 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0253] The display unit 504 can be used to display information input by the object or information provided to the object and various graphic object interfaces of the terminal, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 504 may include a display panel. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor 508 to determine the type of touch event. The processor 508 then provides a corresponding visual output on the display panel according to the type of touch event. Although in Figure 6 In the embodiment, the touch-sensitive surface and the display panel are used as two independent components to realize input and output functions, but in some embodiments, the touch-sensitive surface and the display panel can be integrated to realize input and output functions.

[0254] The terminal may also include at least one sensor 505, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel according to the brightness of the ambient light, and the proximity sensor may turn off the display panel and / or backlight when the terminal is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the terminal, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0255] The audio circuit 506, the speaker, and the microphone can provide an audio interface between the object and the terminal. The audio circuit 506 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 506 and converted into audio data, and then the audio data is output to the processor 508 for processing, and then sent to another terminal through the RF circuit 501, or the audio data is output to the memory 502 for further processing. The audio circuit 506 may also include an earplug jack to provide communication between an external headset and the terminal.

[0256] WiFi is a short-range wireless transmission technology. The terminal can help the object to send and receive emails, browse web pages and access streaming media through the WiFi module 507. It provides wireless broadband Internet access for the object. Figure 6 A WiFi module 507 is shown, but it is understandable that it is not an essential component of the terminal and can be omitted as required without changing the essence of the invention.

[0257] The processor 508 is the control center of the terminal, and uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 502, and calling data stored in the memory 502, the processor 508 performs various functions of the terminal and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 508 may include one or more processing cores; preferably, the processor 508 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, object interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 508.

[0258] The terminal also includes a power supply 509 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 508 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 509 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0259] Although not shown, the terminal may also include a camera, a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the processor 508 in the terminal will load the executable files corresponding to the processes of one or more applications into the memory 502 according to the following instructions, and the processor 508 will run the applications stored in the memory 502 to achieve various functions:

[0260] Obtain labeled data sets and unlabeled data sets, determine human preference rewards and human non-preference rewards based on the labeled data sets, and build a reward model based on the human preference rewards and human non-preference rewards;

[0261] Determining a preset state of target unlabeled data in the unlabeled data set, and determining a first exploration strategy corresponding to the preset state;

[0262] Obtaining an action state data set and an execution result output by the agent in interaction with the environment according to the first exploration strategy and the preset state, and inputting the execution result into the reward model to output a reward value, and setting the reward value in the action state data set;

[0263] Determine the action value corresponding to each action state group in the action state data set, and determine the label value corresponding to each action state group;

[0264] The evaluation value corresponding to each action state group is determined according to the action value and the label value, and the first exploration strategy is updated according to the preset exploration strategy and the evaluation value to obtain the updated first exploration strategy, so as to realize the iterative training of the strategy exploration model, and repeatedly determine the preset state of the target unlabeled data in the unlabeled data set until the number of iterations of the strategy exploration model meets the preset number of iterations. The strategy exploration model training is completed, and the trained strategy exploration model is obtained.

[0265] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the strategy exploration model training method above, which will not be repeated here.

[0266] In an embodiment of the present application, a labeled data set and an unlabeled data set are obtained, human preference rewards and human non-preference rewards are determined according to the labeled data set, and a reward model is constructed according to the human preference rewards and human non-preference rewards; a preset state of the target non-labeled data is determined in the non-labeled data set, and a first exploration strategy corresponding to the preset state is determined; an action state data set and an execution result output by the intelligent agent in interaction with the environment according to the first exploration strategy and the preset state are obtained, and the execution result is input into the reward model to output a reward value, and the reward value is set in the action state data set; the action value corresponding to each action state group in the action state data set is determined, and the label value corresponding to each action state group is determined; the evaluation value corresponding to each action state group is determined according to the action value and the label value, and the first exploration strategy is updated according to the preset exploration strategy and the evaluation value to obtain the updated first exploration strategy, so as to realize the iterative training of the strategy exploration model, and the preset state of the target non-labeled data is determined in the non-labeled data set repeatedly until the number of iterations of the strategy exploration model meets the preset number of iterations, then the strategy exploration model training is completed, and the trained strategy exploration model is obtained. In this way, through the trained strategy exploration model, the exploration strategy can be optimized and enriched, thereby solving the problem of single exploration strategy in related technologies. In this application, the reinforcement learning of the intelligent agent is realized through more exploration strategies, which can improve the efficiency of the reinforcement learning of the intelligent agent.

[0267] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0268] To this end, an embodiment of the present application provides a computer-readable storage medium, which stores a plurality of instructions, which can be loaded by a processor to execute the steps in any one of the strategy exploration model training methods provided in the embodiment of the present application. For example, the instructions can execute the following steps:

[0269] Obtain labeled data sets and unlabeled data sets, determine human preference rewards and human non-preference rewards based on the labeled data sets, and build a reward model based on the human preference rewards and human non-preference rewards;

[0270] Determining a preset state of target unlabeled data in the unlabeled data set, and determining a first exploration strategy corresponding to the preset state;

[0271] Obtaining an action state data set and an execution result output by the agent in interaction with the environment according to the first exploration strategy and the preset state, and inputting the execution result into the reward model to output a reward value, and setting the reward value in the action state data set;

[0272] Determine the action value corresponding to each action state group in the action state data set, and determine the label value corresponding to each action state group;

[0273] The evaluation value corresponding to each action state group is determined according to the action value and the label value, and the first exploration strategy is updated according to the preset exploration strategy and the evaluation value to obtain the updated first exploration strategy, so as to realize the iterative training of the strategy exploration model, and repeatedly determine the preset state of the target unlabeled data in the unlabeled data set until the number of iterations of the strategy exploration model meets the preset number of iterations. The strategy exploration model training is completed, and the trained strategy exploration model is obtained.

[0274] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the policy exploration model training method provided in various optional implementations provided in the above embodiments.

[0275] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0276] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0277] Since the instructions stored in the computer-readable storage medium can execute the steps in any strategy exploration model training method provided in the embodiments of the present application, the beneficial effects that can be achieved by any strategy exploration model training method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0278] The above is a detailed introduction to a strategy exploration model training method, device, computer equipment and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A strategy exploration model training method, characterized in that: include: Obtain a labeled data set and an unlabeled data set, determine a human preference reward and a human non-preference reward according to the labeled data set, and construct a reward model according to the human preference reward and the human non-preference reward; Determining a preset state of target unlabeled data in the unlabeled data set, and determining a first exploration strategy corresponding to the preset state; Obtaining an action state data set and an execution result output by the agent in interaction with the environment according to the first exploration strategy and the preset state, and inputting the execution result into the reward model to output a reward value, and setting the reward value in the action state data set; Determine the action value corresponding to each action state group in the action state data set, and determine the label value corresponding to each action state group; An evaluation value corresponding to each action state group is determined according to the action value and the label value, and the first exploration strategy is updated according to the preset exploration strategy and the evaluation value to obtain an updated first exploration strategy, so as to realize iterative training of the strategy exploration model, and repeatedly determine the preset state of the target unlabeled data in the unlabeled data set until the number of iterations of the strategy exploration model meets the preset number of iterations. Then, the training of the strategy exploration model is completed, and a trained strategy exploration model is obtained.

2. The strategy exploration model training method according to claim 1, characterized in that: The determining of the human preference reward and the human non-preference reward according to the labeled data set includes: Determine first labeled data preferred by humans according to the labeled data set, and determine a total reward value corresponding to each first labeled data; Determine a human preference reward according to the total reward value corresponding to each of the first labeled data; Determine second labeled data that is not preferred by humans according to the labeled data set, and determine a total reward value corresponding to each second labeled data; A human non-preferred reward is determined according to the total reward value corresponding to each of the second labeled data.

3. The strategy exploration model training method according to claim 1, characterized in that: The reward model is constructed according to human preference rewards and human non-preference rewards, including: Construct a human preference model based on human preference rewards and human non-preference rewards; A reward model is constructed based on the human preference model.

4. The strategy exploration model training method according to claim 1, characterized in that: The acquiring agent outputs an action state data set and an execution result by interacting with the environment according to the first exploration strategy and the preset state, including: Obtaining an execution action determined by the agent according to the first exploration strategy in the preset state; Obtain an action state data set and an execution result generated by the agent in the preset state according to the interaction between the execution action and the environment.

5. The strategy exploration model training method according to claim 1, characterized in that: The determining the action value corresponding to each action state group in the action state data set includes: Determine the action, state and reward value corresponding to each action state group in the action state data set; The advantage value and the state value are determined according to the action and the state, and the action value corresponding to each action state group is determined according to the reward value, the advantage value and the state value.

6. The strategy exploration model training method according to claim 1, characterized in that: Determining the evaluation value corresponding to each action state group according to the action value and the label value includes: Determine the square value of the difference between the action value and the label value corresponding to each action state group; Accumulate the square values ​​corresponding to each action state group to obtain the sum of the square values; An evaluation value corresponding to each action state group is determined according to the sum of the square values ​​and the total number of the action state groups.

7. The strategy exploration model training method according to claim 1, characterized in that: The updating of the first exploration strategy according to the preset exploration strategy and the evaluation value to obtain an updated first exploration strategy includes: Determine the action probability distribution corresponding to each action state group; Determine a first KL divergence between the action probability distribution and the action probability distribution corresponding to a preset exploration strategy; Determining a second KL divergence between the action probability distribution and the action probability distribution corresponding to the first exploration strategy; The first exploration strategy is updated according to the first KL divergence, the second KL divergence, and the evaluation value to obtain an updated first exploration strategy.

8. The strategy exploration model training method according to claim 7, characterized in that: The updating of the first exploration strategy according to the first KL divergence, the second KL divergence and the evaluation value to obtain an updated first exploration strategy includes: Determine a preset regularization parameter corresponding to the first KL divergence, and multiply the regularization parameter by the first KL divergence to obtain a first result; Determine a preset learning rate parameter corresponding to the second KL divergence, and divide the second KL divergence by the preset learning rate parameter to obtain a second result; The first exploration strategy is updated according to the first result, the second result and the evaluation value to obtain an updated first exploration strategy.

9. The strategy exploration model training method according to claim 8, characterized in that: After the strategy exploration model training is completed, it also includes: Generate a target first result according to the preset exploration strategy, the preset learning rate parameter and the preset regularization parameter; Generate a target second result according to the current exploration strategy, the preset learning rate parameter and the preset regularization parameter; Determine a target evaluation value corresponding to the state-action group input into the trained strategy exploration model, and generate a target third result according to the target evaluation value, the preset learning rate parameter and the preset regularization parameter; The first result, the second result and the third result are multiplied together to determine a target exploration strategy after updating the current exploration strategy.

10. The strategy exploration model training method according to claim 9, characterized in that: After multiplying the first result, the second result and the third result to determine the target exploration strategy after updating the current exploration strategy, the method further includes: Updating the agent's exploration strategy to the target exploration strategy; The agent is guided to interact with the environment according to the target exploration strategy to perform corresponding target actions.

11. A strategy exploration model training device, characterized in that: include: A first acquisition module is used to acquire a labeled data set and an unlabeled data set, determine a human preference reward and a human non-preference reward according to the labeled data set, and construct a reward model according to the human preference reward and the human non-preference reward; A first determination module, configured to determine a preset state of target unlabeled data in the unlabeled data set, and determine a first exploration strategy corresponding to the preset state; A second acquisition module is used to obtain an action state data set and an execution result output by the agent according to the first exploration strategy and the preset state and the interaction with the environment, and input the execution result into the reward model to output a reward value, and set the reward value in the action state data set; A second determination module is used to determine the action value corresponding to each action state group in the action state data set, and determine the label value corresponding to each action state group; An iteration module is used to determine the evaluation value corresponding to each action state group according to the action value and the label value, and to update the first exploration strategy according to the preset exploration strategy and the evaluation value to obtain the updated first exploration strategy, so as to realize iterative training of the strategy exploration model, and repeatedly determine the preset state of the target non-labeled data in the non-labeled data set until the number of iterations of the strategy exploration model meets the preset number of iterations, then the strategy exploration model training is completed, and the trained strategy exploration model is obtained.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, which are suitable for loading by a processor to execute the strategy exploration model training method described in any one of claims 1 to 10.

13. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the strategy exploration model training method described in any one of claims 1 to 10 is implemented.

Images