Reward model training method and device, computer equipment and storage medium

Abstract

The embodiment of the invention discloses a reward model training method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a sample data set, and enabling each training sample in the sample data set to comprise behavior associated data and a corresponding behavior style label; obtaining a reward model with the same structure for each behavior style; for each behavior style, determining a style matching degree between the training sample and the behavior style through a reward model of the behavior style; according to the reward model of each behavior style, feedback signals are generated for the training samples and the corresponding style matching degrees, and the feedback signals indicate whether the training samples conform to the behavior styles or not; and for the reward model of each behavior style, training the reward model based on the behavior style label of the training sample and the feedback signal to obtain the trained reward model corresponding to each behavior style, so that the difficulty in obtaining the reward models corresponding to different behavior styles can be reduced.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a reward model training method, apparatus, computer equipment and computer-readable storage medium, wherein the storage medium is a computer-readable storage medium. Background Art

[0002] Reinforcement Learning is a machine learning method used to guide agents on how to take actions in a dynamic environment to maximize rewards. In reinforcement learning, the agent perceives the current state in the environment, takes corresponding actions according to the current policy, interacts with the environment, and obtains feedback signals, namely rewards. The agent further adjusts its behavior strategy according to the feedback rewards until the optimal behavior strategy can be generated. Rewards are generally obtained based on reward functions. When the reward function remains unchanged, the agent can obtain a relatively fixed strategy after each training. Since the reward function is fixed, the strategy behaviors generated by the decision model are similar, which makes it easy for players to quickly identify the characters in the game as AI, not real players, and to easily predict their behaviors, resulting in a poor gaming experience.

[0003] In order to achieve strategy diversity, one method is to set different reward functions. Different reward functions are equivalent to different goals, so the corresponding training can often generate diverse strategies. However, setting different reward functions requires a high degree of reliance on relevant expert experience, which makes the setting of reward functions difficult. Summary of the invention

[0004] The embodiments of the present application provide a reward model training method, apparatus, computer device, and computer-readable storage medium, which can reduce the difficulty of obtaining reward models corresponding to different behavior styles.

[0005] A reward model training method provided in an embodiment of the present application includes:

[0006] Acquire a sample data set, wherein each training sample in the sample data set includes behavior association data and a corresponding behavior style label;

[0007] Get a reward model with the same structure for each behavior style;

[0008] For each behavior style, determining the style matching degree between the training sample and the behavior style through the reward model of the behavior style;

[0009] According to the reward model of each behavior style, a feedback signal is generated for the training sample and the corresponding style matching degree, wherein the feedback signal indicates whether the training sample conforms to the behavior style;

[0010] For the reward model of each behavior style, the reward model is trained based on the behavior style label and feedback signal of the training sample to obtain a trained reward model corresponding to each behavior style.

[0011] Accordingly, an embodiment of the present application also provides a reward model training device, comprising:

[0012] A data acquisition unit, used to acquire a sample data set, wherein each training sample in the sample data set includes behavior association data and a corresponding behavior style label;

[0013] A model acquisition unit, used to acquire a reward model with the same structure for each behavior style;

[0014] a determination unit, configured to determine, for each behavior style, a style matching degree between the training sample and the behavior style by using a reward model for the behavior style;

[0015] A generating unit, configured to generate a feedback signal for the training sample and the corresponding style matching degree according to the reward model of each behavior style, wherein the feedback signal indicates whether the training sample conforms to the behavior style;

[0016] The training unit is used to train the reward model for each behavior style based on the behavior style label and feedback signal of the training sample to obtain a trained reward model corresponding to each behavior style.

[0017] Correspondingly, an embodiment of the present application also provides a computer device, including a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute any reward model training method provided in the embodiment of the present application.

[0018] Correspondingly, an embodiment of the present application also provides a computer-readable storage medium, which is used to store a computer program, and the computer program is loaded by a processor to execute any reward model training method provided in the embodiment of the present application.

[0019] The embodiment of the present application obtains a sample data set, wherein each training sample in the sample data set includes behavior association data and a corresponding behavior style label; obtains a reward model with the same structure for each behavior style; for each behavior style, determines the style matching degree between the training sample and the behavior style through the reward model of the behavior style; based on the reward model of each behavior style, generates a feedback signal for the matching degree between the training sample and the corresponding style, and the feedback signal indicates whether the training sample conforms to the behavior style; for the reward model of each behavior style, trains the reward model based on the behavior style label of the training sample and the feedback signal to obtain a trained reward model corresponding to each behavior style.

[0020] In the reward model training method provided in the present application, the reward model can determine the degree of match between the training samples and the behavioral style, and generate a feedback signal based on the degree of match between the training samples and the style. Therefore, the feedback signal can reflect whether the training samples meet the achievement goals corresponding to the reward model; it can also reflect whether the training samples meet the behavioral style, so that the trained reward model can reward strategic behaviors that meet the behavioral style and achievement goals, and punish strategies of other behavioral styles or strategic behaviors that do not meet the achievement goals. In the embodiment of the present application, the reward model is trained with training samples of different behavioral styles to obtain a reward model corresponding to the behavioral style. There is no need to set different reward functions for different behavioral styles, which can reduce the difficulty of obtaining reward models for different behavioral styles. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flow chart of the reward model training method provided in an embodiment of the present application;

[0023] Figure 2 It is a schematic diagram of a strategy behavior generation model provided in an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of a reward model training device provided in an embodiment of the present application;

[0025] Figure 4 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0027] The embodiments of the present application provide a reward model training method, apparatus, computer device and computer readable storage medium. The reward model training apparatus can be integrated in a computer device, which can be a server or a terminal.

[0028] The terminal may include a mobile phone, a wearable smart device, a tablet computer, a laptop computer, a personal computer (PC), and a vehicle-mounted computer.

[0029] Among them, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0030] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0031] This embodiment will be described from the perspective of a reward model training device, which can be integrated into a computer device, which can be a server or a terminal.

[0032] The present application embodiment provides a reward model training method, such as Figure 1 As shown, the specific process of the reward model training method can be as follows:

[0033] 101. Obtain a sample data set, where each training sample in the sample data set includes behavior association data and a corresponding behavior style label.

[0034] The behavior style label is a label corresponding to the behavior style. The sample data set contains training samples of different behavior styles. The behavior style label can be manually labeled, or determined and automatically labeled according to the characteristics of the training samples. That is, in one embodiment, the step of "obtaining a sample data set" may specifically include:

[0035] Get the initial sample data set;

[0036] Determine the behavior style corresponding to each training sample according to the feature information of each training sample in the initial sample data set;

[0037] A behavior style label corresponding to each training sample is generated according to the behavior style of each training sample to obtain a sample data set.

[0038] Among them, the training samples in the initial sample data are not marked with behavioral style labels. The feature information may include the characteristics of strategic behavior, the characteristics of the environment, and the attribute characteristics of the character, etc. The attribute characteristics of the character may be the attributes such as the life value of the game character. According to the characteristics of the pre-set behavioral style and the feature information of the training sample, the behavioral style corresponding to each training sample is determined; according to the behavioral style of each training sample, the behavioral style label corresponding to each training sample is generated to obtain a sample data set.

[0039] Each training sample contains behavior association data and a label of the corresponding behavior style. The behavior association data may include behavior data and at least one of the data related to the behavior. The data related to the behavior may include, for example, the environment state information in which the behavior indicated by the behavior data is executed, and the attribute information of the character, such as the health value of the character, the health value of the character's opponent, etc.

[0040] The data contained in the behavior association data can be flexibly adjusted according to the setting of the behavior style. For example, the movement style of the game character can be divided into fast style and slow style. When in a fixed number of game frames, the character's movement times are greater than 22 times for fast style and less than 10 times for slow style, then the behavior association data can only include the action sequence performed by the character. Under this style setting, the feature information can be the number of actions, and the behavior style label of the training sample is determined according to the number of actions in the behavior association data.

[0041] If the behavior style is set to attack style and defense style, the attack style tends to cause harm to the opponent, and the defense style tends to protect oneself from harm. Under this setting, the behavior-related data may include the sequence of actions performed by the character, the character's health value and the opponent's health value, environmental data of the game environment, etc.

[0042] The sample data set is used to train the reward model so that the trained reward model can reward based on a specific behavior style, that is, the decision model can be reinforced according to the trained reward model so that the decision model can output a strategy that matches the behavior style.

[0043] Reinforcement Learning is a machine learning method used to guide agents on how to take actions in a dynamic environment to maximize rewards. In reinforcement learning, the agent perceives the current state in the environment, takes corresponding actions according to the current policy, interacts with the environment, and obtains feedback signals, i.e. rewards. The agent further adjusts its own behavior strategy according to the feedback rewards until the optimal behavior strategy can be generated.

[0044] In the game scene, the intelligent agent can cooperate or fight with real players in a similar way to how real players manipulate virtual objects. Based on the design principles and implementation methods of various intelligent machines, the artificial intelligence agent of the present disclosure can understand different game strategies and cooperate and fight with real players with different strategies.

[0045] Rewards are generally obtained based on a reward function. When the reward function remains unchanged, the agent can obtain a relatively fixed strategy after each training. Since the reward function remains unchanged, the strategy behaviors generated by the decision-making model are similar, which makes it easy for players to quickly identify that the characters in the game are AI, not real players, and easily predict their behaviors, resulting in a poor gaming experience.

[0046] In order to achieve strategy diversity, one of the representative methods is reinforcement learning combined with population algorithm (Population Based Training, PBT). The PBT method is inspired by the natural process of biological evolution. It generates different offspring through population initialization, crossover mutation, etc. It is a mature global optimization method with high robustness and wide applicability. By treating each offspring as a strategy, the evolutionary algorithm is introduced into reinforcement learning, and then through crossover mutation, the parameters are automatically adjusted and the corresponding diverse strategies are generated. The actual computing resources required for this training method are very large, and it is difficult to put it into actual operation in large-scale scenarios. The success rate of generating effective strategies is also relatively low, so it is difficult to be widely cited.

[0047] Another method is to adjust the rewards. Different reward settings are equivalent to different achievement goals, so the corresponding training can often generate diverse strategies. However, setting different rewards requires a lot of labor costs and is highly dependent on relevant expert experience.

[0048] In the reward model training method provided in the present application, the training samples in the sample data set correspond to behavioral style labels. The reward model for each behavioral style can generate a feedback signal in combination with the style matching degree of the training samples, and train the reward model according to the behavioral style labels, so that the reward model can reward strategic behaviors with specified behavioral styles that can meet the goals achieved corresponding to the reward model, and punish other behavioral styles or strategic behaviors that do not meet the goals. Therefore, through reward models corresponding to different behavioral styles, strategic behaviors that not only meet the goals but also have diverse behavioral styles can be obtained.

[0049] 102. Obtain a reward model with the same structure for each behavior style.

[0050] Among them, each behavior style can be set up with a corresponding reward model, which can be considered a reward model that is independent of the style, and then the reward model is matched with the behavior style through training. The reward model will generate a feedback signal, which represents the "goodness" of the intelligent agent in a specific environmental state and taking a specific action.

[0051] The structure of the reward model corresponding to each behavior style is the same, which may refer to the network topology of each reward model. The number of network layers set in each reward model is the same, and the functions set by the neurons contained in the nth network layer of each reward model and the number of neurons are the same. It can be considered that the same initial reward model is trained. Since each reward model corresponds to a different behavior style, the training samples of each reward model "tendency" are different. For example, assuming that there is a training sample with a style label of behavior style A, for the reward model corresponding to behavior style A, the training sample is a positive sample; for the reward model corresponding to style B, the training sample is a negative sample, so based on the same sample data set, a reward model that can reward behaviors of the corresponding behavior style can be trained. In one embodiment, the reward model may include a matching prediction network, which is used to predict the matching degree between the training sample and the behavior style corresponding to the reward model. Optionally, the matching prediction network may be a classification network.

[0052] In one embodiment, the reward model may further include a reward network, and the reward network may be a neural network modeled based on the reward function.

[0053] 103. For each behavior style, the style matching degree between the training sample and the behavior style is determined through the behavior style reward model.

[0054] For example, for each behavior style, the training sample is input into the reward model corresponding to the behavior style, and the style matching degree between the training sample and the behavior style is obtained through the reward model. The higher the style matching degree, the more the training sample conforms to the behavior style, and the more the reward model needs to reward.

[0055] In one embodiment, the reward model may include a matching prediction network, which is used to determine the matching degree between the training sample and the behavior style corresponding to the reward model. Optionally, the matching prediction network may be a classification network.

[0056] The classification network can be used to determine the probability that the training sample belongs to the corresponding behavior style. That is, in one embodiment, the reward model includes the classification network, and the step of "for each behavior style, predicting the style matching degree between the training sample and the behavior style through the reward model of the behavior style" can specifically include:

[0057] Through the classification network of the reward model of each behavior style, feature extraction is performed on the training samples to obtain the behavior-related feature information of the training samples;

[0058] For each behavior style, the style matching degree with the behavior style is determined according to the behavior-related feature information through the classification network corresponding to the behavior style.

[0059] For example, for one behavior style among multiple behavior styles, the training sample is input into the reward model corresponding to the behavior style, and the features of the training sample are extracted through the classification network of the reward model to obtain the behavior-related feature information of the training sample, and the style matching degree between the training sample and the behavior style is determined based on the behavior-related feature information. Similarly, the training sample is input into the reward model corresponding to other behavior styles, and the style matching degree of the training sample is output through the reward model to train the reward model.

[0060] Among them, the classification network can be set with at least two categories, for example, it can be a binary classification, including a category that is consistent with the behavioral style and a category that is inconsistent with the behavioral style, or it can be a ternary classification, including a category that is consistent with the behavioral style, a category that is inconsistent with the behavioral style, and a category that is similar to the behavioral style.

[0061] For binary classification, the classification network can be a Softmax network layer. The Softmax network layer outputs 1 to indicate that the behavior style is consistent, and outputs 0 to indicate that the behavior style is inconsistent.

[0062] The behavior-related characteristic information may include behavior characteristic information of the strategic behavior, and may also include environmental characteristic information of the environment, etc.

[0063] 104. According to the reward model for each behavior style, a feedback signal is generated for the training sample and the corresponding style matching degree, and the feedback signal indicates whether the training sample conforms to the behavior style.

[0064] Among them, the feedback signal may include one of a reward signal and a penalty signal. The feedback signal may indicate whether the training sample meets the behavioral goal (also called the achievement goal). The behavioral goal indicates the goal to be achieved by the intelligent agent in executing the strategic behavior, for example, it may be to prolong the survival time, or to collect more props in the game, etc. It may also indicate whether the training sample is consistent with the behavioral style. If the feedback signal is a reward signal, it indicates that the reward model believes that the training sample meets the behavioral goal and the behavioral style. If the feedback signal is a penalty signal, it indicates that the reward model believes that the training sample does not meet the behavioral goal or the behavioral style.

[0065] For example, for a reward model of one behavior style, the corresponding feedback signal can be obtained through the reward model according to the behavior-related feature information and style matching degree of the training sample. Similarly, the feedback signal of the reward model of other behavior styles for the training sample can be obtained.

[0066] In one embodiment, the feedback signal may be calculated according to a preset reward function, and the style matching degree is used to adjust the feedback information obtained according to the reward function.

[0067] In another embodiment, the reward function may be modeled to obtain a neural network corresponding to the reward function, i.e., a reward network. Then, the network parameters of the reward network and the matching prediction network in the reward model may be dynamically adjusted through back propagation. That is, the reward model also includes a reward network. The step of "generating a feedback signal for the training sample and the corresponding style matching degree according to the reward model of each behavior style" includes:

[0068] Output an initial feedback signal for the training samples according to the reward model for each behavior style;

[0069] The initial feedback signal is adjusted based on the style matching degree to obtain a feedback signal.

[0070] For example, the reward model for one type of behavioral style can output an initial feedback signal based on whether the training sample is consistent with the behavioral goal. Since the reward model is to reward strategic behaviors that meet both the behavioral goal and the behavioral style, the initial feedback signal can be adjusted based on the style matching degree so that the feedback signal can reflect the relationship between the training sample and the behavioral goal and behavioral style.

[0071] For example, the style matching degree can be in the range of [-1,1]. The smaller the style matching degree, the less consistent the training sample is with the behavior style. The initial feedback signal is the signal output by the reward model. The signal is a scalar. If the initial feedback signal is less than zero, it means punishment; if the initial feedback signal is greater than zero, it means reward; if the initial feedback signal is equal to zero, it means neutrality.

[0072] Therefore, the style matching degree can be used to adjust the initial feedback signal so that the adjusted feedback signal can provide feedback on whether the training sample and the behavioral style match. Specifically, if the style matching degree is less than zero and the initial feedback signal is less than zero, then the absolute value of the style matching degree can be used as a weight to adjust the initial feedback signal so that the more mismatched the style of the training sample is, the greater the penalty will be.

[0073] If the style match is greater than zero and the initial feedback signal is greater than zero, then the style match can be used as a weight to adjust the initial feedback signal so that training samples with more matching styles correspond to larger rewards.

[0074] If the style match is less than zero and the initial feedback signal is greater than zero, or the style match is greater than zero and the initial feedback signal is less than zero, then the initial feedback signal is multiplied by the style match to adjust the initial feedback signal so that training samples with less style matching receive a greater penalty.

[0075] Similarly, the initial feedback information output by the reward model of other behavioral styles can be adjusted through the above process.

[0076] 105. For the reward model of each behavior style, the reward model is trained based on the behavior style label and feedback signal of the training sample to obtain a trained reward model corresponding to each behavior style.

[0077] Since the feedback signal can indicate the conformity between the training sample and the behavior style, the reward model can be trained according to the behavior style label and the feedback signal to obtain a trained reward model. For example, the network parameters of the reward model are adjusted based on the feedback signal through the learning algorithm, so that the reward model can reward behaviors that conform to the corresponding behavior style. The learning algorithm is an algorithm for training the parameters of the reward model, which defines how to optimize the network parameters of the reward model through back propagation. The reinforcement learning algorithm can include the Q learning algorithm, etc. The trained reward model can reward the intelligent agent that performs strategic behaviors that conform to the behavior style and the behavior goal.

[0078] After obtaining the trained reward model, the decision model can be trained according to the trained reward model, so that the decision model can output strategic behaviors that meet the behavioral goals and behavioral styles, and the intelligent agent can be controlled by different decision models, so that the intelligent agent's behavioral style is changeable and not easily identified as a non-game player character. That is, in one embodiment, after the step of "for each behavioral style reward model, the reward model is trained based on the behavioral style label and feedback signal of the training sample to obtain the trained reward model corresponding to each behavioral style", the reward model training method provided in the embodiment of the present application can also include:

[0079] Obtaining a style decision model and an environmental data sample for each behavior style, wherein each style decision model includes a decision model and a post-training reward model;

[0080] Generate strategic behavior data based on environmental data samples through style decision model;

[0081] Generate feedback signals based on the strategy behavior data and the corresponding style matching degree according to the trained reward model of the style decision model;

[0082] The style decision model is trained according to the feedback signal to obtain a trained style decision model corresponding to each behavioral style.

[0083] Among them, each behavioral style corresponds to a style decision model, which can control the intelligent agent to perform behaviors that are consistent with the behavioral goals and behavioral styles. The style decision model includes a decision model and a post-training reward model corresponding to the behavioral style. The decision model can be a model that needs to be trained, and the model structures of decision models of different behavioral styles can be the same.

[0084] The environmental data samples may be data such as the environmental state perceived by the agent in the environment. For example, the environmental data samples may include image frames corresponding to the environment.

[0085] The environmental data samples are input into the style decision model, and the decision model of the style decision model generates strategy behavior data according to the environmental data samples. Then, the feedback signal is generated through the post-training reward model of the style decision model. The style decision model is trained according to the feedback signal to obtain the post-training style decision model corresponding to each behavior style. During the training of the style decision model, the post-training reward model can be fine-tuned, or the parameters of the post-training reward model can be set unchanged.

[0086] Optionally, each style decision model may be a branch model in a policy behavior generation model. The policy behavior generation model may include a style selection network and at least two branch models. The style selection network is used to select at least one branch model for outputting policy behavior according to the environment. Therefore, in one embodiment, each style decision model is a branch model of the policy behavior generation model. The policy behavior generation model also includes a style selection network. The step of "generating policy behavior data based on environment data samples by using the style decision model" may include:

[0087] For each environment data sample, at least one target behavior style is selected through the style selection network;

[0088] Through the decision model of the target behavior style corresponding to the style decision model, strategic behavior data is generated based on the environmental data samples.

[0089] For example, the model structure of the strategic behavior generation model can be as follows Figure 2 As shown, at least one target behavior style can be selected through the style selection network in the strategy behavior generation model; and strategy behavior data can be generated based on the environment data samples through the decision model of the style decision model corresponding to the target behavior style.

[0090] The style decision model of the target behavior style is trained according to the feedback signal, and the style selection network is trained. Each branch model and style selection network in the strategy behavior generation model can be trained through environmental data samples to obtain the trained strategy behavior generation model, and then the style decision model corresponding to different behavior styles can be obtained.

[0091] The environmental data sample set contains environmental data samples corresponding to different behavioral styles. The style selection network can determine which branch model each environmental data sample in the environmental data sample set is used to train, thereby reducing the interference between environmental data samples corresponding to different behavioral styles and speeding up the training speed of each branch model.

[0092] After obtaining the trained behavior generation model, the agent can be made to perform behaviors of different styles. That is, in one embodiment, the trained style decision model corresponding to each behavior style is a branch model of the trained strategy behavior generation model. After the step of "training the style decision model according to the feedback signal to obtain the trained style decision model corresponding to each behavior style", the reward model training method provided in the embodiment of the present application may also include:

[0093] Get the environment data of the agent's current environment;

[0094] Select at least one target behavior style through the style selection grid of the trained policy behavior generation model;

[0095] Generate strategic behavior data based on environmental data through a trained decision model corresponding to the target behavior style;

[0096] Based on the strategic behavior data, the agent is controlled to perform corresponding behaviors in the environment.

[0097] Obtaining environmental data of the environment in which the agent is currently located can be done by obtaining an image frame of the corresponding environment based on the current position of the agent in the environment. Selecting at least one target behavior style through the style selection grid of the trained strategy behavior generation model; generating strategy behavior data that conforms to the target behavior style based on the environmental data through the decision model corresponding to the target behavior style; and controlling the agent to perform corresponding behaviors in the environment based on the strategy behavior data.

[0098] When at least two target behavior styles are selected, the strategic behavior data of a trained style decision model of one target behavior style can be selected to control the agent to perform the corresponding behavior.

[0099] Optionally, the policy behavior data of at least two trained style decision models may be fused to obtain fused policy behavior data, and the intelligent agent may be controlled based on the fused policy behavior data.

[0100] The trained strategy behavior generation model includes a style selection network, which can select the appropriate target behavior style according to the environmental data. The style decision model corresponding to the target behavior style can generate strategy behavior data that is consistent with the behavior goal and target behavior style according to the environmental data. By selecting different behavior styles, the styles of the obtained strategy behavior data are different, thus ensuring the diversity of strategies and allowing the intelligent agent to behave in a variety of behavior styles in the environment.

[0101] As can be seen from the above, the embodiment of the present application obtains a sample data set, each training sample in the sample data set includes behavior association data and a corresponding behavior style label; obtains a reward model with the same structure for each behavior style; for each behavior style, determines the style matching degree between the training sample and the behavior style through the reward model of the behavior style; based on the reward model of each behavior style, generates a feedback signal for the matching degree between the training sample and the corresponding style, and the feedback signal indicates whether the training sample conforms to the behavior style; for the reward model of each behavior style, trains the reward model based on the behavior style label of the training sample and the feedback signal to obtain a trained reward model corresponding to each behavior style.

[0102] In the reward model training method provided in the present application, the reward model can determine the degree of match between the training samples and the behavioral style, and generate a feedback signal based on the degree of match between the training samples and the style. Therefore, the feedback signal can reflect whether the training samples meet the achievement goals corresponding to the reward model; it can also reflect whether the training samples meet the behavioral style, so that the trained reward model can reward strategic behaviors of specified behavioral styles and punish strategic behaviors of other behavioral styles. In the embodiment of the present application, the reward model is trained by training samples of different behavioral styles, so that the reward model corresponding to the behavioral style can be obtained. There is no need to set different reward functions for different behavioral styles, which can reduce the difficulty of obtaining reward models for different behavioral styles.

[0103] In order to facilitate better implementation of the reward model training method provided in the embodiment of the present application, a reward model training device is also provided in one embodiment. The meanings of the terms are the same as those in the above-mentioned reward model training method, and the specific implementation details can refer to the description in the method embodiment.

[0104] The reward model training device can be integrated into a computer device, such as Figure 3 As shown, the reward model training device may include: a data acquisition unit 301, a model acquisition unit 302, a determination unit 303, a generation unit 304 and a training unit 305, which are specifically as follows:

[0105] (1) A data acquisition unit 301 is used to acquire a sample data set, where each training sample in the sample data set includes behavior association data and a corresponding behavior style label.

[0106] In one embodiment, the data acquisition unit 301 may include a data set acquisition subunit, a style determination subunit and a label generation subunit, specifically:

[0107] A data set acquisition subunit, used to acquire an initial sample data set;

[0108] A style determination subunit, used to determine the behavior style corresponding to each training sample according to the feature information of each training sample in the above-mentioned initial sample data set;

[0109] The label generation subunit is used to generate a behavior style label corresponding to each training sample according to the behavior style of each training sample to obtain the above sample data set.

[0110] (2) a model acquisition unit 302, used to acquire a reward model with the same structure for each behavior style;

[0111] (3) A determination unit 303 is used to determine, for each behavior style, a style matching degree between the training sample and the behavior style by using the reward model of the behavior style.

[0112] In one embodiment, the reward model includes a classification network, and the determination unit 303 may include an extraction subunit and a matching degree determination subunit, specifically:

[0113] An extraction subunit, used to extract features from the training samples through a classification network of a reward model for each behavior style, to obtain behavior-related feature information of the training samples;

[0114] The matching degree determination subunit is used to determine, for each behavior style, the style matching degree with the above behavior style according to the above behavior-related feature information through the classification network corresponding to the above behavior style.

[0115] (4) A generating unit 304 is used to generate a feedback signal for the training sample and the corresponding style matching degree according to the reward model of each behavior style, wherein the feedback signal indicates whether the training sample conforms to the behavior style.

[0116] In one embodiment, the generating unit 304 may include a signal generating subunit and a regulating subunit:

[0117] A signal generating subunit, for generating an initial feedback signal for the training sample according to a reward model of each behavior style;

[0118] The adjustment subunit is used to adjust the initial feedback signal based on the style matching degree to obtain the feedback signal.

[0119] (5) A training unit 305 is used to train the reward model for each behavior style based on the behavior style label and feedback signal of the training sample to obtain a trained reward model corresponding to each behavior style.

[0120] In one embodiment, the reward model training device provided in the embodiment of the present application may further include a sample acquisition unit, a strategy generation unit, a feedback signal generation unit and a model training unit, specifically:

[0121] A sample acquisition unit, used to acquire a style decision model and an environmental data sample for each behavior style, wherein each style decision model includes a decision model and a post-training reward model;

[0122] A strategy generating unit, configured to generate strategy behavior data based on the above environment data sample by using the above style decision model;

[0123] A feedback signal generating unit, configured to generate a feedback signal for the strategy behavior data and the corresponding style matching degree through a trained reward model of the style decision model;

[0124] The model training unit is used to train the style decision model according to the feedback signal to obtain a trained style decision model corresponding to each behavior style.

[0125] In one embodiment, each style decision model is a branch model of a strategy behavior generation model, the strategy behavior generation model further includes a style selection network, and the strategy generation unit may include a selection subunit and a data generation subunit, specifically:

[0126] A selection subunit, for selecting at least one target behavior style through the style selection network for each environment data sample;

[0127] The data generation subunit is used to generate strategic behavior data based on the above-mentioned environmental data samples through the decision model of the style decision model corresponding to the above-mentioned target behavior style.

[0128] In one embodiment, the trained style decision model corresponding to each behavior style is a branch model of the trained strategy behavior generation model. The reward model training device provided in the embodiment of the present application may also include an environment data acquisition unit, a style selection unit, a behavior data generation unit and a control unit. Specifically:

[0129] An environmental data acquisition unit, used to acquire environmental data of the environment in which the agent is currently located;

[0130] A style selection unit, configured to select at least one target behavior style through the style selection grid of the trained policy behavior generation model;

[0131] A behavior data generating unit, configured to generate strategy behavior data according to the above-mentioned environment data according to the decision model corresponding to the above-mentioned target behavior style;

[0132] A control unit is used to control the above-mentioned intelligent agent to perform corresponding behaviors in the above-mentioned environment based on the above-mentioned strategic behavior data.

[0133] As can be seen from the above, the reward model training device of the embodiment of the present application obtains a sample data set through the data acquisition unit 301, and each training sample in the sample data set includes behavior association data and a corresponding behavior style label; the model acquisition unit 302 obtains a reward model with the same structure for each behavior style; the determination unit 303 determines the style matching degree between the training sample and the behavior style through the reward model of the behavior style for each behavior style; the generation unit 304 generates a feedback signal for the training sample and the corresponding style matching degree according to the reward model of each behavior style, and the feedback signal indicates whether the training sample conforms to the behavior style; the training unit 305 trains the reward model for each behavior style based on the behavior style label of the training sample and the feedback signal to obtain a trained reward model corresponding to each behavior style.

[0134] In the reward model training method provided in the present application, the reward model can determine the degree of match between the training samples and the behavioral style, and generate a feedback signal based on the degree of match between the training samples and the style. Therefore, the feedback signal can reflect whether the training samples meet the achievement goals corresponding to the reward model; it can also reflect whether the training samples meet the behavioral style, so that the trained reward model can reward strategic behaviors of specified behavioral styles and punish strategic behaviors of other behavioral styles. In the embodiment of the present application, the reward model is trained by training samples of different behavioral styles, so that the reward model corresponding to the behavioral style can be obtained. There is no need to set different reward functions for different behavioral styles, which can reduce the difficulty of obtaining reward models for different behavioral styles.

[0135] Accordingly, the embodiment of the present application also provides a computer device, which may be a terminal. Figure 4 As shown, Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device 500 includes a processor 501 having one or more processing cores, a memory 502 having one or more computer-readable storage media, and a computer program stored in the memory 502 and executable on the processor. The processor 501 is electrically connected to the memory 502. It will be understood by those skilled in the art that the computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0136] The processor 501 is the control center of the computer device 500. It uses various interfaces and lines to connect the various parts of the entire computer device 500, executes various functions of the computer device 500 and processes data by running or loading software programs and / or modules stored in the memory 502, and calling data stored in the memory 502, thereby monitoring the computer device 500 as a whole.

[0137] In the embodiment of the present application, the processor 501 in the computer device 500 will load instructions corresponding to the processes of one or more application programs into the memory 502 according to the following steps, and the processor 501 will run the application programs stored in the memory 502 to implement various functions:

[0138] Obtain a sample data set, wherein each training sample in the sample data set includes behavior association data and a corresponding behavior style label;

[0139] Get a reward model with the same structure for each behavior style;

[0140] For each behavior style, determining the style matching degree between the training sample and the behavior style through the reward model of the behavior style;

[0141] According to the reward model of each behavior style, a feedback signal is generated for the training sample and the corresponding style matching degree, wherein the feedback signal indicates whether the training sample conforms to the behavior style;

[0142] For the reward model of each behavior style, the reward model is trained based on the behavior style label and feedback signal of the training sample to obtain a trained reward model corresponding to each behavior style.

[0143] In one embodiment, the reward model includes a classification network, and the step of “for each behavior style, predicting the style matching degree between the training sample and the behavior style by using the reward model of the behavior style” may include:

[0144] Through the classification network of the reward model of each behavior style, feature extraction is performed on the training samples to obtain behavior-related feature information of the training samples;

[0145] For each behavior style, the style matching degree with the behavior style is determined according to the behavior-related feature information through the classification network corresponding to the behavior style.

[0146] In one embodiment, the reward model further includes a reward network, and the step of "generating a feedback signal for the training sample and the corresponding style matching degree according to the reward model for each behavior style" may include: generating an initial feedback signal for the training sample according to the reward model for each behavior style;

[0147] The initial feedback signal is adjusted based on the style matching degree to obtain the feedback signal.

[0148] In one embodiment, after the above step of "for each behavior style reward model, training the reward model based on the behavior style label and feedback signal of the training sample to obtain a trained reward model corresponding to each behavior style", the reward model training method provided in the embodiment of the present application may include:

[0149] Obtaining a style decision model and an environmental data sample for each behavior style, wherein each style decision model includes a decision model and a post-training reward model;

[0150] Generate strategic behavior data based on the above environmental data samples through the above style decision model;

[0151] Generate a feedback signal for the strategy behavior data and the corresponding style matching degree according to the trained reward model of the style decision model;

[0152] The style decision model is trained according to the feedback signal to obtain a trained style decision model corresponding to each behavior style.

[0153] In one embodiment, each style decision model is a branch model of a strategy behavior generation model, and the strategy behavior generation model further includes a style selection network. The step of “generating strategy behavior data based on the environment data sample by using the style decision model” may include:

[0154] For each environment data sample, at least one target behavior style is selected through the above style selection network;

[0155] The strategy behavior data is generated based on the above-mentioned environment data samples through the decision model of the style decision model corresponding to the above-mentioned target behavior style.

[0156] In one embodiment, the trained style decision model corresponding to each behavior style is a branch model of the trained strategy behavior generation model. After the above step of "training the above style decision model according to the above feedback signal to obtain the trained style decision model corresponding to each behavior style", the reward model training method provided in the embodiment of the present application may further include:

[0157] Get the environment data of the agent's current environment;

[0158] Select at least one target behavior style through the style selection grid of the trained policy behavior generation model;

[0159] Generate strategic behavior data based on the above environmental data through the decision model corresponding to the above target behavior style;

[0160] Based on the above strategy behavior data, the above agent is controlled to perform corresponding behaviors in the above environment.

[0161] In one embodiment, the above step of “obtaining a sample data set” may include:

[0162] Get the initial sample data set;

[0163] Determine the behavior style corresponding to each training sample according to the feature information of each training sample in the initial sample data set;

[0164] A behavior style label corresponding to each training sample is generated according to the behavior style of each training sample to obtain the above sample data set.

[0165] As can be seen from the above, the embodiment of the present application obtains a sample data set, each training sample in the sample data set includes behavior association data and a corresponding behavior style label; obtains a reward model with the same structure for each behavior style; for each behavior style, determines the style matching degree between the training sample and the behavior style through the reward model of the behavior style; based on the reward model of each behavior style, generates a feedback signal for the matching degree between the training sample and the corresponding style, and the feedback signal indicates whether the training sample conforms to the behavior style; for the reward model of each behavior style, trains the reward model based on the behavior style label of the training sample and the feedback signal to obtain a trained reward model corresponding to each behavior style.

[0166] In the reward model training method provided in the present application, the reward model can determine the degree of match between the training samples and the behavioral style, and generate a feedback signal based on the degree of match between the training samples and the style. Therefore, the feedback signal can reflect whether the training samples meet the achievement goals corresponding to the reward model; it can also reflect whether the training samples meet the behavioral style, so that the trained reward model can reward strategic behaviors of specified behavioral styles and punish strategic behaviors of other behavioral styles. In the embodiment of the present application, the reward model is trained by training samples of different behavioral styles, so that the reward model corresponding to the behavioral style can be obtained. There is no need to set different reward functions for different behavioral styles, which can reduce the difficulty of obtaining reward models for different behavioral styles.

[0167] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0168] Optional, such as Figure 4 As shown, the computer device 500 further includes: a touch screen 503, a radio frequency circuit 504, an audio circuit 505, an input unit 506, and a power supply 507. The processor 501 is electrically connected to the touch screen 503, the radio frequency circuit 504, the audio circuit 505, the input unit 506, and the power supply 507, respectively. Those skilled in the art can understand that Figure 4 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0169] The touch display screen 503 can be used to display a graphical user interface and receive operation instructions generated by the user acting on the graphical user interface. The touch display screen 503 may include a display panel and a touch panel. Among them, the display panel may be used to display information input by the user or information provided to the user and various graphical user interfaces of computer equipment, and these graphical user interfaces may be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light emitting diode (OLED, Organic Light-Emitting Diode) and the like. The touch panel may be used to collect the user's touch operation on or near it (such as the user using any suitable object or attachment such as a finger, a stylus, etc. on the touch panel or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Optionally, the touch panel may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 501, and can receive the command sent by the processor 501 and execute it. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 501 to determine the type of touch event, and then the processor 501 provides a corresponding visual output on the display panel according to the type of touch event. In an embodiment of the present application, the touch panel and the display panel can be integrated into the touch display screen 503 to realize the input and output functions. However, in some embodiments, the touch panel and the touch panel can be used as two independent components to realize the input and output functions. That is, the touch display screen 503 can also be used as a part of the input unit 506 to realize the input function.

[0170] The radio frequency circuit 504 may be used to send and receive radio frequency signals, so as to establish wireless communication with a network device or other computer devices through wireless communication, and to send and receive signals between the network device or other computer devices.

[0171] The audio circuit 505 can be used to provide an audio interface between the user and the computer device through a speaker and a microphone. The audio circuit 505 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 505 and converted into audio data, and then the audio data is output to the processor 501 for processing, and then sent to another computer device through the radio frequency circuit 504, or the audio data is output to the memory 502 for further processing. The audio circuit 505 may also include an earphone jack to provide communication between an external headset and the computer device.

[0172] The input unit 506 may be used to receive input numbers, character information or user feature information (such as fingerprint, iris, facial information, etc.), and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0173] The power supply 507 is used to supply power to various components of the computer device 500. Optionally, the power supply 507 can be logically connected to the processor 501 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The power supply 507 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0174] although Figure 4 Not shown, the computer device 500 may also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.

[0175] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0176] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0177] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of computer programs are stored, and the computer program can be loaded by a processor to execute the steps in any virtual item marking method provided in the embodiment of the present application. For example, the computer program can execute the following steps:

[0178] Obtain a sample data set, wherein each training sample in the sample data set includes behavior association data and a corresponding behavior style label;

[0179] Get a reward model with the same structure for each behavior style;

[0180] For each behavior style, determining the style matching degree between the training sample and the behavior style through the reward model of the behavior style;

[0181] According to the reward model of each behavior style, a feedback signal is generated for the training sample and the corresponding style matching degree, wherein the feedback signal indicates whether the training sample conforms to the behavior style;

[0182] For the reward model of each behavior style, the reward model is trained based on the behavior style label and feedback signal of the training sample to obtain a trained reward model corresponding to each behavior style.

[0183] In one embodiment, the reward model includes a classification network, and the step of “for each behavior style, predicting the style matching degree between the training sample and the behavior style by using the reward model of the behavior style” may include:

[0184] Through the classification network of the reward model of each behavior style, feature extraction is performed on the training samples to obtain behavior-related feature information of the training samples;

[0185] For each behavior style, the style matching degree with the behavior style is determined according to the behavior-related feature information through the classification network corresponding to the behavior style.

[0186] In one embodiment, the reward model further includes a reward network, and the step of "generating a feedback signal for the training sample and the corresponding style matching degree according to the reward model for each behavior style" may include: generating an initial feedback signal for the training sample according to the reward model for each behavior style;

[0187] The initial feedback signal is adjusted based on the style matching degree to obtain the feedback signal.

[0188] In one embodiment, after the above step of "for each behavior style reward model, training the reward model based on the behavior style label and feedback signal of the training sample to obtain a trained reward model corresponding to each behavior style", the reward model training method provided in the embodiment of the present application may include:

[0189] Obtaining a style decision model and an environmental data sample for each behavior style, wherein each style decision model includes a decision model and a post-training reward model;

[0190] Generate strategic behavior data based on the above environmental data samples through the above style decision model;

[0191] Generate a feedback signal for the strategy behavior data and the corresponding style matching degree according to the trained reward model of the style decision model;

[0192] The style decision model is trained according to the feedback signal to obtain a trained style decision model corresponding to each behavior style.

[0193] In one embodiment, each style decision model is a branch model of a strategy behavior generation model, and the strategy behavior generation model further includes a style selection network. The step of “generating strategy behavior data based on the environment data sample by using the style decision model” may include:

[0194] For each environment data sample, at least one target behavior style is selected through the above style selection network;

[0195] The strategy behavior data is generated based on the above-mentioned environment data samples through the decision model of the style decision model corresponding to the above-mentioned target behavior style.

[0196] In one embodiment, the trained style decision model corresponding to each behavior style is a branch model of the trained strategy behavior generation model. After the above step of "training the above style decision model according to the above feedback signal to obtain the trained style decision model corresponding to each behavior style", the reward model training method provided in the embodiment of the present application may further include:

[0197] Get the environment data of the agent's current environment;

[0198] Select at least one target behavior style through the style selection grid of the trained policy behavior generation model;

[0199] Generate strategic behavior data based on the above environmental data through the decision model corresponding to the above target behavior style;

[0200] Based on the above strategy behavior data, the above agent is controlled to perform corresponding behaviors in the above environment.

[0201] In one embodiment, the above step of “obtaining a sample data set” may include:

[0202] Get the initial sample data set;

[0203] Determine the behavior style corresponding to each training sample according to the feature information of each training sample in the initial sample data set;

[0204] A behavior style label corresponding to each training sample is generated according to the behavior style of each training sample to obtain the above sample data set.

[0205] As can be seen from the above, the embodiment of the present application obtains a sample data set, each training sample in the sample data set includes behavior association data and a corresponding behavior style label; obtains a reward model with the same structure for each behavior style; for each behavior style, determines the style matching degree between the training sample and the behavior style through the reward model of the behavior style; based on the reward model of each behavior style, generates a feedback signal for the matching degree between the training sample and the corresponding style, and the feedback signal indicates whether the training sample conforms to the behavior style; for the reward model of each behavior style, trains the reward model based on the behavior style label of the training sample and the feedback signal to obtain a trained reward model corresponding to each behavior style.

[0206] In the reward model training method provided in the present application, the reward model can determine the degree of match between the training samples and the behavioral style, and generate a feedback signal based on the degree of match between the training samples and the style. Therefore, the feedback signal can reflect whether the training samples meet the achievement goals corresponding to the reward model; it can also reflect whether the training samples meet the behavioral style, so that the trained reward model can reward strategic behaviors of specified behavioral styles and punish strategic behaviors of other behavioral styles. In the embodiment of the present application, the reward model is trained by training samples of different behavioral styles, so that the reward model corresponding to the behavioral style can be obtained. There is no need to set different reward functions for different behavioral styles, which can reduce the difficulty of obtaining reward models for different behavioral styles.

[0207] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0208] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0209] The above is a detailed introduction to a reward model training method, device, computer equipment and computer storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A reward model training method, characterized in that: include: Acquire a sample data set, wherein each training sample in the sample data set includes behavior association data and a corresponding behavior style label; Get a reward model with the same structure for each behavior style; For each behavior style, determining the style matching degree between the training sample and the behavior style through the reward model of the behavior style; According to the reward model of each behavior style, a feedback signal is generated for the training sample and the corresponding style matching degree, wherein the feedback signal indicates whether the training sample conforms to the behavior style; For the reward model of each behavior style, the reward model is trained based on the behavior style label and feedback signal of the training sample to obtain a trained reward model corresponding to each behavior style.

2. The method according to claim 1, characterized in that The reward model includes a classification network, and for each behavior style, predicting the style matching degree between the training sample and the behavior style through the reward model of the behavior style includes: Extracting features of the training samples through a classification network of a reward model for each behavior style to obtain behavior-related feature information of the training samples; For each behavior style, a style matching degree with the behavior style is determined according to the behavior-related feature information through a classification network corresponding to the behavior style.

3. The method according to claim 2, characterized in that The reward model further includes a reward network. The reward model for each behavior style generates a feedback signal for the training sample and the corresponding style matching degree, including: An initial feedback signal for the training sample according to the reward model of each behavior style; The initial feedback signal is adjusted based on the style matching degree to obtain the feedback signal.

4. The method according to claim 1, characterized in that: The reward model for each behavior style is trained based on the behavior style label and feedback signal of the training sample to obtain a trained reward model corresponding to each behavior style, and the method further includes: Obtaining a style decision model and an environmental data sample for each behavior style, wherein each style decision model includes a decision model and a post-training reward model; Generate strategic behavior data based on the environmental data sample by the style decision model; generating a feedback signal for the strategy behavior data and the corresponding style matching degree according to a trained reward model of the style decision model; The style decision model is trained according to the feedback signal to obtain a trained style decision model corresponding to each behavior style.

5. The method according to claim 4, characterized in that Each style decision model is a branch model of the strategy behavior generation model, and the strategy behavior generation model also includes a style selection network. The strategy behavior data is generated based on the environment data sample by the style decision model, including: For each environment data sample, selecting at least one target behavior style through the style selection network; The strategy behavior data is generated based on the environment data sample through the decision model of the style decision model corresponding to the target behavior style.

6. The method according to claim 5, characterized in that The trained style decision model corresponding to each behavior style is a branch model of the trained strategy behavior generation model. After the style decision model is trained according to the feedback signal to obtain the trained style decision model corresponding to each behavior style, the method further includes: Get the environment data of the agent's current environment; Selecting at least one target behavior style through the style selection grid of the trained policy behavior generation model; Generate strategic behavior data according to the environmental data through a decision model corresponding to the target behavior style; The agent is controlled to perform corresponding behaviors in the environment based on the strategic behavior data.

7. The method according to any one of claims 1 to 6, characterized in that: The obtaining of the sample data set comprises: Get the initial sample data set; Determining a behavior style corresponding to each training sample according to the feature information of each training sample in the initial sample data set; A behavior style label corresponding to each training sample is generated according to the behavior style of each training sample to obtain the sample data set.

8. A reward model training device, characterized in that: include: A data acquisition unit, used to acquire a sample data set, wherein each training sample in the sample data set includes behavior association data and a corresponding behavior style label; A model acquisition unit, used to acquire a reward model with the same structure for each behavior style; a determination unit, configured to determine, for each behavior style, a style matching degree between the training sample and the behavior style by using a reward model for the behavior style; A generating unit, configured to generate a feedback signal for the training sample and the corresponding style matching degree according to the reward model of each behavior style, wherein the feedback signal indicates whether the training sample conforms to the behavior style; The training unit is used to train the reward model for each behavior style based on the behavior style label and feedback signal of the training sample to obtain a trained reward model corresponding to each behavior style.

9. A computer device, characterized in that: It comprises a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the reward model training method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, which is loaded by a processor to execute the reward model training method according to any one of claims 1 to 7.

Images