Model training method, object control method, device, medium and equipment

Through the self-training method of deep reinforcement learning model, the problem of insufficient user data in the uninstalled game was solved, and the model self-training and diversity of decision-making styles were improved. It is suitable for different game scenarios, reducing costs and improving efficiency.

CN114917586BActive Publication Date: 2024-11-26BEIJING ZITIAO NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210621933.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-01
Publication Date
2024-11-26
Estimated Expiration
2042-06-01

AI Technical Summary

Technical Problem

The existing game strategy cannot be effectively extended to all game types, especially games that have not been launched or have a shorter time to go online. Due to insufficient human user data, it is difficult to meet the training requirements and even the user data cannot be obtained.

Method used

Through the training method of deep reinforcement learning model, we can obtain the interactive sequence of virtual objects in the virtual environment, determine the training reward weight parameters and return values, calculate the target loss, conduct model training, and realize the model self-training without real user data, which is suitable for different Game scene.

Benefits of technology

Reduce dependence on real user data, avoid policy spatial constraints, improve the diversity of model decision styles, and is suitable for different decision styles. There is no need to train different models for different styles, reducing costs and improving training efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114917586B_ABST
    Figure CN114917586B_ABST
Patent Text Reader

Abstract

This disclosure relates to a training method for a model, a control method for an object, an apparatus, a medium, and a device. The method includes: acquiring an interaction sequence generated by the interaction between a first virtual object and a second virtual object in a virtual environment, the interaction sequence including multiple sampled data; acquiring a training reward weight parameter corresponding to each interaction sequence, the training reward weight parameter corresponding to a decision style type for training a deep reinforcement learning model; determining a target reward value corresponding to each sampled data based on the training reward weight parameter corresponding to the interaction sequence and the reward value in the interaction sequence; determining a target loss for training the deep reinforcement learning model based on the predicted value of the action determined based on the state characteristics and decision action of each sampled data, and the target reward value corresponding to the sampled data; and training the deep reinforcement learning model based on the target loss.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a model training method, an object control method, a device, a medium and equipment. Background Art

[0002] However, with the development of the computer industry and the game industry, more and more game types have emerged. Game strategies based on the existing learning paradigm cannot be effectively extended to all game types. In related technologies, game user data can usually be used to guide and train virtual object AI to generate strategies. However, through the above scheme, the strategies learned based on game user data cannot be separated from the scope of human users. For some games that have not been launched or have been online for a short time, it is difficult to meet the training requirements due to the small amount of human user data, and even in some cases, user data cannot be obtained. Summary of the invention

[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, the present disclosure provides a method for training a deep reinforcement learning model, the method comprising:

[0005] Acquire an interaction sequence generated by the interaction between a first virtual object and a second virtual object in a virtual environment, wherein the interaction sequence includes a plurality of sampling data, each of the sampling data includes a reward value obtained by the first virtual object performing a decision action under the state characteristics sampled by the virtual environment, the first virtual object is controlled based on a trained deep reinforcement learning model, and the second virtual object is controlled based on a historical deep reinforcement learning model corresponding to the trained deep reinforcement learning model;

[0006] Obtaining a training reward weight parameter corresponding to each of the interaction sequences, wherein the training reward weight parameter corresponds to a decision style type of the training deep reinforcement learning model;

[0007] Determining a target reward value corresponding to each of the sampled data according to a training reward weight parameter corresponding to the interaction sequence and a reward value in the interaction sequence;

[0008] Determine the target loss of the training deep reinforcement learning model according to the estimated action value determined based on the state characteristics and decision actions of each of the sampled data, and the target reward value corresponding to the sampled data;

[0009] The training deep reinforcement learning model is trained based on the target loss.

[0010] In a second aspect, the present disclosure provides a method for controlling a virtual object, the method comprising:

[0011] Determine an interactive virtual object controlled by a user that matches a target virtual object in a target game, wherein the target virtual object is controlled based on a target deep reinforcement learning model, wherein the target deep reinforcement learning model is trained based on the training method for the deep reinforcement learning model described in the first aspect;

[0012] Determining a target reward weight parameter of the target virtual object from a plurality of reward weight parameters according to a behavior type of the target virtual object in the target game, wherein the target reward weight parameter corresponds to a decision style type of the target deep reinforcement learning model;

[0013] Sampling the interaction between the target virtual object and the interactive virtual object in the virtual environment to obtain target state features;

[0014] Based on the target state characteristics, the target reward weight parameters and the target deep reinforcement learning model, a target decision action of the target virtual object under the target state characteristics is determined to control the operation of the target virtual object based on the target decision action.

[0015] In a third aspect, the present disclosure provides a training device for a deep reinforcement learning model, the device comprising:

[0016] A first acquisition module is used to acquire an interaction sequence generated by the interaction between a first virtual object and a second virtual object in a virtual environment, wherein the interaction sequence includes a plurality of sampling data, each of the sampling data includes a reward value obtained by the first virtual object performing a decision action under a state feature sampled by the virtual environment, the first virtual object is controlled based on a trained deep reinforcement learning model, and the second virtual object is controlled based on a historical deep reinforcement learning model corresponding to the trained deep reinforcement learning model;

[0017] A second acquisition module is used to acquire a training reward weight parameter corresponding to each of the interaction sequences, wherein the training reward weight parameter corresponds to a decision style type of the training deep reinforcement learning model;

[0018] A first determination module, configured to determine a target reward value corresponding to each of the sampled data according to a training reward weight parameter corresponding to the interaction sequence and a reward value in the interaction sequence;

[0019] A second determination module is used to determine the target loss of the training deep reinforcement learning model according to the action value estimate determined based on the state characteristics and decision actions of each of the sampled data, and the target reward value corresponding to the sampled data;

[0020] A training module is used to train the training deep reinforcement learning model based on the target loss.

[0021] In a fourth aspect, the present disclosure provides a control device for a virtual object, the device comprising:

[0022] A third determination module is used to determine an interactive virtual object controlled by a user that matches a target virtual object in a target game, wherein the target virtual object is controlled based on a target deep reinforcement learning model, wherein the target deep reinforcement learning model is trained based on the training method of the deep reinforcement learning model described in the second aspect;

[0023] a fourth determination module, configured to determine a target reward weight parameter of the target virtual object from a plurality of reward weight parameters according to a behavior type of the target virtual object in the target game, wherein the target reward weight parameter corresponds to a decision style type of the target deep reinforcement learning model;

[0024] A sampling module, used for sampling the interaction between the target virtual object and the interactive virtual object in the virtual environment to obtain target state characteristics;

[0025] A control module is used to determine a target decision action of the target virtual object under the target state characteristics based on the target state characteristics, the target reward weight parameters and the target deep reinforcement learning model, so as to control the operation of the target virtual object based on the target decision action.

[0026] In a fifth aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect or the second aspect.

[0027] In a sixth aspect, the present disclosure provides an electronic device, including:

[0028] a storage device having a computer program stored thereon;

[0029] A processing device is used to execute the computer program in the storage device to implement the steps of the method described in the first aspect or the second aspect.

[0030] In the above technical solution, an interaction sequence generated by the interaction between the first virtual object and the second virtual object in the virtual environment is obtained, and then the target reward value corresponding to each sampled data can be determined according to the training reward weight parameter corresponding to the interaction sequence and the reward value in the interaction sequence, and the loss calculation is further performed based on the target reward value to train the training deep reinforcement learning model. Therefore, through the above technical solution, model training can be performed based on the interaction sequence generated by the interaction between the virtual objects controlled by the current training deep reinforcement learning model and its corresponding historical deep reinforcement learning model, so that training can be performed directly based on the interaction data generated by the model, without the need for real user operation data, reducing the dependence on real user data during model training, which can avoid the constraints of real user strategies on the exploration strategy space of the model, and can also be applied to the training scenarios of models corresponding to games that have not been launched or have been online for a short time. Moreover, for each interaction sequence, it has a corresponding training reward weight parameter, and each training reward weight parameter corresponds to a decision style type of a training deep reinforcement learning model, so that in the training process of the training deep reinforcement learning model, the training reward weight parameters corresponding to different interaction sequences are used to guide the training deep reinforcement learning model to differentiate into different decision strategy styles, so as to improve the diversity of decision styles in the trained model, and can be applicable to battle control under different styles based on the same model, without the need to train different models for different decision styles, further reducing the training cost and improving the training efficiency of the model.

[0031] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:

[0033] Figure 1 A flowchart of a training method for a deep reinforcement learning model provided according to an embodiment of the present disclosure;

[0034] Figure 2 A schematic diagram of the structure of a training deep reinforcement learning model provided according to an embodiment of the present disclosure;

[0035] Figure 3 A schematic diagram of the structure of a training deep reinforcement learning model provided according to another embodiment of the present disclosure;

[0036] Figure 4A flowchart of a method for controlling a virtual object according to an embodiment of the present disclosure;

[0037] Figure 5 A block diagram of a training device for a deep reinforcement learning model provided according to an embodiment of the present disclosure;

[0038] Figure 6 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0039] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0040] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0041] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0042] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0043] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0044] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0045] All actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0046] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0047] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0048] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0049] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0050] At the same time, it is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0051] Figure 1 As shown, it is a flowchart of a training method of a deep reinforcement learning model provided according to an embodiment of the present disclosure, as shown Figure 1 As shown, the method may include:

[0052] In step 11, an interaction sequence generated by the interaction between the first virtual object and the second virtual object in the virtual environment is obtained, wherein the interaction sequence includes a plurality of sampling data, each sampling data includes a reward value obtained by the first virtual object performing a decision action when the virtual environment is in a sampled state feature, the first virtual object is controlled based on a trained deep reinforcement learning model, and the second virtual object is controlled based on a historical deep reinforcement learning model corresponding to the trained deep reinforcement learning model.

[0053] Among them, the deep reinforcement learning model combines the perception ability of deep learning with the decision-making ability of reinforcement learning. It obtains a high-dimensional observation by interacting with the environment at each sampling moment, and uses deep learning methods to perceive the observation to obtain the specific state feature representation of the observation; then, based on the reinforcement learning method, the decision action under the state feature can be determined. For example, the value function (state value function) of each state and the value function (action value function) of the state-action pair can be evaluated based on the expected return, and the decision strategy is improved based on these two value functions. The decision strategy is used to map the current state to the corresponding decision action. Further, the environment will respond to this decision action and obtain the next observation until the final state, completing a round.

[0054] For example, the virtual environment can be a virtual scene environment generated by a computer, such as a game scene. For example, multimedia data used to interact with the user is rendered, so that the multimedia data can be rendered and displayed as a game scene. The virtual environment provides a multimedia virtual world. The user can control the virtual object AI action through the controls on the operation interface, or directly control the operable virtual objects in the virtual environment, and observe the objects, characters, scenery, etc. in the virtual environment from the perspective of the virtual object, and interact with other virtual objects in the virtual environment through the virtual object. The virtual object can be a virtual image used to simulate the user in the virtual environment, which can be a human image or other animal image, etc.

[0055] In this embodiment, the first virtual object and the second virtual object may be in a virtual environment and controlled by different deep reinforcement learning models. For example, in a one-to-one virtual game environment, a first virtual object O1 and a second virtual object O2 may be included. The attack output by O1 may cause damage to O2, so that O1 wins a reward. O1 may also dodge the attack output by O2 to avoid damage to O1 by the attack output by O2. Similarly, the operation of O2 is similar to that of O1, and the two interact until the round ends.

[0056] For example, the historical deep reinforcement learning model can be any model obtained during the training process of the training deep reinforcement learning model. In order to further ensure the training efficiency and accuracy of the model, one of the models corresponding to the most recent updates can be selected as the historical deep reinforcement learning model. For example, the most recent historical model corresponding to the training deep reinforcement learning model is selected as the historical deep reinforcement learning model, that is, the training deep reinforcement learning model is directly obtained after the parameters of the historical deep reinforcement learning model are updated. In this way, the similarity between the historical deep reinforcement learning model and the training deep reinforcement learning model can be guaranteed to a certain extent, and the game difficulty when the first virtual object and the second virtual object interact can be increased, thereby improving the effectiveness of the interaction sequence.

[0057] In step 12, a training reward weight parameter corresponding to each interaction sequence is obtained, wherein the training reward weight parameter corresponds to a decision style type of the training deep reinforcement learning model.

[0058] Among them, in the training process of the deep reinforcement learning model, it is usually necessary to set a corresponding reward function to evaluate the value of the intelligent agent performing a certain action in the current state, which is usually calculated based on the reward function. The reward function can include multiple reward items. Taking the virtual object as an AI in a battle game as an example, the reward item can include the reward obtained when the first virtual object finally wins, the penalty for the first virtual object being hurt at the current moment, the reward for the first virtual object causing damage to the enemy at the current moment, and the reward for the remaining friendly skills of the first virtual object after the game ends, etc. The reward items and reward functions can be set according to the specific application scenario, and this disclosure does not limit this.

[0059] In the disclosed embodiment, there may be multiple reward weight parameters, and the training reward weight parameter is one of the multiple reward weight parameters. Each reward weight parameter includes a weight corresponding to each reward item in the reward function, so that the decision style can be differentiated by adjusting the weight of the reward item. When sampling the interaction sequence, the corresponding reward weight parameter will be pre-set, and the corresponding reward weight parameter when generating the interaction sequence can be used as the training reward weight parameter corresponding to the interaction sequence.

[0060] In step 13, the target reward value corresponding to each sampled data is determined according to the training reward weight parameter corresponding to the interaction sequence and the reward value in the interaction sequence.

[0061] For example, the target reward value can be the value obtained by weighting each reward item based on the training reward weight parameter. For example, the value of each reward item in the reward function can be determined based on the reward value in the interaction sequence. For example, for the reward items that exist in the reward value corresponding to the sampled data, such as the reward value that includes the reward reward for causing damage to the enemy at the current moment, the value of the corresponding reward item is the reward reward. For the reward items that do not exist in the reward value corresponding to the sampled data, such as the reward obtained when the victory is finally achieved, it will only appear in the sampled data at the end of the round. If the reward item is not included in the reward value corresponding to the sampling number, it can be determined that the value of the reward item is 0.

[0062] Based on the example above, the reward function for fighting games can be expressed as follows:

[0063] r=w1*r win +w2*r my_hp +w3*r your_hp +w4*rf s +...

[0064] Among them, r win Used to indicate the reward for the final victory; my_hp Used to indicate the penalty for taking damage at the current moment, r your_hp Used to indicate the reward for causing damage to the enemy at the current moment, r fs It is used to indicate the remaining friendly skill rewards after the game ends, w1, w2, w3 and w4 respectively represent the weights corresponding to each reward item, that is, the multiple weight values ​​form a set of reward weight parameters. Among them, the type and number of reward items can be set according to the actual application scenario, and the above is only an exemplary description and is not limited.

[0065] In step 14, the target loss for training the deep reinforcement learning model is determined based on the estimated action value determined based on the state characteristics and decision actions of each sampled data, and the target reward value corresponding to the sampled data.

[0066] For example, a deep reinforcement learning model may include a policy network and a value network, which may be implemented through a neural network structure. The policy network is used to generate decision actions based on the current state characteristics and the value determined by the value network, and the value network is used to estimate the reward of the action under the current state characteristics, that is, the state characteristics and actions are input into the value network to obtain the state-action value, that is, the action value estimate. Among them, the action value estimate is used to represent the expected return value of executing the action under the current state characteristics, and the target return value can be used to represent the return value that can be obtained by executing the action under the state characteristics determined in the actual data, so that error calculation can be performed based on the above two, such as the mean square error corresponding to the action value estimate and the target return value can be used as the target loss of the training deep reinforcement learning model.

[0067] In step 15, the deep reinforcement learning model is trained based on the target loss.

[0068] Among them, when the number of training times of the deep reinforcement learning model does not reach the target number, the parameters of the training deep reinforcement learning model can be updated based on the target loss. As described above, the deep reinforcement learning model may include a policy network and a value network. In this embodiment, the gradient information corresponding to the policy network and the value network can be determined respectively based on the target loss, so as to update each parameter in the training deep reinforcement learning model based on the gradient information, thereby ensuring the accuracy of the gradient direction of the parameter update and improving the training update efficiency of the model. Among them, the gradient update method commonly used in the field can be used for updating, and this disclosure does not limit this.

[0069] In the above technical solution, an interaction sequence generated by the interaction between the first virtual object and the second virtual object in the virtual environment is obtained, and then the target reward value corresponding to each sampled data can be determined according to the training reward weight parameter corresponding to the interaction sequence and the reward value in the interaction sequence, and the loss calculation is further performed based on the target reward value to train the training deep reinforcement learning model. Therefore, through the above technical solution, model training can be performed based on the interaction sequence generated by the interaction between the virtual objects controlled by the current training deep reinforcement learning model and its corresponding historical deep reinforcement learning model, so that training can be performed directly based on the interaction data generated by the model, without the need for real user operation data, reducing the dependence on real user data during model training, which can avoid the constraints of real user strategies on the exploration strategy space of the model, and can also be applied to the training scenarios of models corresponding to games that have not been launched or have been online for a short time. Moreover, for each interaction sequence, it has a corresponding training reward weight parameter, and each training reward weight parameter corresponds to a decision style type of a training deep reinforcement learning model, so that in the training process of the training deep reinforcement learning model, the training reward weight parameters corresponding to different interaction sequences are used to guide the training deep reinforcement learning model to differentiate into different decision strategy styles, so as to improve the diversity of decision styles in the trained model, and can be applicable to battle control under different styles based on the same model, without the need to train different models for different decision styles, further reducing the training cost and improving the training efficiency of the model.

[0070] In a possible embodiment, the output layer of the training deep reinforcement learning model includes multiple output branches, and the output branches correspond one-to-one to multiple preset reward weight parameters;

[0071] Accordingly, an exemplary implementation of training the deep reinforcement learning model based on the target loss may include:

[0072] Based on the target loss, a parameter in the trained deep reinforcement learning model corresponding to an output branch corresponding to the training reward weight parameter is updated, and the training reward weight parameter is one of the multiple reward weight parameters.

[0073] In this embodiment, different interaction sequences may be generated based on different reward weight parameters for decision interaction. Therefore, in order to further improve the effective distinction of reward items in the reward function in the early stage of model training, the training reward weight parameters are used as the input of the model in the embodiment of the present disclosure. Figure 2As shown, the output layer of the training deep reinforcement learning model can be set to multiple output branches, that is, a multi-head neural network, and each output branch corresponds to a training reward weight parameter one by one, that is, the training deep reinforcement learning model will output multiple results, each result corresponds to an action output under a decision-making style type.

[0074] Based on this, after the target loss is determined in this embodiment, the parameters corresponding to the output branch corresponding to the training reward weight parameter can be updated based on the target loss, wherein the method of updating the parameters is similar to that described above and will not be repeated here.

[0075] Therefore, through the above technical solution, a corresponding output branch can be set for each training reward weight parameter, that is, each decision style, so as to obtain the output action under the decision style corresponding to the training reward weight parameter. At the same time, during the training process of the deep reinforcement learning model, the parameters of the output branch corresponding to the reward weight parameter in the training deep reinforcement learning model can be updated in a targeted manner based on the interaction sequence generated under the same reward weight parameter, so as to ensure the accuracy of parameter update under each decision style type, and guide the training deep reinforcement learning model to differentiate into different decision strategies through the training reward weight parameter corresponding to the interaction sequence, and at the same time avoid the influence between the training data under different decision style types, so as to ensure the accuracy of model update.

[0076] In a possible embodiment, the interaction sequence may be generated in the following manner:

[0077] Determine the training reward weight parameters for training deep reinforcement learning models.

[0078] Among them, a variety of reward weight parameters can be pre-set according to human experience. Continuing with the example described above, the reward function contains the above 4 reward items, and the reward weight parameters can be set as follows:

[0079] For offensive type: there is no penalty after being damaged by the enemy, so the AI ​​is encouraged to cause more damage than the enemy as much as possible while ensuring victory. The reward weight parameter is set as follows:

[0080] w1=1, w2=0, w3=1, w4=1

[0081] For defensive type: No reward will be given after causing damage to the enemy, which will encourage the AI ​​to avoid the enemy's attack as much as possible and not actively launch an attack. The reward weight parameter is set as follows:

[0082] w1=1, w2=1, w3=0, w4=1

[0083] For balance: the AI ​​will be punished after being damaged by the enemy, and will also be rewarded after damaging the enemy, so as to encourage the AI ​​to cause more damage while avoiding enemy attacks. The reward weight parameters are set as follows:

[0084] w1=1, w2=1, w3=1, w4=1

[0085] Therefore, before generating the interaction sequence, the training reward parameters of the training deep reinforcement learning model can be randomly selected, that is, any one of the three types shown above can be selected to determine the decision style type corresponding to the training deep reinforcement learning model in this interaction round. For example, the determined training reward weight parameters are w1=1, w2=0, w3=1, and w4=1, which means that in this interaction round, the training deep reinforcement learning model determines the decision action and reward to ensure victory while causing as much damage as possible to the enemy.

[0086] Afterwards, the interaction between the first virtual object and the second virtual object in the virtual environment is sampled to obtain a state feature corresponding to the first virtual object.

[0087] Among them, sampling can be performed in the virtual environment based on the game simulator at each preset time interval to obtain the state characteristics corresponding to the first virtual object. For example, the state characteristics may include the position, posture, health, etc. of the first virtual object, the position, posture, health, etc. of the second virtual object, and the reward information and penalty information in the environment state. It should be noted that the characteristic information specifically included in the state characteristics can be set according to the actual application scenario, and this disclosure does not limit this.

[0088] Afterwards, based on the state characteristics, the training reward weight parameters and the training deep reinforcement learning model, the decision action of the first virtual object under the state characteristics is determined to control the operation of the first virtual object based on the decision action, and return to the step of sampling the interaction between the first virtual object and the second virtual object in the virtual environment to obtain the state characteristics corresponding to the first virtual object until the interaction round ends.

[0089] The state feature and the training reward weight parameter can be input into the training deep reinforcement learning model so that the training deep reinforcement learning model outputs a decision action, thereby controlling the operation of the first virtual object. The above steps can then be repeated to perform sampling again, and further generate a decision action based on the state feature obtained by sampling, and control the first virtual object step by step until the interaction round ends, that is, the first virtual object and the second virtual object obtain a battle result.

[0090] The sampled data obtained in the interaction round are sorted in the order of sampling time to obtain the interaction sequence, and the interaction sequence is associated with the training reward weight parameter, wherein each sampled data includes a reward value obtained by executing a decision action when the virtual environment is in a state characteristic.

[0091] Afterwards, the various sampled data sampled in the interaction round can be arranged in sequence to form an interaction sequence, and the interaction sequence can be associated with its corresponding training reward weight parameters, so that in the subsequent process of training deep reinforcement learning based on the interaction sequence, after obtaining the interaction sequence, the training reward weight parameters corresponding to the interaction sequence can be directly determined, thereby ensuring the accuracy of the training process of the deep reinforcement learning model, facilitating the constraint of the decision-making style differentiation of the deep reinforcement learning model based on the training reward weight parameters, and ensuring the update accuracy of the deep reinforcement learning model.

[0092] In a possible embodiment, the output layer of the training deep reinforcement learning model includes multiple output branches, and the output branches correspond one-to-one to multiple preset reward weight parameters;

[0093] Accordingly, an exemplary implementation of determining the decision action of the first virtual object under the state feature based on the state feature, the training reward weight parameter and the training deep reinforcement learning model may include:

[0094] The state features and the training reward weight parameters are input into the training deep reinforcement learning model, so that the feature layer of the training deep reinforcement learning model performs feature extraction on the state features and the training reward weight parameters, and sub-outputs corresponding to multiple output branches of the output layer are obtained based on the extracted features, and the decision action is determined according to the sub-outputs corresponding to the output branches corresponding to the training reward weight parameters in the training deep reinforcement learning model.

[0095] For example, Figure 2 As shown, the feature layer of the training deep reinforcement learning model can be realized by a neural network, and the state features and training reward weight parameters can be input into the training deep reinforcement learning model, so as to extract and encode the state features and training reward weight parameters, and perform subsequent processing based on the extracted features, for example, the action value and state value under the current state features can be estimated based on the extracted features, and the sub-output corresponding to each output branch of the output layer can be further determined based on the action value, state value and decision strategy. Figure 2As shown, the output branch F1 corresponds to the offensive reward weight parameter, the output branch F2 corresponds to the defensive reward weight parameter, and the output branch F3 corresponds to the balanced reward weight parameter, and three actions can be output respectively in the above manner. For example, the sub-output corresponding to the output branch F1 is used to represent the action to be performed when the state feature is in the corresponding offensive decision-making style type.

[0096] Furthermore, the decision action can be determined according to the sub-output corresponding to the output branch corresponding to the training reward weight parameter. For example, if the training reward weight parameter corresponding to this interaction round is initially determined to be the reward weight parameter corresponding to the offensive type, then the action corresponding to the sub-output corresponding to the output branch F1 can be determined as the decision action.

[0097] Therefore, through the above technical scheme, different output branches are formed for each reward weight parameter in the output layer of the training deep reinforcement learning model, so that the corresponding actions under different decision-making style types can be output respectively, and then the final decision action is determined in combination with the decision style type corresponding to this interaction round, thereby ensuring the accuracy of the decision action determination, and at the same time obtaining the interaction sequence corresponding to the decision style type as training data, providing accurate and reliable data support for training deep reinforcement learning models for strategy style differentiation.

[0098] In actual application scenarios, in order to obtain as diverse decision-making strategy styles as possible, a large number of reward weight parameter combinations are often obtained based on some prior knowledge of human users. However, as the number of decision-making style types increases, higher requirements are placed on the training of networks with multiple output branches. Based on this, the present disclosure also provides the following implementation methods.

[0099] In a possible embodiment, another exemplary implementation manner of determining a decision action of the first virtual object under the state feature based on the state feature, the training reward weight parameter, and the training deep reinforcement learning model may include:

[0100] Inputting the state feature and the training reward weight parameter into the training deep reinforcement learning model, and determining the decision action according to the output of the training deep reinforcement learning model;

[0101] The training deep reinforcement learning model 30 includes a neural network feature layer 31, a type feature layer 32 and an attention layer 33, such as Figure 3 shown.

[0102] The neural network feature layer is used to determine the state feature vector corresponding to the state feature and the parameter feature vector corresponding to the training reward weight parameter; the type feature layer is used to determine the type feature vector corresponding to each candidate type under the type feature layer based on the state feature vector and the parameter feature vector, wherein the candidate types correspond one-to-one to the hidden layer decision styles; the attention layer is used to fuse the type feature vectors corresponding to each type according to the parameter feature vector based on the attention mechanism, and determine the output of the training deep reinforcement learning model according to the fusion result.

[0103] Among them, the type feature layer can contain multiple candidate types, and the candidate types correspond to the hidden decision styles one by one, that is, each candidate type corresponds to an implicit representation of the decision style. Then, the type feature vector under the corresponding hidden decision style can be determined through the type feature layer, and the type feature vector represents the action feature under the hidden decision style. Then, the multiple action features are fused through the attention layer to represent more decision style types based on multiple hidden decision styles.

[0104] Among them, based on the attention mechanism, attention can be focused on important information, and some insignificant information extraction will be suppressed. In deep learning, the attention mechanism can be described as a mapping of query vectors (Query) and key-value pair vectors (Key-Value) to attention weights. For example, in this embodiment, each type of feature vector can be used as a key vector and a value vector in a key-value pair vector, and the training reward weight parameter can be used as a query vector for attention processing. Specifically, the query vector can be inner-producted with the key vector in the key-value pair vector to obtain the similarity between the query vector and the key vector, and after the inner-product operation, scaling operations and normalized exponential operations such as softmax function operations are performed in sequence to obtain the attention weight between the query vector and the key vector, and finally the value vector in the key-value pair vector is further inner-producted by the attention weight to obtain the final attention calculation result, that is, the attention weight is determined based on the training reward weight parameter and each type of feature vector, and each type of feature vector is weighted summed based on the attention weight, so as to fuse the type feature vectors, and obtain the fusion result, that is, the fusion feature vector obtained after the weighted summation.

[0105] Therefore, through the above technical scheme, a type feature layer can be set in the training deep reinforcement learning model, and multiple hidden decision styles can be used as implicit features in the model. By fusing the implicit features to represent more decision style features, the efficiency of training a deep reinforcement learning model containing too many decision style types based on multiple output branches can be avoided, the training and learning difficulty of the model training can be reduced, and it is also convenient for the expansion of decision style types.

[0106] In a possible embodiment, the first virtual object and the second virtual object have different character types, and the historical deep reinforcement learning model and the training deep reinforcement learning model respectively correspond to different training reward weight parameters. For example, when a virtual object is playing a game, it can correspond to multiple optional character types, for example, each hero selectable by the user in the game scene can be a character type.

[0107] Accordingly, the character type corresponding to the second virtual object and the training reward weight parameter corresponding to the historical deep reinforcement learning model are determined in the following manner:

[0108] Obtain the training role type corresponding to the first virtual object controlled by the training deep reinforcement learning model. The training role type can be directly obtained after the role of the first virtual object is selected, and the role corresponding to the first virtual object can be specified by the staff according to the training process or randomly selected, which is not limited in the present disclosure.

[0109] Then, determining a first target winning rate between the virtual object corresponding to the training character type and the virtual object corresponding to each candidate character type among a plurality of preset character types except the training character type;

[0110] Determine a second target winning rate between a virtual object controlled by a deep reinforcement learning model corresponding to a first reward weight parameter and a virtual object controlled by a deep reinforcement learning model corresponding to a second reward weight parameter, wherein the first reward weight parameter is a training reward weight parameter corresponding to the trained deep reinforcement learning model, and the second reward weight parameter is each other reward weight parameter among the preset multiple reward weight parameters except the first reward weight parameter.

[0111] For example, during the training process, the role types and reward weight parameters corresponding to the two sides of the game can be recorded, and the win-loss results corresponding to the interaction sequences generated in each interaction round during the training process can be counted, so as to obtain the winning rate corresponding to each role type under each decision-making style type. For example, for an interaction sequence, the two sides are an attacking role X and a defensive role Y, and the winning rate of the battle between the two sides can be determined based on the battle results of each interaction sequence that is the same for the two sides. For example, the winning rate corresponding to each role type under each decision-making style type can be stored based on a two-dimensional matrix.

[0112] As an example, if the determined training character type is U1, then for the training character type U1 and the candidate character type Ux, the average of the winning rates of the virtual object under U1 under each reward weight parameter and the virtual object under Ux under each reward weight parameter can be used as the first target winning rate of the battle between U1 and the virtual object under Ux, that is, the probability of the virtual object under U1 defeating the virtual object under Ux. The first target winning rate between the training character type U1 and other candidate character types can be determined in the same way.

[0113] Similarly, for the first reward weight parameter G1 and the second reward weight parameter Gx, the average winning rate of the virtual objects corresponding to each character type under the first reward weight parameter G1 and the virtual objects corresponding to each character type under the second reward weight parameter Gx can be used as the second target winning rate of the battle between the virtual objects controlled by the deep reinforcement learning models corresponding to G1 and Gx, that is, the probability that the virtual object controlled by the deep reinforcement learning model corresponding to G1 defeats the virtual object controlled by the deep reinforcement learning model corresponding to Gx.

[0114] Furthermore, based on the training character type and the first target win rate, the character type corresponding to the second virtual object is determined from the candidate character types, and based on the training reward weight parameters corresponding to the training deep reinforcement model and the second target win rate, the training reward weight parameters corresponding to the historical deep reinforcement learning model are determined from the second reward weight parameters.

[0115] As an example, the character type corresponding to the second virtual object is determined from the character types whose first target win rate is greater than the first win rate threshold, such as randomly selecting one from the character types whose first target win rate is greater than the first win rate threshold, or selecting the character type corresponding to the first target win rate closest to the first win rate threshold as the character type corresponding to the second virtual object. Similarly, the training reward weight parameter corresponding to the historical deep reinforcement learning model can be determined from the reward weight parameter whose second target win rate is greater than the second win rate threshold, randomly selecting one from the reward weight parameter whose second target win rate is greater than the second win rate threshold, or selecting the reward weight parameter corresponding to the second target win rate closest to the second win rate threshold as the training reward weight parameter corresponding to the historical deep reinforcement learning model. By way of example, the first win rate threshold and the second win rate threshold may be the same or different, and by way of example, the first win rate threshold and the second win rate threshold may both be set to 0.5 to ensure that the win rates of the two players in the battle are similar and to improve the learning efficiency of the model.

[0116] As another example, sampling can be performed based on PFSP (Prioritized Fictitious self-play). Taking the selection of role types as an example, for training role type a, the corresponding role type candidate set C can include any role type except a among multiple role types. Then the probability p of a role type b being selected in the candidate set C can be determined by the following method: b :

[0117] Right now

[0118] Among them, P[a beats b] is used to represent the probability of character type a defeating character type b, that is, the first target probability between character type a and character type b mentioned above, and f(x) can select a linear function such as f(x) = 1-x, or a common function of PFSP in the field, which is not limited. The denominator in the above formula is the sum of the winning rates of character type a against all character types in the candidate set. It can be seen from the above formula that when the winning rate of character type a defeating character type b is lower, the probability of character type b being selected is greater, so that the candidate set can be sampled according to the probability corresponding to the character type to obtain the character type corresponding to the historical deep reinforcement learning model.

[0119] Similarly, the training reward weight parameters corresponding to the historical deep reinforcement learning model can also be sampled in a similar manner to the PFSP described above, which will not be repeated here.

[0120] Therefore, through the above technical scheme, after determining the training role type and training reward weight parameters corresponding to the training deep reinforcement learning model, the training role type and training reward weight parameters corresponding to the historical deep reinforcement learning model can be further determined based on the training role type and training reward weight parameters to ensure the difficulty of interaction between the first virtual object and the second virtual object, so as to improve the strategy optimization efficiency of the training deep reinforcement learning model trained based on the interaction sequence and improve the training efficiency of the training deep reinforcement learning model.

[0121] Figure 4 FIG. 1 is a flowchart of a method for controlling a virtual object according to an embodiment of the present disclosure. The method may include:

[0122] In step 41, an interactive virtual object controlled by a user that matches a target virtual object in a target game is determined. The target virtual object may be controlled based on a target deep reinforcement learning model, which may be trained based on any of the above-mentioned deep reinforcement learning model training methods.

[0123] Among them, the method can be applied to one-on-one game scenarios, such as in a 1v1 fighting game environment, the target virtual object is controlled based on the target deep reinforcement learning model, and the interactive virtual object is a game AI controlled by the user. In a human-computer interactive game, the virtual object controlled by the target deep reinforcement learning model and the virtual object controlled by the user can be matched, and then the game result can be obtained through interaction between the two parties. Accordingly, after the game match is generated, the matched virtual object controlled by the user can be used as the interactive virtual object. In step 42, according to the behavior type of the target virtual object in the target game, the target reward weight parameter of the target virtual object is determined from a plurality of reward weight parameters, wherein the target reward weight parameter corresponds to a decision style type of the target deep reinforcement learning model.

[0124] Among them, there can be multiple decision-making style types for the target deep reinforcement learning model, and each decision-making style type corresponds to a decision-making strategy. For example, in a battle game, the decision-making style types can include offensive, defensive, and balanced. As mentioned above, the offensive decision-making strategy is more inclined to actively initiate an attack, the defensive decision-making strategy is more inclined to defend rather than attack, and the balanced decision-making strategy also values ​​offense and defense to make a balanced choice according to the current situation.

[0125] As an example, when playing a game, the user can select the type of game he wants to play. For example, the user can select the AI ​​style he needs through the selection control in the game interface, so that the AI ​​style can be used as the behavior type of the target virtual object. Among them, the behavior type is used to represent the decision-making style type of the target virtual object, and the corresponding relationship between the behavior type and the reward weight parameter can be pre-set, so that after determining the behavior type of the target virtual object, the target reward weight parameter of the target virtual object can be determined based on the corresponding relationship.

[0126] For example, the behavior type of the target virtual object in the target game is determined in the following manner:

[0127] When the target game starts, the behavior type of the target virtual object is determined based on the historical game data of the interactive virtual object.

[0128] As another example, historical game data of the interactive virtual object corresponding to the user can be obtained with the authorization of the user, and further parameters such as the attack frequency and defense frequency output by the user in the historical game process can be determined, so that the operation parameters of the user can be determined based on the attack frequency and defense frequency, and then the behavior type of the target virtual object can be determined based on the operation parameters. If a corresponding relationship table between the operation parameters and the behavior type can be set in advance, the behavior type of the target virtual object can be determined directly based on the obtained user operation parameters and the corresponding relationship table during the application process.

[0129] As another example, during the target game, if the type update condition is met, the behavior type of the target virtual object is determined based on the game data of the interactive virtual object in the target game.

[0130] Wherein, the update condition may be that the target game has reached the update time, for example, the update time may be set at a preset time interval after the target game is started until the target game ends, for example, an update time may be determined every 5 minutes after the target game starts, so as to determine the behavior type of the target virtual object based on the game data in the current target game. For another example, the update condition may be that the cumulative reward corresponding to the target virtual object in the target time period after the target game is started is less than a threshold value, wherein the cumulative reward may be counted based on the game data in the target game, if the cumulative reward in the target time period is less than the threshold value, it means that the behavior type of the target virtual object determined in the initial state may not match the user's operation parameters, at this time, the behavior type of the target virtual object may be further determined based on the game data in the target game, so as to adjust the behavior type of the target virtual object. Wherein, the method of determining the behavior type of the target virtual object based on the game data in the target game is similar to the method of determining the behavior type of the target virtual object based on the historical game data described above, and will not be repeated here.

[0131] In this embodiment, the behavior type of the target virtual object can be determined, and then the decision-making style type corresponding to the target virtual object can be determined to determine the target reward weight parameter corresponding to the target virtual object, thereby ensuring the matching with the user's control behavior, and in the process of controlling the target virtual object to play the game, the output of the target virtual object can be adaptively updated based on the user's operating parameters to match the user's operating parameters, thereby improving the user's game experience.

[0132] In step 43, the interaction between the target virtual object and the interactive virtual object in the virtual environment is sampled to obtain the target state feature. The implementation method of this step is similar to the method of obtaining the first state feature described above, and will not be repeated here.

[0133] In step 44, based on the target state characteristics, the target reward weight parameters and the target deep reinforcement learning model, the target decision action of the target virtual object under the target state characteristics is determined to control the operation of the target virtual object based on the target decision action.

[0134] Accordingly, the target state characteristics and target reward weight parameters can be input into the target deep reinforcement learning model, so that the decision action output by the target deep reinforcement learning model can be obtained, and then the target virtual object can be controlled based on the decision action to achieve interaction with the interactive virtual object.

[0135] Therefore, through the above technical solution, during the interaction process, the decision-making style of the current target virtual object can be determined according to the behavioral style of the interactive object, so that in the interactive battle with the user, a decision-making style matching the user can be selected for interaction, thereby enhancing user interest and usage experience.

[0136] The present disclosure also provides a training device for a deep reinforcement learning model, such as Figure 5 As shown, the device 10 comprises:

[0137] A first acquisition module 100 is used to acquire an interaction sequence generated by the interaction between a first virtual object and a second virtual object in a virtual environment, wherein the interaction sequence includes a plurality of sampling data, each of the sampling data includes a reward value obtained by the first virtual object performing a decision action under a state feature sampled by the virtual environment, the first virtual object is controlled based on a trained deep reinforcement learning model, and the second virtual object is controlled based on a historical deep reinforcement learning model corresponding to the trained deep reinforcement learning model;

[0138] A second acquisition module 200 is used to acquire a training reward weight parameter corresponding to each of the interaction sequences, wherein the training reward weight parameter corresponds to a decision style type of the training deep reinforcement learning model;

[0139] A first determination module 300, configured to determine a target reward value corresponding to each of the sampled data according to a training reward weight parameter corresponding to the interaction sequence and a reward value in the interaction sequence;

[0140] A second determination module 400 is used to determine the target loss of the training deep reinforcement learning model according to the action value prediction value determined based on the state characteristics and decision actions of each of the sampled data, and the target reward value corresponding to the sampled data;

[0141] The training module 500 is used to train the training deep reinforcement learning model based on the target loss.

[0142] Optionally, the output layer of the training deep reinforcement learning model includes multiple output branches, and the output branches correspond one-to-one to multiple preset reward weight parameters;

[0143] The training module is used to:

[0144] Based on the target loss, a parameter in the trained deep reinforcement learning model corresponding to an output branch corresponding to the training reward weight parameter is updated, and the training reward weight parameter is one of the multiple reward weight parameters.

[0145] Optionally, the interaction sequence is generated by a generation module, and the generation module includes:

[0146] A first determination submodule is used to determine a training reward weight parameter for training a deep reinforcement learning model;

[0147] a sampling submodule, configured to sample the interaction between the first virtual object and the second virtual object in the virtual environment to obtain a state feature corresponding to the first virtual object;

[0148] a second determination submodule, configured to determine a decision action of the first virtual object under the state feature based on the state feature, the training reward weight parameter, and the training deep reinforcement learning model, so as to control the operation of the first virtual object based on the decision action, and return to the step of sampling the interaction between the first virtual object and the second virtual object in the virtual environment to obtain the state feature corresponding to the first virtual object, until the interaction round ends;

[0149] A sorting submodule is used to sort the sampled data obtained in the interaction round in the order of sampling time to obtain the interaction sequence, and associate the interaction sequence with the training reward weight parameter, wherein each sampled data includes a reward value obtained by executing a decision action when the virtual environment is in a state feature.

[0150] Optionally, the output layer of the training deep reinforcement learning model includes multiple output branches, and the output branches correspond one-to-one to multiple preset reward weight parameters;

[0151] The second determining submodule includes:

[0152] The third determination submodule is used to input the state characteristics and the training reward weight parameters into the training deep reinforcement learning model, so that the feature layer of the training deep reinforcement learning model can perform feature extraction on the state characteristics and the training reward weight parameters, obtain sub-outputs corresponding to multiple output branches of the output layer based on the extracted features, and determine the decision action according to the sub-outputs corresponding to the output branches corresponding to the training reward weight parameters in the training deep reinforcement learning model.

[0153] Optionally, the second determining submodule includes:

[0154] A fourth determination submodule, configured to input the state feature and the training reward weight parameter into the training deep reinforcement learning model, and determine the decision action according to the output of the training deep reinforcement learning model;

[0155] The training deep reinforcement learning model includes a neural network feature layer, a type feature layer and an attention layer;

[0156] The neural network feature layer is used to determine the state feature vector corresponding to the state feature and the parameter feature vector corresponding to the training reward weight parameter; the type feature layer is used to determine the type feature vector corresponding to each candidate type under the type feature layer based on the state feature vector and the parameter feature vector, wherein the candidate types correspond one-to-one to the hidden layer decision styles; the attention layer is used to fuse the type feature vectors corresponding to each type according to the parameter feature vector based on the attention mechanism, and determine the output of the training deep reinforcement learning model according to the fusion result.

[0157] Optionally, the first virtual object and the second virtual object have different role types, and the historical deep reinforcement learning model and the training deep reinforcement learning model respectively correspond to different training reward weight parameters;

[0158] The role type corresponding to the second virtual object and the training reward weight parameter corresponding to the historical deep reinforcement learning model are determined in the following manner:

[0159] Obtaining a training role type corresponding to a first virtual object controlled by the training deep reinforcement learning model;

[0160] Determining a first target winning rate between a virtual object corresponding to the training character type and a virtual object corresponding to each candidate character type among a plurality of preset character types except the training character type;

[0161] Determine a second target winning rate between a virtual object controlled by a deep reinforcement learning model corresponding to a first reward weight parameter and a virtual object controlled by a deep reinforcement learning model corresponding to a second reward weight parameter, wherein the first reward weight parameter is a training reward weight parameter corresponding to the training deep reinforcement learning model, and the second reward weight parameter is each other reward weight parameter among a plurality of preset reward weight parameters except the first reward weight parameter;

[0162] Based on the training character type and the first target win rate, the character type corresponding to the second virtual object is determined from the candidate character types, and based on the training reward weight parameters corresponding to the training deep reinforcement learning model and the second target win rate, the training reward weight parameters corresponding to the historical deep reinforcement learning model are determined from the second reward weight parameters.

[0163] The present disclosure also provides a control device for a virtual object, which may include:

[0164] A third determination module is used to determine an interactive virtual object controlled by a user that matches a target virtual object in a target game, wherein the target virtual object is controlled based on a target deep reinforcement learning model, wherein the target deep reinforcement learning model is trained based on any of the above-mentioned deep reinforcement learning model training methods;

[0165] a fourth determination module, configured to determine a target reward weight parameter of the target virtual object from a plurality of reward weight parameters according to a behavior type of the target virtual object in the target game, wherein the target reward weight parameter corresponds to a decision style type of the target deep reinforcement learning model;

[0166] A sampling module, used for sampling the interaction between the target virtual object and the interactive virtual object in the virtual environment to obtain target state characteristics;

[0167] A control module, for determining a target decision action of the target virtual object under the target state feature based on the target state feature, the target reward weight parameter and the target deep reinforcement learning model, so as to control the operation of the target virtual object based on the target decision action

[0168] Optionally, the behavior type of the target virtual object in the target game is determined in the following manner:

[0169] When the target game starts, determining the behavior type of the target virtual object based on the historical game data of the interactive virtual object;

[0170] During the target game, if a type update condition is met, the behavior type of the target virtual object is determined based on the game data of the interactive virtual object in the target game.

[0171] Reference below Figure 6 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0172] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0173] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0174] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0175] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0176] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0177] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0178] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains an interaction sequence generated by the interaction between the first virtual object and the second virtual object in the virtual environment, wherein the interaction sequence includes multiple sampling data, each of the sampling data includes a reward value obtained by the first virtual object performing a decision action under the state characteristics sampled by the virtual environment, the first virtual object is controlled based on a training deep reinforcement learning model, and the second virtual object is controlled based on a historical deep reinforcement learning model corresponding to the training deep reinforcement learning model; obtains a training reward weight parameter corresponding to each of the interaction sequences, wherein the training reward weight parameter corresponds to a decision style type of the training deep reinforcement learning model; determines a target reward value corresponding to each of the sampling data according to the training reward weight parameter corresponding to the interaction sequence and the reward value in the interaction sequence; determines a target loss of the training deep reinforcement learning model according to an estimated value of the action value determined based on the state characteristics and decision actions of each of the sampling data, and the target reward value corresponding to the sampling data; and trains the training deep reinforcement learning model based on the target loss.

[0179] Alternatively, the computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device: determines an interactive virtual object controlled by a user that matches a target virtual object in a target game, wherein the target virtual object is controlled based on a target deep reinforcement learning model, wherein the target deep reinforcement learning model is trained based on any of the training methods for deep reinforcement learning models described above; determines a target reward weight parameter of the target virtual object from a plurality of reward weight parameters according to a behavior type of the target virtual object in the target game, wherein the target reward weight parameter corresponds to a decision style type of the target deep reinforcement learning model; samples the interaction between the target virtual object and the interactive virtual object in a virtual environment to obtain a target state feature; and determines a target decision action of the target virtual object under the target state feature based on the target state feature, the target reward weight parameter and the target deep reinforcement learning model, so as to control the operation of the target virtual object based on the target decision action.

[0180] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0181] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0182] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, the first acquisition module may also be described as a "module for acquiring an interaction sequence generated by the interaction between the first virtual object and the second virtual object in the virtual environment".

[0183] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0184] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0185] According to one or more embodiments of the present disclosure, Example 1 provides a training method for a deep reinforcement learning model, wherein the method includes: obtaining an interaction sequence generated by the interaction of a first virtual object and a second virtual object in a virtual environment, wherein the interaction sequence includes multiple sampling data, each of the sampling data includes a reward value obtained by the first virtual object performing a decision action under the state characteristics sampled by the virtual environment, the first virtual object is controlled based on a training deep reinforcement learning model, and the second virtual object is controlled based on a historical deep reinforcement learning model corresponding to the training deep reinforcement learning model; obtaining a training reward weight parameter corresponding to each of the interaction sequences, wherein the training reward weight parameter corresponds to a decision style type of the training deep reinforcement learning model; determining a target reward value corresponding to each of the sampling data according to the training reward weight parameter corresponding to the interaction sequence and the reward value in the interaction sequence; determining a target loss of the training deep reinforcement learning model according to an estimated value of the action value determined based on the state characteristics and decision actions of each of the sampling data, and the target reward value corresponding to the sampling data; and training the training deep reinforcement learning model based on the target loss.

[0186] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein the output layer of the training deep reinforcement learning model includes multiple output branches, and the output branches correspond one-to-one to a preset plurality of reward weight parameters; the training of the training deep reinforcement learning model based on the target loss includes: based on the target loss, updating the parameters corresponding to the output branches corresponding to the training reward weight parameters in the training deep reinforcement learning model, and the training reward weight parameters are one of the plurality of reward weight parameters.

[0187] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 1, wherein the interaction sequence is generated in the following manner: determining a training reward weight parameter for training a deep reinforcement learning model; sampling the interaction between the first virtual object and the second virtual object in the virtual environment to obtain a state feature corresponding to the first virtual object; determining a decision action of the first virtual object under the state feature based on the state feature, the training reward weight parameter and the training deep reinforcement learning model to control the operation of the first virtual object based on the decision action, and returning to the step of sampling the interaction between the first virtual object and the second virtual object in the virtual environment to obtain the state feature corresponding to the first virtual object until the interaction round ends; sorting the sampled data sampled in the interaction round in chronological order of sampling time to obtain the interaction sequence, and associating the interaction sequence with the training reward weight parameter, wherein each of the sampled data includes a reward value obtained by performing a decision action when the virtual environment is in the state feature.

[0188] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 3, wherein the output layer of the trained deep reinforcement learning model includes multiple output branches, and the output branches correspond one-to-one to multiple preset reward weight parameters; the decision action of the first virtual object under the state characteristics is determined based on the state characteristics, the training reward weight parameters and the training deep reinforcement learning model, including: inputting the state characteristics and the training reward weight parameters into the training deep reinforcement learning model, so that the feature layer of the training deep reinforcement learning model performs feature extraction on the state characteristics and the training reward weight parameters, obtaining sub-outputs corresponding to the multiple output branches of the output layer based on the extracted features, and determining the decision action according to the sub-outputs corresponding to the output branches corresponding to the training reward weight parameters in the training deep reinforcement learning model.

[0189] According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 3, wherein the determining the decision action of the first virtual object under the state feature based on the state feature, the training reward weight parameter and the training deep reinforcement learning model includes: inputting the state feature and the training reward weight parameter into the training deep reinforcement learning model, and determining the decision action according to the output of the training deep reinforcement learning model; wherein the training deep reinforcement learning model includes a neural network feature layer, a type feature layer and an attention layer; the neural network feature layer is used to determine the state feature vector corresponding to the state feature and the parameter feature vector corresponding to the training reward weight parameter, and the type feature layer is used to determine the type feature vector corresponding to each candidate type under the type feature layer based on the state feature vector and the parameter feature vector, wherein the candidate type corresponds one-to-one to the hidden layer decision style; the attention layer is used to fuse the type feature vector corresponding to each type according to the parameter feature vector based on the attention mechanism, and determine the output of the training deep reinforcement learning model according to the fusion result.

[0190] According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 1, wherein the character types of the first virtual object and the second virtual object are different, and the training reward weight parameters corresponding to the historical deep reinforcement learning model and the training deep reinforcement learning model are different; the character type corresponding to the second virtual object and the training reward weight parameter corresponding to the historical deep reinforcement learning model are determined in the following manner: obtaining the training character type corresponding to the first virtual object controlled by the training deep reinforcement learning model; determining a first target winning rate between the virtual object corresponding to the training character type and the virtual object corresponding to each candidate character type among a plurality of preset character types except the training character type; determining a deep reward weight parameter based on the first target winning rate. a second target winning rate between a virtual object controlled by a deep reinforcement learning model and a virtual object controlled by a deep reinforcement learning model corresponding to a second reward weight parameter, wherein the first reward weight parameter is a training reward weight parameter corresponding to the trained deep reinforcement learning model, and the second reward weight parameter is each other reward weight parameter among the preset multiple reward weight parameters except the first reward weight parameter; based on the training character type and the first target winning rate, the character type corresponding to the second virtual object is determined from the candidate character types, and based on the training reward weight parameter corresponding to the training deep reinforcement learning model and the second target winning rate, the training reward weight parameter corresponding to the historical deep reinforcement learning model is determined from the second reward weight parameter.

[0191] According to one or more embodiments of the present disclosure, Example 7 provides a method for controlling a virtual object, the method comprising: determining an interactive virtual object controlled by a user that matches a target virtual object in a target game, wherein the target virtual object is controlled based on a target deep reinforcement learning model, wherein the target deep reinforcement learning model is trained based on the training method of the deep reinforcement learning model described in any one of Examples 1-6; determining a target reward weight parameter of the target virtual object from a plurality of reward weight parameters according to a behavior type of the target virtual object in the target game, wherein the target reward weight parameter corresponds to a decision style type of the target deep reinforcement learning model; sampling the interaction between the target virtual object and the interactive virtual object in a virtual environment to obtain a target state feature; determining a target decision action of the target virtual object under the target state feature based on the target state feature, the target reward weight parameter and the target deep reinforcement learning model, so as to control the operation of the target virtual object based on the target decision action.

[0192] According to one or more embodiments of the present disclosure, Example 8 provides the method of Example 7, wherein the behavior type of the target virtual object in the target game is determined in the following manner: at the start of the target game, the behavior type of the target virtual object is determined based on the historical game data of the interactive virtual object; during the target game, if a type update condition is met, the behavior type of the target virtual object is determined based on the game data of the interactive virtual object in the target game.

[0193] According to one or more embodiments of the present disclosure, Example 9 provides a training device for a deep reinforcement learning model, the device comprising: a first acquisition module, used to acquire an interaction sequence generated by the interaction of a first virtual object and a second virtual object in a virtual environment, wherein the interaction sequence comprises a plurality of sampling data, each of the sampling data comprises a reward value obtained by the first virtual object performing a decision action under the state characteristics sampled by the virtual environment, the first virtual object is controlled based on a training deep reinforcement learning model, and the second virtual object is controlled based on a historical deep reinforcement learning model corresponding to the training deep reinforcement learning model; a second acquisition module, used to acquire a training reward weight parameter corresponding to each of the interaction sequences, wherein the training reward weight parameter corresponds to a decision style type of the training deep reinforcement learning model; a first determination module, used to determine a target reward value corresponding to each of the sampling data according to the training reward weight parameter corresponding to the interaction sequence and the reward value in the interaction sequence; a second determination module, used to determine a target loss of the training deep reinforcement learning model according to an estimated value of the action value determined based on the state characteristics and decision actions of each of the sampling data, and the target reward value corresponding to the sampling data; a training module, used to train the training deep reinforcement learning model based on the target loss.

[0194] According to one or more embodiments of the present disclosure, Example 10 provides a control device for a virtual object, the device comprising: a third determination module, used to determine an interactive virtual object controlled by a user that matches a target virtual object in a target game, wherein the target virtual object is controlled based on a target deep reinforcement learning model, wherein the target deep reinforcement learning model is trained based on the training method of the deep reinforcement learning model described in any one of Examples 1-6; a fourth determination module, used to determine a target reward weight parameter of the target virtual object from a plurality of reward weight parameters according to a behavior type of the target virtual object in the target game, wherein the target reward weight parameter corresponds to a decision style type of the target deep reinforcement learning model; a sampling module, used to sample the interaction between the target virtual object and the interactive virtual object in a virtual environment to obtain a target state feature; a control module, used to determine a target decision action of the target virtual object under the target state feature based on the target state feature, the target reward weight parameter and the target deep reinforcement learning model, so as to control the operation of the target virtual object based on the target decision action.

[0195] According to one or more embodiments of the present disclosure, Example 11 provides a computer-readable medium having a computer program stored thereon, which implements the steps of any of the methods described in Examples 1-8 when executed by a processing device.

[0196] According to one or more embodiments of the present disclosure, Example 12 provides an electronic device, including:

[0197] A storage device having a computer program stored thereon; and a processing device for executing the computer program in the storage device to implement the steps of the method described in any one of Examples 1-8.

[0198] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0199] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0200] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A training method for a deep reinforcement learning model, characterized in that: The method comprises: Acquire an interaction sequence generated by the interaction between a first virtual object and a second virtual object in a virtual environment, wherein the interaction sequence includes a plurality of sampling data, each of the sampling data includes a reward value obtained by the first virtual object performing a decision action under the state characteristics sampled by the virtual environment, the first virtual object is controlled based on a trained deep reinforcement learning model, and the second virtual object is controlled based on a historical deep reinforcement learning model corresponding to the trained deep reinforcement learning model; Obtaining a training reward weight parameter corresponding to each of the interaction sequences, wherein the training reward weight parameter corresponds to a decision style type of the training deep reinforcement learning model; Determining a target reward value corresponding to each of the sampled data according to a training reward weight parameter corresponding to the interaction sequence and a reward value in the interaction sequence; Determine the target loss of the training deep reinforcement learning model according to the estimated action value determined based on the state characteristics and decision actions of each of the sampled data, and the target reward value corresponding to the sampled data; Training the training deep reinforcement learning model based on the target loss; The interaction sequence is generated in the following way: Determine the training reward weight parameters for training deep reinforcement learning models; Sampling the interaction between the first virtual object and the second virtual object in the virtual environment to obtain a state feature corresponding to the first virtual object; Based on the state feature, the training reward weight parameter and the training deep reinforcement learning model, determine a decision action of the first virtual object under the state feature, control the operation of the first virtual object based on the decision action, and return to the step of sampling the interaction between the first virtual object and the second virtual object in the virtual environment to obtain the state feature corresponding to the first virtual object until the interaction round ends; The sampled data obtained in the interaction round are sorted in the order of sampling time to obtain the interaction sequence, and the interaction sequence is associated with the training reward weight parameter, wherein each sampled data includes a reward value obtained by executing a decision action when the virtual environment is in a state characteristic.

2. The method according to claim 1, characterized in that The output layer of the training deep reinforcement learning model includes multiple output branches, and the output branches correspond one-to-one to multiple preset reward weight parameters; The training of the deep reinforcement learning model based on the target loss includes: Based on the target loss, a parameter in the trained deep reinforcement learning model corresponding to an output branch corresponding to the training reward weight parameter is updated, and the training reward weight parameter is one of the multiple reward weight parameters.

3. The method according to claim 1, characterized in that The output layer of the training deep reinforcement learning model includes multiple output branches, and the output branches correspond one-to-one to multiple preset reward weight parameters; The determining, based on the state feature, the training reward weight parameter and the training deep reinforcement learning model, a decision action of the first virtual object under the state feature comprises: The state features and the training reward weight parameters are input into the training deep reinforcement learning model, so that the feature layer of the training deep reinforcement learning model performs feature extraction on the state features and the training reward weight parameters, and sub-outputs corresponding to multiple output branches of the output layer are obtained based on the extracted features, and the decision action is determined according to the sub-outputs corresponding to the output branches corresponding to the training reward weight parameters in the training deep reinforcement learning model.

4. The method according to claim 1, characterized in that: The determining, based on the state feature, the training reward weight parameter and the training deep reinforcement learning model, a decision action of the first virtual object under the state feature comprises: Inputting the state feature and the training reward weight parameter into the training deep reinforcement learning model, and determining the decision action according to the output of the training deep reinforcement learning model; The training deep reinforcement learning model includes a neural network feature layer, a type feature layer and an attention layer; The neural network feature layer is used to determine the state feature vector corresponding to the state feature and the parameter feature vector corresponding to the training reward weight parameter; the type feature layer is used to determine the type feature vector corresponding to each candidate type under the type feature layer based on the state feature vector and the parameter feature vector, wherein the candidate types correspond one-to-one to the hidden layer decision styles; the attention layer is used to fuse the type feature vectors corresponding to each type according to the parameter feature vector based on the attention mechanism, and determine the output of the training deep reinforcement learning model according to the fusion result.

5. The method according to claim 1, characterized in that The first virtual object and the second virtual object have different role types, and the historical deep reinforcement learning model and the training deep reinforcement learning model respectively correspond to different training reward weight parameters; The role type corresponding to the second virtual object and the training reward weight parameter corresponding to the historical deep reinforcement learning model are determined in the following manner: Obtaining a training role type corresponding to a first virtual object controlled by the training deep reinforcement learning model; Determining a first target winning rate between a virtual object corresponding to the training character type and a virtual object corresponding to each candidate character type among a plurality of preset character types except the training character type; Determine a second target winning rate between a virtual object controlled by a deep reinforcement learning model corresponding to a first reward weight parameter and a virtual object controlled by a deep reinforcement learning model corresponding to a second reward weight parameter, wherein the first reward weight parameter is a training reward weight parameter corresponding to the training deep reinforcement learning model, and the second reward weight parameter is each other reward weight parameter among a plurality of preset reward weight parameters except the first reward weight parameter; Based on the training character type and the first target win rate, the character type corresponding to the second virtual object is determined from the candidate character types, and based on the training reward weight parameters corresponding to the training deep reinforcement learning model and the second target win rate, the training reward weight parameters corresponding to the historical deep reinforcement learning model are determined from the second reward weight parameters.

6. A method for controlling a virtual object, characterized in that: The method comprises: Determine an interactive virtual object controlled by a user that matches a target virtual object in a target game, wherein the target virtual object is controlled based on a target deep reinforcement learning model, wherein the target deep reinforcement learning model is trained based on the training method for the deep reinforcement learning model according to any one of claims 1 to 5; Determining a target reward weight parameter of the target virtual object from a plurality of reward weight parameters according to a behavior type of the target virtual object in the target game, wherein the target reward weight parameter corresponds to a decision style type of the target deep reinforcement learning model; Sampling the interaction between the target virtual object and the interactive virtual object in the virtual environment to obtain target state features; Based on the target state characteristics, the target reward weight parameters and the target deep reinforcement learning model, a target decision action of the target virtual object under the target state characteristics is determined to control the operation of the target virtual object based on the target decision action.

7. The method according to claim 6, characterized in that The behavior type of the target virtual object in the target game is determined by: When the target game starts, determining the behavior type of the target virtual object based on the historical game data of the interactive virtual object; During the target game, if a type update condition is met, the behavior type of the target virtual object is determined based on the game data of the interactive virtual object in the target game.

8. A training device for a deep reinforcement learning model, characterized in that: The device comprises: A first acquisition module is used to acquire an interaction sequence generated by the interaction between a first virtual object and a second virtual object in a virtual environment, wherein the interaction sequence includes a plurality of sampling data, each of the sampling data includes a reward value obtained by the first virtual object performing a decision action under a state feature sampled by the virtual environment, the first virtual object is controlled based on a trained deep reinforcement learning model, and the second virtual object is controlled based on a historical deep reinforcement learning model corresponding to the trained deep reinforcement learning model; A second acquisition module is used to acquire a training reward weight parameter corresponding to each of the interaction sequences, wherein the training reward weight parameter corresponds to a decision style type of the training deep reinforcement learning model; A first determination module, configured to determine a target reward value corresponding to each of the sampled data according to a training reward weight parameter corresponding to the interaction sequence and a reward value in the interaction sequence; A second determination module is used to determine the target loss of the training deep reinforcement learning model according to the action value estimate determined based on the state characteristics and decision actions of each of the sampled data, and the target reward value corresponding to the sampled data; A training module, used for training the training deep reinforcement learning model based on the target loss; The interaction sequence is generated by a generation module, and the generation module includes: A first determination submodule is used to determine a training reward weight parameter for training a deep reinforcement learning model; a sampling submodule, configured to sample the interaction between the first virtual object and the second virtual object in the virtual environment to obtain a state feature corresponding to the first virtual object; a second determination submodule, configured to determine a decision action of the first virtual object under the state feature based on the state feature, the training reward weight parameter, and the training deep reinforcement learning model, so as to control the operation of the first virtual object based on the decision action, and return to the step of sampling the interaction between the first virtual object and the second virtual object in the virtual environment to obtain the state feature corresponding to the first virtual object, until the interaction round ends; A sorting submodule is used to sort the sampled data obtained in the interaction round in the order of sampling time to obtain the interaction sequence, and associate the interaction sequence with the training reward weight parameter, wherein each sampled data includes a reward value obtained by executing a decision action when the virtual environment is in a state feature.

9. A control device for a virtual object, characterized in that: The device comprises: A third determination module is used to determine an interactive virtual object controlled by a user that matches a target virtual object in a target game, wherein the target virtual object is controlled based on a target deep reinforcement learning model, wherein the target deep reinforcement learning model is trained based on the training method of the deep reinforcement learning model described in any one of claims 1 to 5; a fourth determination module, configured to determine a target reward weight parameter of the target virtual object from a plurality of reward weight parameters according to a behavior type of the target virtual object in the target game, wherein the target reward weight parameter corresponds to a decision style type of the target deep reinforcement learning model; A sampling module, used for sampling the interaction between the target virtual object and the interactive virtual object in the virtual environment to obtain target state characteristics; A control module is used to determine a target decision action of the target virtual object under the target state characteristics based on the target state characteristics, the target reward weight parameters and the target deep reinforcement learning model, so as to control the operation of the target virtual object based on the target decision action.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processing device, the steps of the method described in any one of claims 1 to 7 are implemented.

11. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 7.