Decision model training method, target object strategy control method and device

Abstract

The application provides a decision model training method, a target object strategy control method and device, the method comprising: obtaining training data of a current state of a target object in a target environment, the training data of the current state comprising N training information of time points, the training information of any t time point in the N time points comprising state information of the t time point, a control strategy of the t time point and a predicted reward of the control strategy of the t time point, the control strategy of the t time point and the predicted reward of the control strategy of the t time point being output of a decision model input with the state information of the t time point, constructing a target loss function according to the training data of the current state, adjusting parameters of the decision model according to the target loss function, inputting state information of an N+1 time point of the target object into the decision model with the adjusted parameters, continuing to train the decision model according to training data of a next state of the target object until a stop training condition is met, and obtaining a trained decision model.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence, and in particular to a decision model training method, a target object strategy control method and device. BACKGROUND

[0002] An agent refers to an entity with intelligence, which refers to any independent entity capable of thinking and interacting with the environment. Strategy refers to the behavior of an agent selecting actions after perceiving the environment. Reinforcement learning, as a subfield of artificial intelligence, can predict the action strategy of an agent through a reinforcement learning model. The decision model trained by reinforcement learning is basically consistent or has a high degree of similarity in action decision when facing different opponent agents in multi-agent game, that is, the action decision of the agent is single and fixed.

[0003] How to realize the diversity strategy of the agent in the multi-player game environment, that is, to adjust the action decision of the agent in real time when facing different opponent agents, is a problem to be solved. SUMMARY

[0004] The present application provides a decision model training method, a target object strategy control method and device, which can realize the diversity strategy of the target object in the target environment.

[0005] In a first aspect, the present application provides a decision model training method, comprising:

[0006] Obtaining training data of a current state of a target object in a target environment, the training data of the current state comprising N training information of time points, any t training information of time points comprising state information of t time point, control strategy of t time point and predicted reward of the control strategy of t time point, the state information of t time point comprising real-time reward of t time point fed back by the target environment, observation information of t time point and region identifier of t time point reached by the target object, the control strategy of t time point and the predicted reward of the control strategy of t time point being the output of the decision model inputted by the state information of t time point, the control strategy of t time point comprising movement action strategy, target task action strategy and target reaching region identifier of t time point, wherein the target reaching region identifier of t time point is the same as the region identifier of t+1 time point reached by the target object, and t and N are positive integers;

[0007] Constructing a target loss function according to the training data of the current state;

[0008] Adjusting and updating the parameters of the decision model according to the target loss function;

[0009] The state information of the target object at the N+1 moment is input into the decision model after updating the parameters, and the decision model is continuously trained according to the training data of the next state of the target object until a stop training condition is met, so as to obtain a trained decision model.

[0010] In a second aspect, the present application provides a strategy control method of a target object, comprising:

[0011] Obtaining state information of the target object at a current moment in a target environment;

[0012] Inputting the state information at the current moment into a decision model to output a control strategy at the current moment, wherein the decision model is trained according to the decision model training method in the first aspect;

[0013] Controlling the target object to perform an action by using the control strategy at the current moment.

[0014] In a third aspect, the present application provides a decision model training device, comprising:

[0015] An obtaining module is configured to obtain training data of a current state of a target object in a target environment, wherein the training data of the current state comprises training information of N moments, the training information of any t moment in the N moments comprises state information at the t moment, a control strategy at the t moment, and a predicted reward of the control strategy at the t moment, the state information at the t moment comprises a real-time reward at the t moment, observation information at the t moment, and a region identifier of a region reached by the target object at the t moment, the control strategy at the t moment and the predicted reward of the control strategy at the t moment are outputs of a decision model inputted with the state information at the t moment, the control strategy at the t moment comprises a moving action strategy, a target task action strategy, and a target region identifier at the t moment, wherein the target region identifier at the t moment is the same as a region identifier of a region reached by the target object at t+1 moment, and t and N are positive integers.

[0016] A constructing module is configured to construct a target loss function according to the training data of the current state.

[0017] An adjusting module is configured to adjust and update parameters of the decision model according to the target loss function.

[0018] A processing module is configured to input state information of the target object at the N+1 moment into the decision model after updating the parameters, and continuously train the decision model according to training data of a next state of the target object until a stop training condition is met, so as to obtain a trained decision model.

[0019] In a fourth aspect, the present application provides a strategy control device of a target object, comprising:

[0020] an acquisition module configured to acquire state information of a target object at a current time in a target environment;

[0021] a control policy output module configured to input the state information at the current time into a decision model to output a control policy at the current time, the decision model being trained according to the decision model training method of the first aspect;

[0022] an action control module configured to control the target object to perform an action by using the control policy at the current time.

[0023] In a fifth aspect, the present application provides a computer device, comprising: a processor and a memory, the memory being configured to store a computer program, and the processor being configured to invoke and run the computer program stored in the memory to execute the method of the first aspect or the second aspect.

[0024] In a sixth aspect, the present application provides a computer readable storage medium comprising instructions which, when executed on a computer program, cause the computer to perform the method of the first aspect or the second aspect.

[0025] In a seventh aspect, the present application provides a computer program product comprising instructions which, when executed on a computer, cause the computer to perform the method of the first aspect or the second aspect.

[0026] In summary, in the present application, by training a decision model for action policy control of a target object in a target environment, during training of the decision model, the real-time reward at time t, the observation information at time t and the region identifier of the target object at time t fed back by the target environment are collectively used as the input of the decision model at time t, and the decision model decides and outputs the control policy at time t (including the movement action policy at time t, the target task action policy, the target region that the target object should reach at next time (i.e. the region indicated by the target region identifier at time t) and the predicted reward of the control policy at time t) according to the three parts of information, wherein the real-time reward at time t fed back by the target environment is the reward obtained after the target object performs the target task with an unknown opponent, the decision model not only learns the movement action policy, but also learns the target task action policy and the target region that the target object should reach at next time, which is used to control the target object to move to a specific region and perform a target task at next time, so that the decision model can learn multiple control policies. Thus, when facing different opponent target objects, the target object can adjust its action decision in real time, and realize the diversity of the target object in the target environment.

[0027] Furthermore, in this embodiment, the decision model training employs a two-stage learning training mode. In the first stage, a post-hoc proximal policy optimization method is used to train the target decision model, which is then used to train the movement action strategy of the target object. In the second stage, the target decision model is used as a teacher model for supervised learning training of the decision model, learning the target task action strategy and the target area the target object should reach in the next moment. This multi-stage training method reduces training difficulty and improves training efficiency. Attached Figure Description

[0028] Figure 1 This application provides an illustration of a decision model training method and a strategy control method for a target object, which serve as embodiments of this application.

[0029] Figure 2 A flowchart illustrating a decision model training method provided in this application embodiment;

[0030] Figure 3 A flowchart illustrating a decision model training method provided in this application embodiment;

[0031] Figure 4 A schematic diagram of a vertical view map corresponding to a target environment;

[0032] Figure 5 To Figure 4 The diagram shows a map after the vertical view map has been divided into regions.

[0033] Figure 6 A schematic diagram illustrating the process by which a decision model, based on the input state information at time t, outputs a control decision at time t, is provided in an embodiment of this application.

[0034] Figure 7 A flowchart illustrating a strategy control method for a target object provided in an embodiment of this application;

[0035] Figure 8 This is a schematic diagram of the structure of a decision model training device provided in an embodiment of this application;

[0036] Figure 9 A schematic diagram of the structure of a strategy control device for a target object provided in an embodiment of this application;

[0037] Figure 10 This is a schematic block diagram of the computer device 700 provided in the embodiments of this application. Detailed Implementation

[0038] With reference to the drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts are within the scope of the present application.

[0039] It should be noted that the terms "first", "second" and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in other than the order illustrated or described herein. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server comprising a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0040] The decision model training method and the target object strategy control method and device provided by the embodiments of the present application relate to artificial intelligence (AI) technology, and specifically relate to machine learning and reinforcement learning in artificial intelligence. Before introducing the technical solutions of the present application, the related knowledge of the present application will be introduced as follows:

[0041] Artificial intelligence: is to use digital computer or digital computer controlled machine simulation, extension and expansion of human intelligence, perception of environment, acquisition of knowledge and use of knowledge to obtain the best results of theory, method, technology and application system. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive subject, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning and other fields.

[0042] Machine Learning (ML): is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory, etc. It is a specialized study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure, and continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning.

[0043] Deep Learning (DL): is a branch of machine learning, which is an algorithm that attempts to use multiple processing layers containing complex structures or multiple nonlinear transformations to abstract data at a high level. Deep learning is a learning process that learns the internal rules and representations of training sample data, which helps to interpret text, images, and sound data. The ultimate goal of deep learning is to enable machines to have analytical learning ability like humans, and to recognize text, images, and sound data. Deep learning is a complex machine learning algorithm, and the results achieved in speech and image recognition far exceed previous related technologies.

[0044] Reinforcement learning is a way of machine learning that can be used to train a decision model for action decision-making of a target object in a target environment. The training process of the decision model includes: in the process of training the decision model, the neural network model based on reinforcement learning is used to learn the action decision of the target object and to learn the control of the target object to adapt to the target environment, and the decision model is obtained. The decision model obtained by training can be used to control the action strategy of the target object in the target environment.

[0045] Neural Network (NN): a deep learning model that simulates the structure and function of biological neural networks in the field of machine learning and cognitive science.

[0046] Target environment: refers to the environment in which the target object is located. The target environment can be a virtual environment or a real environment. The target environment can have various forms of expression, for example, the target environment can be a chessboard, a sports field or a pursuit site, and for example, the target environment can be a virtual chessboard or a virtual pursuit site set up by a computer, etc. various game environments.

[0047] Target object: refers to an active object in the target environment.

[0048] Game environment: a virtual environment displayed by a game application when running on a terminal device. The game environment can be a simulation environment of the real world, a semi-simulation and semi-fictional three-dimensional environment, or a purely fictional three-dimensional environment. Optionally, the game environment is also used for a game environment battle between at least two game characters (such as a multi-player game, a multi-player shooting game, a multi-player chase game, or a multi-player hide-and-seek game), for example, in which there are game resources available for use by at least two game characters. Optionally, the map of the game environment is composed of multiple squares or rectangles.

[0049] Game character: also known as a virtual object, refers to an active object in a game environment. The active object can be at least one of a virtual person, a virtual animal, an animation character, etc. Exemplarily, when the current game environment is a three-dimensional game environment, the game character is a three-dimensional model, each game character has its own shape and volume in the three-dimensional game environment, and occupies a part of the space in the three-dimensional game environment. Optionally, the game character can be a hero, a soldier, or a neutral creature in a multi-player online battle arena (MOBA) game.

[0050] Agent: in the embodiments of the present application, an agent refers to a game character that can interact with a game environment in a game. For example, an agent can communicate, cooperate, or battle with other agents in a certain game environment, and autonomously complete a set target in the game environment according to its own perception of the game environment, according to existing instructions, or through autonomous learning.

[0051] In the related art, the action decision of an agent is single and fixed when facing different opponent agents. To solve this problem, the present application trains a decision model for action policy control of a target object in a target environment. In training the decision model, the state information at time t is taken as the input of the decision model, and the control policy at time t and the predicted reward of the control policy at time t are output. The state information at time t includes the real-time reward of the target environment at time t, the observation information at time t, and the region identifier of the target object at time t. The control policy at time t includes the movement action policy, the target task action policy, and the target arrival region identifier at time t. The target arrival region identifier at time t is the same as the region identifier of the target object at time t+1, that is, the target arrival region identifier at time t is input as the region identifier of the target object at time t+1. The state information at time t, the control policy at time t, and the predicted reward of the control policy at time t constitute the training information at time t. The training information at consecutive N times constitutes the training data of the current state of the target object. The target loss function is constructed according to the training data of the current state, and the parameters of the decision model are adjusted and updated according to the target loss function. Then, the state information of the target object at time N+1 is input into the decision model with updated parameters, and the decision model is continuously trained according to the training data of the next state of the target object until the stopping condition is met, and the trained decision model is obtained. Since the real-time reward of the target environment at time t, the observation information at time t, and the region identifier of the target object at time t are collectively input into the decision model at time t, the decision model decides and outputs the control policy at time t (including the movement action policy at time t, the target task action policy, the target region where the target object should arrive at the next time (i.e. the region indicated by the target arrival region identifier at time t), and the predicted reward of the control policy at time t), wherein the real-time reward of the target environment at time t is the reward obtained by the target object after performing the target task with an unknown opponent. The decision model not only learns the movement action policy, but also learns the target task action policy and the target region where the target object should arrive at the next time. This target region is used to control the movement and target task of the target object at the next time, so that the decision model can learn multiple control policies. Therefore, the target object can adjust its action decision in real time when facing different opponent target objects, and achieve diversity strategy of the target object in the target environment.

[0052] When the target environment is a game environment, multiple application scenarios (such as first-person or third-person) can be adapted. Real opponent information is not needed in the training process, which reduces the training cost and expands the application range.

[0053] Further, the observation information in the embodiments of the present application is first-person observation information, the first-person observation information is target environment information that can be observed by the target object, thereby providing the target object in the first-person multi-player game environment with a decision-making method that can enable the target object to obtain a variety of strategies. The reasoning difficulty and information acquisition difficulty of the first-person environment are much greater than those of the third-person environment and the perfect observation environment.

[0054] Further, in the embodiments of the present application, when training the decision-making model, a two-stage learning training mode is adopted, in the first stage, the target decision-making model is trained using the hindsight proximal policy optimization method, the target decision-making model is used to train the moving action strategy of the target object, in the second stage, the target decision-making model is used as a teacher model of the decision-making model to supervise the learning training of the decision-making model, to learn the target task action strategy and the target region that the target object should reach at the next moment. This multi-stage training mode can reduce the training difficulty and improve the training efficiency.

[0055] Exemplarily, the decision-making model training method and the strategy control method of the target object provided in the embodiments of the present application can be applied to game scenes such as multi-player game, such as multi-player shooting game, multi-player chase game, multi-player hide-and-seek game, etc., and the decision-making model obtained through reinforcement learning is used to control the target object in the game to fight with other target objects.

[0056] Exemplarily, taking the game environment as an example for the target environment and taking the agent as an example for the target object, when the decision-making model training method provided in the embodiments of the present application is applied, the decision-making model trained by the decision-making model training method provided in the embodiments of the present application can be loaded into the multi-player game scene to help the player improve the effective game experience. So that the agent can perform the strategy in a variety of ways as the human player does without cheating, modifying or obtaining the game background parameters, thereby increasing the game playability.

[0057] It can be understood that, in the embodiments of the present application, the target environment is taken as an example for the game environment, and the decision-making model training method and the strategy control method of the target object provided in the embodiments of the present application can also be applied to other scenes, which are not limited in the embodiments of the present application.

[0058] Exemplarily, Figure 1 The application scenario of the decision-making model training method and the strategy control method of the target object provided in the embodiments of the present application is shown in FIG. Figure 1 As shown in FIG. 1, the application scenario of the embodiments of the present application involves a server 1 and a terminal device 2, the terminal device 2 can communicate data with the server 1 through wired network or wireless network.

[0059] In some possible implementation manners, the terminal device 2 refers to a type of device that has rich human-computer interaction modes, has the capability of accessing the Internet, is usually equipped with various operating systems, has strong processing capability, and the like. The terminal device 2 can be a terminal device such as a smart phone, a tablet computer, a portable notebook computer, a desktop computer, and the like, or a telephone watch, but is not limited thereto.

[0060] Figure 1 The server 1 in the method can be a standalone physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The present application does not make any limitation in this regard. In the embodiments of the present application, the server 1 can be a background server of a game application in the terminal device 2, or a server of a game platform.

[0061] In some possible implementation manners, Figure 1 The terminal device and the server are exemplarily shown, and actually, a larger number of terminal devices and servers can be included, and the present application does not make any limitation in this regard.

[0062] Exemplarily, the terminal device 2 can install and run a game client or other clients. Exemplarily, taking the game client as an example, the terminal device 2 can obtain a selected opponent for a game from the server 1, and after a target opponent is selected by a player, the terminal device 2 controls a target object corresponding to the target opponent to fight against a game character controlled by the player by using a trained decision model. The decision model is trained by using the decision model training method provided in the embodiments of the present application.

[0063] The decision model training method provided in the embodiments of the present application can be executed by the server 1, or can be executed by other servers or other terminal devices. Optionally, the strategy control method of the target object provided in the embodiments of the present application can be executed by the server 1, or can be executed by the terminal device 2, or can be executed by the server 1 and the terminal device 2 in cooperation.

[0064] The technical solutions of the present application will be described in detail as follows:

[0065] Figure 2 A flowchart of a decision model training method provided in the embodiments of the present application is shown in FIG. 1. The execution subject of the method can be a server, such as the server 1 shown in FIG. 1, and the method can include the following steps. Figure 2

[0066] ​S101, obtain training data of a current state of a target object in a target environment, the training data of the current state comprising N pieces of training information, any t-th piece of training information in the N pieces of training information comprising state information of the t-th time, a control strategy of the t-th time, and a predicted reward of the control strategy of the t-th time, the state information of the t-th time comprising a real-time reward of the t-th time fed back by the target environment, observation information of the t-th time, and a region identifier of a region reached by the target object at the t-th time, the control strategy of the t-th time and the predicted reward of the control strategy of the t-th time being outputs of a decision model input with the state information of the t-th time, the control strategy of the t-th time comprising a moving action strategy, a target task action strategy, and a target region identifier of the t-th time, wherein the target region identifier of the t-th time is the same as a region identifier of a region reached by the target object at a t+1-th time, and t and N are positive integers.

[0067] Specifically, in the process of training the decision model, one piece of training data of a state of the target object in the target environment is used to update parameters of the decision model once, the one piece of training data of the state can comprise N pieces of training information, that is, the decision model makes N times of control decisions, and one piece of training information can be training information of one time.

[0068] In the embodiment of the application, for any t-th time in the N times, in the process of training the decision model, the state information of the t-th time is input into the decision model, and the control strategy of the t-th time and the predicted reward of the control strategy of the t-th time are output, the state information of the t-th time comprises the real-time reward of the t-th time fed back by the target environment, the observation information of the t-th time, and the region identifier of the region reached by the target object at the t-th time, the control strategy of the t-th time comprises the moving action strategy, the target task action strategy, and the target region identifier of the t-th time, wherein the target region identifier of the t-th time is the same as the region identifier of the region reached by the target object at the t+1-th time, that is, the target region identifier of the t-th time is input as the region identifier of the region reached by the target object at the next t-th time. The state information of the t-th time, the control strategy of the t-th time, and the predicted reward of the control strategy of the t-th time constitute the training information of the t-th time, and the training information of the N times constitutes the training data of the current state of the target object.

[0069] In the embodiment of the application, the moving action strategy can be angle adjustment, action displacement, left turn, right turn, or posture adjustment, and the like.

[0070] In the embodiment, taking the target environment as a game environment as an example, in a game scene, the target task can be a shooting task, a searching task, or an evading task. Correspondingly, the target task action strategy can be shooting, searching, or evading.

[0071] Before taking the state information at time t as the input of the decision model, the state information at time t is obtained. As an implementable way, obtaining the state information at time t can specifically include:

[0072] S1011, obtaining the real-time reward of the target environment feedback at time t.

[0073] Optionally, obtaining the real-time reward of the target environment feedback at time t can specifically be: using the control strategy at time t-1 to control the target object to perform corresponding actions. It is determined that the target object has completed the target task and obtained the real-time reward of the target environment feedback at time t. According to the observation information at time t and the real-time reward of the target environment feedback, the feature information hidden variable is generated, and the feature information hidden variable is stored in the feature information statistical table to obtain an updated feature information statistical table. The updated feature information statistical table is normalized, and the normalized feature information statistical table is determined as the real-time reward of the target environment feedback at time t.

[0074] S1012, obtaining the observation information of the target object at time t.

[0075] S1013, taking the target arrival region identifier of the target object at time t-1 as the region identifier where the target object arrives at time t.

[0076] Wherein, if the output of the decision model is the first control decision, that is, t is 1, at this time the region identifier where the target object arrives at time t can be a preset region identifier.

[0077] S1014, taking the real-time reward of the target environment feedback at time t, the observation information at time t, and the region identifier where the target object arrives at time t as the state information of the target object at time t.

[0078] S102, constructing a target loss function according to the training data of the current state.

[0079] S103, adjusting and updating the parameters of the decision model according to the target loss function.

[0080] Specifically, adjusting and updating the parameters of the decision model according to the target loss function can be adjusting and updating the parameters of the decision model through back propagation.

[0081] S104, inputting the state information of the target object at time N+1 into the decision model with updated parameters, and continuing to train the decision model according to the training data of the next state of the target object until the stopping condition is met, to obtain a trained decision model.

[0082] In an implementable manner in the embodiments of the present application, in the decision model training, the training of the decision model is performed by adopting a two-stage learning training mode. Before S101, the method of the present embodiment can further include:

[0083] S105, training the target decision model using the hindsight proximal policy optimization method, the target decision model being used to train the moving action policy of the target object.

[0084] Specifically, the hindsight proximal policy optimization (HPPO) method modifies the reward score obtained by the target object by replacing the un-reached area with the actually reached area, so that the target object obtains pseudo reward information, so that the target object has a higher opportunity to train using existing data, and improves the data utilization efficiency.

[0085] Specifically, the target decision model is trained using the hindsight proximal policy optimization method, so as to first train the moving action policy of the target object, obtain the target decision model whose moving action policy of the target object is trained, and then use the target decision model as a teacher model of a decision model to be trained, and the decision model as a student model, so as to supervise the learning training of the decision model, learn the target task action policy and the target area that the target object should reach at the next time.

[0086] Optionally, when the target decision model is trained using the hindsight proximal policy optimization method, it can specifically include: first, obtaining first training data of a current state of the target object in the target environment, the first training data of the current state including training information of T time points, Any training information of t time point in the training information of T time points includes first state information of t time point, moving action policy of t time point and predicted reward of t time point, the first state information of t time point includes real-time reward of t time point, observation information of t time point and randomly sampled area identifier fed back by the target environment, the first state information of t time point is input into the target decision model, and the moving action policy of t time point and the predicted reward of t time point are output, the predicted reward of t time point is the reward predicted by the target decision model according to the first state information of t time point and the actually reached area of t time point, wherein is a preset positive integer. Then, a hindsight proximal policy optimization loss function is constructed according to the training data of the current state, the parameters of the target decision model are adjusted and updated according to the constructed hindsight proximal policy optimization loss function, the training information of the T+1 time point of the target object is input into the target decision model after the parameters are updated, then the input and output training of T time points is performed, the first training data of the next state of the target object is obtained, then the target decision model is continuously trained according to the first training data of the next state of the target object, until the stop training condition is met, and the trained target decision model is obtained.

[0087] The input of the target decision model is first state information at time t, the first state information includes real-time reward of the target environment feedback at time t, observation information at time t and randomly sampled region identifier, the output of the target decision model is a moving action strategy at time t and predicted reward at time t, the predicted reward at time t is a reward predicted by the target decision model according to the first state information at time t and the actual arrived region at time t.

[0088] Optionally, the post-hoc proximal policy optimization loss function (also referred to as a constraint target) is as shown in the following formula:

[0089]

[0090] In the formula, and respectively represent the randomly sampled region identifier and the actual arrived region identifier, is the target decision model parameter to be updated this time, is the target decision model parameter before this update, is the target decision model parameter after this update, is the advantage function calculated by the target decision model before this update, the advantage function is equal to the real-time reward r t of the target environment feedback at time t minus the reward V t predicted by the target decision model according to the state information s t at time t and the actual arrived region identifier , that is, r t - V t (s t , a ).a k is the moving action strategy. is an importance sampling weight coefficient, represents the probability of each type of moving action a k predicted by the target decision model before this update according to the state information s k at time k and the actual arrived region identifier , represents the probability of each type of moving action a k predicted by the target decision model before this update according to the state information s k at time k and the randomly sampled region identifier . represents the probability of each type of moving action a k predicted by the target decision model parameter after this update according to the state information s 0, at time k and the actual arrived region identifier . The obtained is the parameter The loss function. In this embodiment, the method, when training the movement control policy, calculates the loss function for each... Update the target decision model parameters every step. Also known as controlling the number of times a decision is made. The first training data representing the current state is (s0, a). 0, r 0, s1, a 1, r 1,…... Taking t=0 as an example, s0 represents the state information at time 0, a0 represents the movement action strategy output by the target decision model at time 0, and r0 represents the real-time reward from the target environment at time 0. Indicates calculation The mean of the post-processing near-end policy optimization loss function corresponding to the training information at each time step, i.e., for The average of the summations of the post-hoc near-end policy optimization loss functions corresponding to the training information at each time step is taken.

[0091] To enhance the stability of the algorithm, a pruning constraint is used in this embodiment. It refers to the clipping range. In the formula above, clip indicates the range calculated from the current value. Less than 1- Then the importance sampling weight coefficient is 1- If the current Greater than 1+ Then the importance sampling weight coefficient is 1+ If the current If the value is greater than 0.8 and less than 1.2, then the importance sampling weight coefficient is taken as follows: This makes the importance sampling weight coefficient in (1- )-(1+ Between ) . For example, it can be set to This ensures that the importance sampling weight coefficient is between 0.8 and 1.2, so that the maximum value of the loss function is no higher than 1.2 times the advantage function and the minimum value is no lower than 0.8 times the advantage function.

[0092] S106. The target decision model is identified as the teacher model of the decision model, so as to conduct supervised learning training on the decision model.

[0093] Optionally, the decision model in this embodiment can be a neural network model.

[0094] In this embodiment, through the multi-stage training manner, the first stage uses the hindsight proximal policy optimization method to train the target decision model, and the target decision model is used to train the moving action policy of the target object. In the second stage, the target decision model is used as a teacher model of the decision model to supervise the learning training of the decision model, and learn the target task action policy and the target region that the target object should reach at the next time. Thus, the training difficulty can be reduced and the training efficiency can be improved.

[0095] Optionally, the target loss function is constructed according to the training data of the current state in S102, which can specifically include:

[0096] S1021, constructing a proximal policy optimization loss function according to the training data of the current state.

[0097] As an implementable manner, the proximal policy optimization loss function is constructed according to the training data of the current state, which can specifically be:

[0098] S1, for the training information of each t time in N times, according to the control policy of t time, the predicted reward of the control policy of t time, the real-time reward of t time fed back by the target environment, the observation information of t time and the region identifier of the target object reached at t time, the proximal policy optimization loss function corresponding to the training information of t time is calculated.

[0099] Optionally, the proximal policy optimization loss function corresponding to the training information of t time is calculated according to the control policy of t time, the predicted reward of the control policy of t time, the real-time reward of t time fed back by the target environment, the observation information of t time and the region identifier of the target object reached at t time, which can specifically be:

[0100] First, the predicted action probability ratio is determined according to the real-time reward of t time fed back by the target environment, the observation information of t time, the region identifier of the target object reached at t time and the control policy of t time.

[0101] Then, the difference between the real-time reward of t time fed back by the target environment and the predicted reward of the control policy of t time is calculated. According to the predicted action probability ratio and the difference between the real-time reward of t time fed back by the target environment and the predicted reward of the control policy of t time, the proximal policy optimization loss function corresponding to the training information of t time is calculated.

[0102] As an implementable manner, the proximal policy optimization loss function corresponding to the training information of t time can be as shown in the following formula:

[0103]

[0104] wherein, For predicting the action probability ratio, the predicted action probability ratio is the ratio of the probability of each type of action a t predicted by the decision model after the current update of the parameters according to the state information s t at time t before the current update of the parameters according to the state information s t predicted by the decision model after the current update of the parameters according to the state information s t at time t before the current update of the parameters according to the state information s is the parameter of the decision model after the current update, is the parameter of the decision model before the current update. is the difference between the real-time reward of time t fed back by the target environment and the predicted reward of the control strategy of time t. In order to enhance the stability of the algorithm, clipping constraints are adopted in the embodiment, is the clipping range, and in the above formula, clip represents that if the current is less than 1- , the predicted action probability ratio is taken as 1- , if the current is greater than 1+ , the predicted action probability ratio is taken as 1+ , if the current is greater than 0.8 and less than 1.2, the predicted action probability ratio is taken as , so that the predicted action probability ratio is between (1- ) and (1+ ). For example, it can be set to , so that the predicted action probability ratio is between 0.8 and 1.2, the maximum value of the loss function is not higher than 1.2 times of , and the minimum value is not lower than 0.8 times of .

[0105] In an implementable manner, the real-time reward of time t fed back by the target environment is the sum of the reward of the exploration strategy of time t and the reward of the utilization strategy of time t, wherein the reward of the exploration strategy of time t is determined according to the number of reached areas at time t and the target reached area at time t, and the reward of the utilization strategy of time t is determined according to the total number of target tasks completed at time t and the target reached area at time t.

[0106] Optionally, the real-time reward r t of time t fed back by the target environment can be as shown in the following formula:

[0107] r t = r t explore + r texploit

[0108] ;

[0109] ;

[0110] wherein r t explore is the reward of the exploration strategy at time t, r t exploit is the reward of the exploitation strategy at time t, is the number of regions reached at time t, is the target region at time t, is the total number of target tasks completed at time t.

[0111] wherein Softmax is a normalized exponential function, and if Vj represents the i-th element in V, then the softmax value of this element is S i :

[0112]

[0113] Substituting i in the above formula with F1(g t ) gives the result r t exploit .

[0114] S2, determining the mean of the proximal policy optimization loss function corresponding to the training information of the N time points as the proximal policy optimization loss function.

[0115] S1022, constructing a supervised learning loss function according to the training data of the current state and the teacher model.

[0116] As an implementable way, the supervised learning loss function can be constructed according to the training data of the current state and the teacher model, and specifically can be:

[0117] For the training information of each t time point in the N time points, the cross-entropy loss function of the moving action strategy in the control strategy at time t and the moving action strategy output by the corresponding teacher model at time t is calculated. The mean of the cross-entropy loss function corresponding to the training information of the N time points is determined as the supervised learning loss function.

[0118] S1023, determining the weighted sum of the proximal policy optimization loss function and the supervised learning loss function as the target loss function.

[0119] Exemplarily, the target loss function may be shown in the following formula:

[0120]

[0121] wherein, is a proximal policy optimization loss function, is a supervised learning loss function, and are two weights, and sum up to 1, for example and are both 0.5.

[0122] In the embodiments of the present application, before the state information of the target object at the N+1 moment is input into the decision model after updating the parameters in S104, the method of the embodiments of the present application can further include: obtaining the state information of the target object at the N+1 moment. The obtaining of the state information at the N+1 moment can be specifically:

[0123] S1041, obtaining the real-time reward of the target environment feedback at the N+1 moment.

[0124] Optionally, the real-time reward of the target environment feedback at the N+1 moment is obtained, which can be specifically:

[0125] The control strategy at the N moment is adopted to control the target object to perform corresponding actions, it is determined that the target object completes the target task at the N+1 moment and obtains the real-time reward of the target environment feedback, the feature information hidden variable is generated according to the observation information at the N+1 moment and the real-time reward of the target environment feedback, and the feature information hidden variable is stored in the feature information statistics table to obtain the updated feature information statistics table. The updated feature information statistics table is normalized, and the normalized feature information statistics table is determined as the real-time reward of the target environment feedback at the N+1 moment.

[0126] S1042, obtaining the observation information of the target object at the N+1 moment.

[0127] S1043, taking the target arrival area identifier of the target object at the N moment as the area identifier of the target object arrival at the N+1 moment.

[0128] S1044, the real-time reward of the target environment feedback at the N+1 moment, the observation information at the N+1 moment and the area identifier of the target object arrival at the N+1 moment constitute the state information of the target object at the N+1 moment.

[0129] In an implementable manner, the decision model after inputting the state information of the target object at the N+1 moment into the updated parameter in S104, the parameter-adjusted decision model is used for: respectively extracting features of the real-time reward of the target environment feedback at the N+1 moment, the observation information at the N+1 moment and the region identifier of the target object at the N+1 moment, to obtain the first hidden variable, the second hidden variable and the third hidden variable. The first hidden variable, the second hidden variable and the third hidden variable are combined to obtain the target hidden variable. The target hidden variable is extracted to obtain target hidden variable feature information. The control strategy at the N+1 moment is generated according to the target hidden variable feature information.

[0130] In the embodiment of the application, the state information at the t moment includes the region identifier of the target object at the t moment, so before S101, the map corresponding to the target environment needs to be regionally divided to obtain a plurality of regions and assign an identifier to each region and save the assigned identifier. Correspondingly, the method of the embodiment can further include, before S101:

[0131] S107, obtaining vertical angle information of the map corresponding to the target environment.

[0132] S108, regionally dividing the map according to the vertical angle information of the map to obtain a plurality of regions.

[0133] S109, assigning an identifier to each region.

[0134] As an implementable manner, S108 regionally divides the map according to the vertical angle information of the map to obtain a plurality of regions, which can specifically include:

[0135] S1081, determining the center point of each region in the plurality of regions according to the vertical angle information of the map and the center point selection method, and the center point selection method is that: the distance between two adjacent center points is the same, and the center point exists in the open region with regular corners and shapes.

[0136] S1082, for each target point in the map except the determined center point, determining the region to which the target point belongs according to the principle that the Euclidean distance between the target point and the center point is the shortest, to obtain a plurality of regions.

[0137] Correspondingly, S109 assigns an identifier to each region, which can specifically be: assigning an identifier to the center point of each region. Optionally, assigning an identifier to the center point of each region can be numbering each center point and taking the one-hot coding result of the number as the identifier of the center point.

[0138] Optionally, the observation information in this application embodiment can be first-person observation information, which is the target environment information that the target object can observe. This provides a decision-making method for the target object in a first-person multiplayer game environment, enabling the target object to obtain diverse strategies. The reasoning difficulty and information acquisition difficulty in the first-person environment are relatively high.

[0139] The decision model training method provided in this embodiment uses the real-time reward at time t from the target environment, the observation information at time t, and the region identifier reached by the target object at time t as inputs to the decision model at time t. The decision model makes decisions based on these three pieces of information and outputs a control strategy at time t (including the movement strategy at time t, the target task action strategy, the target region the target object should reach in the next moment (i.e., the region indicated by the target arrival region identifier at time t), and the predicted reward for the control strategy at time t). The real-time reward at time t from the target environment is the reward obtained by the target object after performing a target task with an unknown opponent. The decision model learns not only the movement strategy but also the target task action strategy and the target region the target object should reach in the next moment. This target region is used to control the target object's movement and target task in a specific area in the next moment, allowing the decision model to learn multiple control strategies. Therefore, the target object can adjust its action decisions in real time when facing different opponent targets, achieving diverse strategies in a multi-player game environment. Furthermore, it can be adapted to various application scenarios (such as first-person or third-person perspectives), does not require real opponent information during training, reduces training costs, and expands the application scope.

[0140] The following is combined Figure 3 The decision model training method provided in this application will be described in detail using a specific embodiment.

[0141] Figure 3 This application provides a flowchart of a decision model training method, the execution entity of which can be a server, such as... Figure 3 As shown, the method may include:

[0142] S201. Obtain the vertical perspective information of the map corresponding to the target environment.

[0143] In this embodiment, the target environment is a game environment as an example. Accordingly, the target object can be an intelligent agent.

[0144] S202. Divide the map into regions based on the vertical perspective information of the map to obtain multiple regions.

[0145] S203. Assign an identifier to each region and save the correspondence between regions and identifiers.

[0146] In one embodiment, the map is divided into regions based on the map's vertical perspective information, resulting in multiple regions. Specifically, this may include:

[0147] S2011. Based on the vertical perspective information of the map and the center point selection method, determine the center point of each area in multiple areas. The center point selection method is as follows: the distance between two adjacent center points is the same, and there is a center point in corners and open areas with regular shapes.

[0148] S2012. For each target point on the map other than the determined center point, determine the region to which the target point belongs based on the principle of the shortest Euclidean distance between the target point and the center point, thus obtaining multiple regions.

[0149] Accordingly, in S203, an identifier is assigned to each region. Specifically, an identifier is assigned to the center point of each region. Optionally, assigning an identifier to the center point of each region can be done by assigning a numerical number to the center point of each region and using the one-hot encoding result of the numerical number as the identifier of the center point.

[0150] For example, Figure 4 This is a schematic diagram of a vertical view map corresponding to a target environment. Figure 5 To Figure 4 The diagram shown is an illustration of a map after regional division from a vertical perspective. Figure 5 As shown, in this embodiment, Figure 3 The vertical view map shown is divided into 20 areas ( Figure 5 As shown in the numbers 0-19, the single-hot encoding result of each number is used as the identifier of each corresponding area.

[0151] S204. Input the state information of the target object in the target environment at time t into the decision model, and output the control strategy at time t and the predicted reward of the control strategy at time t. The state information at time t includes the real-time reward at time t fed back by the target environment, the observation information at time t, and the area identifier reached by the target object at time t. The control strategy at time t includes the movement action strategy, the target task action strategy, and the target area identifier reached at time t.

[0152] Specifically, before inputting the state information of the target object at time t in the target environment into the decision model, it is necessary to obtain the state information at time t. Obtaining the state information at time t can specifically include:

[0153] S2041. Obtain the real-time reward at time t from the target environment feedback.

[0154] Optionally, the real-time reward of the target environment feedback at time t is obtained, which can be specifically: using the control strategy at time t-1 to control the target object to perform corresponding actions. It is determined that the target object completes the target task at time t and obtains the real-time reward of the target environment feedback. The feature information hidden variable is generated according to the observation information at time t and the real-time reward of the target environment feedback, and the feature information hidden variable is stored in the feature information statistical table to obtain an updated feature information statistical table. The updated feature information statistical table is normalized, and the normalized feature information statistical table is determined as the real-time reward of the target environment feedback at time t.

[0155] S2042, the observation information of the target object at time t is obtained.

[0156] S2043, the target arrival area identifier of the target object at time t-1 is taken as the area identifier arrived by the target object at time t.

[0157] Wherein, if the output of the decision model is the first control decision, that is, t is 1, at this time the area identifier arrived by the target object at time t can be a preset area identifier.

[0158] S2044, the real-time reward of the target environment feedback at time t, the observation information at time t and the area identifier arrived by the target object at time t are taken as the state information of the target object at time t.

[0159] The following will be described in detail Figure 6 After the state information at time t is input into the decision model, the processing process of the decision model, Figure 6 A process diagram of the decision model outputting the control decision at time t according to the input state information at time t provided by the embodiment of the application is shown in Figure 6 As shown in the figure, the process of the decision model outputting the control decision at time t according to the input state information at time t can include:

[0160] S11, the decision model respectively extracts features from the real-time reward of the target environment feedback at time t, the observation information at time t and the area identifier arrived by the target object at time t, to obtain first, second and third hidden variables.

[0161] In the embodiment, optionally, the above feature extraction can also introduce a self-attention mechanism for different parts of the input to further extract features, or use a long short-term memory network (LSTM) to enable the network to process time sequence features to enhance performance.

[0162] S12, the decision model combines the first, second and third hidden variables to obtain a target hidden variable.

[0163] S13, the decision model extracts features of the target hidden variable to obtain target hidden variable feature information.

[0164] S14, the decision model generates a control strategy at time t according to the target hidden variable feature information.

[0165] Specifically, according to the target hidden variable feature information, three independent neural network strategy modules can be generated, including a movement control strategy module, a target task (such as shooting / searching / avoiding) control strategy module, and a target arrival area output strategy module. These three modules respectively output a movement action strategy, a target task action strategy, and a target arrival area identifier at time t. The target arrival area identifier at time t is used as the identifier of the area where the target object arrives at the next time (i.e., time t+1).

[0166] When the decision model generates and outputs the control strategy at time t, it will also output the predicted reward of the control strategy at time t.

[0167] S205, using the control strategy at time t to control the target object to perform an action, obtaining a real-time reward at time t+1 fed back by the target environment, and using the target arrival area identifier at time t as the identifier of the area where the target object arrives at time t+1.

[0168] S206, inputting the state information of the target object at time t+1 in the target environment into the decision model, and outputting a control strategy at time t+1 and a predicted reward of the control strategy at time t+1.

[0169] S207, when the Nth control decision is completed, obtaining training data of the current state, the training data of the current state including training information at N time points, and the training information at any time t of the N time points including state information at time t, a control strategy at time t, and a predicted reward of the control strategy at time t.

[0170] Specifically, for example, from time t to time t+N, training information at these N time points is obtained.

[0171] S208, constructing a target loss function according to the training data of the current state.

[0172] S209, adjusting and updating the parameters of the decision model according to the target loss function.

[0173] S210, inputting the state information of the target object at time N+1 into the decision model with updated parameters, and continuing to train the decision model according to the training data of the next state of the target object until a stop training condition is met, to obtain a trained decision model.

[0174] Specifically, the detailed process of S208-S210 can be found in Figure 2The descriptions in the illustrated embodiments will not be repeated here.

[0175] The method in this embodiment enables a target object, in any environment during multi-player games, to record its current reward using a feature information statistics table after interacting with other unknown opponents (such as shooting, searching, or dodging and gaining reward points). This information is then input into a decision model for feature extraction. The decision model then determines the target area the target object should reach in the next moment. This target area controls the target object's movement and target tasks within a specific area, thus achieving strategic diversity when facing different opponents. This method enables the target object to exhibit effective and diverse strategies against unknown opponents in multi-target game environments with arbitrary perspectives, maximizing its own task rewards. It reduces technical implementation costs, eliminates the need for opponent information for training and application, and expands the practical application scope and scenarios.

[0176] Figure 7 A flowchart illustrating a strategy control method for a target object provided in this application embodiment. The execution entity of this method can be a server, such as... Figure 7 As shown, the method may include:

[0177] S301. Obtain the current state information of the target object in the target environment.

[0178] S302. Input the current state information into the decision model and output the control strategy for the current moment.

[0179] Among them, the decision model is based on Figure 2 or Figure 3 The decision model was trained using the method shown.

[0180] S303. Use the control strategy at the current moment to control the target object to perform actions.

[0181] The strategy control method for a target object provided in this embodiment obtains the current state information of the target object in the target environment, inputs the current state information into a decision model, outputs the current control strategy, and uses the current control strategy to control the target object's actions. This allows the target object to adjust its action decisions in real time when facing different opponent target objects, realizing diverse strategies for the target object in a multi-player game environment.

[0182] By using the agent strategy control method provided in this embodiment, the agent is controlled to conduct one-on-one comparisons with other opposing agents in the same game environment. The opposing agents are, for example, F1, Axon, Marvin, Yanshi, or other agents. In most game environments, it has shown better performance.

[0183] Figure 8 A structural schematic diagram of a decision model training device provided by an embodiment of the present application is shown in FIG. 1. The device can include an acquisition module 11, a construction module 12, an adjustment module 13, and a processing module 14. Figure 8

[0184] The acquisition module 11 is configured to acquire training data of a current state of a target object in a target environment. The training data of the current state includes N pieces of training information. Any piece of training information at a time t of the N pieces of training information includes state information at the time t, a control policy at the time t, and a predicted reward of the control policy at the time t. The state information at the time t includes a real-time reward at the time t fed back by the target environment, observation information at the time t, and a region identifier of a region reached by the target object at the time t. The control policy at the time t and the predicted reward of the control policy at the time t are outputs of a decision model input with the state information at the time t. The control policy at the time t includes a moving action policy, a target task action policy, and a target region identifier at the time t. The target region identifier at the time t is the same as a region identifier of a region reached by the target object at a time t+1. t and N are positive integers.

[0185] The construction module 12 is configured to construct a target loss function according to the training data of the current state.

[0186] The adjustment module 13 is configured to adjust and update parameters of the decision model according to the target loss function.

[0187] The processing module 14 is configured to input state information at a time N+1 of the target object into the decision model with the updated parameters, continue training the decision model according to training data of a next state of the target object, until a stop training condition is met, and obtain a trained decision model.

[0188] Optionally, the acquisition module 11 is configured to:

[0189] train a target decision model using a hindsight proximal policy optimization method, the target decision model being configured to train a moving action policy of the target object.

[0190] determine the target decision model as a teacher model of the decision model, to perform supervised learning training on the decision model.

[0191] Optionally, the construction module 12 is configured to:

[0192] construct a proximal policy optimization loss function according to the training data of the current state.

[0193] construct a supervised learning loss function according to the training data of the current state and the teacher model.

[0194] ​A weighted sum of the proximal policy optimization loss function and the supervised learning loss function is determined as the target loss function.

[0195] Optionally, the constructing module 12 is specifically configured to:

[0196] For the training information of each t moment in the N moments, the proximal policy optimization loss function corresponding to the training information of the t moment is calculated according to the control strategy of the t moment, the predicted reward of the control strategy of the t moment, the real-time reward of the t moment fed back by the target environment, the observation information of the t moment and the region identifier of the target object arrived at the t moment;

[0197] The mean of the proximal policy optimization loss functions corresponding to the training information of the N moments is determined as the proximal policy optimization loss function.

[0198] Optionally, the constructing module 12 is specifically configured to:

[0199] The predicted action probability ratio is determined according to the real-time reward of the t moment fed back by the target environment, the observation information of the t moment, the region identifier of the target object arrived at the t moment and the control strategy of the t moment.

[0200] The difference between the real-time reward of the t moment fed back by the target environment and the predicted reward of the control strategy of the t moment is calculated.

[0201] The proximal policy optimization loss function corresponding to the training information of the t moment is calculated according to the predicted action probability ratio and the difference.

[0202] Optionally, the real-time reward of the t moment fed back by the target environment is the sum of the reward of the exploration strategy of the t moment and the reward of the utilization strategy of the t moment.

[0203] The reward of the exploration strategy of the t moment is determined according to the number of regions arrived at the t moment and the target arrived region of the t moment.

[0204] The reward of the utilization strategy of the t moment is determined according to the total number of target tasks completed at the t moment and the target arrived region of the t moment.

[0205] Optionally, the constructing module 12 is specifically configured to:

[0206] For the training information of each t moment in the N moments, the cross-entropy loss function of the moving action strategy in the control strategy of the t moment and the moving action strategy of the t moment output by the corresponding teacher model is calculated.

[0207] The mean of the cross-entropy loss functions corresponding to the training information of the N moments is determined as the supervised learning loss function.

[0208] Optionally, the acquiring module 11 is further configured to acquire the real-time reward of the (N+1)th moment fed back by the target environment.

[0209] obtaining observation information of the target object at the N+1 moment;

[0210] taking the target arrival area identifier of the target object at the N moment as the area identifier where the target object arrives at the N+1 moment;

[0211] taking the real-time reward of the target environment feedback at the N+1 moment, the observation information at the N+1 moment and the area identifier where the target object arrives at the N+1 moment as the state information of the target object at the N+1 moment.

[0212] Optionally, the obtaining module 11 is specifically configured to:

[0213] adopting the control strategy at the N moment to control the target object to perform corresponding actions;

[0214] determining that the target object completes the target task at the N+1 moment and obtains the real-time reward of the target environment feedback;

[0215] generating feature information hidden variables according to the observation information at the N+1 moment and the real-time reward of the target environment feedback, and storing the feature information hidden variables into a feature information statistical table to obtain an updated feature information statistical table;

[0216] performing normalization processing on the updated feature information statistical table, and determining the normalized feature information statistical table as the real-time reward of the target environment feedback at the N+1 moment.

[0217] Optionally, the decision model after the adjustment of the parameters is configured to:

[0218] respectively performing feature extraction on the real-time reward of the target environment feedback at the N+1 moment, the observation information at the N+1 moment and the area identifier where the target object arrives at the N+1 moment to obtain first hidden variables, second hidden variables and third hidden variables;

[0219] merging the first hidden variables, the second hidden variables and the third hidden variables to obtain target hidden variables;

[0220] performing feature extraction on the target hidden variables to obtain target hidden variable feature information;

[0221] generating the control strategy at the N+1 moment according to the target hidden variable feature information.

[0222] Optionally, the obtaining module 11 is further configured to:

[0223] obtaining vertical viewing angle information of a map corresponding to the target environment;

[0224] performing region division on the map according to the vertical viewing angle information of the map to obtain a plurality of regions;

[0225] Assign an identity to each region.

[0226] Figure 9 A structural schematic diagram of a policy control device of a target object provided by an embodiment of the present application is shown in Figure 9 The device can include an acquisition module 21, a control policy output module 22, and an action control module 23, wherein,

[0227] The acquisition module is configured to acquire state information of the target object at a current moment in a target environment.

[0228] The control policy output module is configured to input the state information at the current moment into a decision model, and output a control policy at the current moment, the decision model being trained according to a decision model training method shown in Figure 2

[0229] The action control module is configured to control the target object to perform an action by using the control policy at the current moment.

[0230] It should be understood that the device embodiments and the method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, details are not described here. Specifically, Figure 8 The decision model training device shown in Figure 9 The policy control device of the target object shown in can perform the method embodiments corresponding to the computer device, and the aforementioned and other operations and / or functions of each module in the device are respectively for implementing the method embodiments corresponding to the computer device. For brevity, details are not described here.

[0231] The decision model training device and the policy control device of the target object of the embodiments of the present application are described above from the perspective of functional modules in combination with the drawings. It should be understood that the functional modules can be implemented in the form of hardware, or in the form of instructions of software, or in the form of a combination of hardware and software modules. Specifically, each step of the method embodiments in the embodiments of the present application can be completed by integrated logic circuits and / or software instructions in the hardware of the processor, and the steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processors for execution, or executed by a combination of hardware and software modules in the code processor. Alternatively, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory, and combines the hardware to complete the steps in the above method embodiments.

[0232] Figure 10 is a schematic block diagram of a computer device 700 provided by an embodiment of the present application.

[0233] As Figure 10 ​As shown, the computer device 700 can include:

[0234] The memory 710 is configured to store a computer program, and transmit the program code to the processor 720. In other words, the processor 720 can call and run the computer program from the memory 710 to implement the method in the embodiments of the present application.

[0235] For example, the processor 720 can be configured to execute the method embodiments described above according to the instructions in the computer program.

[0236] In some embodiments of the present application, the processor 720 can include but is not limited to:

[0237] A general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and the like.

[0238] In some embodiments of the present application, the memory 710 includes but is not limited to:

[0239] The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), synch link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0240] In some embodiments of the present application, the computer program can be divided into one or more modules, which are stored in the memory 710 and executed by the processor 720 to complete the method provided by the present application. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the electronic device.

[0241] As shown in Figure 9 The computer device can further include:

[0242] The transceiver 730 can be connected to the processor 720 or the memory 710.

[0243] The processor 720 can control the transceiver 730 to communicate with other devices, specifically, can send information or data to other devices, or receive information or data sent by other devices. The transceiver 730 can include a transmitter and a receiver. The transceiver 730 can further include an antenna, and the number of antennas can be one or more.

[0244] It should be understood that the various components within the electronic device are connected by a bus system, which includes, in addition to a data bus, a power bus, a control bus, and a state signal bus.

[0245] The application also provides a computer storage medium, which stores a computer program, and the computer program enables a computer to execute the method of the method embodiment when executed by the computer. Alternatively, the application embodiment also provides a computer program product containing instructions, and the instructions enable the computer to execute the method of the method embodiment when executed by the computer.

[0246] When implemented by using software, the computer program product can be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. When loaded and executed by a computer, the computer program instructions produce the flow or function of the embodiment of the application in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through a wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, digital video disc (DVD)), or a semiconductor medium (for example, solid state disk (SSD)) and the like.

[0247] Those skilled in the art can realize that the modules and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0248] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the division of the above-described device embodiments is only a logical function division, and there can be another division manner for the actual implementation, for example, multiple devices or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different parts can be indirect couplings or communication connections through some interfaces, devices or modules, and can be electrical, mechanical or in other forms.

[0249] The modules explained as separated components can or can not be physically separated, and the components shown as modules can or can not be physical modules, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. For example, the functional modules in the embodiments of the present application can be integrated into a processing module, or each module can be physically present separately, or two or more modules can be integrated into one module.

[0250] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for training a decision model, characterized in that, include: Training data for the current state of the target object in the target environment is obtained. This training data includes training information from N time points. The training information at any time point t includes the state information at time t, the control policy at time t, and the predicted reward of the control policy at time t. The state information at time t includes the real-time reward from the target environment at time t, the observation information at time t, and the region identifier reached by the target object at time t. The control policy at time t and the predicted reward of the control policy at time t are input to the output of the decision model based on the state information at time t. The control policy at time t includes a movement action policy, a target task action policy, and the predicted reward at time t. The target arrival area identifier at time t is the same as the area identifier reached by the target object at time t+1. t and N are both positive integers. The real-time reward from the target environment at time t is the reward obtained by the target object after performing a target task with an unknown opponent. The area indicated by the target arrival area identifier at time t is the target area that the target object should reach at the next time. The target area is used to control the target object's movement and target task in a specific area at the next time. The observation information at time t is first-person observation information, which is the target environment information that the target object can observe. Construct a target loss function based on the training data of the current state; Based on the target loss function, adjust and update the parameters of the decision model; The state information of the target object at time N+1 is input into the updated decision model. The decision model is then trained based on the training data of the next state of the target object until the training stop condition is met, thus obtaining the trained decision model.

2. The method according to claim 1, characterized in that, Before obtaining the training data of the current state of the target object in the target environment, the method further includes: A target decision model is trained using a post-hoc proximal strategy optimization method, and the target decision model is used to train the movement action strategy of the target object. The target decision model is identified as the teacher model of the decision model, so as to conduct supervised learning training on the decision model.

3. The method according to claim 2, characterized in that, The step of constructing the target loss function based on the training data of the current state includes: Construct a near-end policy optimization loss function based on the training data of the current state; Construct a supervised learning loss function based on the training data of the current state and the teacher model; The target loss function is determined by the weighted sum of the proximal policy optimization loss function and the supervised learning loss function.

4. The method according to claim 3, characterized in that, The step of constructing a near-end policy optimization loss function based on the training data of the current state includes: For the training information at each time t in the N time points, the near-end policy optimization loss function corresponding to the training information at time t is calculated based on the control policy at time t, the predicted reward of the control policy at time t, the real-time reward of the target environment at time t, the observation information at time t, and the region identifier reached by the target object at time t. The mean of the proximal policy optimization loss function corresponding to the training information at the N time points is determined as the proximal policy optimization loss function.

5. The method according to claim 4, characterized in that, The step of calculating the near-end policy optimization loss function corresponding to the training information at time t based on the control policy at time t, the predicted reward of the control policy at time t, the real-time reward at time t fed back by the target environment, the observation information at time t, and the region identifier reached by the target object at time t includes: Based on the real-time reward at time t fed back by the target environment, the observation information at time t, the area identifier reached by the target object at time t, and the control strategy at time t, the predicted action probability ratio is determined; Calculate the difference between the real-time reward at time t from the target environment feedback and the predicted reward of the control policy at time t; Based on the predicted action probability ratio and the difference, the proximal policy optimization loss function corresponding to the training information at time t is calculated.

6. The method according to claim 5, characterized in that, The real-time reward at time t from the target environment feedback is the sum of the reward for the exploration strategy at time t and the reward for the utilization strategy at time t. The reward for the exploration strategy at time t is determined based on the number of areas reached at time t and the target area reached at time t. The reward for the strategy at time t is determined based on the total number of target tasks completed at time t and the target area reached at time t.

7. The method according to claim 3, characterized in that, The step of constructing a supervised learning loss function based on the training data of the current state and the teacher model includes: For the training information at each time t in the N time points, calculate the cross-entropy loss function between the movement action policy in the control policy at time t and the corresponding movement action policy output by the teacher model at time t. The mean of the cross-entropy loss function corresponding to the training information at the N time points is determined as the supervised learning loss function.

8. The method according to claim 1, characterized in that, Before inputting the state information of the target object at time N+1 into the decision model after updating the parameters, the method further includes: Obtain the real-time reward at the (N+1)th time step from the target environment feedback; Obtain the observation information of the target object at time N+1; The target area identifier reached by the target object at time N is used as the area identifier reached by the target object at time N+1. The real-time reward at time N+1, the observation information at time N+1, and the region identifier reached by the target object at time N+1 constitute the state information of the target object at time N+1.

9. The method according to claim 8, characterized in that, The real-time reward obtained at the (N+1)th time from the target environment feedback includes: The control strategy at time N is adopted to control the target object to perform corresponding actions; It is determined that the target object completed the target task at the (N+1)th time and received a real-time reward from the target environment. Based on the observation information at the (N+1)th time and the real-time reward feedback from the target environment, latent variable of feature information is generated, and the latent variable of feature information is stored in the feature information statistics table to obtain the updated feature information statistics table; The updated feature information statistics table is normalized, and the normalized feature information statistics table is determined as the real-time reward at the N+1 time of the target environment feedback.

10. The method according to claim 8, characterized in that, The updated decision model is used for: Feature extraction is performed on the real-time reward at time N+1 of the target environment feedback, the observation information at time N+1, and the region identifier reached by the target object at time N+1, respectively, to obtain the first latent variable, the second latent variable, and the third latent variable; The first latent variable, the second latent variable, and the third latent variable are combined to obtain the target latent variable; Feature extraction is performed on the target latent variables to obtain target latent variable feature information; The control strategy at time N+1 is generated based on the target latent variable feature information.

11. The method according to claim 1, characterized in that, The method further includes: Obtain the vertical perspective information of the map corresponding to the target environment; The map is divided into regions based on its vertical perspective information, resulting in multiple regions. Assign an identifier to each region.

12. The method according to claim 11, characterized in that, The map is divided into regions based on its vertical perspective information, resulting in multiple regions, including: Based on the vertical perspective information of the map and the center point selection method, the center point of each of the multiple regions is determined. The center point selection method is as follows: the distance between two adjacent center points is the same, and there are center points in corners and open areas with regular shapes. For each target point in the map other than the determined center point, the region to which the target point belongs is determined according to the principle of the shortest Euclidean distance between the target point and the center point, thus obtaining the multiple regions; Assigning an identifier to each region includes: Assign an identifier to the center point of each region.

13. A strategy control method for a target object, characterized in that, include: Obtain the current state information of the target object in the target environment; The current state information is input into the decision model, and the current control strategy is output. The decision model is trained using the decision model training method according to any one of claims 1-12. The target object is controlled to perform actions using the control strategy at the current moment.

14. A decision model training device, characterized in that, include: The acquisition module is used to acquire training data of the current state of the target object in the target environment. The training data includes training information from N time points. The training information at any time point t includes the state information at time t, the control policy at time t, and the predicted reward of the control policy at time t. The state information at time t includes the real-time reward fed back by the target environment at time t, the observation information at time t, and the region identifier reached by the target object at time t. The control policy at time t and the predicted reward of the control policy at time t are input to the output of the decision model based on the state information at time t. The control policy at time t includes a movement action policy and a target task action policy. The target arrival area identifier at time t is the same as the area identifier reached by the target object at time t+1. t and N are both positive integers. The real-time reward at time t, fed back by the target environment, is the reward obtained by the target object after performing a target task with an unknown opponent. The area indicated by the target arrival area identifier at time t is the target area that the target object should reach at the next moment. The target area is used to control the target object's movement and target task in a specific area at the next moment. The observation information at time t is first-person observation information, which is the target environment information that the target object can observe. A construction module is used to construct a target loss function based on the training data of the current state; An adjustment module is used to adjust and update the parameters of the decision model according to the target loss function; The processing module is used to input the state information of the target object at time N+1 into the decision model after updating the parameters, and continue to train the decision model according to the training data of the next state of the target object until the training stop condition is met, so as to obtain the trained decision model.

15. A strategy control device for a target object, characterized in that, include: The acquisition module is used to acquire the current state information of the target object in the target environment. A control strategy output module is used to input the current state information into the decision model and output the control strategy at the current time. The decision model is trained by the decision model training method according to any one of claims 1-12. The motion control module is used to control the target object to perform actions using the control strategy at the current moment.

16. A computer device, characterized in that, include: A processor and a memory, the memory being used to store a computer program, the processor being used to invoke and run the computer program stored in the memory to perform the method of any one of claims 1 to 12 or 13.

17. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer program, cause the computer to perform the method as described in any one of claims 1 to 12 or 13.

18. A computer program product containing instructions, characterized in that, When the instructions are executed on a computer, the computer causes the computer to perform the method of any one of claims 1 to 12 or 13.

Images