A game control method and device, and storage medium

Abstract

The present invention discloses a game control method and device, and a storage medium, wherein the method comprises: when detecting that a target video game starts, obtaining a current video frame; based on the current video frame and a preset online value network, obtaining a current grayscale image and a current action, and controlling the target video game to perform the current action, obtaining a current reward value and a next video frame, and then obtaining a current quintuple and saving it to a preset database; when the number of the current quintuples obtained is greater than or equal to the preset number of samples, and the preset total number of training rounds is greater than zero, updating the parameters of the preset online value network based on the preset target value network and the preset database; when the number of the current quintuples is an integer multiple of the preset total training step length, replacing the parameters of the preset target value network with the parameters of the preset online value network, and reducing the preset total number of training rounds by one; using the next video frame as the current video frame, and continuing the above process. Through the present invention, the control effect of events can be improved.

Description

Technical Field

[0001] The present invention relates to dynamic optimization technology, and in particular to a game control method and device, and a storage medium. Background Art

[0002] Currently, deep reinforcement learning is a hot area in artificial intelligence research. It combines the perception ability of deep learning with the decision-making ability of reinforcement learning, providing a new control idea for dynamic optimization-related events (for example, video games).

[0003] Due to the instability and overestimation problems of deep reinforcement learning, the related technology uses an improved algorithm of deep Q network in the control of video games to try to alleviate the overestimation problem of deep reinforcement learning algorithm. However, since the Q-Learning algorithm is a different strategy algorithm, this itself will increase the instability of the deep reinforcement learning algorithm. In addition, the greedy strategy is used every time to select the action to be performed in the video game, and the Max operator is used every time to select the optimal action, which will amplify the overestimation problem of the deep reinforcement learning algorithm. Therefore, the instability and overestimation problems in deep reinforcement learning still exist, thereby reducing the control effect of the event. Summary of the invention

[0004] The present invention provides a game control method and device, and a storage medium, aiming to improve the control effect of events.

[0005] The technical solution of the present invention is achieved in this way:

[0006] An embodiment of the present invention provides a game control method, the method comprising:

[0007] When it is detected that the target video game starts, the current video frame is obtained;

[0008] Based on the current video frame and the preset online value network, a current grayscale image and a current action are obtained, and the target video game is controlled to execute the current action to obtain a current reward value and a next video frame;

[0009] Based on the current grayscale image, the current action, the current reward value, the next video frame and the preset online value network, a current five-tuple is obtained and saved to a preset database;

[0010] When the number of current quintuples obtained is greater than or equal to the preset number of samples, and the preset total number of training rounds is greater than zero, updating the parameters of the preset online value network based on the preset target value network and the preset database;

[0011] When the number of the obtained current quintuples is an integer multiple of the preset total training step length, the parameters of the preset target value network are replaced by the parameters of the preset online value network, and the preset total number of training rounds is reduced by one;

[0012] The next step video frame is used as the current video frame, and the current grayscale image and the current action are obtained based on the current video frame and the preset online value network.

[0013] In the above scheme, the preset online value network and the preset target value network are both networks composed of three convolutional layers and two fully connected layers.

[0014] In the above scheme, the current grayscale image and the current action are obtained based on the current video frame and the preset online value network, including:

[0015] Performing grayscale processing on the current video frame to obtain the current grayscale image;

[0016] Input the current grayscale image and each of the preset actions to be selected into the preset online value network to obtain the current evaluation value corresponding to each of the actions to be selected;

[0017] According to the current evaluation value corresponding to each of the candidate actions, determine the optimal action with the largest current evaluation value from the preset candidate actions;

[0018] The current action is obtained from the preset candidate actions or the optimal action with the maximum current evaluation value by using the ε-greedy strategy.

[0019] In the above scheme, the current five-tuple is obtained based on the current grayscale image, the current action, the current reward value, the next video frame and the preset online value network and saved to the preset database, including:

[0020] Grayscale processing is performed on the next step video frame to obtain a next step grayscale image;

[0021] According to the next step grayscale map, the preset online value network and the ∈-greedy strategy, the preset candidate actions are evaluated and selected to obtain the next step action;

[0022] The current five-tuple is composed of the current grayscale image, the current action, the current reward value, the next grayscale image and the next action and is saved in the preset database.

[0023] In the above scheme, the ∈-greedy strategy includes exploration probability and optimal probability; the step of evaluating and selecting the preset candidate actions according to the next step grayscale map, the preset online value network and the ∈-greedy strategy to obtain the next step action includes:

[0024] Input the next step grayscale image and each of the preset actions to be selected into the preset online value network to obtain the next step evaluation value corresponding to each of the actions to be selected;

[0025] According to the next step evaluation value corresponding to each of the candidate actions, determine the optimal action with the largest next step evaluation value from the preset candidate actions;

[0026] The next action is obtained by selecting from the preset candidate actions according to the exploration probability and selecting the optimal action with the largest evaluation value for the next step according to the optimal probability.

[0027] In the above scheme, the exploration probability and the real-time step length are in inverse proportion, and the real-time step length is the remainder obtained by dividing the preset total training step length by the number of the current quintuples obtained.

[0028] In the above solution, the parameter updating of the preset online value network based on the preset target value network and the preset database includes:

[0029] Obtaining the preset number of sample quintuples from the preset database;

[0030] According to the preset target value network and each sample quintuple in the preset number of sample quintuples, obtaining a target value corresponding to each sample quintuple;

[0031] According to the preset online value network and each sample quintuple, obtaining an evaluation value corresponding to each sample quintuple;

[0032] The root mean square back propagation algorithm is used to update the parameters of the preset online value network according to the target value, the evaluation value and the preset error calculation function.

[0033] In the above scheme, obtaining the target value corresponding to each sample quintuple according to the preset target value network and each sample quintuple of the preset number of samples includes:

[0034] When the next grayscale image in each sample quintuple belongs to the end video frame, the current reward value in each sample quintuple is used as the target value corresponding to each sample quintuple;

[0035] When the next grayscale image in each sample quintuple does not belong to the end video frame, the next grayscale image and the next action in each sample quintuple are input into the preset target value network to obtain the initial target value; based on the preset discount factor, the initial target value and the current reward value in each sample quintuple, the target value corresponding to each sample quintuple is calculated.

[0036] In the above scheme, the use of the root mean square back propagation algorithm to update the parameters of the preset online value network according to the target value, the evaluation value and the preset error calculation function includes:

[0037] Using a preset error calculation function, performing a root mean square calculation on the target value and the evaluation value to obtain an error corresponding to each sample quintuple;

[0038] Utilizing a root mean square back propagation algorithm, parameters of the preset online value network are updated according to the error corresponding to each sample quintuple.

[0039] In the above solution, obtaining a preset number of sample quintuples from the preset database includes:

[0040] Randomly extracting the preset number of samples of original five-tuples from the preset database;

[0041] For each original quintuple of the preset number of samples, a plurality of related current grayscale images adjacent to the current grayscale image in each original quintuple and a plurality of related next step grayscale images adjacent to the next step grayscale image in each original quintuple are obtained from the preset database;

[0042] A sample quintuple is formed by each original quintuple, the multiple related current grayscale images and the multiple related next step grayscale images, thereby obtaining the sample quintuple with the preset number of samples.

[0043] An embodiment of the present invention provides a game control device, the device comprising:

[0044] A control unit, configured to obtain a current video frame when detecting that a target video game has started; and based on the current video frame and a preset online value network, obtain a current grayscale image and a current action, and control the target video game to execute the current action, thereby obtaining a current reward value and a next video frame;

[0045] A data construction unit, configured to obtain a current five-tuple based on the current grayscale image, the current action, the current reward value, the next video frame and the preset online value network, and save the current five-tuple to a preset database;

[0046] A parameter updating unit, configured to update the parameters of the preset online value network based on the preset target value network and the preset database when the number of the current quintuples obtained is greater than or equal to the preset number of samples and the preset total number of training rounds is greater than zero; and to replace the parameters of the preset target value network with the parameters of the preset online value network and reduce the preset total number of training rounds by one when the number of the current quintuples obtained is an integer multiple of the preset total training step length;

[0047] The control unit is further configured to use the next step video frame as the current video frame, and continue to obtain a current grayscale image and a current action based on the current video frame and a preset online value network.

[0048] In the above scheme, the preset online value network and the preset target value network are both networks composed of three convolutional layers and two fully connected layers.

[0049] In the above scheme, the control unit is also used to grayscale the current video frame to obtain the current grayscale image; and input the current grayscale image and each of the preset candidate actions into the preset online value network to obtain the current evaluation value corresponding to each of the candidate actions; and determine the optimal action with the largest current evaluation value from the preset candidate actions based on the current evaluation value corresponding to each of the preset candidate actions; and use the ∈-greedy strategy to obtain the current action from the preset candidate actions or the optimal action with the largest current evaluation value.

[0050] In the above scheme, the data construction unit is also used to grayscale the next step video frame to obtain the next step grayscale image; and evaluate and select the preset candidate actions according to the next step grayscale image, the preset online value network and the ∈-greedy strategy to obtain the next step action; and the current five-tuple is composed of the current grayscale image, the current action, the current reward value, the next step grayscale image and the next step action and saved in the preset database.

[0051] In the above scheme, the ∈-greedy strategy includes exploration probability and optimal probability;

[0052] The data construction unit is also used to input the next step grayscale image and each of the preset candidate actions into the preset online value network to obtain the next step evaluation value corresponding to each of the candidate actions; and determine the optimal action with the largest next step evaluation value from the preset candidate actions according to the next step evaluation value corresponding to each of the candidate actions; and select from the preset candidate actions according to the exploration probability, and select the optimal action with the largest next step evaluation value according to the optimal probability to obtain the next step action.

[0053] In the above scheme, the exploration probability and the real-time step length are in inverse proportion, and the real-time step length is the remainder obtained by dividing the preset total training step length by the number of the current quintuples obtained.

[0054] In the above scheme, the parameter updating unit is also used to obtain the preset number of sample quintuples from the preset database; and obtain the target value corresponding to each sample quintuple according to the preset target value network and each sample quintuple of the preset number of sample quintuples; and obtain the evaluation value corresponding to each sample quintuple according to the preset online value network and each sample quintuple; and use the root mean square back propagation algorithm to update the parameters of the preset online value network according to the target value, the evaluation value and the preset error calculation function.

[0055] In the above scheme, the parameter updating unit is also used to, when the next grayscale image in each sample quintuple belongs to the end video frame, use the current reward value in each sample quintuple as the target value corresponding to each sample quintuple; and when the next grayscale image in each sample quintuple does not belong to the end video frame, input the next grayscale image and the next action in each sample quintuple into the preset target value network to obtain the initial target value; based on the preset discount factor, the initial target value and the current reward value in each sample quintuple, calculate the target value corresponding to each sample quintuple.

[0056] In the above scheme, the parameter updating unit is also used to use a preset error calculation function to perform root mean square calculation on the target value and the evaluation value to obtain the error corresponding to each sample quintuple; and use a root mean square back propagation algorithm to update the parameters of the preset online value network according to the error corresponding to each sample quintuple.

[0057] In the above scheme, the parameter updating unit is also used to randomly extract the preset number of original quintuples from the preset database; and for each original quintuple of the preset number of original quintuples, obtain from the preset database a plurality of related current grayscale images adjacent to the current grayscale image in each original quintuple, and a plurality of related next-step grayscale images adjacent to the next-step grayscale image in each original quintuple; and form a sample quintuple by each original quintuple, the plurality of related current grayscale images and the plurality of related next-step grayscale images, thereby obtaining the preset number of sample quintuples.

[0058] An embodiment of the present invention provides a game control device, which includes: a processor, a memory and a communication bus, wherein the memory communicates with the processor via the communication bus, and the memory stores one or more programs executable by the processor. When the one or more programs are executed, the steps of any of the above-mentioned game control methods are performed by the processor.

[0059] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program, and when the program is executed by at least one processor, the steps of any of the above-mentioned game control methods are implemented.

[0060] The embodiment of the present invention provides a game control method and device, and a storage medium, which adopts the above-mentioned technical implementation scheme. First, a current action is obtained based on a current video frame and a preset online value network, and a target video game is controlled to execute the current action. Since the current action executed by the target video game is a determined action obtained based on the current video frame and the preset online value network, the stability of controlling the target video game is improved. Secondly, before replacing the parameters of the preset target value network with the parameters of the preset online value network after parameter update, the current five-tuple is obtained based on the preset online value network before the parameter update and saved to a preset database. Therefore, the current five-tuple in the preset database is not all obtained by the preset online value network after parameter update, nor is it all obtained by the preset target value network after parameter replacement. The coupling degree between the preset target value network after parameter replacement and the preset database is reduced, and then the overestimation when the preset online value network is used to update the parameters is reduced. In summary, the stability of controlling the target video game is improved, and the overestimation of controlling the target video game is reduced, and finally the control effect of the video game is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A flowchart of a game control method provided by an embodiment of the present invention;

[0062] Figure 2 A schematic diagram of a game control device provided by an embodiment of the present invention Figure 1 ;

[0063] Figure 3 A schematic diagram of a game control effect provided by an embodiment of the present invention Figure 1 ;

[0064] Figure 4 A schematic diagram of a game control effect provided by an embodiment of the present invention Figure 2 ;

[0065] Figure 5 A schematic diagram of a game control device provided by an embodiment of the present invention Figure 2 ;

[0066] Figure 6 A schematic diagram of a game control device provided by an embodiment of the present invention Figure 3 . DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0068] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present invention, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0069] In order to alleviate the instability and overestimation problems of deep reinforcement learning, the related technology adopts an improved algorithm of deep Q network in the control of video games. First, the improved algorithm adopts a replay memory unit and a dual network with a target value network to improve the stability of the deep reinforcement learning algorithm; then the Q-Learning algorithm is used to realize the real-time update of the dual network; finally, a correction function is used to increase the gap between the optimal value and the non-optimal value. On the premise of increasing the gap between the optimal value and the non-optimal value, the property of the exponential function is used to make the reduction of the non-optimal value more reasonable, thereby alleviating the overestimation problem of the deep reinforcement learning algorithm; however, since the Q-Learning algorithm is an Strategy algorithm, which itself will increase the instability of deep reinforcement learning algorithm. Secondly, greedy strategy is used every time to select the actions to be performed in the video game, and Max operator is used every time to select the optimal action, which will amplify the over-estimation problem of deep reinforcement learning algorithm. In addition, the quadruple is stored in the playback memory unit. The quadruple only collects the current state, the action taken in the current state, the immediate reward value and the next state, but does not collect the next action taken in the next state, resulting in insufficient use of information. The dual network consists of only two convolutional layers and two fully connected layers, which is not deep enough, and the information extraction of the original data is not sufficient, making it difficult to discover deeper rules of the original data.

[0070] The related technology also uses deep reinforcement learning of double Q learning, realizes action selection through online value network, realizes action evaluation through target value network, realizes action decoupling from action evaluation and selection, and reduces over-estimation problem; however, since target value network is periodically copied to online value network, each time when the copying process is just completed, the parameters of these two networks are exactly the same, and the difference is not significant. The difference gradually increases with the increase of training time. Therefore, in the early stage after the copying operation is completed, the selection and evaluation of actions by online value network and target value network are exactly the same as the selection and evaluation of actions by only online value network. Therefore, the selection and evaluation of actions are not decoupled, so the over-estimation problem is not reduced, so that the average reward value obtained by deep reinforcement learning algorithm when controlling video games is not significantly improved.

[0071] The related technology also adopts a new Q-Learning algorithm to alleviate the evaluation error caused by the Max operator and reduce the over-estimation problem. However, since the function approximation method is not introduced to generalize the state space, it is only suitable for processing problems with smaller state spaces and is not suitable for processing problems with larger state spaces.

[0072] Based on the above content, in order to solve the problems of related technologies, the present invention proposes a game control method, which proposes to replace the Q-Learning algorithm with the same strategy by the SARSA algorithm, and replace the greedy strategy with the ∈-greedy strategy for action selection. Since the SARSA algorithm has higher stability than the Q-Learning algorithm, the present invention using the SARSA algorithm has higher stability; secondly, the ∈-greedy strategy can randomly select actions with a probability of ∈, and select the optimal action with a probability of 1-∈, which also reduces the over-estimation problem; in addition, a five-tuple data structure is used to introduce the next action for the four-tuple, so that the algorithm can make more full use of information; an online value network and a target value network including three convolutional layers and two fully connected layers are constructed to increase the depth of the network structure, and at the same time, the moving step size of the filter in the convolutional layer is modified to improve the network's ability to discover deep data features of the original data, and finally improve the ability to discover the optimal value, that is, to improve the average reward value obtained in the control of video games.

[0073] Embodiment 1

[0074] The embodiment of the present invention provides a game control method, such as Figure 1 As shown, the method includes:

[0075] S101, when it is detected that the target video game starts, obtaining the current video frame;

[0076] When the device detects that the target video game starts, it uses the first frame image when the target video game starts as the current video frame, and the current video frame represents the current state.

[0077] In some embodiments, when the device detects that the target video game has started, it also sets a playback memory unit, a total capacity of the playback memory unit, a preset total number of training rounds, a preset total training step length for each round of training, a preset online value network, and a preset target value network; wherein the playback memory unit is used to store data generated when controlling the target video game; and the preset total training step length is greater than the preset number of samples.

[0078] Exemplarily, the total capacity N of the replay memory unit D is set to 1000000, the total number of preset training rounds is 215, 215 training epochs, the preset total training step size B of each epoch is 10000, and the parameters θ of the preset online value network and the parameters θ' of the preset target value network are set with random values.

[0079] In some embodiments, the preset online value network represents the evaluation value corresponding to the current image and action; the preset target value network represents the target value corresponding to the next image and the next action.

[0080] In some embodiments, the preset online value network and the preset target value network are both networks consisting of three convolutional layers and two fully connected layers.

[0081] Exemplarily, a dual network structure including a preset online value network and a preset target value network is constructed, wherein the preset online value network is used to calculate the evaluation value, and the preset target value network is used to calculate the target value; the preset online value network and the preset target value network both include three convolutional layers and two fully connected layers, wherein each convolutional layer performs a convolution operation on the data for feature extraction, and the convolution operation ensures that the input of each neuron in the convolutional layer is connected to the local receptive field of the previous layer, thereby reducing the number of parameters that need to be stored and ensuring that the output function changes in the same way as the input function; the target value finally output by the fully connected layer is combined with the ∈-greedy strategy to complete the action selection.

[0082] Exemplarily, the three convolutional layers include the first convolutional layer, the second convolutional layer and the third convolutional layer. The first convolutional layer has 16 4×4 filters, and the step size of the convolution operation is 2; the second convolutional layer has 32 4×4 filters, and the step size of the convolution operation is 2; the third convolutional layer has 64 4×4 filters, and the step size of the convolution operation is 2; taking the input of the network as a set of cascaded grayscale images, each set of cascaded grayscale images contains 4 images, and the size of each grayscale image is 84×84, so it can be stored in a 84×84×4 vector space, and after the first convolutional layer 16 4× After the convolution operation with 4 filters in the first convolution layer, the output vector is 41×41×16, and the edge of the output vector is padded with 1, and the padded output vector becomes 42×42×16; after the padded output vector is convolved with 32 4×4 filters in the second convolution layer, the output vector is 20×20×32; after the output vector of the second convolution layer is convolved with 64 4×4 filters in the third convolution layer, the output vector is 9×9×64; wherein, the activation function corresponding to each convolution layer can be a linear rectification function (ReLU, Rectified Linear Unit).

[0083] Exemplarily, the two fully connected layers include a first fully connected layer and a second fully connected layer, the first fully connected layer has 256 neurons, and the activation function can be ReLU; the number of neurons in the second fully connected layer is the number of preset actions to be selected, and the preset actions to be selected are all optional game operations in the target video game.

[0084] It should be noted that the preset online value network and the preset target value network used are both convolutional neural networks. Compared with the Q table, the convolutional neural network realizes the generalization of the state space, so that the algorithm can respond to all state spaces, and ultimately can handle problems with larger state spaces.

[0085] S102, based on the current video frame and the preset online value network, obtain the current grayscale image and the current action, and control the target video game to execute the current action, obtain the current reward value and the next video frame;

[0086] The device obtains a current grayscale image and a current action based on a current video frame and a preset online value network; executes the current action in a target video game, the target video game outputs a current reward value, and jumps from the current video frame to the next video frame.

[0087] In some embodiments, the device performs grayscale processing on the current video frame to obtain a current grayscale image; inputs the current grayscale image and each of the preset candidate actions into a preset online value network to obtain a current evaluation value corresponding to each candidate action; determines the optimal action with the largest current evaluation value from the preset candidate actions based on the current evaluation value corresponding to each candidate action; and utilizes an ∈-greedy strategy to obtain the current action from the preset candidate actions or the optimal action with the largest current evaluation value.

[0088] In some embodiments, the ∈-greedy strategy includes an exploration probability ∈ and an optimal probability ∈', ∈'=1-∈; the device selects from the preset candidate actions according to the exploration probability, and selects the optimal action with the largest current evaluation value according to the optimal probability to obtain the current action.

[0089] In some embodiments, the exploration probability is inversely proportional to the real-time step length, and the real-time step length is the remainder obtained by dividing the preset total training step length by the number of current quintuples obtained.

[0090] Exemplarily, the real-time step size represents the step size increasing from zero in each round of training, and the real-time step size is zero when each round of training ends; in each round of training, the exploration probability gradually decreases from 1 to 0.1 as the real-time step size increases.

[0091] S103, based on the current grayscale image, the current action, the current reward value, the next video frame and the preset online value network, obtain the current five-tuple and save it to the preset database;

[0092] The device obtains a next grayscale image and a next action based on a next video frame and a preset online value network; a current five-tuple is formed by the current grayscale image, the current action, the current reward value, the next grayscale image and the next action, and the obtained current five-tuple is saved to a preset database; wherein the five-tuple stores historical data of controlling a target video game; the preset database includes a playback memory unit.

[0093] In some embodiments, the device grayscales the next video frame to obtain a next grayscale image; evaluates and selects preset candidate actions based on the next grayscale image, a preset online value network and an ∈-greedy strategy to obtain a next action; and the current five-tuple is composed of the current grayscale image, the current action, the current reward value, the next grayscale image and the next action.

[0094] The device uses a preset online value network to evaluate the next grayscale image and each candidate action in the preset candidate actions, and then uses the ∈-greedy strategy to select the preset candidate actions to obtain the next action, and then obtains a current quintuple and saves it.

[0095] It should be noted that the present invention controls a video game, so each frame of the video game is a state. When the video game starts, the state in which no action is selected is the current state. The current video frame in the current state is s, and the action performed in the current state is the current action a selected in the current state. When a new state is reached, the new video frame is s', the reward value fed back by the video game is r, and the next action taken in the new state is a'. A quintuple is used to represent the five data s, a, s', r and a'.

[0096] In some embodiments, the total capacity N of the preset database is set, and each time a current quintuple is obtained, it is saved to the preset database by a cyclic insertion method. When the total number of quintuples in the preset database is equal to N, the earliest quintuple is overwritten from the beginning and inserted cyclically.

[0097] Exemplarily, a quintuple for storing historical data is constructed, and each current quintuple is saved to a preset database or a playback memory unit D, which can store N quintuples; each current quintuple stores five parts of content, expressed as (φ, a, r, φ', a'), wherein φ is a current grayscale image representing the current state, a is the current action selected in the current state, φ' is a next step grayscale image representing the next state, r is the current reward value, and a' is the next step action selected in the next state.

[0098] Exemplarily, the current quintuple is inserted into the playback memory unit by a cyclic insertion method. Taking the total capacity N of the playback memory unit as 1,000,000 as an example, when the stored quintuples are greater than 1,000,000, the earliest quintuple is overwritten from the beginning and inserted cyclically.

[0099] In some embodiments, the ∈-greedy strategy includes exploration probability and optimal probability; the device inputs the next step grayscale map and each of the preset candidate actions into a preset online value network to obtain the next step evaluation value corresponding to each candidate action; according to the next step evaluation value corresponding to each candidate action, the optimal action with the largest next step evaluation value is determined from the preset candidate actions; the next step action is obtained by selecting from the preset candidate actions according to the exploration probability and selecting the optimal action with the largest next step evaluation value according to the optimal probability.

[0100] The device inputs the next step grayscale image and each candidate action into the preset online value network, obtains the next step evaluation value corresponding to each candidate action, and then determines the optimal action with the largest next step evaluation value; then selects from the preset candidate actions according to the exploration probability, and selects the optimal action with the largest next step evaluation value according to the optimal probability, to obtain the next step action.

[0101] S104: when the number of the current quintuples obtained is greater than or equal to the preset number of samples, and the preset total number of training rounds is greater than zero, updating the parameters of the preset online value network based on the preset target value network and the current quintuples obtained;

[0102] The device counts the number of current quintuples obtained, and determines whether the number of current quintuples obtained is not less than the preset number of samples; when the number of current quintuples obtained is not less than the preset number of samples, determines whether the preset total number of training rounds is greater than zero, otherwise, takes the next video frame as the current video frame, and continues to obtain the current grayscale image and the current action based on the current video frame and the preset online value network; when the preset total number of training rounds is greater than zero, based on the preset target value network and the current quintuple obtained, updates the parameters of the preset online value network, and continues to use the preset online value network after the parameter update, otherwise, takes the next video frame as the current video frame, and continues to obtain the current grayscale image and the current action based on the current video frame and the preset online value network.

[0103] It should be noted that when the preset total number of training rounds is equal to zero, through continuous iterations, the performance of the preset online value network and the preset target value network obtained at this time is optimal. Using the preset online value network and the preset target value network obtained at this time to control the target video game can obtain a higher average reward value.

[0104] Exemplarily, when training the preset online value network, 32 quintuples are randomly selected from the playback memory unit as sample quintuples and used for training; during the training process, the current grayscale image and the current action are used as input data of the preset online value network, and the next grayscale image state, the current reward value, and the next action are used as input data of the preset target value network.

[0105] In some embodiments, the device obtains a preset number of sample quintuples from a preset database; obtains a target value corresponding to each sample quintuple based on a preset target value network and each sample quintuple in the preset number of sample quintuples; obtains an evaluation value corresponding to each sample quintuple based on a preset online value network and each sample quintuple; and updates the parameters of the preset online value network based on the target value, the evaluation value and the preset error calculation function using a root mean square back propagation algorithm.

[0106] The device obtains a preset number of sample quintuples from a preset database or a playback memory unit; uses a preset target value network to calculate the target value corresponding to each sample quintuple; uses a preset online value network to calculate the evaluation value corresponding to each sample quintuple; uses a root mean square back propagation algorithm to update the parameters of the preset online value network according to the target value, the evaluation value and a preset error calculation function; wherein the preset error calculation function represents the error value corresponding to the target value and the evaluation value; wherein different data in each sample quintuple are respectively input into a preset online value network and a preset target value network to obtain an evaluation value and a target value, respectively.

[0107] In some embodiments, when the next grayscale image in each sample quintuple belongs to the end video frame, the device uses the current reward value in each sample quintuple as the target value corresponding to each sample quintuple; when the next grayscale image in each sample quintuple does not belong to the end video frame, the next grayscale image and the next action in each sample quintuple are input into a preset target value network to obtain an initial target value; based on the preset discount factor, the initial target value and the current reward value in each sample quintuple, the target value corresponding to each sample quintuple is calculated.

[0108] The device determines whether the next grayscale image in each sample quintuple belongs to the end video frame; when the next grayscale image belongs to the end video frame, the current reward value r in each sample quintuple is directly used as the target value y corresponding to each sample quintuple, y=r; otherwise, the next grayscale image and the next action in each sample quintuple are input into the preset target value network to obtain the initial target value, and then the target value is calculated.

[0109] Furthermore, the device uses the target value calculation formula (1) to calculate the initial target value and the current reward value in each sample quintuple to obtain the target value corresponding to each sample quintuple:

[0110] y=r+γQ′(φ′,a′,θ′) (1)

[0111] Among them, γ is the preset discount factor, which can be taken as 0.95; Q' is the initial target value, φ' is the next gray value, a' is the next action, and θ' is the parameter of the preset target value network.

[0112] In some embodiments, the device uses a preset error calculation function to perform root mean square calculation on the target value and the evaluation value to obtain the error corresponding to each sample quintuple; and uses the root mean square back propagation algorithm to update the parameters of the preset online value network according to the error corresponding to each sample quintuple.

[0113] The device inputs the target value and the evaluation value into a preset error calculation function to obtain the error corresponding to each sample quintuple; the preset online value network and the errors corresponding to all sample quintuples are input into the root mean square back propagation algorithm to obtain the preset online value network after parameter update.

[0114] In some embodiments, the preset error calculation function is shown in formula (2):

[0115] e=(yQ(φ,a,θ)) 2 (2)

[0116] Among them, y is the target value corresponding to each sample quintuple, Q is the evaluation value corresponding to each sample quintuple, and e is the error corresponding to each sample quintuple.

[0117] In some embodiments, the device randomly extracts a preset number of original quintuples from a preset database; for each original quintuple in the preset number of original quintuples, obtains from the preset database a plurality of related current grayscale images adjacent to the current grayscale image in each original quintuple, and a plurality of related next-step grayscale images adjacent to the next-step grayscale image in each original quintuple; a sample quintuple is formed by each original quintuple, a plurality of related current grayscale images, and a plurality of related next-step grayscale images, thereby obtaining a preset number of sample quintuples.

[0118] The device randomly extracts a preset number of original quintuples from a preset database; for each original quintuple, a plurality of related current grayscale images adjacent to the current grayscale image in each original quintuple and a plurality of related next step grayscale images adjacent to the next step grayscale image in each original quintuple are obtained to form a sample quintuple; wherein the current grayscale image and the plurality of related current grayscale images are time-continuous, and the next step grayscale image and the plurality of related next step grayscale images are time-continuous.

[0119] Furthermore, the device uses the current grayscale image and multiple related current grayscale images in each sample quintuple as the current image group; uses the next step grayscale image and multiple related current grayscale images in each original quintuple as the next step image group; inputs the current image group and current action in each sample quintuple into a preset online value network to obtain an evaluation value corresponding to each sample quintuple; inputs the next step image group and next step action in each sample quintuple into a preset target value network to obtain an initial target value.

[0120] Exemplarily, the input original color image data is grayed to obtain a grayscale image. In order to provide continuous state information, three related current grayscale images adjacent to the current grayscale image in each sample quintuple are obtained. The current grayscale image and the three related current grayscale images are four continuous current grayscale images. The four current grayscale images are cascaded to form a new current image group. The current image group is represented by (φ t-2, ,φ t-1, ,φ t, ,φ t+1, ), and use the current image group as the input of the preset online value network; where t is the current moment corresponding to the current grayscale image.

[0121] S105, when the number of the current quintuples obtained is an integer multiple of the preset total training step length, the parameters of the preset target value network are replaced by the parameters of the preset online value network, and the preset total number of training rounds is reduced by one;

[0122] The device also determines whether the number of current quintuples obtained is an integer multiple of the preset total training step length; when the number of current quintuples obtained is an integer multiple of the preset total training step length, it indicates that one round of training process is completed, the parameters (or weights) of the preset online value network are copied to the parameters (or weights) of the preset target value network, and the preset target value network after parameter replacement is continued to be used, and the preset total number of training rounds is reduced by one, otherwise, the next step video frame is used as the current video frame, and the current grayscale image and the current action are continued to be obtained based on the current video frame and the preset online value network.

[0123] For example, taking the preset total training step length B as 10,000, after 10,000 steps, the parameter values ​​of the preset online value network are assigned to the preset target value network, and then the parameters of the preset target value network continue to remain unchanged, and the parameters of the preset online value network are updated at each step until after another 10,000 steps, the parameter values ​​of the preset online value network are assigned to the preset target value network.

[0124] It should be noted that after obtaining the current quintuple, the device learns the optimal strategy through the SARSA algorithm, that is, optimizes the parameters of the preset online value network and the preset target value network. Specifically, in the SARSA algorithm, only the parameters of the preset online value network are updated in real time, and the parameters of the preset target value network are not updated in real time. The parameters of the target value network are periodically copied to the online value network; the preset target value network is used to calculate the target value, and the preset online value network is used to calculate the evaluation value. The preset online value network is evaluated by calculating the error value through the target value and the evaluation value, thereby updating the parameters of the preset online value network, and through continuous iteration, the performance of the optimization algorithm in video game control is optimized.

[0125] For example, Figure 2A game control device is shown, which includes a target video game 21, a preset online value network 22, a preset target value network 23, a playback memory unit 24 and a parameter updating unit 25; first, the total capacity N of the playback memory unit 24 is initialized to 1000000, the discount factor is 0.95, the number of training epochs is 215, the total number of training epochs is preset to 215, 215 training epochs, the preset total training step length B of each epoch is 10000, and the parameters θ of the preset online value network 22 and the parameters θ' of the preset target value network 23 are set with random values; within 215 epochs, for each epoch, first, the real-time step length steps=0, and each game in the target video game 21 is defined as a segment episode from the beginning to the end, and each epoch can contain 1 to n episodes, where n is an integer greater than zero.

[0126] Further, when the target video game 21 starts, the preset online value network 22 obtains the first frame image when the video game starts from the target video game 21 as the current video frame s; the preset online value network 22 performs image grayscale processing on the current video frame s to obtain the current grayscale image φ, and then evaluates and selects the preset candidate actions according to the current grayscale image φ and the ∈-greedy strategy to obtain the current action a; controls the target video game 21 to execute the current action a, obtains the current reward value r and the next video frame s' from the target video game 21; performs grayscale processing on the next video frame s' to obtain the next grayscale image φ'; evaluates and selects the preset candidate actions according to the next grayscale image and the ∈-greedy strategy to obtain the next action a'; the current five-tuple (φ, a, r, φ', a') is composed of the current grayscale image φ, the current action a, the current reward value r, the next grayscale image φ' and the next action a', and is saved to the playback memory unit 24; when the playback memory When the number of quintuples in the memory unit 24 is greater than or equal to the preset number of samples, obtain the preset number of sample quintuples, input (φ, a) in each sample quintuple to the preset online value network 22, so that the preset online value network 22 calculates the evaluation value Q (φ, a, θ); input (r, φ', a') in each sample quintuple to the preset target value network 23, so that the preset target value network 23 calculates the target value y, y = r or y = r + γQ' (φ', a', θ ′); the parameter updating unit 25 inputs y and Q into the preset error calculation function, calculates the error value corresponding to each sample quintuple, and then uses the root mean square directional propagation algorithm to update the parameters of the preset online value network 22 according to the error value corresponding to each sample quintuple; after every 10,000 steps, the parameters of the preset online value network 22 are copied to the preset target value network 23, and the parameters of the preset target value network 23 at other times are not changed, and this process is repeated until 215 epochs are trained.

[0127] It can be understood that when the number of current quintuples obtained is greater than or equal to the preset number of samples, the current quintuples obtained can be used to update the parameters of the preset online value network; until the number of current quintuples obtained is greater than or equal to the preset total training step length, the parameters of the preset target value network are replaced with the parameters of the preset online value network. Since the preset total training step length is greater than the preset number of samples, the current quintuple obtained by the preset online value network after parameter update will not be directly selected as the sample quintuple for updating the parameters of the preset online value network. Therefore, in the evaluation of the preset online value network, there is always a distance of K / batch steps between the preset online value network and the preset target value network, which ultimately makes the evaluation value and the target value when the parameters of the preset online value network are updated decoupled. At this time, the preset target value network is only used for valuation, not for selecting actions; wherein, when the number M of the current quintuples obtained is greater than or equal to N, K=N, otherwise K=M, K and M are integers greater than 0; batch is the preset number of samples.

[0128] S106: Taking the next video frame as the current video frame, and continuing to obtain the current grayscale image and the current action based on the current video frame and the preset online value network.

[0129] The device takes the next video frame as the current video frame, and continues to obtain the current grayscale image and the current action based on the current video frame and the preset online value network, and then obtains a current five-tuple, which is saved in the revisit memory unit.

[0130] It should be noted that the present invention improves the stability of game control by replacing the Q-Learning algorithm with a different strategy with the SARSA algorithm with the same strategy, and replacing the quadruple with the quintuple; the evaluation value and the target value are decoupled when the parameters of the preset online value network are updated, reducing over-estimation; and the state space is generalized by using the online value network and the target value network, both of which include three convolutional layers and two fully connected layers, so that the present invention can handle video games with a larger state space.

[0131] For example, Figure 3 A schematic diagram of a game control effect is shown, in which the game control method provided by an embodiment of the present invention, the improved algorithm of the deep Q network, and the deep reinforcement learning of double Q learning are used to control the video game Amidar. Different average reward values ​​are obtained on the video game Amidar. The horizontal axis is the number of training rounds, and the vertical axis is the average reward value. It can be seen that the embodiment of the present invention obtains the highest reward value.

[0132] For example, Figure 4A schematic diagram of a game control effect is shown, in which the game control method of an embodiment of the present invention, the improved algorithm of the deep Q network, and the deep reinforcement learning algorithm of double Q learning are used to control the video game Amidar. Different average reward values ​​are obtained on the video game Breakout. The horizontal axis is the number of training rounds, and the vertical axis is the average reward value. It can be seen that the embodiment of the present invention obtains the highest reward value.

[0133] It can be understood that, firstly, the device obtains the current action based on the current video frame and the preset online value network, and controls the target video game to execute the current action. Since the current action executed by the target video game is a determined action obtained based on the current video frame and the preset online value network, the stability of controlling the target video game is improved; secondly, before replacing the parameters of the preset target value network with the parameters of the preset online value network after the parameter update, the current five-tuple has been obtained based on the preset online value network before the parameter update and saved to the preset database. Therefore, the current five-tuple in the preset database is not all obtained by the preset online value network after the parameter update, nor is it all obtained by the preset target value network after the parameter replacement. The coupling degree between the preset target value network after the parameter replacement and the preset database is reduced, and then the over-estimation when using it to update the parameters of the preset online value network is reduced; in summary, the stability of controlling the target video game is improved, and the over-estimation of controlling the target video game is reduced, and finally the control effect of the video game is improved, and better performance is achieved in the control of the video game, and a higher reward value is obtained.

[0134] Embodiment 2

[0135] Based on the same inventive concept of the first embodiment, further description is given.

[0136] An embodiment of the present invention provides a game control device, such as Figure 5 As shown, the game control device 3 includes:

[0137] The control unit 31 is used to obtain the current video frame when detecting the start of the target video game; and based on the current video frame and the preset online value network, obtain the current grayscale image and the current action, and control the target video game to execute the current action, and obtain the current reward value and the next video frame;

[0138] A data construction unit 32, for obtaining a current five-tuple based on a current grayscale image, a current action, a current reward value, a next video frame and a preset online value network and saving the result to a preset database;

[0139] The parameter updating unit 33 is used to update the parameters of the preset online value network based on the preset target value network and the preset database when the number of the current quintuples obtained is greater than or equal to the preset number of samples and the preset total number of training rounds is greater than zero; and when the number of the current quintuples obtained is an integer multiple of the preset total training step length, replace the parameters of the preset target value network with the parameters of the preset online value network, and reduce the preset total number of training rounds by one;

[0140] The control unit 31 is further configured to use the next video frame as the current video frame, and continue to obtain the current grayscale image and the current action based on the current video frame and the preset online value network.

[0141] In some embodiments, the preset online value network and the preset target value network are both networks consisting of three convolutional layers and two fully connected layers.

[0142] In some embodiments, the control unit 31 is also used to grayscale the current video frame to obtain a current grayscale image; and input the current grayscale image and each of the preset candidate actions into a preset online value network to obtain a current evaluation value corresponding to each candidate action; and determine the optimal action with the largest current evaluation value from the preset candidate actions based on the current evaluation value corresponding to each candidate action; and use the ∈-greedy strategy to obtain the current action from the preset candidate actions or the optimal action with the largest current evaluation value.

[0143] In some embodiments, the data construction unit 32 is also used to grayscale the next video frame to obtain a next grayscale image; and evaluate and select the preset candidate actions according to the next grayscale image, the preset online value network and the ∈-greedy strategy to obtain the next action; and form a current five-tuple consisting of the current grayscale image, the current action, the current reward value, the next grayscale image and the next action and save it to a preset database.

[0144] In some embodiments, the ∈-greedy strategy includes an exploration probability and an optimal probability;

[0145] The data construction unit 32 is also used to input the next step grayscale image and each candidate action in the preset candidate actions into the preset online value network to obtain the next step evaluation value corresponding to each candidate action; and determine the optimal action with the largest next step evaluation value from the preset candidate actions according to the next step evaluation value corresponding to each candidate action; and select from the preset candidate actions according to the exploration probability, and select the optimal action with the largest next step evaluation value according to the optimal probability to obtain the next step action.

[0146] In some embodiments, the exploration probability is inversely proportional to the real-time step length, and the real-time step length is the remainder obtained by dividing the preset total training step length by the number of current quintuples obtained.

[0147] In some embodiments, the parameter updating unit 33 is also used to obtain a preset number of sample quintuples from a preset database; and obtain the target value corresponding to each sample quintuple based on the preset target value network and each sample quintuple of the preset number of sample quintuples; and obtain the evaluation value corresponding to each sample quintuple based on the preset online value network and each sample quintuple; and use the root mean square back propagation algorithm to update the parameters of the preset online value network according to the target value, the evaluation value and the preset error calculation function.

[0148] In some embodiments, the parameter updating unit 33 is also used to use the current reward value in each sample quintuple as the target value corresponding to each sample quintuple when the next grayscale image in each sample quintuple belongs to the end video frame; and when the next grayscale image in each sample quintuple does not belong to the end video frame, input the next grayscale image and the next action in each sample quintuple into the preset target value network to obtain the initial target value; based on the preset discount factor, the initial target value and the current reward value in each sample quintuple, calculate the target value corresponding to each sample quintuple.

[0149] In some embodiments, the parameter updating unit 33 is also used to use a preset error calculation function to perform a root mean square calculation on the target value and the evaluation value to obtain the error corresponding to each sample quintuple; and use a root mean square back propagation algorithm to update the parameters of the preset online value network according to the error corresponding to each sample quintuple.

[0150] In some embodiments, the parameter updating unit 33 is also used to randomly extract a preset number of original quintuples from a preset database; and for each original quintuple in the preset number of original quintuples, obtain from the preset database a plurality of related current grayscale images adjacent to the current grayscale image in each original quintuple, and a plurality of related next grayscale images adjacent to the next grayscale image in each original quintuple; and form a sample quintuple by each original quintuple, a plurality of related current grayscale images and a plurality of related next grayscale images, thereby obtaining a preset number of sample quintuples.

[0151] It should be noted that, in actual applications, the above-mentioned control unit 31, data construction unit 32 and parameter update unit 33 can be implemented by a processor 34 located on the game control device 4, specifically a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA).

[0152] The embodiment of the present invention also provides a game control device, such as Figure 6 As shown, the game control device 4 includes: a processor 34, a memory 35 and a communication bus 36. The memory 35 communicates with the processor 34 via the communication bus 36. The memory 35 stores one or more programs executable by the processor 34. When one or more programs are executed, any one of the game control methods described in the above embodiments is executed by the processor 34.

[0153] In practical applications, the memory 35 can be a volatile memory (volatile memory), such as a random access memory (Random-Access Memory, RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (Read-Only Memory, ROM), a flash memory (flash memory), a hard disk (Hard Disk Drive, HDD) or a solid-state drive (Solid-State Drive, SSD); or a combination of the above types of memory, and provide programs and data to the processor 34.

[0154] An embodiment of the present invention provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors. When the program is executed by the processor 34, any one of the game control methods described in the above embodiments is implemented.

[0155] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0156] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0157] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0158] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. A game control method, characterized in that: The method comprises: When it is detected that the target video game starts, the current video frame is obtained; Based on the current video frame and the preset online value network, a current grayscale image and a current action are obtained, and the target video game is controlled to execute the current action to obtain a current reward value and a next video frame; Based on the current grayscale image, the current action, the current reward value, the next video frame and the preset online value network, a current five-tuple is obtained and saved to a preset database; When the number of current quintuples obtained is greater than or equal to the preset number of samples, and the preset total number of training rounds is greater than zero, updating the parameters of the preset online value network based on the preset target value network and the preset database; When the number of the obtained current quintuples is an integer multiple of the preset total training step length, the parameters of the preset target value network are replaced by the parameters of the preset online value network, and the preset total number of training rounds is reduced by one; Taking the next step video frame as the current video frame, and continuing to obtain the current grayscale image and the current action based on the current video frame and the preset online value network; The obtaining of the current grayscale image and the current action based on the current video frame and the preset online value network includes: Performing grayscale processing on the current video frame to obtain the current grayscale image; Input the current grayscale image and each of the preset actions to be selected into the preset online value network to obtain the current evaluation value corresponding to each of the actions to be selected; According to the current evaluation value corresponding to each of the candidate actions, determine the optimal action with the largest current evaluation value from the preset candidate actions; The current action is obtained from the preset candidate actions or the optimal action with the maximum current evaluation value by using the ε-greedy strategy.

2. The method according to claim 1, characterized in that: The preset online value network and the preset target value network are both networks composed of three convolutional layers and two fully connected layers.

3. The method according to claim 1, characterized in that The step of obtaining a current five-tuple based on the current grayscale image, the current action, the current reward value, the next video frame and the preset online value network and saving the five-tuple to a preset database includes: Grayscale processing is performed on the next step video frame to obtain a next step grayscale image; According to the next step grayscale map, the preset online value network and the ∈-greedy strategy, the preset candidate actions are evaluated and selected to obtain the next step action; The current five-tuple is composed of the current grayscale image, the current action, the current reward value, the next grayscale image and the next action and is saved in the preset database.

4. The method according to claim 3, characterized in that The ∈-greedy strategy includes exploration probability and optimal probability; the step of evaluating and selecting the preset candidate actions according to the next step grayscale map, the preset online value network and the ∈-greedy strategy to obtain the next step action includes: Input the next step grayscale image and each of the preset actions to be selected into the preset online value network to obtain the next step evaluation value corresponding to each of the actions to be selected; According to the next step evaluation value corresponding to each of the candidate actions, determine the optimal action with the largest next step evaluation value from the preset candidate actions; The next action is obtained by selecting from the preset candidate actions according to the exploration probability and selecting the optimal action with the largest evaluation value for the next step according to the optimal probability.

5. The method according to claim 4, characterized in that The exploration probability is inversely proportional to the real-time step length, and the real-time step length is the remainder obtained by dividing the preset total training step length by the number of the current quintuples obtained.

6. The method according to any one of claims 1 to 5, characterized in that: The updating of parameters of the preset online value network based on the preset target value network and the preset database includes: Obtaining the preset number of sample quintuples from the preset database; According to the preset target value network and each sample quintuple in the preset number of sample quintuples, obtaining a target value corresponding to each sample quintuple; According to the preset online value network and each sample quintuple, obtaining an evaluation value corresponding to each sample quintuple; The root mean square back propagation algorithm is used to update the parameters of the preset online value network according to the target value, the evaluation value and the preset error calculation function.

7. The method according to claim 6, characterized in that The step of obtaining a target value corresponding to each sample quintuple according to the preset target value network and each sample quintuple of the preset number of samples comprises: When the next grayscale image in each sample quintuple belongs to the end video frame, the current reward value in each sample quintuple is used as the target value corresponding to each sample quintuple; When the next grayscale image in each sample quintuple does not belong to the end video frame, the next grayscale image and the next action in each sample quintuple are input into the preset target value network to obtain the initial target value; based on the preset discount factor, the initial target value and the current reward value in each sample quintuple, the target value corresponding to each sample quintuple is calculated.

8. The method according to claim 6, characterized in that The method of using the root mean square back propagation algorithm to update the parameters of the preset online value network according to the target value, the evaluation value and the preset error calculation function includes: Using a preset error calculation function, performing a root mean square calculation on the target value and the evaluation value to obtain an error corresponding to each sample quintuple; Utilizing a root mean square back propagation algorithm, parameters of the preset online value network are updated according to the error corresponding to each sample quintuple.

9. The method according to claim 6, characterized in that The step of obtaining a preset number of sample quintuples from the preset database includes: Randomly extracting the preset number of samples of original five-tuples from the preset database; For each original quintuple of the preset number of samples, a plurality of related current grayscale images adjacent to the current grayscale image in each original quintuple and a plurality of related next step grayscale images adjacent to the next step grayscale image in each original quintuple are obtained from the preset database; A sample quintuple is formed by each original quintuple, the multiple related current grayscale images and the multiple related next step grayscale images, thereby obtaining the sample quintuple with the preset number of samples.

10. A game control device, characterized in that: The device comprises: A control unit, configured to obtain a current video frame when detecting that a target video game has started; and based on the current video frame and a preset online value network, obtain a current grayscale image and a current action, and control the target video game to execute the current action, thereby obtaining a current reward value and a next video frame; A data construction unit, configured to obtain a current five-tuple based on the current grayscale image, the current action, the current reward value, the next video frame and the preset online value network, and save the five-tuple to a preset database; A parameter updating unit, configured to update the parameters of the preset online value network based on the preset target value network and the preset database when the number of the current quintuples obtained is greater than or equal to the preset number of samples and the preset total number of training rounds is greater than zero; and to replace the parameters of the preset target value network with the parameters of the preset online value network and reduce the preset total number of training rounds by one when the number of the current quintuples obtained is an integer multiple of the preset total training step length; The control unit is further configured to use the next video frame as the current video frame, and continue to obtain a current grayscale image and a current action based on the current video frame and a preset online value network; The control unit is also used to grayscale the current video frame to obtain the current grayscale image; and input the current grayscale image and each of the preset candidate actions into the preset online value network to obtain the current evaluation value corresponding to each of the candidate actions; and determine the optimal action with the largest current evaluation value from the preset candidate actions according to the current evaluation value corresponding to each of the candidate actions; and use the ∈-greedy strategy to obtain the current action from the preset candidate actions or the optimal action with the largest current evaluation value.

11. The device according to claim 10, characterized in that The preset online value network and the preset target value network are both networks composed of three convolutional layers and two fully connected layers.

12. The device according to claim 10, characterized in that The data construction unit is also used to grayscale the next step video frame to obtain a next step grayscale image; and evaluate and select preset candidate actions according to the next step grayscale image, the preset online value network and the ∈-greedy strategy to obtain the next step action; and form the current quintuple with the current grayscale image, the current action, the current reward value, the next step grayscale image and the next step action and save it to the preset database.

13. The device according to claim 12, characterized in that The ∈-greedy strategy includes exploration probability and optimal probability; The data construction unit is further used to input the next step grayscale image and each of the preset actions to be selected into the preset online value network to obtain the next step evaluation value corresponding to each action to be selected; And according to the next step evaluation value corresponding to each of the candidate actions, determine the optimal action with the largest next step evaluation value from the preset candidate actions; and select from the preset candidate actions according to the exploration probability, and select the optimal action with the largest next step evaluation value according to the optimal probability to obtain the next action.

14. The device according to claim 13, characterized in that The exploration probability is inversely proportional to the real-time step length, and the real-time step length is the remainder obtained by dividing the preset total training step length by the number of the current quintuples obtained.

15. The device according to any one of claims 10 to 14, characterized in that The parameter updating unit is further used to obtain the preset number of sample quintuples from the preset database; and obtain the target value corresponding to each sample quintuple according to the preset target value network and each sample quintuple of the preset number of sample quintuples; and obtain the evaluation value corresponding to each sample quintuple according to the preset online value network and each sample quintuple; And using a root mean square back propagation algorithm, parameters of the preset online value network are updated according to the target value, the evaluation value and a preset error calculation function.

16. The device according to claim 15, characterized in that The parameter updating unit is further used for, when the next grayscale image in each sample quintuple belongs to the end video frame, using the current reward value in each sample quintuple as the target value corresponding to each sample quintuple; and when the next grayscale image in each sample quintuple does not belong to the end video frame, inputting the next grayscale image and the next action in each sample quintuple into the preset target value network to obtain an initial target value; Based on the preset discount factor, the initial target value and the current reward value in each sample quintuple, the target value corresponding to each sample quintuple is calculated.

17. The device according to claim 15, characterized in that The parameter updating unit is also used to use a preset error calculation function to perform a root mean square calculation on the target value and the evaluation value to obtain the error corresponding to each sample quintuple; and use a root mean square back propagation algorithm to update the parameters of the preset online value network according to the error corresponding to each sample quintuple.

18. The device according to claim 15, characterized in that The parameter updating unit is further used to randomly extract the preset number of samples of original quintuples from the preset database; and for each of the preset number of samples of original quintuples, obtain from the preset database a plurality of related current grayscale images adjacent to the current grayscale image in each original quintuple and a plurality of related next step grayscale images adjacent to the next step grayscale image in each original quintuple; And each original quintuple, the multiple related current grayscale images and the multiple related next step grayscale images form a sample quintuple, and then obtain the sample quintuple of the preset number of samples.

19. A game control device, characterized in that: The device includes: a processor, a memory and a communication bus, the memory communicates with the processor via the communication bus, the memory stores one or more programs executable by the processor, and when the one or more programs are executed, the method according to any one of claims 1 to 9 is executed by the processor.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, and when the program is executed by at least one processor, the method according to any one of claims 1 to 9 is implemented.

Images