Method Of Evaluating Reinforcement Learning Model By Using Transformer Model And System Thereof
The transformer model-based evaluation system aligns reinforcement learning behaviors with actual user actions, providing a quantitative and objective assessment of reinforcement learning models in online games.
Patent Information
- Application Number
- KR1020220163473
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-11-29
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2042-11-29
AI Technical Summary
Reinforcement learning models in game algorithms face challenges in determining desirable agent behavior, as learning is evaluated by accumulated reward or loss values, which may not align with actual user actions.
A method and system using a transformer model to evaluate reinforcement learning models by comparing agent behaviors with actual user behaviors, adjusting reinforcement learning behaviors based on cost changes to align with user actions.
Quantitative and objective evaluation of reinforcement learning models is achieved, enhancing their performance to resemble actual user behavior in online games.
Smart Images

Figure 112022128176014-PAT00002_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a method and system for evaluating a reinforcement learning model using a transformer model, and more specifically, to a method and system for evaluating a reinforcement learning model using a transformer model for evaluating a reinforcement learning model applied to a specific online game. Background Technology
[0002] Reinforcement learning is the process of an agent learning the best course of action from its current state. Learning proceeds in a direction that maximizes the reward received from the environment for the actions performed by the reinforcement learning model. Reinforcement learning can be applied to fields that require adaptation to various environments and decision-making tailored to each specific situation (e.g., game algorithms, stock trading, robotics, and autonomous vehicles).
[0003] However, in the field of game algorithms, reinforcement learning has a limitation in that it is difficult to determine whether an agent's behavior is desirable in a given environment, as learning is evaluated by comparing accumulated reward or loss values based on the agent's actions. Furthermore, it can identify and execute behaviors as desirable even if the actual user playing the game does not perform them.
[0004] Meanwhile, the Transformer is a neural network that learns context and meaning by tracking the relationships between multiple sequential data, such as words within a sentence, and can be used in fields such as machine translation, abstract summarization, and speech recognition.
[0005] The technology forming the background of the present invention is disclosed in Korean Registered Patent Publication No. 10-2342458 (registered on Dec. 20, 2021). The problem to be solved
[0006] As such, according to the present invention, a method for evaluating a reinforcement learning model using a transformer model and a system for evaluating a reinforcement learning model applied to a specific online game are provided. means of solving the problem
[0007] According to an embodiment of the present invention for achieving such technical challenges, a method for evaluating a reinforcement learning model performed by a reinforcement learning model evaluation system using a transformer model comprises: receiving environmental information of a specific online game; deriving reinforcement learning behaviors through a previously trained reinforcement learning model using the received environmental information of the online game; storing a plurality of sets of derived reinforcement learning behaviors from the time an episode regarding a specific situation within the online game begins until the time an episode ends; receiving actual behavior data in the online game of a plurality of users who have played the online game; deriving transformer behaviors through a previously trained transformer model using the received actual behavior data; storing a plurality of sets of derived transformer behaviors from the time an episode regarding a specific situation within the online game begins until the time an episode ends; comparing the stored sets of reinforcement learning behaviors with the sets of transformer behaviors; if a change in reinforcement learning behavior is required according to the comparison result, changing the reinforcement learning behavior while increasing the cost; and evaluating the reinforcement learning model through the final cost.
[0008] The step of changing reinforcement learning behaviors while increasing the cost above may increase the cost by 1 when additional behaviors need to be input into the set of reinforcement learning behaviors, when specific behaviors need to be deleted, or when specific behaviors need to be modified.
[0009] In the step of evaluating the reinforcement learning above, the lower the final cost, the better the reinforcement learning performance of the reinforcement learning model can be evaluated.
[0010] The above environmental information is information representing the state of the online game at the current time and may include at least one of the following: game character location, item location, game character health, game character mana, information on various obstacles on the game screen, information on various buildings appearing on the game screen, game character job, game character skill, health of monsters in the game, and game character level.
[0011] The above episode is determined differently depending on the type of online game, and the step of comparing the stored reinforcement learning behavior set and the Transformer behavior set can measure the cost by comparing at least one of the total movement direction, total number of moves, movement direction at a specific point, and number of consecutive moves of the character in the online game from the start time to the end time of the above episode.
[0012] According to another embodiment of the present invention, a reinforcement learning model evaluation system using a transformer model may include: a reinforcement learning unit that receives specific online game environment information, derives reinforcement learning behaviors through a pre-trained reinforcement learning model using the input online game environment information, and stores a plurality of derived sets of reinforcement learning behaviors from the time an episode regarding a specific situation within the online game begins until the time an episode ends; a transformer learning unit that receives actual behavior data on an online game from a plurality of users who have played the online game, derives transformer behaviors through a pre-trained transformer model using the input behavior data, and stores a plurality of derived transformer behaviors by collecting them from the time an episode regarding a specific situation within the online game begins until the time an episode ends; a control unit that compares the stored set of reinforcement learning behaviors with the set of transformer behaviors, and if a change in reinforcement learning behavior is required according to the comparison result, changes the reinforcement learning behavior while increasing the cost; and an evaluation unit that evaluates the reinforcement learning model using the comparison result. Effects of the invention
[0013] As such, according to the present invention, the performance of a reinforcement learning model applied to an online game can be evaluated quantitatively and objectively by comparing the reinforcement learning model using a transformer model to which the user's actual behavior is input.
[0014] In addition, by generalizing to the actual behavior of a vast number of users using a Transformer model, it is possible to maximize rewards so that reinforcement learning resembles actual user behavior according to a specific online game environment. Brief explanation of the drawing
[0015] Figure 1 is a configuration diagram of a reinforcement learning model evaluation system using a transformer model according to an embodiment of the present invention. FIG. 2 is a flowchart of a reinforcement learning model evaluation method using a transformer model according to an embodiment of the present invention. Figure 3 is a conceptual diagram of the reinforcement learning model evaluation method illustrated in Figure 2. Figure 4 is a diagram illustrating a reinforcement learning model evaluation method applied to the Ms. Pacman game. Figure 5 is an example diagram to specifically explain the S290 method in the reinforcement learning model evaluation method applied to the Ms. Pacman game. Specific details for implementing the invention
[0016] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification have been given similar reference numerals.
[0017] Throughout the specification, when a part is described as "including" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.
[0018] Embodiments of the present invention are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the invention.
[0019] Figure 1 is a configuration diagram of a reinforcement learning model evaluation system using a transformer model according to an embodiment of the present invention.
[0020] Figure 1 is a configuration diagram of a reinforcement learning model evaluation system using a transformer model according to an embodiment of the present invention.
[0021] Referring to FIG. 1, a reinforcement learning model evaluation system (100) using a transformer model according to an embodiment of the present invention includes a reinforcement learning unit (110), a transformer learning unit (120), a control unit (130), and an evaluation unit (140).
[0022] First, the reinforcement learning unit (110) receives specific online game environment information and derives reinforcement learning behaviors through a previously trained reinforcement learning model using the input online game environment information, and stores a set of multiple reinforcement learning behaviors derived from the time an episode regarding a specific situation in the online game begins until the time the episode ends. Here, the environment information is information representing the state of the online game at the current time and may include at least one of the following: game character location, item location, game character health, game character mana, various obstacle information on the game screen, various building information appearing on the game screen, game character job, game character skill, health of monsters in the game, and game character level.
[0023] In addition, episodes may be determined differently depending on the type of online game.
[0024] Next, the Transformer learning unit (120) receives actual behavioral data in the online game from multiple users who have played the online game, derives Transformer behaviors through a previously learned Transformer model using the input behavioral data, and collects and stores multiple Transformer behaviors derived from the time an episode about a specific situation in the online game starts until the time an episode ends.
[0025] And, the control unit (130) compares the stored set of reinforcement learning behaviors with the set of transformer behaviors, and if a change in the reinforcement learning behavior is required based on the comparison result, changes the reinforcement learning behavior while increasing the cost.
[0026] Specifically, the control unit (130) can increase the cost by 1 if, depending on the comparison result, additional input to the reinforcement learning behavior set is required, a specific behavior is deleted, or a specific behavior is modified.
[0027] Additionally, the control unit (130) can measure the cost by comparing at least one of the total movement direction, total number of movements, movement direction at a specific point, and number of consecutive movements of a character in an online game from the start time of an episode to the end time.
[0028] And, the evaluation unit (140) evaluates the reinforcement learning model using the comparison results.
[0029] Specifically, the evaluation unit (140) can evaluate that the lower the final cost, the better the reinforcement learning performance of the reinforcement learning model.
[0030] Hereinafter, a reinforcement learning model evaluation system (100) according to an embodiment of the present invention will be described in detail using FIGS. 2 to 5. In the embodiment described below, a specific example is provided for evaluating a reinforcement learning model for the Ms. Pacman game. However, the present invention is not limited to this game.
[0031] FIG. 2 is a flowchart of a reinforcement learning model evaluation method using a transformer model according to an embodiment of the present invention, FIG. 3 is a conceptual diagram of the reinforcement learning model evaluation method illustrated in FIG. 2, and FIG. 4 is a diagram illustrating a reinforcement learning model evaluation method applied to the Ms. Pacman game.
[0032] As illustrated in FIGS. 2 and 3, the reinforcement learning unit (110) receives environment information of a specific online game (S210).
[0033] According to one embodiment, the reinforcement learning unit (110) may receive environment information as environment information of the Ms. Pacman game, including at least one of Pacman's health, at least one ghost movement, the location of an obstacle on the game screen, a game end condition, and the number of pellets on the screen.
[0034] And, the reinforcement learning unit (110) derives reinforcement learning behaviors through a previously learned reinforcement learning model using environment information of the input online game (S220).
[0035] According to one embodiment, the reinforcement learning unit (110) can derive a plurality of reinforcement learning behaviors configured to acquire the user's pellets and avoid ghosts through a reinforcement learning model learned such that the reward increases as the pellets are acquired.
[0036] The reinforcement learning unit (110) stores a set of multiple reinforcement learning behaviors derived from the time an episode about a specific situation in an online game starts until the time an episode ends (S230).
[0037] As illustrated in FIG. 4a, according to one embodiment, the reinforcement learning unit (110) can store a set of reinforcement learning behaviors (set of directions for acquiring pellets and avoiding ghosts) derived from the time a round starts in the Ms. Pacman game until the round ends when all pellets are acquired or Pacman's health is reduced to the point when the round ends, such as {←,←,↓,↓,↓,↓,→,→,→,→,←,←,←,←,←,←,←,←,←,←,↑,↑,↑,→,→,↑,↑,↑,↑,↑,↑,↑,↑,↑,↑,↑,↑,↑,↑,↑,↑,→,→,→,→,→,→,→,→,→,→,→,→,→,→,↓,↓,↓,↓,↓,↓,↓,↓,↓,↑,↑,↑,↑,↑,↑,↑,←,←,←,←,←,←,←,←,←}.
[0038] Next, the Transformer learning unit (120) receives actual behavioral data in the online game from multiple users who have played the online game (S240).
[0039] According to one embodiment, the Transformer learning unit (120) can receive actual behavior data in the same game environment as the environment information of the Ms. Pacman game input to the reinforcement learning unit (110) of a plurality of users.
[0040] Next, the transformer learning unit (120) derives transformer behavior through a previously learned transformer model using input multiple actual behavior data (S250).
[0041] According to one embodiment, the transformer learning unit (120) can derive a plurality of transformer actions configured to acquire the user's pellet and avoid ghosts.
[0042] Next, the Transformer learning unit (120) stores a set of multiple Transformer actions derived from the time an episode about a specific situation in the online game starts until the time an episode ends (S260).
[0043] As illustrated in FIG. 4b, according to one embodiment, the Transformer learning unit (110) can store a set of Transformer learning behaviors (a set of directions for acquiring pellets and avoiding ghosts) derived from the time a round starts in the Ms. Pacman game until the round ends when all pellets are acquired or Pacman's health is reduced to the point when the round ends, such that the round ends when all pellets are acquired or Pacman's health is reduced to the point when all of Pacman's health is reduced, namely {←,←,↓,↓,↓,↓,→,→,→,→,→,←,←,←,←,←,←,←,←,←,↑,↑,↑,→,→,→,↑,↑,↑,↑,↑,↑,↑,↑,↑,↑,↑,↑,↑,←,←,←,←,→,→,→,→,→,→,→,→,→,→,↓,↓,↓,↓,↓,↓,←,↓,↓,↑,↑,↑,↑,↑,↑,↑,←,←,←,←,↑,↑,↑,↑,↑,←,←,←,→,↓,↑,↑}. there is.
[0044] Next, the control unit (130) compares the stored set of reinforcement learning behaviors with the set of transformer behaviors (S270).
[0045] If the control unit (130) needs to change the reinforcement learning behavior based on the comparison result, it changes the reinforcement learning behavior while increasing the cost (S280).
[0046] Specifically, the control unit (130) can increase the cost by 1 when additional actions need to be input into the reinforcement learning action set, when specific actions need to be deleted, or when specific actions need to be modified, until the point where the reinforcement learning action set becomes identical to the transformer action set.
[0047] The evaluation unit (140) evaluates the reinforcement learning model through the final cost (S290).
[0048] Specifically, the evaluation unit (140) can evaluate that the reinforcement learning performance of the reinforcement learning model is superior as the final cost accumulated until the point where the reinforcement learning behavior set becomes identical to the transformer behavior set is lower.
[0049] Figure 5 is an example diagram to specifically explain the S290 method in the reinforcement learning model evaluation method applied to the Ms. Pacman game.
[0050] As illustrated in FIG. 5, the evaluation unit (140) can evaluate whether the performance of the reinforcement learning model is excellent through the final cost accumulated while changing the reinforcement learning behaviors in the control unit (130) until the set of reinforcement learning behaviors becomes identical to the set of transformer behaviors.
[0051] Additionally, the evaluation unit (140) can provide the reinforcement learning model evaluation results to a user or a terminal connected to the reinforcement learning model evaluation system (100) via a network.
[0052] As such, according to an embodiment of the present invention, the performance of a reinforcement learning model applied to an online game can be evaluated quantitatively and objectively by comparing the reinforcement learning model using a transformer model into which the user's actual behavior is input.
[0053] The present invention has been described with reference to the embodiments illustrated in the drawings, but this is merely illustrative, and those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the technical spirit of the following claims. Explanation of the symbols
[0054] 100: Reinforcement Learning Model Evaluation System 110: Reinforcement Learning Unit 120: Transformers Learning Department 130: Control unit 140: Evaluation Department
Claims
Claim 1 A method for evaluating a reinforcement learning model performed by a reinforcement learning model evaluation system using a Transformer model, comprising: receiving environmental information of a specific online game; deriving reinforcement learning behaviors through a pre-trained reinforcement learning model using the received environmental information of the online game; storing a set of multiple derived reinforcement learning behaviors from the time an episode regarding a specific situation within the online game begins until the time an episode ends; receiving actual behavior data in the online game of multiple users who have played the online game; deriving Transformer behaviors through a pre-trained Transformer model using the received actual behavior data; storing a set of multiple derived Transformer behaviors from the time an episode regarding a specific situation within the online game begins until the time an episode ends; comparing the stored set of reinforcement learning behaviors with the set of Transformer behaviors; and, if a change in the reinforcement learning behavior is required according to the comparison result, changing the reinforcement learning behavior while increasing the cost. A method for evaluating a reinforcement learning model that includes the step of evaluating the reinforcement learning model through a final cost, wherein the step of changing reinforcement learning behaviors while increasing the cost increases the cost by 1 when additional behaviors must be input into the set of reinforcement learning behaviors, when specific behaviors must be deleted, or when specific behaviors must be modified. Claim 2 delete Claim 3 In claim 1, the step of evaluating the reinforcement learning model is a reinforcement learning model evaluation method in which the lower the final cost, the better the reinforcement learning performance of the reinforcement learning model is evaluated. Claim 4 A method for evaluating a reinforcement learning model according to claim 1, wherein the environment information is information representing the state of an online game at a current time, and includes at least one of the following: game character location, item location, health of the game character, mana of the game character, various obstacle information on the game screen, various building information appearing on the game screen, job of the game character, skill of the game character, health of a monster in the game, and level of the game character. Claim 5 In claim 1, the episode is determined differently depending on the type of online game, and the step of comparing the stored reinforcement learning behavior set and the Transformer behavior set is a reinforcement learning model evaluation method that measures the cost by comparing at least one of the total movement direction, total number of moves, movement direction at a specific point, and number of consecutive moves of a character in the online game from the start time to the end time of the episode. Claim 6 In a reinforcement learning model evaluation system using a Transformer model, the system comprises: a reinforcement learning unit that receives specific online game environment information, derives reinforcement learning behaviors through a pre-trained reinforcement learning model using the input online game environment information, and stores a set of multiple derived reinforcement learning behaviors from the time an episode regarding a specific situation within the online game begins until the time the episode ends; a Transformer learning unit that receives actual behavior data in the online game from multiple users who have played the online game, derives Transformer behaviors through a pre-trained Transformer model using the input behavior data, and stores a set of multiple derived Transformer behaviors from the time an episode regarding a specific situation within the online game begins until the time the episode ends; and a control unit that compares the stored set of reinforcement learning behaviors with the set of Transformer behaviors, and if a change in reinforcement learning behavior is required based on the comparison result, changes the reinforcement learning behavior while increasing the cost. A reinforcement learning model evaluation system comprising an evaluation unit that evaluates a reinforcement learning model using the above comparison results, wherein the control unit increases the cost by 1 when, according to the above comparison results, an action must be additionally input into a set of reinforcement learning actions, a specific action must be deleted, or a specific action must be modified. Claim 7 delete Claim 8 In paragraph 6, the above evaluation unit is a reinforcement learning model evaluation system that evaluates the reinforcement learning performance of a reinforcement learning model as being superior when the final cost is lower. Claim 9 In claim 6, the above-mentioned environment information is information representing the state of the online game at a current time, and includes at least one of the game character location, item location, game character health, game character mana, various obstacle information on the game screen, various building information appearing on the game screen, game character job, game character skill, health of monsters in the game, and game character level, in a reinforcement learning model evaluation system. Claim 10 In claim 6, the above episode is determined differently depending on the type of online game, and the control unit is a reinforcement learning model evaluation system that measures the cost by comparing at least one of the total movement direction, total number of moves, movement direction at a specific point, and number of consecutive moves of a character in the online game from the start time to the end time of the above episode.
Citation Information
Patent Citations
Memory-based reinforcement learning method for storing optional information in streaming data and system therefore
KR1020210105803A
System and method for training ai character type of NPC for virtual training
KR1020220015523A
