Method and device for generating sub-action determination artificial intelligence model based on importance assessment, and method and device for setting monte carlo tree search space using same

Abstract

One embodiment of the present invention proposes an artificial intelligence model generation method performed by at least one processor. The method comprises a step for generating an artificial intelligence model trained to output a set of one or more major sub-actions for an input specific state variable according to repeating the steps of: selecting, on the basis of an inference problem, major sub-actions from among sub-actions corresponding to the current state variable of the inference problem, and generating the next state variable on the basis of the current state variable and the major sub-actions; and selecting, on the basis of the inference problem, major sub-actions from among the sub-actions corresponding to the current state variable of the inference problem by using the next state variable as a new current state variable, and generating the next state variable on the basis of the current state variable and the major sub-actions.

Description

Method and device for creating an artificial intelligence model for sub-action decision-making based on importance evaluation, and method and device for setting a Monte Carlo tree search space using the same

[0001] The present invention relates to a method and device for generating an artificial intelligence model for determining sub-actions based on importance evaluation, and a method and device for setting a Monte Carlo tree search space using the same. More specifically, the present invention relates to a method and device for generating an artificial intelligence model trained to output a set of one or more major sub-actions for an input specific state variable, and setting a Monte Carlo tree search space using the artificial intelligence model.

[0002] Monte Carlo Tree Search (MCTS) is an algorithm that models the decision-making process using probabilistic sampling to arrive at optimal decisions in uncertain environments. It operates by pre-evaluating the outcomes of possible actions through simulation and selecting the optimal action based on this evaluation.

[0003] However, when the action space is very large or the rules of the problem to be solved are complex, the number of possible paths to be explored increases exponentially, leading to a decrease in search efficiency. For example, games like Go or chess present numerous possible paths per turn, dramatically increasing the time, resources, and cost required to explore all possible paths.

[0004] Particularly in the field of robot control, the problem of combining multiple sub-actions to form a single action frequently arises. In this case, the size of the action space is calculated as the product of the possible scenarios for each sub-action. Therefore, applying Monte Carlo tree search poses the problem of a very large search space.

[0005] Recently, in order to improve the speed of the algorithm for spatial keyword queries, a prior art 1 was proposed for a method of estimating spatial costs and reducing the search range based on the estimated spatial costs.

[0006] However, prior art 1 has the limitation that it only adjusts the search range based on the spatial cost of the input candidate group and does not reduce the search space itself, and does not selectively consider only important sub-behaviors.

[0007] Accordingly, there is a need for a method for generating an artificial intelligence model that reduces the search space of a Monte Carlo tree search algorithm by selectively considering only important sub-behaviors, and a method for setting a Monte Carlo tree search space using the same.

[0008] Prior Art 1: Republic of Korea Patent Publication No. 10-2202500, "Search Range Reduction Technique for Improving Search Speed ​​in Spatial Keyword Cover Problems" (Registered January 7, 2021)

[0009] In order to solve the above-described technical problem, we aim to provide a method and device for creating an artificial intelligence model that determines sub-actions based on an evaluation of the importance of sub-actions corresponding to the current state variables of an inference problem.

[0010] In addition, in order to solve the above-described technical problem, the present invention provides a method and device for setting a Monte Carlo tree search space using an importance evaluation-based lower-order action decision artificial intelligence model.

[0011] The technical problems to be solved by the present invention are not limited to the technical problems described above, and other technical problems of the present invention can be derived from the following description.

[0012] In order to solve the above-described technical problem, one embodiment of the present invention provides a method for generating an artificial intelligence model for determining sub-actions based on importance assessment, which is performed by at least one processor. The method includes a step of selecting major sub-actions from among sub-actions corresponding to a current state variable of an inference problem based on an inference problem, generating a next state variable based on the current state variable and the major sub-actions, and a step of selecting major sub-actions from among sub-actions corresponding to a current state variable of the inference problem based on the logic problem with the next state variable as a new current state variable, and generating a next state variable based on the current state variable and the major sub-actions, thereby generating an artificial intelligence model trained to output a set of one or more major sub-actions for an input specific state variable.

[0013] In addition, another embodiment of the present invention provides an artificial intelligence model generation device for determining sub-actions based on importance evaluation. The device includes a communication module, at least one processor, and a memory electrically connected to the processor and storing at least one code to be executed by the processor. The memory stores a code that, when executed through the processor, causes the processor to select major sub-actions from among sub-actions corresponding to a current state variable of the inference problem based on an inference problem, generate a next state variable based on the current state variable and the major sub-actions, and select major sub-actions from among sub-actions corresponding to a current state variable of the inference problem based on the inference problem with the next state variable as a new current state variable, and generate a next state variable based on the current state variable and the major sub-actions, thereby generating an artificial intelligence model trained to output a set of one or more major sub-actions for an input specific state variable.

[0014] In addition, another embodiment of the present invention provides a method for setting a Monte Carlo tree search space using an artificial intelligence model. The method includes a step of selecting major sub-actions from among sub-actions corresponding to a current node of a Monte Carlo tree node based on a preset Monte Carlo tree node, generating a next node based on the current node and the major sub-actions, and a step of selecting major sub-actions from among sub-actions corresponding to a current node of the Monte Carlo tree node based on the preset Monte Carlo tree node with the next node as a new current node, and generating a next node based on the current node and the major sub-actions, thereby setting a next node for the specific node as a search space using an artificial intelligence model trained to output a set of one or more major sub-actions for the specific node.

[0015] In addition, another embodiment of the present invention provides a device for setting a Monte Carlo tree search space using an artificial intelligence model. The device includes a communication module, at least one processor, and a memory electrically connected to the processor and storing at least one code to be executed by the processor. The memory stores a code that, when executed by the processor, causes the processor to select major sub-actions from among sub-actions corresponding to a current node of the Monte Carlo tree node based on a preset Monte Carlo tree node, generate a next node based on the current node and the major sub-actions, and select major sub-actions from among sub-actions corresponding to the current node of the Monte Carlo tree node based on the preset Monte Carlo tree node with the next node as a new current node, and to repeat the process of generating a next node based on the current node and the major sub-actions, thereby outputting a set of one or more major sub-actions for a specific node, using an artificial intelligence model trained to set a next node for the specific node as a search space.

[0016] According to the solution to the problem of the present invention described above, it is possible to learn the relationship between a specific state and a sub-action from high-dimensional observation data.

[0017] In addition, according to the solution to the problem of the present invention described above, it is possible to perform a search for a task with high importance as a priority within limited resources.

[0018] In addition, according to the solution to the problem of the present invention described above, it is possible to perform fluid action abstraction depending on the state.

[0019] In addition, according to the solution to the problem of the present invention described above, the search space is reduced, enabling search with less computational effort.

[0020] In addition, according to the solution to the problem of the present invention described above, the interpretability of the decision-making process of the Monte Carlo tree search model can be increased.

[0021] The effects of the present invention are not limited to the effects described above, and include all effects understood from the following description.

[0022] FIG. 1 is a drawing showing a detailed configuration of a device according to one embodiment of the present invention.

[0023] Figure 2 is a diagram illustrating a process in which the device illustrated in Figure 1 trains an artificial intelligence model.

[0024] FIG. 3 is a diagram illustrating the process of setting up a Monte Carlo tree search space by the device illustrated in FIG. 1.

[0025] Figures 4 to 9 are drawings illustrating an embodiment using the device illustrated in Figure 1.

[0026] FIG. 10 is a flowchart illustrating the sequence of a method for creating an artificial intelligence model for determining subordinate actions based on importance evaluation according to another embodiment of the present invention.

[0027] Figures 11 and 12 are flowcharts illustrating additional steps included in the method for creating a sub-action decision artificial intelligence model based on importance assessment illustrated in Figure 10.

[0028] Figure 13 is a flowchart illustrating the sequence of a method for setting a Monte Carlo tree search space using an artificial intelligence model according to another embodiment of the present invention.

[0029] FIGS. 14 and 15 are flowcharts illustrating additional steps included in the Monte Carlo tree search space setting method using the artificial intelligence model illustrated in FIG. 13.

[0030] Hereinafter, the present invention will be described in detail with reference to the attached drawings. However, the present invention can be implemented in various different forms and is not limited to the embodiments described herein. In addition, the attached drawings are only intended to facilitate understanding of the embodiments of the invention disclosed herein, and the technical ideas disclosed herein are not limited by the attached drawings. All terms, including technical and scientific terms, used herein should be interpreted as having meanings generally understood by a person of ordinary skill in the art to which the present invention pertains. Terms defined in the dictionary should be interpreted as having additional meanings consistent with the relevant technical literature and the present invention, and shall not be interpreted in an extremely ideal or restrictive sense unless otherwise defined.

[0031] In order to clearly explain the present invention in the drawings, parts irrelevant to the description have been omitted, and the size, shape, and appearance of each component shown in the drawings may be modified in various ways. Identical / similar parts throughout the specification are given identical / similar drawing reference numerals.

[0032] The suffixes "module" and "function" used in the following description for components are assigned or used interchangeably solely for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. Furthermore, in describing the embodiments disclosed herein, detailed descriptions of related known technologies have been omitted if they are deemed to obscure the gist of the embodiments disclosed herein.

[0033] Throughout the specification, when a part is said to be "connected (connected, in contact with, or coupled)" to another part, this includes not only cases where it is "directly connected (connected, in contact with, or coupled)" but also cases where it is "indirectly connected (connected, in contact with, or coupled)" with another member in between. Furthermore, when a part is said to "include (have or provide)" a certain component, this does not mean that it excludes other components, but rather that it may "include (have or provide)" other components, unless otherwise specifically stated.

[0034] As used herein, ordinal terms such as "first," "second," etc., are used solely to distinguish one component from another and do not limit the order or relationship of the components. For example, the first component of the present invention may be referred to as the "second component," and similarly, the second component may also be referred to as the "first component." As used herein, singular expressions should be construed to include plural expressions, unless explicitly stated otherwise.

[0035] FIG. 1 is a drawing showing a detailed configuration of a device according to one embodiment of the present invention.

[0036] In one example, the device (100) may be at least one of an importance evaluation-based sub-action decision artificial intelligence model generation device (hereinafter referred to as an “AI model generation device”) and an AI model-based Monte Carlo tree search space setting device (hereinafter referred to as a “search space setting device”). In this case, the AI ​​model generation device and the search space setting device may be referred to as an AI model generation system and a search space setting system, respectively.

[0037] The artificial intelligence model generation device may include a communication module (110), a processor (120) that performs operations according to code stored in a memory (130), and a memory (130) that stores the code.

[0038] The AI ​​model generation device can be implemented as a computer or portable terminal that can connect to a server or other terminal via a network. Here, the computer includes, for example, a notebook, desktop, or laptop equipped with a web browser, and the portable terminal can include, for example, a wireless communication device that guarantees portability and mobility, and can include all types of handheld-based wireless communication devices such as various types of communication-based terminals, smartphones, and tablet PCs. In addition, the portable terminal can be an edge device or on-device AI having at least one processor capable of AI model inference work. The network can be implemented as a wired network such as a Local Area Network (LAN), a Wide Area Network (WAN), or a Value Added Network (VAN), or any type of wireless network such as a mobile radio communication network or satellite communication network.

[0039] In the artificial intelligence model generation device, the communication module (110) may include a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals through wired or wireless connections with other network devices. In a mobile communication network constructed according to technical standards or communication methods for mobile communication used in the mobile communication module (e.g., GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA (Wideband CDMA), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTEA (Long Term Evolution-Advanced), etc.), a wireless signal is transmitted and received with at least one of a base station, an external terminal, and a server.

[0040] In the artificial intelligence model generation device, the processor (120) may include various types of devices that control and process data. The processor (120) may refer to a data processing device built into hardware that has a physically structured circuit to perform a function expressed by a code or command included in a program. In one example, the processor (120) may be implemented in the form of a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., but the scope of the present invention is not limited thereto.

[0041] In the artificial intelligence model generation device, the memory (130) can store at least one of information and data input to the communication module (110), information and data required for a function performed by the processor (120), and data generated according to the execution of the processor (120).

[0042] In the artificial intelligence model generation device, the memory (130) should be interpreted as a general term for a non-volatile storage device that maintains stored information even when no power is supplied and a volatile storage device that requires power to maintain the stored information. In addition to a volatile storage device that requires power to maintain the stored information, the memory (130) may include cloud storage, SSD, magnetic storage media, or flash storage media, but the scope of the present invention is not limited thereto.

[0043] In the artificial intelligence model generation device, the memory (130) is electrically connected to the processor (120) and stores at least one code that is executed by the processor (120). The memory (130) stores a code that causes the processor (120) to perform the following functions and procedures when executed by the processor (120).

[0044] In an artificial intelligence model generation device, the memory (130) stores code that causes the processor (120) to select a primary sub-action from among sub-actions corresponding to the current state variables of the inference problem based on the inference problem, and to generate the next state variable based on the current state variable and the primary sub-actions. The inference problem may include at least one of an image and text representing a specific situation and information about the specific situation. The inference problem may include any environment in which a specific task is selected based on multiple scenarios, such as a game, robot control, customized service provision, or autonomous driving system. The current state variable may include the current situation of the user in the inference problem (environment). For example, in a Go environment, it may include the current arrangement of black and white stones; in a game environment, it may include the current location of surrounding obstacles and players; and in a customized service provision environment, it may include the current user's needs and a list of required services. The sub-actions corresponding to the current state variable may include all actions that the user can select for the current situation, and each sub-action may be performed independently of the other. For example, in a Go environment, it could include all possible positions for placing a Go stone, such as placing a black stone at coordinate K10, placing a black stone at coordinate D3, etc. In a game environment, it could include all possible moves based on the given situation, such as moving right or moving left.

[0045] In the artificial intelligence model generation device, the memory (130) stores a code that causes the processor (120) to generate an artificial intelligence model trained to output a set of one or more major sub-behaviors for an input specific state variable by repeating the process of selecting major sub-behaviors from among sub-behaviors corresponding to the current state variable of an inference problem based on an inference problem with the next state variable as a new current state variable and the process of generating the next state variable based on the current state variable and the major sub-behaviors. The artificial intelligence model may be trained using specific inference problems, current state variables for the specific inference problems, and sub-behaviors corresponding to the current state variables for the specific inference problems as learning data.

[0046] In the artificial intelligence model generation device, the memory (130) may further store code that causes the processor (120) to set sub-actions corresponding to the current state variables. The memory (130) may further store code that causes the processor (120) to calculate the importance of the sub-actions and select a primary sub-action from among the sub-actions based on the importance. The memory (130) may further store code that causes the processor (120) to generate a variable of the next state predicted after the current state variable based on the current state variable and the primary sub-action.

[0047] In the artificial intelligence model generation device, the memory (130) may further store a code that causes the processor (120) to calculate the importance of sub-actions and select a major sub-action from among the sub-actions based on the importance, learn the relationship between the current state variable and the sub-actions using a preset deep neural network to calculate the importance, and select major sub-actions by masking sub-actions that have an importance that satisfies a preset criterion among the sub-actions. The preset deep neural network may be set based on an encoder / decoder structure, and may be implemented based on a convolutional neural network (CNN) to process state information in the form of an image.

[0048] The search space setting device may include a communication module (110), a processor (120) that performs operations according to code stored in a memory (130), and a memory (130) that stores the code.

[0049] The search space setting device can be implemented as a computer or portable terminal that can connect to a server or other terminal via a network. Here, the computer includes, for example, a notebook, desktop, or laptop equipped with a web browser, and the portable terminal can include, for example, a wireless communication device that guarantees portability and mobility, and can include all types of handheld-based wireless communication devices such as various types of communication-based terminals, smartphones, and tablet PCs. In addition, the portable terminal can be an edge device or on-device AI having at least one processor capable of AI model inference work. The network can be implemented as a wired network such as a Local Area Network (LAN), a Wide Area Network (WAN), or a Value Added Network (VAN), or any type of wireless network such as a mobile radio communication network or a satellite communication network.

[0050] In the search space setting device, the communication module (110) may include a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals through wired or wireless connections with other network devices. In a mobile communication network constructed according to technical standards or communication methods for mobile communication used in the mobile communication module (e.g., GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA (Wideband CDMA), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTEA (Long Term Evolution-Advanced), etc.), a wireless signal is transmitted and received with at least one of a base station, an external terminal, and a server.

[0051] In the search space setting device, the processor (120) may include various types of devices that control and process data. The processor (120) may refer to a data processing device built into hardware that has a physically structured circuit to perform a function expressed by a code or command included in a program. In one example, the processor (120) may be implemented in the form of a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., but the scope of the present invention is not limited thereto.

[0052] In the search space setting device, the memory (130) can store at least one of information and data input to the communication module (110), information and data required for a function performed by the processor (120), and data generated according to the execution of the processor (120).

[0053] In the search space setting device, the memory (130) should be interpreted as a general term for a non-volatile storage device that maintains stored information even when no power is supplied and a volatile storage device that requires power to maintain the stored information. In addition to a volatile storage device that requires power to maintain the stored information, the memory (130) may include cloud storage, SSD, magnetic storage media, or flash storage media, but the scope of the present invention is not limited thereto.

[0054] In the search space setting device, the memory (130) is electrically connected to the processor (120) and stores at least one code that is executed by the processor (120). The memory (130) stores a code that causes the processor (120) to perform the following functions and procedures when executed by the processor (120).

[0055] In the search space setting device, the memory (130) stores a code that causes the processor (120) to select major sub-actions from among sub-actions corresponding to the current node of the Monte Carlo tree node based on the preset Monte Carlo tree node, and to generate the next node based on the current node and the major sub-actions.

[0056] In a search space setting device, a memory (130) stores a code that causes the processor (120) to set the next node for the specific node as a search space by using an artificial intelligence model trained to output a set of one or more major sub-behaviors for a specific node, as the processor (120) repeats the process of selecting major sub-behaviors from among sub-behaviors corresponding to the current node of the Monte Carlo tree node based on the preset Monte Carlo tree node with the next node as a new current node and the process of generating a next node based on the current node and the major sub-behaviors. The artificial intelligence model may be trained using a Monte Carlo tree node, a current node for the Monte Carlo tree node, and sub-behaviors corresponding to the current nodes for the Monte Carlo tree node as learning data.

[0057] In the search space setting device, the memory (130) may further store code that causes the processor (120) to set sub-actions corresponding to the current node. The memory (130) may further store code that causes the processor (120) to calculate the importance of the sub-actions and select a primary sub-action among the sub-actions based on the importance. The memory (130) may further store code that causes the processor (120) to generate a next node predicted to follow the current node based on the current node and the primary sub-action.

[0058] In the search space setting device, the memory (130) may further store a code that causes the processor (120) to calculate the importance of sub-actions and select a major sub-action from among the sub-actions based on the importance, learn the relationship between the current node and the sub-actions using a preset deep neural network to calculate the importance, and perform masking processing on sub-actions having an importance that satisfies a preset criterion among the sub-actions to select the major sub-actions.

[0059] Figure 2 is a diagram illustrating a process in which the device illustrated in Figure 1 trains an artificial intelligence model.

[0060] The device (100) may be at least one of an artificial intelligence model generation device and a search space generation device. The process described below may be performed by the processor (130) and memory (120) of the device (100) described above with reference to FIG. 1. Therefore, any details that overlap with the above description will be omitted.

[0061] The device (100) is a deep neural network implemented based on a Convolutional Neural Network (CNN). The following operations can be performed using the device (100). The device (100) provides status information for a specific situation. Encode the current state variable (S11), extract sub-actions corresponding to the current state variable from the generated current state variable, and determine the importance of the sub-actions. (S12), the device (100) represents the state information as a potential state You can create a current state variable by mapping it to Importance from can be calculated as in the mathematical formula 1 below. The device (100) From mask It can be sampled as in mathematical formula 2 below.

[0062]

[0063] In the above mathematical expression 1, each represents the importance of each sub-action and can be a parameter of the Bernoulli distribution.

[0064]

[0065] In the above mathematical expression 2, each refers to masking data for each sub-action.

[0066] The device (100) is an important sub-behavior whose importance is below a preset threshold. is selected (S13). The device (100) is a state variable , important sub-actions Current state variables based on Next predicted state variable Creates (S14).

[0067] The device (100) is a predicted state variable The importance calculation step (S12) and the next state variable creation step (S14) are repeated with the new current state variable as the new current state variable. In more detail, Calculate the importance of the corresponding sub-actions, Important sub-actions among the sub-actions whose importance is below a preset threshold Select and and Based on this, the next predicted state variable is Repeat the process of creating .

[0068] The device (100) uses state information, state variables for the state information, and sub-behaviors corresponding to the state variables as learning data, and uses the predicted state variables as state variables in the importance calculation step (S12) of the sub-behaviors, and repeats the importance calculation step (S12) to the next state variable generation step (S14), thereby generating an artificial intelligence model trained to output a set of one or more major sub-behaviors for an input specific state variable.

[0069] FIG. 3 is a diagram illustrating the process of setting up a Monte Carlo tree search space by the device illustrated in FIG. 1.

[0070] The device (100) may be at least one of an artificial intelligence model generation device and a search space setting device. The process described below may be performed by the processor (130) and memory (120) of the device (100) described above with reference to FIG. 1. Therefore, any content overlapping with the above description will be omitted. The process of the device (100) setting the Monte Carlo tree search space may include the importance calculation step (S11) to the next state variable generation step (S14) described with reference to FIG. 2, and any content overlapping with the above description will be omitted below.

[0071] The device (100) is an abstracted action space defined by the following mathematical formula 3 in the importance calculation step (S12) described with reference to FIG. 2. can be constructed. The abstracted action space can include sampling from the Bernoulli distribution.

[0072]

[0073] In the above mathematical formula 3, refers to a behavioral variable, When it is related to state transition, Otherwise, it has nothing to do with state transitions.

[0074] The device (100) sets a threshold value according to the following mathematical expression 4 for the abstracted action space defined by the above mathematical expression 3. The search space can be set by masking sub-actions with the following importance levels.

[0075]

[0076] In the above mathematical formula 4, means a behavioral variable, and each represents the importance of each sub-behavior.

[0077] The device (100) sets a sub-action set according to the mathematical expression 5 below in the search space set according to the mathematical expression 4. is optionally considered.

[0078]

[0079] In the above mathematical expression 5, refers to the initial probability distribution (prior) of the policy. is the same abstraction space Actions mapped to can be marginalized.

[0080] Figures 4 to 11 are drawings illustrating embodiments using the device illustrated in Figure 1. The present invention will be further described below using embodiments. The embodiments were evaluated through the following tests.

[0081] This embodiment augments MuZero using a Conditional Structure Inference Network. Throughout this embodiment, MuZero is used in its default state, with no enhancements or modifications. In this embodiment, the abstraction threshold is used. We use . MuZero is trained with over 100,000 gradient steps in all environments, and the performance of each run is evaluated with 32 seeds every 2,000 steps. All experiments were performed using JAX and Haiku on an NVIDIA RTX 3090 GPU.

[0082] This embodiment is a process of solving a problem in a Doorkey environment using an MCTS agent to which the present invention is applied. Fig. 4 is a diagram illustrating a Doorkey environment. Referring to Fig. 4, in the Doorkey environment, an agent moves in a grid-shaped action space and must obtain a key and open a door to reach a target point. The action space is is factorialized, where class corresponds to the agent's movement, class corresponds to the interaction between a key and a door. The state includes the agent's location, the key's location, and the door's state (locked / unlocked). In this embodiment, state information in the form of an image is used. The action includes multiple sub-actions such as 'move forward', 'move backward', 'turn left', 'turn right', 'pick up the key', and 'open the door'. Each sub-action can be performed independently, and various actions can be taken by combining sub-actions. The agent receives a reward of -0.1 for each step. The door configuration, key, wall, and initial position of the agent are randomly initialized at the beginning of each episode. The properties of the door and key are also randomly initialized in each episode. There are three difficulty levels: EASY, NORMAL, and HARD, and the number of properties for each setting is 2, 3, and 4, respectively.

[0083] In this embodiment, the agent uses a conditional structure inference network to selectively consider only the sub-actions that are important in the current state. For example, if the agent has already obtained the key, actions like "pick up the key" no longer need to be considered. By eliminating sub-actions irrelevant to the current state, the search space is effectively reduced.

[0084] Conditional Structure Inference Network learns the CSI relationship between state and action variables. In this embodiment, the output from different states of Doorkey Visualize the output here is identical to the above mathematical expression 1.

[0085] Figure 5 is a diagram illustrating the process of selectively considering only important sub-actions in the current state using a conditional structure inference network. The vertical axis of the graph in Figure 5 represents probability, while the horizontal axes 0 and 1 represent sub-actions related to the agent's location, 2 represents sub-actions related to the key, and 3 represents sub-actions related to the door.

[0086] Referring to Figure 5, (a) shows that the sub-action (2) corresponding to the key and the sub-action (3) corresponding to the door are assigned a probability of 0 because the agent has already obtained the key and cannot interact with the door at this moment. (b) shows that the sub-action (2) corresponding to the key is assigned a probability of 1 because the agent can pick up the key. The sub-action (3) corresponding to the door is assigned a probability of 0 because the agent still cannot interact with the door. Similarly, (c) shows that the sub-action (2) corresponding to the key and the sub-action (3) corresponding to the door are assigned a probability of 0 because the agent has already obtained the key and opened the door. (d) shows that the sub-action (3) corresponding to the key is assigned a probability of 0 and the sub-action (3) corresponding to the door is assigned a probability of 1 because the agent can interact with the door after it has already been opened.

[0087] Figure 6 is a diagram illustrating the importance prediction of a conditional structure inference network for sub-actions corresponding to keys and doors along the path of Doorkey.

[0088] Conditional structure inference networks can be implemented based on convolutional neural networks (CNNs) and can process state information in image form. Conditional structure inference networks transform input states into latent state representations and output importance probability values ​​for each sub-action.

[0089] Referring to Figure 6, when the agent is close to the key but has not yet picked it up (t=2), the conditional structure inference network and Infer from here represents the importance of each sub-action. The reason for this reasoning is that the agent cannot open the door at present, so the sub-action Because there is no need to consider it. After that, the agent picks up the key and moves toward the door (t=5), and from this point on, the prediction of the conditional structure inference network begins to change. When the agent approaches the door (t=7) and opens the door (t=10), the conditional structure inference network and This suggests that the present invention effectively captures the combinatorial relationship between the current state and behavioral variables that change in different states and time steps.

[0090] By repeatedly training the conditional structure inference network and the process of this embodiment, the present invention creates an artificial intelligence model that outputs sub-actions with an importance lower than a preset value. The artificial intelligence model creates a masked action vector based on the importance output from the conditional structure inference network. For example, if the importance of the "pick up key" action is determined to be lower than the preset value, the value corresponding to the action is masked to 0, and the values ​​for the remaining actions are kept unchanged. The masked action vector is then received as input and the next state is predicted.

[0091] Figures 7 through 9 illustrate the results of the embodiments described with reference to Figures 4 through 6. Referring to Figure 7, MuZero did not significantly improve performance in a vast action space, while the embodiment applying the present invention achieved optimal performance in all environments. Referring to Figures 8 and 9, MuZero's performance deteriorated as the number of available actions increased, while the embodiment applying the present invention produced better results.

[0092] FIG. 10 is a flowchart illustrating the sequence of a method for creating an artificial intelligence model for determining sub-actions based on importance evaluation according to another embodiment of the present invention.

[0093] The method for generating an artificial intelligence model for determining a sub-action based on importance assessment, which will be described below (hereinafter referred to as the “AI model generating method”), can be performed by the AI ​​model generating device described above with reference to FIGS. 1 to 9. Therefore, the contents of the embodiments of the present invention described above with reference to FIGS. 1 to 9 can be equally applied to the embodiments to be described below, and any overlapping descriptions with the above descriptions will be omitted. The steps described below do not necessarily have to be performed in order, the order of the steps can be set in various ways, and the steps can be performed almost simultaneously.

[0094] Referring to FIG. 10, the method for creating an artificial intelligence model may include a step of selecting a main sub-behavior and creating a next state variable (S1100) and a step of creating an artificial intelligence model (S1200).

[0095] The main sub-action selection and next state variable generation step (S1100) is a step of selecting a main sub-action from among sub-actions corresponding to the current state variable of the inference problem based on the inference problem, and generating the next state variable based on the current state variable and the main sub-actions. The inference problem may include at least one of a specific situation and an image and text representing information about the specific situation.

[0096] The artificial intelligence model creation step (S1200) is a step of creating an artificial intelligence model that is trained to output a set of one or more major sub-behaviors for an input specific state variable by repeating the step of selecting major sub-behaviors and creating the next state variable (S1100) with the next state variable as a new current state variable, using specific inference problems, the current state variables for the inference problems, and the sub-behaviors corresponding to the current state variables for the specific inference problems as learning data.

[0097] Referring to FIG. 11, the main sub-action selection and next state variable generation step (S1100) may include a sub-action setting step (S1110), a main sub-action selection step according to importance (S1120), and a next state variable generation step (S1130).

[0098] The sub-action setting step (S1110) is a step for setting sub-actions corresponding to the current state variable. The main sub-action selection step (S1120) is a step for calculating the importance of sub-actions and selecting the main sub-action among the sub-actions based on the importance. The next state variable generation step (S1130) is a step for generating the next state variable predicted after the current state variable based on the current state variable and the main sub-actions.

[0099] Referring to Fig. 12, the step of selecting major sub-actions according to importance (S1120) may include an importance calculation step (S1121) and a step of selecting major sub-actions through masking processing (S1122).

[0100] The importance calculation step (S1121) is a step that calculates importance by learning the relationship between the current state variable and sub-actions using a preset deep neural network. The key sub-action selection step (S1122) is a step that selects key sub-actions by masking sub-actions that satisfy the preset criteria for importance.

[0101] Figure 13 is a flowchart illustrating the sequence of a method for setting a Monte Carlo tree search space using an artificial intelligence model according to another embodiment of the present invention.

[0102] The Monte Carlo tree search space setting method using an artificial intelligence model described below (hereinafter referred to as the “search space setting method”) can be performed by the search space setting device described above with reference to FIGS. 1 to 9. Therefore, the contents of the embodiments of the present invention described above with reference to FIGS. 1 to 9 can be equally applied to the embodiments to be described below, and any overlapping descriptions with the above descriptions will be omitted below. The steps described below do not necessarily have to be performed in order, the order of the steps can be set in various ways, and the steps can be performed almost simultaneously.

[0103] Referring to FIG. 13, the search space setting method includes a main sub-action selection and next node creation step (S2100) and a search space setting step (S2200).

[0104] The main sub-action selection and next node generation step (S2100) is a step of selecting main sub-actions from among sub-actions corresponding to the current node of the Monte Carlo tree node based on a preset Monte Carlo tree node, and generating the next node based on the current node and main sub-actions.

[0105] The search space setting step (S2200) is a step of setting the next node for a specific node as a search space by using an artificial intelligence model trained to output a set of one or more major sub-actions for a specific node by repeating the step of selecting major sub-actions and generating the next node (S2100) with the next node as a new current node. The artificial intelligence model may be trained using Monte Carlo tree nodes, the current node for the Monte Carlo tree nodes, and sub-actions corresponding to the current nodes for the Monte Carlo tree nodes as learning data.

[0106] Referring to FIG. 14, the main sub-action selection and next node creation step (S2100) may include a sub-action setting step (S2110), a main sub-action selection step according to importance (S2120), and a next node creation step (S2130).

[0107] The sub-action setting step (S2110) is a step for setting sub-actions corresponding to the current node. The main sub-action selection step (S2120) is a step for calculating the importance of sub-actions and selecting the main sub-action among the sub-actions based on the importance. The next node generation step (S2130) is a step for generating the next node predicted to follow the current node based on the current node and the main sub-actions.

[0108] Referring to Fig. 15, the step of selecting major sub-actions according to importance (S2120) may include an importance calculation step (S2121) and a step of selecting major sub-actions through masking processing (S2122).

[0109] The importance calculation step (S2121) uses a preset deep neural network to learn the relationship between the current node and sub-actions and calculate the importance. The key sub-action selection step (S2122) uses masking processing to select key sub-actions by masking sub-actions that satisfy preset criteria for importance.

[0110] The method for generating an artificial intelligence model for determining a subordinate action based on importance assessment of the present invention and the method for setting a Monte Carlo tree search space using the artificial intelligence model described so far can also be implemented in the form of a recording medium including computer-executable instructions, such as program modules executed by a computer. The computer-readable medium may be any available medium that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable medium may include a computer storage medium. The computer storage medium includes both volatile and nonvolatile, removable and non-removable media implemented by any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0111] Those skilled in the art will appreciate that the present invention can be easily modified into other specific forms without changing the technical spirit or essential characteristics of the present invention based on the above description. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The scope of the present invention is indicated by the following claims, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be interpreted as being included in the scope of the present invention. The scope of the present application is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be interpreted as being included in the scope of the present application.

[0112] The form for carrying out the invention is substantially the same as the best form for carrying out the invention.

[0113] The present invention can be used in a technique for reducing the search space of a Monte Carlo tree search algorithm, and thus has industrial applicability.

Claims

1. A method for generating an artificial intelligence model performed by at least one processor, a) A step of selecting main sub-actions from among sub-actions corresponding to the current state variables of the inference problem based on the inference problem, and generating the next state variables based on the current state variables and the main sub-actions; b) a step of generating an artificial intelligence model trained to output a set of one or more major sub-behaviors for an input specific state variable by repeating step a) with the next state variable as a new current state variable, A method for creating an artificial intelligence model for sub-action decision-making based on importance evaluation.

2. In paragraph 1, The above inference problem is, Containing at least one image and text representing a specific situation and information about said specific situation, A method for creating an artificial intelligence model for sub-action decision-making based on importance evaluation.

3. In paragraph 1, Step a) above, a-1) A step of setting sub-actions corresponding to the current state variables; a-2) A step of calculating the importance of the above sub-actions and selecting a main sub-action among the above sub-actions based on the importance; and a-3) A step of generating a next state variable predicted after the current state variable based on the current state variable and the main sub-behavior, A method for creating an artificial intelligence model for sub-action decision-making based on importance evaluation.

4. In paragraph 3, Step a-2) above, A step of calculating the importance by learning the relationship between the current state variable and the sub-actions using a preset deep neural network; and It includes a step of selecting the main sub-behaviors by masking the sub-behaviors that have an importance level that satisfies a preset standard among the above sub-behaviors. A method for creating an artificial intelligence model for sub-action decision-making based on importance evaluation.

5. In paragraph 1, The above artificial intelligence model is, It is trained using specific inference problems, current state variables for the specific inference problems, and sub-actions corresponding to the current state variables for the specific inference problems as learning data. A method for creating an artificial intelligence model for sub-action decision-making based on importance evaluation.

6. Communication module; at least one processor; and A memory electrically connected to the processor and storing at least one code to be executed by the processor, The above memory, when executed through the above processor, The above processor selects major sub-actions from among sub-actions corresponding to the current state variable of the inference problem based on the inference problem, generates a next state variable based on the current state variable and the major sub-actions, and selects major sub-actions from among sub-actions corresponding to the current state variable of the inference problem based on the inference problem with the next state variable as a new current state variable, and generates a next state variable based on the current state variable and the major sub-actions, thereby causing an artificial intelligence model trained to output a set of one or more major sub-actions for an input specific state variable to be generated. A device for generating an artificial intelligence model for sub-action decision-making based on importance evaluation.

7. In paragraph 6, The above inference problem is, Containing at least one image and text representing a specific situation and information about said specific situation, A device for generating an artificial intelligence model for sub-action decision-making based on importance evaluation.

8. In paragraph 6, The above memory is, The processor further stores a code that causes the processor to set sub-actions corresponding to the current state variable, calculate the importance of the sub-actions, select a main sub-action among the sub-actions based on the importance, and generate a next state variable predicted after the state variable based on the current state variable and the main sub-action. A device for generating an artificial intelligence model for sub-action decision-making based on importance evaluation.

9. In paragraph 8, The above memory is, When the processor calculates the importance of the sub-actions and selects a main sub-action among the sub-actions based on the importance, A code is further stored that learns the relationship between the current state variable and the sub-actions using a preset deep neural network to calculate the importance, and selects the main sub-actions by masking the sub-actions that have an importance that satisfies a preset criterion among the sub-actions. A device for generating an artificial intelligence model for sub-action decision-making based on importance evaluation.

10. In paragraph 6, The above artificial intelligence model is, It is trained using specific inference problems, current state variables for the specific inference problems, and sub-actions corresponding to the current state variables for the specific inference problems as learning data. A device for generating an artificial intelligence model for sub-action decision-making based on importance evaluation.

11. In a method for setting up a Monte Carlo tree search space using an artificial intelligence model, i) a step of selecting main sub-actions from among sub-actions corresponding to the current node of the Monte Carlo tree node based on a preset Monte Carlo tree node, and generating the next node based on the current node and the main sub-actions; and ii) a step of setting the next node for the specific node as a search space by using an artificial intelligence model trained to output a set of one or more major sub-actions for the specific node, by repeating step i) with the next node as a new current node. A method for setting up a Monte Carlo tree search space using an artificial intelligence model.

12. In paragraph 11, Step i) above, i-1) A step of setting sub-actions corresponding to the current node; i-2) A step of calculating the importance of the above sub-actions and selecting a main sub-action among the above sub-actions based on the importance; and i-3) A step of generating a next node predicted to follow the current node based on the current node and the main sub-action, A method for setting up a Monte Carlo tree search space using an artificial intelligence model.

13. In paragraph 12, The above step i-2) is, A step of calculating the importance by learning the relationship between the current node and the sub-behaviors using a preset deep neural network; and It includes a step of selecting the main sub-behaviors by masking the sub-behaviors that have an importance level that satisfies a preset standard among the above sub-behaviors. A method for setting up a Monte Carlo tree search space using an artificial intelligence model.

14. In paragraph 11, The above artificial intelligence model is, The Monte Carlo tree node, the current node for the Monte Carlo tree node, and the sub-actions corresponding to the current nodes for the Monte Carlo tree node are trained as learning data. A method for setting up a Monte Carlo tree search space using an artificial intelligence model.

15. Communication module; at least one processor; and A memory electrically connected to the processor and storing at least one code to be executed by the processor, The above memory, when executed through the above processor, The above processor selects major sub-actions from among sub-actions corresponding to the current node of the Monte Carlo tree node based on a preset Monte Carlo tree node, generates a next node based on the current node and the major sub-actions, and repeats the process of selecting major sub-actions from among sub-actions corresponding to the current node of the Monte Carlo tree node based on the preset Monte Carlo tree node and generating a next node based on the current node and the major sub-actions with the next node as a new current node, thereby storing a code that causes the next node for the specific node to be set as a search space using an artificial intelligence model trained to output a set of one or more major sub-actions for a specific node. A device for setting up a Monte Carlo tree search space using an artificial intelligence model.

16. In paragraph 15, The above memory is, The processor further stores a code that causes the processor to set sub-actions corresponding to the current node, calculate the importance of the sub-actions, select a main sub-action among the sub-actions based on the importance, and generate a next node predicted after the current node based on the current node and the main sub-action. A device for setting up a Monte Carlo tree search space using an artificial intelligence model.

17. In paragraph 16, The above memory is, When the processor calculates the importance of the sub-actions and selects a main sub-action among the sub-actions based on the importance, A code is further stored that learns the relationship between the current node and the sub-actions using a preset deep neural network to calculate the importance, and selects the main sub-actions by masking sub-actions that have an importance that satisfies a preset standard among the sub-actions. A device for setting up a Monte Carlo tree search space using an artificial intelligence model.

18. In paragraph 16, The above artificial intelligence model is, The Monte Carlo tree node, the current node for the Monte Carlo tree node, and the sub-actions corresponding to the current nodes for the Monte Carlo tree node are trained as learning data. A device for setting up a Monte Carlo tree search space using an artificial intelligence model.

Images