Apparatus and Method for designing a transformative structure behavior tree based on reinforcement learning

Abstract

The present invention relates to a reinforcement learning-based dynamic behavior tree design apparatus and method. A reinforcement learning-based dynamic behavior tree design device and method according to one embodiment of the present invention comprises: a data collection unit that collects data on the result of a robot's task performance and environment data; a reinforcement learning unit that calculates the task sequence of the robot by applying a reinforcement learning algorithm to the task performance result data and environment data; a behavior tree rearrangement unit that rearranges the structure of the behavior tree based on the calculated task sequence; and a robot control unit that controls the task of the robot based on the rearranged behavior tree.

Description

Technology Field

[0001] The present invention relates to a reinforcement learning-based transformable structure behavior tree design device and a method of operation thereof. More specifically, the invention relates to a reinforcement learning-based transformable structure behavior tree design device and a method of operation thereof that can improve work efficiency and accuracy by learning a work sequence using robot work performance result data and environment data as inputs, and dynamically reconstructing the structure of a behavior tree based thereon. Background Technology

[0002] For a robot to perform tasks autonomously in a given environment, technology capable of efficiently managing task sequences and execution plans is essential.

[0003] To this end, the Behavior Tree (BT), used as a conventional task planning technique, has been widely used as a methodology for hierarchically designing and managing tasks in robots or AI systems.

[0004] Behavior trees have structural advantages such as controlling the flow of work and performing alternative tasks in the event of task failure, but they had the following limitations.

[0005] First of all, the static structure of the behavior tree makes it difficult to respond flexibly to environmental changes or robot task failures.

[0006] For example, action trees have limitations in establishing an efficient optimal path because the order of operations is fixed.

[0007] In addition, work efficiency may be reduced because, even in the event of a robot task failure, a fixed alternative path is executed instead of the optimal path derived from analyzing the cause of the failure.

[0008] To complement this, approaches utilizing Reinforcement Learning (RL) are gaining attention. Reinforcement Learning is a technology in which an agent interacts with the environment and learns an optimal behavior policy based on rewards.

[0009] Reinforcement learning has shown high performance, particularly in the field of autonomous robots, but it has the following problems.

[0010] First, reinforcement learning cannot structurally manage the order of tasks, and training time becomes excessively long when the state space is large or complex.

[0011] In addition, there was a problem with the lack of a system to reflect learning results in work plans in real time.

[0012] In response to this, there have been attempts to realize autonomous robot tasks by combining action trees and reinforcement learning; however, since the output values ​​of reinforcement learning were limited to merely using auxiliary information, there were limitations in dynamically transforming the action tree structure itself or optimizing the path in real time in the event of task failure. Prior art literature

[0013] Republic of Korea Registered Patent Publication No. 10-2734051 The problem to be solved

[0014] The present invention was devised to solve the problems described above, and the objective of the present invention is to provide a reinforcement learning-based dynamic action tree design apparatus and method that enables dynamic reconstruction of the action tree and real-time path optimization in the event of task failure, and further improves task efficiency and environmental adaptability by reflecting the output value (TaskID array) of the reinforcement learning algorithm into the action tree.

[0015] The objectives of the present invention are not limited to the problems mentioned above, and other objectives and advantages of the present invention not mentioned may be understood from the following description and will be more clearly understood by the embodiments of the present invention. Furthermore, it will be understood that the objectives and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims. means of solving the problem

[0016] For the above purpose, a reinforcement learning-based transformable structure behavior tree design device according to one embodiment of the present invention comprises: a data collection unit that collects data on the result of a robot's task performance and environment data; a reinforcement learning unit that calculates the task sequence of the robot by applying a reinforcement learning algorithm to the task performance result data and environment data; a behavior tree rearrangement unit that rearranges the structure of the behavior tree based on the calculated task sequence; and a robot control unit that controls the task of the robot based on the rearranged behavior tree.

[0017] And, a reinforcement learning-based transformable structure behavior tree design method according to one embodiment of the present invention comprises: collecting data on the result of a robot's task performance and environmental data; applying a reinforcement learning algorithm to the data on the result of the task performance and environmental data to calculate the task sequence of the robot; rearranging the structure of the behavior tree based on the calculated task sequence; and controlling the task of the robot based on the rearranged behavior tree.

[0018] And, according to one embodiment of the present invention, a reinforcement learning-based dynamic action tree design device comprises: a memory; and at least one processor, wherein the at least one processor collects data on the result of a task performed by a robot and environment data, applies a reinforcement learning algorithm to the data on the result of a task performed by the robot and the environment data to calculate the task sequence of the robot, rearranges the structure of an action tree based on the calculated task sequence, and controls the task of the robot based on the rearranged action tree. Effects of the invention

[0019] According to the present invention, the dynamically reconfigurable behavior tree structure has the effect of enabling real-time response to task failures or environmental changes.

[0020] In addition, according to the present invention, by reflecting the output value (TaskID array) of a reinforcement learning algorithm in a behavior tree to optimize the task order, it has the effect of minimizing task time and resource consumption.

[0021] Accordingly, according to the present invention, it has the effect of providing high flexibility and reliability in various application fields such as autonomous robots, logistics optimization, and game AI, where environmental adaptability and operational stability are required. Brief explanation of the drawing

[0022] FIG. 1 is a configuration diagram of a reinforcement learning-based transformable structure behavior tree design device according to one embodiment of the present invention. FIG. 2 is a flowchart of a reinforcement learning-based transformable structure behavior tree design method according to one embodiment of the present invention. FIG. 3 is a diagram showing the structure of a behavior tree (BT) according to one embodiment of the present invention. FIG. 4 is a diagram briefly showing the overall structure of an RLTS-BT according to one embodiment of the present invention. FIG. 5 is a diagram showing the RL-NODE subtree structure of RLTS-BT according to one embodiment of the present invention. FIG. 6 is a diagram showing the detailed structure of an RLTS-BT according to one embodiment of the present invention. FIG. 7 is a diagram visually showing the work sequence and path for a patrol mission of a robot according to one embodiment of the present invention. FIG. 8 is a schematic diagram showing a work planning framework configured based on an RLTS-BT structure to perform a patrol mission simulation of a robot according to one embodiment of the present invention. Specific details for implementing the invention

[0023] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols are assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the invention.

[0024] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0025] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0027] In one embodiment of the present invention, a technology is described that combines a reinforcement learning algorithm with a behavior tree (BT) to dynamically optimize the order of operations and adapt to changes in the environment.

[0028] For robots to perform tasks autonomously, a structured method for planning and executing task sequences is required, and for this purpose, hierarchical task planning techniques such as behavior trees are widely used.

[0029] More specifically, a behavior tree manages the flow of work through a tree structure of nodes designed hierarchically to perform tasks, where each node defines specific conditions and actions.

[0030] Through the interaction between these components, the behavior tree operates in such a way that the parent node evaluates the status returned by each node to determine whether to execute the next task.

[0031] As such, behavior trees have the advantage of visually representing workflows and efficiently managing conditions and sequences between tasks.

[0032] However, existing behavior trees have limitations in that they cannot respond flexibly to environmental changes or task failures because their task sequence and tree structure are fixed.

[0033] For example, when an autonomous robot encounters unexpected obstacles or fails a task in a complex environment, existing behavior trees suffer from significantly reduced efficiency because they repeatedly attempt the same path and task sequence.

[0034] Meanwhile, reinforcement learning is a technology in which an agent interacts with an environment to learn an optimal behavior policy based on rewards, and it demonstrates high performance, particularly in areas such as game AI, robot control, and logistics optimization.

[0035] In other words, unlike supervised learning, reinforcement learning can be utilized to solve various problems because the agent learns on its own without explicit correct answer data.

[0036] However, reinforcement learning cannot structurally manage the sequence of tasks and has limitations in adapting to environmental changes in real time. Furthermore, the learning process is time-consuming, and there is a lack of a system to directly incorporate learning results into work plans.

[0037] To solve these problems, the present invention proposes RLTS-BT (Reinforcement Learning-Based Transformative Structure Behavior Tree), an algorithm with a structure that can dynamically reconstruct the order of operations by including reinforcement learning nodes (RL nodes) in the sub-nodes of the behavior tree.

[0038] Furthermore, we propose a method to link the output of the reinforcement learning algorithm (TaskID array) with a behavior tree to set an alternative task sequence in the event of task failure and to adapt to environmental changes in real time.

[0039] In particular, in the present invention, learning can be iteratively improved by feeding back task performance result data to the reinforcement learning algorithm through a data exchange interface between the reinforcement learning algorithm and the behavior tree.

[0040] This allows for the minimization of working time and resource consumption, and provides high flexibility and reliability in various application environments.

[0041] Accordingly, in one embodiment of the present invention, a reinforcement learning-based transformable structural behavior tree design device and method combining a behavior tree and a reinforcement learning algorithm are proposed to significantly improve the optimization of the work sequence and environmental adaptability.

[0042] In this regard, FIG. 1 is a configuration diagram of a reinforcement learning-based transformable structure behavior tree design device according to one embodiment of the present invention, and FIG. 2 is a flowchart of a reinforcement learning-based transformable structure behavior tree design method according to one embodiment of the present invention.

[0043] Referring to FIG. 1, a reinforcement learning-based transformable structure behavior tree design device (100) may include a data collection unit (110), a reinforcement learning unit (120), a behavior tree rearrangement unit (130), and a robot control unit (140).

[0044] In an embodiment, the reinforcement learning-based transformable structure behavior tree design device (100) may be a device that generates a work sequence for a robot to perform a task, controls the task by dynamically reconstructing the structure of the behavior tree, and supports the improvement of learning efficiency and adaptation to environmental changes by reflecting the results of the task performance in reinforcement learning, but is not limited thereto.

[0045] The data collection unit (110) can collect data on the robot's work performance results and environmental data in real time and provide input data required for the reinforcement learning unit (120) (see step S210).

[0046] More specifically, the data collection unit (110) collects and processes the following data.

[0047] The task execution result data includes at least one of the task completion time and task execution accuracy.

[0048] For example, the task completion time may be information indicating the time taken for a robot to complete a specific task.

[0049] Furthermore, task performance accuracy can be information representing a numerical value evaluating how well a task meets required criteria. For example, task performance accuracy can be calculated based on the difference between the target position and the actual position in a task of moving an object to a specific location, or based on the deviation between the actual path and the ideal path in a task of moving along a defined path.

[0050] Environmental data can refer to dynamic information in the environment where a robot performs a task. For example, environmental data can refer to the location of surrounding obstacles (e.g., Lidar or camera sensor data) or the robot's movement path within the environment (e.g., sensor network data).

[0051] The data collection unit (110) can collect data from the results of these tasks and environmental data in real time through sensors.

[0052] In addition, the data collection unit (110) can perform the role of supporting iterative improvement of learning by feeding back data of the robot's task performance results based on the rearranged behavior tree described later to the reinforcement learning unit (120).

[0053] The reinforcement learning unit (120) applies a reinforcement learning algorithm to the task execution result data and environment data to calculate the task sequence of the robot (see step S220).

[0054] For example, the aforementioned reinforcement learning algorithm takes task execution result data and environment data as input and is an algorithm trained to produce a robot task sequence that satisfies at least one of minimum task time and high task execution accuracy.

[0055] In addition, the reinforcement learning algorithm of the present invention may be a reinforcement learning algorithm based on the Proximal Policy Optimization (PPO) algorithm.

[0056] In addition, the reinforcement learning unit (120) can produce a TaskID array as a result learned through a reinforcement learning algorithm.

[0057] Here, the TaskID array represents the robot's task order and can be calculated to enable the robot to visit VISITPOINTS (task points) in an optimal order. Additionally, the TaskID array can be used as a criterion for determining the task execution order in the behavior tree.

[0058] For example, the TaskID array [3, 5, 1, 2, 4] may mean that tasks are performed in the order VISITPOINT-3 → VISITPOINT-5 → VISITPOINT-1 → VISITPOINT-2 → VISITPOINT-4.

[0059] In addition, the TaskID array can be optimized through the reward function of the reinforcement learning algorithm.

[0060] For example, the reward function can adjust the TaskID array based on one or more of minimum task time and high task performance accuracy.

[0061] The behavior tree rearrangement unit (130) rearranges the structure of the behavior tree based on the calculated work order (see step S230).

[0062] For example, the behavior tree of the behavior tree rearrangement unit (130) is a reinforcement learning-based behavior tree, and by integrating a reinforcement learning algorithm into the behavior tree structure, it can adapt in real time to environmental changes and task failures.

[0063] Reinforcement learning-based behavior trees control the sequence and execution flow of tasks through hierarchically organized nodes, and can provide an optimized task plan through components such as Reactive Fallback nodes, Reinforcement Learning (RL) nodes, and Task-Sequence nodes.

[0064] The robot control unit (140) controls the robot's work based on the rearranged action tree (see step S240).

[0065] And, the robot control unit (140) executes the robot's movements and tasks according to the task sequence (TaskID array) provided in the rearranged action tree within the RLTS-BT structure.

[0066] At this time, the data collection unit (110) continuously collects data on the results of the robot's work performance based on the rearranged action tree and transmits the collected data on the results of the robot's work performance based on the rearranged action tree to the reinforcement learning unit (130).

[0067] For example, the data collection unit (110) can transmit data on the result of a robot's task performance based on a rearranged action tree to the reinforcement learning unit (120) based on a ROS2 message interface.

[0068] More specifically, the ROS2 message can be composed of data such as "Task completion time: 15 seconds, obstacle avoidance success: True, task accuracy: 95%".

[0069] And, the reinforcement learning unit (120) receives real-time data from the ROS2 message, calculates a reward value, and can rearrange the order of operations based on the calculated reward value.

[0070] At this time, the reinforcement learning unit (120) can analyze the robot's task performance result data and environment data based on the rearranged action tree, evaluate information such as the reduction of work time, the success rate of obstacle avoidance, and the accuracy of the task, and determine a reward value.

[0071] For example, if the path is blocked by an obstacle in a work plan where a robot visits a specific VISITPOINT, the reinforcement learning unit (120) can set a low reward value through the received data and rearrange the work order to bypass the VISITPOINT or specify an alternative path.

[0072] In this process, the reinforcement learning unit (120) can generate a new TaskID array by utilizing data including obstacle status, distance traveled, and unfinished tasks through ROS2 messages.

[0073] And, the behavior tree rearrangement unit (130) can rearrange the behavior tree within the RLTS-BT structure based on a new TaskID array.

[0074] This behavior tree rearrangement process combines the policy learning results of reinforcement learning algorithms with ROS2 message data to adapt to environmental changes in real time and maximize operational efficiency.

[0075] Above, a reinforcement learning-based transformable structure behavior tree design apparatus and method capable of dynamically rearranging the order of operations and adapting to environmental changes in real time based on the structure of an RLTS-BT (reinforcement learning-based transformable structure behavior tree) according to one embodiment of the present invention have been described in detail. Below, a more specific explanation will be provided regarding the configuration of the RLTS-BT and the role of each component to realize this.

[0076] For reference, the following content relates to a case in which RLTS-BT (Reinforcement Learning-based Transformable Structure Behavior Tree) according to an embodiment of the present invention is applied in the work planning and execution process for a robot's patrol mission.

[0077] Specifically, the patrol mission consists of a process in which the robot sequentially moves to designated points (Visitpoints), overcomes or bypasses obstacles, completes tasks at each point, and returns to the starting point.

[0078] In this regard, FIGS. 3 to 6 are drawings showing the structure of an RLTS-BT (Learning-Based Modification Structure Behavior Tree) according to an embodiment of the present invention.

[0079] For example, FIG. 3 is a diagram showing the structure of a behavior tree (BT) according to one embodiment of the present invention, FIG. 4 is a diagram showing the overall structure of an RLTS-BT according to one embodiment of the present invention, FIG. 5 is a diagram showing the RL-NODE subtree structure of an RLTS-BT according to one embodiment of the present invention, and FIG. 6 is a diagram showing the detailed structure of an RLTS-BT according to one embodiment of the present invention.

[0080] And, FIG. 7 is a diagram visually showing the work sequence and path for a patrol mission of a robot according to one embodiment of the present invention.

[0081] Referring to FIGS. 3 to 7, for a patrol mission of a robot with enhanced environmental adaptability according to one embodiment of the present invention, a reinforcement learning (RL) node can be integrated into a subtree of an existing behavior tree (BT; Behavior Tree).

[0082] RLTS-BT is hierarchically organized around the ROOT node, and its main components include the MAINLOOP node, RECEPTION-CHECK node, TASK-SEQUENCE node, RL-NODE subtree, and BACK2ORIGIN node.

[0083] First, ROOT, the top node of RLTS-BT, is responsible for starting the execution of the entire tree and can call MAINLOOP nodes that control the order of operations.

[0084] MAINLOOP is a Sequence node that adopts a sequential execution method and executes child nodes in a defined order, and can manage the main execution flow of RLTS-BT. For example, it proceeds to the next node only if the current node succeeds, and stops execution if even one of the child nodes fails.

[0085] At this time, the MAINLOOP of RLTS-BT was designed to re-execute all nodes sequentially from the leftmost node, regardless of the previous success of the child nodes, when a Tick signal was given in the next iteration.

[0086] On the other hand, the MAINLOOP of the RL-NODE operates as a Sequence Star (Sequence*) node, so that in the next iteration, the previously successful node is not re-executed, and execution starts again from the failed node.

[0087] In this way, even though they are the same MAINLOOP, the operation methods of the Sequence node and the Sequence* node are differentiated to manage the re-execution of failed tasks more effectively.

[0088] The RECEPTION-CHECK-1 node is a Reactive Fallback node that receives an array of TaskIDs, which are the outputs of a reinforcement learning algorithm, from a ROS2 message, verifies their validity, and passes a valid array of TaskIDs to the SAVE-TOPIC-VALUE node to save the array.

[0089] The TASK-SEQUENCE node can control the execution order of the child node (RL-NODE) ​​subtree based on the TaskID array.

[0090] Here, each element of the TaskID array determines the execution order of the RL-NODE subtree, which allows VISITPOINT nodes to be executed dynamically.

[0091] For example, if the TaskID array is [3, 5, 1, 2, 4], RL-NODE-1, RL-NODE-2, RL-NODE-3, RL-NODE-4, and RL-NODE-5 are executed sequentially, and each RL-NODE checks the corresponding element of the TaskID array and determines the order of operations.

[0092] More specifically, the RL-NODE subtree is a subtree that checks each element of the TaskID array and executes a task accordingly. Its child nodes may include a TASK-CHECK node, a VISITPOINT node, etc.

[0093] First, the TASK-CHECK node is a node that checks a specific value in the TaskID array and can include an ID-CHECK node, which is a Condition node. Then, the ID-CHECK node can return SUCCESS or FAILURE by checking whether the current element value of the TaskID array matches an expected value.

[0094] Here, if the element value of the TaskID array matches the expected value, SUCCESS is returned, and the connected VISITPOINT node can be executed.

[0095] Alternatively, if the element value of the TaskID array does not match the expected value, FAILURE is returned and the next TASK-CHECK node can be executed.

[0096] Also, the VISITPOINT node is a node that controls the robot to visit a designated work point by overcoming or bypassing obstacles.

[0097] To elaborate on the operation of the aforementioned RL-NODE, when RL-NODE-1 is running and the first value of the TaskID array is 3, the ID-CHECK node of the TASK-CHECK-1 node first checks whether the first value (3) of the TaskID array matches 1, and returns FAILURE because it does not match.

[0098] Likewise, the TASK-CHECK-2 node also returns FAILURE because the first value (3) of the TaskID array does not match 2.

[0099] The TASK-CHECK-3 node returns SUCCESS because the ID-CHECK node matches the first value (3) of the TaskID array with 3.

[0100] Then, upon the SUCCESS return of TASK-CHECK-3, the VISITPOINT-3 node is executed, and subsequently, the execution of RL-NODE-1 is terminated.

[0101] In this way, the TASK-SEQUENCE node sequentially executes the RL-NODE subtree according to the order of the TaskID array, and each RL-NODE can be executed by checking the corresponding element of the TaskID array.

[0102] For example, if the TaskID array is [3, 5, 1, 2, 4], VISITPOINT-3 from RL-NODE-1, VISITPOINT-5 from RL-NODE-2, VISITPOINT-1 from RL-NODE-3, etc. can be executed sequentially.

[0103] Consequently, tasks can be dynamically controlled in the order of VISITPOINT-3 → VISITPOINT-5 → VISITPOINT-1 → VISITPOINT-2 → VISITPOINT-4 depending on the value of the TaskID array.

[0104] The PUBLISH-TOPIC-VALUE node can write the total execution time of the tasks in ROS2 message format after all tasks are completed. This data is passed to the reinforcement learning unit and used as a reward value, contributing to learning that improves task efficiency.

[0105] The BACK2ORIGIN node is a node that commands the robot to return to the origin after the last task is completed.

[0106] In this way, RLTS-BT according to one embodiment of the present invention can significantly improve environmental adaptability and work efficiency in robot patrol tasks by integrating a reinforcement learning (RL) algorithm into an existing behavior tree (BT) structure.

[0107] In particular, it overcomes the limitations of the existing fixed task order by maintaining the static structure of the behavior tree while dynamically rearranging the task order based on the TaskID array and providing a structure that can flexibly respond even in the event of task failure.

[0108] In addition, ROS2 messages are received in real time through the RECEPTION-CHECK-1 node, and the task order is adjusted by reflecting them in the TASK-SEQUENCE and RL-NODE subtrees, and tasks at each point can be performed reliably through the VISITPOINT node.

[0109] Furthermore, the learning efficiency of the reinforcement learning unit was enhanced by transmitting task execution result data as ROS2 messages through the PUBLISH-TOPIC-VALUE node, and the patrol mission was optimized by completing the process of returning to the origin through the BACK2ORIGIN node.

[0110] Accordingly, RLTS-BT according to one embodiment of the present invention can effectively combine a reinforcement learning algorithm and a behavior tree structure to maximize the efficiency of a robot's task performance, and can be applied to various application environments of autonomous robots, and can have high practicality in path search and task performance, especially in complex environments.

[0111] And, FIG. 8 is a schematic diagram showing a work planning framework configured based on an RLTS-BT structure to perform a patrol mission simulation of a robot according to one embodiment of the present invention.

[0112] Referring to Fig. 8, the simulation module simulates the robot's movements and tasks by virtually modeling the real environment.

[0113] For example, the simulation environment includes various obstacles (e.g., ramps, sandy terrain, stairs, etc.), and the robot performs tasks by overcoming or bypassing them.

[0114] And, the simulation module generates data such as the task completion time ( / sim_time) and passes it to the RL Module.

[0115] The Navigation Module is a module that performs basic path search and control, and controls the robot to move to a designated point within the environment and perform tasks according to the work commands of RLTS-BT.

[0116] The Behavior Tree Module (BT Module) executes the RLTS-BT (Reinforcement Learning-based Transformable Structure Behavior Tree) structure, rearranges the task order based on the TaskID array, and transmits task commands to the Navigation Module. It also generates task result data as ROS2 messages and feeds them back to the Reinforcement Learning Module (RL Module).

[0117] The reinforcement learning module (RL Module) executes a reinforcement learning algorithm to generate a TaskID array and passes it to the BT Module. Then, it continuously optimizes the learning process by utilizing the task result data as a reward value.

[0118] For example, the reward value can be expressed as a function of [Equation 1] based on the minimum work time.

[0119] [Formula 1]

[0120]

[0121] (Here, R is the compensation function, t is the elapsed time, and i is each work point (VISITPOINT))

[0122] Through this, the results of comparing the RLTS-BT structure and the existing patrol operation method based on the simulation results are as follows.

[0123] The existing method took 23.676 minutes for the robot to visit all VISITPOINTS sequentially in a counterclockwise direction and return to the origin.

[0124] However, the RLTS-BT method learned the TaskID array to generate an optimal path, and as a result, completed the patrol mission in 14.584 minutes.

[0125] As a result, the RLTS-BT method according to one embodiment of the present invention achieved a time reduction of approximately 38.401% compared to the conventional method and provided a time efficiency approximately 1.623 times higher.

[0127] Meanwhile, the functional operations and embodiments of the subject described herein may be implemented as digital electronic circuits, or as computer software, firmware, or hardware comprising the structures and structural equivalents disclosed herein, or as a combination of one or more of these. The embodiments of the subject described herein may be implemented as one or more computer program products, that is, as one or more modules relating to computer program instructions encoded on a tangible program storage medium for processing the operation of a processing system or for execution by means of such processing.

[0128] A computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, or a combination of one or more of these.

[0129] In this specification, the terms “system” or “device” encompass all apparatuses, devices, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A processing system may, in addition to hardware, include code that forms an execution environment for a computer program upon request, such as, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0130] A computer program (also known as a program, software, software application, script, or code) may be written in any form of a programming language, including compiled or interpreted languages, or a priori or procedural languages, and may be deployed in any form, including a standalone program, a module, a component, a subroutine, or other unit suitable for use in a computer environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a single file provided to the requested program, in multiple interacting files (e.g., a file storing one or more modules, subprograms, or parts of code), or in a part of a file containing other programs or data (e.g., one or more scripts stored within a markup language document). A computer program may be deployed to be executed on multiple computers or a single computer located at a single site or distributed across multiple sites and interconnected by a communication network.

[0131] Meanwhile, computer-readable media suitable for storing computer program instructions and data may include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices like EPROM, EEPROM, and flash memory devices, magnetic disks like internal hard disks or external disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or integrated with special-purpose logic circuits.

[0132] In the specification of this disclosure (particularly in the claims), the use of the term "above" and similar descriptive terms may be in both singular and plural. Furthermore, where a range is described in this disclosure, it is to include an invention to which individual values ​​belonging to said range are applied (unless otherwise stated), as is equivalent to describing each individual value constituting said range in the detailed description of the invention.

[0133] Unless explicitly stated otherwise, the steps constituting the method according to the present disclosure may be performed in a suitable order. The present disclosure is not necessarily limited by the order in which the steps are described. The use of all examples or exemplary terms (e.g., etc.) in the present disclosure is merely for the purpose of describing the present disclosure in detail, and the scope of the present disclosure is not limited by such examples or exemplary terms unless limited by the claims. Furthermore, a person skilled in the art will understand that various modifications, combinations, and changes may be made according to design conditions and factors within the scope of the claims or equivalents to which they are added.

[0134] Accordingly, the scope of the present disclosure should not be limited to the embodiments described above, and all scopes equivalent to or equivalently modified from the claims set forth below, as well as the claims set forth below, shall be considered to fall within the scope of the scope of the present disclosure.

Claims

Claim 1 A reinforcement learning-based transformable structure behavior tree design device comprising: a data collection unit for collecting robot task performance result data and environment data; a reinforcement learning unit for calculating the task sequence of the robot by applying a reinforcement learning algorithm to the task performance result data and the environment data; a behavior tree rearrangement unit for rearranging the structure of the behavior tree based on the calculated task sequence; and a robot control unit for controlling the robot's task based on the rearranged behavior tree. Claim 2 In claim 1, the behavior tree rearrangement unit rearranges the structure of the behavior tree based on the array of TaskIDs of the calculated task sequence, a reinforcement learning-based dynamic behavior tree design device. Claim 3 In paragraph 2, the behavior tree rearrangement unit rearranges the node order and subtree structure of the behavior tree according to the TaskID array, and the subtree structure is composed of a Reactive Fallback node and a Sequence node, a reinforcement learning-based dynamic behavior tree design device. Claim 4 A reinforcement learning-based transformable structural behavior tree design device according to claim 1, wherein the data collection unit transmits data of a robot's task performance result based on the rearranged behavior tree to the reinforcement learning unit. Claim 5 In claim 4, the data collection unit transmits data on the result of a robot's task performance based on the rearranged action tree to the reinforcement learning unit based on a ROS2 message interface, a reinforcement learning-based dynamic action tree design device. Claim 6 In claim 5, the reinforcement learning unit calculates a reward value based on data of the robot's task performance results based on the rearranged action tree, and updates the robot's task sequence based on the calculated reward value, a reinforcement learning-based dynamic action tree design device. Claim 7 A reinforcement learning-based dynamic behavior tree design device according to claim 1, wherein the reinforcement learning algorithm is an algorithm learned to take work performance result data and environment data as inputs and to produce a robot work sequence satisfying at least one of minimum work time and high work performance accuracy. Claim 8 A reinforcement learning-based dynamic behavior tree design device according to claim 1, wherein the task performance result data includes at least one of task completion time and task performance accuracy. Claim 9 A reinforcement learning-based transformable structure behavior tree design method comprising: collecting data on the result of a robot's task performance and environment data; applying a reinforcement learning algorithm to the data on the result of the task performance and environment data to calculate the task sequence of the robot; rearranging the structure of the behavior tree based on the calculated task sequence; and controlling the robot's task based on the rearranged behavior tree. Claim 10 In claim 9, the step of rearranging the structure of the behavior tree comprises a process of rearranging the structure of the behavior tree based on the array of TaskIDs of the calculated task sequence, a reinforcement learning-based dynamic behavior tree design method. Claim 11 A reinforcement learning-based dynamic behavior tree design method, wherein in claim 10, the step of rearranging the structure of the behavior tree further includes the process of rearranging the node order and subtree structure of the behavior tree according to the TaskID array, and the subtree structure is composed of Reactive Fallback nodes and Sequence nodes. Claim 12 In claim 9, the step of collecting the data includes the process of transmitting the robot's task performance result data based on the rearranged behavior tree to the reinforcement learning unit, a reinforcement learning-based transformable structural behavior tree design method. Claim 13 A reinforcement learning-based dynamic behavior tree design method, wherein the process of transmitting to the reinforcement learning unit further includes the process of transmitting robot task performance result data based on the rearranged behavior tree to the reinforcement learning unit based on a ROS2 message interface. Claim 14 In claim 13, the step of calculating the task sequence of the robot further comprises: a process of calculating a reward value based on task performance result data of the robot based on the rearranged task tree; and a process of updating the task sequence of the robot based on the calculated reward value, a reinforcement learning-based dynamic action tree design step. Claim 15 In claim 9, the reinforcement learning algorithm is an algorithm learned to take work execution result data and environment data as inputs and to produce a robot work sequence satisfying at least one of minimum work time and high work execution accuracy, a reinforcement learning-based dynamic action tree design device. Claim 16 In claim 9, the above-mentioned task performance result data comprises at least one of task completion time and task performance accuracy, a reinforcement learning-based dynamic behavior tree design device. Claim 17 A reinforcement learning-based dynamic action tree design device comprising: a memory; and at least one processor, wherein the at least one processor collects data on the result of a task performed by a robot and environment data, applies a reinforcement learning algorithm to the data on the result of a task performed by the robot and the environment data to calculate a task sequence of the robot, rearranges the structure of a action tree based on the calculated task sequence, and controls the task of the robot based on the rearranged action tree.

Images