Method and apparatus for training palletizing robot based on reinforcement learning using attention mechanism of vision transformer

Abstract

A robot training apparatus according to an embodiment may perform the operations of: acquiring state information including a state of a pallet and the size of an object to be loaded onto the pallet; converting the state information into a patch of a predefined size in order to input same into a vision transformer model; on the basis of an actor network of a reinforcement learning model including a policy function for determining a loading position of the object in the direction that maximizes an expected value of the final loading rate of the pallet, determining, from the patch, a position to load the object onto the pallet; on the basis of a critic network of the reinforcement learning model including a value function for deriving the expected value according to the determination, deriving an expected value according to the determination; and updating parameters of the policy function and the value function in the direction that minimizes a loss of a loss function calculated on the basis of the determination and the expected value.

Description

A method and device for learning a palletizing robot based on reinforcement learning utilizing the attention mechanism of a vision transformer.

[0001] The present invention relates to a technology for teaching a palletizing robot to efficiently load objects of various sizes onto a pallet based on a reinforcement learning model utilizing the attention mechanism of a vision transformer.

[0002] This application claims priority to Korean Patent Application No. 10-2024-0078481, filed on June 17, 2024, and Korean Patent Application No. 10-2024-0115483, filed on August 28, 2024, the entire contents of which are disclosed in the specification and drawings of the above applications are incorporated herein by reference.

[0003] Meanwhile, the present invention was supported by the following national research and development project.

[0004] Assignment ID: 171193129

[0005] Assignment Number: 22019-0-01190-005

[0006] Ministry of Science and ICT

[0007] Project Management Agency Name: Information and Communications Technology Planning and Evaluation Institute

[0008] Research Project Name: SW Computing Industry Core Technology Development

[0009] Research Project Name: (SW Star Lab) Robot Learning: Efficient, Safe, and Socially Friendly Machine Learning

[0010] Project implementation organization name: Seoul National University Industry-Academic Cooperation Foundation

[0011] Research period: January 1, 2024 - December 31, 2024

[0012] Assignment ID: 1711193420

[0013] Assignment Number: 2022-0-00480-002

[0014] Ministry of Science and ICT

[0015] Project Management Agency Name: Information and Communications Technology Planning and Evaluation Institute

[0016] Research Project Name: Development of Core Source Technologies for Human-Centered Artificial Intelligence

[0017] Research Project Title: Development of Goal-Oriented Artificial Intelligence Generation and Inference Technology

[0018] Project implementation organization name: Seoul National University Industry-Academic Cooperation Foundation

[0019] Research period: January 1, 2024 - December 31, 2024

[0020] The problem of stably loading rectangular objects of various sizes within a confined three-dimensional space is commonly known as the "3D bin packing" problem. 3D bin packing is a critical challenge in various industries, including logistics, manufacturing, and distribution.

[0021] Existing techniques for solving the 3D bin packing problem have primarily used heuristic algorithms or rule-based approaches. While these methods can provide rapid solutions to given problems under specific conditions, their performance deteriorates rapidly as the conditions or problem complexity increase.

[0022] For example, when given pallets with different conditions and objects of varying sizes, existing technologies struggle to adaptively predict the loading order and location, making efficient loading difficult. Thus, existing rule-based approaches face limitations in their ability to flexibly adapt to new conditions or variables.

[0023] To address the issues with these existing technologies, approaches utilizing reinforcement learning have recently gained attention. Reinforcement learning is a machine learning method in which an agent learns optimal behavior by interacting with its environment. It can be effectively applied to complex optimization problems such as the 3D bin packing problem.

[0024] However, existing reinforcement learning algorithms have limited performance in solving real-time loading problems because they cannot effectively handle the complexity of state and action spaces.

[0025] In particular, in real-world situations where objects of different sizes must be loaded onto pallets of different sizes in real time, there is a problem that the performance of existing reinforcement learning algorithms is significantly reduced.

[0026] The present invention aims to overcome the limitations of existing reinforcement learning methods by introducing a vision transformer model into the network of a reinforcement learning model so that the loading state of objects, which is generally expressed as a two-dimensional grid map in a 3D bin packing problem, can be effectively applied to a reinforcement learning algorithm.

[0027] Specifically, the present invention aims to improve the accuracy of state representation in reinforcement learning by patching the shape of a pallet and the state of loading objects using a vision transformer model, since existing heuristic algorithms or rule-based approaches have a problem in that they cannot properly handle the complexity of the state space, resulting in low loading efficiency.

[0028] Furthermore, the present invention aims to develop a reinforcement learning algorithm that can flexibly handle pallets of various sizes. While existing reinforcement learning algorithms can demonstrate a certain level of performance for fixed-size pallets, they suffer from a significant reduction in loading efficiency for pallets of varying sizes. The present invention proposes a technique that gradually expands the receptive area of ​​the attention mechanism with each iteration, thereby maintaining high loading efficiency for pallets of various sizes.

[0029] Furthermore, the present invention seeks to improve the performance of reinforcement learning through data augmentation techniques. The present invention utilizes a data augmentation technique that extracts a portion of the training data or applies rotational transformations, taking advantage of the symmetrical shape of pallets used to load objects. This technique generates new training data from existing training data, thereby increasing the training data for the reinforcement learning algorithm and maximizing its performance.

[0030] Through this, the present invention seeks to solve the problem of maximizing loading efficiency, realizing automation and optimization of work, and increasing practical industrial applicability in various industrial fields such as logistics, manufacturing, and distribution.

[0031] Meanwhile, the technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below.

[0032] A method performed by a robot learning device operated by a processor according to one embodiment may include: obtaining state information including a state of a pallet and a size of an object to be loaded onto the pallet; converting the state information into a patch of a predefined size for inputting into a vision transformer model; determining a position to load the object onto the pallet from the patch based on an actor network of a reinforcement learning model including a policy function that determines a position to load the object in a direction that maximizes an expected value for a final loading rate of the pallet; deriving an expected value according to the decision based on a critic network of the reinforcement learning model including a value function that derives the expected value according to the decision; and updating parameters of the policy function and the value function in a direction that minimizes a loss of a loss function calculated based on the decision and the expected value.

[0033] Additionally, the state of the pallet can be specified by a grid map including a grid indicating location information on the pallet and index information indicating the height of an object loaded at a location on the grid.

[0034] Additionally, the acquiring operation may include an operation of rotating the grid map by 90 degrees to augment data to be used as state information.

[0035] Additionally, the acquiring operation may include an operation of capturing a portion of the grid map and reallocating height information of the remaining grid map excluding the captured portion to preset height information to augment data to be used as state information.

[0036] Additionally, the size of the above object can be specified by the length, width, and height of the object, respectively.

[0037] Additionally, the above-described converting operation may include a first patch set that divides the grid map into windows of a preset size; and a second patch set that includes height information of six states in which the object can be placed based on a preset position in the grid of the window size.

[0038] In addition, the value function may be set to calculate the total volume of goods loaded on the pallet as the final loading rate until the pallet is full to a preset height or becomes unloadable before reaching a preset height, and may be set to output a reward proportional to the expected value of the final loading rate.

[0039] Additionally, the policy function may be configured to compute a probability distribution for the final loading rate when the agent determines a specific location in the pallet of state information.

[0040] In addition, the policy function may be set to determine that loading is not possible if there is no case in which a preset ratio or more of the area of ​​the bottom of the object can be supported in the state information.

[0041] Additionally, the critic network may include a patch encoder, a positional embedding, and an attention layer as vision transformer-based layers designed to update the parameters of the value function.

[0042] In addition, the attention layer can be trained to limit a receptive field of a preset size to be used for learning in the pallet based on an attention mechanism and to calculate a final loading rate according to a predetermined loading position based on the receptive field.

[0043] Additionally, the attention layer may be set to gradually increase the receptive area as the number of executions of the attention mechanism increases.

[0044] Additionally, the attention layer can expand the receptive area by one grid as the number of executions of the attention mechanism increases.

[0045] In addition, after the above updating operation, the operation may further include generating information about the status of the pallet on which the object is loaded at the location and the size of the object to be loaded next according to the decision, and re-performing the obtaining operation or the updating operation.

[0046] In one embodiment, a robot learning device includes a memory including a command; and a processor that performs a predetermined operation based on the command, wherein the operation of the processor may include: an operation of obtaining state information including a state of a pallet and a size of an object to be loaded onto the pallet; an operation of converting the state information into a patch of a predefined size for inputting into a vision transformer model; an operation of determining a position to load the object onto the pallet from the patch based on an actor network of a reinforcement learning model including a policy function that determines a position to load the object in a direction that maximizes an expected value for a final loading rate of the pallet; an operation of deriving an expected value according to the decision based on a critic network of the reinforcement learning model including a value function that derives the expected value according to the decision; and an operation of updating parameters of the policy function and the value function in a direction that minimizes a loss of a loss function calculated based on the decision and the expected value.

[0047] In one embodiment, a computer program stored in a computer-readable recording medium may include instructions that, when executed on at least one processor, cause the processor to perform the following operations: obtaining state information including a state of a pallet and a size of an object to be loaded onto the pallet; converting the state information into a patch of a predefined size for inputting into a vision transformer model; determining a position to load the object onto the pallet from the patch based on an actor network of a reinforcement learning model including a policy function that determines a position to load the object in a direction that maximizes an expected value for a final loading rate of the pallet; deriving an expected value according to the decision based on a critic network of the reinforcement learning model including a value function that derives the expected value according to the decision; and updating parameters of the policy function and the value function in a direction that minimizes a loss of a loss function calculated based on the decision and the expected value.

[0048] The present invention provides an artificial intelligence model that loads objects of various sizes onto pallets of various sizes based on reinforcement learning utilizing the attention mechanism of a vision transformer, thereby achieving the following effects.

[0049] The present invention expresses the loading status of objects as a grid map and converts it into patches, thereby enabling a reinforcement learning model to utilize a network based on a vision transformer, thereby learning more accurate status information and significantly improving loading efficiency.

[0050] Furthermore, the present invention proposes a technique for gradually expanding the attention mechanism's receptive area with each execution cycle, enabling the AI ​​model to be trained to maintain high loading efficiency for pallets of various sizes. This is useful for practical application in various industrial settings, and offers significant benefits in logistics and manufacturing, which handle objects of various sizes.

[0051] Furthermore, the present invention can maximize the performance of reinforcement learning through data augmentation techniques. Specifically, it leverages the symmetry of load states to generate new training data from existing training data, thereby increasing the quantity of training data and enhancing the performance of reinforcement learning algorithms. This data augmentation technique secures training data diversity, thereby enhancing the model's generalization ability and applicability in real-world environments.

[0052] Therefore, the present invention can solve the 3D bin packing problem more efficiently and flexibly by utilizing a vision transformer in a reinforcement learning model, and can play an important role in maximizing loading efficiency and realizing automation and optimization of work in various industrial fields such as logistics, manufacturing, and distribution.

[0053] Meanwhile, the effects of the present invention are not limited to those mentioned above, and other technical effects not mentioned can be clearly understood by those skilled in the art from the description below.

[0054] Figure 1 is a configuration diagram of a robot learning device according to one embodiment.

[0055] Figure 2 is a flowchart showing the steps of operations performed by a robot learning device according to one embodiment.

[0056] Figure 3 is a conceptual diagram briefly illustrating an operation performed by a robot learning device according to one embodiment.

[0057] FIG. 4 is an exemplary diagram of an operation of rotating a pallet to augment state information to augment learning data according to one embodiment.

[0058] FIG. 5 is an example diagram of an operation of augmenting state information by retaining some information of a pallet and injecting other information to augment learning data according to one embodiment.

[0059] FIG. 6 is an example diagram of an operation for learning a value function by setting the receptive area of ​​state information to expand as the number of execution rounds of the attention mechanism increases in a critic network according to one embodiment.

[0060] Figure 7 is a flowchart of operations for using a reinforcement learning model learned according to one embodiment in an actual environment.

[0061] Figure 8 is an example diagram showing how a reinforcement learning model learned according to one embodiment is used in a real environment.

[0062] Figure 9 is a comparison table comparing the performance of the embodiment of this document with existing algorithms.

[0063] Figures 10a to 10d are tables comparing the performance of the embodiment of this document and existing algorithms on pallets of various sizes.

[0064] The purpose, technical configuration, and resulting operational effects of the present invention will be more clearly understood through the following detailed description based on the drawings attached to the specification of the present invention. The following describes embodiments of the present invention in detail with reference to the attached drawings.

[0065] The embodiments disclosed herein should not be construed or used to limit the scope of the present invention. Those skilled in the art will readily appreciate that the descriptions herein, including the embodiments, have a wide range of applications. Therefore, any embodiments described in the detailed description of the present invention are intended to serve as illustrative examples to better illustrate the present invention and are not intended to limit the scope of the present invention to the embodiments.

[0066] The functional blocks depicted in the drawings and described below are merely examples of possible implementations. Other implementations may utilize other functional blocks without departing from the spirit and scope of the detailed description. Furthermore, while one or more functional blocks of the present invention are depicted as individual blocks, one or more of the functional blocks of the present invention may be a combination of various hardware and software configurations that perform the same function.

[0067] Additionally, the expression “including certain components” is an “open” expression, simply indicating the presence of those components, and should not be understood as excluding additional components.

[0068] Furthermore, when it is said that a component is “connected” or “connected” to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

[0069] Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that various modifications, equivalents, and / or alternatives of the embodiments of the present invention are included.

[0070] FIG. 1 is a configuration diagram of a robot learning device (100) (hereinafter referred to as 'device (100)') according to one embodiment.

[0071] Referring to FIG. 1, a device (100) according to one embodiment may each include a memory (110), a processor (120), an input / output interface (130), and a communication interface (140).

[0072] The memory (110) can store data acquired from an external device or data generated by the processor. The memory (110) can store commands that can cause the processor (120) to perform operations. For example, the memory (110) can store multiple state information, reinforcement learning models, etc. as learning data to be used in training a reinforcement learning model, which will be described later.

[0073] The processor (120) is a computing device that controls overall operations. The processor (120) can execute instructions stored in the memory (110). The operation of the device (100) according to the embodiment of this document can be understood as an operation performed by the processor (120).

[0074] The input / output interface (130) may include a hardware interface or software interface for inputting or outputting information.

[0075] The communication interface (140) enables information to be transmitted and received via a communication network. To this end, the communication interface (140) may include a wireless communication module or a wired communication module.

[0076] The device (100) may be implemented in various forms of devices capable of performing operations through a processor (120) and transmitting and receiving information through a network. For example, the device may be implemented in the form of a server, a computer device, a portable communication device, a smart phone, a portable multimedia device, a laptop, a tablet PC, etc., but is not limited to these examples.

[0077] FIG. 2 is a flowchart illustrating operations performed by a device (100) according to one embodiment. FIG. 3 is a conceptual diagram briefly illustrating operations performed by a device (100) according to one embodiment. The operations of the device (100) according to the embodiments of FIGS. 2 and 3 can be understood as operations performed by a processor (120).

[0078] Each step disclosed in FIGS. 2 and 3 is merely a preferred embodiment for achieving the purpose of the present invention, and some steps may be added or deleted as needed, and one step may be included and performed in another step. The order of each operation disclosed in FIGS. 2 and 3 is merely an order arranged for convenience of understanding, and this order is not limited to a chronological order, and the order may be changed and performed differently depending on the designer's choice.

[0079] Referring to FIGS. 2 and 3 together, at step S1010, the device (100) can acquire state information to be used as training data for a reinforcement learning model. The reinforcement learning model according to the embodiment of the present document aims to train a palletizing robot to solve the 'ONLINE 3D BIN PACKING' problem, which enables it to efficiently load objects of various sizes presented in real time onto a pallet of a predetermined size.

[0080] Accordingly, state information may include information necessary for determining the actions to be performed by the agent of the reinforcement learning model (e.g., determining the location of items to be loaded onto a pallet and the orientation of the items). State information may be obtained from a pre-stored database or an external database.

[0081] For example, the status information may include information about the status of the pallet and the size of the items to be loaded onto the pallet.

[0082] According to the example of Fig. 3, the state of the pallet can be specified as a grid map including a 4x4 grid indicating location information on the pallet and index information (e.g., a number corresponding to the height) indicating the height of an item already loaded at each location among the 4x4 grid. In addition, according to the example of Fig. 3, the size of the item is specified as the length, width, and height of the item, respectively (e.g., 1x1x2). Meanwhile, the size of the pallet and the size of the item according to the example of Fig. 3 are merely examples and are not limited thereto.

[0083] Meanwhile, the device (100) can enhance the state information by modifying it as in the embodiments of FIGS. 4 and 5 to enhance the learning data.

[0084] FIG. 4 is an exemplary diagram of an operation of rotating a pallet to augment state information to augment learning data according to one embodiment.

[0085] Referring to FIG. 4, the device (100) can generate new state information as in FIG. 4 (B, C, D) by repeatedly rotating the original grid map of FIG. 4 (A) 90 degrees clockwise or counterclockwise to augment learning data.

[0086] FIG. 5 is an example diagram of an operation of augmenting state information by retaining some information of a pallet and injecting other information to augment learning data according to one embodiment.

[0087] Referring to FIG. 5, the device (100) captures a part 3x3 (FIG. 5(B)) of the original 4x4 grid map of FIG. 5(A) as in FIG. 5(C), and reallocates the height information of the remaining grid map excluding the capture to preset height information (ex. 0) to generate new state information, FIG. 5(D), thereby augmenting learning data.

[0088] In addition, referring to FIG. 5, the device (100) can capture a part 2x2 (FIG. 5(E)) of the original 4x4 grid map of FIG. 5(A) as in FIG. 5(F), and reallocate the height information of the remaining grid map excluding the capture to preset height information (ex. 0) to generate new state information, FIG. 5(G), thereby augmenting learning data.

[0089] The reinforcement learning model according to the embodiment of this document can receive state information in the form of patches, as the critic network is designed in the form of a Vision Transformer model. That is, in order for the reinforcement learning model according to the embodiment of this document to receive patch information as input data for the Vision Transformer, the device (100) can convert the state information into the form of patches by performing the following step S1020.

[0090] At step S1020, the device (100) can convert the state information into patch information of a predefined size to input into the vision transformer model.

[0091] According to the example of FIG. 3, the device (100) can generate a first patch set by dividing a 4x4 grid map that specifies the state of the pallet into windows of a preset size (e.g., 2x2). For example, if the reference position where an object is placed in a 2x2 window is set to be the 'upper left' of the window, when a grid located on the right side (or bottom side) of the grid map is captured as a window, a null value is generated in the window. In order to specify the case of such a null value, the device (100) can additionally attach a dummy grid having an index of '-1' to the right side and the bottom side of the 4x4 grid map and capture it. In this case, the device (100) can generate a first patch set of 16 (Np) 2x2 (Lp x Lp) sizes, captured so that each of the 16 grids is located on the upper left side of the 2x2 window.

[0092] According to the example of FIG. 3, the device (100) can generate a second patch set including height information of six states in which an object (e.g., a rectangular solid, a cube, etc.) can be placed based on the position of the 'upper left' in a grid of a 2x2 window size based on the size of the object.

[0093] The reinforcement learning model according to the embodiment of this document can recognize state information through a patch converted according to the above-described process.

[0094] Meanwhile, steps S1030 and S1040 are operations for training a reinforcement learning model according to the embodiment of this document. Before explaining the operations, the configuration of the reinforcement learning model according to the embodiment of this document will first be explained.

[0095] The reinforcement learning model according to the embodiment of this document can be designed as an actor-critic based reinforcement learning model.

[0096] Actor-critic-based reinforcement learning trains an agent to perform optimal actions while interacting with a given environment. The agent decides on a specific action from a given state, receives a reward proportional to the expected value, and moves to the next state. Through this process, the agent learns a policy that maximizes the total reward to best achieve its goal.

[0097] To this end, an actor-critic-based reinforcement learning model can be composed of an actor network that learns a policy function (e.g., a function that calculates the probability of selecting action a in state s) that allows the agent to make the optimal decision given state information, and a critic network that learns a value function (e.g., a Q function, a state-value function, a state-action value function) that calculates the expected value according to the agent's decision given state information.

[0098] The actor-critic-based reinforcement learning model is structured in such a way that, when specific state information is given, the actor network makes the agent make a decision that maximizes the cumulative expected value among various decisions, and the critic network determines the expected value according to the agent's decision, thereby learning the network's parameters through interaction with each other.

[0099] The policy function of the actor network and the value function of the critic network according to the embodiment of this document can be designed according to the Tsallis Actor-Critic algorithm, and the policy function and value function used in Tsallis Actor-Critic and the loss function that sets the learning direction of the parameters of each function are known in the art, so a detailed description thereof will be omitted.

[0100] The reinforcement learning model according to the embodiment of this document performs reinforcement learning according to the actor-critic based algorithm described above, but proposes a new structure of the neural network included in the actor network and critic network of the reinforcement learning model, and a method of updating the parameters of the neural network according to the attention mechanism.

[0101] Referring to Fig. 3, the actor network can be designed to learn the parameters of a policy function, including convolutional layers and fully-connected layers based on a convolutional neural network (CNN). The policy function can be configured to calculate a probability distribution based on the predicted final loading rate when the agent determines a specific location in a pallet of state information. The method of configuring the actor network based on a convolutional neural network (CNN) is a well-known technique, and a detailed description thereof will be omitted.

[0102] Referring to Fig. 3, the critic network can be designed to learn the parameters of a value function, including a layer based on a vision transformer. At this time, the value function is set to calculate the final loading rate as the total volume of the loaded items on the pallet until the pallet is full to a preset height or becomes unloadable before reaching the preset height, and the reward can be set in a direction that maximizes the final loading rate. Here, the unloadable setting can be set to determine that loading is unloadable if there is no case in which a preset percentage (e.g., 60% or more) of the area of ​​the bottom of the item can be supported in the given state information.

[0103] For example, a critic network may include a patch encoder, a positional embedding, an attention layer, and a fully-connected layer.

[0104] The patch encoder converts each patch into a vector understandable by the vision transformer model, and positional embedding adds patch location information to each converted vector, enabling the vision transformer model to recognize the location of each patch. The attention layer consists of multiple layers that perform the attention mechanism. The fully connected layer can combine input data to produce the final output.

[0105] Here, the attention mechanism, a technique primarily used in computer vision (CV), is designed to focus on important parts of input data. The attention mechanism allows the model to assign weight to more important parts of a particular input (e.g., the receptive field) and ignore less important parts, enabling more effective learning.

[0106] The configuration of the patch encoder, positional embedding, attention layer, and fully connected layer itself corresponds to the configuration mainly used in the vision transformer model, but the attention layer included in the critic network according to the embodiment of this document can perform a new operation as in the embodiment of FIG. 6 when learning is performed through the attention mechanism.

[0107] FIG. 6 is an example diagram of an operation for learning a value function by setting the receptive area of ​​state information to expand as the number of execution rounds of the attention mechanism increases in a critic network according to one embodiment.

[0108] Referring to FIG. 6, the device (100) can learn the value function of the critic network by setting the acceptance area of ​​the grid map for calculating the reward to gradually increase as the number of execution rounds of the attention mechanism increases.

[0109] For example, in the first execution round of the attention mechanism, the device (100) does not cause the attention layer to compute the expected value over the entire grid map,

[0110] By setting the patch range including the loading location of the object decided by the agent (black dot) and the surrounding location (empty circle) spaced one grid block apart from the loading location as a receptive area (blue and red area), the device (100) can proceed with attention mechanism learning so that the attention layer recognizes the pallet as the size set as the receptive area of ​​the first round, masks the area that is not the receptive area, and calculates the expected value according to the agent's action based on the receptive area.

[0111] Accordingly, in the attention mechanism of the first execution round, patches including the loading position (black dot) and surrounding positions (empty circles) can be used for learning. Assuming that the patch corresponding to the loading position and surrounding positions is selected based on the 'lower left' position in the patch, in the case of Figure 6, a total of six patches including the loading position and five surrounding surrounding positions can be considered for learning. In this case, the blue area represents the patch area including the loading position, and the red area represents the patch area including the surrounding positions. The sum of these two areas can constitute the receptive field of the first attention mechanism.

[0112] Afterwards, when moving to the second attention mechanism, the device (100) can proceed with attention mechanism learning by setting the patch range including the surrounding locations (empty circles) spaced two grid blocks apart from the loading location as the receptive area (blue and red areas), so that the attention layer recognizes the pallet as the size set as the receptive area of ​​the second round, and calculates the expected value according to the agent's action based on the receptive area.

[0113] In the second attention mechanism training, seven additional grids not included in the first attention are included, expanding the overall receptive field. In other words, the critique network learns a value function by incorporating a wider range of surrounding information than the first attention mechanism through the second attention mechanism. Accordingly, the attention mechanism according to the embodiment of this paper starts from a small area of ​​the grid map, and as the number of iterations of the attention mechanism increases, the receptive field gradually expands, incorporating new surrounding information with each iteration. Furthermore, the attention mechanism is repeated until the entire grid map is reached. In this way, the critique network can learn a more accurate value function by considering state information in a broader context.

[0114] This gradual expansion of the receptive field enables rapid learning using local information about the loading location and its surroundings in the early stages of learning, and then expanding the receptive field as the training progresses, allowing learning to reflect the overall context. Accordingly, the reinforcement learning model according to the embodiment of this document can be trained to adapt to various pallet sizes and estimate expected values ​​based on loading locations within the grid area. This allows a palletizing robot equipped with a reinforcement learning model to more efficiently determine the loading location of items, ultimately maximizing pallet loading efficiency.

[0115] Meanwhile, the example of Fig. 6 assumes a 4x4 grid map for ease of understanding and explains the first and second rounds. However, if the grid map is larger, the range of the spacing grid can be gradually increased as the rounds progress, so that the receptive area to be included can be set to expand from a small area to the entire area by repeating multiple rounds.

[0116] According to the design of the above-described reinforcement learning model, in step S1030, the device (100) can be trained according to the actor-critic reinforcement learning algorithm so that the actor network determines an action (=determining the location and placement direction of loading an object on the pallet) in a direction that maximizes the expected value for the final loading rate of the pallet. The device (100) can allow the agent to determine the location and placement direction of loading an object on the pallet from patch information based on the actor network including a policy function. At this time, the policy function calculates a probability distribution for the expected final loading rate when the agent determines a specific location in the pallet of the state information, and the agent can determine the location where the object is to be loaded on the grid map, the bottom area of ​​the object to be loaded at the corresponding location, and the direction of the object to be loaded. At this time, the reinforcement learning policy may include a policy that determines that loading is impossible if a preset ratio or more of the bottom area of ​​the object cannot be supported.

[0117] At step S1040, the device (100) can be trained according to the actor-critic reinforcement learning algorithm so that the critic network can derive an expected value based on the agent's decision. As in the embodiment of FIG. 6 described above, the device (100) can be trained so that the attention layer limits a receptive area of ​​a preset size to be used for learning in the pallet based on the attention mechanism, and calculates a final loading rate according to a predetermined loading position based on this receptive area.

[0118] At step S1050, the device (100) can update the parameters of the policy function and the value function in a direction in which the loss according to the loss function of the policy function and the loss function of the value function is minimized based on the decision of the agent at step S1030 and the expected value according to the decision calculated at step S1040.

[0119] Meanwhile, the structure of the loss function of the policy function and value function according to the embodiment of this document can be designed according to the Tsallis Actor-Critic algorithm, and the parameters can be updated according to the parameter update method used in the Tsallis Actor-Critic algorithm.

[0120] In step S1060, the device (100) can newly generate the status of the pallet loaded with the object based on the decision made in step S1030. For example, the area occupied by the object loaded based on the decision made in step S1030 and the height information can be reflected in the grid map of step S1010.

[0121] Additionally, in step S1060, after the loaded object is reflected, the device (100) obtains information on the size of the new object to be loaded next, and can re-perform the operations of steps S1010 to S1050 described above. The number of times the re-performed operation according to step S1060 is performed may be repeated according to the settings of the designer who trains the reinforcement learning model, or may be repeated until the reinforcement learning model determines that the object cannot be loaded.

[0122] FIG. 7 is a flowchart of an operation of using a reinforcement learning model learned in an actual environment according to one embodiment, and FIG. 8 is an example diagram of using a reinforcement learning model learned in an actual environment according to one embodiment.

[0123] The operations according to the embodiments of FIGS. 7 and 8 may be understood as operations performed by a device (hereinafter, exemplified as a "palletizing robot") that uses a reinforcement learning model learned according to the embodiment of FIG. 2. For example, the palletizing robot may include a processor that operates the learned reinforcement learning model, or a communication module that communicates with a server that stores the learned reinforcement learning model.

[0124] Each step disclosed in FIGS. 7 and 8 is merely a preferred embodiment for achieving the purpose of the present invention, and some steps may be added or deleted as needed, and one step may be included and performed within another step. The order of each operation disclosed in FIG. 7 is merely arranged for the convenience of understanding, and this order is not limited to a chronological order, and the order may be changed and performed differently depending on the designer's choice.

[0125] Referring to FIGS. 7 and 8, at step S2010, the palletizing robot can observe current status information, including the size of the pallet and the items to be loaded onto the pallet. For example, the palletizing robot can collect status information, including the status of the pallet and the size and location of the items. For example, the palletizing robot can include a camera module, and can observe the pallet, the items already loaded onto the pallet, and the items to be loaded onto the pallet through the camera module.

[0126] In step S2020, the palletizing robot can determine the position to load an object on the pallet and the placement direction of the object based on the current state information based on the reinforcement learning model learned according to the embodiment of this document. For example, the palletizing robot can determine the position to load an object and the placement direction of the object to maximize the loading rate on the pallet using the policy function of the reinforcement learning model. In this case, the learned reinforcement learning model can receive the state information as a patch (e.g., the same process as the operations of steps S1010 and S1020 described above), calculate the Q-value for each position of the pallet, and select the position with the highest Q-value as the optimal position to load the object.

[0127] In step S2030, the palletizing robot can verify the output from the reinforcement learning model. If the reinforcement learning model outputs that there is no suitable loading location, the palletizing robot can terminate the loading process for the pallet. If the reinforcement learning model determines that there is a loading location, the palletizing robot proceeds to step S2040.

[0128] In step S2040, the palletizing robot can load objects in a determined layout direction at a location determined by the reinforcement learning model. For example, the palletizing robot can use its robot arm to move the objects to a loading location and load the objects. After loading, the palletizing robot returns to step S2010 to observe the current status information of newly loaded objects and to observe newly loaded objects. As shown in FIG. 8(B), the entire process can be repeated until the pallet is full of objects to a preset height or the pallet becomes unloadable.

[0129] Figure 9 is a comparison table comparing the performance of the embodiment of this document with existing algorithms. Figure 9 shows the performance of each algorithm on three different datasets (Random, CUT-1, and CUT-2), and the performance indicators were evaluated based on pallet loading efficiency.

[0130] Among the datasets, Random is a dataset that randomly presents the sizes of objects, CUT-1 is a dataset that arranges each object in descending order of height when all of the objects are loaded without gaps, and CUT-2 is a dataset that randomly arranges the order within the range that can be loaded when all of the objects are loaded without gaps.

[0131] Here, the embodiments of this document correspond to Base, GRE, and GREViT, and Base corresponds to an embodiment in which the attention mechanism is used for a uniform receptive area without using data augmentation, GRE corresponds to an embodiment in which the attention mechanism is used in a gradual expansion manner (Fig. 6) without using data augmentation, and GREViT corresponds to an embodiment in which the data augmentation is used (Figs. 4 and 5) and the attention mechanism is used in a gradual expansion manner (Fig. 6).

[0132] In the table in Figure 9, the first column indicates the algorithm name, the second column indicates performance on the Random dataset, the third column indicates performance on the CUT-1 dataset, and the fourth column indicates performance on the CUT-2 dataset. Performance on each dataset is measured by pallet loading efficiency, with a higher value indicating higher loading efficiency.

[0133] Table 9 shows the performance of several existing algorithms, including CP, EP, EMS, HM, DBL, LSAH, and MACS. For example, the CP algorithm showed a performance of 0.469 on the Random dataset, and 0.501 on the CUT-1 and CUT-2 datasets, respectively. In contrast, the EP algorithm showed a performance of 0.510 on the Random dataset, 0.532 on the CUT-1 dataset, and 0.541 on the CUT-2 dataset. In addition, Zhao et al. The algorithm and PCT algorithm showed performances of 0.662 ± 0.024 and 0.687 ± 0.003 on the Random dataset, 0.675 ± 0.012 and 0.703 ± 0.001 on the CUT-1 dataset, and 0.670 ± 0.019 and 0.679 ± 0.005 on the CUT-2 dataset, respectively.

[0134] Among the examples in this paper, GREViT showed the highest loading efficiency across all datasets, with performances of 0.704 ± 0.004 on the Random dataset, 0.731 ± 0.013 on the CUT-1 dataset, and 0.730 ± 0.004 on the CUT-2 dataset. In other words, the examples in this paper confirm that combining vision transformers and reinforcement learning can more effectively solve the pallet loading problem.

[0135] Figures 10a to 10d are tables comparing the performance of the embodiment of this document and existing algorithms on pallets of various sizes. Figure 10 shows the average loading ratios for 10x10 pallets, 9x9 pallets, 8x8 pallets, and 7x7 pallets, respectively, and the dataset consists of three types: Random, CUT-1, and CUT-2.

[0136] Among the datasets, Random is a dataset that randomly presents the sizes of objects, CUT-1 is a dataset that arranges each object in descending order of height when all of the objects are loaded without gaps, and CUT-2 is a dataset that randomly arranges the order within the range that can be loaded when all of the objects are loaded without gaps.

[0137] Here, the embodiments of this document correspond to Base, GRE, and GREViT, and Base corresponds to an embodiment in which the attention mechanism is used for a uniform receptive area without using data augmentation, GRE corresponds to an embodiment in which the attention mechanism is used in a gradual expansion manner (Fig. 6) without using data augmentation, and GREViT corresponds to an embodiment in which the data augmentation is used (Figs. 4 and 5) and the attention mechanism is used in a gradual expansion manner (Fig. 6).

[0138] Figure 10a shows the performance on a 10x10 sized pallet. In the case of GREViT among the examples in this paper, it shows a performance of 0.704 ± 0.004 on the Random dataset, 0.731 ± 0.013 on the CUT-1 dataset, and 0.730 ± 0.004 on the CUT-2 dataset, confirming that it shows the best loading ratio performance on each dataset.

[0139] Figure 10b shows the performance on a 9x9 sized pallet. In the case of GREViT among the examples in this paper, it shows a performance of 0.698 ± 0.020 on the Random dataset, 0.719 ± 0.008 on the CUT-1 dataset, and 0.707 ± 0.006 on the CUT-2 dataset, confirming that it shows the best loading ratio performance on each dataset.

[0140] Figure 10c shows the performance in size pallets. In the case of GREViT among the examples in this paper, it shows performance of 0.647 ± 0.022 in the Random dataset, 0.721 ± 0.012 in the CUT-1 dataset, and 0.713 ± 0.008 in the CUT-2 dataset, confirming that it shows the best loading ratio performance in each dataset.

[0141] Figure 10d shows the performance in size pallets. In the case of GREViT among the examples in this paper, it shows performance of 0.639 ± 0.008 in the Random dataset, 0.728 ± 0.016 in the CUT-1 dataset, and 0.705 ± 0.003 in the CUT-2 dataset, confirming that it shows the best loading ratio performance in each dataset.

[0142] That is, the experimental results for pallets of various sizes. Among the examples in this document, GREViT shows the highest loading rate for all pallet sizes.

[0143] According to the above-described embodiment, the present invention provides an artificial intelligence model that loads objects of various sizes onto pallets of various sizes based on reinforcement learning utilizing the attention mechanism of a vision transformer, thereby achieving the following effects.

[0144] The present invention expresses the loading status of objects as a grid map and converts it into patches, thereby enabling a reinforcement learning model to utilize a network based on a vision transformer, thereby learning more accurate status information and significantly improving loading efficiency.

[0145] Furthermore, the present invention proposes a technique for gradually expanding the attention mechanism's receptive area with each execution cycle, enabling the AI ​​model to be trained to maintain high loading efficiency for pallets of various sizes. This is useful for practical application in various industrial settings, and offers significant benefits in logistics and manufacturing, which handle objects of various sizes.

[0146] Furthermore, the present invention can maximize the performance of reinforcement learning through data augmentation techniques. Specifically, it leverages the symmetry of load states to generate new training data from existing training data, thereby increasing the quantity of training data and enhancing the performance of reinforcement learning algorithms. This data augmentation technique secures training data diversity, thereby enhancing the model's generalization ability and applicability in real-world environments.

[0147] Therefore, the present invention can solve the 3D bin packing problem more efficiently and flexibly by utilizing a vision transformer in a reinforcement learning model, and can play an important role in maximizing loading efficiency and realizing automation and optimization of work in various industrial fields such as logistics, manufacturing, and distribution.

[0148] The various embodiments and terms used in this document are not intended to limit the technical features described in this document to specific embodiments, but should be understood to encompass various modifications, equivalents, or alternatives of the embodiments. In connection with the description of the drawings, similar reference numerals may be used to refer to similar or related components. The singular form of a noun corresponding to an item may include one or more items, unless the context clearly indicates otherwise.

[0149] In this document, the phrases "A or B", "at least one of A and B", "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" can each include all possible combinations of the items listed together in that phrase. Terms such as "1", "2", or "first" or "second" may be used merely to distinguish the corresponding component from other corresponding components and do not limit the corresponding components in any other respect (e.g., importance or order). When a (e.g., a first) component is referred to as "coupled" or "connected" to another (e.g., a second) component, with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0150] The term "module" as used herein may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0151] Various embodiments of the present document may be implemented as software (e.g., a program) including one or more instructions stored in a storage medium (e.g., a memory) readable by a device (e.g., an electronic device). The storage medium may include random access memory (RAM), a memory buffer, a hard drive, a database, erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), read-only memory (ROM), and / or the like.

[0152] Additionally, the processor of the embodiments of the present document can call at least one command among one or more commands stored from a storage medium and execute it. This enables the device to operate to perform at least one function according to the at least one command called. These one or more commands may include code generated by a compiler or code executable by an interpreter. The processor may be a general-purpose processor, a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), and / or the like.

[0153] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0154] The methods according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated on a machine-readable storage medium, such as a manufacturer's server, an application store's server, or a server's memory.

[0155] According to various embodiments, each component (e.g., a module or a program) of the described components may include a single or multiple entities. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. A method for performing a robot learning device operated by a processor, An operation of obtaining status information including the status of a pallet and the size of an item to be loaded on the pallet; An operation of converting the above state information into a patch of a predefined size for inputting into a vision transformer model; An operation of determining a location to load the object on the pallet from the patch based on an actor network of a reinforcement learning model including a policy function for determining a location to load the object in a direction that maximizes an expected value for the final loading rate of the pallet; An operation of deriving an expected value according to the decision based on a critic network of the reinforcement learning model including a value function that derives the expected value according to the decision; and Including an operation of updating the parameters of the policy function and the value function in a direction in which the loss of the loss function calculated based on the above decision and expected value is minimized. method.

2. In paragraph 1, The condition of the above pallet is It is specified as a grid map including a grid indicating location information on the pallet and index information indicating the height of an object loaded at a location on the grid. method.

3. In paragraph 2, The above acquisition action is Including an operation to rotate the above grid map by 90 degrees to augment data to be used as state information. method.

4. In paragraph 2, The above acquisition action is An operation of capturing a portion of the above grid map and reallocating the height information of the remaining grid map excluding the capture to preset height information to augment data to be used as state information, method.

5. In paragraph 2, The size of the above item is Specified by the length, width, and height of each of the above objects, method.

6. In paragraph 5, The above conversion operation is A first patch set that divides the above grid map into windows of a preset size; and A second patch set including height information of six states in which the object can be placed based on a predetermined position on a grid of the above window size, method.

7. In paragraph 1, The above value function is The total volume of the goods loaded on the pallet is set to be calculated as the final loading rate until the goods are full to the preset height on the pallet or the loading state becomes unavailable before the preset height is reached, and a compensation proportional to the expected value of the final loading rate is output. method.

8. In paragraph 7, The above policy function is The agent is set to compute a probability distribution for the expected final load factor when it determines a specific location in the pallet of the above state information. method.

9. In paragraph 8, The above policy function is In the above status information, if there is no case in which the area of ​​the bottom of the object can be supported by a preset ratio or more, it is determined that the object cannot be loaded. method.

10. In paragraph 7, The above critic network A vision transformer-based layer designed to update the parameters of the above value function, comprising a patch encoder, a positional embedding, and an attention layer. method.

11. In paragraph 10, The above attention layer Based on the attention mechanism, a receptive field of a preset size to be used for learning in the pallet is limited, and the learning is performed to calculate the final loading rate according to a predetermined loading position based on the receptive field. method.

12. In paragraph 11, The above attention layer As the number of executions of the attention mechanism increases, the receptive area is set to gradually increase. method.

13. In paragraph 12, The above attention layer As the number of executions of the attention mechanism increases, the receptive area is expanded by one grid at a time. method.

14. In paragraph 1, After the above updating action, Further comprising an operation of generating information about the status of the pallet on which the object is loaded at the location and the size of the object to be loaded next according to the above decision, and re-performing the obtaining operation or the updating operation. method.

15. Memory containing instructions; and Includes a processor that performs a predetermined operation based on the above command, The operation of the above processor is as follows: An operation of obtaining status information including the status of a pallet and the size of an item to be loaded on the pallet; An operation of converting the above state information into a patch of a predefined size for inputting into a vision transformer model; An operation of determining a location to load the object on the pallet from the patch based on an actor network of a reinforcement learning model including a policy function for determining a location to load the object in a direction that maximizes an expected value for the final loading rate of the pallet; An operation of deriving an expected value according to the decision based on a critic network of the reinforcement learning model including a value function that derives the expected value according to the decision; and Including an operation of updating the parameters of the policy function and the value function in a direction in which the loss of the loss function calculated based on the above decision and expected value is minimized. Robot learning device.

16. When performed on at least one processor, The above processor An operation of obtaining status information including the status of a pallet and the size of an item to be loaded on the pallet; An operation of converting the above state information into a patch of a predefined size for inputting into a vision transformer model; An operation of determining a location to load the object on the pallet from the patch based on an actor network of a reinforcement learning model including a policy function for determining a location to load the object in a direction that maximizes an expected value for the final loading rate of the pallet; An operation of deriving an expected value according to the decision based on a critic network of the reinforcement learning model including a value function that derives the expected value according to the decision; and Including a command that causes an operation to update the parameters of the policy function and the value function in a direction in which the loss of the loss function calculated based on the above decision and expected value is minimized. A computer program stored on a computer-readable recording medium.

Images