DEVICE AND METHOD FOR TRAINING REINFORCEMENT LEARNING AGENT FOR AUTOMATING THE COLLAGE GENERATION PROCESS

Abstract

To provide a device and method for training reinforcement learning agent for automating a collage generation process by learning a decision-making process required for collage generation by using a reinforcement learning model.SOLUTION: A reinforcement learning agent training device for automating a collage generation process includes: a memory storing a reinforcement learning agent training program; and a processor which executes the program stored in the memory. Therein the reinforcement learning agent training program causes the processor to execute: a step S110 of determining an action for a collage once state information including a canvas, a material, a target, and a remaining number of times is input; a step S120 of applying the action to a first canvas to render a second canvas; a step S130 of updating a reward based upon similarity between the first canvas and second canvas and the target; and a step S140 of repeating the respective steps S110 to S130 to train a reinforcement learning agent.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a reinforcement learning agent learning apparatus and method for automating the collage generation process. [Background technology]

[0002] Collage is an artistic genre in which unrelated image fragments from newspapers, magazines, photographs, etc. are arranged to create a unified image (target image). Traditionally, collages are completed through the artist's own process of selecting and arranging materials, requiring considerable skill and time to create a high-quality collage image. Using computers can shorten this process. Existing computer-based techniques for automatically generating collage images include simple calculation-based methods and pixel-based generation methods. Simple calculation-based methods divide the target image into a checkerboard-like pattern and insert images with similar colors into each area, or randomly fill in photos according to a predetermined shape. These existing methods are the simplest form of automated collage and are easy to implement, but the range of expression they can produce is very limited and lacks artistic quality. Pixel-based generation methods convert a target image into a collage-style image or generate a collage-style image from text. Furthermore, pixel-based generation methods are not specifically for collages, but rather are a technology for image generation itself. Advances in deep learning technology have made it possible to generate a wider range of images than ever before, and this is widely known among the general public. However, these pixel-based generation methods generate images statistically on a pixel-by-pixel basis, relying solely on learning data to imitate collage styles and not involving the process of creating an actual collage. Therefore, it is not possible to specify the materials for the collage (virtual materials), and it is difficult to set a clear target image (virtual target image).

[0003] Reinforcement learning, an artificial intelligence technique, allows a model to directly interact with the environment through actions, accumulate experience (trial and error), and learn behaviors that maximize rewards based on that experience. The advantage of reinforcement learning is that it does not require predefined data to learn behaviors. This distinguishes it from supervised learning techniques, which require a large amount of data. Furthermore, because reinforcement learning does not rely on data, it can independently discover and acquire good behavioral structures. In certain artistic fields, it is difficult to obtain available data due to issues such as absolute numbers and copyright, making supervised learning difficult to apply. Furthermore, data on the artistic process is even more difficult to obtain, significantly limiting the application of supervised learning. Therefore, reinforcement learning allows a model to independently discover and learn the artistic process without relying on data. Collage, in particular, requires a much more complex decision-making process due to the involvement of materials, making it necessary to develop an efficient learning environment and method tailored to collage. However, conventional computation-based methods are extremely restrictive in terms of the form and artistic quality of collages, and pixel-based image generation methods have the limitation that they statistically imitate collage styles and arrange pixels, rather than being generated through the decision-making process required for actual collages.

[0004] Therefore, the present invention proposes a method for an AI to learn the decision-making process required for creating a collage by itself using a reinforcement learning model, and a method for automating collage creation based on the learned results, thereby enabling the creation of a collage image by automating all the processes required for actually creating a collage, such as material selection, cutting, and pasting. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Korean Patent No. 10-2405324 (Name of invention: Method for learning drawing on a terminal) Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention is intended to solve the above-mentioned problems, and aims to provide a reinforcement learning agent learning device and method for automating the collage generation process, which uses a reinforcement learning model to learn the decision-making process required for collage generation.

[0007] However, the technical problem that this embodiment aims to achieve is not limited to the above-mentioned technical problem, and other technical problems may exist. [Means for solving the problem]

[0008] As a technical means for solving the above-mentioned technical problems, a reinforcement learning agent learning device for automating a collage generation process according to one embodiment of the present invention includes a memory storing a reinforcement learning agent learning program, and a processor for executing the program stored in the memory, wherein the reinforcement learning agent learning program includes: (a) a step of determining an action for the collage when state information including a canvas, a material, a goal, and a remaining number of times is input; (b) a step of applying the action to a first canvas to render a second canvas; (c) a step of updating a reward based on the similarity between the first canvas, the second canvas, and the goal; and (d) a step of repeating steps (a) to (c) to train the reinforcement learning agent.

[0009] A training method for a reinforcement learning agent training device for automating a collage generation process according to another embodiment of the present invention includes: (a) determining an action for a collage when state information including a canvas, a material, a goal, and a remaining number of times is input; (b) applying the action to a first canvas to render a second canvas; (c) updating a reward based on the similarity between the first canvas, the second canvas, and the goal; and (d) repeating steps (a) to (c) to train the reinforcement learning agent. [Effects of the Invention]

[0010] According to any one of the above-mentioned solutions to the problems of the present invention, prior knowledge of the method for autonomously creating collages through reinforcement learning is not required, so there is no cost for collecting and refining collage data. Depending on the method for defining the reward required for reinforcement learning, the direction of learning can be set differently for each user.

[0011] In addition, the present invention allows users to know the entire process of creating a collage, specify the materials, clearly define the target image, and even allows users to participate in the collage creation process and input a set of rules for individualization. [Brief explanation of the drawings]

[0012] FIG. 1 is a configuration diagram of a reinforcement learning agent learning device according to an embodiment of the present invention.

[0013] 2 and 3 are diagrams illustrating a reinforcement learning agent learning process according to an embodiment of the present invention.

[0014] 4, 5, 6, 7 and 8 are diagrams illustrating detailed modules of a reinforcement learning agent learning device according to an embodiment of the present invention.

[0015] FIG. 9 is a diagram illustrating a collage generation process of a reinforcement learning agent according to an embodiment of the present invention.

[0016] 10, 11, 12, and 13 are diagrams illustrating an example of collage generation for a reinforcement learning agent according to an embodiment of the present invention.

[0017] 14 and 15 are diagrams for comparing and explaining an example of collage generation of a reinforcement learning agent according to an embodiment of the present invention with an existing pixel-based methodology. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily carry out the present invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not relevant to the description will be omitted, and similar parts will be designated by similar reference numerals throughout the specification.

[0019] Throughout this specification, a part being "coupled" to another part includes not only a part being "directly coupled" to another part, but also a part being "electrically coupled" to another part via another element therebetween. Furthermore, when a part "comprises" a certain component, this does not mean that the part excludes other components, but that the part can further include other components, unless otherwise specified.

[0020] As used herein, the term "module" includes hardware-implemented units, software-implemented units, and units implemented using both hardware and software. Furthermore, one unit may be implemented using two or more pieces of hardware, or two or more units may be implemented by a single piece of hardware. Meanwhile, a "module" is not limited to software or hardware; a "module" may reside on an addressable storage medium or implement one or more processors. Thus, by way of example, a "module" includes components such as software components, object-oriented software components, class components, and task components, processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The components and functions provided within a "module" may be combined into fewer components and "modules" or further separated into additional components and "modules." Furthermore, a component and a "module" may be embodied to implement one or more CPUs within a device.

[0021] A network refers to a connection structure that allows information exchange between nodes such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), the Internet (WWW), wired and wireless data communication networks, telephone networks, wired and wireless television communication networks, etc. Examples of wireless data communication networks include, but are not limited to, 3G, 4G, 5G, 3GPP (registered trademark) (3rd Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth (registered trademark), infrared communication, ultrasonic communication, visible light communication (VLC), LiFi, etc.

[0022] FIG. 1 is a block diagram of a reinforcement learning agent learning device according to an embodiment of the present invention, and FIGS. 2 and 3 are diagrams illustrating a reinforcement learning agent learning process according to an embodiment of the present invention.

[0023] Referring to FIG. 1, the reinforcement learning agent learning device (100) may include a communication module (110), a memory (120), a processor (130), and a database (140).

[0024] The reinforcement learning agent learning device 100 may be implemented as a computer or a mobile terminal that can connect to a network. Here, the computer may include, for example, a notebook computer, a desktop computer, a laptop computer, etc., and the mobile terminal may include, for example, any kind of handheld-based wireless communication device, such as various smartphones, tablet PCs, smart watches, etc., as a wireless communication device that ensures portability and mobility.

[0025] In addition, the reinforcement learning agent learning device 100 can function as a server that provides the reinforcement learning agent's learning results using state information including canvas, material, goal, and remaining number of times to an external computing device. In this case, the server can operate in a cloud computing service model such as SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service), or can be constructed in the form of a private cloud, public cloud, or hybrid cloud.

[0026] The communication module 110 may be a device that includes the necessary hardware and software to transmit and receive signals, such as control signals or data signals, with other network devices through wired or wireless connections.

[0027] The memory 120 may have a reinforcement learning agent learning program recorded therein. The reinforcement learning agent learning program includes a step of determining an action for the collage when state information including a canvas, a material, a goal, and a remaining number of times is input (S110), a step of applying the action to a first canvas to render a second canvas (S120), a step of updating a reward based on the similarity between the first canvas, the second canvas, and the goal (S130), and a step of training the reinforcement learning agent 200 by repeating steps S110 to S130 (S140). Here, the memory 120 may include a magnetic storage medium or a flash storage medium in addition to a volatile storage device that requires power to maintain stored information, but the scope of the present invention is not limited thereto.

[0028] The memory 120 may store a separate program, such as an operating system for processing and controlling the processor 130, and may also perform a function for temporarily storing input or output data.

[0029] The processor 130 executes a reinforcement learning agent learning program (hereinafter, "program") stored in the memory 120, and provides a function for controlling the hardware of the reinforcement learning agent learning device 100 through the execution of the program. That is, the processor 130 executes the program to perform hardware control functions such as necessary file systems, memory allocation, networking, basic libraries, timers, device control (display, media, input device, 3D, etc.), and other utilities.

[0030] 2 and 3, the processor 130 receives state information including the canvas, material, goal, and remaining number of times, and then performs the steps of determining an action for the collage (S110), applying the action to the first canvas to render the second canvas (S120), updating a reward based on the similarity between the first canvas, the second canvas, and the goal (S130), and repeating steps S110 to S130 to train the reinforcement learning agent 200 (S140). Specific steps of the reinforcement learning agent's learning process through program execution will be described later with reference to FIGS. 4 to 8.

[0031] The processor 130 may include any type of device capable of processing data. For example, it may refer to a data processing device built into hardware having circuits physically structured to perform functions expressed by code or commands contained in a program. Examples of such data processing devices built into hardware include a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., but the scope of the present invention is not limited to these.

[0032] The database 140 stores or provides data required for the reinforcement learning agent learning device 100 under the control of the processor 130. For example, the database 140 may store results generated during the reinforcement learning agent learning process. The database 140 may be included as a separate component from the memory 120 or may be constructed in a partial area of ​​the memory 120.

[0033] 4 to 8 are diagrams illustrating detailed modules of a reinforcement learning agent learning device according to an embodiment of the present invention.

[0034] 4, the processor 130 may implement detailed modules that perform various functions by executing a reinforcement learning agent learning program. For example, the reinforcement learning agent learning program may be executed by the processor 130 to implement an actor 210, a renderer 220, a critic 230, and a discriminator 240.

[0035] FIG. 5 is a diagram for explaining a self-learning method for a collage generation process by the reinforcement learning agent learning device of the present invention.

[0036] For example, referring to FIG. 5, when the actor (210) observes state information including the canvas, material, and goal and decides on an action, the renderer (220) can manipulate the material according to the action to render the first canvas onto the second canvas. At this time, the canvas before manipulation is the first canvas, and the canvas after manipulation is the second canvas. At this time, the actor (210) deforms the canvas so that it approaches the goal. Therefore, the reward for the actor's (210) action can be set based on the difference between the first similarity between the first canvas and the goal (the similarity between the canvas and the goal before manipulation) and the second similarity between the second canvas and the goal (the similarity between the canvas and the goal after manipulation). The critic (230) learns how the action decided by the actor (210) affects future rewards, and the actor (210) may continuously learn actions that will bring relatively greater rewards. Therefore, the reinforcement learning agent learning device 100 allows the reinforcement learning agent 200 to learn using various materials and goals. The trained reinforcement learning agent 200 can generate a collage according to an automated collage generation process that brings the canvas closer to the goal, regardless of the goal or material given.

[0037] Specifically, the actor (210) can determine the actions required to create a collage when state information including the canvas, material, goal, and remaining number of times is input. The actor (210) can observe state information including the first canvas, material, goal, and remaining number of times to determine the actions to take regarding the collage.

[0038] Here, the canvas is a background image onto which collage materials are pasted, and may be a virtual video frame rather than a real object, with a target image already drawn on it. The first canvas is the image before the materials are manipulated, and the second canvas is the image after the materials are manipulated. Materials can be newspapers, magazines, and various other conventional image data used to create collage works. The target is the collage image that is ultimately to be generated by repeating the collage generation process of manipulating materials on the canvas a remaining number of times. The remaining number of times refers to the number of times remaining available materials.

[0039] As an example, the state information may be in the form of a matrix that can be input to the actor (210), where the canvas, material, and target are each RGB three-channel images, and the remaining number may be a scalar value rather than an image value.

[0040] For example, if the size of an image is n*n (width, height), the image can be expressed as an n*n*3 matrix (the 3 means the three RGB channels). If the canvas, material, and target image are all set to the same n*n, the images can be merged in the channel dimension and expressed as an n*n*9 matrix. The remaining number of times can be expressed as n*n*1 by copying the scalar value. As a result, the status information can be expressed as n*n*10 by merging an n*n*9 matrix and an n*n*1 matrix in the channel dimension.

[0041] The actor (210) can observe the input state information and determine an action. The actor (210) can select a first action element to cut out a piece of material and a second action element to paste the cut piece of material onto the canvas. In this case, the action includes a first action element and a second action element, where the first action element includes a virtual frame, a position of the virtual frame on the material, the horizontal and vertical lengths of the virtual frame, and the positions of points on each side of the virtual frame, and the second action element includes the piece of material, the position of the piece of material on the canvas, and the rotation angle of the piece of material.

[0042] For example, the actor 210 may determine a first behavior element and a second behavior element. The first behavior element may determine a rectangular frame for cutting the material, a position on the material where the rectangular frame will be placed, the horizontal and vertical dimensions of the rectangular frame for cutting the material, and the positions of points on each side of the rectangular frame (the positions of the points on each side are determined proportionally). Then, pieces of material may be generated according to the shape of the rectangle formed by connecting the four points. That is, a cutting operation may be performed by the first behavior element, and pieces of material may be cut from the material. The second behavior element may determine pieces of material to be pasted on a canvas, a position on the first canvas where the pieces of material will be placed, and a rotation angle at which the pieces of material will be pasted. That is, a pasting operation may be performed by the second behavior element, and the pieces of material may be pasted on the first canvas, creating a second canvas.

[0043] FIG. 6 is a diagram for explaining the renderer (220) of the present invention.

[0044] 6, the renderer (220) can apply an operation to a first canvas to render a second canvas, i.e., the renderer (220) can render the first canvas onto the second canvas through an operation of manipulating material (cut and paste).

[0045] For example, the state transition process through the renderer 220 can be divided into a first canvas before pasting a new piece of material (State 1), pasting a new piece of material onto the first canvas (Action), and a second canvas with the new piece of material pasted (State 2). In this case, the rendering process can use differentiable image operations and a separate pre-trained network (shaper).

[0046] For example, the renderer (220) can generate a mask corresponding to the virtual frame based on a first behavioral element, use the mask to cut out material to generate pieces of material, and paste the pieces of material onto the first canvas based on a second behavioral element to generate a second canvas.

[0047] For example, the renderer 220 can output a mask having the shape of a piece of material determined based on the input first behavior element. Multiplying the material by the mask can cut out pieces of material according to the shape of the mask. The renderer 220 can then paste the pieces of material onto a first canvas based on the input second behavior element. Differentiable image operations (transition and rotation) and arithmetic operations can be applied to the second behavior element and pasted onto the first canvas. As a result, when a first canvas and behavior (first behavior element / second behavior element) are input to the renderer 220, a differentiable series of processes can be performed to output the next second canvas. The renderer 220 uses a state transition function, which can be designed to be differentiable for compatibility with reinforcement learning algorithms.

[0048] For example, the present invention can train a reinforcement learning agent (200) using Equation 1, which is the objective function of model-based SAC (Soft Actor-Critic), which is constructed by modifying SAC (Soft Actor-Critic). As a result, the present invention allows the reinforcement learning agent (200) to know the state transition function and reward function (model), which is effective for collage learning.

[0049]

number

[0050] Here, V is the critic (value function), Jv is the objective function for critic learning, D is the experience data storage (replay buffer) for learning, t is time, s is the state, a is the action, π is the actor (policy), Jπ is the objective function for actor learning, r is the reward function, P is the state transition function, γ (gamma) is the depreciation rate, and α (alpha) is the entropy adjustment coefficient in SAC.

[0051] FIG. 7 is a diagram for explaining the critic (230) of the present invention.

[0052] Referring to FIG. 7, in the process of selecting a material from the status information, the critic 230 can output the highest status value for a material having the highest difference between the first similarity and the second similarity for each material.

[0053] For example, the critic (230) can be trained to infer state values ​​by replacing only the materials with various types of materials while keeping the canvas, goal, and remaining number of times constant among the state information.

[0054] During the collage creation process, the actor (210) determines his / her behavior based on the given material, so any material can be provided at any time. However, if multiple materials are provided at the same time, the optimal material will exist depending on the state of the first canvas and the goal at that moment. When the critic (230) receives a canvas (C), a goal (I), and multiple materials (M), it can infer a state value using the first and second similarities calculated by the discriminator (240). For example, the critic (230) can select, from the multiple materials, the material that outputs the highest difference (state value) between the first similarity between the first canvas (C) and the goal (I) and the second similarity between the second canvas (C) and the goal (I).

[0055] FIG. 8 is a diagram for explaining the discriminator (240) of the present invention.

[0056] For example, the discriminator 240 can update the reward based on the similarity between the first canvas and the target, the second canvas, and the target. Specifically, the discriminator 240 can use the difference between the first similarity between the first canvas and the target and the second similarity between the second canvas and the target as the reward.

[0057] Referring to Figure 8, the discriminator (240) can collect pairs of canvases (first canvas / second canvas) generated during learning and their corresponding targets, and can learn using a regression method to classify the pair of target (I) and canvas (C) as 0 and the pair of target (I) and target (I) as 1. In other words, the discriminator (240) can output a value closer to 1 the closer the canvas is to the target, and can play a role in evaluating the similarity.

[0058] For example, if there is a first canvas and a second canvas generated by applying an action, the discriminator (240) can calculate the first and second similarities and calculate how much the second canvas has increased in similarity to the target compared to the first canvas. The reinforcement learning agent (200) can define this calculated value as a reward for that action. In addition, the discriminator (240) can continue learning as reinforcement learning progresses.

[0059] Therefore, as the performance of the reinforcement learning agent (200) increases, the performance of the discriminator (240) also increases, which has the advantage of providing a relative standard that is tailored to the performance of the reinforcement learning agent (200) rather than an absolute standard such as MSE. In addition, since it is trained to grasp the similarity with respect to the collage task itself, it is possible to evaluate the similarity in the context of the collage rather than simply comparing pixels.

[0060] When the user inputs a goal for a collage work, the reinforcement learning agent (200) observes state information based on the goal, determines an action for the collage, applies the action to the first canvas, and renders the second canvas. The collage generation process is repeated according to the remaining number of times, and the collage work corresponding to the goal can be output.

[0061] FIG. 9 is a diagram illustrating a collage generation process of a reinforcement learning agent according to an embodiment of the present invention.

[0062] When a target image is input to the reinforcement learning agent 200, the reinforcement learning agent 200 repeats a collage generation process of cutting and pasting materials onto a first canvas to generate a second canvas, and can provide a reconstructed image that is highly similar to the target image as a collage work. As an example, in the collage generation process for the target image of the number 3, materials are pasted sequentially from the first canvas with a white background to complete the collage work, but the first canvas is not limited to a white background.

[0063] 10 to 13 are diagrams illustrating an example of collage generation for a reinforcement learning agent according to an embodiment of the present invention.

[0064] In this embodiment of the present invention, the actor (210) and critic (230) are configured using a CNN (Convolutional Neural Network)-based ResNet-18 network, and a reward function is designed and trained in a way that derives the similarity between the target and canvas images using a network with a GAN (Generative Adversarial Network) discriminator (240) structure, using SAC as the reinforcement learning algorithm. As a result, as shown in Figure 10, a graph of image similarity (MSE) for each image dataset based on the training results can be seen.

[0065] For example, the target image during training can be randomly changed each time a work is completed, and the material image can be randomly changed each time a material is used. In addition, the material and target images ensure diversity in the image dataset (e.g., image net) that is brought in, and the reinforcement learning agent (200) is trained to select actions according to the given situation each time, as it is impossible to predict the next material or target image.

[0066] FIG. 10 shows that as the reinforcement learning agent of the present invention continues to learn, the image similarity (MSE criterion) between the target and the second canvas decreases.

[0067] Figure 11 shows the collage results output by the reinforcement learning agent of the present invention after learning target images classified into digit images (MNIST), flower images (Flowers), landscape images (Scene), and graphic images (ImageNet). In this case, the material images were provided as images with various textures and colors.

[0068] FIG. 12 shows the collage results obtained by inputting target images divided into flower images and landscape images into the reinforcement learning agent of the present invention, and then learning and outputting the collage results by limiting the types of material images provided in the collage generation process to material images similar to the target image.

[0069] FIG. 13 shows an example in which the reinforcement learning agent of the present invention is applied to a large-scale image in an expanded divisional execution method. For example, since a picture is completed by assembling parts, it is easier to perform a collage using the divisional method for high-resolution images. Here, the example picture shown in FIG. 13 has been worked on in more detail for complex parts. The trained reinforcement learning agent (200) does not generate a collage all at once, but generates it by making decisions at each stage, so other behavioral elements are relatively free to intervene in the overall collage generation process. Therefore, the reinforcement learning agent (200) is capable of individualization.

[0070] 14 and 15 are diagrams for comparing and explaining an example of collage generation by a reinforcement learning agent according to an embodiment of the present invention with an existing pixel-based methodology.

[0071] Figure 14 compares the image generation results of existing pixel-based methodologies (Gatys / Perceptual / AdaIN / AdaAttn / StyTR-2) with the reinforcement learning agent (200).

[0072] As shown in Figure 14, the pixel-based methodologies generated images that were unique but far from the actual collage. However, the reinforcement learning agent (200) of the present invention generated images that were similar to the actual collage.

[0073] FIG. 15 shows the results of comparing the image generation results of FIG. 14 through the large-scale language model CLIP and LPIPS, which is one of the image similarity measurement methods.

[0074] As shown in Figure 15, the comparison using CLIP examined the degree to which text such as "collage" and "human-made work" matched the resulting image. Similarity measurements were performed using CLIP trained on millions of text-image pairs. This allowed us to determine the degree to which CLIP agreed that the resulting image was an actual collage. Images generated by the reinforcement learning agent (200) of the present invention could also be numerically compared with images generated using a pixel-based methodology. Furthermore, by examining the degree to which the text (e.g., "red apple") for the target image used to obtain the resulting image matched the resulting image, we could also compare the degree to which the target content was distorted when generating the collage. The comparison results showed that the collage generated by the reinforcement learning agent (200) of the present invention was more similar to the actual collage than works generated using a pixel-based method, and better preserved the target content. Similarly, a comparison of the similarity (difference) between the target image and the generated result image using LPIPS shows that the collage work generated by the reinforcement learning agent (200) of the present invention obtained a higher similarity (lower difference).

[0075] In the following, the description of the same configuration as above will be omitted.

[0076] Referring again to FIG. 3, the learning method of the reinforcement learning agent learning device for automating the collage generation process includes a step of determining an action for the collage when state information including a canvas, a material, a goal, and a remaining number of times is input (S110), a step of applying the action to a first canvas to render a second canvas (S120), a step of updating a reward based on the similarity between the first canvas, the second canvas, and the goal (S130), and a step of learning the reinforcement learning agent by repeating steps S110 to S130 (S140).

[0077] In step S110, a first action element is selected for cutting out a piece of material, and a second action element is selected for pasting the cut out piece of material onto the canvas, where the first action element includes a virtual frame, the position of the virtual frame on the material, the horizontal and vertical lengths of the virtual frame, and the positions of points on each side of the virtual frame, and the second action element includes the piece of material, the position of the piece of material on the canvas, and the rotation angle of the piece of material.

[0078] In step S120, a mask corresponding to a virtual frame is generated based on a first behavioral element, material is cut out using the mask to generate pieces of material, and the pieces of material are pasted on the first canvas based on a second behavioral element to generate a second canvas.

[0079] In step S130, the difference between the first similarity between the first canvas and the target and the second similarity between the second canvas and the target can be used as a reward.

[0080] In the process of selecting materials from the state information before step S110, the highest state value may be output for a material having the highest difference between the first similarity and the second similarity for each material.

[0081] When the user inputs a goal for a collage work, the reinforcement learning agent (200) observes state information based on the goal, determines an action for the collage, applies the action to the first canvas, and renders the second canvas. The collage generation process is repeated according to the remaining number of times, and the collage work corresponding to the goal can be output.

[0082] A learning method according to an embodiment of the present invention may be embodied in the form of a recording medium containing computer-executable commands, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer, including volatile and nonvolatile media, removable and non-removable media. Also, a computer-readable medium may include computer storage media. A computer storage medium includes all volatile and non-volatile, removable and non-removable media embodied in any method or technology for storing information, such as computer-readable commands, data structures, program modules, or other data.

[0083] Although the apparatus and methods of the present invention have been described with reference to particular embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.

[0084] The above description of the present invention is for illustrative purposes only, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical spirit or essential characteristics of the present invention. Therefore, the above-described embodiments should be understood to be illustrative in all respects and not restrictive. For example, each component described as a single component may be implemented in a distributed form, and similarly, each component described as a distributed form may be implemented in a combined form.

[0085] The scope of the present invention is indicated by the claims that follow rather than by the above detailed description, and all modifications and variations that fall within the meaning and scope of the claims and their equivalents should be construed as being included within the scope of the present invention. [Explanation of symbols]

[0086] 100: Reinforcement learning agent learning device 110: Communication module 120:Memory 130: Processor 140: Database 200: Reinforcement learning agent 210: Actor 220: Renderer 230: Critic 240: Discriminator

Claims

1. In a reinforcement learning agent learning device for automating the collage generation process, A memory in which the reinforcement learning agent learning program is stored, and a processor for executing a program stored in the memory; The reinforcement learning agent learning program includes: (a) a step of determining an action for a collage when state information including a first canvas before material manipulation, a material, a goal, and a remaining number of times is input; (b) a step of applying the action to the first canvas and rendering a second canvas after material manipulation; (c) a step of updating a reward based on similarities between the first canvas, the second canvas, and the goal; and (d) a step of repeating steps (a) to (c) using the second canvas rendered in step (b) as the first canvas in a next iteration step and learning a reinforcement learning agent using the reward calculated in step (c).

2. Claim 1: The reinforcement learning agent learning program described is In the step (a), A first action element is selected to cut out a piece of material, and a second action element is selected to paste the cut out piece of material onto the canvas. the first behavioral element includes a virtual frame, a position of the virtual frame on the material, horizontal and vertical lengths of the virtual frame, and positions of points on each side of the virtual frame; A reinforcement learning agent learning device, wherein the second behavioral element includes a piece of material, a position of the piece of material on a canvas, and a rotation angle of the piece of material.

3. Claim 2: The reinforcement learning agent learning program described is In the step (b), a reinforcement learning agent learning device that generates a mask corresponding to the virtual frame based on the first behavior element, cuts out material using the mask to generate pieces of the material, and pastes the pieces of material onto the first canvas based on the second behavior element to generate the second canvas.

4. Claim 1: The reinforcement learning agent learning program described is In the step (c), A reinforcement learning agent learning device that utilizes a difference between a first similarity between the first canvas and a target and a second similarity between the second canvas and a target as the reward.

5. Claim 4: The reinforcement learning agent learning program In the process of selecting a material from the state information, A reinforcement learning agent learning device that outputs the highest state value for a material having the highest difference between the first similarity and the second similarity for each material.

6. Claim 1: The reinforcement learning agent described is A reinforcement learning agent learning device that, when a user inputs a goal for a collage work, observes state information based on the goal, determines an action for the collage, applies the action to the first canvas, and renders the second canvas by repeating the collage generation process according to the remaining number of times, and outputs a collage work that corresponds to the goal.

7. A learning method for a reinforcement learning agent learning device for automating a collage generation process, (a) determining an action to be taken on the collage when state information including a first canvas before material manipulation, material, goal, and remaining number of times is input; (b) applying the behavior to the first canvas to render a material-manipulated second canvas; (c) updating a reward based on the similarity between the first canvas and the second canvas and the goal; and (d) using the second canvas rendered in step (b) as the first canvas in a next iteration, and repeating steps (a) to (c) while training a reinforcement learning agent using the reward calculated in step (c).

8. Claim 7: At the step (a) described above, A first action element is selected to cut out a piece of material, and a second action element is selected to paste the cut out piece of material onto the canvas. the first behavioral element includes a virtual frame, a position of the virtual frame on the material, horizontal and vertical lengths of the virtual frame, and positions of points on each side of the virtual frame; A learning method for a reinforcement learning agent learning device, wherein the second behavioral element includes a piece of material, a position of the piece of material on a canvas, and a rotation angle of the piece of material.

9. In claim 8, At the step (b) of the description, A learning method for a reinforcement learning agent learning device, comprising: generating a mask corresponding to the virtual frame based on the first behavior element; cutting out material using the mask to generate pieces of the material; and pasting the pieces of material onto the first canvas based on the second behavior element to generate the second canvas.

10. Claim 7: At the step (c) described above, A learning method for a reinforcement learning agent learning device, wherein a difference between a first similarity between the first canvas and a target and a second similarity between the second canvas and the target is used as the reward.

11. Claim 10: In the process of selecting materials from the listed condition information, A learning method for a reinforcement learning agent learning device, which outputs the highest state value for a material having the highest difference between the first similarity and the second similarity for each material.

12. Claim 7: The reinforcement learning agent described is A learning method for a reinforcement learning agent learning device, which, when a goal of a collage work is input by a user, observes state information based on the goal, determines an action for the collage, applies the action to the first canvas, and renders the second canvas, repeating this collage generation process according to the remaining number of times, and outputs a collage work corresponding to the goal.

Images