Method and apparatus for controling robot arm using of hierachical reinforcement learning model, method and apparatus for providing hierachical reinforcement learning for robot arm control

Abstract

The present invention relates to the control of a robot arm by a robot arm control device comprising a learning model for controlling a robot arm, and may include the steps of: acquiring, respectively, a target image to be drawn by the robot arm, a virtual canvas image to reflect the drawing work of the robot arm for the target image, and a virtual pen nib image of the robot arm whose position changes as the drawing work is reflected in steps, which are virtual strokes, within the virtual canvas image; processing the target image to determine the amount of position change of the virtual pen nib image within the virtual canvas image when the target image is input to the learning model; and processing to output a joint angle change amount for driving the robot arm based on the amount of position change.

Description

Technology Field

[0001] The present invention relates to robot arm control technology and neural network-based reinforcement learning for the same. Background Technology

[0003] Due to advancements in robotics, various types of robot control technologies are being applied to the field of art.

[0004] Conventional robot control technology for drawing employs methods that utilize mathematical techniques to decompose a target image in order to obtain the sequential process for drawing a target image, or learns to mimic drawing process data generated directly by humans and manipulates the robot using inverse kinematics.

[0005] Because this method relies heavily on human design, its scope of application is not only limited but also incurs significant costs. Prior art literature

[0007] Registered Patent Publication No. 10-1358270 (Registration announced February 7, 2014) Registered Patent Publication No. 10-1266374 (Registration announced May 22, 2013) The problem to be solved

[0008] In an embodiment of the present invention, we propose a robot arm control technology that implements drawing using model-free reinforcement learning based on neural networks and computer vision deep learning.

[0009] In an embodiment of the present invention, we propose a hierarchical reinforcement learning technique applicable to the control of a drawing robot arm by separately performing learning on the position of the virtual pen nib of the robot arm and learning on the amount of change in joint angles.

[0010] The problems that the present invention aims to solve are not limited to those mentioned above, and other unmentioned problems to be solved will be clearly understood by those skilled in the art to which the present invention pertains from the descriptions below. means of solving the problem

[0012] According to an embodiment of the present invention, a hierarchical reinforcement learning device for controlling a robot arm may be provided, comprising: a storage unit storing a first learning model and a second learning model for controlling a robot arm; an acquisition unit acquiring, respectively, a target image to be drawn by the robot arm, a virtual canvas image to reflect the drawing work of the robot arm for the target image, and a virtual pen nib image of the robot arm whose position changes as the drawing work is reflected in steps, which are virtual strokes, within the virtual canvas image; and a processing unit that trains the first learning model so that the virtual canvas image corresponds to the target image based on the position of the virtual pen nib image within the virtual canvas image, and trains the second learning model so that the amount of change in the joint angle of the robot arm corresponds to the amount of change in the target position of the virtual pen nib image.

[0013] Here, the first learning model comprises: a commander that determines a position change amount for changing the position of the virtual pen nib image within the virtual canvas image; and a discriminator that determines a similarity change amount between the virtual canvas image and the target image, in which the virtual drawing operation is reflected in steps based on the position change amount, and determines a reward value according to the similarity change amount; wherein the commander can perform reinforcement learning based on the reward value.

[0014] In addition, the second learning model may include a stroker that receives the joint angle of the robot arm and the change in the target position as input and determines the change in the joint angle of the robot arm.

[0015] In addition, the processing unit determines the similarity between the change in position of the virtual pen nib changed by the change in joint angle and the change in target position, and the stroker can perform reinforcement learning based on a compensation value according to the similarity.

[0016] In addition, the above target position change amount can be updated whenever the above joint angle change amount is determined.

[0017] According to an embodiment of the present invention, a robot arm control device may be provided, comprising: a storage unit for storing a program for controlling a robot arm; an acquisition unit for acquiring, respectively, a target image to be drawn by the robot arm, a virtual canvas image to reflect the drawing work of the robot arm for the target image, and a virtual pen nib image of the robot arm whose position changes as the drawing work is reflected in steps, which are virtual strokes, within the virtual canvas image; and a processing unit that, when the target image is input into a pre-learned learning model within the program, processes to determine the amount of position change of the virtual pen nib image within the virtual canvas image in the learning model, and processes to output a joint angle change amount for driving the robot arm based on the amount of position change.

[0018] Here, the learning model may include a commander that determines the amount of change in position; and a stroker that outputs the amount of change in joint angle.

[0019] In addition, the commander may be trained to output a position change amount that satisfies the target position change amount of the robot arm when the target image is input, and the stroker may be trained to output a joint angle change amount of the robot arm when the position change amount and the joint angle of the robot arm are input.

[0020] In addition, the processing unit can measure the position information of the actual pen nib mounted on the robot arm when the robot arm is driven based on the change in the joint angle, and reflect the position information of the actual pen nib in the virtual pen nib image.

[0021] According to an embodiment of the present invention, a hierarchical reinforcement learning method for a robot arm control comprising a first learning model and a second learning model for the control of a robot arm can be provided, comprising the steps of: acquiring, respectively, a target image to be drawn by the robot arm, a virtual canvas image to reflect the drawing work of the robot arm for the target image, and a virtual pen nib image of the robot arm whose position changes as the drawing work is reflected in steps, which are virtual strokes, within the virtual canvas image; training the first learning model so that the virtual canvas image corresponds to the target image based on the position of the virtual pen nib image within the virtual canvas image; and training the second learning model so that the amount of change in the joint angle of the robot arm corresponds to the amount of change in the target position of the virtual pen nib image.

[0022] Here, the step of training the first learning model may include: a step of determining a position change amount as an action to change the position of the virtual pen nib image within the virtual canvas image; and a step of determining a similarity change amount between the virtual canvas image and the target image, in which the drawing work is reflected in step units based on the position change amount, and determining a compensation value according to the similarity change amount.

[0023] In addition, the step of determining the amount of change in position may include a step of performing reinforcement learning based on the reward value.

[0024] In addition, the step of training the second learning model may include the step of receiving the joint angle of the robot arm and the change in the target position as inputs and determining the change in the joint angle of the robot arm.

[0025] Additionally, the above-described hierarchical reinforcement learning method may further include: a step of determining the similarity between the change in position of the virtual pen nib changed by the change in joint angle and the change in target position; and a step of performing reinforcement learning based on a reward value according to the similarity.

[0026] In addition, the above target position change amount can be updated whenever the above joint angle change amount is determined.

[0027] According to an embodiment of the present invention, a method for controlling a robot arm using a hierarchical reinforcement learning model can be provided, comprising: a step of acquiring, respectively, a target image to be drawn by the robot arm, a virtual canvas image to reflect the drawing work of the robot arm for the target image, and a virtual pen nib image of the robot arm whose position changes as the drawing work is reflected in steps, which are virtual strokes, within the virtual canvas image; a step of processing to determine the amount of position change of the virtual pen nib image within the virtual canvas image when the target image is input to the learning model; and a step of processing to output a joint angle change amount for driving the robot arm based on the amount of position change.

[0028] Here, the learning model can be trained to output a position change amount that satisfies the target position change amount of the robot arm as an action when the target image is input, and can be trained to output a joint angle change amount of the robot arm when the position change amount and the joint angle of the robot arm are input.

[0029] Additionally, the robot arm control method may further include: a step of measuring position information of an actual pen nib mounted on the robot arm when the robot arm is driven based on the amount of change in the joint angle; and a step of reflecting the position information of the actual pen nib in the virtual pen nib image.

[0030] According to an embodiment of the present invention, a computer-readable recording medium storing a computer program, wherein the computer program includes instructions for a processor to perform a control method of a robot arm using a hierarchical reinforcement learning model, and the method may include the steps of: acquiring, respectively, a target image to be drawn by the robot arm, a virtual canvas image to reflect the drawing work of the robot arm for the target image, and a virtual pen nib image of the robot arm whose position changes as the drawing work is reflected in steps, which are virtual strokes, within the virtual canvas image; processing such that when the target image is input to the hierarchical reinforcement learning model, the hierarchical reinforcement learning model determines the amount of position change of the virtual pen nib image within the virtual canvas image; and processing such that the amount of position change of the robot arm is output to the joint angle change amount for driving the robot arm based on the amount of position change.

[0031] According to an embodiment of the present invention, a computer program stored on a computer-readable recording medium, wherein the computer program includes instructions for a processor to perform a hierarchical reinforcement learning method for controlling a robot arm, and the method may include the steps of: acquiring, respectively, a target image to be drawn by the robot arm, a virtual canvas image to reflect the drawing work of the robot arm for the target image, and a virtual pen nib image of the robot arm whose position changes as the drawing work is reflected in steps, which are virtual strokes, within the virtual canvas image; training a first learning model so that the virtual canvas image corresponds to the target image based on the position of the virtual pen nib image within the virtual canvas image; and training a second learning model so that the amount of change in the joint angle of the robot arm corresponds to the amount of change in the target position of the virtual pen nib image. Effects of the invention

[0033] According to an embodiment of the present invention, a hierarchical reinforcement learning model is constructed by separately learning the position of a virtual pen nib of a robot arm and the amount of change in joint angles, and by controlling the robot arm through the constructed hierarchical reinforcement learning model to reach a target drawing, a result for a natural target drawing can be obtained at low cost without prior design based on knowledge. Brief explanation of the drawing

[0035] FIG. 1 is a block diagram showing a device (100) for controlling a robot arm or learning a learning model for controlling a robot arm according to an embodiment of the present invention. FIG. 2 is a diagram showing the configuration of the storage unit (120) of FIG. 1 in detail, and is a conceptual diagram for explaining the learning process. FIG. 3 is a diagram for explaining the configuration and learning process of the first learning model (124) of FIG. 2. Figure 4 is a detailed conceptual diagram of Figure 3. FIG. 5 is a diagram for explaining the configuration and learning process of the second learning model (126) of FIG. 2. Figure 6 is a detailed conceptual diagram of Figure 5. FIG. 7 is a diagram showing the configuration of the storage unit (120) of FIG. 1 in detail, and is a conceptual diagram for explaining the process of execution after learning. FIG. 8 is a diagram illustrating the configuration of the robot arm control learning model (124 / 126) of FIG. 7 and the robot arm control process through it. Figure 9 is a detailed conceptual diagram of Figure 8. Specific details for implementing the invention

[0036] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the scope of the present invention is defined only by the claims.

[0037] In describing the embodiments of the present invention, specific descriptions of known functions or configurations will be omitted unless actually necessary for describing the embodiments of the present invention. Furthermore, the terms described below are defined in consideration of the functions in the embodiments of the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification.

[0038] In an embodiment of the present invention, drawing is implemented using model-free reinforcement learning based on a neural network and computer vision deep learning, and a hierarchical reinforcement learning technique is proposed that can be applied to the control of a robot arm for drawing by separately performing learning on the position of a virtual pen nib of the robot arm and learning on the amount of change in joint angles.

[0040] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0041] FIG. 1 is a block diagram showing a device (100) for controlling a robot arm or learning a learning model for controlling a robot arm according to an embodiment of the present invention.

[0042] That is, the configuration of the embodiment of FIG. 1 can apply a hierarchical reinforcement learning function for robot arm control and a robot arm control function through such hierarchical reinforcement learning in parallel.

[0043] First, a device (100) that performs a hierarchical reinforcement learning function for robot arm control will be described in detail with reference to FIGS. 1 to 6.

[0044] As illustrated in FIG. 1, a hierarchical reinforcement learning device (100) according to an embodiment of the present invention may include an acquisition unit (110), a storage unit (120), and a processing unit (130).

[0045] First, the acquisition unit (110) can acquire an arbitrary target image that the robot arm must draw, a virtual canvas image to which the drawing work of the robot arm for the target image is reflected, and a virtual pen nib image of the robot arm whose position changes as the drawing work is reflected in steps, which are virtual strokes, within the virtual canvas image.

[0046] As shown in FIG. 2, the storage unit (120) of the hierarchical reinforcement learning device (100) may store a robot arm control program (122) and information necessary for the execution of the robot arm control program (122), and the robot arm control program (122) may include a first learning model (124) and a second learning model (126) for robot arm control. As described below, the first learning model (124) in the hierarchical reinforcement learning device (100) may include a commander (124a) and a discriminator (124b), and the second learning model (126) may include a stroker (126a).

[0047] The processing unit (130) can train a first learning model (124) based on the position of the virtual pen nib image within the virtual canvas image so that the virtual canvas image corresponds to (reaches) the target image, and train a second learning model (126) so that the amount of change in the joint angle of the robot arm corresponds to the amount of change in the target position of the virtual pen nib image.

[0048] FIG. 3 is a diagram for explaining the configuration and learning process of the first learning model (124) of FIG. 2.

[0049] As illustrated in FIG. 3, the first learning model (124) may include a commander (124a) and a discriminator (124b).

[0050] The commander (124a) can determine the amount of position change to change the position of the virtual pen nib image within the virtual canvas image.

[0051] The discriminator (124b) can determine the amount of change in similarity between the virtual canvas image and the target image, to which the drawing work of the robot arm is reflected in step units, based on the amount of change in position of the commander (124a), and determine a compensation value according to the amount of change in similarity. For example, the discriminator (124b) can determine a positive (+) compensation value if the amount of change in similarity is greater than or equal to a threshold, and a negative (-) compensation value if the amount of change in similarity is less than the threshold.

[0052] At this time, the commander (124a) can perform reinforcement learning based on the reward value of this discriminator (124b).

[0053] Accordingly, the commander (124a) can be trained to receive a high reward value, and the discriminator (124b) can also be trained at the same time.

[0054] The first learning model (124) of FIG. 3 will be explained in more detail with reference to FIG. 4.

[0055] First, the target image (g c ), when the virtual canvas image (V) and the virtual pen nib image (pos) are input through the acquisition unit (110), the processing unit (130) determines that the virtual canvas image (V) is a target image (g) based on the position of the virtual pen nib image (pos) within the virtual canvas image (V). c The first learning model (124) can be trained to reach )

[0056] Specifically, the commander (124a) in the first learning model (124) changes the position of the virtual pen nib image (pos) within the virtual canvas image (V) using a position change amount (a c ) can be determined, and accordingly, the first learning model (124) performs drawing work in steps to change the virtual canvas image (V t ->V t+1 ) and the modified virtual pen nib image (pos t ->pos t+1 It can reflect ).

[0057] Accordingly, the discriminator (124b) is a change in position amount (a c A virtual canvas image (V) in which drawing work is reflected step by step based on ) t ->V t+1 ) and target image (g c Determine the amount of change in similarity between ), and the compensation value (r) based on the determined amount of change in similarity c ) can be determined and provided to the commander (124a).

[0058] The commander (124a) is the reward value (r) of this discriminator (124b). c Reinforcement learning can be performed based on ).

[0059] FIG. 5 is a diagram for explaining the configuration and learning process of the second learning model (126) of FIG. 2.

[0060] The second learning model (126) may include a stroker (126a) that receives the joint angle of the robot arm and the change in target position as input and determines the change in the joint angle of the robot arm. This stroker (126a) can perform learning independently in an environment independent of the commander (124a) described above.

[0061] The second learning model (126) of FIG. 5 will be explained in more detail with reference to FIG. 6.

[0062] First, the joint angle (s) of the robot arm with the stroker (126a) s) and target position change amount (g s When ) is input, the stroker (126a) [is] this joint angle (s s ) and target position change amount (g s Based on ), the amount of change in the joint angle of the robot arm (a s ) can be determined.

[0063] Accordingly, the processing unit (130) [requires] this amount of change in joint angle (a s The amount of change in the position of the virtual nib changed by ) and the amount of change in the target position (g s The similarity between ) can be determined, and the stroker (126a) has a compensation value (r) according to the similarity of the processing unit (130). s Reinforcement learning can be performed based on ).

[0064] At this time, the aforementioned target position change amount (g s ) is the amount of change in joint angle (a s It can be updated whenever ) is determined.

[0066] Meanwhile, a robot arm control device (100) using hierarchical reinforcement learning will be described in detail with reference to FIGS. 1, FIGS. 7, FIGS. 8 and FIGS. 9.

[0067] As illustrated in FIG. 1, the robot arm control device (100) may include an acquisition unit (110), a storage unit (120), and an acquisition unit (130).

[0068] The acquisition unit (110) can acquire an arbitrary target image that the robot arm must draw, a virtual canvas image in which the drawing work of the robot arm for the target image is reflected, and a virtual pen nib image of the robot arm in which the position is changed as the virtual drawing work is reflected in steps, which are virtual strokes, within the virtual canvas image.

[0069] As shown in FIG. 7, the storage unit (120) of the hierarchical reinforcement learning device (100) may store a robot arm control program (122) and information necessary for the execution of the robot arm control program (122), and the robot arm control program (122) may include a robot arm control learning model (124 / 126) for robot arm control. As described below, the learning model (124 / 126) in the robot arm control device (100) may include a commander (124a) and a stroker (126a).

[0070] When a target image is input into a robot arm control learning model (124 / 126) within a program, the processing unit (130) processes to determine the amount of change in position of a virtual pen nib image within a virtual canvas image in the robot arm control learning model (124 / 126), and processes to output a change in joint angle for driving the robot arm based on the amount of change in position.

[0071] FIG. 8 is a diagram illustrating the configuration of the robot arm control learning model (124 / 126) of FIG. 7 and the robot arm control process through it.

[0072] The robot arm control learning model (124 / 126) of FIG. 8 may include a commander (124a) and a stroker (126a). That is, in the embodiment of the present invention, a single learning model (124 / 126) is implemented by combining the commander (124a) and the stroker (126a) that were independently learned in FIG. 1 to FIG. 6, respectively, and through this, the robot arm control device (100) can complete the target drawing by accurately reflecting the position of the pen nib and the joint angle of the robot arm.

[0073] As illustrated in FIG. 8, the commander (124a) can determine the amount of change in the position of the virtual pen tip when a target image is input.

[0074] The stroker (126a) can output a change in the joint angle of the robot arm based on the change in position determined by the commander (124a).

[0075] FIG. 9 is a diagram for explaining in more detail the configuration and learning process of the robot arm control learning model (124 / 126) of FIG. 8.

[0076] First, the target image (g c ), when the virtual canvas image (V) and virtual pen nib image (pos) are input through the acquisition unit (110), the processing unit (130) [determines] the target position change amount (g) of the robot arm through the commander (124a) of the robot arm control learning model (124 / 126). s The amount of change in position of a virtual nib satisfying ) (a c It can be processed to output ).

[0077] Subsequently, the processing unit (130) [determines] the amount of change in the joint angle (a) of the robot arm through the stroker (126a) of the robot arm control learning model (124 / 126). s It can be processed to output ). Specifically, the processing unit (130) can process the amount of position change of the virtual pen nib (a c ) and robot arm joint angles(s s Input ) into the stroker (126a) to change the joint angle of the robot arm (a s It can be processed so that ) is output.

[0078] This amount of change in joint angle (a s When this is reflected in the robot arm control device (100), the processing unit (130) can generate a control value for driving the actual robot arm. This control value can be applied to a robot arm driving unit (not shown), and the robot arm driving unit can generate a joint angle change amount (a s It can be driven at a certain angle according to the control value based on ).

[0079] When the robot arm is driven by the robot arm drive unit, the processing unit (130) can measure the position information of the actual pen nib mounted on the robot arm. This position information measurement can be performed independently by the robot arm drive unit.

[0080] When the position information of the actual pen nib is measured, the processing unit (130) can reflect the measured position information in the virtual pen nib image (pos) within the virtual canvas image (V).

[0081] Afterwards, the commander (124a) again changes the position amount (a c ) determines, and the stroker (126a) determines the amount of change in joint angle (a s While repeating the process of determining ), the target image (g c ...will be completed.

[0083] According to the embodiment of the present invention as described above, a hierarchical reinforcement learning model is constructed by separately learning the position of the virtual pen nib of the robot arm and the amount of change in the joint angle, and by controlling the robot arm through the constructed hierarchical reinforcement learning model to reach the target drawing, a result of a natural target drawing can be obtained at low cost without prior design based on knowledge. In addition, in the embodiment of the present invention, since actions are determined while drawing on a virtual canvas and drawing on the actual drawing through the robot arm, there is no need to check the actual drawing, so there is an advantage that equipment such as a camera is not required.

[0084] Meanwhile, combinations of each block of the attached block diagram and each step of the flowchart may be executed by computer program instructions. Since these computer program instructions can be loaded into the processor of a general-purpose computer, a specialized computer, or other programmable data processing equipment, the instructions executed through the processor of the computer or other programmable data processing equipment create a means to perform the functions described in each block of the block diagram.

[0085] Since these computer program instructions may be stored in a computer-available or computer-readable recording medium (or memory), etc., which can be directed toward a computer or other programmable data processing equipment to implement a function in a specific way, the instructions stored in the computer-available or computer-readable recording medium (or memory) may also be used to produce a manufactured item containing instruction means that perform the function described in each block of the block diagram.

[0086] And, since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that perform a series of operation steps on a computer or other programmable data processing equipment to create a process executed by a computer and perform the computer or other programmable data processing equipment may also provide steps for executing the functions described in each block of the block diagram.

[0087] Additionally, each block may represent a module, segment, or part of code containing at least one executable instruction for executing a specified logical function(s). Also, it should be noted that in some alternative embodiments, the functions mentioned in the blocks may occur out of order. For example, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may be executed in reverse order according to the corresponding function. Explanation of the symbols

[0089] 100: Robot arm control device (Hierarchical reinforcement learning device for robot arm control) 110: Acquisition Department 120: Storage section 122: Robot Arm Control Program 124: First learning model 124a: Commander 124b: Discriminator 126: Second learning model 126a: Stroker 130: Processing unit

Claims

Claim 1 delete Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 delete Claim 6 A robot arm control device comprising: a storage unit for storing a program for controlling a robot arm; an acquisition unit for acquiring, respectively, a target image to be drawn by the robot arm, a virtual canvas image to reflect the drawing work of the robot arm for the target image, and a virtual pen nib image of the robot arm whose position changes as the drawing work is reflected in steps, which are virtual strokes, within the virtual canvas image; and a processing unit that, when the target image is input into a pre-learned learning model within the program, processes to determine the amount of position change of the virtual pen nib image within the virtual canvas image in the learning model, and processes to output a joint angle change amount for driving the robot arm based on the amount of position change. Claim 7 In claim 6, the learning model comprises a commander that determines the amount of position change; and a stroker that outputs the amount of joint angle change; a robot arm control device. Claim 8 A robot arm control device according to claim 7, wherein the commander is learned to output a position change amount satisfying a target position change amount of the robot arm when the target image is input, and the stroker is learned to output a joint angle change amount of the robot arm when the position change amount and the joint angle of the robot arm are input. Claim 9 In claim 6, the processing unit measures position information of an actual pen nib mounted on the robot arm when the robot arm is driven based on the amount of change in the joint angle, and reflects the position information of the actual pen nib in the virtual pen nib image, thereby forming a robot arm control device. Claim 10 A hierarchical reinforcement learning method for a hierarchical reinforcement learning device comprising a first learning model and a second learning model for controlling a robot arm, comprising: a step of acquiring, respectively, a target image to be drawn by the robot arm, a virtual canvas image to reflect the drawing work of the robot arm for the target image, and a virtual pen nib image of the robot arm whose position changes as the drawing work is reflected in steps, which are virtual strokes, within the virtual canvas image; a step of training the first learning model so that the virtual canvas image corresponds to the target image based on the position of the virtual pen nib image within the virtual canvas image; and a step of training the second learning model so that the amount of change in the joint angle of the robot arm corresponds to the amount of change in the target position of the virtual pen nib image; wherein the step of training the first learning model includes a step of determining the amount of change in position to change the position of the virtual pen nib image within the virtual canvas image as an action. A hierarchical reinforcement learning method for robot arm control comprising: a step of determining the amount of change in similarity between the virtual canvas image and the target image, in which the drawing work is reflected in the step unit based on the amount of change in position, and determining a compensation value according to the amount of change in similarity. Claim 11 delete Claim 12 A hierarchical reinforcement learning method for robot arm control according to claim 10, wherein the step of determining the position change amount includes the step of performing reinforcement learning based on the reward value. Claim 13 A hierarchical reinforcement learning method for robot arm control according to claim 10, wherein the step of training the second learning model includes the step of receiving the joint angle of the robot arm and the change amount of the target position and determining the change amount of the joint angle of the robot arm. Claim 14 A hierarchical reinforcement learning method for robot arm control, further comprising: a step of determining the similarity between the change in position of the virtual pen tip changed by the change in joint angle and the change in position of the target position in claim 13; and a step of performing reinforcement learning based on a compensation value according to the similarity. Claim 15 A hierarchical reinforcement learning method for robot arm control in which the target position change amount is updated whenever the joint angle change amount is determined, according to claim 14. Claim 16 A method for controlling a robot arm using a hierarchical reinforcement learning model, comprising a learning model for controlling a robot arm, wherein the method comprises: a step of acquiring, respectively, a target image to be drawn by the robot arm, a virtual canvas image to reflect the drawing work of the robot arm for the target image, and a virtual pen nib image of the robot arm whose position changes as the drawing work is reflected in steps, which are virtual strokes, within the virtual canvas image; a step of processing such that when the target image is input to the learning model, the learning model determines the amount of position change of the virtual pen nib image within the virtual canvas image; and a step of processing such that the learning model outputs a joint angle change amount for driving the robot arm based on the amount of position change; wherein the learning model is learned to output a position change amount satisfying the target position change amount of the robot arm as an action when the target image is input, and is learned to output a joint angle change amount of the robot arm when the position change amount and the joint angle of the robot arm are input. Claim 17 delete Claim 18 A method for controlling a robot arm using a hierarchical reinforcement learning model, further comprising: a step of measuring position information of an actual pen nib mounted on the robot arm when the robot arm is driven based on the change in joint angle of the above-mentioned joint angle; and a step of reflecting the position information of the actual pen nib in the virtual pen nib image. Claim 19 A computer-readable recording medium storing a computer program, wherein the computer program includes instructions for a processor to perform a control method of a robot arm using a hierarchical reinforcement learning model, and the method comprises: a step of acquiring, respectively, a target image to be drawn by the robot arm, a virtual canvas image to reflect the drawing work of the robot arm for the target image, and a virtual pen nib image of the robot arm whose position changes as the drawing work is reflected in steps, which are virtual strokes, within the virtual canvas image; a step of processing such that when the target image is input to the hierarchical reinforcement learning model, the hierarchical reinforcement learning model determines the amount of position change of the virtual pen nib image within the virtual canvas image; and a step of processing such that a joint angle change amount for driving the robot arm is output based on the amount of position change; wherein the hierarchical reinforcement learning model is learned to output a position change amount satisfying the target position change amount of the robot arm as an action when the target image is input, and is learned to output a joint angle change amount of the robot arm when the position change amount and the joint angle of the robot arm are input. Claim 20 A computer program stored on a computer-readable recording medium, wherein the computer program includes instructions for a processor to perform a hierarchical reinforcement learning method for controlling a robot arm, and the method comprises: a step of acquiring, respectively, a target image to be drawn by the robot arm, a virtual canvas image to reflect the drawing work of the robot arm for the target image, and a virtual pen nib image of the robot arm whose position changes as the drawing work is reflected in steps, which are virtual strokes, within the virtual canvas image; a step of training a first learning model so that the virtual canvas image corresponds to the target image based on the position of the virtual pen nib image within the virtual canvas image; and a step of training a second learning model so that the amount of change in the joint angle of the robot arm corresponds to the amount of change in the target position of the virtual pen nib image; wherein the step of training the first learning model includes a step of determining as an action the amount of change in position to change the position of the virtual pen nib image within the virtual canvas image. A computer program stored in a recording medium comprising: a step of determining the amount of change in similarity between the virtual canvas image and the target image, in which the virtual drawing work is reflected in the step unit based on the amount of change in position, and determining a compensation value according to the amount of change in similarity.

Images