Control device, control method and program
The control device enhances robot behavior precision by integrating rough and corrected behavior stages, addressing the limitations of existing reinforcement learning methods in robot behavior determination.
Patent Information
- Application Number
- JP2023076716
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-08
- Publication Date
- 2024-11-20
- Estimated Expiration
- 2043-05-08
AI Technical Summary
Existing reinforcement learning methods for robot behavior determination focus on local image parts, neglecting the influence of overall state changes, leading to poor sample efficiency and precision in robot behavior decision-making.
A control device that divides the action decision into two stages: determining a rough behavior from the entire image and then correcting it using a cropped local image, integrating the two processes to enhance learning efficiency and precision.
Improves the precision of robot behavior determination by reflecting overall state changes and enhances learning efficiency through coordinated learning of pixel importance and behavior calculation.
Smart Images

Figure 2024161737000001_ABST
Abstract
Description
[Technical field]
[0001] An embodiment of the present invention relates to a control device, a control method, and a program. [Background technology]
[0002] In picking tasks using a robot (one example of a control target), a technology has been proposed that uses reinforcement learning to learn a strategy for determining the robot's behavior from images. With this technology, the success or failure of picking can basically be determined only from the images of the end effector and the surrounding area of the grasped object (the object to be picked). Therefore, it is sufficient to focus on only localized parts of the image as a strategy.
[0003] However, in such a technique, the behavior of the controlled object is determined only from the peripheral images of the most important pixels, so it is difficult to reflect the influence of the overall state, for example, the change in the camera angle, on the behavior. [Prior art documents] [Non-patent literature]
[0004] [Non-Patent Document 1] James, S., & Davison, AJ, “Q-attention: Enabling efficient learning for vision-based robotic manipulation,” IEEE Robotics and Automation Letters, 7(2), 1612-1619, (2022). Summary of the Invention [Problem to be solved by the invention]
[0005] An object of the present invention is to provide a control device, a control method, and a program that can determine the behavior of a controlled object with higher accuracy. [Means for solving the problem]
[0006] The control device of the embodiment includes an acquisition unit, a first action calculation unit, an observation processing unit, a second action calculation unit, a determination unit, and a target control unit. The acquisition unit acquires first observation information used to determine the action of the control target. The first action calculation unit calculates a first action for the first observation information using a first model trained to input the first observation information and output the first action of the control target. The observation processing unit processes the first observation information based on the first action and outputs second observation information. The second action calculation unit calculates a second action for the second observation information using a second model trained to input the second observation information and output the second action of the control target. The determination unit determines the action of the control target based on the first action and the second action. The target control unit controls the control target to take the determined action. [Brief description of the drawings]
[0007] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a control system according to an embodiment. [Diagram 2] Block diagram of a control device according to an embodiment. [Diagram 3] FIG. 11 is a diagram showing an example of processing of observation information. [Figure 4] 4 is a flowchart of a control process according to the embodiment. [Diagram 5] 4 is a flowchart of a learning process in the embodiment. [Figure 6] FIG. 11 is a block diagram of a control device according to a modified example. [Figure 7] FIG. 2 is a hardware configuration diagram of a control device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0008] A preferred embodiment of a control device according to the present invention will be described in detail below with reference to the accompanying drawings. In the following, a control system including a control device for controlling a robot (picking robot) that grasps an object (item) will be described as an example. The controlled object is not limited to a robot, and may be any object.
[0009] As described above, a technology has been proposed that uses reinforcement learning to learn a strategy for determining a robot's behavior from an image. For example, a technology has been proposed in which the importance of each pixel in an image is first calculated, the image is cut out around the most important pixel, and the behavior of the picking robot is determined from the surrounding image. Even with this technology, since the behavior is determined only from the surrounding image of the most important pixel, it is difficult to reflect the influence of changes in the overall state in the behavior determination.
[0010] In addition, in the above-mentioned technology, the part that calculates the importance of each pixel and the part that calculates the behavior of the picking robot from the surrounding images are trained independently during learning, which generally results in poor sample efficiency.
[0011] In the following embodiment, by appropriately modifying the above two parts, it becomes possible to link them together for learning, thereby improving learning efficiency.
[0012] Specifically, the decision on an action using a strategy is divided into the following two stages. In the first stage, a rough action is decided instead of the importance of pixels. Hereinafter, the rough action is called action AA (first action). In the second stage, the image is cut out according to action AA, and a corrected action is decided from the cut-out image. Hereinafter, the corrected action is called action AB (second action).
[0013] This allows the entire information to be reflected in the action AA. The output of the first stage (action AA) and the output of the second stage (action AB) are equivalent in terms of the value of the action. Therefore, during learning, the action value can be calculated in a common form for both, improving the sample efficiency of learning.
[0014] First, we will briefly explain reinforcement learning, which is used to learn the control of robots and the like.
[0015] Reinforcement learning is a method to learn the input state s at a certain time t. t Action from a tThis is a method of learning a policy for determining the state s t corresponds to the control information of the action a t The policy corresponds to, for example, the probability distribution π(a t |s t ) In the following, a policy represented by a probability distribution π may be expressed as π.
[0016] The observation information includes, for example, an image including the object to be grasped captured by an imaging device such as a camera, information on the internal state of the robot, and information indicating auxiliary states (such as the current control time.) The control information is information for controlling the robot to be controlled, and is, for example, information indicating the coordinates and rotation angle of an end effector equipped on the robot.
[0017] The end effector may be referred to as a hand, a manipulator, etc. The end effector may be configured to include, for example, at least one of a gripper that grasps an object to be grasped by pinching it, and a suction pad that holds the object to be grasped by suction.
[0018] action a t can be expressed in any form. For a robot, action a t may be represented as eight-dimensional information including, for example, the following elements: - 3D information indicating position coordinates (x, y, z coordinates, etc.) - Four-dimensional information indicating rotation (quaternion) One-dimensional information showing gripper opening and closing
[0019] The policy is learned, for example, by a neural network model (hereinafter, sometimes simply referred to as a neural network) that outputs a probability value or a parameter of a probability model. In other words, the policy is represented by a model such as a neural network. Then, learning of the policy is realized by learning this model.
[0020] For example, reinforcement learning uses a policy π(a t |s t ) is the purpose.
number
[0021] Here, r(s t ,a t ) is the state s t Action a t is the reward obtained at time t as a result of performing the above. γ is the discount rate, a parameter between 0 and 1 that adjusts how much consideration is given to rewards in the distant future when deciding on actions. γ also plays the role of regularization to stabilize learning.
[0022] There are various algorithms for reinforcement learning. Below, we use the action value function Q(s t ,a t We will explain an example of using an algorithm that indirectly learns a policy through learning the action value function Q(s t ,a t ) is the state s t Action a t It is an estimate of the discounted cumulative reward obtained by acting according to the current policy after taking s. In TD learning (Temporal Difference Learning), the action value function Q(s t ,a t ) is learned using the update formula expressed by the following formula (2), where α represents the learning rate.
number
[0023] In general, the term in equation (2) expressed by the following equation (3) is difficult to calculate, so π(a|s t+1 ) is the action value function Q(s t+1 ,a) is used.
number
[0024] Action value function Q(s t ,a t ) is trained using models such as linear models and neural networks. In other words, the action value function Q(s t ,a t ) is expressed by models such as linear models and neural networks. By using the action value function, policy learning is realized by maximizing the action value function.
[0025] The following describes problems with conventional reinforcement learning. When images are mainly used as observed information, the policy and action-value function need to be calculated based on features extracted from the high-dimensional input of images.
[0026] In tasks such as picking an object (goods), the image that is important for calculating the behavior is often only a portion of the image (for example, an area including the object to be grasped and its surroundings). For this reason, by using an image that is a cut-out portion of the entire image (hereinafter, local image) rather than the entire image (hereinafter, entire image), control with higher precision is possible. In addition, since local images are less susceptible to the influence of the surrounding environment (background, etc.), high robustness can be expected. On the other hand, there are many elements that can be determined only from the entire image. For example, if the angle of the camera changes, the effect of the change in angle cannot be calculated in the local image. Therefore, it is preferable that the behavior is determined from both the entire image and the local image.
[0027] Next, the configuration of the control system of this embodiment will be described. Fig. 1 is a diagram showing an example of the configuration of a control system 10 of this embodiment. As shown in Fig. 1, the control system 10 includes a control device 100, a picking robot 200, and a camera 250.
[0028] The control device 100 controls the behavior (operation) of the picking robot 200. The picking robot 200 is a robot for picking (grasping) objects 210a and 210b, which are objects to be grasped. The camera 250 is an imaging device that captures images including the objects 210a and 210b. The images captured by the camera 250 may be in any format, such as an RGB image, a depth image, and a point cloud.
[0029] There may be one or more cameras 250. For example, two or more cameras 250 may capture images of the objects 210a and 210b from different angles.
[0030] The control device 100 inputs observation information and outputs control information for controlling the behavior of the picking robot 200. The observation information includes, for example, an image from the camera 250, information on the internal state of the picking robot 200 (joint angles, positions, etc.), and information indicating auxiliary states (current control time, etc.). The control information of the picking robot 200 includes, for example, the coordinates and rotation angle of an end effector provided to the picking robot 200.
[0031] Furthermore, the control device 100 learns a policy for outputting optimal control information (control information for taking an action) for gripping the object 210. The policy may be learned by operating the actual picking robot 200, or may be learned in a simulation environment.
[0032] 2 is a block diagram showing an example of the configuration of the control device 100. The control device 100 learns a policy such that a general action is determined from a whole image, a local image is cut out according to the determined action, and the action is corrected from the local image.
[0033] As shown in FIG. 2, the control device 100 includes a memory unit 151, a display unit 152, an acquisition unit 101, a memory control unit 102, an output control unit 103, an estimation unit 110, a learning unit 120, and a target control unit 130.
[0034] The storage unit 151 stores various information used by the control device 100. For example, the storage unit 151 stores the observation information acquired by the acquisition unit 101. The storage unit 151 can be configured with any commonly used storage medium such as a flash memory, a memory card, a RAM (Random Access Memory), a HDD (Hard Disk Drive), and an optical disk.
[0035] The display unit 152 is a display device for displaying various information used in the control device 100, and is realized by, for example, a liquid crystal display. For example, the display unit 152 displays the estimation result (determined action, etc.) by the estimation unit 110 under the control of the output control unit 103.
[0036] The acquisition unit 101 acquires various information used by the control device 100. For example, the acquisition unit 101 acquires observation information used to determine the behavior of the picking robot 200. In the following, the time at which the picking robot 200 is controlled is represented as a control time t. Also, the observation information output from the picking robot 200 at the control time t is represented as s t and the action determined at control time t is a t The control time t may be determined in any unit. For example, the control time t may be determined for each control cycle of the picking robot 200, or may be determined as a unit including a plurality of control cycles.
[0037] In addition, in the following, the observation information used when determining the behavior of the picking robot 200 using the learned policy (model) and when learning the model may be referred to as observation information IA (first observation information).
[0038] The acquisition unit 101 further acquires a past action a determined at the control time t-1. t-1 Reward obtained by r t-1may be acquired from the picking robot 200. The reward may be calculated by the picking robot 200 or may be set by a user or the like. The acquisition unit 101 uses the storage control unit 102 to store the acquired various information in the storage unit 151 and to obtain the observation information s t to the behavior calculation unit 111.
[0039] The acquiring unit 101 may acquire parameters necessary for learning, which are input by a user, etc. The input of information by a user, etc. may be performed by an input device, such as a keyboard and a pointing device, such as a mouse.
[0040] The storage control unit 102 controls the storage of various types of information in the storage unit 151. For example, the storage control unit 102 stores observation information in the storage unit 151 in accordance with an instruction from the acquisition unit 101.
[0041] The output control unit 103 controls the output of various information used in the control device 100. For example, the output control unit 103 may output (display) on the display unit 152 at least one of the processed observation information and the estimated (determined) behavior (behavior AB).
[0042] The estimation unit 110 corresponds to a function of estimating (determining) an action using a learned policy (model), and the learning unit 120 corresponds to a function of learning a policy (model).
[0043] First, a description will be given of the estimation unit 110. The estimation unit 110 includes a behavior calculation unit 111 (first behavior calculation unit), an observation processing unit 112, a behavior calculation unit 113 (second behavior calculation unit), and a determination unit 114.
[0044] The behavior calculation unit 111 calculates an action AA, which is a general behavior of the picking robot 200, based on the observation information IA. The behavior calculation unit 111 corresponds to a function for determining the above-mentioned first stage behavior.
[0045] For example, the action calculation unit 111 calculates the action AA using a learned policy PA (first model) used to calculate the action AA. For example, the policy PA currently obtained by learning is calculated by dividing the policy PA by a probability distribution π 1 (a 1 t |s t In this case, the behavior calculation unit 111 calculates the obtained observation information s t Using the probability distribution π 1 (a 1 t |s t )Act according to a 1 t For example, the behavior calculation unit 111 calculates a probability distribution π 1 (a 1 t |s t ) with the probability of one action a 1 t Action a 1 t corresponds roughly to Action AA.
[0046] The observation processing unit 112 processes the observation information IA based on the behavior AA. For example, if the observation information IA is an image, the observation processing unit 112 performs at least one of image cropping, rotation, and perspective transformation. Hereinafter, the processed observation information may be referred to as observation information IB (second observation information).
[0047] As the action AA and the observation information IA at the control time t, the action a 1 t and observational information t In this case, the observation processing unit 112 inputs the action a 1 t and the observation information s t The processed observation information IB is obtained by processing the observation information s 2 t =f(s t ,a 1 t ) to output.
[0048] 3 is a diagram showing an example of processing of observation information. Image 301 corresponds to observation information IA before processing. Image 302 corresponds to observation information IB after processing. Image 302 corresponds to an image obtained by, for example, cutting out a part of image 301 and rotating it.
[0049] Any method may be used to process the observation information based on the behavior, but for example, the following method may be applied: The following example is an example in which the observation information is an image. - Coordinates in the image that correspond to the position coordinates included in the action are found, and a portion of the area around the found coordinates is cropped. Rotate or perspective transform the image according to the rotation information included in the action. For example, rotate or perspective transform the image so that the angle with respect to the end effector rotated according to the rotation value included in the action is constant.
[0050] The correspondence between the position coordinates included in the action and the coordinates in the image can be obtained in advance, for example, from the correspondence between the position coordinates of the end effector when the picking robot 200 (end effector) grasps an appropriate object and the image of the object in the image. The observation processing unit 112 linearly interpolates the correspondence obtained in advance to obtain the calculated action a 1 t It is possible to obtain coordinates in the image that correspond to the position coordinates included in the image. Note that errors when obtaining the correspondence can be absorbed by learning by the learning unit 120. Therefore, there is no need to perform highly accurate calibration of the correspondence.
[0051] The observation processing unit 112 performs the action a 1 t Information indicating the action a 1 t At least one of the reward calculated for and the current control time may be added to the observation information IA as auxiliary information.
[0052] The observation and processing unit 112 may also perform a plurality of types of processing. For example, the observation and processing unit 112 may perform both the above-mentioned image cropping and image rotation.
[0053] In addition, when the dimension of the observation information is high, in order to reduce the computational cost, the obtained observation information s t In some cases, a reduced image or only a portion of a plurality of images taken from a plurality of angles may be used as the observation information s. t Instead of using directly, we use observational information s t The observation information IB may be calculated using information related to the observation information s t The relevant information is, for example, the following information: Observation information t A higher resolution image than the image (image before reduction) Observation information t A plurality of images including an image in which the above-mentioned image is taken from an angle different from that of the above-mentioned image.
[0054] The multiple images taken from multiple angles may be, for example, images taken by multiple cameras 250. t The information related to is stored in, for example, the storage unit 151 and is read out from the storage unit 151 by the observation and processing unit 112 during processing.
[0055] 2, the behavior calculation unit 113 calculates the behavior AB of the controlled object based on the processed observation information IB. The behavior calculation unit 113 corresponds to the function of determining the second stage behavior described above.
[0056] For example, the action calculation unit 113 calculates the action AB using a learned policy PB (second model) used to calculate the action AB. 2 t In addition, the policy PB currently obtained by learning is expressed as a probability distribution π 2 (a 2 t |s 2t In this case, the behavior calculation unit 113 calculates the observed information s 2 t Using the probability distribution π 2 (a 2 t |s 2 t )Act according to a 2 t For example, the behavior calculation unit 113 calculates the probability distribution π 2 (a 2 t |s 2 t ) with the probability of one action a 2 t Action a 2 t is action a 1 t Observation information processed according to 2 t Because we use the action a 1 t This can be interpreted as action AB corrected by the above.
[0057] The decision unit 114 decides whether to 1 t and Action a 2 t For example, the determination unit 114 determines the behavior of the control target based on the behavior a 1 t and Action a 2 t By combining these, the final action a = g(a 1 t ,a 2 t The determining unit 114 may perform a conversion process such as scaling on the determined action, and output the processed action as control information.
[0058] action a 1 t and Action a 2 t Any method may be used for combining the above signals. For example, a method using at least one of linear transformation and multiplication may be applied.
[0059] The linear transformation is a synthesis method that can be applied to, for example, information indicating position coordinates included in an action. For example, the determination unit 114 may perform a synthesis of g(a 1 t ,a 2 t )=a 1 t +c×a 2 t We use a linear transformation such as:
[0060] Multiplication is a synthesis method that can be applied to information indicating a rotation included in an action (e.g., a quaternion). For example, the determination unit 114 may 1 t ,a 2 t )=a 1 t ×a 2 t Multiplication such as:
[0061] In the case of an action that can be determined only from the post-processing observation information IB, such as opening and closing the gripper, the determination unit 114 determines g(a 1 t ,a 2 t )=a 2 t Act like a 2 t Only from this may the final action be determined.
[0062] Next, the learning unit 120 will be described. The learning unit 120 corresponds to a function that learns a policy (model) used to determine an action. The learning unit 120 includes an observation processing unit 121, an update unit 122 (second update unit), and an update unit 123 (first update unit). Note that the method of processing the observation information IA based on the action AA is common between the estimation time and the learning time. Therefore, the observation processing unit 112 in the estimation unit 110 and the observation processing unit 121 in the learning unit 120 may be common.
[0063] During learning at control time t+1, the storage unit 151 stores the following information acquired by the acquisition unit 101: Observation information s at control time t t ·Action takena t ·Action a t Reward obtained by r t Observation information s at control time t+1 t+1
[0064] The observation processing unit 121 processes the observation information selected (sampled) from the observation information stored in the storage unit 151 into an appropriate action AA (action a 1 t ) under the observation information s 2 t =f(s t ,a 1 t ) to convert the observation information s 2 t corresponds to the processed observation information IB. The action AA can be obtained, for example, by the following method. ·Actual action taken a t · Probability distribution π of the current policy PA 1 (a 1 t |s t ) Action a selected with probability according to 1 t
[0065] The update unit 122 uses the observation information IB to update the action-value function Q 2 (s 2 t ,a 2 t ) is calculated as the action value AVB (second action value). 2 (s 2 t ,a 2 t ) is an example of a second action-value function.
[0066] Furthermore, the update unit 122 updates (learns) the policy PB using the calculated action value AVB. As described above, the learning of the policy PB is performed by the action value function Q 2 (s 2t ,a 2 t ) is achieved by maximizing
[0067] That is, the update unit 122 updates, for example, the information (s 2 t ,a t ,r t ,s 2 t+1 ,a 1 t ) The action value function Q 2 (s 2 t ,a 2 t ) is the value of the action value AVB, and the action value function Q 2 (s 2 t ,a 2 t For example, when TD learning is used, the update unit 122 updates the action value function Q 2 (s 2 t ,a 2 t ) to learn.
number
[0068] where a 2 t is a t =g(a 1 t ,a 2 t When g is given by linear transformation and multiplication, the update unit 122 satisfies a t and a 1 t From analytically, 2 t The update unit 122 may obtain a by sampling. 2 t The update unit 122 may obtain a term shown in the following equation (5) from equation (4) by sampling. This term is also calculated using the action value function Q 1The value of AVA (action value) may be replaced by the following equation (6).
number
number
[0069] The update unit 123 updates the observation information IA and the action value AVB (action value function Q 2 (s 2 t ,a 2 t ) and the value of ) are used to learn the policy PA. 1 (s t ,a 1 t ) is calculated as the action value AVA (first action value). 1 (s t ,a 1 t ) is an example of a first action-value function.
[0070] Furthermore, the update unit 123 uses the information (observation information IA) stored in the storage unit 151 and the calculated action value AVA to calculate the action value function Q 1 (s t ,a 1 t ) is updated (learned). As mentioned above, the learning of the policy PA is done by updating the action value function Q 1 (s t ,a 1 t ) is achieved by maximizing
[0071] For example, the update unit 123 calculates the action value function Q by using an update formula assuming that the action AB is not executed and the action value AVA. 1 (s t ,a 1 t ) is updated (learning the policy PA). Not executing the action AB means, for example, that the decision unit 114 decides on the action AA as the final action a. In this case, the action-value function Q 1(s t ,a 1 t ) and the action value function Q 2 (s 2 t ,a 2 t ) satisfies the following equation (7).
number
[0072] Therefore, the update unit 123 updates the action value function Q 1 (s t ,a 1 t ) to update the
number
[0073] Further, the update unit 123 uses the information (observation information IA) stored in the storage unit 151 to update the action value function Q 1 (s t ,a t ) may be learned directly by TD learning.
number
[0074] For example, when the determination unit 114 determines the final action while ignoring some components of the action AA, the effect of calculating the action value when the action AB is not executed is small. In such a case, the update unit 123 calculates the action value function Q by using an update formula assuming that the action AB is executed and the action value AVA. 1 (s t ,a 1 t ) may be updated (learning the policy PA).
[0075] For example, action a is divided into a=(x,h) based on the coordinate x and the gripper opening / closing h, and the function that combines action AA and action AB is g(x1 ,x 2 )=x 1 +cx 2 , and g(h 1 ,h 2 )=h 2 Consider the case where:
[0076] In this case, there is no need to consider the h component when calculating the action value AVA. Therefore, the action value function is Q 1 (s t ,x 1 t ) In this case, the action value function Q 1 and the action value function Q 2 satisfies the following formula (10). Therefore, the update unit 123 may perform learning using an update formula shown in the following formula (11) instead of formula (8).
number
number
[0077] Here, the term shown in equation (12) is h 2 Policy (probability distribution) π 2 This means the expected value when sampling according to
number
[0078] In order to actively explore previously unobserved states and states where the policy output is uncertain, a method of adding the entropy of the action to the normal reward may be used. The entropy of the action is an example of information that represents the randomness of the action. For example, in equations (4) and (9), r t But with the entropy term H(π) added, t +H(π). In this embodiment, two strategies PA and PB are used. Therefore, the entropy term H(π) is the sum of the two entropies H(π 1 ) and H(π2 ), or two entropies H(π 1 ) and H(π 2 ) may also include statistical information such as the average
[0079] The target control unit 130 uses the strategy learned (updated) as described above to control the control target so as to take the action determined by the estimation unit 110. For example, the target control unit 130 receives control information corresponding to the action determined by the estimation unit 110 (determination unit 114) from the estimation unit 110, and controls the picking robot 200 according to the received control information.
[0080] Each of the above units (acquisition unit 101, memory control unit 102, output control unit 103, estimation unit 110, learning unit 120, and target control unit 130) is realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU (Central Processing Unit) execute a program, that is, by software. Each of the above units may be realized by a processor such as a dedicated IC (Integrated Circuit), that is, by hardware. Each of the above units may be realized by using a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.
[0081] Next, the control process by the control device 100 of this embodiment will be described. Fig. 4 is a flowchart showing an example of the control process in this embodiment. The control process is a process of determining the behavior of the control target (the picking robot 200) using a learned model, and controlling the control target by the determined behavior. The control process is executed at each control time t.
[0082] The acquisition unit 101 acquires observation information from the picking robot 200 (step S101). The acquisition unit 101 acquires the acquired observation information s t to the behavior calculation unit 111. Also, the acquisition unit 101 outputs the observation information stー1 , observation information s tー1 Action a determined using t-1 , action a t-1 The reward r obtained from the picking robot 200 by t-1 , and the observation information s at the control time t t is stored in the storage unit 151 as experience information.
[0083] The behavior calculation unit 111 calculates the observed information s t Using Action AA, Action a t (Step S102). The behavior calculation unit 111 calculates the observed information s t and action AA are output to the observation processor 112 and the decision unit 114.
[0084] The observation processing unit 112 processes the input observation information s t and outputs the processed observation information IB to the behavior calculation unit 113 (step S103).
[0085] The behavior calculation unit 113 calculates the behavior AB using the observation information IB input from the observation processing unit 112, and outputs the behavior AB to the determination unit 114 (step S104).
[0086] The determination unit 114 determines an action by combining the action AA input from the action calculation unit 111 and the action AB input from the action calculation unit 113 (step S105). The determination unit 114 may convert the determined action into control information and output it.
[0087] The object control unit 130 controls the picking robot 200 by outputting control information corresponding to the determined action to the picking robot 200 (step S106).
[0088] Next, the learning process by the control device 100 of this embodiment will be described. Fig. 5 is a flowchart showing an example of the learning process in this embodiment. The learning process is a process for learning a model used in the control process and the like. Before the learning process, for example, a plurality of pieces of experience information are stored in the storage unit 151 by operating the control process shown in Fig. 4.
[0089] The learning unit 120 acquires (samples) a certain number (an integer equal to or greater than 1) of empirical information from the plurality of pieces of empirical information stored in the storage unit 151, and outputs the acquired empirical information to the observation processing unit 121 (step S201).
[0090] The observation processing unit 121 processes the observation information IA included in the input experience information using the behavior included in the experience information as an action AA, and outputs the processed observation information IB to the update unit 122 (step S202). The value of the behavior AA may be a value calculated by the behavior calculation unit 111 using the observation information IA included in the experience information.
[0091] The update unit 122 uses the experience information including the processed observation information IB to update the action value function Q 2 (Policy PB) is updated (step S203).
[0092] The update unit 122 judges whether or not to end the update of the policy PB (step S204). Any method may be used to judge the end, but for example, the following method may be applied. When the value of the error term becomes equal to or less than a threshold (threshold of the error term), it is determined that the process is finished. Note that the error term is a term for calculating the difference included in the right-hand side of the update equation such as equation (2). -When the number of updates exceeds a threshold (threshold for number of updates), it is determined that the process has ended.
[0093] If the update of the policy PB is not to be ended (step S204: No), the process returns to step S202 and the process is repeated.
[0094] When updating of the policy PB is to be ended (step S204: Yes), the update unit 123 updates the experience information and the action value AVB (action value function Q 2 (s 2 t ,a 2 t ) and the value of the action value function Q 1 The update unit 123 updates the action value function Q 2 Instead of the value of 1 (Potency PA) may be updated directly.
[0095] The update unit 123 judges whether or not to end the update of the policy PA (step S206). Any method may be used to judge the end, but the same method as that for judging the end of the update of the policy PB can be applied.
[0096] If the update of the policy PA is not to be ended (step S206: No), the process returns to step S205 and the process is repeated.
[0097] When updating of the policy PA is to be ended (step S206: Yes), the learning unit 120 judges whether or not to end the learning process (step S207). Any method may be used to judge whether or not the learning is to be ended, and for example, the following method may be applied. When the performance of the policy is equal to or greater than a threshold (performance threshold), the process is determined to be terminated. The performance of the policy is calculated, for example, based on the success rate of the action determined by the policy. When the number of times the processes from step S201 to step S206 are repeated exceeds a threshold (threshold for the number of times the processes are repeated), it is determined that the process has ended.
[0098] Next, a description will be given of an example of information displayed on the display unit 152 by the output control unit 103. The output control unit 103 displays on the display unit 152, for example, a part or all of the following information: Information showing the results of processing by the observation and processing unit 121 - The value of the action AB calculated during learning -Graph showing statistical information (variance, average, etc.) of the values of multiple actions A and B calculated during training
[0099] These pieces of information can be used to determine the progress of learning. For example, if the processing result by the observation processing unit 121 always contains information necessary for correcting the action AA (such as when the grasped object is always near the center), it can be determined that the policy PA has been sufficiently learned. Also, if the processing result by the observation processing unit 121 always contains information necessary for correcting the action AA and a high reward is always obtained, it can be determined that the policy PB has been sufficiently learned.
[0100] By the output control unit 103 displaying the above-mentioned information, the user can judge the performance of the control device 100. In addition, for example, by the user adjusting parameters required for learning, such as the learning rate, depending on the judgment result, it becomes possible to prevent performance degradation due to overlearning and reduce the learning time.
[0101] (Modification) In the modified example, a function of processing behavior (behavior processing unit) is added. Fig. 6 is a block diagram showing an example of the configuration of a control device 100-2 of the modified example. As shown in Fig. 6, the control device 100-2 includes a memory unit 151, a display unit 152, an acquisition unit 101, a memory control unit 102, an output control unit 103, an estimation unit 110, a learning unit 120-2, and a target control unit 130.
[0102] The modification is different from the above embodiment in that the learning unit 120-2 further includes a behavior processing unit 124-2, and the behavior input to the observation processing unit 121-2 is replaced with the behavior output from the behavior processing unit 124-2. Since the other configurations are the same as those in FIG. 2, the same reference numerals are used and the description is omitted.
[0103] The behavior processing unit 124-2 processes the behavior of the control target. The behavior to be processed may be an actually executed behavior, or may be a behavior calculated according to the current policy PA.
[0104] Any method may be used to process the behavior, but for example, a method of randomly changing some or all of the elements contained in the behavior may be applied. By processing the behavior in this way, it is possible to increase the amount of learning data. As a result, it is possible to further improve the learning efficiency.
[0105] The observation processor 121-2 differs from the observation processor 121 of the above embodiment in that the observation processor 121-2 processes observation information using behavior processed by a behavior processor 124-2.
[0106] As described above, according to the embodiment, the behavior of the controlled object can be determined with higher accuracy.
[0107] Next, a hardware configuration of the control device of the embodiment will be described with reference to Fig. 7. Fig. 7 is an explanatory diagram showing an example of a hardware configuration of the control device of the embodiment.
[0108] The control device of the embodiment includes a control device such as a CPU 51, storage devices such as a ROM (Read Only Memory) 52 and a RAM 53, a communication I / F 54 that connects to a network and performs communication, and a bus 61 that connects each unit.
[0109] The programs executed by the control device of the embodiment are provided in a state that they are pre-installed in the ROM 52 or the like.
[0110] The program executed by the control device of the embodiment may be configured to be provided as a computer program product by being recorded in an installable or executable format on a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), or a DVD (Digital Versatile Disk).
[0111] Furthermore, the program executed by the control device of the embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the program executed by the control device of the embodiment may be provided or distributed via a network such as the Internet.
[0112] The program executed by the control device of the embodiment can cause a computer to function as each unit of the control device described above. In this computer, the CPU 51 can read the program from a computer-readable storage medium onto a main storage device and execute the program.
[0113] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]
[0114] 100, 100-2 control device 101 Acquisition Department 102 Memory control unit 103 Output control section 110 Estimation part 111 Behavior Calculation Department 112 Observation and Processing Department 113 Behavior Calculation Department 114 Decision Section 120, 120-2 Learning Section 121, 121-2 Observation and Processing Section 122 Update Department 123 Update Department 124-2 Behavior processing department 130 Target control section 151 Storage section 152 Display section
Claims
1. an acquisition unit that acquires first observation information used to determine a behavior of a control target; a first behavior calculation unit that calculates the first behavior in response to the first observation information by using a first model that has been trained to input the first observation information and output a first behavior of the controlled object; an observation processing unit that processes the first observation information based on the first behavior and outputs second observation information; a second behavior calculation unit that calculates the second behavior in response to the second observation information by using a second model that has been trained to input the second observation information and output a second behavior of the controlled object; a determination unit that determines an action of the control target based on the first action and the second action; a target control unit that controls the control target so as to take a determined action; A control device comprising:
2. a second updating unit that calculates a second action value used to update the second model using the second observation information, and updates the second model using the calculated second action value; a first updating unit that calculates a first action value used to update the first model using the second observation information and the second action value, and updates the first model using the calculated first action value; Further comprising: The control device according to claim 1 .
3. A behavior processing unit that processes the behavior of the control target is further provided, the observation processing unit processes the first observation information based on the processed behavior and outputs the second observation information. The control device according to claim 2.
4. The first update unit further calculates the first action value using the first observation information, and updates the first model using the calculated first action value. The control device according to claim 2.
5. the first update unit updates the first model using an update equation assuming that the second action is not executed and the first action value; The control device according to claim 2.
6. The first update unit updates the first model by using an update equation assuming that the second action is executed and the first action value. The control device according to claim 2.
7. The second update unit updates the second model using an update equation including a term in which a reward for a determined action is added with randomness of the determined action. The control device according to claim 2.
8. The first update unit updates the first model using an update equation including a term in which a reward for a determined action is added with randomness of the determined action. The control device according to claim 2.
9. the first observation information is an image, The observation processing unit performs at least one of cropping, rotating, and perspective transformation of the image. The control device according to claim 1 .
10. the observation processing unit adds at least one of information indicating the first action, a reward calculated for the first action, and a time to the first observation information; The control device according to claim 1 .
11. the determination unit determines the action of the control target by at least one of a linear transformation using the first action and the second action and a multiplication of the first action and the second action. The control device according to claim 1 .
12. The observation and processing unit includes: processing the first observation information using information related to the first observation information; The control device according to claim 1 .
13. An output control unit that outputs the second action calculated by the second action calculation unit, The control device according to claim 1 .
14. A control method executed by a control device, comprising: An acquisition step of acquiring first observation information used to determine a behavior of a control target; a first behavior calculation step of calculating the first behavior in response to the first observation information by using a first model trained to input the first observation information and output a first behavior of the controlled object; an observation processing step of processing the first observation information based on the first behavior and outputting second observation information; a second behavior calculation step of calculating the second behavior in response to the second observation information by using a second model trained to input the second observation information and output a second behavior of the controlled object; a determining step of determining an action of the control target based on the first action and the second action; an object control step of controlling the control object so as to take a determined action; A control method comprising:
15. On the computer, An acquisition step of acquiring first observation information used to determine a behavior of a control target; a first behavior calculation step of calculating the first behavior in response to the first observation information by using a first model trained to input the first observation information and output a first behavior of the controlled object; an observation processing step of processing the first observation information based on the first behavior and outputting second observation information; a second behavior calculation step of calculating the second behavior in response to the second observation information by using a second model trained to input the second observation information and output a second behavior of the controlled object; a determining step of determining an action of the control target based on the first action and the second action; an object control step of controlling the control object so as to take a determined action; A program for executing the above.
Citation Information
Patent Citations
Information processing apparatus and program
JP2022115640A
