Posture estimation device

Abstract

To provide a posture estimation device that can easily estimate an appropriate posture even when performing large movements. [Solution] The posture estimation device is a posture estimation device that estimates the target posture of a robot capable of performing robotic work using a learning-based method. The posture estimation device comprises an acquisition unit and an estimation unit. The estimation unit has a hand evaluation value estimation unit that estimates a hand evaluation value, which is an evaluation value of posture related to hand movements, which are movements that do not involve arm movement, based on environmental data. The estimation unit has an arm evaluation value estimation unit that estimates an arm evaluation value, which is an evaluation value of posture related to arm movements, which are movements that involve arm movement, based on environmental data. The posture estimation device comprises an integration unit that integrates the hand evaluation value and the arm evaluation value to calculate an overall evaluation value, which is an evaluation value of the movement posture in robotic work, and an output unit that outputs the overall target posture of the robot based on the overall evaluation value.

Description

【Technical Field】 【0001】 The disclosure in this specification relates to a posture estimation device. 【Background Art】 【0002】 Patent Document 1 discloses grasping learning using a modular neural network that separates a high-dimensional neural network into a low-dimensional neural network. The description of the prior art document is incorporated herein by reference as an explanation of the technical elements in this specification. 【Prior Art Documents】 【Patent Documents】 【0003】 【Patent Document 1】 Japanese Patent Application Laid-Open No. 2022-187984 【Summary of the Invention】 【Problems to be Solved by the Invention】 【0004】 In the configuration of the prior art document, only the relationship between the hand and the object is focused on in grasping learning. Therefore, whether the movement of the entire robot including the arm can be executed is not considered. Thus, when the robot moves significantly by executing an operation after reaching an optimal operation posture, the operation may not be executable. From the above viewpoints or other viewpoints not mentioned, further improvement is required for the posture estimation device. 【0005】 One object of the disclosure is to provide a posture estimation device that can easily estimate an appropriate operation posture even when executing a large movement. 【Means for Solving the Problems】 【0006】 The above objectives are achieved by the combination of features described in the independent claims. Subordinate claims further specify advantageous examples. The reference numerals in parentheses in the claims indicate correspondences with specific embodiments described later in the embodiments, and do not limit the disclosed technical scope. The objectives, features, and effects disclosed in this specification will become clearer with reference to the subsequent detailed description and the accompanying drawings. 【0007】 One disclosure for achieving the above objective is a posture estimation device that estimates the target posture of a robot capable of performing a robot operation, which is an operation performed on an object by a robot (10) having a hand (13) and an arm (12) that are connected so as to be rotatable relative to each other, using a learning-based method. An acquisition unit (141) acquires environmental data including data about an object captured by a sensor (120), The estimation unit (150) includes a hand evaluation value estimation unit (158) that estimates a hand evaluation value, which is an evaluation value of posture related to hand movements, which are robot operations that do not involve arm movement, based on environmental data, and an arm evaluation value estimation unit (159) that estimates an arm evaluation value, which is an evaluation value of posture related to arm movements, which are robot operations that involve arm movement, based on environmental data, An integration unit (146) combines the hand evaluation value and the arm evaluation value to calculate an overall evaluation value, which is an evaluation value of the movement posture in robot work, This posture estimation device includes an output unit (149) that outputs the overall target posture of the robot based on the overall evaluation value. 【0008】 The disclosed posture estimation device includes an integration unit that combines hand evaluation values ​​and arm evaluation values ​​to calculate a comprehensive evaluation value, which is an evaluation value of the operating posture during robot operation. Therefore, the comprehensive evaluation value can be calculated based on two evaluation values: hand evaluation values, which are evaluation values ​​of movements without arm movement, and arm evaluation values, which are evaluation values ​​of movements with arm movement. Thus, it is possible to estimate an appropriate target posture in robot operation that includes large movements involving arm movement. In other words, it is possible to provide a posture estimation device that makes it easy to estimate an appropriate operating posture even when performing large movements. [Brief explanation of the drawing] 【0009】 [Figure 1] This is a block diagram showing the configuration of a robotic work system. [Figure 2] This is a diagram showing the configuration of the robot. [Figure 3] This is a block diagram showing the functions that a processor possesses. [Figure 4] This is a flowchart related to robotic operations. [Figure 5] This is a flowchart illustrating the process in step S110 of Figure 4. [Figure 6] This is a flowchart illustrating the process in step S130 of Figure 4. [Figure 7] This is a block diagram showing the functions provided by the processor in the second embodiment. [Figure 8] This is a flowchart relating to the robot operation in the second embodiment. [Figure 9] Figure 8 is a flowchart illustrating the process in step S280. [Modes for carrying out the invention] 【0010】 Multiple embodiments will be described with reference to the drawings. In multiple embodiments, functionally and / or structurally corresponding and / or related parts may be given the same reference numeral, or reference numerals that differ by hundreds or more digits. For corresponding and / or related parts, refer to the description of other embodiments. 【0011】 (First Embodiment) In Figure 1, the robotic work system 1 is a system for performing various tasks using the robot 10. The robotic work system 1 can be used, for example, as an automated harvesting system for automatically harvesting crops. The robotic work system 1 can also be used, for example, as an automated manufacturing system for automatically sorting and assembling parts in the manufacturing of industrial products. In particular, in the agricultural sector, there are challenges such as a declining workforce and the need to shorten process design time due to variable-product, variable-volume production, and there is a strong demand for automation and efficiency improvements using the robot 10. 【0012】 The robotic work system 1 is capable of performing tasks that involve various movements as robotic tasks. For example, consider a case where the robot performs a harvesting task, such as harvesting tomatoes. In this case, the robotic tasks include actions such as imaging, grasping, picking, and placing. The imaging action is the action of imaging the tomatoes and their surroundings, which are the objects of the harvesting task. The grasping action is the action of grasping the tomatoes. The picking action is the action of twisting and pulling the grasped tomatoes. The placing action is the action of placing the harvested tomatoes into the harvesting box. The robotic tasks also include posture change actions, which change the posture of the robot 10 to the posture required to perform each action. 【0013】 In the tomato harvesting operation, it may not include the above-described operations or may include operations other than the above-described operations. For example, as an operation replacing the picking operation, it may include a cutting operation for cutting the stalk portion connected to the grasped tomato. When performing the cutting operation, it is preferable to perform the grasping operation simultaneously and cut the stalk portion while grasping the tomato. Thus, depending on the content of the operation, a plurality of operations may be performed simultaneously. 【0014】 The robot work system 1 estimates a target posture using a learning-based method and automatically executes robot work. For example, assume a case where a tomato harvesting operation is performed as robot work. The shapes of tomatoes are different from one another, and the positions and orientations where the tomatoes are borne are also different from each other. Therefore, it is very difficult to successfully harvest tomatoes by repeatedly executing the same harvesting operation according to a predetermined rule. In other words, it is required to select different postures and execute the harvesting operation according to the shape of each tomato. Therefore, it can be said that it is useful to perform posture estimation that achieves both high speed and reliability by performing environmental understanding and prediction using a learning-based method. 【0015】 The robot work system 1 includes a sensor 120, a posture estimation device 100, a drive control device 190, and a robot 10. In the robot work system 1, the posture estimation device 100 estimates a target posture based on the data acquired by the sensor 120. Then, the drive control device 190 outputs a drive signal to the robot 10 so as to realize the target posture, and the robot 10 is driven according to the drive signal. 【0016】 In FIG. 2, the robot 10 has a configuration including a base 11, five arms 12, and one hand 13. The base 11 is installed on a workbench or the like. The five arms 12 are the first arm 12a, the second arm 12b, the third arm 12c, the fourth arm 12d, and the fifth arm 12e. When not distinguishing them, they are described as the arm 12. 【0017】 The base 11 and the first arm 12a are configured to be relatively rotatable by a built-in motor. Similarly, the second arm 12b and the third arm 12c, the third arm 12c and the fourth arm 12d, the fourth arm 12d and the fifth arm 12e, and the fifth arm 12e and the hand 13 are also configured to be relatively rotatable. The robot 10 configured in this way has a configuration with six rotational axes. With the six rotational axes, seven parts, namely the base 11, the five arms 12, and the hand 13, rotate relative to the parts connected by the rotational axes. Since it has six rotational axes, the posture of the robot 10 is defined by the rotational angles of the six axes. The rotational axes are sometimes called joints. The number of rotational axes is not limited to six and may be more than six or less than six. 【0018】 The hand 13 is capable of performing a gripping operation to grip an object. In the gripping operation, methods such as sandwiching the object with the fingertips or adsorbing the object using suction force can be adopted. The hand 13 is capable of performing a twisting operation of rotating and moving the hand 13 while maintaining the state of gripping the object. The hand 13 may be configured to be capable of performing a cutting operation of gripping and cutting the object by adding a scissors shape. 【0019】 In FIG. 1, the sensor 120 is a device for acquiring environmental data that is data indicating information on the surrounding environment of the robot 10. The sensor 120 is an imaging device for imaging obstacles and objects existing around the robot 10. The imaging data by the imaging device provides an example of the environmental data. The imaging data imaged by the sensor 120 can be an RGB image including color information or a depth image including depth information corresponding to each pixel. The imaging data is point cloud data having attribute information such as color and depth corresponding to the coordinate information of points in a three-dimensional space. 【0020】 A hand-mounted camera can be used as the sensor 120. However, the mounting position of the sensor 120 is not limited to the hand 13; the sensor 120 may also be mounted on the arm 12. Furthermore, the sensor 120 may be mounted in a location different from the robot 10. In addition, the sensor 120 may be composed of multiple imaging devices. 【0021】 The drive control device 190 is a device that generates and outputs drive signals for driving the robot 10. The drive control device 190 generates and outputs drive signals related to the rotation angle of each rotation axis of the robot 10 and drive signals related to the end effector of the hand 13. The drive control device 190 may control the driving of the robot 10 by feeding back the results of the robot's driving, or it may control the driving without using the feedback results. 【0022】 The posture estimation device 100 is a device that estimates the posture of the robot 10. Posture is information that indicates the position and orientation of each component of the robot 10 when the robot 10 is performing a task. The posture estimated by the posture estimation device 100 includes information on the position and orientation of the end effector, which is the tip of the robot 10, as three-dimensional information. The position information of the end effector can be, for example, the x, y, and z coordinates of three-dimensional coordinates. The orientation information of the end effector can be, for example, the roll angle, which is the rotation around the x axis, the pitch angle, which is the rotation around the y axis, and the yaw angle, which is the rotation around the z axis. 【0023】 The posture estimation device 100 comprises an input device 110 and a control device 130. The input device 110 is a keyboard or the like. The input device 110 is a device that can input simulation conditions such as robot information, which is information about the structure and posture of the robot 10. The robot information may include information indicating the number of links, length, and shape of the arm 12. The robot information may also include information indicating the shape of the hand 13, such as whether it is a gripper or scissors, and information indicating the size of the hand 13. The robot information may also include information indicating the type of joint, such as a rotary joint, linear joint, torsional joint, spherical joint, plane joint, or universal joint. The robot information may also include information about the range of motion of the joints and the current joint angle. 【0024】 The control device 130 includes a processor 140, a ROM 160, and a RAM 170. An attitude estimation device 100 with such a configuration can also be implemented using a general-purpose computer. The attitude estimation device 100 and the drive control device 190 may reside in the same computer, or they may be configured in separate computers. 【0025】 The ROM 160, a non-volatile storage medium, stores the attitude estimation program that the processor 140 executes. By executing the attitude estimation program stored in the ROM 160 while utilizing the temporary storage function of the RAM 170, the processor 140 executes the parts shown in Figure 3. The execution of the parts shown in Figure 3 by the processor 140 means that the attitude estimation method corresponding to the attitude estimation program is executed. 【0026】 In Figure 3, the processor 140 comprises an acquisition unit 141, an estimation unit 150, an integration unit 146, and an output unit 149. The acquisition unit 141 acquires robot information input from the input device 110 and point cloud data captured by the sensor 120. The integration unit 146 integrates multiple estimation results estimated by the estimation unit 150 to calculate an overall evaluation value. The overall evaluation value is an evaluation value for evaluating a series of robot operations as a combination of the motion postures of each movement included in the robot operation. The output unit 149 determines the overall target posture, which is the target posture for the robot 10 as a whole, and outputs it to the drive control device 190. 【0027】 The estimation unit 150 estimates various information necessary to output the overall target posture in robot operations. The estimation unit 150 can also be described as an evaluation unit that estimates evaluation values ​​for evaluating whether or not to adopt a particular candidate motion posture as the overall target posture. 【0028】 The estimation unit 150 stores data obtained when the robot actually performs harvesting work as training data and utilizes it as training data for supervised learning. However, the estimation unit 150 can employ various learning-based methods, not limited to supervised learning, as its learning-based method. The estimation unit 150 can also be described as a learning unit that learns to estimate the evaluation value of the movement posture using a learning-based method based on the training data and simulation conditions. 【0029】 An example of training data used in a learning-based method is information about the motion posture performed by robot 10. More specifically, this includes coordinate information indicating the position of the end-effector in the motion posture performed by robot 10, and information on the roll angle, pitch angle, and yaw angle indicating the orientation of the end-effector. The information about the motion posture performed by robot 10 may also be information on the rotation angles of each joint of robot 10. The information about the motion posture performed by robot 10 may also be information on the approach vector in the robot 10's motion. 【0030】 Examples of training data include point cloud data of objects and obstacles acquired by sensor 120. Another example of training data is information on the robot 10's posture during harvesting and data indicating whether the harvesting was successful. This data can also be considered data for the robot 10's coordinate system. Yet another example of training data is data showing the center position of a tomato and the position of its calyx. This data can be used to represent the object's coordinate system by defining the direction from the tomato's center to its calyx as the z-axis. Finally, another example of training data is data showing the shape and size of the hand 13. This data allows for posture estimation based on the data closest to the current shape and size of the robot 10's hand 13, making accurate posture estimation easier. By using data showing the object's coordinate system and data showing the shape and size of the hand 13 as training data, the learning convergence rate can be improved. 【0031】 The estimation unit 150 includes an operation success rate estimation unit 151, an operability estimation unit 152, an interference estimation unit 153, and a work success rate estimation unit 154. The estimation unit 150 also includes a hand evaluation value estimation unit 158 ​​and an arm evaluation value estimation unit 159. 【0032】 The motion success rate estimation unit 151 estimates the probability of success for each motion included in the robot operation. The motion success rate estimation unit 151 comprises an occlusion rate estimation unit 151A, a gripping success rate estimation unit 151B, and a motion feasibility estimation unit 151C. 【0033】 The occlusion rate estimation unit 151A estimates the occlusion rate, which is an index indicating the percentage of an object that is hidden by an obstacle when the object is imaged. If the entire object is visible, the occlusion rate is 0%, and if the entire object is hidden by an obstacle, the occlusion rate is 100%. For example, if the object is a tomato, and half of the tomato is hidden by leaves, the occlusion rate will be 50%. 【0034】 The method for estimating the occlusion rate is not limited to estimating it by the proportion of the object that is hidden relative to the whole object. For example, in the action of grasping a tomato, if it is sufficient to image only the fruit portion of the tomato, the calyx and pedicel of the tomato can be ignored, and the occlusion rate can be estimated from the proportion of the fruit portion that is hidden. 【0035】 In the process of cutting a tomato, if it is sufficient to image only the stem portion of the tomato, the occlusion rate can be estimated from the proportion of the stem that is hidden, without considering the tomato fruit or calyx. In this case, instead of estimating the occlusion rate from the proportion of the hidden portion of the stem to the total stem, the occlusion rate can also be estimated from the proportion of the stem that is imaged relative to the portion of the stem required for the cutting operation. For example, consider a case where it is sufficient to image 2 cm of the stem during the cutting operation. In this case, if the portion of the stem included in the image data is 2 cm or more, the occlusion rate is estimated to be 0%. On the other hand, if the portion of the stem included in the image data is 1 cm, the occlusion rate is estimated to be 50%. 【0036】 The gripping success rate estimation unit 151B estimates the gripping success rate, which is an index indicating the success rate of the action of gripping an object. In estimating the gripping success rate, it is considered whether the forces applied by the hand 13, which functions as a gripper, to the object cancel each other out and whether the object does not rotate unintentionally due to the force applied by the hand 13. In other words, the gripping success rate is estimated by considering the extent to which force closure is achieved when gripping the object with the hand 13. Furthermore, when gripping using suction force, the gripping success rate is estimated by considering the magnitude of the suction force. 【0037】 The motion feasibility estimation unit 151C estimates motion feasibility, which is an indicator of whether the robot 10 can assume the motion posture to be estimated. The presence of obstacles is not considered when estimating motion feasibility. That is, assuming that there are no obstacles in the surroundings, it estimates whether the robot 10 can assume the posture changes from the start posture to the end posture of the motion. If there are no posture changes between the start posture and the end posture of the motion, it estimates whether the robot can assume one posture that is both the start posture and the end posture of the motion. If the motion feasibility estimation unit 151C estimates that the robot 10 can assume the posture changes from the start posture to the end posture of the motion, it estimates that motion is possible. On the other hand, if it estimates that there is even one posture that the robot 10 cannot assume during the posture changes from the start posture to the end posture of the motion, it determines that motion is impossible. 【0038】 The operation feasibility estimation unit 151C can estimate whether the device is operational by performing inverse kinematics calculations for its operational posture. If a solution exists for the inverse kinematics calculations, it can be estimated that the device is operational. On the other hand, if a solution does not exist for the inverse kinematics calculations, it can be estimated that the device is not operational. 【0039】 The maneuverability estimation unit 152 estimates the maneuverability, which is an index indicating the size of the range in which the robot 10 can operate. The maneuverability can be estimated by calculating the maneuverability ellipsoid for the robot 10. More specifically, the posture of the target hand 13 is expressed in terms of joint variables, and a Jacobian matrix is ​​derived. Singular value decomposition is performed on this Jacobian matrix to calculate singular values. The maneuverability ellipsoid can be calculated using these singular values. The larger the maneuverability ellipsoid, the larger the operable range and the greater the maneuverability. 【0040】 The interference estimation unit 153 estimates whether the robot 10 will interfere with surrounding obstacles. The interference estimation unit 153 estimates the interference probability, which is an indicator of the possibility that the robot 10 may not be able to perform its actions due to interference with an obstacle. The interference estimation unit 153 also estimates the presence or absence of adverse effects, which is an indicator of the possibility that the robot 10 may damage the obstacle or the robot 10 itself due to interference with an obstacle. For example, in tomato harvesting work, the arm 12 may come into contact with the stem of a tomato or another tomato, damaging the stem or the other tomato. In such a case, since the robot 10 interferes and damages the stem or the other tomato, it is estimated that there is an adverse effect. 【0041】 The presence or absence of interference with an obstacle and the presence or absence of adverse effects are estimated using different criteria. For example, if an obstacle that interferes with the robot 10 is unlikely to cause damage even if it comes into contact with the robot, or if damage to the obstacle has no effect, then it is estimated that there is no adverse effect. An example of an obstacle that is unlikely to cause damage even if it comes into contact with the robot 10 and whose damage has no effect is a tomato leaf. Alternatively, the presence or absence of interference with an obstacle may be estimated as occurring if the distance between the obstacle and the robot 10 is 2 mm or less, and the presence or absence of adverse effects may be estimated as occurring if the distance between the obstacle and the robot 10 is zero, i.e., if there is contact. 【0042】 The task success rate estimation unit 154 estimates the task success rate, which is an indicator of the probability that the robot 10 will successfully perform a robot task. The task success rate estimation unit 154 estimates the task success rate by dividing the number of successful robot tasks by the number of times the robot task has been performed. For example, if the robot task is performed 10 times and 8 of those times are successful, the task success rate is estimated to be 80%. 【0043】 When a robot operation involves multiple actions such as grasping, picking, and placing, the operation is considered successful only if all actions are performed successfully. Therefore, if the grasping and picking actions are successful in a harvesting operation, but the placing action fails, the harvesting operation will be considered a failure. 【0044】 The hand evaluation value estimation unit 158 ​​estimates a hand evaluation value, which is an evaluation value of a hand action that does not involve the movement of the arm 12. Hand actions do not include actions that involve the movement of both the arm 12 and the hand 13. However, even if an action involves the movement of the arm 12, if the movement of the arm 12 is very small and there is no need to consider interference between the arm 12 and an obstacle, it may be considered a hand action. Hand actions include, for example, imaging actions to image an object, grasping actions to grasp an object, cutting actions to cut an object, and placing actions to place an object. 【0045】 The hand evaluation value estimation unit 158 ​​estimates the hand evaluation value in relation to the motion start posture, which is the posture in which the robot 10 starts the hand movement. In other words, the hand evaluation value estimation unit 158 ​​changes the motion start posture and estimates the hand evaluation value for each posture in which the hand movement is performed. For example, the hand evaluation value estimation unit 158 ​​estimates a first hand evaluation value, which is the hand evaluation value when the hand movement is performed in the first posture, and a second hand evaluation value, which is the hand evaluation value when the hand movement is performed in the second posture. Here, the first posture and the second posture are different postures from each other. For example, a deep learning model can be used in estimating the hand evaluation value. Details of the hand evaluation value estimation method will be explained later. The hand evaluation value is an evaluation value that is mainly determined by the relationship between the hand 13 and the object, and can be said to be an evaluation value that is influenced by the coordinate system of the object. 【0046】 The arm evaluation value estimation unit 159 estimates an arm evaluation value, which is an evaluation value of an arm action, which is an action involving the movement of the arm 12. An arm action includes actions involving the movement of both the arm 12 and the hand 13. Examples of arm actions include a tearing action, where an object is grasped and then twisted and pulled, and a posture change action, where the arm 12 is moved from one position to another. 【0047】 The tearing motion will be explained in more detail. The tearing motion is an action that can be achieved by combining, for example, a gripping motion, a rotational motion, and a linear motion. The rotational motion is the motion of rotating the hand 13 along the axis of rotation, and is defined by the direction of rotation (clockwise or counterclockwise), the angle of rotation, the speed of rotation, the axis of rotation, the torque of rotation, etc. The linear motion is the motion of moving the hand 13 in a straight line, and is defined by the direction of movement, the distance of movement, the speed of movement, etc. An action that performs rotational and linear motions simultaneously can also be called a helical motion or a twisting motion. The twisting motion will be defined by various elements such as curvature, torsion ratio, and helical distance. 【0048】 The tearing action may also be performed by combining actions other than those described above. For example, the tearing action may be performed by combining a gripping action, a rotation action, and a swivel action. The swivel action is an action that rotates the joints of the arm 12 to change the orientation of the hand 13, and is defined by the direction of rotation (clockwise or counterclockwise), the angle of rotation, the speed of rotation, etc. 【0049】 The arm evaluation value estimation unit 159 estimates the arm evaluation value in relation to the starting posture, which is the posture in which the robot 10 begins arm movement. In other words, the arm evaluation value estimation unit 159 changes the starting posture and estimates the arm evaluation value for each posture in which the arm movement is performed. For example, the arm evaluation value estimation unit 159 estimates a first arm evaluation value, which is the arm evaluation value when the arm movement is performed in the first posture, and a second arm evaluation value, which is the arm evaluation value when the arm movement is performed in the second posture. Here, the first posture and the second posture are different postures from each other. For example, a deep learning model can be used in estimating the arm evaluation value. Details of the arm evaluation value estimation method will be explained later. The arm evaluation value is an evaluation value that is mainly determined by the posture of the arm 12, and can be said to be an evaluation value that is influenced by the coordinate system of the robot 10. 【0050】 This section explains the sequence of robot operations using robot work system 1 as an automated harvesting system, specifically focusing on the automated harvesting of tomatoes, with diagrams illustrating the process. The following explanation assumes that the harvesting operation includes five steps: imaging, approach, picking, placement, and retraction. The imaging step includes two actions: a change in posture from the initial posture to the imaging posture, and the imaging action itself. Here, the initial posture refers to the posture of robot 10 at the start of the robot operation. The approach step includes one action: a change in posture from the imaging posture to the gripping posture. The picking step includes two actions: a gripping action and a picking action. The placement step includes two actions: a change in posture from the post-picking posture to the placement posture, and the placement action. The retraction step includes two actions: a change in posture from the placement posture to the retraction posture, and a change in posture from the retraction posture back to the initial posture. 【0051】 In Figure 4, when the robot work system 1 starts harvesting tomatoes, in step S101, the robot information acquisition unit 141 acquires the input information. In addition to robot information, it also acquires information that has been input as simulation conditions to be used in the simulation. After acquiring the input information, the process proceeds to step S102. 【0052】 In step S102, the acquisition unit 141 acquires point cloud data, which is sensor information. The point cloud data contains information about the target object, the tomato, and any obstacles in its surroundings. For example, after acquiring the sensor information, the process proceeds to step S110. 【0053】 In step S110, the hand evaluation value estimation unit 158 ​​estimates the hand evaluation value distribution. The details of the hand evaluation value distribution estimation process are described below. 【0054】 In Figure 5, when the hand evaluation value distribution estimation process is started, the estimation unit 150 selects the operation posture for each operation for which the hand evaluation value is estimated in step S111. The operation posture is the posture at the start of that operation, and is randomly selected from various posture candidates. For example, consider the case where hand evaluation values ​​are estimated for imaging, grasping, and placement operations. In this case, a posture is randomly selected from the posture candidates as the imaging posture, which is the posture at which the imaging operation starts, and this is designated as the first imaging posture. Furthermore, a posture is randomly selected from the posture candidates as the grasping posture, which is the posture at which the grasping operation starts, and this is designated as the first grasping posture. Furthermore, a posture is randomly selected from the posture candidates as the placement posture, which is the posture at which the placement operation starts, and this is designated as the first placement posture. 【0055】 The movement posture is not limited to the posture at the start of the movement, but may also include intermediate postures that can be taken during the execution of the movement. For example, in a grasping movement, the posture in which the fingertips of hand 13 are extended may be included in the movement posture as an intermediate grasping posture. If an intermediate posture exists, the hand evaluation value will be estimated for both the movement start posture and the intermediate posture. Below, the estimation process for the arm evaluation value distribution will be explained using the case where an intermediate posture is not included as an example. After selecting the movement posture, proceed to step S113. 【0056】 In step S113, the interference estimation unit 153 estimates the interference probability of the hand movement. The interference probability includes the starting posture interference probability and the movement interference probability. The starting posture interference probability is the probability that interference with an obstacle occurs at the time the movement is to start, making it impossible to maintain that posture. For example, if an obstacle is present in the first imaging posture and the first imaging posture cannot be maintained, the starting posture interference probability in the first imaging posture will be 100%. 【0057】 The probability of an action interfering is the probability that an action will be unable to be performed due to interference with an obstacle during the execution of that action. For example, in the first gripping position, when attempting to extend the hand to grip, it may be impossible to extend the hand due to interference with an obstacle. In this case, the probability of action interfering in the first gripping position is 100%. However, in actions where the movement to be performed is small and interference with obstacles is considered very unlikely, the probability of action interfering may be considered 0%. This effectively eliminates the need to consider the probability of action interfering, thus reducing the processing load on the interference estimation unit 153 compared to when the probability of action interfering is estimated. 【0058】 The interference estimation unit 153 estimates the interference probability as 0% if both the starting posture interference probability and the motion interference probability are 0%. On the other hand, if either the starting posture interference probability or the motion interference probability is 100%, it estimates the interference probability as 100%. After estimating the interference probability, the process proceeds to step S115. 【0059】 In step S115, the success rate estimation unit 151 estimates the success rate of the hand movement. The success rate of the imaging movement is estimated based on the occlusion rate estimated by the occlusion rate estimation unit 151A. The lower the occlusion rate, the higher the estimated success rate of the imaging movement in that imaging posture. The estimation of the success rate of the imaging movement is not limited to the method described above. For example, in addition to the occlusion rate, the success rate of the movement may also be estimated by considering the proportion of information in the imaging data that can be judged as an obvious error. 【0060】 The success rate of the gripping operation is determined by the gripping success rate estimation unit 151B. Therefore, if a gripping posture allows for a gripping operation that satisfies the force closure, the gripping success rate for that posture is estimated to be high. On the other hand, if a gripping posture does not allow for a gripping operation that satisfies the force closure, the gripping success rate for that posture is estimated to be low. The estimation of the success rate of the gripping operation is not limited to the method described above, and various factors that contribute to the success of the gripping operation, such as frictional force, may also be considered. 【0061】 The success rate of the placement operation is estimated to be higher the shorter the distance from the placement position to the place where the object is placed. Furthermore, the success rate of the placement operation is estimated to be higher the wider the placement area. In other words, the wider the range in which successful placement is considered acceptable, the higher the estimated success rate. However, the estimation of the success rate of the placement operation is not limited to the methods described above. 【0062】 The success rate of a hand movement may be calculated by dividing the number of successes by the number of attempts. Here, the number of successes is the number of times the movement was performed successfully. The number of attempts is the number of times the movement was performed. For example, if the same gripping movement is repeated 10 times and succeeded 9 times, the success rate of the movement in that gripping position can be estimated to be 90%. After estimating the success rate of the movement, proceed to step S116. 【0063】 In step S116, the success rate estimation unit 154 estimates the success rate of a series of harvesting operations, including hand movements. The success rate is estimated in relation to the posture of each movement. For example, consider a harvesting operation that includes an imaging operation in the first imaging posture, a grasping operation in the first grasping posture, and a placing operation in the first placing posture. In this case, if the harvesting operation is successful, the number of executions and successes of each movement in each posture are increased by 1. For example, for the grasping operation in the first grasping posture, the number of executions is 1 and the number of successes is 1, so the success rate is estimated to be 100%. On the other hand, if the harvesting operation fails, the number of executions of each movement in each posture is increased by 1, but the number of successes is not increased. For example, for the grasping operation in the first grasping posture, the number of executions is 1 and the number of successes is 0, so the success rate is estimated to be 0%. 【0064】 The success rate of the operation is estimated by accumulating the number of executions and successes of other operations that include the same motion posture. For example, consider a scenario where a harvesting operation including imaging in the first imaging posture, grasping in the first grasping posture, and placing in the first placing posture is performed successfully, and then a harvesting operation including imaging in the second imaging posture, grasping in the first grasping posture, and placing in the first placing posture fails. In this case, for the imaging operation in the first imaging posture, the number of executions is 1 and the number of successes is 1, so the success rate of the operation is estimated to be 100%. On the other hand, for the imaging operation in the second imaging posture, the number of executions is 1 and the number of successes is 0, so the success rate of the operation is estimated to be 0%. Furthermore, for the grasping operation in the first grasping posture and the placing operation in the first placing posture, the number of executions is 2 and the number of successes is 1, so the success rate of the operation is estimated to be 50%. After estimating the success rate of the operation, proceed to step S117. 【0065】 In step S117, the hand evaluation value estimation unit 158 ​​estimates the hand evaluation value. The hand evaluation value estimation unit 158 ​​estimates the hand evaluation value for each movement posture based on the interference probability, the movement success rate, and the work success rate. The hand evaluation value can be estimated as a numerical value between 1.0 and 0. The higher the hand evaluation value, the higher the evaluation, meaning that it is a movement posture that should be adopted as the target posture. 【0066】 The hand evaluation value estimation unit 158 ​​estimates a higher hand evaluation value the lower the interference probability. For example, if the interference probability is 0%, the hand evaluation value is set to 1.0, and if the interference probability is 100%, the hand evaluation value is set to 0. 【0067】 The hand evaluation value estimation unit 158 ​​estimates a higher hand evaluation value the higher the success rate of the action. For example, if the success rate of the action is above the upper threshold, the hand evaluation value is set to 1.0, and if the success rate of the action is below the lower threshold, the hand evaluation value is set to 0. Also, if the success rate of the action is between the upper and lower thresholds, the hand evaluation value is set between 1.0 and 0. In this case, the higher the success rate of the action, the higher the hand evaluation value is set. 【0068】 The hand evaluation value estimation unit 158 ​​estimates a higher hand evaluation value the higher the work success rate. For example, if the work success rate is above the upper threshold, the hand evaluation value is set to 1.0, and if the work success rate is below the lower threshold, the hand evaluation value is set to 0. Also, if the work success rate is between the upper and lower thresholds, the hand evaluation value is set to a higher value between 1.0 and 0, with the hand evaluation value increasing as the work success rate increases. 【0069】 Furthermore, while the hand evaluation value is estimated for each movement posture, the hand evaluation value for each item may be used as the estimated result, or the hand evaluation values ​​for each item may be combined to form the estimated result. For example, suppose that for the first gripping posture, the hand evaluation value based on interference probability is 1.0, the hand evaluation value based on movement success rate is 0.9, and the hand evaluation value based on work success rate is 0.9. In this case, if the hand evaluation values ​​for each item are combined to form the estimated result, the hand evaluation value for the first gripping posture can be estimated to be 0.81, which is the product of the hand evaluation values ​​for each item. After estimating the hand evaluation value for each movement posture, proceed to step S118. 【0070】 In step S118, the hand evaluation value estimation unit 158 ​​estimates whether the number of hand evaluations, which is the number of movement postures for which hand evaluation values ​​have been estimated, is equal to or greater than the hand evaluation threshold. The hand evaluation count is counted for each movement posture. For example, if hand evaluation values ​​have been estimated for nine gripping postures, from the first gripping posture to the ninth gripping posture, in a gripping movement, the hand evaluation count for the gripping movement is 9. 【0071】 If the number of hand evaluations is equal to or greater than the hand evaluation threshold, it is determined that a sufficient number of evaluations have been performed, and the process proceeds to step S119. If the robot operation involves multiple hand movements, the process proceeds to step S119 only if the number of hand evaluations for each hand movement is equal to or greater than the hand evaluation threshold. On the other hand, if the number of hand evaluations is less than the hand evaluation threshold, it is determined that a sufficient number of evaluations have not been performed, and the process proceeds to step S111. If even one hand movement has a hand evaluation count below the hand evaluation threshold, the process proceeds to step S111. As a result, the system repeatedly selects a different movement posture and estimates the hand evaluation value until the number of hand evaluations equals or exceeds the hand evaluation threshold. The hand evaluation threshold may be a common value for all hand movements, or it may be a different value for each hand movement. 【0072】 In step S119, the hand evaluation value estimation unit 158 ​​estimates the hand evaluation value distribution. The hand evaluation value distribution is a distribution that shows the magnitude of the hand evaluation value for each operating posture. For example, the hand evaluation value distribution is estimated by plotting the hand evaluation value for each imaging posture, the hand evaluation value for each grasping posture, and the hand evaluation value for each placement posture. Various types of distributions can be estimated as the hand evaluation value distribution, such as a uniform distribution, a normal distribution, and a mixed distribution. After estimating the hand evaluation value distribution, the hand evaluation value distribution estimation process is completed, and the process proceeds to step S130. 【0073】 In step S130, the arm evaluation value estimation unit 159 estimates the arm evaluation value distribution. The details of the arm evaluation value distribution estimation process are described below. 【0074】 In Figure 6, when the arm evaluation value distribution estimation process is started, the estimation unit 150 selects the operating posture for each arm operation for which the arm evaluation value is to be estimated in step S131. The operating postures include, for example, the initial posture, imaging posture, grasping posture, tearing posture, placement posture, and retraction posture. Below, we assume the case where arm evaluation values ​​are estimated for posture changes from the initial posture to the imaging posture, posture changes from the imaging posture to the grasping posture, posture changes during the tearing operation, posture changes from the posture after the tearing operation to the placement posture, posture changes from the placement posture to the retraction posture, and posture changes from the retraction posture to the initial posture. In this case, the operation start posture, which is the posture at which each operation begins, and the operation end posture, which is the posture at which each operation ends, are selected randomly. However, if there are no multiple posture candidates for the robot 10, the operating posture will not be selected randomly, but rather one operating posture will be continuously selected. 【0075】 In the arm evaluation value distribution estimation process, similar to the hand evaluation value estimation process, the movement posture may include intermediate movement postures. For example, in arm movement, the posture change from the imaging posture to the grasping posture may be divided into two posture changes: the posture change from the imaging posture to the intermediate movement posture, and the posture change from the intermediate movement posture to the grasping posture. Alternatively, multiple intermediate movement postures may be set, dividing the movement into three or more posture changes. If intermediate movement postures exist, the arm evaluation value will be estimated for both the starting posture and the intermediate movement posture. Below, the arm evaluation value distribution estimation process will be explained using the case where intermediate movement postures are not included as an example. After selecting the movement posture, proceed to step S132A. 【0076】 In step S132A, the feasibility estimation unit 151C estimates whether the arm movement is possible. The feasibility estimation unit 151C estimates whether the robot 10 can take on the posture change from the start posture to the end posture based on the existence of a solution in the inverse kinematics calculation. Here, multiple candidates, i.e., multiple movement trajectories, can be assumed for the posture change from the start posture to the end posture. Therefore, if there is a movement trajectory for which even one solution in the inverse kinematics calculation exists, it can be estimated that the robot 10 can take on the posture change from the start posture to the end posture. 【0077】 For example, consider a case where we estimate whether or not the robot can perform a change in posture from the initial posture to the imaging posture. In this case, we estimate whether or not the robot 10 can perform the change in posture from the initial posture to the imaging posture. If the robot 10 cannot perform the change in posture from the initial posture to the imaging posture, we estimate that it cannot perform the action. After estimating whether or not it can perform the action, we proceed to step S132B. 【0078】 In step S132B, the maneuverability estimation unit 152 estimates the maneuverability. The maneuverability is estimated with respect to the starting position of the arm movement. For example, when estimating the maneuverability for the change in position from the initial position to the imaging position, the maneuverability in the initial position is estimated by calculating the maneuverable ellipsoid in the initial position. After estimating the maneuverability, the process proceeds to step S132C. 【0079】 In step S132C, the estimation unit 150 estimates the number of candidate arm movement trajectories. The number of candidate trajectories is the number of selectable movement trajectories from the start of movement to the end of movement. For example, suppose that in the approach process, both a clockwise approach and a counterclockwise approach are operable and selectable without interference from obstacles as movement trajectories from the imaging position to the grasping position. In this case, it can be estimated that at least two movement trajectories are selectable. If the clockwise approach interferes with an obstacle and that movement trajectory cannot be taken, the clockwise approach is not included in the selectable movement trajectories. 【0080】 For example, suppose there are three possible movement trajectories for a tearing operation: pulling downwards, pulling towards the user, and pulling upwards. If these three movement trajectories are feasible and do not interfere with obstacles, it can be estimated that at least three movement trajectories are selectable: downwards, towards the user, and upwards. After estimating the number of trajectory candidates, proceed to step S133. 【0081】 In step S133, the interference estimation unit 153 estimates the interference probability of the arm movement. Similar to the interference probability of the hand movement, the interference probability of the arm movement includes the starting posture interference probability and the movement interference probability. The movement interference probability includes the probability that the arm will interfere with an obstacle during the posture change movement from the starting posture to the ending posture, and will not be able to take that posture. For example, in a tearing motion, if an obstacle is present in the trajectory in which the pulling motion is performed and the arm cannot change posture to the ending posture of the pulling motion, the interference probability is set to 100%. 【0082】 Even if interference occurs with an obstacle, if it does not affect the execution of the operation, the interference probability may be set to 0%. For example, in the pulling motion of a picking operation, if the arm 12 interferes with the tomato support and cannot assume the desired pulling motion position, the interference probability is set to 100%. On the other hand, in the pulling motion, if the arm 12 interferes with the tomato leaves, but the arm 12 can still assume the desired position, the interference probability is set to 0%. 【0083】 The interference probability is determined by selecting the higher of the two probabilities: the starting position interference probability and the motion interference probability. For example, if the starting position interference probability is 0% and the motion interference probability is 100%, the interference probability is set to 100%. After estimating the interference probability, proceed to step S134. 【0084】 In step S134, the interference estimation unit 153 estimates whether the arm movement has any adverse effects on the surroundings. For example, if the arm 12 interferes with the tomato stem during the picking operation, causing the stem to break, it is estimated that there is an adverse effect. For example, if the arm 12 interferes with the support pole that holds the tomato during the picking operation, causing damage to both the arm 12 and the tomato pole, it is estimated that there is an adverse effect. After estimating whether there is an adverse effect, the process proceeds to step S135. 【0085】 In step S135, the success rate estimation unit 151 estimates the success rate of the arm movement. The success rate is estimated by considering factors other than those considered in the feasibility of movement and interference probability. For example, the success rate is estimated by whether the tomato can be held from the start position to the end position of the picking movement. In this case, the success rate of the picking movement can be estimated based on whether a grip that satisfies the force closure can be maintained. 【0086】 The success rate of arm movements can be calculated by dividing the number of successful attempts by the number of attempts. For example, if the same posture change from the first initial posture to the first imaging posture is repeated 10 times and succeeds 9 times, the number of attempts is 10 and the number of successes is 9. Therefore, the success rate of that posture change can be estimated to be 90%. 【0087】 If there are no factors other than the feasibility of operation and the probability of interference that contribute to the success of the operation, the estimation of the success rate of the operation may be omitted. For example, when changing the posture from the initial posture to the imaging posture, no fine motor work using the hands, such as grasping, is involved. For this reason, the estimation of the success rate of the operation can be omitted. After estimating the success rate of the operation, proceed to step S136. 【0088】 In step S136, the success rate estimation unit 154 estimates the success rate of a series of harvesting operations, including arm movements. The success rate is estimated in relation to the movement posture and posture change of each movement. For example, consider a harvesting operation that includes a posture change from a first initial posture to a first imaging posture. If the harvesting operation is successful, the number of executions and successes of the posture change from the first initial posture to the first imaging posture are increased by 1. Furthermore, the number of executions and successes of each movement and posture change included in this harvesting operation are also increased by 1. On the other hand, if the harvesting operation fails, the number of executions of the posture change from the first initial posture to the first imaging posture is increased by 1, but the success count is not increased. Furthermore, the number of executions and successes of each movement and posture change included in this harvesting operation are also increased by 1, but the success count is not increased. After estimating the success rate, the process proceeds to step S137. 【0089】 In step S137, the arm evaluation value estimation unit 159 estimates the arm evaluation value. The arm evaluation value estimation unit 159 estimates the arm evaluation value based on whether it can be operated, the degree of operability, the number of trajectory candidates, the interference probability, the presence or absence of adverse effects, the success rate of the operation, and the success rate of the work. The arm evaluation value, like the hand evaluation value, is a numerical value from 1.0 to 0, and a higher number indicates a higher evaluation, meaning it is an operation posture that should be adopted as the target posture. 【0090】 The arm evaluation value estimation unit 159 estimates a higher arm evaluation value when the arm is operational compared to when it is not operational. For example, if the arm is operational, the arm evaluation value is set to 1.0, and if the arm is not operational, the arm evaluation value is set to 0. 【0091】 The arm evaluation value estimation unit 159 estimates a higher arm evaluation value the higher the degree of maneuverability. For example, if the degree of maneuverability is equal to or greater than the upper threshold, the arm evaluation value is set to 1.0, and if the degree of maneuverability is less than the lower threshold, the arm evaluation value is set to 0. Also, if the degree of maneuverability is between the upper and lower thresholds, the arm evaluation value is set between 1.0 and 0. In this case, the higher the degree of maneuverability, the higher the arm evaluation value is set. 【0092】 The arm evaluation value estimation unit 159 estimates a higher arm evaluation value the more trajectory candidates there are. For example, if the number of trajectory candidates is greater than or equal to the upper threshold, the arm evaluation value is set to 1.0, and if the number of trajectory candidates is less than the lower threshold, the arm evaluation value is set to 0. Also, if the number of trajectory candidates is between the upper and lower thresholds, the arm evaluation value is set between 1.0 and 0. In this case, the more trajectory candidates there are, the higher the arm evaluation value is set. 【0093】 The arm evaluation value estimation unit 159 estimates a higher arm evaluation value the lower the interference probability. For example, if the interference probability is 0%, the arm evaluation value is set to 1.0, and if the interference probability is 100%, the arm evaluation value is set to 0. 【0094】 The arm evaluation value estimation unit 159 estimates a higher arm evaluation value when there are no adverse effects on the surroundings compared to when there are adverse effects. For example, the arm evaluation value is set to 1.0 when there are no adverse effects on the surroundings, and to 0 when there are no adverse effects. 【0095】 The arm evaluation value estimation unit 159 estimates a higher arm evaluation value the higher the success rate of the operation. For example, if the success rate of the operation is equal to or greater than the upper threshold, the arm evaluation value is set to 1.0, and if the success rate of the operation is less than the lower threshold, the arm evaluation value is set to 0. Also, if the success rate of the operation is between the upper and lower thresholds, the arm evaluation value is set between 1.0 and 0. In this case, the higher the success rate of the operation, the higher the arm evaluation value is set. 【0096】 The arm evaluation value estimation unit 159 estimates a higher arm evaluation value the higher the work success rate. For example, if the work success rate is above the upper threshold, the arm evaluation value is set to 1.0, and if the work success rate is below the lower threshold, the arm evaluation value is set to 0. Also, if the work success rate is between the upper and lower thresholds, the arm evaluation value is set between 1.0 and 0. In this case, the higher the work success rate, the higher the arm evaluation value is set. After estimating the arm evaluation value, the process proceeds to step S138. 【0097】 In step S138, the arm evaluation value estimation unit 159 estimates whether the number of arm evaluations, which is the number of arm evaluation values ​​estimated, is equal to or greater than the arm evaluation threshold. If the number of arm evaluations is equal to or greater than the arm evaluation threshold, it is determined that a sufficient number of evaluations have been performed and the process proceeds to step S139. On the other hand, if the number of arm evaluations is less than the arm evaluation threshold, it is determined that a sufficient number of evaluations have not been performed and the process proceeds to step S131. As a result, the estimation of arm evaluation values ​​is repeatedly performed by selecting a different operating posture and posture change until the number of arm evaluations becomes equal to or greater than the arm evaluation threshold. 【0098】 It is preferable to set the arm evaluation threshold to a value greater than the hand evaluation threshold. This allows for the estimation of the arm's posture using more training data for arm movements, which tend to be more complex and involve larger movements than hand movements, compared to the case of hand movements. Therefore, it is easier to estimate the posture of arm movements with greater accuracy. In addition, it is easier to reduce the processing load and processing time involved in estimating the posture of hand movements. 【0099】 In step S139, the arm evaluation value estimation unit 159 estimates the arm evaluation value distribution. The arm evaluation value distribution is a distribution that shows the magnitude of the arm evaluation value for each operating posture. Similar to the hand evaluation value distribution, the arm evaluation value distribution can be estimated using various types of distributions such as a uniform distribution, a normal distribution, or a mixed distribution. After estimating the arm evaluation value distribution, the arm evaluation value distribution estimation process is completed, and the process proceeds to step S151. 【0100】 In step S151, the integration unit 146 integrates the hand evaluation value and arm evaluation value, which are the estimation results of the estimation unit 150. Here, the hand evaluation value is the evaluation value of the hand movement, which is an action that does not involve the movement of the arm 12, and the arm evaluation value is the evaluation value of the arm movement, which is an action that involves the movement of the arm 12. Therefore, by integrating the hand evaluation value and the arm evaluation value, an overall evaluation value is calculated, which is an evaluation value of the movement posture and posture change in the actions of each process necessary for robot work. 【0101】 The integration unit 146 can calculate an overall evaluation value by multiplying the hand evaluation value and the arm evaluation value. More specifically, the hand evaluation value and the arm evaluation value are multiplied for the same operating posture. For example, the hand evaluation value in the first imaging posture is multiplied by the arm evaluation value during the posture change from the first imaging posture to the gripping posture. In this way, the overall evaluation value for the entire robot operation is calculated by multiplying the hand evaluation value and the arm evaluation value for the same operating posture. 【0102】 The method for calculating the overall evaluation value by the integration unit 146 is not limited to the example described above. For example, the integration unit 146 may calculate the overall evaluation value from a linear sum of weighted values ​​of the hand evaluation value and the arm evaluation value. The weighting of the movement postures in each posture can be set according to their importance in the robot task. For example, if the robot task includes even one movement posture that the robot 10 cannot assume, the robot 10 will not be able to perform that task. For this reason, the weight of the arm evaluation value based on whether or not it can move may be set heavier than that of the other evaluation values. 【0103】 The hand and arm evaluation values ​​for work success rate can change due to various factors. For example, if the shape of the tomato to be harvested is irregular and outside the standard size, and the grasping operation fails, the grasping operation may have been successful even with the same grasping posture if the tomato shape was within the standard size. Thus, the work success rate is related to various factors other than the movements of the robot 10. For this reason, the weight of the evaluation values ​​for hand and arm movements based on the work success rate may be lighter than that of other evaluation values. In other words, the more an evaluation value is related to factors other than the movements of the robot 10, the lighter its weight may be. 【0104】 The integration unit 146 changes the weighting method according to the content of the operation. For example, consider the case where the arm evaluation value for the posture change from the imaging posture to the grasping posture and the hand evaluation value for the grasping posture are integrated. In this case, the further away the position is from the target object (tomato), the heavier the arm evaluation value for the posture change from the imaging posture to the grasping posture is made, and the closer the position is to the tomato, the lighter the arm evaluation value for the posture change from the imaging posture to the grasping posture is made. This is because, in the operation leading up to grasping, the further away the position is from the target object (tomato), the higher the possibility of interference with obstacles during the approach process, which is the posture change from the imaging posture to the grasping posture, and therefore the importance of the arm evaluation value is relatively higher than that of the hand evaluation value. 【0105】 Weighting based on the nature of the action can be achieved by defining action intervals according to the nature of the action and changing the weighting method for each action interval. For example, the action interval in which the action in the approach process is performed is called the approach action interval, and the action interval in which the action in the ripping process is performed is called the ripping action interval. In the approach action interval, the further away from the object, the heavier the weight of the arm evaluation value and the lighter the weight of the hand evaluation value. On the other hand, in the ripping action interval, the weights of the hand evaluation value and the arm evaluation value are set to the same weight. 【0106】 In this way, for the same motion posture common to both hand and arm movements, weights are assigned to the hand evaluation value and the arm evaluation value, and the evaluation values ​​are integrated. Therefore, it is possible to change which of the hand evaluation value and the arm evaluation value is prioritized depending on the movements included in the robot operation. This weighting is not limited to integration by a linear sum of weights; the same approach can be applied to multiplication or other methods. For example, different weights may be assigned to the hand evaluation value and the arm evaluation value, and then the evaluation value may be integrated by multiplication of the hand evaluation value and the arm evaluation value to estimate the overall evaluation value. 【0107】 The integration unit 146 may prioritize the hand evaluation value and the arm evaluation value to calculate the overall evaluation value. For example, the evaluation value with the larger number of evaluations between the hand evaluation value and the arm evaluation value may be prioritized. This allows the system to adopt the evaluation value that has a larger number of evaluations and is likely to have been evaluated with higher accuracy for any given movement posture where two evaluation values ​​exist. However, the method of prioritizing is not limited to the number of evaluations. For example, if the arm evaluation value based on feasibility of movement is 0, the arm evaluation value may be prioritized, and if the arm evaluation value based on feasibility of movement is 1.0, the hand evaluation value may be prioritized. Alternatively, if the amount of movement from the start posture to the end posture is greater than or equal to a threshold, the arm evaluation value may be prioritized, and if the amount of movement from the start posture to the end posture is less than a threshold, the hand evaluation value may be prioritized. After calculating the overall evaluation value, the system proceeds to step S191. 【0108】 In step S191, the output unit 149 outputs the overall target posture. For the overall target posture, the operating posture with the highest overall evaluation value for each operation is selected. More specifically, the posture with the highest overall evaluation value is selected as the target posture for each of the initial posture, imaging posture, grasping posture, detachment posture, mounting posture, and retraction posture. Then, the combination of the selected target postures is determined as the overall target posture and output to the drive control device 190. This completes the posture estimation control using the posture estimation device 100. 【0109】 The effects of the above-described embodiment will now be explained. According to the above-described embodiment, the posture estimation device 100 includes an integration unit 146 that integrates the hand evaluation value and the arm evaluation value to calculate a total evaluation value, which is an evaluation value of the operating posture in robot work. Therefore, it is possible to evaluate not only the operating posture of the hand 13 but also the overall operating posture of the robot 10, including the arm 12, and calculate the total evaluation value. Accordingly, it is easier to estimate an appropriate target posture in robot work compared to a configuration that calculates the total evaluation value by considering only the hand evaluation value. 【0110】 Furthermore, the posture estimation device 100 estimates hand evaluation values ​​and arm evaluation values ​​separately and integrates the estimation results. Therefore, the estimation of hand evaluation values ​​and arm evaluation values ​​can be performed by different estimation entities. Consequently, even if the hand movement is changed, if the arm movement remains unchanged, the previously estimated arm evaluation values ​​can be maintained, and the posture estimation process can be executed. 【0111】 Furthermore, the overall operating posture of the robot 10, including the arm 12, is also evaluated to calculate a comprehensive evaluation value. Therefore, even for operations involving arm movements, such as twisting movements that simultaneously perform rotational and linear movements, the target posture can be accurately estimated. Consequently, the robot work system 1 can be applied even to robotic operations involving large movements of the arm 12, such as crop harvesting. In other words, the types of robotic operations to which the robot work system 1 can be applied can be expanded. 【0112】 The integration unit 146 integrates the hand evaluation value and the arm evaluation value using one of the following methods: multiplication, weighted linear sum, or prioritization. Therefore, it is easier to appropriately integrate the evaluation values ​​compared to simply adding the hand evaluation value and the arm evaluation value together. For example, when integrating by simple summation, even if the arm evaluation value for operation feasibility is 0, if the other evaluation values ​​are high, the overall evaluation value may become the highest. However, an arm evaluation value of 0 for operation feasibility means that the robot 10 cannot take on the posture change from its operation start posture to its operation end posture. In other words, an operation posture that is actually impossible to perform is included in the target posture, making it unsuitable as a robot operation. Thus, by using one of the following methods: multiplication, weighted linear sum, or prioritization, it is easier to suppress the occurrence of a high overall evaluation value for a robot operation that includes an operation posture unsuitable for robot operations. 【0113】 The hand evaluation value estimation unit 158 ​​estimates the hand evaluation value based on at least the interference probability and the work success rate. Therefore, it can estimate the hand evaluation value with greater accuracy compared to estimating the hand evaluation value based on either the interference probability or the work success rate alone. 【0114】 The arm evaluation value estimation unit 159 estimates the arm evaluation value based on at least the feasibility of operation, the probability of interference, and the success rate of the operation. Therefore, it can estimate the arm evaluation value with greater accuracy compared to cases where the arm evaluation value is estimated without considering feasibility of operation, the probability of interference, or the success rate of the operation. 【0115】 The arm evaluation value estimation unit 159 estimates the arm evaluation value based on the feasibility of movement and the interference probability. Here, feasibility of movement is an indicator that shows whether the robot 10 can perform the movement assuming that there are no obstacles in the surroundings. On the other hand, the interference probability is an indicator that shows the possibility that the robot 10 may not be able to perform the movement due to interference with an obstacle. Therefore, it is possible to evaluate separately situations in which movement cannot be performed regardless of obstacles and situations in which movement cannot be performed due to obstacles. Consequently, it is easier to learn evaluation values ​​appropriately compared to when it is not possible to distinguish whether the inability to perform movement is due to an obstacle or not. Thus, an improvement in learning convergence and a reduction in the amount of training data can be expected. 【0116】 The integration unit 146 changes the weighting method according to the content of the operation and integrates the hand evaluation value and the arm evaluation value. Therefore, when integrating the hand evaluation value and the arm evaluation value, it is easier to estimate an appropriate overall evaluation value compared to when no weighting is applied. 【0117】 The estimation unit 150 includes a work success rate estimation unit 154 that estimates the work success rate obtained by dividing the number of times the robot work was successfully performed by the number of times the robot work was performed. Therefore, it is possible to estimate the evaluation value while taking into account how successful the work actually is. Consequently, it is easier to estimate the appropriate operating posture with greater accuracy compared to when the evaluation value is estimated without considering the work success rate. 【0118】 When using robotic work system 1 as an automated harvesting system, the shape and size of the crops to be harvested are not uniform but irregular. Therefore, in order to accurately estimate the posture of robot 10, it is very useful to understand the environment surrounding robot 10 and to make predictions using a learning-based method. 【0119】 (Second Embodiment) This embodiment is a modification based on the preceding embodiment. In this embodiment, the search unit 248 searches for the overall evaluation value distribution and outputs the overall target attitude based on the search results. 【0120】 In Figure 7, the processor 140 includes a search unit 248. The search unit 248 searches the overall evaluation value distribution to find a suitable combination of operating postures from among various candidate operating postures for robot work. Using the automatic harvesting of tomatoes as an example, the search process, which is a difference from the first embodiment, will be explained below with reference to the figures. 【0121】 In Figure 8, after the integration unit 146 integrates the estimation results in step S151, the process proceeds to step S280. In step S280, the search unit 248 performs a search based on the integrated overall evaluation value distribution. In the search, a combination of motion postures necessary for the robot operation is set, and a determination is made as to whether the robot operation can be executed with that combination of motion postures to achieve the task. Details of the search process are described below. 【0122】 In Figure 9, when the search process is started, the search unit 248 selects a search interval in step S281A. In selecting the search interval, the search interval is selected from intervals that are included in the maneuverable ellipsoid. This is because it is clear without searching that the robot 10 cannot assume any operating posture that is not included in the maneuverable ellipsoid. 【0123】 In selecting the search interval, priority is given to selecting intervals that contain movement postures with high overall evaluation values ​​in the overall evaluation value distribution. At this time, it is preferable to select the search interval with a coarser granularity than the granularity of the intervals that show the overall evaluation value of movement postures in the overall evaluation value distribution. For example, nine intervals are selected as the search interval, consisting of the one interval with the highest overall evaluation value and eight intervals adjacent to that interval with the highest overall evaluation value. However, any intervals among the nine that are not included in the manipulable ellipsoid are excluded from the search interval. In this way, the intervals surrounding the interval containing the movement posture with the highest overall evaluation value are selected as the search interval. 【0124】 However, the section containing the operating posture with the highest overall evaluation value does not necessarily contain the optimal operating posture for actual harvesting work. Therefore, it is not always necessary to select the section containing the operating posture with the highest overall evaluation value and the surrounding sections as the search section. In particular, if the distribution of overall evaluation values ​​is multimodal, the overall evaluation value may be low in the sections surrounding the section showing the highest peak, and the sections showing peaks in overall evaluation values ​​may be concentrated adjacent to each other in other sections. In this case, it may be possible to efficiently select an appropriate target posture by selecting the section containing the section where the sections showing peaks in overall evaluation values ​​are concentrated adjacent to each other as the search section. After selecting the search section, proceed to step S281B. 【0125】 In step S281B, the search unit 248 selects the operating posture. The operating postures include the initial posture, imaging posture, grasping posture, retrieval posture, placement posture, and retraction posture, and the search unit selects the appropriate posture. In the search section, the operating posture with the highest overall evaluation value is selected preferentially. More specifically, the operating postures are selected in order of highest overall evaluation value. For example, if the overall evaluation value of the first grasping posture is higher than that of the first grasping posture, the first grasping posture is selected before the second grasping posture is selected. However, the selection of operating postures is not limited to the order of highest overall evaluation value. 【0126】 For example, the probability of selecting a movement posture in the search may be set so that the higher the overall evaluation value, the higher the probability of selecting that posture. In this case, since a movement posture with a high overall evaluation value is more likely to be selected, the search can be performed more efficiently. Furthermore, even if the overall evaluation value is low, there is still a small chance that it may be selected as a movement posture. Therefore, even if a movement posture with a low overall evaluation value is actually the optimal movement posture, it may be possible to find that movement posture early through the search. This is particularly useful when the estimation accuracy of the overall evaluation value is low, such as when there is a large discrepancy between the situation of the learning data for harvesting work and the situation of the current harvesting work. After selecting a movement posture, proceed to step S282. 【0127】 In step S282, the feasibility estimation unit 151C estimates whether the arm movement is possible. The feasibility estimation unit 151C estimates that the movement is possible if a solution for the inverse kinematics calculation exists, and estimates that the movement is impossible if a solution for the inverse kinematics calculation does not exist. After estimating the feasibility, the process proceeds to step S283. 【0128】 In step S283, the interference estimation unit 153 estimates the probability of interference between the hand movement and the arm movement. If the robot 10 interferes with an obstacle in that movement position and cannot assume that position, the interference probability is estimated to be 100%. On the other hand, if the robot 10 does not interfere in that movement position, or if there is only minor interference but it can still assume that position, the interference probability is estimated to be 0%. After estimating the interference probability, the process proceeds to step S284. 【0129】 In step S284, the interference estimation unit 153 estimates whether the arm movement has any adverse effects on the surroundings. After estimating whether or not there are adverse effects, the process proceeds to step S286. 【0130】 In step S286, the success rate estimation unit 154 estimates the success rate of a series of harvesting operations, including hand and arm movements. The series of harvesting operations is repeated on the simulator with the selected movement posture, and the number of times the operations are performed and the number of successful operations are recorded. The success rate is obtained by dividing the number of successful operations by the number of times the operations are performed. After estimating the success rate, the process proceeds to step S287. 【0131】 In step S287, the search unit 248 estimates a search evaluation value for a series of harvesting operations combining different movement postures. The search evaluation value can be estimated as a numerical value between 0 and 1.0. The higher the search evaluation value, the higher the evaluation, meaning that it is a movement posture that should be adopted as the target posture. 【0132】 The search unit 248 estimates a search evaluation value based on the feasibility of operation, the probability of interference, the presence or absence of adverse effects, and the success rate of the work. The search unit 248 sets the search evaluation value to 1.0 if operation is possible, the interference probability is 0%, there are no adverse effects, and the success rate of the work is above the upper threshold. In other words, the search unit 248 sets the search evaluation value to 1.0 if it can assume that the operation posture can be taken while considering obstacles, there are no adverse effects on the surroundings, and the success rate of the work is estimated to be high. On the other hand, if operation is impossible or the interference probability is 100%, the search evaluation value is set to 0. Furthermore, if there are adverse effects or the success rate of the work is below the upper threshold, the search evaluation value is set to a value lower than 1.0. 【0133】 The method for estimating the exploration evaluation value is not limited to the examples described above. For example, the exploration evaluation value may be estimated based on feasibility of operation, interference probability, and operation success rate, without considering the presence or absence of adverse effects. Alternatively, the exploration evaluation value may be estimated by additionally considering the operation success rate. After estimating the exploration evaluation value, proceed to step S288. 【0134】 In step S288, the search unit 248 determines whether the search evaluation value is above a threshold. If the search evaluation value is above the threshold, it is determined that the harvesting operation can be performed without problems by using that combination of movement postures as the target posture, and the process proceeds to step S289. On the other hand, if the search evaluation value is below the threshold, it is determined that the harvesting operation cannot be performed properly with that combination of movement postures, and the process proceeds to step S271. 【0135】 In step S271, the search unit 248 increments the search count by one. The search count is information indicating the number of times the search evaluation value has been estimated for each combination of motion postures in that search interval. The higher the search count, the more times multiple combinations of motion postures have been selected and the search evaluation value estimated in that search interval. For example, if the search count is 9, it means that 9 patterns of multiple combinations of motion postures have been tried and the search evaluation value for each has been estimated. If it is the first time a search is performed in that search interval, the search count is 1. If 9 searches have already been performed in that search interval, the search count increases by one to 10. After incrementing the search count by one, the process proceeds to step S272. 【0136】 In step S272, the search unit 248 determines whether the number of searches is equal to or greater than a threshold. If the number of searches is equal to or greater than a threshold, it determines that there is not enough room for further searching in that search interval and that it is necessary to perform a search in a different search interval, and proceeds to step S273. On the other hand, if the number of searches is less than a threshold, it determines that there is still room for further searching in that search interval and that it is necessary to continue searching in the same search interval, and proceeds to step S281B. As a result, the search is repeated in the same search interval, changing the combination of operating postures, until the number of searches is equal to or greater than a threshold. 【0137】 In step S273, the search unit 248 resets the search count. In other words, the search count is set to 0, and the process proceeds to step S281A. This selects a new search interval, and the search process is repeated until a combination of operating postures is found in which the search evaluation value is equal to or greater than the threshold. 【0138】 In step S289, the search unit 248 outputs the search results. The search results output are combinations of operating postures for which the search evaluation value is equal to or greater than a threshold. After outputting the search results, the search process ends and the process proceeds to step S291. 【0139】 In Figure 8, in step S291, the output unit 149 outputs the overall target attitude. The overall target attitude is selected from a combination of operating attitudes in which the search evaluation value is equal to or greater than a threshold. The overall target attitude is output to the drive control device 190, and the attitude estimation control using the attitude estimation device 100 is terminated. 【0140】 The effects of the above-described embodiment will now be explained. According to the above-described embodiment, the search is prioritized over searching for intervals with high overall evaluation values ​​in the distribution of overall evaluation values. Here, operating postures with high overall evaluation values ​​tend to have higher search evaluation values ​​than operating postures with low overall evaluation values. Therefore, it is easier to efficiently find operating postures where the search evaluation value is above a threshold. 【0141】 The search unit 248 estimates the search evaluation value of the operating posture in the searched section based on at least the feasibility of operation, the probability of interference, and the success rate of the task. Therefore, the search evaluation value can be estimated from two perspectives: the feasibility perspective, which indicates whether the robot 10 can actually assume that operating posture, and the success probability perspective, which indicates whether the robot task can be successfully performed by executing that operating posture. Consequently, it is easier to estimate the search evaluation value with greater accuracy compared to estimating the search evaluation value without including the two perspectives of feasibility and success probability. 【0142】 The hand evaluation value estimation unit 158 ​​estimates the distribution of hand evaluation values ​​using a learning-based method, and the arm evaluation value estimation unit 159 estimates the distribution of arm evaluation values ​​using a learning-based method. Therefore, even for robot operations that the robot 10 has never performed before, evaluation values ​​can be estimated with high accuracy. Consequently, the distribution of evaluation values ​​can be estimated more efficiently compared to estimating the distribution of hand evaluation values ​​and arm evaluation values ​​without using a learning-based method. 【0143】 The estimation unit 150 includes an operability estimation unit 152 that estimates the operability based on robot information input to the input device 110. Therefore, it is easier to estimate the operability with greater accuracy compared to estimating the operability without using robot information. Furthermore, compared to a configuration in which the robot 10 is actually moved to directly estimate and learn the range of motion, it is possible to estimate and learn the range of motion with a smaller number of parameters. 【0144】 The search unit 248 determines the search range from within the range of the maneuverable ellipsoid, and excludes areas outside the range of the maneuverable ellipsoid. This prevents the robot 10 from searching for operating postures beyond its actual range of motion. Consequently, it prevents unnecessary searches and enables efficient searching. 【0145】 In the search process, it is not necessary to directly use the overall evaluation value distribution obtained from training using a deep learning model. For example, the search can be performed using two networks: one based on a deep learning model and another trained using a method called CVAE (Conditional Variational Auto Encoder). CVAE stands for Conditional Variational Auto Encoder, and it is a method for efficiently generating data that meets the objective by incorporating additional conditions and information into the model. The granularity of the operating posture of a network trained with CVAE can be finer than the granularity of the operating posture of a network based on a deep learning model. 【0146】 The following describes a case where search processing is performed using two networks. The deep learning model network is input with robot information such as joint angles and raw point cloud data as environmental data, and a single trial is performed to obtain its output. The output obtained is, for example, the end-effector position of hand 13, the approach vector, and the environmental data. Then, the obtained output is input to the network trained with CVAE, and the search is performed multiple times until the optimal solution is found, obtaining the end-effector rotation angle as the optimal solution. In this way, by combining rough sampling using the deep learning model network with fine-grained sampling using the CVAE-trained network, the search can be performed efficiently. 【0147】 (Other embodiments) The disclosures in this specification and drawings are not limited to the exemplary embodiments. The disclosures include the exemplary embodiments and variations thereof by those skilled in the art. For example, the disclosures are not limited to combinations of parts and / or elements shown in the embodiments. The disclosures are implementable in a variety of combinations. The disclosures may have additional parts that can be added to the embodiments. The disclosures include those in which parts and / or elements of the embodiments have been omitted. The disclosures include substitutions or combinations of parts and / or elements between one embodiment and another. The scope of the disclosed technical areas is not limited to the descriptions of the embodiments. Some of the scope of the disclosed technical areas are indicated by the claims and should be understood to include all modifications within the meaning and scope equivalent to the claims. 【0148】 The disclosures in the specification and drawings are not limited by the claims. The disclosures in the specification and drawings encompass the technical ideas described in the claims and extend to a wider and more diverse range of technical ideas than those described in the claims. Therefore, a variety of technical ideas can be extracted from the disclosures in the specification and drawings without being bound by the claims. 【0149】 The control unit and method described herein may be implemented by a dedicated computer comprising a processor programmed to perform one or more functions embodied by a computer program. Alternatively, the apparatus and method described herein may be implemented by a dedicated hardware logic circuit. Alternatively, the apparatus and method described herein may be implemented by one or more dedicated computers comprising a combination of a processor that executes a computer program and one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by the computer on a computer-readable non-transitional tangible recording medium. [Explanation of Symbols] 【0150】 1 Robot work system, 10 Robot, 12 Arm, 13 Hand, 100 Pose estimation device, 110 Input device, 120 Sensor, 130 Control device, 140 Processor, 141 Acquisition unit, 146 Integration unit, 149 Output unit, 150 Estimation unit, 151 Motion success rate estimation unit, 151A Occlusion rate estimation unit, 151B Gripping success rate estimation unit, 151C Operation feasibility estimation unit, 152 Operability estimation unit, 153 Interference estimation unit, 154 Work success rate estimation unit, 158 Hand evaluation value estimation unit, 159 Arm evaluation value estimation unit, 190 Drive control device, 248 Search unit.

Claims

[Claim 1] A posture estimation device that estimates the target posture of a robot (10) capable of performing robotic work, which is a task performed on an object by a robot (10) having a hand (13) and an arm (12) that are connected so as to be rotatable relative to each other, using a learning-based method, An acquisition unit (141) acquires environmental data including data relating to the object captured by the sensor (120), An estimation unit (150) having a hand evaluation value estimation unit (158) that estimates a hand evaluation value, which is an evaluation value of posture related to hand movements, which are movements of the robot that do not involve the movement of the arm, based on the environmental data, and an arm evaluation value estimation unit (159) that estimates an arm evaluation value, which is an evaluation value of posture related to arm movements, which are movements of the robot that involve the movement of the arm, based on the environmental data, An integration unit (146) integrates the hand evaluation value and the arm evaluation value to calculate an overall evaluation value, which is an evaluation value of the movement posture in the robot operation, A posture estimation device comprising an output unit (149) that outputs the overall target posture of the robot based on the overall evaluation value. [Claim 2] The posture estimation device according to claim 1, wherein the integration unit integrates the hand evaluation value and the arm evaluation value using one of the following methods: multiplication, weighted linear sum, or prioritization. [Claim 3] The estimation unit includes an operation feasibility estimation unit (151C) that estimates whether the arm movement is possible without considering surrounding obstacles, An interference estimation unit (153) estimates the probability of interference between the robot and the surrounding obstacles, It includes a work success rate estimation unit (154) that estimates the work success rate obtained by dividing the number of times the robot work was successfully performed by the number of times the robot work was performed, The hand evaluation value estimation unit estimates the hand evaluation value based at least on the interference probability and the work success rate, The posture estimation device according to claim 2, wherein the arm evaluation value estimation unit estimates the arm evaluation value based on at least the feasibility of operation, the interference probability, and the work success rate. [Claim 4] The hand evaluation value estimation unit estimates the distribution of the hand evaluation values ​​using the learning-based method, The arm evaluation value estimation unit estimates the distribution of the arm evaluation values ​​using the learning-based method, The integration unit integrates the distribution of the hand evaluation value and the distribution of the arm evaluation value to estimate the distribution of the overall evaluation value. The system includes a search unit (248) that prioritizes searching for intervals with high overall evaluation values ​​in the distribution of the overall evaluation values ​​over intervals with low overall evaluation values, and estimates the search evaluation value of the operating posture in the searched interval based on at least the feasibility of operation, the interference probability, and the work success rate. The attitude estimation device according to claim 3, wherein the output unit outputs the overall target attitude based on the search results of the search unit. [Claim 5] The system includes an input device (110) capable of inputting robot information, which is information relating to the robot, The estimation unit includes an operability estimation unit (152) that estimates the operability based on the robot information input to the input device. The posture estimation device according to claim 4, wherein the arm evaluation value estimation unit estimates the arm evaluation value based at least on the degree of maneuverability. [Claim 6] The maneuverability estimation unit calculates the maneuverable ellipsoid, The posture estimation device according to claim 5, wherein the search unit determines the search range from within the range of the manipulable ellipsoid, and excludes areas outside the range of the manipulable ellipsoid from the search range. [Claim 7] The estimation unit includes an operation feasibility estimation unit (151C) that estimates whether the arm movement is possible without considering surrounding obstacles, The system includes an interference estimation unit (153) that estimates the probability of interference between the robot and surrounding obstacles, The posture estimation device according to claim 1, wherein the arm evaluation value estimation unit estimates the arm evaluation value based at least on the feasibility of operation and the interference probability. [Claim 8] The estimation unit estimates an evaluation value for each of the multiple actions if the robot operation includes multiple actions. The posture estimation device according to claim 1, wherein the integration unit integrates the hand evaluation value and the arm evaluation value by changing the weighting method according to the content of the operation. [Claim 9] The estimation unit includes a work success rate estimation unit (154) that estimates the work success rate obtained by dividing the number of times the robot work was successful by the number of times the robot work was performed. The hand evaluation value estimation unit estimates the hand evaluation value based at least on the work success rate, The posture estimation device according to claim 1, wherein the arm evaluation value estimation unit estimates the arm evaluation value based at least on the work success rate.

Images