Robot system, robot control method, and robot control program

Abstract

A robot system according to the present invention comprises a skeleton estimation unit that estimates skeletal information that represents the skeleton of a workpiece in a real space on the basis of a workpiece image that includes the workpiece, a completion state determination unit that determines the completion state of work performed on the workpiece on the basis of the estimated skeletal information and constraint condition information that indicates constraint conditions related to work on the workpiece, and a robot control unit that makes a robot provided in the real space perform work that corresponds to the determined completion state.

Description

ROBOT SYSTEM, ROBOT CONTROL METHOD, AND ROBOT CONTROL PROGRAM

[0001] One aspect of the present disclosure relates to a robot system, a robot control method, and a robot control program.

[0002] Robots that handle items such as food and agricultural products have been developed.

[0003] International Publication No. 2021 / 085345

[0004] However, the workpieces that can be handled reliably are limited to regular shaped objects, and it is difficult to apply robots to work on irregularly shaped workpieces, for example.

[0005] Therefore, one aspect of the present disclosure aims to improve the reliability of work performed on a workpiece.

[0006] A robot system according to one aspect of the present disclosure includes a skeleton estimation unit that estimates skeleton information indicating the skeleton of a workpiece based on a workpiece image including the workpiece existing in real space, a completion state determination unit that determines the completion state of work to be performed on the workpiece based on constraint condition information that indicates constraint conditions related to work on the workpiece and the estimated skeletal information, and a robot control unit that causes a robot placed in real space to perform work according to the determined completion state.

[0007] According to one aspect of the present disclosure, it is possible to improve the reliability of work performed on a workpiece.

[0008] FIG. 1 is a diagram illustrating an example of the configuration of a robot system. FIG. 2 is a diagram illustrating a workpiece and its skeletal information. FIG. 3 is a diagram illustrating an example of the operation of the robot system. FIG. 4 is a diagram illustrating an example of the operation of the robot system. FIG. 5 is a diagram illustrating a result of the operation of the robot system. FIG. 6 is a diagram illustrating a result of the operation of the robot system. FIG. 7 is a diagram illustrating an example of the hardware configuration of a computer used for the robot system. FIG. 8 is a diagram illustrating the configuration of a robot system according to a modified example. FIG. 9 is a diagram illustrating an example of estimated key points of a workpiece. FIG. 10 is a diagram illustrating an example of a process for selecting a workpiece. FIG. 11 is a diagram illustrating an example of a processed workpiece. FIG. 12 is a diagram illustrating the configuration of a robot system according to a modified example. FIG. 13 is a diagram illustrating an example of a user interface for generating constraint information.

[0009] Various examples of the present disclosure will be described in detail below with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements are designated by the same reference numerals, and redundant description will be omitted.

[0010] [Configuration of Robot System] The configuration of an example robot system 100 will be described with reference to Figures 1, 2A, and 2B. Figure 1 is a diagram showing an example of the configuration of the robot system 100. Figures 2A and 2B are both diagrams showing a workpiece W and skeletal information K of the workpiece W.

[0011] The robot system 100 is a control system for causing a robot R placed in real space to process a workpiece W present in the real space. As shown in Fig. 1 , the robot system 100 includes a camera C, a skeleton estimation unit 110, a workpiece memory unit 112, a learning data generation unit 113, a learning unit 114, a workpiece selection unit 120, a start state determination unit 130, a work information memory unit 132, a completion state determination unit 140, a constraint information memory unit 141, an end state determination unit 142, a path generation unit 150, a robot control unit 160, and the robot R.

[0012] The robot R is a device or equipment that performs work on a workpiece W. The robot R is placed in real space. The robot R has an end effector that applies the action required for that work to the workpiece W. In the following explanation, it is assumed that the robot R is a vertical articulated robot, but the robot R may also be other types of robots, such as a horizontal articulated robot or a direct-acting robot.

[0013] The operation refers to a process performed by the robot R on the workpiece W. The operation can be a process that affects the shape of the workpiece W. For example, the operation includes at least one of handling the workpiece W and processing the workpiece W. Examples of handling include pick-and-place operations, alignment operations, packing operations, unpacking operations, and assembly operations. Examples of processing include welding operations, painting operations, grinding operations, polishing operations, cutting operations, and printing operations. In the following explanation, it is assumed that the operation is pick-and-place operations of the workpiece W by the robot R, and other operations will be explained as appropriate.

[0014] The robot system 100 can cause the robot R to perform work on various workpieces W. For example, the robot system 100 causes the robot R to perform work on an irregular workpiece W. The irregular workpiece W can be a workpiece that is difficult to deform but has individual differences in shape, such as broccoli, fried shrimp, or fried chicken. Alternatively, the irregular workpiece W can be a workpiece, i.e., a flexible object, such as a harness, cable, or handkerchief, where individual differences in shape are negligible but the degree of deformation between individual pieces can vary greatly. Alternatively, the irregular workpiece W can be a workpiece that combines the characteristics of these two types of workpieces. FIG. 2A shows broccoli as an example of an irregular workpiece W that has individual differences in shape.

[0015] The camera C captures an image of the workpiece W and generates a workpiece image that is an image including the workpiece W (i.e., an image that captures the workpiece W). The image may be a two-dimensional image, or may include a distance image in addition to the two-dimensional image. For example, the camera C is fixed to the tip of the arm of the robot R. The camera C may be fixed to another position on the robot R, or may be fixed to the ceiling or a pillar. Multiple cameras C may be arranged.

[0016] The skeleton estimation unit 110 is a functional module that estimates skeleton information K that indicates the skeleton of the workpiece W based on an image including the workpiece W. The skeleton information K indicates the skeleton of the workpiece W to an extent that allows the robot R to plan work on the workpiece W. The skeleton of the workpiece W represents the external shape of the workpiece W or the approximate positional relationship of the parts of the workpiece W. The workpiece W does not need to have a "bone." For example, broccoli has a structure in which multiple stalks extend from one stalk and each stalk connects to a bunch. If the workpiece W is broccoli, the skeleton information K indicates the approximate positional relationship of the parts to an extent that allows the robot R to perform work on the broccoli.

[0017] The skeletal information K of the workpiece W includes multiple keys, which are information indicating the approximate positional relationships of parts of the workpiece W. The multiple keys include multiple key points, which are information indicating the positions of specific parts of the workpiece W, and one or more skeletons, each of which is a virtual line connecting two key points. A skeleton is also called a bone. In the example of FIG. 2A , the skeletal information K of the workpiece W includes multiple keys, which include multiple key points P (P1 to P5) and multiple skeletons S (S1 to S4) of the workpiece W. When the workpiece image includes multiple workpieces W as in the example of FIG. 2B , the skeleton estimation unit 110 can estimate skeletal information K for each of the two or more workpieces W.

[0018] In one example, the skeleton estimation unit 110 estimates skeletal information K of the workpiece W using a skeleton estimation model 111. The skeleton estimation model 111 is an inference device that receives an input of a workpiece image including the workpiece W and estimates the skeletal information K of the workpiece W. The skeletal information K is estimated by the skeleton estimation unit 110 inputting a workpiece image of the workpiece W for which skeletal information K is to be estimated into the skeleton estimation model 111. The skeletal estimation model 111 can be realized using keypoint detection technology, such as OpenPose, RTMPoses, and YOLOX-Pose. In keypoint detection, the positions of specific parts of the workpiece W to be recognized are identified as keypoints P, and virtual lines (edges) representing the relationship between the two positions are inferred as skeletons S (bones). These keypoints P and skeletons S are processed as skeletal information K.

[0019] The skeleton estimation model 111 is generated in advance by machine learning using training data including a sample image, which is an image based on a three-dimensional model (3D model) of the workpiece W, and multiple data records in which multiple keys set in the 3D model are associated with each other. Therefore, the skeleton estimation model 111 is a trained model. The multiple keys in each data record also include multiple key points and one or more skeletons, each of which connects two key points. The robot system 100 has a work memory unit 112, a training data generation unit 113, and a training unit 114 as components for performing the training.

[0020] The work storage unit 112 is a functional module that stores, as learning data, sample images based on at least one 3D model of the work W and skeletal information K for each 3D model. The learning data generation unit 113 is a functional module that transforms a 3D model represented by at least one of the multiple data records of the learning data stored in the work storage unit 112 to generate new data records of the learning data and stores the new data records in the work storage unit 112. When a 3D model is transformed, the skeletal information K (multiple keys) for the transformed 3D model can be determined because the skeletal information K for the original 3D model is known. Therefore, a new data record in which the transformed 3D model and the determined skeletal information K are associated with each other can be stored in the work storage unit 112. The learning data generation unit 113 can increase the amount of learning data through such processing. In the present disclosure, the transformation of a 3D model may include not only a process of changing the outer shape of the 3D model to create individual differences in the outer shape of the work W, but also a process of creating individual differences in the degree of deformation of a 3D model of a flexible and therefore deformable object. The learning unit 114 is a functional module that trains the skeleton estimation model 111 by machine learning using training data that may include new data records, to generate the skeleton estimation model 111. The generation of the skeleton estimation model 111 by the learning unit 114 may include updating the skeleton estimation model 111.

[0021] The workpiece selection unit 120 is a functional module that selects at least one workpiece W to be processed in the operation based on skeletal information K estimated for each of the multiple workpieces W included in the workpiece image. If the workpiece image includes a single workpiece W, the workpiece selection unit 120 selects that workpiece W. As shown in FIG. 2B , the workpiece image may include multiple workpieces W. In this case, the workpiece selection unit 120 selects at least one workpiece W to be processed in the operation from the workpiece image.

[0022] For example, the workpiece selection unit 120 selects at least one workpiece based on the skeletal information K of each of the multiple workpieces W. The workpiece selection unit 120 may select a workpiece W having skeletal information K with a degree of deformation within a predetermined range, such as a straight workpiece W or a workpiece W with a small degree of bending. The workpiece selection unit 120 may select the current workpiece W based on its relationship with previously selected workpieces W so that the variation in the skeletal information K of the workpieces W to be processed sequentially falls within a predetermined threshold.

[0023] Alternatively, the workpiece selection unit 120 calculates the area on the workpiece image for each of the multiple workpieces W. Then, based on the area and skeletal information K of each of the multiple workpieces W, the workpiece selection unit 120 selects a workpiece that is entirely exposed in the workpiece image without being hidden or buried by other workpieces W. Based on the area and skeletal information K of each of the multiple workpieces W, the workpiece selection unit 120 may select a workpiece W whose skeletal information K is entirely exposed, or a workpiece W whose proportion of exposed skeletal information K is equal to or greater than a predetermined threshold.

[0024] The start state determination unit 130 is a functional module that determines a start state, which is the state of the robot R at the start of work, based on the skeletal information K of the workpiece W. The start state represents, for example, at least one of the position and orientation of the robot R when work on the workpiece W is started. The position and orientation of the robot R may be the position and orientation of the end effector. In a pick-and-place or alignment process, the start state represents, for example, the position and orientation of the end effector when picking up the workpiece W. In an assembly or machining process, the start state represents, for example, the position and orientation of the end effector at the start of the process.

[0025] For example, the start state determination unit 130 may analyze skeletal information K of the workpiece W, convert the skeletal information K into at least one of the position and orientation of the end effector when the end effector acts on the workpiece W (e.g., picks), and determine the result of the conversion as the start state. The start state determination unit 130 may perform the conversion using a function that receives input of the skeletal information K and calculates the start state, or may extract the start state by geometrically analyzing the skeletal information K. As another example, the start state determination unit 130 may perform the conversion using an inference unit that receives input of at least the skeletal information K and outputs the start state. Alternatively, the start state determination unit 130 may simulate various start states based on the skeletal information K, evaluate each simulation result using a predetermined evaluation method, and determine the start state that will yield the optimal simulation result.

[0026] The start state determination unit 130 may determine the start information based on the work information that is set according to the type of workpiece W or the work and stored in the work information storage unit 132, and the estimated skeletal information K. In one example, the work information is defined for each type of workpiece W, regardless of individual differences such as the shape of the workpiece W, and is associated with at least one of a plurality of keys of the workpiece W. The work information represents information about the work performed on the workpiece W. For example, the work information includes at least one of the position and posture of the end effector acting on the workpiece W in the start state of the work performed on the workpiece W.

[0027] In the case of a pick-and-place operation of the workpiece W by the robot R, the operation information includes pick information indicating, for each of at least one key on the workpiece W, whether the robot R can pick the position on the workpiece W corresponding to that key. The pick information may also indicate whether the robot R can pick the area around each of the key points P and the skeleton S. The start state determination unit 130 determines a pick position A on the workpiece W by the robot R as the start state based on the pick information and the estimated skeleton information K. The pick position A refers to the position at which the robot R acts on the workpiece W to pick it. For example, if the end effector is a hand, the pick position A represents a gripping position, and if the end effector is a suction pad, the pick position A represents a suction position. In the case of a processing operation, the operation information may also indicate, for each of multiple keys (each key point P and each skeleton S) on the workpiece W, whether processing can be started around that key.

[0028] The completion state determination unit 140 is a functional module that determines the completion state of the work performed on the workpiece W based on constraint condition information that represents constraint conditions related to the work on the workpiece W and the estimated skeletal information K. The robot system 100 causes the robot R to perform the work according to the determined completion state.

[0029] The constraints represent conditions related to the work performed on the workpieces W and are stored in advance as constraint information in the constraint information storage unit 141. The constraints are defined for each type of workpiece W, regardless of individual differences such as the shape of the workpieces W. For example, the constraints include conditions for each of multiple types of workpieces W regarding the state of one or more workpieces W after the work is completed. If the work is handling, the constraints may be conditions regarding the posture (e.g., orientation) of each workpiece W, conditions regarding the evaluation results for the state in which one or more workpieces W are arranged or aligned, or conditions regarding the state of the multiple workpieces W after assembly is completed. The evaluation results may be calculated by applying physical measurements of one or more arranged or aligned workpieces W to a predetermined algorithm, or may be calculated by inputting images of one or more arranged or aligned workpieces W into a trained model such as a neural network. If the work is processing, the constraints may be conditions regarding the state of one or more workpieces W after the processing is completed.

[0030] The completion state represents the predicted state of a specific workpiece W processed by the robot R when the work to be performed by the robot R is completed. While the constraint conditions are defined for each type of workpiece W, regardless of individual differences such as the shape of the workpiece W, the completion state is defined according to the shape of each individual workpiece W to be processed by the work to be performed. If the work is handling, the constraint condition may be a condition that a specific portion of the workpiece W is placed facing a specific direction, or a condition regarding the evaluation value of the state in which multiple workpieces W are placed. In response to such constraint conditions, the completion state determination unit 140 may determine a post-placement state, which is the state after the workpiece W has been placed, as the completion state, according to the individual shape of the workpiece W (i.e., the skeleton information K). The completion state determination unit 140 may determine at least the posture (e.g., orientation) of the placed workpiece W as the post-placement state, and may further determine at least one of the position and shape of the placed workpiece W. For example, the completion state determination unit 140 determines at least the posture of the placed work W by specifying not only the direction of a specific part of the work W but also the direction of other parts, and further determines at least one of the position and shape of the placed work W.

[0031] The completion state determination unit 140 may use the estimated skeletal information K to convert the constraint information into values ​​or conditions specific to the individual workpiece W to be processed, and determine the conversion result as the completion state. For example, if the constraint information specifies the position and orientation of a specific part of the workpiece W, the completion state determination unit 140 identifies a specific part of the workpiece W from the estimated skeletal information K and specifies the position and orientation of the individual workpiece W so that the identified part is located at the position and orientation specified by the constraint information. The completion state determination unit 140 then determines the state of the workpiece W arranged at the identified position and orientation as the completion state. The completion state determination unit 140 may determine the completion state using a function that receives input of the skeletal information K and calculates the completion state based on the constraint information, or may extract the completion state by geometrically analyzing the skeletal information K based on the constraint information. As another example, the completion state determination unit 140 may determine the completion state using an inference unit that receives input of at least the constraint information and the skeletal information K and outputs the completion state. Alternatively, the completion state determination unit 140 may simulate various completion states based on the skeleton information K and the constraint condition information, evaluate each simulation result using a predetermined evaluation method, and determine the completion state that will produce the optimal simulation result.

[0032] As described above, the robot system 100 causes the robot R to perform a task corresponding to the determined completion state. In one example, the task corresponding to the completion state is a task that transitions the state of the workpiece W to the completion state. In this case, the robot system 100 can achieve the completion state as a result of the task. As another example, in the case of a task whose completion state is determined at the time the robot R is caused to start the task, the robot system 100 can also achieve the completion state by determining the start state of the task. For example, in a case where the start position of welding on the workpiece W is specified as a constraint, the robot system 100 may determine an individual welding start position for the workpiece W as the completion state based on the constraint in accordance with individual differences in the workpiece W, and cause the robot R to start the task from that welding start position.

[0033] The end state determination unit 142 is a functional module that determines an end state, which is the state of the robot R in the determined completion state, based on the determined completion state. If the robot R can be operated by the completion state without using the end state, the process of determining the end state may be omitted. In this case, the end state can be appropriately interpreted as the completion state. Examples of when an end state is unnecessary include a case where the completion state directly represents the end state, and a case where the operation of the robot R is uniquely determined by determining the completion state.

[0034] The end state represents, for example, at least one of the position and orientation of the robot R when an operation on the workpiece W is completed. The position and orientation of the robot R may be the position and orientation of the end effector. In a pick-and-place or alignment process, the end state represents, for example, the position and orientation of the end effector when placing the workpiece W. In an assembly or machining process, the end state represents, for example, the position and orientation of the end effector when the operation is completed.

[0035] For example, the end state determination unit 142 may convert the end state of the workpiece W based on the configuration of the robot R or the end effector, identify either the position or posture of the tip of the arm or the end effector of the robot R, and determine the identified result as the end state. The scope of the present disclosure also includes a case where the end state of the workpiece W is determined indirectly rather than explicitly, that is, where the end state determination unit 140 and the end state determination unit 142 are integrated into a series of processes, and the end state is determined as if from the constraint condition information and the skeleton information K. By configuring the end state determination unit 140 and the end state determination unit 142 separately, the respective processes do not depend on each other, making it easier to build a robot system that can be applied to various workpieces W and various robots R.

[0036] The path generation unit 150 is a functional module that generates a path for the robot R based on the determined start state and end state. Path generation is also called path planning. A path represents, for example, the trajectory of the robot R's movement from the start point to the end point. The path generation unit 150 may collect information on the surrounding environment, such as obstacles, in addition to the start state and end state, and generate a path that prevents the robot R and the transported object from interfering with the obstacles. In the case of pick-and-place, the transported object is a workpiece W. The path generation unit 150 may also generate a path using other methods.

[0037] At least one of the start state and the end state may include a predetermined motion of the robot R that is uniquely determined by that state, and the predetermined motion may include an approach point. In this case, the path generating unit 150 may generate a path from or to the approach point.

[0038] When the path generation unit 150 is unable to generate a path, it may execute at least one of the following: the end state determination unit 142 redetermines the end state; and the start state determination unit 130 redetermines the start state. The path generation unit 150 may then generate (regenerate) a path based on at least one of the redetermined end state and the redetermined start state. For a given task such as pick-and-place, the path generation unit 150 may plan a change of gripping of the workpiece W during the process of generating or regenerating a path. As an example, consider a case where the robot system 100 temporarily places the workpiece W on a placement table, changes the way the workpiece W is held, and continues the pick-and-place. In this case, the robot system 100 may process both the task of transporting the workpiece W to the placement table and the task of transitioning the workpiece W placed on the placement table to a completed state as pick-and-place. In certain situations, such as when it is not possible to generate a path from the start state to the end state, the robot system 100 (path generation unit 150) may divide the task of directly transitioning the workpiece W to the completed state into the task of placing the workpiece W on a stand and the task of transitioning the workpiece W placed on the stand to the completed state, and generate a path for each of the divided tasks.

[0039] The robot control unit 160 is a functional module that causes the robot R to perform a task corresponding to the determined completion state. For example, the robot control unit 160 controls the robot R so that the robot R uses its end effector to perform a task that transitions the workpiece W to the completion state. The robot control unit 160 may cause the robot R to perform a task corresponding to the completion state based on at least one of the start state and the end state. For example, the robot control unit 160 causes the robot R to perform a task based on a path generated based on the start state and the end state. The robot control unit 160 may cause the robot R to perform a task based on a path generated (regenerated) based on at least one of a redetermined start state and a redetermined end state. Such robot control allows the robot R to perform a task according to the skeletal information K of the workpiece W, thereby improving the accuracy or reliability of the task performed on the workpiece W and increasing the variety of workpieces W that can be processed by the robot R.

[0040] [Operation of Robot System] The operation of the robot system 100 will be described with reference to Figures 3A, 3B, 4A, and 4B, and an example of a robot control method according to the present disclosure will be described. Figure 3A is a diagram showing an example of the operation of the robot system 100 related to learning the skeleton estimation model 111. Figure 3B is a diagram showing an example of the operation of the robot system 100 related to the control of the robot R. Therefore, the robot system 100 executes the processing shown in Figures 3A and 3B. Figures 4A and 4B are diagrams for explaining the results of the operation of the robot system 100.

[0041] First, referring to FIG. 3A , an example of training of the skeleton estimation model 111 will be described. Training of the skeleton estimation model 111 begins in step S101. In step S101, the training data generation unit 113 acquires a 3D model and skeletal information K for a specific workpiece W corresponding to the type of workpiece to be estimated by the skeleton estimation model 111. The training data generation unit 113 acquires a 3D model of the workpiece W based on a sample image stored in the work storage unit 112, and acquires skeletal information K corresponding to the sample image. This skeletal information K includes a plurality of keys set in the 3D model. As described above, the multiple keys include a plurality of key points and one or more skeletons, each of which connects two key points.

[0042] In step S102, the learning data generation unit 113 generates a new data record of the learning data. The learning data generation unit 113 deforms the acquired 3D model of the workpiece W, updates the skeletal information K (multiple keys) corresponding to the deformed 3D model, and generates a new sample image obtained from the deformed 3D model. The learning data generation unit 113 associates the new sample image with the updated skeletal information K (multiple keys) to generate a new data record. The learning data generation unit 113 stores the new data record in the workpiece storage unit 112 as at least a part of the learning data.

[0043] In step S103, the learning unit 114 checks whether the amount of learning data necessary for learning the skeleton estimation model 111 has been prepared. If the amount of learning data is insufficient, the process returns to step S102, and the learning data generation unit 113 generates new data records. If a sufficient amount of learning data has been prepared, the process proceeds to step S104.

[0044] In step S104, the learning unit 114 generates the skeleton estimation model 111 by machine learning using the training data.

[0045] In step S105 , the learning unit 114 stores the generated skeleton estimation model 111 in the skeleton estimation unit 110 .

[0046] According to this learning method, learning data can be prepared by deforming a 3D model of the workpiece W, and the learning data can be used to learn the skeleton estimation model 111. Since the amount of learning data required for learning can be prepared without manually creating a large number of 3D models, the construction cost of the robot system 100 can be reduced.

[0047] Next, an example of control of the robot R will be described with reference to Fig. 3B. Control of the robot R starts from step S201. In step S201, the camera C acquires a workpiece image including the workpiece W. For example, the camera C captures a workpiece image as shown in Fig. 2B.

[0048] In step S202, the skeleton estimation unit 110 estimates skeletal information K of the workpiece W included in the workpiece image. In the example of Fig. 2B, the skeleton estimation unit 110 estimates skeletal information K for each of the multiple workpieces (broccoli) W in the workpiece image.

[0049] In step S203, the workpiece selection unit 120 selects a workpiece W to be processed in the next job based on the skeleton information K of each of the one or more workpieces W.

[0050] In step S204, the start state determination unit 130 determines pick position A as the start state. In the example of Fig. 2B, a broccoli near the upper center is selected, and the start state determination unit 130 determines pick position A for the selected broccoli.

[0051] In step S205, the completion state determination unit 140 determines the completion state of the work performed on the selected workpiece W. FIGS. 4A and 4B show examples of the completion state of the work. In these examples, the work is assumed to be pick-and-place of a workpiece (broccoli) W. FIG. 4A shows a state in which the broccoli pieces are placed so that their orientations are not aligned, and FIG. 4B shows a state in which the broccoli pieces are placed so that their orientations are aligned. For example, it is assumed that placing the broccoli pieces in the same orientation is set as at least part of the constraint condition. In this case, the completion state determination unit 140 determines a state in which the orientations of the left and right broccoli pieces are aligned as the completion state when placing the left broccoli shown in FIG. 4B. That is, the completion state determination unit 140 determines the position and orientation of the left broccoli after placement so that the orientations of the left and right broccoli pieces are aligned.

[0052] In step S206, the end state determination unit 142 determines the end state of the robot R in the determined completion state.

[0053] In step S207, the path generating unit 150 generates a path connecting the start state and the end state. As a result, it is determined that the robot R picks up the workpiece W in the start state and places the workpiece W in the end state, and a path connecting the start state and the end state is prepared.

[0054] In step S208, it is confirmed whether the path generation unit 150 was able to generate a path. If a path cannot be generated, the process returns to step S204. In this case, at least one of the end state determination unit 142 redetermines the end state and the start state determination unit 130 redetermines the start state, and the path generation unit 150 generates (regenerates) a path based on at least one of the redetermined end state and the redetermined start state. If a path can be generated, the process proceeds to step S209.

[0055] In step S209, the robot control unit 160 causes the robot R to perform the task corresponding to the determined completion state. If the task is pick-and-place, the robot R picks up the workpiece W in the determined start state, transports the workpiece W along the generated path, and places the workpiece W in the determined end state. In the example of FIG. 4B , even if the workpiece W has an irregular shape like broccoli, the robot system 100 picks and places each workpiece W so that the orientation of the multiple workpieces W is aligned. This mechanism improves the quality of the task and increases the variety of workpieces that can be processed.

[0056] In step S210, the workpiece selection unit 120 checks whether or not there is a workpiece W to be processed in the next job (i.e., whether or not the job is to be repeated). If such a workpiece W exists, the robot system 100 repeats the process from step S201 or step S203. If such a workpiece W does not exist, the robot system 100 ends the process.

[0057] [Hardware Configuration] The above-described series of processes may be executed by dedicated hardware or software. When the series of processes are executed by software, the above-described series of processes can be realized by causing a general-purpose or dedicated computer to execute a program. The computer may have a CPU (Central Processing Unit), a recording device such as an HDD (Hard Disk Drive), a ROM (Read Only Memory), or a RAM (Random Access Memory), a communication device connected to a network such as a LAN (Local Area Network) or the Internet, an input device such as a mouse or a keyboard, a drive for reading and writing removable storage media such as a magnetic disk such as a flexible disk, various optical disks such as CDs (Compact Discs), MO (Magneto Optical) disks, and DVDs (Digital Versatile Discs), and semiconductor memories, and an output device such as a display device such as a monitor, and an audio output device such as a speaker or headphones. This computer may perform the above series of processes by executing a program recorded on a recording device or a removable storage medium, or a program obtained via a network.

[0058] 5 is a diagram showing an example of the hardware configuration of a computer 1000 used for a robot system according to the present disclosure. In this example, the computer 1000 includes a main body 1100, an output device 1200, and an input device 1300.

[0059] The main body 1100 is a device having a circuit 1600. The circuit 1600 has a processor 1601, a memory 1602, a storage 1603, an input / output port 1604, and a communication port 1605. The number of each hardware component may be one or more. The storage 1603 records programs for configuring each functional module of the main body 1100. The storage 1603 is a computer-readable recording medium such as a hard disk, a non-volatile semiconductor memory, a magnetic disk, or an optical disk. The memory 1602 temporarily stores programs loaded from the storage 1603, calculation results of the processor 1601, and the like. The processor 1601 configures each functional module by executing programs in cooperation with the memory 1602. The input / output port 1604 inputs and outputs electrical signals to and from the output device 1200 or the input device 1300 in response to commands from the processor 1601. The input / output port 1604 may also input and output electrical signals to and from other devices. The communication port 1605 performs data communication with other devices via the communication network N in accordance with instructions from the processor 1601 .

[0060] The output device 1200 is a device for outputting information from the main body 1100. Examples of the output device 1200 include display devices such as various displays and speakers.

[0061] The input device 1300 is a device for inputting information to the main body 1100. Examples of the input device 1300 include operation interfaces such as a keypad, a mouse, and an operation controller.

[0062] The output device 1200 and the input device 1300 may be integrated as a touch panel. For example, the main body 1100, the output device 1200, and the input device 1300 may be integrated as in a tablet computer.

[0063] Each functional module of the robot system is realized by loading a robot control program into the processor 1601 or memory 1602 and having the processor 1601 execute the program. The robot control program includes code for realizing each functional module of the robot system. The processor 1601 operates the input / output port 1604 and communication port 1605 in accordance with the robot control program, and executes reading and writing of data from and to the memory 1602 or storage 1603.

[0064] The robot control program may be provided in the form of a non-transitory recording medium such as a CD-ROM, a DVD-ROM, or a semiconductor memory, or may be provided via a communications network as a data signal superimposed on a carrier wave.

[0065] [Modification] (Pick Position Estimation Model) In the above example, the start state determination unit 130 determines the pick position A as the start state based on the skeleton information K, but the start state determination unit may also determine the pick position A directly from the workpiece image. This modification will be described with reference to FIG. 6 .

[0066] 6 is a diagram showing the configuration of a robot system 200 according to a modified example. The robot system 200 differs from the robot system 100 in that it includes a start state determination unit 230 instead of the start state determination unit 130. A description of the same configuration as the robot system 100 will be omitted.

[0067] The start state determination unit 230 has a pick position estimation model 231. The pick position estimation model 231 is generated in advance by machine learning using learning data including multiple data records. Each data record indicates a correspondence between a sample image based on a 3D model of the workpiece W and pick information indicating whether the robot R can pick a position on the 3D model corresponding to a key set in the 3D model. The pick information is associated with at least one of multiple keys set in the 3D model. The multiple keys include multiple key points P and one or more skeletons S, each connecting two key points P. In the learning phase, the pick position estimation model 231 is trained using the pick information associated with the keys as learning data. However, in the inference phase (operation phase), when the pick position estimation model 231 receives input of a workpiece image including the workpiece W, it processes the workpiece image without using a key and outputs a pick position A. Therefore, the start state determination unit 230 inputs the workpiece image into the pick position estimation model 231 to determine the pick position A to be performed by the robot R on the workpiece W. The pick position estimation model 231 may be realized by, for example, a neural network model such as CNN. In one example, the robot control unit 160 causes the robot R to perform an operation on the workpiece W based on the end state and the pick position.

[0068] (Processing Based on Score) As described above, the workpiece processed by the robot may be a flexible object whose individual shape difference is negligible but whose degree of deformation may vary greatly from one object to another. In this case, the skeleton estimation unit 110 may estimate multiple key points of the workpiece, calculate scores for the multiple key points, and estimate the posture (e.g., orientation) of the workpiece as at least part of the skeleton information based on the scores.

[0069] Estimation of the workpiece posture based on the score will be described with reference to FIG. 7 . FIG. 7 is a diagram showing an example of key points P of the workpiece W estimated using the skeleton estimation model 111. In this example, the skeleton estimation unit 110 estimates the orientation of the workpiece W as at least part of the skeleton information. This estimation is also called head-to-tail determination. The workpiece W is a conversion cable for transmitting signals between input / output terminals with different specifications. The skeleton estimation model 111 is assumed to have been trained to estimate up to five key points P for the workpiece W. In estimating the key points P, the skeleton estimation model 111 assigns an identifier to each key point P to distinguish them. In situation 301, the skeleton estimation model 111 estimates five key points P with identifiers 0 to 4 in order from the input / output terminal on the head side (the terminal on the right side in FIG. 7 ) to the input / output terminal on the rear side (the terminal on the left side in FIG. 7 ), which is an ideal estimation. In contrast, in situation 302, the skeleton estimation model 111 estimates only four key points P, and the identifiers are not arranged in order along the extension direction of the workpiece W. The skeleton estimation unit 110 executes a process for accurately determining the head and tail even when the skeleton estimation model 111 does not accurately estimate the key point P, as in situation 302 .

[0070] In one example, the skeleton estimation unit 110 defines in advance a weight corresponding to an identifier for each of a plurality of key points P that can be detected. The skeleton estimation unit 110 also sets a status for each key point P indicating whether the key point has been detected. The status is 1 if the key point has been detected, and 0 if the key point has not been detected. The skeleton estimation unit 110 divides the workpiece W into two regions based on the midpoint of a line segment connecting the first and second ends of the workpiece W, and determines in which of the two divided regions each detected key point P is located. For each of a plurality of key points P that can be detected, the skeleton estimation unit 110 calculates the product of the weight, the status, and a predetermined area value assigned to the divided region in which the key point P is located. The skeleton estimation unit 110 then calculates a score based on the sum of multiple products corresponding to the plurality of key points P. If the score satisfies a predetermined condition, the skeleton estimation unit 110 determines that the first end is a head-side input / output terminal. If the score does not satisfy the predetermined condition, the skeleton estimation unit 110 determines that the second end is a head-side input / output terminal.

[0071] That is, the skeleton estimation unit 110 estimates the identifier and position of each of the multiple key points P of the workpiece W. Then, the skeleton estimation unit 110 calculates a score based on the identifier and position of each of the multiple key points P, and estimates the posture of the workpiece W as at least a part of the skeleton information based on the score.

[0072] The workpiece selection unit 120 may use the score to select at least one workpiece from the multiple workpieces included in the workpiece image. In this case, the skeleton estimation unit 110 calculates a score for each of the multiple workpieces. The workpiece selection unit 120 selects at least one workpiece as a workpiece to be processed in the job based on the score of each of the multiple workpieces. For example, the workpiece selection unit 120 may select a workpiece whose score is equal to or greater than a predetermined threshold, or may select a workpiece with the highest score.

[0073] FIG. 8 is a diagram showing an example of a process for selecting a workpiece. In this example, the workpiece image G includes multiple workpieces (conversion cables) W stored in a box. The skeleton estimation unit 110 calculates a score for each of the multiple workpieces W and performs head-to-tail determination for each workpiece W. In the example of FIG. 8, the workpiece selection unit 120 selects the workpiece W located at the top of the box, i.e., the workpiece W located closest to the camera C, based on the scores of the multiple workpieces W. In the example of FIG. 8, the results of the head-to-tail determination for the selected workpiece W are indicated by the labels "head" and "tail," respectively.

[0074] (Checking the Processed Workpiece) The robot systems 100 and 200 may check the processed workpiece. In this example, the camera C photographs and acquires a new workpiece image including the workpiece processed in the work performed by the robot R. The skeleton estimation unit 110 estimates new skeleton information indicating the skeleton of the workpiece based on the new workpiece image. The completion state determination unit 140 determines whether the processed workpiece satisfies the constraints based on the new skeleton information. If the processed workpiece satisfies the constraints, the robot systems 100 and 200 determine whether to have the robot R perform work on another workpiece. This process corresponds to step S210 described above. If the processed workpiece does not satisfy the constraints, the robot control unit 160 may have the robot R perform work again according to the new skeleton information so that the workpiece satisfies the constraints. Alternatively, the completion state determination unit 140 may notify the user of the robot systems 100 and 200 of a message indicating that the constraints are not satisfied.

[0075] FIG. 9 is a diagram illustrating an example of a processed workpiece. In this example, the constraint condition includes a condition that the workpiece (conversion cable) W be placed within the placement area D. In both situations 311 and 312, the skeleton estimation unit 110 estimates new skeleton information K including multiple key points P of the processed workpiece W. The completion state determination unit 140 determines whether the workpiece W satisfies the constraint condition based on the skeleton information K. For example, the completion state determination unit 140 determines that the constraint condition is satisfied if the proportion of the workpiece W that fits within the placement area D is equal to or greater than a predetermined threshold, and determines that the constraint condition is not satisfied if the proportion is less than the threshold. The completion state determination unit 140 may determine that the constraint condition is satisfied if the number of key points P that fit within the placement area D is equal to or greater than a predetermined threshold, and determines that the constraint condition is not satisfied if the number is less than the threshold. In situation 311, the completion state determination unit 140 determines that the workpiece W satisfies the constraint condition. In situation 312, the completion state determination unit 140 determines that the workpiece W does not satisfy the constraint condition. As described above, in situation 312, the robot control unit 160 may cause the robot R to again perform the task of placing the workpiece W within the placement area D in accordance with the new skeleton information K.

[0076] (Generation of Constraint Information) The robot system may include a functional module that generates constraint information. This modified example will be described with reference to FIG. 10. FIG. 10 is a diagram showing the configuration of a robot system 100A according to the modified example. The robot system 100A differs from the robot system 100 in that it further includes a constraint generation unit 143. A description of the same configuration as the robot system 100 will be omitted.

[0077] The constraint generation unit 143 is a functional module that generates constraint information based on a teaching image that indicates the target state of the workpiece when the work is completed. The teaching image may be generated by photographing a real workpiece or by drawing a virtual workpiece using computer graphics. The constraint generation unit 143 estimates teaching skeleton information that indicates the skeleton of the workpiece in the teaching image and generates constraint information based on this teaching skeleton information. Like the above-mentioned skeleton information K, the teaching skeleton information also includes multiple keys, each of which includes multiple key points and one or more skeletons that connect two key points. Like the skeleton estimation unit 110, the constraint generation unit 143 may estimate the teaching skeleton information from the teaching image using the skeleton estimation model 111. The constraint generation unit 143 may set the posture (e.g., orientation) of the workpiece indicated by the teaching skeleton information as a constraint and generate constraint information that indicates this constraint. Alternatively, the constraint generating unit 143 may display a user interface for generating constraint information, and generate the constraint information based on the estimated teaching skeleton information and user input received via the user interface. As described above, the constraint information is generated for each type of workpiece.

[0078] 11 is a diagram showing an example of a user interface for generating constraint information for a pick-and-place operation. The user interface 400 shown in this example includes an image display area 401 that displays a teach image T specified by the user, an add area button 402 that is a button for defining a placement area D where a workpiece W is placed by the pick-and-place operation, a label 403 for specifying the type of workpiece, and a setting completion button 404 for confirming the constraint information.

[0079] In one example, the user first displays a desired teaching image T in the image display area 401. The teaching image T is an image showing the ideal state of the workpiece (conversion cable) W to be placed by the pick-and-place operation, i.e., the state of the workpiece W desired by the user for the robot R after placement. Next, the user clicks or taps the add area button 402 and specifies the area where the workpiece W should be placed by drawing a frame F with a pointing device. Next, the user drags the label 403 corresponding to the workpiece W in the teaching image T to the workpiece W on the teaching image T, thereby setting the type of workpiece W as a conversion cable. In response to this setting, the constraint generation unit 143 estimates teaching skeleton information indicating at least the orientation of the workpiece W represented by the skeleton of the workpiece W. Then, based on the estimated orientation and the specified frame F, the constraint generation unit 143 generates constraint condition information indicating the placement area D as at least part of the constraint conditions for the conversion cable. In response to the user clicking or tapping the setting completion button 404, the constraint generation unit 143 stores the generated constraint information in the constraint information storage unit 141. The constraint information can be referenced by the completion state determination unit 140 to determine the completion state of the pick-and-place operation performed on each workpiece (conversion cable) W.

[0080] As a modified example of the constraint generating unit 143 and the user interface 400, the constraint generating unit 143 may generate constraint condition information indicating the placement area D based on the attitude of the workpiece indicated by the teaching skeleton information, without receiving user input of the frame F. As another example, the constraint generating unit 143 may display a dialog screen for adjusting the automatically set placement area D in response to a predetermined user operation on the user interface 400, and change the position, shape, orientation, etc. of the placement area D based on the user input on the dialog screen.

[0081] Various examples of the present disclosure have been described above in detail with reference to the accompanying drawings. However, it goes without saying that the scope of the technical idea of ​​the present disclosure is not limited to the above examples. It is clear that a person skilled in the art of the present disclosure can conceive of various changes, modifications, combinations, etc. within the scope of the technical idea of ​​the present disclosure as set forth in the claims. Therefore, technologies to which these changes, modifications, and combinations are applied also naturally fall within the scope of the technical idea of ​​the present disclosure.

[0082] In this specification, the process flows described in flowcharts include not only processes that are performed chronologically in the order described, but also processes that are performed in parallel or individually, rather than chronologically. Needless to say, even when processes are performed chronologically, the order of each step can be changed.

[0083] The hardware configuration of the system is not limited to a configuration in which each functional module is realized by executing a program. For example, at least some of the functional modules may be configured by logic circuits specialized for the functions, or may be configured by an ASIC (Application Specific Integrated Circuit) that integrates the logic circuits.

[0084] When comparing the magnitude of two numbers within a computer system or computer, either of the two criteria "greater than or equal to" and "greater than" can be used, or either of the two criteria "less than or equal to" and "under".

[0085] [Additional Notes] As can be seen from the various examples above, the present disclosure includes the following aspects.

[0086] (Aspect 1) A robot system comprising: a skeleton estimation unit that estimates skeleton information indicating the skeleton of a workpiece based on a workpiece image including the workpiece present in real space; a completion state determination unit that determines a completion state of the workpiece to be performed based on constraint condition information indicating constraint conditions related to the workpiece and the estimated skeleton information; and a robot control unit that causes a robot located in the real space to perform the work according to the determined completion state. In this case, the completion state of the workpiece to be performed is determined based on the skeleton of the workpiece, and the robot is controlled based on the completion state. This processing enables the robot to perform delicate work according to the shape of the workpiece actually being processed or changes in the shape. Therefore, the reliability of work on the workpiece can be improved. For example, the accuracy of work on the workpiece can be increased. In one example, it is possible to increase the types of workpieces that can be processed by the robot, i.e., to broaden the range of robot systems available.

[0087] (Aspect 2) The robot system according to Aspect 1, wherein the operation includes a pick-and-place operation of the workpiece by the robot, and the completion state determination unit determines, as the completion state, at least a post-placement state that is a state after the workpiece has been placed. In this case, the state of the workpiece placed by the robot is determined as the completion state, so that the pick-and-place operation can be controlled with higher precision.

[0088] (Aspect 3) The robot system according to Aspect 2, wherein the work is an irregular work, the operation includes picking and placing the irregular work by the robot, and the completion state determination unit determines at least the posture of the placed work as the post-placement state. In this case, the posture of the placed irregular work is determined as the post-placement state (completion state), so that the pick-and-place operation of the irregular work can be controlled with higher precision.

[0089] (Aspect 4) The robot system according to any one of Aspects 1 to 3, further comprising an end state determination unit that determines, based on the determined completion state, an end state that is the state of the robot in the determined completion state, and the robot control unit causes the robot to perform the task based on the determined end state. In this case, the end state that is the state of the robot when the task is completed is determined based on the completion state, and the robot is controlled based on that end state, so that the robot can be operated more flexibly and accurately in response to the shape of the workpiece.

[0090] (Aspect 5) The robot system according to Aspect 4, further comprising a start state determination unit that determines a start state, which is the state of the robot at the start of the task, based on the skeletal information, and a robot control unit that causes the robot to perform the task based on the determined start state and the end state. In this case, the start state, which is the state of the robot at the start of the task, is determined based on the skeletal state, making it possible to increase the success rate of the robot starting the task and to more accurately control the result at the completion of the task, which is affected by the state at the start of the task.

[0091] (Aspect 6) The robot system according to Aspect 5 further includes a path generation unit that generates a path for the robot based on the start state and the end state, wherein the path generation unit, when unable to generate the path, causes the end state determination unit to redetermine the end state or the start state determination unit to redetermine the start state; generates the path based on at least one of the redetermined end state and the redetermined start state; and a robot control unit causes the robot to perform the task based on the generated path. In this case, a path for the robot during the task is generated, and when unable to generate a path, a path is generated after redetermining the end state and the start state. Such automatic path generation reduces the frequency of tasks that cannot be performed due to inability to generate a path, thereby increasing the probability of completing tasks while adapting to various situations. For example, it can increase the probability of reliably performing tasks depending on the surrounding environment, such as obstacles, or the state of the workpiece.

[0092] (Aspect 7) The robot system according to Aspect 5 or 6, wherein the skeletal information includes a plurality of keys, each including a plurality of key points of the workpiece and one or more skeletons connecting two of the key points, and the start state determination unit determines the start state based on task information associated with at least one of the plurality of keys set for each type of workpiece and the estimated skeletal information. In this case, by referencing task information corresponding to the workpiece in addition to the skeletal information of the workpiece, the robot can be more reliably caused to start work. Furthermore, by setting the task information, the user's intention for the work can be reflected in the robot control.

[0093] (Aspect 8) The robot system according to Aspect 7, wherein the task includes a pick-and-place task of the workpiece by the robot, the task information includes pick information indicating, for each of the at least one key, whether the robot can pick a position on the workpiece corresponding to the key, and the start state determination unit determines, as the start state, a pick position on the workpiece by the robot based on the pick information and the skeleton information. In this case, a pick position is set as task information for each type of workpiece, and the pick position of the workpiece to be processed by the task is determined according to the task information and the skeleton information. This process increases the success rate of pick-and-place and can control the state of the workpiece after placement.

[0094] (Aspect 9) The robot system according to Aspect 4, further comprising: a start state determination unit that determines a pick position of the workpiece to be picked by the robot, the operation including pick-and-place operation of the workpiece by the robot, the start state determination unit having a pick position estimation model trained using the training data, the pick position estimation model including learning data including a plurality of data records indicating a correspondence between a sample image based on a 3D model of the workpiece and pick information indicating whether the robot can pick a position on the 3D model corresponding to a key set in the 3D model, the pick information being associated with at least one of the plurality of keys set in the 3D model, the plurality of keys including a plurality of key points and one or more skeletons each connecting two of the key points, the start state determination unit determining the pick position based on the workpiece image and the pick position estimation model, and a robot control unit causing the robot to perform the operation based on the end state and the determined pick position. In this case, the pick position estimation model trained in advance using learning data indicating a correspondence between a sample image and pick information associated with skeleton information can determine a pick position to operate the robot without determining a start state. Furthermore, by introducing a pick position estimation model, the cost of building a robot system can be reduced and the accuracy of the pick position can be improved.

[0095] (Aspect 10) The robot system according to any one of Aspects 1 to 9, further comprising a workpiece selection unit that selects at least one workpiece from a plurality of the workpieces included in the workpiece image, wherein the skeleton estimation unit estimates the skeleton information for each of the plurality of workpieces, and the workpiece selection unit selects the at least one workpiece as a workpiece to be processed in the task based on the skeleton information for each of the plurality of workpieces. In this case, workpieces predicted to be appropriate to be processed are selected based on the skeleton of each workpiece, thereby increasing the probability of completing the task and improving the quality of the task results.

[0096] (Aspect 11) The robot system according to aspect 10, wherein the workpiece selection unit selects a workpiece that is entirely exposed in the workpiece image as the workpiece to be processed in the task based on the area and skeletal information of each of the plurality of workpieces. In this case, a workpiece that is entirely exposed in the workpiece image, i.e., a workpiece that is expected to be easy to process, is selected, thereby further increasing the probability of completing the task.

[0097] (Aspect 12) The robot system according to Aspect 10, wherein the workpiece selection unit selects a workpiece whose degree of deformation is within a predetermined range as the workpiece to be processed in the operation based on the skeletal information of each of the plurality of workpieces. In this case, a workpiece that is relatively close to a typical posture of a workpiece, i.e., a workpiece that is expected to be easy to process, is selected, thereby further improving the quality of the operation result.

[0098] (Aspect 13) The robot system according to any one of Aspects 1 to 12, wherein the skeleton estimation unit calculates scores for a plurality of key points of the workpiece, and estimates the posture of the workpiece as at least a part of the skeleton information based on the scores. In this case, the posture of the workpiece is estimated based on the scores calculated for a plurality of key points of the workpiece, so that the posture of the workpiece can be estimated more accurately. As a result, the accuracy of work performed on the workpiece can be further improved.

[0099] (Aspect 14) The robot system according to any one of Aspects 1 to 13, wherein the skeleton estimation unit estimates new skeleton information indicating the skeleton of the processed workpiece based on a new workpiece image including the workpiece processed in an operation performed by the robot, and the completion state determination unit determines whether the processed workpiece satisfies the constraint conditions based on the estimated new skeleton information. In this case, whether the processed workpiece satisfies the constraint conditions is determined based on the skeleton of the workpiece. This mechanism makes it possible to automatically monitor whether the robot has properly processed the workpiece.

[0100] (Aspect 15) The robot system according to Aspect 14, wherein the robot control unit causes the robot to perform the task again in accordance with the estimated new skeletal information if the processed workpiece does not satisfy the constraints. In this case, if the processed workpiece does not satisfy the constraints, the robot processes the workpiece again based on the current skeletal information of the workpiece. This processing can further improve the quality of the work results.

[0101] (Aspect 16) The robot system according to any one of Aspects 1 to 15, further comprising a constraint generating unit that, based on a teach image that indicates a target state of the workpiece when the task is completed, estimates teach skeletal information that indicates a skeleton of the workpiece in the teach image, and generates the constraint condition information based on the teach skeletal information. In this case, since the constraint condition information is generated based on the workpiece indicated by the teach image, it is possible to easily set constraint conditions for the task on the workpiece.

[0102] (Aspect 17) The robot system according to Aspect 16, wherein the operation includes a pick-and-place operation of the workpiece by the robot, and the constraint generation unit estimates an orientation of the workpiece in the teaching image as at least a part of the teaching skeleton information, and generates the constraint condition information based on the estimated orientation, the constraint condition information representing, as at least a part of the constraint condition, a placement area that is an area where the workpiece is placed by the pick-and-place operation. In this case, the placement area of ​​the workpiece is set based on the orientation of the workpiece in the teaching image, making it possible to easily set an appropriate placement area suited to the type of workpiece.

[0103] (Aspect 18) A robot control method executed by a robot system including at least one processor, comprising: a step of estimating skeletal information indicating a skeleton of a workpiece based on a workpiece image including the workpiece present in real space; a step of determining a completion state of the workpiece to be performed based on the estimated skeletal information and constraint condition information indicating constraint conditions related to the workpiece; and a step of causing a robot located in the real space to perform the work corresponding to the determined completion state. In this case, the completion state of the workpiece to be performed is determined based on the skeleton of the workpiece, and the robot is controlled based on the completion state. This processing enables the robot to perform delicate work corresponding to the shape of the workpiece actually being processed or changes in the shape. Therefore, the reliability of work on the workpiece can be improved. For example, the accuracy of work on the workpiece can be increased. In one example, the types of workpieces processed by the robot can be increased, i.e., the range of robot systems available can be expanded.

[0104] (Aspect 19) A robot control program that causes a computer to execute the following steps: estimating skeletal information indicating the skeleton of a workpiece based on a workpiece image including the workpiece present in real space; determining a completion state of the workpiece to be performed based on the estimated skeletal information and constraint condition information indicating constraint conditions related to the workpiece; and causing a robot located in the real space to perform the work corresponding to the determined completion state. In this case, the completion state of the workpiece to be performed is determined based on the skeleton of the workpiece, and the robot is controlled based on the completion state. This processing enables the robot to perform delicate work corresponding to the shape of the workpiece actually being processed or changes in the shape. Therefore, the reliability of work on the workpiece can be improved. For example, the accuracy of work on the workpiece can be increased. In one example, the types of workpieces that can be processed by the robot can be increased, i.e., the range of robot systems available can be expanded.

[0105] (Aspect 20) The robot system according to any one of Aspects 1 to 17, wherein the skeleton estimation unit estimates the skeletal information based on the workpiece image using a skeleton estimation model trained using training data including a sample image based on a 3D model of the workpiece and a plurality of data records including a plurality of keys set in the 3D model, the plurality of keys including a plurality of key points and one or more skeletons each connecting two of the key points. In this case, the skeletal information is estimated using a skeleton estimation model trained in advance using training data indicating correspondence between a sample image and a plurality of keys corresponding to the workpiece. Introducing the skeleton estimation model can reduce the cost of building a robot system and improve the accuracy of estimating the skeleton of a workpiece.

[0106] (Aspect 21) The robot system according to Aspect 20, further comprising: a training data generation unit that transforms the 3D model represented by one data record of the plurality of data records to generate a new data record of the training data; and a training unit that trains the skeleton estimation model using at least the new data record. In this case, the amount of training data for training the skeleton estimation model automatically increases by transforming an existing 3D model, thereby reducing the cost of implementing the training.

[0107] (Aspect 22) The robot system according to Aspect 13, further comprising a workpiece selection unit that selects at least one workpiece from a plurality of the workpieces included in the workpiece image, wherein the skeleton estimation unit calculates the score for each of the plurality of workpieces, and the workpiece selection unit selects the at least one workpiece as a workpiece to be processed in the task based on the score for each of the plurality of workpieces. In this case, workpieces predicted to be appropriate to be processed are selected based on the score of each workpiece, thereby increasing the probability of completing the task and improving the quality of the task result.

[0108] 100, 100A, 200...Robot system, 110...Skeleton estimation unit, 111...Skeleton estimation model, 112...Work memory unit, 113...Learning data generation unit, 114...Learning unit, 120...Work selection unit, 130...Start state determination unit, 132...Work information memory unit, 140...Completion state determination unit, 141...Constraint condition information memory unit, 142...End state determination unit, 143...Constraint generation unit, 150...Path generation unit, 160...Robot control unit, 230...Start state determination unit, 231...Pick position estimation model, 400...User interface, A...Pick position, C...Camera, D...Placement area, G...Work image, K...Skeleton information, P...Key point, R...Robot, S...Skeleton, T...Teaching image, W...Work.

Claims

1. A robot system comprising: a skeleton estimation unit that estimates skeleton information indicating the skeleton of a workpiece based on a workpiece image including the workpiece existing in real space; a completion state determination unit that determines a completion state of the work being performed on the workpiece based on constraint condition information indicating constraint conditions related to the work on the workpiece and the estimated skeletal information; and a robot control unit that causes a robot positioned in the real space to perform the work according to the determined completion state.

2. The robot system according to claim 1, wherein the operation includes a pick-and-place operation of the workpiece by the robot, and the completion state determination unit determines, as the completion state, at least a post-placement state which is a state after the workpiece has been placed.

3. The robot system according to claim 2, wherein the work is an irregular work, the operation includes a pick-and-place operation of the irregular work by the robot, and the completion state determination unit determines at least the posture of the placed work as the post-placement state.

4. A robot system according to any one of claims 1 to 3, further comprising an end state determination unit which determines an end state, which is the state of the robot at the determined completion state, based on the determined completion state, and the robot control unit causes the robot to perform the task based on the determined end state.

5. The robot system according to claim 4, further comprising a start state determination unit that determines a start state, which is the state of the robot at the start of the work, based on the skeletal information, and a robot control unit that causes the robot to perform the work based on the determined start state and the end state.

6. A robot system as described in claim 5, further comprising a path generation unit which generates a path for the robot based on the start state and the end state, wherein the path generation unit, when unable to generate the path, causes at least one of the end state determination unit to redetermine the end state and the start state determination unit to redetermine the start state, generates the path based on at least one of the redetermined end state and the redetermined start state, and a robot control unit causes the robot to perform the task based on the generated path.

7. The robot system described in claim 5, wherein the skeletal information includes a plurality of keys, each of which includes a plurality of key points of the work and one or more skeletons connecting two of the key points, and the starting state determination unit determines the starting state based on work information associated with at least one of the plurality of keys set for each type of work and the estimated skeletal information.

8. The robot system described in claim 7, wherein the operation includes a pick-and-place operation of the workpiece by the robot, the operation information includes pick information indicating, for each of the at least one key, whether the robot is able to pick a position on the workpiece corresponding to the key, and the starting state determination unit determines, as the starting state, a pick position on the workpiece by the robot based on the pick information and the skeletal information.

9. The robot system according to claim 4, further comprising a start state determination unit which determines a pick position on the workpiece by the robot, the operation including a pick-and-place operation of the workpiece by the robot, the start state determination unit having a pick position estimation model learned from the learning data, the pick position estimation model including learning data including a plurality of data records indicating a correspondence between a sample image based on a 3D model of the workpiece and pick information indicating whether the robot can pick a position on the 3D model corresponding to a key set on the 3D model, the pick information being associated with at least one of the plurality of keys set on the 3D model, the plurality of keys including a plurality of key points and one or more skeletons each connecting two of the key points, and determining the pick position based on the workpiece image and the pick position estimation model, and a robot control unit causing the robot to perform the operation based on the end state and the determined pick position.

10. The robot system of claim 1, further comprising a work selection unit that selects at least one work from a plurality of workpieces included in the work image, wherein the skeleton estimation unit estimates the skeleton information for each of the plurality of workpieces, and the work selection unit selects the at least one workpiece as a workpiece to be processed in the operation based on the skeleton information for each of the plurality of workpieces.

11. The robot system according to claim 10, wherein the work selection unit selects a workpiece that is entirely exposed in the workpiece image as the workpiece to be processed in the operation based on the area of ​​each of the plurality of workpieces and the skeletal information.

12. The robot system according to claim 10, wherein the workpiece selection unit selects a workpiece having a degree of deformation within a predetermined range as the workpiece to be processed in the operation based on the skeletal information of each of the plurality of workpieces.

13. The robot system of claim 1, wherein the skeleton estimation unit calculates scores for a plurality of key points of the workpiece, and estimates the posture of the workpiece as at least a part of the skeleton information based on the scores.

14. The robot system of claim 1, wherein the skeleton estimation unit estimates new skeleton information indicating the skeleton of the processed work based on a new work image including the work processed in the operation performed by the robot, and the completion state determination unit determines whether the processed work satisfies the constraint conditions based on the estimated new skeleton information.

15. The robot system according to claim 14, wherein the robot control unit causes the robot to perform the work again in accordance with the estimated new skeletal information when the processed workpiece does not satisfy the constraint conditions.

16. The robot system according to claim 1, further comprising a constraint generation unit that estimates teaching skeleton information indicating the skeleton of the workpiece in a teaching image that indicates a target state of the workpiece when the task is completed, based on the teaching image, and generates the constraint condition information based on the teaching skeleton information.

17. The robot system described in claim 16, wherein the operation includes a pick-and-place operation of the workpiece by the robot, and the constraint generation unit estimates the posture of the workpiece in the teaching image as at least part of the teaching skeleton information, and generates the constraint condition information based on the estimated posture, the constraint condition information representing a placement area, which is an area where the workpiece is placed by the pick-and-place operation, as at least part of the constraint condition.

18. A robot control method executed by a robot system having at least one processor, comprising: a step of estimating skeletal information indicating the skeleton of a workpiece based on a workpiece image including the workpiece existing in real space; a step of determining a completion state of the work to be performed on the workpiece based on constraint condition information indicating constraint conditions related to the work on the workpiece and the estimated skeletal information; and a step of having a robot placed in the real space perform the work according to the determined completion state.

19. A robot control program that causes a computer to execute the steps of: estimating skeletal information indicating the skeleton of a workpiece based on a workpiece image including the workpiece existing in real space; determining a completion state of the work being performed on the workpiece based on constraint condition information indicating constraint conditions related to the work on the workpiece and the estimated skeletal information; and having a robot placed in the real space perform the work according to the determined completion state.