Attitude detection device and data generation program

Abstract

To provide an attitude detection system capable of stably detecting an attitude of an object and a data generation program capable of generating learning data to be used by the attitude detection system.SOLUTION: An attitude detection device for detecting attitudes of an objects W on the basis of an image T by imaging the object W having no skeletons estimates pseudo skeletons S obtained by combining a plurality of key points K determined as feature points of the object W and line segment-shaped bones B connecting two key points in the image T, and detects attitudes of the objects W.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an attitude detection device that detects the attitude of an object, and a data generation program that generates learning data used in the device. [Background technology]

[0002] To allow a robot to grasp an object such as a workpiece, it is necessary to detect the object's posture in addition to its shape and size. The "posture" referred to here includes information on the overall orientation of the object (in the case of a fish, which direction is the head facing when the object is positioned) and the position of the object's main parts relative to the whole, and this applies hereinafter as well unless otherwise specified.

[0003] When a robot grasps and moves multiple objects of the same size and shape one by one, the position and orientation of the objects can be detected by applying a matching process to the image captured by a camera using the design data (CAD data) of the objects. Therefore, if information about the part of the object to be grasped is given to the robot in advance, the robot can detect the position and orientation of the part of the object to be grasped based on the captured image and grasp the object.

[0004] Furthermore, for objects whose shapes vary slightly from one object to another, it is possible to control the robot to grasp the object by defining a primitive shape that is the basis of the object's shape and performing a fitting process, as described in Non-Patent Document 1. A primitive shape is a simplified version of the actual shape of the object, such as a cylinder or a cone. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Yuma Yoshimoto and Hakaru Tamukoh, "Automatic Approximation of Primitive Shapes using Point Clouds," The 2022 International Conference on Artificial Life and Robotics (ICAROB2022), OS15-5, 2022. Summary of the Invention [Problem to be solved by the invention]

[0006] However, for example, if an object has a front face but the primitive shape is a shape whose front face cannot be defined, such as a cylinder, there is a problem in that it is not possible to have a robot grasp the object with its front face oriented in a predetermined direction, or to place the grasped object with its front face oriented in a predetermined direction. For this reason, a system using primitive shapes is not suitable for equipment that performs tasks such as grasping and placing oriented objects in an extremely small space, or plating food ingredients.

[0007] Furthermore, if the differences in shape or orientation between objects exceed a certain level, a system that uses primitive shapes cannot be adopted. Therefore, in order to perform processing on an object with a certain orientation according to the object's orientation, it is important to be able to stably detect the object's orientation. This applies not only to scenes in which an object is grasped, but also to other scenes in which some kind of processing is performed on the object.

[0008] The present invention has been made in consideration of the above circumstances, and aims to provide an attitude detection system that can stably detect the attitude of an object, and a data generation program that can generate learning data to be used in the system. [Means for solving the problem]

[0009] A posture detection device according to a first invention that meets the above-mentioned objective is a posture detection device that detects the posture of an object that does not have a skeleton based on an image of the object, and detects the posture of the object by estimating a pseudo-skeleton that combines a plurality of key points defined as feature points of the object in the image and line-shaped bones that connect two of the key points.

[0010] A data generation program according to a second invention that meets the above-mentioned objective is a data generation program that causes a computer to generate learning data to be used for training a posture detection device that detects the posture of an object that does not have a skeleton based on an image of the object, and that causes the computer to perform processing that receives data on a virtual space, data on a three-dimensional model of the object represented in the virtual space, and information on each of the three-dimensional models of a pseudo-skeleton that combines a plurality of key points defined as feature points of the object and line-shaped bones connecting two of the key points, and outputs the learning data that has information on the positions of the key points in the model-included virtual space and the lengths, positions, and orientations of the bones in the model-included virtual space.

[0011] A data generation program according to a third invention that meets the above-mentioned objective is a data generation program that causes a computer to generate learning data to be used for training a posture detection device that detects the posture of an object without a skeleton based on an image of the object, and that causes the computer to perform a process in which a background image, a plurality of object images of the object, and information for each of the plurality of object images of a pseudo-skeleton that combines a plurality of key points defined as feature points of the object captured in the object images and line-shaped bones connecting two of the key points, are given to the computer, in which the computer is given information for each of the plurality of object images of a pseudo-skeleton that combines a plurality of key points defined as feature points of the object captured in the object images and line-shaped bones connecting two of the key points, and outputs the learning data that has an integrated image in which the plurality of object images are arranged on the background image, and information on the positions of the key points in the integrated image, and the lengths, positions, and orientations of the bones in the integrated image. [Effects of the Invention]

[0012] The posture detection device of the first invention detects the posture of an object by estimating a pseudo-skeleton that combines multiple key points defined as feature points of the object in an image and line-shaped bones connecting two key points, so that it can stably detect the posture of objects whose shapes vary from individual to individual.

[0013] The data generation programs according to the second and third aspects of the present invention can be used to generate learning data that can be used by the attitude detection device according to the first aspect of the present invention, and therefore the data generation programs are compatible with the attitude detection device. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is an explanatory diagram of a posture detection device according to an embodiment of the present invention; [Figure 2] FIG. 2 is an explanatory diagram of an image of an object. [Figure 3] FIG. 1 is an explanatory diagram of a pseudo-skeleton of an object. [Figure 4] FIG. 10 is an explanatory diagram of a process for generating learning data. DETAILED DESCRIPTION OF THE INVENTION

[0015] Next, with reference to the accompanying drawings, embodiments embodying the present invention will be described to aid in understanding the present invention. 1 and 3, a posture detection device 10 according to an embodiment of the present invention is a device that detects the posture of an object W, which does not have a skeleton, based on an image T captured of the object W. These will be described in detail below.

[0016] In this embodiment, the object W is a flexible, shapeless object (a shape that can be changed) that varies in shape, and is, for example, fried shrimp, which is an example of a food ingredient, as shown in Fig. 2. Examples of food ingredients that vary in shape include sautéed chicken, strawberries, and broccoli. The posture detection device 10 can be configured by hardware such as a CPU and memory, and software. In this embodiment, as shown in FIG. 1, the posture detection device 10 is connected to a hand-type robot 12 having a camera 11 attached to the tip side (the side where the part for grasping an object is provided).

[0017] The posture detection device 10 includes a camera driver 13 that exchanges signals with the camera 11, a pseudo-skeleton estimation unit 14 that analyzes the image T (see Figure 3) captured by the camera 11 and derives predetermined information about the object W, a coordinate conversion unit 15 that converts two-dimensional coordinates in the image T into three-dimensional coordinates, a planning unit 16 that creates a path plan for the robot 12 based on the output from the coordinate conversion unit 15, and an arm control unit 17 that controls the robot 12 in accordance with the created path plan.

[0018] In this embodiment, a plurality of objects W of one type are placed in a predetermined location that can be imaged by the camera 11. The orientation detection device 10 is designed to grasp the plurality of objects W (meaning "plurality," the same applies hereinafter) placed in the predetermined location one by one and place them in different positions with the same orientation, but is not limited to this. For example, the objects grasped by the robot 12 may be a plurality of types of objects W, or a process of applying seasoning to a predetermined location of the object W may be performed instead of the process of grasping the object W.

[0019] The camera 11 is placed at a reference position (for example, at a height of 50 cm directly above where the objects W will be placed) by the robot 12, and captures images of multiple objects W placed in predetermined locations. The pseudo-skeleton estimation unit 14 has a software program (artificial intelligence in this embodiment), acquires the image T captured by the camera 11 via the camera driver 13, and estimates (identifies) a pseudo-skeleton S in the image T, which is a combination of multiple key points K defined as feature points of the objects W and one or more line-shaped bones B connecting two key points K, as shown in FIG. 3 .

[0020] Skeleton estimation algorithms (including AI) for estimating a pseudo-skeleton S are mainly of the bottom-up type and the top-down type, and this embodiment employs the bottom-up type. The bottom-up type first estimates (searches) all key points K in an image T in which multiple objects W are captured, and then connects the key points K to estimate bones B. In contrast, the top-down type uses an object detection algorithm to detect objects W in the image T, and then estimates key points K and bones B for each detected object W.

[0021] Generally, the bottom-up type is more robust to occlusion and noise than the top-down type. However, the top-down type can also be adopted. In FIG. 3, the rectangular frame surrounding the object W is a bounding box. The pseudo-skeleton estimation unit 14 may be designed to estimate the bounding box in addition to the pseudo-skeleton S. Furthermore, segmentation may be designed.

[0022] Furthermore, when the camera 11 captures images of multiple types of objects W and the robot 12 performs a predetermined process on the types of objects W, the processing can be performed by adding a classifier to the posture detection device that identifies and classifies each object captured in the image T, and by providing the pseudo-skeleton estimation unit with an algorithm corresponding to each type of object W.

[0023] In this embodiment, as shown in Figure 2, the tip of the head side (an example of an end), the belly, and the tail of the object W, which is a fried shrimp, are defined as feature points of the object W, and the pseudo-skeleton S of the object W is defined to have three key points K corresponding to each of the three feature points of the object W, as well as a linear bone B connecting the key point K at the tip of the head side to the key point K at the belly, and a linear bone B connecting the key point K at the tip of the belly to the key point K at the tail.

[0024] In this embodiment, a portion that often bends in a fried shrimp (an example of a bent portion) is set as a feature point of the belly. This is because, when feature points are set at bent portions, more detailed information about the posture of the target object W can be output compared to when feature points are set at straight portions. It is not necessary to use only these three feature points for the fried shrimp, and any part other than the tip of the head, the belly, and the tail may be used as the feature points, and the number of feature points may be two or four or more. In this embodiment, in addition to the three feature points and the two bones B, a gripping part G is also defined as a part of the object W to be gripped.

[0025] Whether the object W is fried shrimp or something else, in order to stably detect the posture of the object W, each object W must have two or more (i.e., multiple) key points K (feature points) and one or more bones B. In this embodiment, the general rule is that for any type of object W, all key points K are connected to one or more bones B (sometimes multiple bones B).

[0026] Here, the number and positions of key points K (feature points) for the object W, the number, positions, orientations, and lengths of bones B, and the position of the gripping part G can be determined in advance by a person or dedicated software (including artificial intelligence). In the case of an object that has branches like broccoli (the part where the stalk splits into two or more branches), it is conceivable to determine the branching points as key points K (feature points).

[0027] As shown in FIG. 1, the pseudo skeleton estimation unit 14 is connected to a data generation device 18 that generates training data for training the pseudo skeleton estimation unit 14. The data generation device 18 is composed of a computer, a data generation program (software program) that causes the computer to generate training data, and the like. The training data includes, for each of a plurality of objects W, information on the image of the object W, as well as information on key points K and bones B of the image of the object W. The "image of object W" in the "information on the image of object W" is preferably an image of the object W that is different in shape, or an image of the object W seen from a different viewpoint, even if the object W is of the same type.

[0028] As shown in FIG. 4, the image information of the object W is data of a three-dimensional model M of the object W that is represented in a virtual three-dimensional space (hereinafter referred to as "virtual space R") (applicable to virtual space R), or an image (hereinafter also referred to as "object image") of an actual object W or a pseudo object W. Note that the data of the virtual space R and the three-dimensional model M are data that can be handled by a three-dimensional simulator. When the image information of the object W is data of the three-dimensional model M of the object W, the data generation program causes the computer of the data generation device 18 to perform the following process.

[0029] In other words, the relevant process is a process in which data on the virtual space R, information on the pseudo-skeleton S for each of multiple three-dimensional models M, and information on the gripping parts G of each three-dimensional model M are given to the computer, and the computer outputs learning data that includes a model-included virtual space MR in which multiple three-dimensional models M are arranged in the virtual space R, the position of key points K in the model-included virtual space MR, information on the length, position, and orientation of bones B in the model-included virtual space MR, and information on the position of the gripping parts G in the model-included virtual space MR.

[0030] Furthermore, when the image information of the object W is an image obtained by capturing an image of the object W, the data generation program causes the computer of the data generating device 18 to perform the following process. That is, in this process, a background image, a plurality of object images obtained by capturing an image of the object W, information on the pseudo-skeleton S for each of the plurality of object images, and information on the gripping portion G of each object image are provided to the computer, and the computer outputs the learning data having an integrated image in which the plurality of object images are arranged on the background image, information on the positions of key points K in the integrated image, information on the lengths, positions, and orientations of bones B in the integrated image, and information on the positions of the gripping portions G in the integrated image.

[0031] The pseudo-skeleton estimation unit 14 learns based on the training data and creates an algorithm for detecting the pseudo-skeleton S of the object W in an image, i.e., multiple key points K, one or more bones B, and gripping parts G, from an image of one or more objects W captured by the camera 11 from a reference position. The training data does not have to include the gripping parts G. If the training data does not include the gripping parts G, the pseudo-skeleton estimation unit 14 learns (creates) an algorithm for estimating the pseudo-skeleton S from the image captured by the camera 11, as well as an algorithm for estimating the gripping parts G for the pseudo-skeleton S, based on the training data.

[0032] After the pseudo-skeleton estimation unit 14 estimates all pseudo-skeleton S and the gripping parts G of each pseudo-skeleton S in the image T, the coordinate conversion unit 15 converts the two-dimensional coordinates of each key point K and gripping part G for each estimated pseudo-skeleton S into three-dimensional coordinates using a known method (for example, a method using trigonometric functions). Information on the three-dimensional coordinates of the bones S can be derived from information on the three-dimensional coordinates of the key points K.

[0033] The planning unit 16 determines the next object W to be grasped by the robot 12 from among the multiple objects W captured in the image T, estimates each key point K of the object W, the three-dimensional coordinates of each bone B (i.e., the three-dimensional coordinates of the pseudo-skeleton S), and the position of the grasping part G, and creates a path plan for the robot 12. Here, since estimating the three-dimensional coordinates of the pseudo-skeleton S of the object W corresponds to estimating the posture of the object W, the posture detection device 10 detects the posture of the object W captured in the image T based on the image T. [Example]

[0034] Next, an experiment conducted to confirm the effects of the present invention will be described. For the experiment, 325 sets of experimental samples (data) were created based on 325 images of five loosely stacked fried shrimp models, including the images, pseudo-skeleton information, and bounding box information (position and size information). The pseudo-skeleton information included position information for three key points corresponding to the tip of the head, belly, and tail of the fried shrimp, as well as the position, length, and orientation of each of the two bones connecting the three key points. Of the 325 experimental samples, 300 sets were used for training the posture detection system, and the remaining 25 sets were used for testing the posture detection system.

[0035] First, the trained posture detection system was given 25 images corresponding to 25 sets of test samples to detect the pseudo-skeleton. The detected pseudo-skeleton information was compared with a predetermined pseudo-skeleton for the 25 sets of test samples, and the detection accuracy of the pseudo-skeleton by the posture detection system was calculated using mAP. The experimental result was mAP of 0.795. Furthermore, when a similar experiment was conducted on images of a human, the mAP was 0.724, so the mAP value in this experiment was at the same level as the value when using humans as the target.

[0036] Next, a hand-type robot equipped with a camera was connected to the trained posture detection device, and the camera was made to capture images of multiple model fried shrimp piled up in bulk. The image was then fed to the posture detection device, which was made to detect the pseudo-skeleton of the model fried shrimp in the image. The posture detection device was then controlled to make the hand-type robot perform the task of grasping each model fried shrimp and placing them in a different position. This experiment was conducted four times, and the number of model fried shrimp that the hand-type robot was able to grasp in the specified orientation was verified.

[0037] The results of the experiment were as follows: In the following, "number of grasps" refers to the number of model fried shrimp grasped by the hand-type robot, "number of successes" refers to the number of model fried shrimp grasped in a specified direction by the hand-type robot, and "success rate" refers to the ratio of the number of successes to the number of grasps.

[0038] 1st time: Number of grasps: 7, number of successes: 7, success rate 100% 2nd time: Number of grasps: 8, number of successes: 8, success rate 100% 3rd time: Number of grasps: 9, number of successes: 8, success rate 88.8% 4th time: Number of grasps: 7, number of successes: 7, success rate 100% Total for 1st to 4th attempts: 31 grasps, 30 successes, success rate 96.8%

[0039] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and all changes in conditions that do not depart from the gist of the present invention are within the scope of application of the present invention. For example, it is not necessary to use the ends of an object as key points, and even if the object has a bent part, it is not necessary to use the bent part as a key point. Also, it is possible to use a pseudo-skeleton in which three or more bones are connected to one key point.

[0040] Furthermore, the orientation detection device does not have to be used with a robot, but can be used with a device other than a robot, or can be used simply to detect the orientation of an object. When the orientation detection device is used with a robot, the robot does not have to be a hand-type robot, and there are no restrictions on the robots that can be used with the orientation detection device (for example, it can be used with a mobile manipulator). Furthermore, the posture detection device does not need to handle the concept of a gripper (the pseudo-skeleton may or may not have a gripper). [Explanation of symbols]

[0041] 10: Posture detection device, 11: Camera, 12: Robot, 13: Camera driver, 14: Pseudo skeleton estimation unit, 15: Coordinate conversion unit, 16: Planning unit, 17: Arm control unit, 18: Data generation device, B: Bone, G: Grasping unit, K: Key point, M: 3D model, MR: Virtual space with model, R: Virtual space, S: Pseudo skeleton, T: Image, W: Object

Claims

1. A posture detection device that detects the posture of an object that does not have a skeleton based on an image of the object, A posture detection device characterized by detecting the posture of the object by estimating a pseudo-skeleton that combines a plurality of key points defined as feature points of the object in the image and line-shaped bones connecting two of the key points.

2. 2. The posture detection device according to claim 1, wherein the pseudo-skeleton has the key points corresponding to the ends of the object.

3. 3. The posture detection device according to claim 1, wherein the pseudo-skeleton has key points corresponding to bent portions of the object.

4. A data generation program that causes a computer to generate learning data used for training a posture detection device that detects the posture of an object that does not have a skeleton based on an image of the object, the data generation program comprising: The computer, Virtual space data and data of a three-dimensional model of the object represented in the virtual space; Information on each of the plurality of three-dimensional models of a pseudo-skeleton that combines a plurality of key points defined as feature points of the object and line segment bones connecting two of the key points is given, outputting the learning data including a model-included virtual space in which a plurality of the three-dimensional models are arranged in the virtual space, and information on the positions of the key points in the model-included virtual space and the lengths, positions, and orientations of the bones in the model-included virtual space; A data generation program characterized by causing a process to be performed.

5. A data generation program that causes a computer to generate learning data used for training a posture detection device that detects the posture of an object that does not have a skeleton based on an image of the object, the data generation program comprising: The computer, A background image and A plurality of object images obtained by capturing images of the object; information for each of the plurality of object images, which is a pseudo-skeleton combining a plurality of key points defined as feature points of the object captured in the object images and line segment bones connecting two of the key points; outputting the learning data having an integrated image in which the plurality of object images are arranged on the background image, and information on the positions of the key points in the integrated image and the lengths, positions, and orientations of the bones in the integrated image; A data generation program characterized by causing a process to be performed.

Images