Data Acquisition and Control Equipment

Abstract

To provide a data collection device and control device, which enable efficient collection of datasets for machine learning.SOLUTION: A data collection device is provided, comprising an acquisition unit for acquiring captured image data of an object deformed by a robot arm, a determination unit configured to determine whether the object is flexible or not based on the amount of deformation of the object, and a storage unit configured to store the image data of the object in association with a determination result of the determination unit in order to use the data for controlling the robot arm.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a data collection device and a control device that collect data sets for AI (artificial intelligence) learning. [Background technology]

[0002] Recent advances in robotics technology have led to increased automation using robotic systems in factories and logistics warehouses. In order to apply such robotic systems to sites such as stores where many general consumers reside, the robotic systems must be robust to the environment. However, with regard to image recognition in particular, it is difficult to ensure robustness in a changing environment. To achieve image recognition that is robust to the environment, AI-based image recognition technology is required. By using AI image recognition to recognize the type, position, and orientation of an object, a robot can grasp and manipulate that object.

[0003] However, image recognition using AI requires a huge amount of training data set. Because collecting such a large amount of training data set is costly, reducing the collection cost is of great value. Furthermore, in order to recognize the position and orientation of an object, it is important not only to photograph the object but also to label the captured image with the object's position and orientation information. Therefore, a method has been disclosed in which the image of the object and markers placed at known positions around the object are recognized, and the position and orientation of the object are estimated based on the relative position information and orientation information from the markers (Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 189661 Summary of the Invention [Problem to be solved by the invention]

[0005] However, while the conventional methods described above can label the position and orientation of rigid objects in images, they always capture images of flexible objects in the same deformed state. This may prevent the inclusion of various deformed states of flexible objects in the dataset. Furthermore, if a flexible object is deformed in a different state from when the dataset was collected, the correct position and orientation of the object may not be estimated.

[0006] The present disclosure aims to provide a data collection device and a control device that efficiently collect datasets for machine learning. [Means for solving the problem]

[0007] The data collection device of the present disclosure includes an acquisition unit that acquires image data of an object deformed by a robot arm, and a device that acquires image data of the object deformed by a robot arm. The area of ​​the object changes depending on a determination unit that determines whether the object is a flexible object based on the result of the determination by the determination unit, and a storage unit that stores image data of the object in association with the determination result by the determination unit for use in controlling the robot arm. The acquisition unit acquires image data of the object before deformation and image data of the object after deformation, and calculates the area of ​​the object in each image data. When the object is determined to be a flexible object, the storage unit stores image data of the object deformed by the robot arm at a position where the contour width in a predetermined width direction of the object in an undeformed state is longest, in association with the determination result of the determination unit. .

[0008] The control device of the present disclosure generates data in which image data of an object deformed by a robot arm is associated with a determination result of whether the object is flexible or not based on the amount of deformation of the object. and, if the object is determined to be a flexible object, image data of the object deformed by the robot arm at a position where the contour width in a predetermined width direction is longest in the region of the object before deformation. and a control unit that controls the robot arm based on the control model stored in the storage unit. The data collection device disclosed herein includes an acquisition unit that acquires image data of an object deformed by a robot arm, a judgment unit that determines whether the object is a flexible object based on the amount of deformation of the object, and a memory unit that stores image data of the object in association with the judgment result of the judgment unit for use in controlling the robot arm.When the object is determined to be a flexible object, the memory unit stores image data of the object deformed by the robot arm at a position where the contour width in a specified width direction is longest in the undeformed object in association with the judgment result of the judgment unit. [Effects of the Invention]

[0009] According to the present disclosure, datasets for machine learning can be collected efficiently. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram showing a configuration of a robot arm device including a data collection device according to a first embodiment. [Figure 2] FIG. 10 is a diagram showing a state in which the determination result and the position and orientation of the object relative to the photographing position are labeled for the image data. [Figure 3] FIG. 10 is a diagram showing how the gripping position of the robot arm is calculated. [Figure 4] 1 is a block diagram showing a configuration of a data collection device according to a first embodiment. [Figure 5] 3 is a flowchart showing the operation of the first embodiment. [Figure 6] FIG. 10 is a diagram showing how the gripping position of the robot arm is calculated. [Figure 7] FIG. 10 is a block diagram showing the configuration of a learning device according to a second embodiment. [Figure 8] FIG. 10 is a diagram showing the configuration of a robot arm device including a control device according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] (Embodiment 1) The data collection device disclosed herein operates a robot arm and a rotary table to change the relative position between the camera and the object and the object's orientation in order to accurately estimate the position and orientation of an object consisting of a flexible object that can undergo various deformations. The data collection device then photographs the object with a camera while applying deformations to the object using a robot hand, and labels the position and orientation of the object relative to the camera, calculated from the geometric relationship (geometric information) of the robot arm, in the image data.

[0013] First, the configuration of a robot arm device 100 including a data collection device 21 according to the present disclosure will be described. Fig. 1 shows the overall configuration of a robot arm device 100 according to the first embodiment.

[0014] The robot arm device 100 includes a robot arm 7, a camera 4, a rotary table 5, a control device 8, and a data collection device 21. The control device 8 and the data collection device 21 are electrically connected to each other. The control device 8 and the data collection device 21 are also electrically connected to the robot arm 7, the camera 4, and the rotary table 5, respectively.

[0015] The robot arm 7 includes a vertically articulated robot 2 in which adjacent links are connected by multiple joints, and a robot hand 3 that functions as a hand. The vertically articulated robot 2 has an elongated shape, and its base end is fixed to a fixing base 1. This fixing base 1 is used to fix the robot arm 7 and the rotary table 5, and is fixed to the floor, for example. The robot hand 3 is used to grasp an object T placed on the rotary table 5, and is attached to the tip of the vertically articulated robot 2.

[0016] The camera 4 is attached to the tip of the vertical articulated robot 2 and photographs an object T placed on a rotary table 5 .

[0017] The rotary table 5 changes the attitude (photographing angle) of the object T relative to the camera 4, and is fixed to the fixing base 1. The rotary table 5 is rotated by a servo motor, and may rotate, for example, in the circumferential direction 6 of a rotation axis extending in the vertical direction. Note that, although the rotary table 5 is used in this embodiment, the present invention is not limited to this as long as the photographing angle of the object T can be changed. For example, multiple cameras may be arranged instead of using the rotary table 5. This allows for efficient photographing.

[0018] The data collection device 21 includes an acquisition unit 22, a determination unit 23, and a storage unit 24.

[0019] When the object T deformed by the robot hand 3 of the robot arm 7 is photographed by the camera 4, the acquisition unit 22 acquires image data of the photographed object T from the camera 4 in order to determine the flexibility of the object T in the determination unit 23. The acquisition unit 22 also has a position / posture calculation unit 22a and an area calculation unit 22b.

[0020] The area calculation unit 22b acquires image data of the object T before and after deformation by the robot hand 3, and calculates the area of ​​the object T in each image data. For example, the area calculation unit 22b may acquire, in advance, background image data obtained by photographing the turntable 5 in a state in which the object T is not present from the camera 4, and extract an object area, which is the area where the object T exists, based on the background image data by subtracting the background portion from the image data obtained by photographing the object T. Then, the area calculation unit 22b calculates the area of ​​the object T based on the extracted object area. This area changes depending on the amount of deformation of the object T, and can therefore be an index indicating the flexibility of the object T.

[0021] The position / posture calculation unit 22a calculates the position and posture of the object T relative to the shooting position of the image data, for example, based on the geometric information of the robot arm 7. For example, the position / posture calculation unit 22a may change the position and posture of the object T, which has been determined by the determination unit 23 to be a flexible object or not, into various positions and postures, and sequentially calculate the position and posture of the object T relative to the shooting position of the image data.

[0022] At this time, the robot arm 7, camera 4, and turntable 5 are geometrically constrained via the fixing base 1. Furthermore, the position and orientation calculation unit 22a has preset therein a camera coordinate system 12 indicating the position and orientation of the camera 4, a base coordinate system 13 indicating the position and orientation of the base end of the robot arm 7, and an object coordinate system 14 indicating the position and rotation direction of the turntable 5. Here, the camera coordinate system 12 is, for example, as shown in FIG. 1, a set of unit vectors (e x ,e y ,e z), and may be set so that the position and orientation change together with the camera 4. Similarly, the base coordinate system 13 and the object coordinate system 14 are respectively configured by unit vectors (e x ,e y ,e z ), and may be set so that the position and orientation change together with the base end of the robot arm 7 and the turntable 5. Note that since the position and orientation of the object T change depending on the position and rotation direction of the turntable 5, the object coordinate system 14 indicates the position and orientation of the object T. Therefore, the position / orientation calculation unit 22a may calculate changes in these coordinate systems based on the angles of the joints of the robot arm 7 and the rotation angle of the turntable 5, and calculate the position and orientation of the object T relative to the camera 4 based on the calculated changes in the coordinate systems.

[0023] Specifically, the position / orientation calculation unit 22a calculates the position and orientation of the base coordinate system 13 of the robot arm 7 relative to the camera coordinate system 12, based on the angles of each joint of the robot arm 7. Furthermore, the position / orientation calculation unit 22a calculates the position and orientation of the object coordinate system 14 of the object T relative to the base coordinate system 13 of the robot arm 7, based on the rotation angle of the turntable 5. This associates the camera coordinate system 12 with the object coordinate system 14 via the base coordinate system 13 of the robot arm 7. Therefore, the position / orientation calculation unit 22a may calculate the position and orientation of the object coordinate system 14 relative to the camera coordinate system 12, based on the position and orientation of the base coordinate system 13 relative to the camera coordinate system 12 and the position and orientation of the object coordinate system 14 relative to the base coordinate system 13. For example, if the position (x, y, z) of the camera coordinate system 12 relative to the base coordinate system 13 is (-2, -2, -1) and the position of the object coordinate system 14 relative to the base coordinate system 13 is (-2, -2, 0), the position of the object coordinate system 14 relative to the camera coordinate system 12 will be calculated to be (0, 0, 1). Similarly, a change in the attitude of the object coordinate system 14 relative to the camera coordinate system 12 may be calculated based on, for example, the direction of the unit vector of the camera coordinate system 12 relative to the unit vector of the base coordinate system 13 and the direction of the unit vector of the object coordinate system 14 relative to the unit vector of the base coordinate system 13.

[0024] The determination unit 23 determines whether or not the object T is a flexible object based on the amount of deformation of the object T. For example, the determination unit 23 may determine whether or not the object T is a flexible object based on the area of ​​the region of the object T calculated by the area calculation unit 22b.

[0025] The storage unit 24 stores a data set in which the image data of the object T is associated with the determination result of the determination unit 23, i.e., "flexible object" or "not flexible object," for use in controlling the robot arm 7. The storage unit 24 may also store the position and orientation of the object T with respect to the shooting position calculated by the position / orientation calculation unit 22a in further association with the image data.

[0026] 2, the storage unit 24 may label and store image data D of an image of an object T with the determination result and the position and orientation of the object T relative to the photographing position. In this way, data that is grouped by labeling the determination result and the position and orientation of the object T for the image data D can be used for training a machine learning model, as will be described later.

[0027] The determination unit 23 may determine whether the object T is a rigid object. For example, the determination unit 23 may determine that an object T other than a flexible object is a rigid object. In this case, the storage unit 24 may store image data of not only the object T made of a flexible object but also the object T made of a rigid object. In this case, the image data D stored in the storage unit 24 may be captured using different methods depending on whether the object T is a flexible object or a rigid object. For example, if the object T is a flexible object, the object T may be captured after applying an external force with the robot hand 3, and the image data D may be stored in the storage unit 24. This makes it possible to obtain image data D of the object T deformed into various shapes. On the other hand, if the object T is a rigid object, the object T may be captured without applying an external force, and the image data D may be stored in the storage unit 24.

[0028] The control device 8 controls each part of the robot arm device 100 to collect a data set for machine learning, including image data D obtained by photographing the target object T, in the data collection device 21.

[0029] Next, an example of how the determination unit 23 determines the object T will be described.

[0030] First, the control device 8 operates each joint of the robot arm 7 based on the camera coordinate system 12, the base coordinate system 13, and the object coordinate system 14, and moves the tip of the robot arm 7, on which the camera 4 is disposed, to a preset initial position and initial posture. This initial position and initial posture may be set in advance, for example, so that the entire object T is included near the center of the shooting area of ​​the camera 4.

[0031] When the control device 8 places the robot arm 7 in an initial position and initial posture, it controls the camera 4 to capture an image of the object T. The camera 4 outputs the captured image data D to the area calculation unit 22b of the acquisition unit 22. Then, the area calculation unit 22b extracts an object region corresponding to the object T from the image data D captured by the camera 4.

[0032] Next, the control device 8 controls the robot hand 3 to grip the object T with a preset reference gripping force, and releases the object T several seconds after the gripping state. The robot arm 7 is then returned to its initial position and initial posture, and the object T is photographed again by the camera 4. The area calculation unit 22b then extracts an object region from the image data D photographed by the camera 4. The area calculation unit 22b calculates the area of ​​the object region in the image data D based on the extracted object region, and calculates the reduction rate of the area of ​​the object region after gripping relative to the area of ​​the object region before gripping using the following equation 1.

[0033]

number

[0034] When the reduction rate of the area of ​​the object region is equal to or greater than the reference value, the determination unit 23 determines that the object T is a flexible object. On the other hand, when the reduction rate of the area of ​​the object region is less than the reference value, the determination unit 23 determines that the object T is a rigid object.

[0035] Next, the acquisition unit 22 collects image data D of various deformation states of the object T determined to be a flexible object by the determination unit 23. At this time, the control device 8 may control the robot hand 3 to grip the object T and apply an external force to the object T in order to deform the object T.

[0036] Next, an example of deforming the target object T with the robot hand 3 in order to collect image data D will be described.

[0037] The robot hand 3 deforms the flexible object T by clamping the object T. In this case, in order to efficiently collect deformation pattern images of the object T, it is necessary to change the object T into various deformation states in a short time, and to achieve this, it is necessary to quickly determine the gripping force and gripping position of the robot hand 3.

[0038] 3, the area calculation unit 22b extracts an object region R from image data D obtained by photographing the object T in an undeformed state (a state in which no force is applied by the robot hand 3). Next, the area calculation unit 22b calculates the edge width W in the width direction D1 parallel to the turntable 5 for multiple height positions (positions in the height direction D2 perpendicular to the turntable 5) based on the edge of the extracted object region R. Then, the control device 8 selects the positions with the largest edge width W as gripping positions P1 and P2, and controls the robot hand 3 to grip the gripping positions P1 and P2.

[0039] As a result, when the determination unit 23 determines that the object T is a flexible object, the storage unit 24 stores image data D of the object T deformed by gripping the positions P1 and P2. That is, the storage unit 24 stores image data D of the object T deformed by the robot arm 7 at positions P1 and P2 where the edge width W in the predetermined width direction D1 is longest in the undeformed object T, in association with the determination result of the determination unit 23. In this way, by the robot hand 3 gripping the positions P1 and P2 where the edge width W is longest in the object T, the object T can be deformed more greatly, and deformation pattern images can be collected efficiently.

[0040] In this case, the gripping force of the robot hand 3 may be set according to the size of the contour width W (gripping contour width) of the gripping positions P1 and P2. For example, the control device 8 may determine the gripping force of the robot hand 3 according to the size of the gripping contour width W relative to a preset reference contour width. This allows the widest (largest bulge) position on the object T to be gripped with a gripping force according to that width. The formula for calculating the gripping force of the robot hand 3 is shown in Equation 2 below.

[0041]

number

[0042] When the gripping contour width W is the same as the reference contour width, the control device 8 controls the robot hand 3 to grip with a preset reference gripping force. When the gripping contour width W is larger than the reference contour width, the control device 8 controls the robot hand 3 to grip with a gripping force greater than the reference gripping force, depending on the increase in the gripping contour width W relative to the reference contour width. At this time, if the gripping force exceeds a preset maximum gripping force, the control device 8 controls the robot hand 3 to grip with the maximum gripping force. This makes it possible to collect image data D with approximately the same amount of deformation applied to various objects T.

[0043] In the above-described first embodiment, in order to deform the object T, an external force is applied to the object T by pinching the object T with the robot hand 3. This pinching method of applying an external force applies the same external force inward from both sides of the object T (grasping positions P1 and P2), so it is possible to apply an external force to the object T without the object T moving significantly on the turntable 5.

[0044] FIG. 4 shows the hardware configuration of the data collection device 21.

[0045] The data collection device 21 includes a storage unit 24, a processor 25, and a user interface (UI) 26, which are interconnected via a bus B.

[0046] In addition, the programs or instructions that realize the various functions and processes described below in the data collection device 21 may be downloaded from any external device via a network, or may be provided from a removable storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or flash memory.

[0047] The memory unit 24 is realized by one or more non-transitory storage media, such as random access memory, flash memory, or a hard disk drive, and stores installed programs or instructions as well as files, data, etc. used in executing the programs or instructions.

[0048] The processor 25 may be realized by one or more central processing units (CPUs), graphics processing units (GPUs), processing circuitry, etc., each of which may be configured with one or more processor cores. The processor 25 executes various functions and processes of the control device 8, which will be described later, in accordance with programs, instructions, data such as parameters required to execute the programs or instructions, etc., stored in the storage unit 24.

[0049] The UI 26 may be composed of input devices such as a keyboard, a mouse, a camera, a microphone, etc., output devices such as a display, a speaker, a headset, a printer, etc., and input / output devices such as a touch panel, and realizes an interface between the user and the control device 8. For example, the user may operate the control device 8 by operating a GUI (Graphical User Interface) displayed on a display or a touch panel using a keyboard, a mouse, etc.

[0050] It should be noted that the above-described hardware configuration is merely an example, and the control device 8 according to the present disclosure may be realized by any other appropriate hardware configuration.

[0051] Next, the data collection operation of this embodiment will be described with reference to the flowchart shown in FIG.

[0052] First, in step S1, the control device 8 moves the robot arm 7 so that it assumes a preset initial position and initial posture. This initial position and initial posture may be set, for example, so that the entire object T placed on the turntable 5 is included near the center of the imaging area of ​​the camera 4. Based on geometric information of the robot arm 7, such as the camera coordinate system 12, the base coordinate system 13, and the object coordinate system 14, the control device 8 drives each joint of the robot arm 7 so that the tip of the robot arm 7 assumes the initial position and initial posture.

[0053] Next, control device 8 controls camera 4 to capture an image of object T. Camera 4 outputs the captured image data D to area calculation unit 22b of acquisition unit 22. Then, as shown in FIG. 6 , area calculation unit 22b extracts object region R corresponding to object T from image data D and calculates the area of ​​object region R in step S2.

[0054] Once the object T has been photographed in this manner, the control device 8 controls the robot arm 7 in step S3 to grip predetermined positions Pa1 and Pa2 of the object T with a reference gripping force. At this time, the robot arm 7 may be controlled to grip, for example, the predetermined positions Pa1 and Pa2, side portions corresponding to the central portion C of the object region R in the height direction D2 from the photographing direction. That is, the control device 8 controls the robot arm 7 to grip two points Pa1 and Pa2 where a straight line passing through the central portion C of the object region R in the height direction D2 and extending in the width direction D1 intersects with the outline of the object region R.

[0055] Thereafter, in step S4, control device 8 moves the robot arm back to its initial position and initial posture. Control device 8 also controls camera 4 to capture an image of object T, and the captured image data D is output from camera 4 to acquisition unit 22. Then, in step S5, area calculation unit 22b of acquisition unit 22 extracts object region R from the image data D and calculates the area of ​​object region R.

[0056] Subsequently, in step S6, the determination unit 23 determines whether or not the object T is a flexible object based on the reduction rate of the area of ​​the object region R after gripping relative to the area of ​​the object region R before gripping.

[0057] If it is determined that the object T is not a flexible object, that is, if it is determined that the object T is a rigid object, then in step S7, the control device 8 controls the camera 4 to rotate the turntable 5 by a certain angle and then capture an image of the object T. Next, the control device 8 controls the camera 4 to further rotate the turntable 5 by the same certain angle as before and then capture an image of the object T. In this way, the control device 8 repeats capturing images of the object T from different angles until the turntable 5 makes one rotation.

[0058] When the turntable 5 makes one rotation, the control device 8 changes the angle of each joint of the robot arm 7, i.e., changes the position (e.g., height position) of the object T relative to the imaging position, and controls the camera 4 to image the object T again while rotating the turntable 5 by a certain angle. The control device 8 repeats imaging the object T from different angles until the turntable 5 makes one rotation. In this way, the object T is imaged at various positions and postures. Then, in step S8, the storage unit 24 stores the image data D of the object T imaged at the various positions and postures in association with the determination result of the determination unit 23 that the object is a "rigid object." The storage unit 24 may also store information related to the object region in association with the image data D of the rigid object.

[0059] In this way, the data collection device 21 can efficiently collect a data set for machine learning that includes image data D of rigid objects. Furthermore, by changing the angle of each joint of the robot arm 7, it is possible to photograph the target object T at different angles of view, and to collect image data D of rigid objects photographed at various positions and postures.

[0060] At this time, the position / posture calculation unit 22a may calculate the position and posture of the object T relative to the shooting position of the image data D based on geometric information of the robot arm 7, for example, the camera coordinate system 12, the base coordinate system 13, and the object coordinate system 14. Then, the storage unit 24 stores the image data D of the object T photographed at various positions and angles in association with the position and posture of the object T relative to the shooting position.

[0061] This makes it possible to efficiently collect training data in which the position and orientation relationship between the camera 4 and the target object T is labeled in the image data D of the rigid object.

[0062] Furthermore, in step S9, the control device 8 determines whether or not the joint angles of the robot arm 7 have been changed a predetermined number of times or more. If the joint angles of the robot arm 7 have not been changed a predetermined number of times or more, the control device 8 changes the joint angles of the robot arm 7 in step S10, and then returns to step S7 to control the camera 4 to capture an image of the target T. The control device 8 repeats steps S7, S8, S9, and S10 until it is determined that the joint angles of the robot arm 7 have been changed a predetermined number of times or more. Then, in step S9, the control device 8 ends collection of image data D.

[0063] On the other hand, if it is determined in step S6 that the object T is a flexible object, the process proceeds to step S11, where the area calculation unit 22b extracts an object region R from the image data D. As shown in Fig. 3, the control device 8 determines the gripping positions P1 and P2 and gripping force of the robot hand 3 based on the contour width W of the extracted object region R. Next, in step S12, the control device 8 controls the robot hand 3 so as to apply an external force to the object T using the determined gripping positions P1 and P2 and gripping force.

[0064] In step S13, the control device 8 rotates the turntable 5 by a certain angle, and then controls the camera 4 to capture an image of the object T. Next, the control device 8 further rotates the turntable 5 by the same certain angle as before, and then controls the camera 4 to capture an image of the object T. In this way, the control device 8 repeats capturing images of the object T from different angles until the turntable 5 makes one rotation.

[0065] When the turntable 5 makes one rotation, the control device 8 changes the joint angles of the robot arm 7, i.e., changes the position (e.g., height position) of the object T relative to the imaging position, and controls the camera 4 to image the object T again while rotating the turntable 5 by a certain angle. The control device 8 repeatedly images the object T from different angles until the turntable 5 makes one rotation. In this way, the object T is imaged at various positions and postures. Then, in step S14, the storage unit 24 stores the image data D of the object T imaged at the various positions and postures in association with the determination result of whether the object T is a "flexible object" or a "rigid object." As a result, the storage unit 24 stores the image data D of the object T in association with the determination result of whether the object T is a "flexible object" or a "rigid body." At this time, the storage unit 24 may store two sets of image data D of the object T before and after deformation by the robot arm 7 (before and after gripping) in association with the determination result of whether the object T is a flexible object. Furthermore, the storage unit 24 may store information relating to the object region R in further association with the image data D of the flexible object.

[0066] In this way, the data collection device 21 can efficiently collect a data set for machine learning that includes the flexible object image data D. Furthermore, by changing the angle of each joint of the robot arm 7, it is possible to photograph the target object T at different angles of view, and to collect the flexible object image data D photographed at various positions and postures.

[0067] At this time, the position / posture calculation unit 22a may calculate the position of the object T and the posture of the object T relative to the photographing position of the image data D based on the geometric information of the robot arm 7. Then, the storage unit 24 stores the image data D of the object T photographed at various positions and postures in association with the position and posture of the object T.

[0068] This makes it possible to efficiently collect teacher data in which the image data D of the flexible object is labeled with the position and orientation relationship between the camera 4 and the object T. In this way, the storage unit 24 collects teacher data in which the image data D is labeled with the determination result of whether the object is a flexible object or a rigid object, the position of the object T relative to the shooting position, and the orientation of the object T relative to the shooting orientation.

[0069] Furthermore, in step S15, the control device 8 determines whether the joint angles of the robot arm 7 have been changed a predetermined number of times or more. If the joint angles of the robot arm 7 have not been changed a predetermined number of times or more, the control device 8 changes the joint angles of the robot arm 7 in step S16, and then returns to step S11 to determine the gripping positions P1 and P2 and the gripping force of the robot hand 3 based on the newly extracted object region R. The control device 8 repeats steps S11, S12, S13, S14, S15, and S16 until it is determined that the joint angles of the robot arm 7 have been changed a predetermined number of times or more. If the joint angles of the robot arm 7 have been changed a predetermined number of times or more in step S15, the control device 8 ends collection of image data D.

[0070] According to this embodiment, image data D of an object T deformed by the robot arm 7 is acquired, and the determination unit 23 determines whether the object T is a flexible object based on the amount of deformation of the object T. Then, the image data D of the object T is stored in the storage unit 24 in association with the determination result of the determination unit 23 for use in controlling the robot arm 7. This makes it possible to efficiently collect a data set for machine learning.

[0071] (Embodiment 2) Hereinafter, a second embodiment of the present disclosure will be described. Here, differences from the first embodiment will be mainly described, and common reference numerals will be used for common features with the first embodiment, and detailed description thereof will be omitted.

[0072] In the second embodiment, a control model for the robot arm 7 is constructed using the data set collected in the first embodiment as training data.

[0073] 7, a learning device 31 may be connected to the data collecting device 21 of the first embodiment. The learning device 31 includes a learning unit 32 and a control model storage unit 33.

[0074] The learning unit 32 performs machine learning using the data set collected by the data collecting device 21 as training data to construct a control model to be used for controlling the robot arm 7. That is, the control model is trained using training data in which image data D of an object T deformed by the robot arm 7 is associated with a determination result of whether the object T is a flexible object. In this case, the control model may be trained using training data in which two sets of image data D of the object T before and after deformation by the robot arm 7 (before and after gripping) are associated with a determination result of the object T. In this way, the control model is constructed to receive the image data D of the object T as input and output whether the object T is a flexible object. The control model may also be trained using a data set of the image data D, the determination result, and the position and posture of the object T as training data. In this way, the control model is constructed to further output the position and posture of the object T in response to the input of the image data D of the object T.

[0075] The control model storage unit 33 stores the control model constructed by the learning unit 32.

[0076] The hardware configuration of the learning device 31 is the same as that of the data collecting device 21 in the first embodiment, and therefore a description thereof will be omitted.

[0077] According to this embodiment, the learning unit 32 constructs a control model trained using as training data data in which image data D of the object T deformed by the robot arm 7 is associated with a determination result as to whether the object T is a flexible object based on the amount of deformation of the object T. In this way, the use of an efficiently collected dataset for machine learning makes it possible to easily construct a control model.

[0078] (Embodiment 3) Hereinafter, a third embodiment of the present disclosure will be described. Here, differences from the first and second embodiments will be mainly described, and common reference numerals will be used for common features with the first and second embodiments, and detailed descriptions thereof will be omitted.

[0079] In the third embodiment, the control model constructed in the second embodiment is used to control the robot arm 7.

[0080] 8 shows the configuration of a robot arm device 100 equipped with a control device 41 according to the present disclosure. The robot arm device 100 includes a robot arm 7, a camera 4, a turntable 5, and the control device 41. The control device 41 is electrically connected to the robot arm 7, the camera 4, and the turntable 5.

[0081] The control device 41 includes a control unit 42 and a storage unit 43 .

[0082] The storage unit 43 stores the control model constructed by the learning unit 32 of the second embodiment. That is, the storage unit 43 stores a control model trained using as training data data in which image data D of the object T deformed by the robot arm 7 is associated with a determination result of whether the object T is a flexible object based on the amount of deformation of the object T. For example, the storage unit 43 may store a control model trained using as training data data in which the determination result is associated with the position and posture of the object T for the image data D.

[0083] The control unit 42 controls each unit of the robot arm device 100 based on the control model stored in the storage unit 43. For example, when information indicating that the object T is a flexible object and the position and posture of the object T are output from the control model in response to input of image data D, the control unit 42 may control the robot arm 7 to grip a predetermined position with an appropriate gripping force based on this information.

[0084] The hardware configuration of the control device 41 is the same as that of the data collecting device 21 of the first embodiment, and therefore a description thereof will be omitted.

[0085] Next, the operation of controlling the robot arm of this embodiment will be described.

[0086] First, the control unit 42 controls the robot arm 7 to grip the object T and place it on the turntable 5. Then, the control unit 42 controls the camera 4 to capture an image of the object T when it is not being gripped by the robot arm 7, i.e., the object T before deformation. Next, the control unit 42 controls the robot arm 7 to grip and deform the object T placed on the turntable 5. For example, the control unit 42 may control the robot arm 7 to grip the object T with a preset reference gripping force. Then, the control unit 42 controls the camera 4 to again capture an image of the object T placed on the turntable 5.

[0087] When the object T before and after deformation are photographed in this manner, the control unit 42 inputs the two image data D before and after deformation into a control model stored in the storage unit 43. As a result, whether or not the object T is a flexible object is output from the control model. If the object T is a flexible object, the control unit 42 controls the robot arm 7 to grip the object T with a flexible object gripping force that is preset according to the flexible object, and to perform a predetermined task such as store work. On the other hand, if the object T is not a flexible object, the control unit 42 controls the robot arm 7 to grip the object T with, for example, a gripping force greater than the flexible object gripping force.

[0088] Generally, stores handle products with various degrees of flexibility. Therefore, the robot arm 7 performing store operations is required to grip the product with a gripping force that corresponds to the flexibility of the product. Therefore, in the present disclosure, the control unit 42 controls the robot arm 7 based on the above control model so that the robot arm 7 grips the object T with a gripping force that corresponds to the flexibility of the object T. This allows the robot arm 7 to appropriately perform operations on the object T.

[0089] Furthermore, when the position and posture of the object T are further output from the control model, the control unit 42 controls the robot arm 7 to grasp the object T based on the output position and posture of the object T. This allows the robot arm 7 to grasp the object T at an appropriate position and perform the task.

[0090] According to this embodiment, the control unit 42 controls the robot arm 7 based on a control model trained using training data in which image data D of the object T deformed by the robot arm 7 is associated with a determination result of whether or not the object T is a flexible object based on the amount of deformation of the object T. This allows the control unit 42 to control the robot arm 7 to grip the object T with an appropriate gripping force based on whether or not the object T is a flexible object.

[0091] Although specific examples of the present disclosure have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and alterations of the specific examples exemplified above. [Industrial Applicability]

[0092] The data collection device and control device of the present invention can efficiently perform AI learning in automated robots that manipulate various objects. [Explanation of symbols]

[0093] 1 Fixing base 2. Vertically articulated robot 3. Robot Hand 4. Camera 5 Rotating table 6 Circumferential direction 7. Robotic Arm 8 Control Device 12 Camera Coordinate System 13 Base Coordinate System 14 Object coordinate system 21 Data collection equipment 22 Acquisition Department 22a Position / attitude calculation section 22b Area calculation part 23 Judgment section 24 Memory section 25 processors 26 User Interface 31 Learning Device 32 Learning Department 33 Control model memory unit 41 Control device 42 Control Unit 43 Storage section 100 Robot arm device C central part D1 width direction D2 Height direction P1,P2,Pa1,Pa2 Gripping position R Object Region T object W Contour width

Claims

1. an acquisition unit that acquires image data of an object deformed by the robot arm; a determination unit that determines whether the object is a flexible object based on an area of ​​the object that changes depending on an amount of deformation of the object; a storage unit that stores image data of the object in association with the determination result of the determination unit for use in controlling the robot arm, the acquisition unit acquires image data of the object before deformation and image data of the object after deformation, and calculates a region area of ​​the object in each image data; When the object is determined to be a flexible object, the storage unit stores image data of the object deformed by the robot arm at a position where the contour width in a predetermined width direction of the object in an undeformed state is longest, in association with the determination result of the determination unit. Data collection equipment.

2. the acquisition unit calculates a position and orientation of the object relative to a photographing position of the image data based on geometric information of the robot arm; The storage unit stores the position and orientation of the object in association with the image data. The data collection device of claim 1 .

3. a storage unit that stores data in which image data of an object deformed by a robot arm is associated with a determination result of whether the object is a flexible object based on the amount of deformation of the object, and, when the object is determined to be a flexible object, a control model trained using as training data image data of the object deformed by the robot arm at a position in the region of the object in a state before deformation where the contour width in a predetermined width direction is longest; a control unit that controls the robot arm based on the control model stored in the storage unit, Control device.

4. An acquisition unit that acquires image data of an object deformed by a robot arm; a determination unit that determines whether the object is a flexible object based on the deformation amount of the object; a storage unit that stores image data of the object in association with the determination result of the determination unit for use in controlling the robot arm, When the object is determined to be a flexible object, the storage unit stores image data of the object deformed by the robot arm at a position where the contour width in a predetermined width direction of the object in an undeformed state is longest, in association with the determination result of the determination unit. Data collection equipment.

Images