Data generation system, data generation method, and program

Abstract

To generate learning data that can cause a robot to learn various operations.SOLUTION: A data generation system, which generates learning data for generating a learned model in which operation of a robot including an arm part and a hand part is learned, comprises: arm image acquisition means that acquires images of multiple kinds of arm parts, and applies identification information to an image of each arm part; hand image acquisition means which acquires images of multiple kinds of hand parts, and applies identification information to an image of each hand part; and learning data generation means that generates learning data by selecting one arm part from the plurality of acquired arms, selecting one hand part from the plurality of acquired hand parts, generating an image when operating the robot in which the selected hand part is connected to the selected arm part, and applying identification information of the selected arm part and identification information of the selected hand part respectively to the generated image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a data generation system, a data generation method, and a program for generating learning data for robot movements. [Background technology]

[0002] A system is known in which a training data set including images of a robot taken from various angles is acquired in a simulation environment in order to generate a trained model that has learned the grasping motion of a robot (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6671694 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, if the type or size of the object to be grasped exceeds the range that the trained model can handle, the robot's grasping operation may become difficult, and restrictions on the robot's operation may occur.

[0005] The present disclosure has been made to solve such problems, and its main purpose is to provide a data generation system, a data generation method, and a program for generating learning data that enables a robot to learn a variety of movements. [Means for solving the problem]

[0006] In order to achieve the above object, one aspect of the present disclosure is to A data generation system that generates learning data for generating a trained model that has learned the behavior of a robot including an arm unit and a hand unit connected to a tip of the arm unit, an arm image acquisition means for acquiring images of a plurality of types of the arm portions and assigning identification information for identifying each of the acquired images of the arm portions; a hand image acquisition means for acquiring images of a plurality of types of hand portions and assigning identification information for identifying each of the acquired hand portion images; a learning data generation means for selecting one arm from the plurality of arm parts acquired by the arm image acquisition means, selecting one hand from the plurality of hand parts acquired by the hand image acquisition means, generating an image of a robot having the selected hand part connected to the selected arm part when the robot is operated, and assigning identification information of the selected arm part and identification information of the selected hand part to the generated image, respectively, thereby generating the learning data; Equipped with Data Generation System is. In this aspect, The learning data generation means may generate the learning data by operating a robot with the selected hand unit connected to the selected arm unit, generating images of the robot photographed from a plurality of different positions, and assigning identification information of the position, identification information of the selected arm unit, and identification information of the selected hand unit to each of the generated images. In this aspect, the robot performs an action on an object, further comprising an object image acquisition means for acquiring images of a plurality of types of objects and assigning identification information for identifying each of the acquired object images, The learning data generation means may select one object from the plurality of objects acquired by the object image acquisition means, generate an image of a robot having the selected hand unit connected to the selected arm unit operating on the selected object, and generate the learning data by assigning identification information of the selected arm unit, identification information of the selected hand unit, and identification information of the object to the generated image. In this aspect, the hand unit is configured as a gripping unit having a housing unit, at least two claw units connected to the housing unit and configured to grip an object, and a connecting unit connecting the housing unit and the claw units, the hand image acquisition means generates a plurality of types of the housing parts, the claw parts, and the connection parts, and assigns identification information to each of the generated housing parts, each claw part, and each connection part; The learning data generation means may select a housing part, a claw part, and a connection part from the housing part, the claw part, and the connection part generated by the hand image acquisition means, respectively, to form the gripping part, generate an image of a robot in which the formed gripping part is connected to the selected arm part and is operated, and generate the learning data by assigning identification information of the selected arm part and identification information of the housing part, the claw part, and the connection part of the formed gripping part to the generated image. In this aspect, The plurality of types of hand units may include at least one of a gripping unit that grips an object, a magnetically attracting unit that magnetically attracts an object, and a suction unit that attracts an object. In order to achieve the above object, one aspect of the present disclosure is to A data generation method for generating training data for generating a trained model that has learned the behavior of a robot including an arm unit and a hand unit connected to a tip of the arm unit, acquiring images of a plurality of types of the arm section, and assigning identification information for identifying each of the acquired images of the arm section; acquiring images of a plurality of types of hand units, and assigning identification information for identifying each of the acquired hand unit images; a step of generating the learning data by selecting one arm unit from the acquired plurality of arm units, selecting one hand unit from the acquired plurality of hand units, generating an image of a robot connected to the selected arm unit and operated, and assigning identification information of the selected arm unit and identification information of the selected hand unit to the generated image, respectively; Including, Data generation method is. In order to achieve the above object, one aspect of the present disclosure is to A program for generating learning data for generating a trained model that has learned the behavior of a robot including an arm unit and a hand unit connected to a tip of the arm unit, a process of acquiring images of a plurality of types of arm units and assigning identification information for identifying each arm unit to each of the acquired images of the arm units; A process of acquiring images of a plurality of types of hand units and assigning identification information for identifying each of the acquired hand unit images; a process of generating the learning data by selecting one arm from the acquired plurality of arm sections, selecting one hand from the acquired plurality of hand sections, generating an image of a robot connected to the selected arm section and operating the robot, and assigning identification information of the selected arm section and identification information of the selected hand section to the generated image; to the computer, program is. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to provide a data generation system, a data generation method, and a program that generate learning data that enables a robot to learn a variety of movements. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a schematic hardware configuration of a data generation system according to an embodiment of the present invention. [Figure 2] 1 is a block diagram showing a schematic system configuration of a data generation system according to an embodiment of the present invention. [Figure 3] 10 is a flowchart showing an example of a flow of a method for generating an arm portion. [Figure 4]FIG. 2 is a diagram illustrating a configuration example of a hand unit. [Figure 5] 10 is a flowchart showing an example of the flow of a method for generating a hand unit. [Figure 6] FIG. 10 is a diagram showing a method for selecting a hand unit. [Figure 7] 10 is a flowchart showing an example of the flow of a robot generation method. [Figure 8] 10 is a flowchart showing an example of the flow of a data generation method according to the present embodiment. [Figure 9] 1 is a block diagram showing a schematic system configuration of a data generation system according to an embodiment of the present invention. [Figure 10] FIG. 1 is a diagram illustrating an example of the configuration of an object. DETAILED DESCRIPTION OF THE INVENTION

[0009] Embodiment 1 The present embodiment will be described below with reference to the drawings. The data generation system according to the present embodiment generates learning data for generating a trained model that is trained by machine learning the behavior of a robot.

[0010] The robot is configured as, for example, a multi-joint arm robot including an arm having a plurality of link sections and joint sections (elbow joints, wrist joints, etc.) connecting the link sections, and a hand section connected to the tip of the arm section. The hand section is configured as, for example, a gripping section for gripping an object, a magnetic section for magnetically attracting the object, a suction section for attracting the object, etc.

[0011] The training data generated by the data generation system is input to a learning device such as a neural network, for example. The learning device learns the input training data and generates the trained model.

[0012] 1 is a block diagram showing a schematic hardware configuration of a data generation system according to this embodiment. The data generation system 1 according to this embodiment has the hardware configuration of a typical computer, including a processor 11 such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), an internal memory 12 such as a RAM (Random Access Memory) or a ROM (Read Only Memory), a storage device 13 such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), an input / output I / F 14 for connecting peripheral devices such as a display, and a communication I / F 15 for communicating with devices external to the device.

[0013] However, as factory production becomes more flexible, the arms and hands (material handling) that make up robots are becoming more diverse. Generating training data to operate such diverse robots in a real environment is extremely time-consuming and difficult.

[0014] In contrast, the data generation system 1 according to this embodiment can easily and quickly generate a composite image with annotations (identification information) using a simulator that operates a robot in a virtual space, as will be described later. The user can use the generated composite image as learning data for robot operation learning.

[0015] 2 is a block diagram showing a schematic system configuration of a data generation system according to this embodiment. The data generation system 1 according to this embodiment includes an arm image acquisition unit 2, a hand image acquisition unit 3, and a learning data generation unit 4.

[0016] The arm image acquisition unit 2 is a specific example of arm image acquisition means. The arm image acquisition unit 2 acquires images of multiple types of arm sections and assigns label information for identifying each arm section to each acquired image of the arm section. The label information is a specific example of identification information.

[0017] For example, images of multiple types of arm parts may be set in advance in a memory unit such as a storage device. The arm image acquisition unit 2 can acquire images of multiple types of arm parts from the memory unit. Information on images of multiple types of arm parts may be input to the arm image acquisition unit 2 via an input device or the like.

[0018] The arm image acquisition unit 2 may generate images of multiple types of arm units by itself. This makes it possible to easily generate multiple types of arm units. Figure 3 is a flowchart showing an example of the flow of a method for generating an arm unit.

[0019] The arm image acquisition unit 2 determines whether the joint number (Joint_num) is 0 or not (S11).

[0020] When the arm image acquisition unit 2 determines that the joint number (Joint_num) is 0 (YES in S11), it reads a preset base part (S12) and places the read base part (S13). The arm image acquisition unit 2 increments the joint number (Joint_num) (S14).

[0021] The arm image acquiring unit 2 determines whether the joint number is equal to or greater than a predetermined value (S15). If the arm image acquiring unit 2 determines that the joint number is equal to or greater than the predetermined value (YES in S15), it ends this process.

[0022] On the other hand, when the arm image acquiring unit 2 determines that the joint number is not equal to or greater than the predetermined value (NO in S15), the process returns to the above (S11).

[0023] If the arm image acquisition unit 2 determines that the joint number (Joint_num) is not 0 (NO in S11), it reads the sth link part that has been set in advance (S16), and then reads the s-1th link part that has been set in advance (S17).

[0024] The arm image acquisition unit 2 places the s-1th link and places the sth link for the placed s-1th link (S18).The arm image acquisition unit 2 randomly selects a joint from a preset joint list (S19).

[0025] The arm image acquisition unit 2 places the selected joint between the s-1th link and the sth link (S20), and returns to the above process (S14).

[0026] Hand image acquisition unit 3 is a specific example of a hand image acquisition means. Hand image acquisition unit 3 acquires images of multiple types of hand parts and assigns label information for identifying each hand part to each acquired image of the hand part.

[0027] For example, images of the multiple types of hand parts may be set in advance in a storage unit such as a storage device, etc. The multiple types of hand parts include, for example, a gripping part, a magnetic part, and a suction part.

[0028] The hand image acquisition unit 3 can acquire images of multiple types of hand parts from a storage unit or the like. Information on images of multiple types of hand parts may be input to the hand image acquisition unit 3 via an input device or the like. Note that the hand image acquisition unit 3 may generate images of multiple types of hand parts itself. This makes it possible to easily generate images of multiple types of hand parts.

[0029] For example, as shown in Fig. 4, the hand unit may be configured as a gripping unit having a body unit (m_body), at least two nail units (m_nail) connected to the body unit for gripping an object, and a connecting unit (not shown) for connecting the body unit and the nail units. In Fig. 4, the left hand unit is configured with two nail units, and the right hand unit is configured with three nail units. The connecting unit may be, for example, a hinge mechanism, a fixed and hinge mechanism, a sliding mechanism, or a fixed and sliding mechanism.

[0030] The hand image acquisition unit 3 may generate a plurality of types of housing parts, claw parts, and connection parts, and may respectively assign label information to each generated housing part, each claw part, and each connection part.

[0031] FIG. 5 is a flowchart showing an example of a flow of a method for generating a hand part. The hand image acquisition unit 3 reads the size of a preset housing part (S21). The hand image acquisition unit 3 reads the size of a preset claw part and the number of claw parts (nail_num) (S22). The hand image acquisition unit 3 arranges the housing part and each claw part (S23), and arranges a connection part between the housing part and each claw part (S24).

[0032] The learning data generation unit 4 is a specific example of learning data generation means. The learning data generation unit 4 randomly selects, for example, one hand part from among the plurality of hand parts acquired by the hand image acquisition unit 3. As shown in FIG. 6, the learning data generation unit 4 may select one hand part (gripping part (n < 0.1), magnetic attachment part, (0.1 < n < 0.2) suction part (0.2 < n < 0.3), ···) based on a random number n.

[0033] The learning data generation unit 4 connects the selected hand part to the selected arm part via a joint part to generate a robot.

[0034] FIG. 7 is a flowchart showing an example of a flow of a method for generating a robot. The learning data generation unit 4 acquires the center position information, size information, posture information, etc. of the arm part (S31). The learning data generation unit 4 calculates the tip position of the arm part based on the acquired information (S32).

[0035] The learning data generation unit 4 calculates a simultaneous transformation matrix T (S33). The learning data generation unit 4 performs coordinate transformation of the hand part using the calculated simultaneous transformation matrix T, and moves the hand part to the tip of the arm part (S34). The learning data generation unit 4 arranges a joint part between the tip of the arm part and the hand part (S35).

[0036] The learning data generation unit 4 generates images of the robot generated as described above when it is operated. The learning data generation unit 4 operates the robot by, for example, randomly changing the joint angles of each joint of the robot within a predetermined range. The predetermined range may be preset as an initial value in the learning data generation unit 4 or the like.

[0037] The learning data generation unit 4 generates learning data by assigning label information of the selected arm unit and label information of the selected hand unit to the generated image. Note that, as described above, if the hand image acquisition unit 3 assigns label information to each housing unit, each claw unit, and each connection unit, the learning data generation unit 4 may assign label information of the selected arm unit and label information of the housing unit, claw unit, and connection unit of the configured gripping unit to the image when the robot is operated.

[0038] Next, a data generation method according to this embodiment will be described below. Fig. 8 is a flowchart showing an example of the flow of the data generation method according to this embodiment.

[0039] The arm image acquisition unit 2 acquires images of a plurality of types of arm portions, and assigns label information for identifying each arm portion to each of the acquired images of the arm portions (S41).

[0040] The hand image acquisition unit 3 acquires images of a plurality of types of hand parts, and assigns label information for identifying each hand part to each acquired image of the hand part (S42).

[0041] The learning data generation unit 4 randomly selects one arm part from the multiple arm parts acquired by the arm image acquisition unit 2 (S43). The learning data generation unit 4 randomly selects one hand part from the multiple hand parts acquired by the hand image acquisition unit 3 (S44).

[0042] The learning data generation unit 4 generates a robot in which the selected hand unit is connected to the selected arm unit (S45). The learning data generation unit 4 generates an image of the generated robot when it is in operation (S46). The learning data generation unit 4 generates learning data by adding label information of the selected arm unit and label information of the selected hand unit to the generated image (S47).

[0043] As described above, the data generation method according to this embodiment makes it possible to easily and quickly generate images of various robots with identification information. Then, by referring to the identification information, a user can use the synthesized images of the robots as learning data for learning the various movements of various robots.

[0044] Embodiment 2 The robot may perform an action on an object, such as grasping the object. Fig. 9 is a block diagram showing a schematic system configuration of a data generation system according to this embodiment. In addition to the above configuration, the data generation system 20 according to this embodiment further includes an object image acquisition unit 5.

[0045] The object image acquisition unit 5 is a specific example of an object image acquisition means. The object image acquisition unit 5 acquires images of a plurality of types of objects.

[0046] For example, images of multiple types of objects may be stored in advance in a storage unit such as a storage device, etc. The multiple types of objects include, for example, round, square, spherical, cubic, rectangular parallelepiped objects, parts, etc.

[0047] The object image acquisition unit 5 can acquire images of multiple types of objects from a storage unit or the like. Information on images of multiple types of objects may also be input to the object image acquisition unit 5 via an input device or the like. The object image acquisition unit 5 may also generate images of multiple types of objects by itself. The object image acquisition unit 5 assigns identification information for identifying each object to each acquired image of the object.

[0048] The learning data generation unit 4 selects, for example, one object randomly from among the plurality of objects acquired by the object image acquisition unit 5. As shown in FIG. 10, the learning data generation unit 4 may select one object (round object (n < 0.1), square object (0.1 < n < 0.2), spherical object (0.2 < n < 0.3), ···) based on the random number n.

[0049] The learning data generation unit 4 operates the selected robot with the selected hand part connected to the selected arm part on the selected object. The learning data generation unit 4 generates an image when the robot operates. The learning data generation unit 4 generates learning data by attaching the identification information of the selected arm part, the identification information of the selected hand part, and the identification information of the object to the generated image.

[0050] According to this embodiment, it is possible to generate the learning data of the robot in consideration of the object on which the robot operates.

[0051] Also, in this embodiment, the learning data generation unit 4 may operate the robot with the selected hand part connected to the selected arm part as described above, and generate images when the robot is photographed (overlooked) from a plurality of different positions.

[0052] The learning data generation unit 4 may generate learning data by attaching the identification information of the photographed position, the identification information of the selected arm part, and the identification information of the selected hand part to each of the generated images. Thereby, it is possible to generate the learning data of the robot in consideration of the photographing position (overlooking position) of the robot.

[0053] Although several embodiments of the present disclosure have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims.

[0054] The present disclosure can also be implemented by causing a processor to execute a computer program to perform the processes shown in, for example, FIG.

[0055] The program can be stored and supplied to a computer using various types of non-transitory computer readable media. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)).

[0056] The program may be provided to the computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can provide the program to the computer via a wired communication path such as an electrical wire or optical fiber, or via a wireless communication path.

[0057] Each component of the data generation system 1, 20 according to the above-described embodiment can be realized not only by a program, but also in part or in whole by dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). [Explanation of symbols]

[0058] 1 Data generation system, 2 Arm image acquisition unit, 3 Hand image acquisition unit, 4 Learning data generation unit, 11 Processor, 12 Internal memory, 13 Storage device, 14 Input / output I / F, 15 Communication I / F, 20 Data generation system

Claims

1. A data generation system that generates learning data for generating a trained model that has learned the behavior of a robot including an arm unit and a hand unit connected to a tip of the arm unit, an arm image acquisition means for acquiring images of a plurality of types of the arm portions and assigning identification information for identifying each of the acquired images of the arm portions; a hand image acquisition means for acquiring images of a plurality of types of hand portions and assigning identification information for identifying each of the acquired hand portion images; a learning data generation means for selecting one arm from the plurality of arm parts acquired by the arm image acquisition means, selecting one hand from the plurality of hand parts acquired by the hand image acquisition means, generating an image of a robot having the selected hand part connected to the selected arm part when the robot is operated, and assigning identification information of the selected arm part and identification information of the selected hand part to the generated image, respectively, thereby generating the learning data; Equipped with Data generation system.

2. 2. The data generation system of claim 1, the learning data generation means operates a robot having the selected hand unit connected to the selected arm unit, generates images of the robot photographed from a plurality of different positions, and generates the learning data by assigning identification information of the position, identification information of the selected arm unit, and identification information of the selected hand unit to each of the generated images. Data generation system.

3. 2. The data generation system of claim 1, the robot performs an action on an object, further comprising an object image acquisition means for acquiring images of a plurality of types of objects and assigning identification information for identifying each of the acquired object images, the learning data generation means selects one object from the plurality of objects acquired by the object image acquisition means, generates an image of a robot having the selected hand unit connected to the selected arm unit operating on the selected object, and generates the learning data by assigning identification information of the selected arm unit, identification information of the selected hand unit, and identification information of the object to the generated image; Data generation system.

4. 2. The data generation system of claim 1, the hand unit is configured as a gripping unit having a housing unit, at least two claw units connected to the housing unit and configured to grip an object, and a connecting unit connecting the housing unit and the claw units, the hand image acquisition means generates a plurality of types of the housing parts, the claw parts, and the connection parts, and assigns identification information to each of the generated housing parts, each claw part, and each connection part; the learning data generation means selects a housing part, a claw part, and a connection part from the housing part, the claw part, and the connection part generated by the hand image acquisition means, respectively, to configure the gripping part, generates an image of a robot in which the configured gripping part is connected to the selected arm part and is operated, and generates the learning data by assigning identification information of the selected arm part and identification information of the housing part, the claw part, and the connection part of the configured gripping part to the generated image. Data generation system.

5. 2. The data generation system of claim 1, the plurality of types of hand units include at least one of a gripping unit that grips an object, a magnetic attracting unit that magnetically attracts an object, and a suction unit that attracts an object; Data generation system.

6. A data generation method for generating training data for generating a trained model that has learned the behavior of a robot including an arm unit and a hand unit connected to a tip of the arm unit, acquiring images of a plurality of types of the arm section, and assigning identification information for identifying each of the acquired images of the arm section; acquiring images of a plurality of types of hand units, and assigning identification information for identifying each of the acquired hand unit images; a step of generating the learning data by selecting one arm unit from the acquired plurality of arm units, selecting one hand unit from the acquired plurality of hand units, generating an image of a robot connected to the selected arm unit and operated, and assigning identification information of the selected arm unit and identification information of the selected hand unit to the generated image, respectively; Including, Data generation method.

7. A program for generating learning data for generating a trained model that has learned the behavior of a robot including an arm unit and a hand unit connected to a tip of the arm unit, a process of acquiring images of a plurality of types of arm units and assigning identification information for identifying each arm unit to each of the acquired images of the arm units; A process of acquiring images of a plurality of types of hand units and assigning identification information for identifying each of the acquired hand unit images; a process of generating the learning data by selecting one arm from the acquired plurality of arm sections, selecting one hand from the acquired plurality of hand sections, generating an image of a robot connected to the selected arm section and operating the robot, and assigning identification information of the selected arm section and identification information of the selected hand section to the generated image; to the computer, program.

Images