Training data generating device and training data generating method, and machine learning device and machine learning method using training data

Abstract

This device for generating learning data is provided with a measuring device, such as a visual sensor, that measures an arrangement region of a plurality of workpieces and acquires an image. The device for generating learning data is provided with a learning data generation unit that generates learning data including an image acquired by the visual sensor and a workpiece removal position. The device for generating learning data generates a plurality of sets of learning data by repeating: control of the movement of the workpieces using a movement device so as to change the arrangement pattern of the workpieces; measurement of the arrangement region of the plurality of workpieces by the visual sensor; and generation of learning data by the learning data generation unit.

Description

[Technical field]

[0001] The present invention relates to a training data generating device and a training data generating method, and a machine learning device and a machine learning method that use training data. [Background technology]

[0002] Conventionally, a technology for transporting a workpiece using a robot device equipped with an articulated robot has been known. The robot can transport the workpiece to a target position and posture by changing the position and posture while holding the workpiece. To achieve this, the robot device needs to change the position and posture of the robot according to the position and posture of the workpiece.

[0003] It is known that in order for a robot device to hold a workpiece, a visual sensor captures an image of the workpiece placed at a predetermined position and controls the robot based on the information obtained from the visual sensor. For example, it is known that a robot control device calculates the position and orientation of the workpiece from the image obtained by the visual sensor and controls the position and orientation of the robot according to the position and orientation of the workpiece.

[0004] In recent years, it has been known that machine learning is used to control machines in various fields (for example, JP 2019-34836 A). Known examples of machine learning include reinforcement learning, supervised learning, and unsupervised learning.

[0005] In a robot device that transports a workpiece, it is known that machine learning is performed to estimate the position and orientation of the workpiece that the robot picks up from an image of the workpiece captured by a visual sensor. For example, machine learning is performed using teacher data to generate a learning model. Then, a technology is known that uses the generated learning model to calculate the position and orientation of the workpiece that the robot picks up from the image captured by the visual sensor (for example, JP 2019-56966 A and JP 2018-161692 A). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2019-34836 A [Patent Document 2] JP 2019-56966 A [Patent Document 3] JP 2018-161692 A Summary of the Invention [Problem to be solved by the invention]

[0007] In order to transport the workpieces, the robot device can capture an image of an area where multiple workpieces are placed using a visual sensor and calculate the pick-up position for picking up one workpiece. In order to perform machine learning such as supervised learning, it is necessary to generate data including images of the workpieces and the pick-up positions of each workpiece as learning data.

[0008] In machine learning, a learning model that can accurately estimate the pick-up position and orientation of a workpiece can be generated by using a large amount of learning data. For example, there is a task of picking up multiple workpieces that are randomly stacked. Or, there is a task of picking up aligned workpieces. In either task, it is preferable to perform machine learning using a large amount of learning data.

[0009] However, there is a problem that generating a large amount of learning data requires a large amount of work and time. In particular, when the workpiece is large or heavy, a large amount of effort is required for the worker to change the workpiece arrangement pattern. In other words, there is a problem that it is difficult to obtain a large amount of learning data while changing the workpiece arrangement pattern.

[0010] For example, learning data may be obtained by changing the packaging (layout pattern) of a large workpiece that is difficult for one worker to carry. In order to collect learning data, multiple workers must move the workpiece. This makes it inefficient to generate learning data. Furthermore, if the workpiece is large, it may be difficult for multiple workers to transport the workpiece.

[0011] On the other hand, learning data can be generated by the simulation device. For example, a virtual three-dimensional space can be generated by the simulation device using a three-dimensional CAD model created by three-dimensional CAD (Computer Aided Design) software or the like. In the virtual three-dimensional space, a layout pattern of a plurality of workpieces is generated using the three-dimensional CAD model of the workpiece. The simulation device can generate an image captured by a virtual visual sensor from a predetermined position in the virtual three-dimensional space. The image and the position of the workpiece at this time can be used as learning data.

[0012] However, there is a problem that the image generated by the simulation device is different from the image obtained when performing actual work in the real world. For example, when actually performing the work of picking up a workpiece, the shadow of an object reflected on the workpiece or the glossiness of the surface of the workpiece may change depending on the surrounding lighting conditions. Or, halation may occur in the image. Furthermore, in the real world, there are variations in the presence or absence of dirt on the workpiece, its position and size, the presence or absence of scratches, its position and size, or the position of an object (e.g., a label or tape) actually attached to the workpiece. The image generated by the simulation device is an ideal image that does not reflect these conditions. In addition, the learning data does not reflect the size error or manufacturing variation when the workpiece is manufactured. Furthermore, the workpiece may be deformed when it is transported. When learning data is generated by the simulation device, there is a problem that it is not possible to generate realistic learning data that reflects the deformation or manufacturing variation of the workpiece and corresponds to the real state.

[0013] In this way, when an ideal image different from an image acquired in an actual work is used as learning data to generate a learning model, it may not be possible to accurately estimate the pick-up position and posture of the workpiece handled in the actual work. As a result, there is a high possibility that the robot device will not be able to hold the workpiece in the actual work. In other words, there is a problem that the success rate of the robot device in transporting the workpiece decreases. [Means for solving the problem]

[0014] A first aspect of the present disclosure is a device for generating learning data used in machine learning. The device for generating learning data includes a measuring device that measures an arrangement area of ​​a plurality of workpieces to obtain at least one of a two-dimensional image and a three-dimensional image. The device for generating learning data includes a moving device that moves at least one workpiece, and a control unit that controls the operation of the moving device. The device for generating learning data includes a learning data generating unit that generates learning data including an image acquired by the measuring device and workpiece removal position information for removing the workpiece. The device for generating learning data generates a plurality of learning data by repeating control of moving the workpieces with the moving device so as to change the arrangement pattern of the workpieces, measurement of the arrangement area of ​​the plurality of workpieces with the measuring device, and generation of learning data with the learning data generating unit.

[0015] A second aspect of the present disclosure is a machine learning device including the above-mentioned learning data generation device. The machine learning device includes a learning unit that performs machine learning based on the learning data generated by the learning data generation unit and generates a learning model that estimates a removal position of the workpiece from an image of the workpiece placement area. The machine learning device includes an inference unit that estimates the removal position of the workpiece from an image acquired by a measuring instrument based on the learning model generated by the learning unit.

[0016] A third aspect of the present disclosure is a method for generating learning data used in machine learning. The method for generating learning data includes a measurement step in which a measuring device measures an arrangement area of ​​a plurality of workpieces to obtain at least one of a two-dimensional image and a three-dimensional image. The method for generating learning data includes a movement step in which a moving device moves at least one workpiece to change the arrangement pattern of the workpieces, and a learning data generation step in which learning data including the image obtained in the measurement step and workpiece removal position information for removing the workpieces is generated. The method for generating learning data generates a plurality of learning data by repeating the movement step, the measurement step, and the learning data generation step.

[0017] A fourth aspect of the present disclosure is a machine learning method including the above-mentioned learning data generation method. The machine learning method includes a learning step of performing machine learning based on the learning data generated in the learning data generation step and generating a learning model that estimates a removal position of the workpiece from an image of the workpiece placement area. The machine learning method includes an inference step of estimating a removal position of the workpiece from an image acquired by a measuring device based on the learning model generated in the learning step. Effect of the Invention

[0018] According to aspects of the present disclosure, it is possible to provide a training data generation device and a training data generation method that can generate training data used in machine learning efficiently and realistically in response to real-world conditions. It is also possible to provide a machine learning device including the training data generation device and a machine learning method including the training data generation method. [Brief description of the drawings]

[0019] [Figure 1] FIG. 2 is a perspective view of a first training data generating device according to an embodiment. [Diagram 2] FIG. 2 is a block diagram of a device for generating first training data in the embodiment. [Diagram 3] 10 is a flowchart showing control of the first learning data generating device. [Figure 4]FIG. 2 is a perspective view of the first learning data generating device when the workpiece arrangement pattern is being changed. [Diagram 5] FIG. 1 is a perspective view of a first robot system that transports a workpiece using a learning model generated by a learning data generation device. [Figure 6] FIG. 2 is a block diagram of a first robot system. [Figure 7] FIG. 11 is a perspective view of a second robot system that transports a workpiece using a learning model generated by a learning data generation device. [Figure 8] 13 is a flowchart of control of a third training data generation device in an embodiment. [Figure 9] FIG. 13 is a perspective view of a fourth training data generating device in the embodiment. [Figure 10] FIG. 13 is a block diagram of a fourth training data generating device. [Figure 11] 13 is a flowchart of control of a fourth learning data generation device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] A learning data generating device, a learning data generating method, a machine learning device, and a machine learning method according to the embodiment will be described with reference to Figures 1 to 11. In the robot system according to the embodiment, a robot device transports a plurality of workpieces placed on a floor or a pallet to a predetermined target position.

[0021] The machine learning device of this embodiment generates a learning model that calculates a pick-up position and posture for picking up a workpiece based on an image of an arrangement area of ​​multiple workpieces. The machine learning device of this embodiment performs supervised learning. The robot system calculates the pick-up position and posture of the workpiece based on an image acquired by a measuring instrument such as a visual sensor and a learning model generated by machine learning performed in advance. Then, the robot system controls the position and posture of the robot according to the pick-up position and posture of the workpiece calculated by the machine learning device. The learning data generation device of this embodiment generates learning data as teacher data for generating a learning model.

[0022] (First learning data generating device) FIG. 1 shows a perspective view of a plurality of carrier vehicles, which are moving devices, a plurality of workpieces, and a visual sensor, which is a measuring instrument, constituting a first learning data generating device in this embodiment. FIG. 2 shows a block diagram of the first learning data generating device in this embodiment. In this embodiment, an example using a rectangular parallelepiped workpiece is described. For example, a cardboard box corresponds to such a workpiece. However, the workpiece is not limited to this form, and a workpiece of any shape can be adopted.

[0023] The first learning data generation device 1 includes a visual sensor 32 as a measuring instrument that measures an arrangement area in which a plurality of workpieces 91 are arranged and acquires at least one of a two-dimensional image and a three-dimensional image. The visual sensor 32 captures images of the plurality of workpieces 91 and the background of the workpieces 91. In the first learning data generation device 1, a two-dimensional camera that captures a two-dimensional image is arranged as the visual sensor 32. In this embodiment, the visual sensor 32 is arranged above the area in which the plurality of workpieces 91 are arranged, but is not limited to this form. The visual sensor 32 may be arranged diagonally above the area in which the plurality of workpieces 91 are arranged.

[0024] The plurality of workpieces 91 are disposed within the visual field 32a of the visual sensor 32. The visual sensor 32 is fixed and supported by a support member 92. The support member 92 is fixed to a stand (not shown) that is fixed to the environment, but is not limited to this form. The support member 92 may be fixed to a moving mechanism (not shown) that can move relative to the stand fixed to the environment. The moving mechanism may include a mechanism driven by a motor or one robot device. The visual sensor 32 in this embodiment is configured to capture an image of the upper surface 91a of the workpiece 91, but is not limited to this form. The visual sensor 32 may be configured to capture an image of the upper surface 91a and the side surface 91b of the workpiece 91 while moving in accordance with the operation of the moving mechanism described above. The visual sensor 32 also captures an image of the background of the plurality of workpieces 91.

[0025] The learning data generation device 1 includes a transport system including one or more transport vehicles 31 as a moving device for moving at least one workpiece. The transport vehicle 31 travels on a floor surface with one or more workpieces 91 on board. The transport vehicle 31 may be configured to be able to raise and lower the workpiece while traveling, or to tilt the workpiece. The transport vehicle 31 in this embodiment is an automated guided vehicle (AGV) that travels autonomously. The transport system in this embodiment includes a plurality of transport vehicles 31.

[0026] The learning data generation device 1 includes a calculation processing device 10 that controls a plurality of carrier vehicles 31 and a visual sensor 32 to generate learning data. The calculation processing device 10 is configured as a digital computer having a CPU (Central Processing Unit) as a processor. The calculation processing device 10 includes a RAM (Random Access Memory) and a ROM (Read Only Memory) connected to the CPU via a bus.

[0027] The arithmetic processing device 10 includes a storage unit 12 that stores any information related to the generation of learning data. The storage unit 12 stores data such as images acquired by the visual sensor 32. The storage unit 12 can be configured with a non-transitory storage medium capable of storing information. The storage unit 12 can be configured with a storage medium such as a volatile memory, a non-volatile memory, a magnetic storage medium, or an optical storage medium.

[0028] For example, the storage unit can be configured with a hard disk drive (HDD) or a solid state drive (SSD) arranged in the edge device or the cloud. Alternatively, the storage unit may include a storage medium such as a universal serial bus (USB) memory connected to the edge device. The storage unit may be arranged in another arithmetic processing device, a server, or a cloud connected to the arithmetic processing device via an electric communication line.

[0029] The arithmetic processing device 10 includes a reception unit 11 that receives information from the outside. The reception unit 11 receives predetermined learning data generation conditions 25 and a predetermined learning data generation program 26, for example, by an operator operating an input device such as a keyboard and a mouse. The input device may include a touch panel type display panel. Alternatively, the reception unit 11 may receive information from the outside (for example, a server) in the form of a file.

[0030] The arithmetic processing device 10 includes a processing unit 13 that performs pre-processing to generate an arrangement pattern before capturing an image with the visual sensor 32. The processing unit 13 includes an arrangement pattern generation unit 13a that generates an arrangement pattern of a plurality of workpieces 91. The arrangement pattern generation unit 13a generates the arrangement pattern of the workpieces based on the generation conditions 25 of the learning data.

[0031] The arithmetic processing device 10 includes a control unit 14 that controls the operation of the multiple transport vehicles 31. The control unit 14 includes a motion plan generating unit 14a that generates a motion plan for the multiple transport vehicles 31 using the position and posture of each workpiece in each of the multiple arrangement patterns of the multiple workpieces generated by the arrangement pattern generating unit 13a as target values. The control unit 14 includes a motion command generating unit 14b that generates multiple motion commands to operate the multiple transport vehicles 31 based on the motion plans for the multiple transport vehicles 31 generated by the motion plan generating unit 14a. The multiple motion commands generated by the motion command generating unit 14b are transmitted to the multiple transport vehicles 31 by wireless communication. Each transport vehicle 31 moves the workpiece 91 according to the motion command.

[0032] The control unit 14 includes an imaging command generating unit 14c that transmits an imaging start command to the visual sensor 32. The imaging command generating unit 14c transmits the imaging start command to the visual sensor 32 after each transport vehicle 31 is placed at each target position defined in the operation plan. Based on the imaging start command, the visual sensor 32 captures an image of the arrangement area where the multiple workpieces 91 placed on the multiple transport vehicles are arranged.

[0033] The arithmetic processing device 10 includes a learning data generating unit 15 that generates learning data including an image acquired by the visual sensor 32 and removal position information of the workpiece for removing the workpiece. The learning data generating unit 15 includes a data acquiring unit 15a that acquires data for generating the learning data. The data acquiring unit 15a acquires an image captured by the visual sensor 32. The data acquiring unit 15a acquires removal position information of the workpiece held by the hand in order to perform the workpiece transport operation by the robot device. The data acquiring unit 15a can calculate the removal position of each of the multiple workpieces 91 based on the arrangement pattern of the multiple workpieces 91 generated by the arrangement pattern generating unit 13a. The data acquiring unit 15a can calculate removal position information for each workpiece 91 as the position information of the workpiece.

[0034] The learning data generating unit 15 of the arithmetic processing device 10 includes a judging unit 15b that judges whether to store or discard the learning data based on a predetermined criterion for judging whether the learning data is acceptable. The judging unit 15b judges whether the learning data, which includes the pick-up position and posture of the workpiece, the outer shape of the workpiece, and an image of the workpiece, is appropriate for learning. Examples of the judging criteria include whether the image is out of focus, too dark, too bright, whether there are many overlapping images, or whether there are many images that do not include the workpiece. If the learning data meets the judging criteria, the learning data is stored in the storage unit 12. If the learning data does not meet the judging criteria, the judging unit 15b discards the learning data.

[0035] The arithmetic processing device 10 includes an image processing unit 16 that performs image processing of an image captured by the visual sensor 32. For example, when acquiring a two-dimensional image from the visual sensor 32, the image processing unit 16 estimates the pick-up position and posture of the workpiece in the two-dimensional image by performing pattern matching of the two-dimensional image.

[0036] The above-mentioned reception unit 11, processing unit 13, and arrangement pattern generation unit 13a included in the processing unit 13 correspond to a processor driven in accordance with the learning data generation program 26. The control unit 14, and the motion plan generation unit 14a, motion command generation unit 14b, and imaging command generation unit 14c included in the control unit 14 correspond to a processor driven in accordance with the learning data generation program 26. The image processing unit 16, learning data generation unit 15, and the data acquisition unit 15a and determination unit 15b included in the learning data generation unit 15 correspond to a processor driven in accordance with the learning data generation program 26. The processors function as the respective units by performing processing defined in the learning data generation program 26.

[0037] FIG. 3 shows a flow chart of the control of the learning data generation method in this embodiment. With reference to FIG. 1 to FIG. 3, in step 101, the reception unit 11 acquires the learning data generation conditions 25. The learning data generation conditions 25 include, for example, at least one of a target value of the number of learning data, a range of the number of types of workpieces, a range of workpiece sizes, and a condition of the arrangement pattern of the workpieces. The condition of the arrangement pattern of the workpieces may include at least one of a range of the number of layers in which a plurality of workpieces are stacked, and a condition of the gap between the workpieces. The learning data generation conditions 25 may also include information on the outer shape, size, and number of the actual workpieces used in generating the learning data.

[0038] The worker can specify the desired learning data generation conditions 25 via the input device. For example, the worker can specify the target value for the number of learning data as 100 sets. The worker can set the range of the number of work types to 1 type or more and 15 types or less. The worker can set the width, height, and depth as the range of the work size to 150 mm or more and 800 mm or less.

[0039] The worker can specify the conditions for the arrangement pattern of the workpieces so that the workpieces overlap in a range of 1 to 5 layers. The gap between the workpieces can be set to, for example, 5 mm to 10 mm. The learning data generation conditions 25 accepted by the accepting unit 11 are stored in the memory unit 12.

[0040] Next, in step 102, the arrangement pattern generating unit 13a of the processing unit 13 generates a plurality of arrangement patterns of a plurality of workpieces. The arrangement pattern generating unit 13a in this embodiment is configured to execute a three-dimensional simulation. The arrangement pattern generating unit 13a generates a plurality of arrangement patterns of a plurality of workpieces by a three-dimensional simulation based on the learning data generation conditions 25. For example, one arrangement pattern of a plurality of workpieces 91 in three rows and four columns as shown in FIG. 1 is generated. Note that the arrangement pattern generating unit is not limited to an arrangement pattern in which the workpieces are stacked in one layer as shown in FIG. 1, and may generate an arrangement pattern of two or more layers.

[0041] The arrangement pattern generating unit 13a can be configured, for example, by a three-dimensional simulation device using a three-dimensional CAD model generated by three-dimensional CAD software. The arrangement pattern generating unit 13a generates an arrangement pattern based on the learning data generation condition 25 by using a three-dimensional CAD model of the work in a virtual three-dimensional environment generated by the simulation device. The arrangement pattern generating unit 13a can generate a virtual three-dimensional space so as to correspond to an area where the work is imaged as shown in Fig. 1. Then, it generates a plurality of arrangement patterns in which a plurality of workpieces are arranged inside an area where the workpiece can be imaged.

[0042] The arrangement pattern generating unit 13a generates, for example, CAD models of 60 workpieces, for example, using 3D CAD software, with the width, height, and height of the workpieces being 150 mm or more and 800 mm or less, respectively, within a range of 15 different sizes. These 60 CAD models of workpieces are used to generate a plurality of arrangement patterns, and for example, the CAD models of the 60 workpieces can be generated so that they are stacked in a range of 1 layer or more and 5 layers or less. In addition, an arrangement pattern can be generated so that the gap between the CAD models of the workpieces is within a range of 5 mm or more and 10 mm or less. The arrangement pattern generating unit 13a may generate, for example, 100 sets of arrangement patterns.

[0043] In this way, the arrangement pattern generation unit 13a can generate any arrangement pattern that satisfies the learning data generation condition 25. The arrangement pattern generation unit 13a can generate an irregular arrangement pattern of workpieces such that the CAD models of all workpieces are arranged within the field of view 32a of the virtual visual sensor 32 in the virtual three-dimensional simulation environment.

[0044] In addition, the arrangement pattern generating unit 13a can calculate the three-dimensional position and orientation of each workpiece in each arrangement pattern and output it as a processing result. For example, in a virtual three-dimensional space generated by the simulation device, a model of a virtual visual sensor is generated according to the position and orientation of the actual visual sensor. In each arrangement pattern, the arrangement pattern generating unit 13a calculates the three-dimensional relative position and relative orientation of each workpiece with respect to the virtual visual sensor. That is, the arrangement pattern generating unit 13a can calculate the three-dimensional position and orientation of each workpiece in the sensor coordinate system of the virtual visual sensor. For example, the position of the workpiece 91 can be determined as the center of gravity of the upper surface 91a of the workpiece 91. The orientation of the upper surface 91a of the workpiece 91 can be calculated as the orientation of the workpiece 91. The storage unit 12 can store the processing result of the arrangement pattern generating unit 13a.

[0045] The arrangement pattern generating unit 13a can generate an arrangement pattern of one layer, or an arrangement pattern of two or more layers in which a plurality of workpieces overlap. The arrangement pattern generating unit 13a may generate an arrangement pattern in which no workpieces are arranged within the visual field (image capture area) of the visual sensor. In other words, the arrangement pattern generating unit 13a may generate an arrangement pattern in which only the background of a transport vehicle for transporting the workpieces, or a pallet or tray, is captured.

[0046] 3, in step 103, the motion plan generating unit 14a of the control unit 14 formulates motion plans for the multiple guided vehicles 31. The motion plan generating unit 14a generates motion plans for the multiple guided vehicles 31 based on the processing result of the placement pattern generating unit 13a. Then, the motion command generating unit 14b generates multiple motion commands based on the motion plans thus generated.

[0047] For example, the arrangement pattern generating unit 13a generates 100 sets of different arrangement patterns of a plurality of workpieces. The arrangement pattern generating unit 13a calculates and outputs information on the three-dimensional relative position and relative orientation of each workpiece with respect to the virtual visual sensor for each arrangement pattern. With reference to FIG. 1, calibration is performed in advance between the visual sensor 32 that actually captures images in the real world and the origin position from which each transport vehicle 31 starts moving. For example, the position of the charging station of each transport vehicle 31 can be used as the origin position of the transport vehicle 31.

[0048] The motion plan generating unit 14a converts the relative position and orientation of each workpiece 91 with respect to the visual sensor 32 output from the arrangement pattern generating unit 13a into a three-dimensional position and orientation of each workpiece 91 with respect to the origin position of the transport vehicle 31. That is, the coordinate values ​​in the sensor coordinate system of the visual sensor 32 can be converted into coordinate values ​​in a coordinate system with the origin position of the transport vehicle 31 as the origin. Then, the motion plan generating unit 14a generates a motion command for the operation of the transport vehicle 31 carrying the workpiece 91 so that each workpiece 91 actually reaches the target position and orientation generated by the arrangement pattern generating unit 13a, using the three-dimensional position and orientation of each workpiece 91 with respect to the origin position of the transport vehicle 31 as target values. The motion plan generating unit 14a performs a motion plan for each transport vehicle 31 so that each workpiece 91 reaches the target position and orientation generated by the arrangement pattern generating unit 13a.

[0049] The motion plan generating unit 14a can plan the motion of each transport vehicle 31 by the above-mentioned method for one arrangement pattern of a plurality of workpieces generated by the arrangement pattern generating unit 13a. The motion plan generating unit 14a generates a motion plan for moving the plurality of transport vehicles 31 so as not to collide with each other when each transport vehicle 31 moves toward a target position and a target posture with the workpieces 91 loaded thereon. For example, the motion plan generating unit 14a can generate a plan for moving the transport vehicles 31 one by one to a target position and a target posture. For example, the motion plan generating unit 14a can generate a plan for moving the transport vehicles 31 carrying the workpieces one by one so that the workpieces are arranged in order from the end of the arrangement pattern when the arrangement pattern of the workpieces is viewed in a plan view from a visual sensor.

[0050] Alternatively, the motion plan generating unit 14a simultaneously operates the multiple guided vehicles 31 and calculates the distance between each of the guided vehicles 31. The motion plan generating unit 14a may generate motion plans for the multiple guided vehicles 31 so that the distance is greater than 0. The motion command generating unit 14b of the control unit 14 generates motion commands for each guided vehicle 31 based on the motion plans for each guided vehicle 31 thus generated.

[0051] In step 104, the transport vehicle 31 moves according to the operation command to transport the workpiece 91. Each workpiece 91 is transported by each transport vehicle 31 so as to be in a predetermined position and posture in the arrangement pattern generated by the arrangement pattern generation unit 13a. There are various methods for controlling the transport vehicle 31 so that it departs from a predetermined origin position (e.g., a charging station) with the workpiece 91 loaded thereon and reaches a target position and target posture. As a method for controlling the transport vehicle based on the operation plan of the transport vehicle, for example, a marker is attached to the transport vehicle, a visual sensor capable of capturing an image of the space in which the transport vehicle moves is disposed, and the position and posture of the marker on the image captured by the visual sensor at a certain time interval is detected by image processing of an image processing unit described later, and the position and posture of the transport vehicle can be calculated in real time.

[0052] For example, the image processing unit performs image processing of images captured by a visual sensor. The position and attitude of the marker, i.e., the current position and attitude of the transport vehicle, are detected from images captured at regular time intervals. The motion command generating unit calculates the differences between the current position and attitude of the transport vehicle and the target position and target attitude, respectively, so that the transport vehicle reaches the three-dimensional target position and target attitude. Then, based on these differences, a target speed vector along which the transport vehicle should move can be calculated, and a speed command along which the transport vehicle should move can be calculated as a motion command.

[0053] Alternatively, an acceleration sensor may be arranged on the transport vehicle, and the current acceleration measured by the acceleration sensor may be integrated twice to calculate the current position. Alternatively, a GPS (Global Positioning System) receiver may be arranged on the transport vehicle to measure the current position of the transport vehicle. The motion command generating unit may constantly correct the speed command of the transport vehicle based on the real-time measurement results of the GPS device, and transmit the corrected speed command to the transport vehicle. Also, the difference between multiple two-dimensional images captured by a visual sensor at regular time intervals may be compared, and a stop command may be transmitted to the transport vehicle 31 after the four corners of the workpiece 91 are completely captured in the two-dimensional image.

[0054] Next, in step 105, the imaging command generating unit 14c of the control unit 14 transmits an imaging start command to the visual sensor 32 to capture an image. The visual sensor 32 captures an image after one arrangement pattern generated by the arrangement pattern generating unit 13a is completed. Here, a two-dimensional image is captured. The two-dimensional image can be image data composed of one image including an image of the upper surfaces 91a of the multiple workpieces 91. In addition, when a moving mechanism for moving the visual sensor 32 is provided, the control unit 14 can control the position and attitude of the visual sensor 32 by controlling the operation of the moving mechanism. In this case, the two-dimensional image can be image data composed of one or more images including the upper surfaces 91a and side surfaces 91b of the multiple workpieces 91.

[0055] Next, in step 106, the data acquisition unit 15a of the learning data generation unit 15 acquires a two-dimensional image captured by the visual sensor 32. The data acquisition unit 15a of the learning data generation unit 15 also acquires the position and orientation of each workpiece from the arrangement pattern generation unit 13a. The data acquisition unit 15a acquires, for example, the position and orientation of each workpiece 91 in the sensor coordinate system of the visual sensor 32. Then, the data acquisition unit 15a acquires the size of the workpiece and the outer shape of the workpiece from the arrangement pattern generation unit 13a. Then, the data acquisition unit 15a calculates the removal position and orientation of the workpiece for the robot device to perform the work of removing the workpiece. Next, the learning data generation unit 15 generates learning data including at least one of the removal position and orientation and the outer shape of the workpiece in addition to the image data. For example, learning data including image data and removal position information may be generated, learning data including image data and removal position and orientation information may be generated, or learning data including image data, removal position and orientation, and outer shape information of the workpiece may be generated.

[0056] Next, in step 107, the determination unit 15b of the learning data generation unit 15 determines whether or not one piece of generated learning data satisfies a criterion for determining whether the learning data is acceptable. The determination unit 15b can determine whether or not the learning data including the image captured by the visual sensor 32 is appropriate for machine learning, based on the result of image processing by the image processing unit 16. These criteria for determining whether or not the learning data is acceptable can be included in the generation conditions 25 of the learning data.

[0057] For example, image processing unit 16 performs image processing on candidate images of learning data to detect workpieces in the images. Judgment unit 15b calculates the number of images in which no workpieces are detected among the images of learning data generated up to now. In other words, it calculates the number of images in which no workpieces are shown. If the number of images exceeds a predetermined threshold (e.g., 1), judgment unit 15b may judge the learning data including the images and information such as the removal positions corresponding to the images to be unsatisfactory learning data.

[0058] Alternatively, the image processing unit 16 calculates the difference between the currently captured two-dimensional image of the candidate and a previously captured two-dimensional image, and outputs the calculation result to the storage unit 12. The determination unit 15b can determine that a plurality of images with small differences are duplicated images and are unacceptable learning data. The determination unit 15b may then leave one set of learning data and delete the other duplicated learning data.

[0059] Furthermore, the image processing unit 16 can calculate the brightness or out-of-focus of a two-dimensional image that is a candidate for learning data, for example, by performing a fast Fourier transform (FFT). The judgment unit 15b acquires the result of the image processing. The judgment unit 15b selects an image that is too dark, too bright, or out-of-focus based on a predetermined judgment threshold. The judgment unit 15b may judge learning data that includes these images as unacceptable learning data.

[0060] In step 107, if the generated learning data satisfies the criteria for judging whether the learning data is acceptable, control proceeds to step 109. In step 109, the memory unit 12 stores the learning data. In step 107, if the generated learning data does not satisfy the criteria for judging whether the learning data is acceptable (if the learning data is unacceptable), control proceeds to step 108. In step 108, the judgment unit 15b discards the learning data. Note that after all the learning data generated by the learning data generation unit is stored in the memory unit 12, the judgment unit 15b may discard the learning data that does not satisfy the criteria for judging the learning data.

[0061] Next, in step 110, the judgment unit 15b judges whether or not the number of pieces of successful learning data stored in the storage unit 12 has reached a predetermined target value. For example, the judgment unit 15b judges whether or not 100 sets of learning data, which is the target value, have been generated. In step 110, if the number of pieces of learning data has reached the target value, the judgment unit 15b judges that the number of pieces of learning data is successful. In other words, it can be judged that the learning data required for machine learning has been generated. Then, this control ends.

[0062] On the other hand, in step 110, if the number of pieces of learning data is less than the target value, control returns to step 102. In step 102, the arrangement pattern generation unit 13a generates another arrangement pattern. Alternatively, if the arrangement pattern generation unit 13a has already generated a plurality of arrangement patterns, it selects a new arrangement pattern that has not been selected. Then, the control from step 102 to step 110 is repeatedly performed until the number of pieces of learning data reaches the target value.

[0063] In this manner, the learning data generating device in this embodiment repeats the control of moving the workpieces 91 by the transport vehicle 31 so as to change the arrangement pattern of the multiple workpieces 91, the imaging of the arrangement area in which the multiple workpieces 91 are arranged by the visual sensor 32, and the generation of learning data by the learning data generating unit 15. The learning data generating method in this embodiment includes a measurement step in which the visual sensor 32, which is a measuring instrument, images the arrangement area in which the multiple workpieces 91 are arranged to obtain at least one of a two-dimensional image and a three-dimensional image, and a movement step in which the transport vehicle 31 moves at least one workpiece 91 to change the arrangement pattern of the workpieces. The learning data generating method includes a learning data generating step in which learning data including the image captured by the visual sensor and workpiece removal position information for removing the workpieces is generated. The movement step, the measurement step, and the learning data generating step are repeated to generate multiple pieces of learning data.

[0064] The learning data generating device and the learning data generating method of the present embodiment can generate learning data efficiently. In particular, for heavy or large workpieces, the transportation work required when changing the arrangement pattern of such multiple workpieces can be automated. The burden on the worker can be reduced and the work time can be shortened. For example, a heavy workpiece can be a workpiece weighing 10 kg or more or a workpiece weighing 20 kg or more, which is a heavy burden for one worker to carry. An example of a large workpiece can be a workpiece with a diameter of 1 m or more or a workpiece with a diameter of 2 m or more, which is a heavy burden for one worker to carry.

[0065] In the learning data generating device and the learning data generating method of the present embodiment, learning data is generated using images of workpieces handled in actual work, not in a simulation. For this reason, it is possible to generate real learning data that reflects lighting conditions, variations in workpiece manufacturing, deformation of the workpiece during transportation, and the like. By using such real learning data for machine learning, it is possible to correctly detect the workpiece in response to various situations in an actual real environment. As a result, it is possible to suppress failures in holding the workpiece, and to improve work efficiency.

[0066] In addition, image data in the case where the detection of the workpiece fails or the workpiece is not detected can also be collected and learned. It is possible to generate learning data that is closer to the actual work. In addition, images showing the background in which the workpiece is placed during the actual work can also be used as learning data. In particular, when multiple workpieces overlap, images including surrounding workpieces that become the background can be included in the learning data. In this case, since it is difficult to distinguish between the workpieces close to the surrounding workpiece to be taken out and the target work, it is possible to generate learning data that includes the relationship between the target workpiece to be taken out and the detailed and complex background around it. As a result, it is possible to improve the accuracy of detecting workpieces from images including detailed and complex backgrounds in machine learning.

[0067] Fig. 4 shows a perspective view of the workpieces and the visual sensor after the workpiece arrangement pattern has been changed once. In the arrangement pattern shown in Fig. 4, nine workpieces 91 are arranged within the field of view 32a. On the other hand, in the arrangement pattern shown in Fig. 1, twelve workpieces 91 are arranged within the field of view 32a. In each arrangement pattern, the type (size), number, and positions and postures of the workpieces arranged within the field of view 32a are different from each other.

[0068] In this way, the arrangement pattern of the workpieces can be changed in any manner. For example, the number of workpieces can be increased one by one from a state in which no workpieces 91 are arranged. Or, the number of workpieces in the field of view 32a imaged by the visual sensor 32 can be decreased. When changing the arrangement pattern, the number of workpieces can be increased or decreased one by one. Or, the arrangement pattern can be changed by increasing or decreasing the number of workpieces by any number. In addition to the number of workpieces, for example, the number of layers in which the workpieces are piled up, the number of workpieces arranged in each layer, the number of types of workpiece sizes, the number of workpieces of each size, and the position and posture of each workpiece can be changed arbitrarily.

[0069] 1 and 4, the workpieces are arranged so as to be aligned when viewed from the visual sensor 32, but this is not limiting. The workpieces may be arranged so that the orientation of the workpieces when viewed from the visual sensor is irregular. The workpieces 91 may be arranged so that the top surface 91a (the surface with the cutting line) of the workpiece 91 is inclined when viewed from the visual sensor 32. Moreover, the workpieces 91 may be arranged so that the side surface 91b (the surface without the cutting line) of the workpiece faces upward when viewed from the visual sensor 32.

[0070] The determination unit 15b of the learning data generation unit 15 may determine whether or not the workpieces are arranged according to the arrangement pattern determined by the arrangement pattern generation unit 13a. The arrangement pattern generation unit 13a sets an imaging plane for a virtual visual sensor in a three-dimensional simulation. The arrangement pattern generation unit 13a may be configured to output a projected image in which a plurality of workpieces are projected onto the imaging plane. The image processing unit 16 calculates a difference between the projected image output from the arrangement pattern generation unit 13a and an image captured by the visual sensor 32. The determination unit 15b may determine whether or not the workpieces are arranged according to the target arrangement pattern generated in the simulation based on a determination of whether or not the calculated difference is equal to or smaller than a predetermined threshold value. For example, the determination unit 15b can determine that the workpieces are arranged according to the target arrangement pattern generated in the simulation when the difference between the projected image and the image captured by the visual sensor 32 is smaller than a predetermined determination threshold value.

[0071] In addition, the visual sensor 32 of the learning data generation device 1 may capture an image of a background in which no workpieces are present. For example, the visual sensor 32 may capture an image of a container, tray, or palette in which no workpieces are placed. The arrangement pattern generation unit 13a may generate an arrangement pattern in which no workpieces are placed. The learning data generation unit 15 may generate learning data including an image consisting only of the background of the workpieces. By including an image in which no workpieces are placed in the learning data and performing machine learning, the difference from an image in which a workpiece is placed can be well learned. This can reduce failure cases in which noise in the background is erroneously detected as a workpiece.

[0072] The two-dimensional camera that captures the two-dimensional image of the workpiece can be any device that captures a two-dimensional image. The two-dimensional image capture device can be a visible light camera, such as a camera that captures black and white images or a camera that captures color images. Alternatively, an infrared camera that captures an image of a heated high-temperature metal workpiece, or an ultraviolet camera that captures an ultraviolet image that can detect scratches that are not visible with visible light, etc. may be used.

[0073] The visual sensor in this embodiment is fixed by a support member, i.e., the position of the visual sensor is fixed, but this is not limited to the above. The visual sensor may be configured to be movable. For example, the visual sensor may be fixed to a support member fixed to a transport vehicle and move together with the transport vehicle. Also, the visual sensor may be fixed to the hand of a robot arm and move together with the hand of the robot.

[0074] In addition, in the learning data generation device 1 of this embodiment, a transport vehicle 31 is adopted as a moving device that moves the workpiece 91, but this is not limited to this form. In particular, in this embodiment, an unmanned transport vehicle that moves autonomously is adopted, but this is not limited to this form. It may be a transport vehicle that is moved manually by remote control by an operator, or a transport vehicle that can be switched between autonomous and manual. As the transport vehicle, a transport vehicle having a wheel drive mechanism or a transport vehicle having a crawler drive mechanism can be adopted. Alternatively, it may be a transport vehicle that has both of these drive mechanisms and moves while switching between autonomous and manual depending on the condition of the floor surface.

[0075] The moving device may also be configured to perform an operation to change the three-dimensional position and attitude of the workpiece. For example, a coordinate system having an X-axis and a Y-axis extending horizontally can be set on the moving device. The moving device may have a locking mechanism that holds the workpiece so that the positions of the workpiece in the X-axis and Y-axis directions do not change when the workpiece is placed on the moving device. The moving device may also have a lifting mechanism that changes the position of the workpiece in the Z-axis direction. By employing this mechanism, the moving device can move in the X-axis and Y-axis directions with the workpiece placed on it and drive the lifting mechanism to change the three-dimensional position of the workpiece.

[0076] The moving device may also have a mechanism for changing the attitude of the member on which the workpiece is placed. For example, the moving device may be configured to have a rotation mechanism (e.g., a hinge and a spring) for tilting the member on which the workpiece is placed about the X-axis or the Y-axis. Furthermore, the moving device may have a mechanism for rotating about the Z-axis with the workpiece placed thereon, thereby changing the three-dimensional attitude of the workpiece. For example, the moving device may translate at +10 mm in the X-axis direction, -20 mm in the Y-axis direction, and +50 mm in the Z-axis direction to change the position and attitude of the workpiece. Furthermore, the moving device may rotate at +20° around the X-axis, -30° around the Y-axis, and +60° around the Z-axis. In this way, the arrangement pattern of multiple workpieces can be changed by changing the three-dimensional position and attitude of at least one workpiece. The moving device of this embodiment may be configured to have at least one of the above-mentioned mechanisms.

[0077] In this embodiment, one workpiece is placed on one transport vehicle, but this is not limited to the embodiment. Multiple workpieces can be placed on one transport vehicle. For example, multiple workpieces can be stacked or arranged side by side on one transport vehicle.

[0078] (Robot system equipped with machine learning device) FIG. 5 shows a perspective view of a first robot system in this embodiment that detects an actual workpiece and transfers it. FIG. 6 shows a block diagram of the robot system in this embodiment. With reference to FIGS. 5 and 6, the first robot system 8 includes a robot device including a robot 3 and a hand 4. The robot 3 in this embodiment is an articulated robot having a plurality of joints. The hand 4 in this embodiment includes a suction pad 4a. The hand 4 is configured to hold the workpiece 91 by suction. The robot device is not limited to this form, and any robot device having a robot and a hand capable of transferring a workpiece can be adopted.

[0079] The robot system 8 includes a visual sensor 32 that captures an image of an area where the workpieces 91 are placed. When actually performing the task of transporting the workpieces 91, a plurality of workpieces 91 arranged on a pallet 33 in the placement area are transported by any method. The workpieces 91 may be, for example, cardboard boxes containing products that have arrived at a warehouse in a logistics center. The workpieces 91 are placed on the pallet 33 and transported to an area where the robot device operates.

[0080] The visual sensor 32 captures an image of an arrangement area in which a plurality of workpieces 91 are arranged. The visual sensor 32 is fixed to a support member 92. The visual sensor 32 is fixed so that all of the workpieces 91 are arranged within a field of view 32a, but this is not limited to the form. The visual sensor 32 may be fixed to the hand of the robot 3 together with the support member 92 and moved with the operation of the robot 3. The visual sensor 32 is arranged above the area in which the workpieces 91 are arranged so as to mainly capture an image of the upper surface 91a of the workpiece 91, similar to the learning data generation device, but this is not limited to the form. The visual sensor 32 may be arranged diagonally above the area in which the workpieces 91 are arranged so as to capture an image of the upper surface 91a and the side surface 91b of the workpiece 91. Alternatively, the visual sensor 32 may be arranged so as to capture an image of the upper surface 91a and the side surface 91b of the workpiece 91 while being moved with the operation of the robot 3. As the visual sensor 32, a two-dimensional camera, a three-dimensional camera, and a measuring instrument including a two-dimensional camera and a three-dimensional measuring instrument can be adopted.

[0081] The robot system 8 includes a robot control device 2 (not shown) that controls the robot device. The robot control device 2 includes a calculation processing device (computer) having a CPU (Central Processing Unit) as a processor. The robot control device 2 includes a storage unit 42 that stores any information related to the robot system 8. The storage unit 42, like the storage unit 12 of the calculation processing device 10, can be configured of a non-transitory storage medium capable of storing information.

[0082] The robot control device 2 may be input with an operation program 41 that is generated in advance to perform the operations of the robot 3 and the hand 4, or may be configured to generate the operation program 41 internally. The operation control unit 43 transmits an operation command for driving the robot 3 based on the operation program 41 to the arm driving unit 44. The arm driving unit 44 includes an electric circuit that drives a driving motor, and provides an electric command to the arm driving device 46 based on the operation command. The operation control unit 43 also transmits an operation command for driving the hand driving device 47 to the hand driving unit 45. The hand driving unit 45 includes an electric circuit that drives, for example, an air pump, and provides an electric command to the air pump based on the operation command.

[0083] The operation control unit 43 corresponds to a processor that is driven according to the operation program 41. The processor reads the operation program 41 and executes the control defined in the operation program 41, thereby functioning as the operation control unit 43.

[0084] The robot system 8 of this embodiment includes a machine learning device. The machine learning device of this embodiment includes the above-mentioned learning data generation device 1 and a robot control device 2. The robot control device 2 includes a machine learning unit 51 that performs machine learning. The machine learning unit 51 acquires characteristics, features, and the like included in the input learning data by learning.

[0085] The machine learning unit 51 includes a data acquisition unit 52 and a learning unit 54 that generates a learning model 55. The machine learning unit 51 of this embodiment performs supervised learning. In supervised learning, learning data 57, i.e., a plurality of sets of input data (including image data and label data), are input to the machine learning unit 51. The machine learning unit 51 learns the relationship between the image data and the label data included in the input data set. The machine learning unit 51 generates a model (learning model) that estimates a label from an image, i.e., a model that acquires the relationship of the label to the image.

[0086] In this embodiment, the data acquisition unit 52 acquires an image 58 included in the learning data 57 as an input. The data acquisition unit 52 also acquires at least one of a pick-up position and posture 59 and an outer shape 60 of a workpiece included in the learning data 57 as a label. The learning unit 54 acquires, for example, 100 sets of learning data 57 including the labels as input data. The learning unit 54 performs deep learning using the learning data as input data for MASK R-CNN (Region Based Convolutional Neural Networks) to generate a learning model 55.

[0087] The machine learning unit 51 corresponds to a processor driven according to a machine learning program. The processor performs control defined in the program, thereby functioning as the machine learning unit 51. Furthermore, each of the data acquisition unit 52, the learning unit 54, and the inference unit 56 corresponds to a processor driven according to the machine learning program.

[0088] The data acquisition unit 52 acquires, for example, an image captured by a two-dimensional camera as image 58. The data acquisition unit 52 also acquires information on the position and orientation of each workpiece and outer shape information in each of a plurality of arrangement patterns of a plurality of workpieces generated by the arrangement pattern generation unit 13a as removal position and orientation 59 and outer shape 60 of the workpiece. The learning unit 54 can generate a learning model 55 that estimates the removal position and orientation of the workpiece and the outer shape of the workpiece from the image of the arrangement area of ​​the workpiece captured by the two-dimensional camera.

[0089] The inference unit 56 acquires the learning model 55 generated by the learning unit 54. The inference unit 56 uses the learning data 57, which is a two-dimensional image captured by the visual sensor 32 of the robot system 8 as input data, to estimate, for example, the three-dimensional removal position and posture of the upper surface 91a of the workpiece 91 shown in the image and the outer shape of the workpiece.

[0090] The pick-up position and orientation of the workpiece 91 can be calculated, for example, in the sensor coordinate system of the visual sensor 32. A world coordinate system that remains immovable even if the position and orientation of the robot 3 change is set in the robot device. Calibration between the sensor coordinate system and the world coordinate system set in the robot device can be performed in advance.

[0091] The operation control unit 43 of the robot control device 2 acquires the pick-up position and posture of the workpiece 91 expressed in the sensor coordinate system from the inference unit 56. The operation control unit 43 converts the pick-up position and posture of the workpiece 91 expressed in the world coordinate system. When the pick-up position and posture 59 of the workpiece as the label data of the learning data 57 are already expressed in the world coordinate system, the operation control unit 43 of the robot control device 2 acquires the pick-up position and posture of the workpiece 91 expressed in the world coordinate system from the inference unit 56. The operation control unit 43 calculates the position and posture of the robot and the hand when the robot picks up the workpiece based on the pick-up position and posture of the workpiece expressed in the world coordinate system. The operation control unit 43 can control the robot 3 and the hand 4 to hold the workpiece 91.

[0092] The operation control unit 43 can perform the task of picking up the workpieces 91 in a predetermined order. For example, the operation control unit 43 performs control to hold and transport the workpieces 91 to a predetermined position in order starting from the workpiece 91 arranged at the end or the highest position of the arrangement pattern. The robot device can transport each workpiece 91 to, for example, a conveyor arranged on the side of the working area where the workpieces 91 are arranged.

[0093] In this manner, the machine learning device of the present embodiment includes a learning data generation device that generates learning data. The machine learning device includes a learning unit that generates a learning model that estimates the removal position of the work from an image of the work placement area. The learning unit performs machine learning based on the learning data generated by the learning data generation unit. The machine learning device includes an inference unit that estimates the removal position of the work from an image acquired by a measuring instrument. The machine learning method also includes a learning step of generating a learning model that estimates the removal position of the work from an image of the work placement area based on the learning data generated by the above-mentioned learning data generation method. In the learning step, machine learning is performed based on the learning data generated in the learning data generation step. The machine learning method includes an inference step of estimating the removal position of the work from an image acquired by a measuring instrument based on the learning model generated in the learning step. In the machine learning device or machine learning method of the present embodiment, a learning model with excellent accuracy in estimating the removal position can be generated in order to use the learning data generated by the learning data generation device of the present embodiment. Alternatively, the inference unit can accurately estimate the removal position from the image of the work placement area.

[0094] The learning data used for machine learning may be configured to acquire images captured in the past, removal position information, and the like. For example, the learning data generating unit may acquire images, removal positions and postures of the workpieces, and external shape information from a database of the cloud or a specified device. The learning data generating unit may also acquire two-dimensional images or removal position information recorded in a storage medium such as a memory via a network such as a LAN (Local Area Network). Alternatively, the learning data generating unit may be configured to remotely acquire images captured by a visual sensor installed in a remote location and removal position information via a network.

[0095] Although the machine learning device of the present embodiment performs supervised learning, the present invention is not limited to this form and can perform any machine learning. For example, machine learning such as semi-supervised learning, unsupervised learning, or reinforcement learning can be performed using the learning data generated by the learning data generation device.

[0096] 7 shows a perspective view of a second robot system that performs actual work in this embodiment. In the first robot system 8 described above, the position of the robot device is fixed, but this is not limited to the embodiment. The second robot system 9 includes a slide device 34 for moving the robot 3. The slide device 34 includes a moving stage 35 to which the robot 3 is fixed. The moving stage 35 is configured to be movable in the direction in which the slide device 34 extends, as shown by an arrow 99.

[0097] In this way, the robot 3 may be fixed to a moving device. By adopting this configuration, even a robot 3 with a short stroke can pick up all of the workpieces. In other words, the robot can be made smaller. In the second robot system 9, a slide device is adopted as a device for moving the entire robot, but this is not limited to the embodiment. For example, a transport vehicle may be adopted as a device for moving the entire robot. In other words, the robot may be fixed to a transport vehicle and formed to be movable in any direction.

[0098] (Second learning data generating device) The second learning data generating device in this embodiment will be described. The configuration of the second learning data generating device is the same as that of the first learning data generating device 1 (see Figs. 1 and 2). In the first learning data generating device 1, the take-out position and attitude 59 of the work included in the learning data 57 are the three-dimensional take-out position and attitude calculated in a simulation executed by the arrangement pattern generating unit 13a. Also, for the outer shape 60 of the work, information on the three-dimensional CAD model of the work used in the simulation is used.

[0099] 1, an arrangement pattern is generated by moving a transport vehicle 31 carrying a workpiece 91. However, the position and orientation of the workpiece 91 transported by the transport vehicle 31 may deviate slightly from the position and orientation of the workpiece calculated by the simulation. A method for correcting and calculating this deviation will be described below.

[0100] 2, in the second learning data generation device, the image processing unit 16 performs image processing of the two-dimensional image acquired by the visual sensor 32. The above-mentioned projection image generated by the arrangement pattern generation unit 13a is a projection image of an arrangement pattern generated so as to satisfy the learning data generation condition 25 specified by the worker, and is an image that strictly reflects the intention of the worker. This projection image is stored in the storage unit 12 as a reference image. In each reference image, the removal position of each workpiece is calculated and determined by the simulation device.

[0101] The image processing unit 16 calculates the difference between a two-dimensional image capturing an actual arrangement pattern created in the real world by the operation of multiple transport vehicles 31 and the reference image, thereby performing calculations to correct the difference or amount of deviation from the position and posture or external shape of the work in the reference image, and calculates the removal position of the work and the external shape of the work in the actually captured two-dimensional image. This makes it possible to generate misaligned label data (removal position and posture, and external shape information) for two-dimensional image data capturing an actual arrangement pattern created in the real world by the operation of transport vehicles 31, and to generate excellent learning data including misaligned label data and image data.

[0102] The learning data generating unit 15 generates learning data including the two-dimensional image obtained from the visual sensor 32, the two-dimensional removal position and posture of the workpiece in the two-dimensional image detected by the image processing unit 16, and the outer shape of the workpiece. With reference to Fig. 6, the learning unit 54 of the machine learning device can perform supervised learning using the learning data. The learning unit 54 can generate a learning model 55 that estimates the removal position and posture of the workpiece in the two-dimensional image and the outer shape of the workpiece from the two-dimensional image.

[0103] For example, 100 two-dimensional images are acquired for 100 sets of different workpiece arrangement patterns. Image processing including pattern matching is performed on each image. Information on the removal position and orientation of the workpiece in the two-dimensional image and information on the outer shape of the workpiece are acquired for each image. These 100 sets of learning data are generated, and deep learning using MASK R-CNN is performed to generate a learning model.

[0104] The robot system that actually transports the workpiece may have a configuration similar to that of the robot system in Figures 5 and 6. Here, a measuring instrument capable of acquiring two-dimensional images and three-dimensional point cloud data may be used as the visual sensor 32. For example, a stereo camera may be used as the visual sensor 32, which is a measuring instrument.

[0105] In the work of transporting a workpiece, the inference unit 56 estimates the removal position and posture of the workpiece and the outer shape of the workpiece in the two-dimensional image contained in the image based on the two-dimensional image captured by the visual sensor 32. The visual sensor 32 outputs the two-dimensional image and also outputs three-dimensional point cloud data.

[0106] The inference unit 56 acquires three-dimensional point cloud data from the visual sensor 32. The inference unit 56 acquires three-dimensional point cloud data acquired by, for example, a stereo camera. The inference unit 56 calculates a two-dimensional take-out position (position of a pixel in an image) and an orientation in a two-dimensional image. Next, the inference unit 56 calculates a three-dimensional take-out position of a point in a three-dimensional space corresponding to the take-out position (pixel position) in the two-dimensional image from the three-dimensional point cloud data based on the relationship of the calibration of the measuring instrument. Regarding the orientation of the workpiece, when the learning data 57 includes the take-out orientation of the workpiece, the orientation of the workpiece estimated by the inference unit 56 can be adopted. In this way, the inference unit 56 can calculate the three-dimensional take-out position and orientation of the workpiece from the two-dimensional image by using measurement data from a measuring instrument having the functions of a two-dimensional measuring instrument and a three-dimensional measuring instrument.

[0107] In addition, when the learning data 57 does not include the take-out orientation of the workpiece, the inference unit 56 may calculate the orientation of the workpiece in the three-dimensional space based on the three-dimensional point cloud data. For example, the three-dimensional orientation of the workpiece can be calculated based on the two-dimensional orientation in the calculated two-dimensional image and the point cloud data around the three-dimensional take-out position.

[0108] Next, the operation control unit 43 calculates the three-dimensional position and orientation of the robot and the hand when the robot picks up the workpiece, based on the three-dimensional pick-up position of the workpiece 91 calculated from the three-dimensional point cloud data and the orientation of the workpiece at the pick-up position. Then, the operation control unit 43 controls the position and orientation of the robot 3 and the hand 4.

[0109] In this way, the second learning data generating device can obtain information on the position and posture of the workpiece in the two-dimensional image and information on its outer shape by using the two-dimensional image. A measuring instrument capable of obtaining two-dimensional images and three-dimensional point cloud data can be used as a visual sensor for the robot that actually performs the work. The second learning data generating device can generate excellent learning data in which the deviation of the actual workpiece position in the real world from the workpiece position in the simulation is corrected.

[0110] In the above embodiment, pattern matching is performed to detect the removal position of the workpiece, but the present invention is not limited to this embodiment. For example, image recognition techniques such as blob detection or cylinder detection may be used to detect the removal position, posture, and outer shape of the workpiece in the image.

[0111] Furthermore, the measuring instrument used in the actual transport work is not limited to a stereo camera, and any measuring instrument capable of acquiring 2D images and 3D point cloud data can be used. For example, a measuring instrument in which any 3D measuring instrument is attached to a 2D camera can be used. A measuring instrument including one 2D camera and a distance sensor, or a measuring instrument including one 2D camera and a laser scanner can be used.

[0112] Alternatively, a configuration in which one two-dimensional camera is attached to a movable device may be adopted. Images equivalent to those of a stereo camera can be acquired by capturing images of the same arrangement pattern of the workpiece from two different positions or angles that are determined in advance using one camera. In addition, although three-dimensional point cloud data is adopted in the above embodiment, this is not limiting. A distance image may be adopted as the three-dimensional information.

[0113] In the above-mentioned training data generating device, the training data is configured to use two-dimensional images, but the present invention is not limited to this configuration, and three-dimensional point cloud data may be used as the training data. That is, instead of the two-dimensional measuring device, a three-dimensional measuring device that acquires three-dimensional point cloud data may be disposed as a measuring device of the training data generating device. For example, a stereo camera may be disposed instead of the two-dimensional camera.

[0114] When the image processing unit acquires three-dimensional point cloud data using a three-dimensional measuring device, the image processing unit can also generate a distance image from the three-dimensional point cloud data. For example, a distance image can be generated in which pixel density changes according to the distance from the origin position of the three-dimensional measuring device or an arbitrary reference position in three-dimensional space to each work or background. Then, learning data including the distance image can be used to implement control similar to that for two-dimensional images.

[0115] Furthermore, information on the external shape of the workpiece and information on the removal position and orientation may be included in the learning data together with the distance image acquired by the three-dimensional measuring device. The measuring device performs three-dimensional measurement of the placement area of ​​the workpiece, thereby obtaining three-dimensional information as three-dimensional point cloud data. The learning data generating unit may generate learning data including three-dimensional removal position information of the workpiece. In this case, even in a robot system that uses a learning model generated by performing machine learning to infer the removal position of the workpiece and transport the workpiece, a three-dimensional measuring device can be adopted as a measuring device that measures the image and three-dimensional point cloud data included in the learning data. Then, for example, the removal position and orientation and external shape of the workpiece can be estimated based on a distance image or the like generated by measuring the placement area of ​​the newly placed workpiece.

[0116] In addition, when a three-dimensional measuring device capable of acquiring three-dimensional information is used, three-dimensional point cloud data can be acquired together with the two-dimensional image. The image processing unit can perform a matching process between the three-dimensional point cloud data and the arrangement pattern in the three-dimensional simulation generated by the arrangement pattern generating unit.

[0117] Next, the image processing unit can convert the three-dimensional take-out position and orientation of the workpiece in the three-dimensional simulation (position and orientation expressed in the coordinate system of the virtual visual sensor in the simulation) and the outer shape of the workpiece into the three-dimensional take-out position and orientation of the workpiece in the three-dimensional point cloud data (position and orientation expressed in the coordinate system of the three-dimensional measuring device) and the outer shape of the workpiece. Then, the learning data generating unit may generate learning data including at least one of the three-dimensional point cloud data, the three-dimensional take-out position and orientation of the workpiece in the three-dimensional point cloud data, and the outer shape of the workpiece. The machine learning device may employ this learning data to perform machine learning.

[0118] The arithmetic processing device of the learning data generating device may have a display unit that displays the generated learning data. The display unit may be configured with a display panel such as a liquid crystal display panel or a touch panel. The display unit displays the learning data in which labels (such as information on the removal position and posture of the workpiece and the outer shape of the workpiece) are automatically attached to the image (including the distance image) or the three-dimensional point cloud data. The worker can check whether or not there is an erroneously attached label in the image or the three-dimensional point cloud data. The worker may delete or correct a label that is determined to be incorrect using an input device such as a keyboard, a mouse, or a touch pen. Alternatively, the worker may add new label information. In this way, by performing machine learning using the learning data corrected by the worker, a learning model with fewer errors and high robustness can be generated.

[0119] Other configurations, actions, and effects are similar to those of the first learning data generation device and the machine learning device equipped with the first learning data generation device, and therefore will not be described repeatedly here.

[0120] (Third learning data generating device) A third learning data generating device of this embodiment will be described. In the above-mentioned first learning data generating device 1 and second learning data generating device, control is performed to change the position and posture of the workpieces imaged by the visual sensor and to increase the number of workpieces. In the third learning data generating device, a plurality of carrier vehicles are used to change the arrangement pattern of the workpieces captured within the visual field of the visual sensor.

[0121] First, multiple workpieces are placed inside the visual field of the visual sensor. For example, multiple workpieces may be placed inside the visual field of the visual sensor by using the method implemented in the first learning data generation device or the second learning data generation device. Next, control is performed to change the placement pattern by moving at least one or more workpieces outside the visual field of the visual sensor. The configuration of the third learning data generation device is similar to that of the first learning data generation device (see Figures 1 and 2). In addition, a two-dimensional camera capable of acquiring two-dimensional images is adopted as the visual sensor 32.

[0122] FIG. 8 shows a flowchart of the control performed by the third learning data generating device. With reference to FIG. 1, FIG. 2, and FIG. 8, in step 121, the transport vehicle 31 places a plurality of workpieces 91 inside the field of view 32a of the visual sensor 32. For example, as shown in FIG. 1, many workpieces 91 are placed within the range imaged by the visual sensor 32. This control can be performed by the same control as that of the first learning data generating device. For example, in the control of the first learning data generating device of FIG. 3, it can be performed by the control from step 101 to step 104. Alternatively, the operator may manually transport the transport vehicle 31 to place a plurality of workpieces 91 inside the field of view 32a of the visual sensor 32.

[0123] Next, steps 105 to 110 are similar to steps 105 to 110 in the control of the first learning data generation device 1 (see FIG. 3). In step 110, if the number of learning data is less than the target value, control proceeds to step 125.

[0124] In step 125, the motion plan generating unit 14a of the control unit 14 generates a motion plan for the moving transport vehicle 31. Here, an embodiment in which control is performed to reduce the number of workpieces 91 one by one will be described. For example, the image processing unit 16 performs image processing of the two-dimensional image captured in step 105 to detect the positions of multiple workpieces. The motion plan generating unit 14a can generate motion plans for multiple transport vehicles carrying multiple workpieces at the positions of the detected multiple workpieces based on the processing results of the image processing unit 16. Alternatively, the moving transport vehicle 31 may be determined by the placement pattern generating unit 13a generating a placement pattern. The motion plan generating unit 14a can predetermine the order of movement of the multiple moving transport vehicles 31. For example, the transport vehicles 31 can be moved in order from the end of the placement pattern.

[0125] The motion plan generating unit 14a may generate a motion plan for the transport vehicle 31 so that the transport vehicle 31 moves a workpiece arranged inside the field of view 32a to the outside. For example, the motion plan generating unit 14a can generate a motion plan so that the transport vehicle 31 carrying the workpiece 91 leaves the field of view 32a of the visual sensor 32 and returns to a predetermined origin position or start position.

[0126] Next, in step 126, the motion command generating unit 14b of the control unit 14 drives the transport vehicle 31 based on the motion plan generated in step 125. The motion command generating unit 14b can transmit a speed command or a movement path command for the transport vehicle 31. After this, the control returns to step 105. Then, the control from step 105 to step 110 can be repeated.

[0127] In this way, the arrangement pattern can be changed by generating an arrangement pattern in which the number of workpieces is reduced one by one. Note that the movement of one workpiece is not limited, and learning data may be generated by taking an image every time two or more workpieces are moved. If the number of workpieces present inside the field of view 32a of the visual sensor 32 is reduced and there are no more workpieces to move, the control shown in FIG. 8 can be repeated again.

[0128] When multiple transport vehicles 31 are movable, the motion plan generating unit 14a can generate a motion plan such that each transport vehicle 31 carrying a workpiece 91 leaves the visual field of the visual sensor one by one. Alternatively, the motion plan generating unit 14a may generate a motion plan for moving multiple transport vehicles 31 at once. The imaging command generating unit 14c can transmit a command to the visual sensor 32 to capture an image after the transport vehicle 31 leaves the visual field 32a of the visual sensor 32 and returns to a predetermined position such as the origin position.

[0129] The other configurations, actions, and effects are similar to those of the first learning data generation device, the second learning data generation device, and the machine learning device equipped with the first learning data generation device, so the description will not be repeated here.

[0130] (Fourth learning data generating device) Fig. 9 shows a perspective view of a fourth learning data generating device in this embodiment. Fig. 10 shows a block diagram of the fourth learning data generating device in this embodiment. With reference to Figs. 9 and 10, in the fourth learning data generating device 5, a robot device including a robot 3 and a hand 4 is used to change the arrangement pattern of a plurality of workpieces 91 inside a visual field 32a of a visual sensor 32. As the visual sensor 32, a two-dimensional camera capable of acquiring a two-dimensional image is adopted.

[0131] The fourth learning data generation device 5 includes a robot device including a robot 3 and a hand 4 as a moving device that moves a workpiece. The fourth learning data generation device 5 includes a robot control device 6. The configurations of the operation control unit 43, arm driving unit 44, and hand driving unit 45 of the robot control device 6 are similar to the configurations of the robot control device 2 of the robot system 8 that actually transports the workpiece (see FIG. 6). The configurations of the robot 3 and hand 4 of the robot device are similar to the configurations of the robot system 8 that actually transports the workpiece. In addition, the configuration of the arithmetic processing device 10 is similar to the configuration of the arithmetic processing device 10 of the first learning data generation device 1 (see FIG. 2).

[0132] In the fourth learning data generation device 5, a robot device is used to move the workpiece 91 from inside the field of view 32a of the visual sensor 32 to outside the field of view 32a. That is, the workpiece 91 is moved using a robot device instead of the transport vehicle 31 of the third learning data generation device.

[0133] A flow chart of control of the fourth learning data generation device is shown in Fig. 11. With reference to Figs. 9 to 11, step 121 and steps 105 to 110 are similar to those of the third learning data generation device (see Fig. 8). In step 110, when the number of pieces of learning data is less than the target value, control proceeds to step 128.

[0134] In step 128, the motion plan generating unit 14a of the control unit 14 generates a motion plan for the robot device so as to move the multiple workpieces located at the positions of the multiple workpieces detected as a result of image processing performed by the image processing unit 16 on the image captured in step 105. The control for selecting the workpieces to be moved in this manner can be the same as that in step 125 of the third learning data generating device (see FIG. 8). The motion plan generating unit 14a generates a motion plan for the robot device so as to move the workpiece 91 to a predetermined position, for example, onto a conveyor.

[0135] In step 129, the motion command generator 14b generates a motion command for the robot device based on the motion plan for the robot device generated by the motion plan generator 14a. The motion command generator 14b then transmits the generated motion command for the robot device to the motion control unit 43 of the robot controller 6. The motion command generator 14b transmits the position and posture of the robot and the hand for picking up the workpiece to the motion control unit 43 of the robot controller 6.

[0136] The operation control unit 43 drives the robot 3 and the hand 4 based on the above-mentioned operation command. Then, after the hand 4 holds a predetermined workpiece 91, the workpiece 91 is placed outside the field of view 32a. For example, the workpiece 91 can be transported to a temporary placement area or temporary placement shelf placed outside the field of view 32a. After this, the control returns to step 105. Then, the control from step 105 to step 110 is repeated until the number of learning data reaches the target value. When there are no more workpieces 91 to be moved, the control of FIG. 11 can be repeated.

[0137] In this way, even if a robot device is used as the moving device, the arrangement pattern of the workpieces can be changed by moving the workpieces arranged inside the visual field of the visual sensor to the outside of the visual field.

[0138] If the robot control device 6 fails to cause the robot device to remove the workpiece 91, the transport vehicle 31 carrying the workpiece may move to move the workpiece 91 outside the field of view 32a of the visual sensor 32. For example, the control unit 14 may control the transport vehicle 31 carrying the workpiece to leave the field of view 32a of the visual sensor 32 and move to the origin position. After the transport vehicle moves to the origin position, generation of learning data including capturing an image by the visual sensor can be resumed.

[0139] In addition, in the fourth learning data generating device, the arrangement pattern is changed by moving the workpieces so that they move from inside to outside the visual field of the visual sensor. That is, the robot device is controlled so that the number of workpieces present inside the visual field of the visual sensor is reduced, but this is not limited to the above. The robot device may transport the workpieces so that the number of workpieces present inside the visual field of the visual sensor is increased.

[0140] The other configurations, actions, and effects are similar to those of the first learning data generation device to the third learning data generation device, and the machine learning device equipped with the first learning data generation device, so the description will not be repeated here.

[0141] In the third learning data generating device and the fourth learning data generating device, the arrangement pattern is changed by moving the work transported by the transport vehicle from inside to outside the visual field of the visual sensor, but this is not limited to the above. The arrangement pattern may be changed by moving the work transported by the transport vehicle from outside to inside the visual field of the visual sensor. Alternatively, the arrangement pattern may be changed by changing the position of the work transported by the transport vehicle as the transport vehicle moves inside the visual field of the visual sensor. Alternatively, the arrangement pattern may be changed by changing the posture of the transport vehicle without changing the position of the work and changing the posture of the work carried on the transport vehicle inside the visual field of the visual sensor.

[0142] In the above-mentioned first to fourth learning data generating devices, the learning data mainly includes two-dimensional images captured by a visual sensor, but the present invention is not limited to this form. For example, when the learning data generating device includes a three-dimensional measuring device, a distance image may be generated from three-dimensional point cloud data measured by the three-dimensional measuring device, and learning data including the distance image and label data such as the outer shape of the workpiece and the removal position and posture of the workpiece may be generated. The machine learning device may perform machine learning using such learning data.

[0143] In the above-mentioned first to fourth learning data generating devices, the learning data mainly includes a two-dimensional image captured by a visual sensor, but the present invention is not limited to this form. For example, when the learning data generating device includes a three-dimensional measuring device, a matching process is performed between the three-dimensional point cloud data measured by the three-dimensional measuring device and the three-dimensional arrangement pattern generated by the arrangement pattern generating unit. Using the result of the matching process, the three-dimensional take-out position, orientation, and external shape information in the three-dimensional simulation of the multiple workpieces calculated by the arrangement pattern generating unit are changed to the three-dimensional take-out position, orientation, and external shape information in the three-dimensional point cloud data measured in the real world. The changed take-out position, orientation, and external shape information may be used as label data for the three-dimensional point cloud data to generate learning data including the three-dimensional point cloud data. The machine learning device may perform machine learning using such learning data.

[0144] The first to fourth learning data generating devices described above are configured to generate learning data mainly using information such as the workpiece removal position and posture and the workpiece outer shape calculated by the arrangement pattern generating unit as label data. Alternatively, learning data is generated using information such as the workpiece removal position and posture and the workpiece outer shape detected as a result of image processing by the image processing unit as label data, but is not limited to these forms. For example, a distance image generated from an image captured by a visual sensor or three-dimensional point cloud data measured by a three-dimensional measuring device, or three-dimensional point cloud data may be displayed on a display unit such as a monitor, and an operator may instruct information such as the workpiece removal position and posture and the workpiece outer shape on the image or three-dimensional point cloud data using an input device such as a mouse. That is, the label data included in the learning data may be teaching data generated by an operator.

[0145] An example of application of the learning data generating device and the machine learning device of the present embodiment is a system that transports cardboard boxes containing products as workpieces in a warehouse of a logistics center. For example, cardboard boxes containing products delivered in a receiving process in a warehouse of a logistics center are placed on a transport vehicle such as an AGV, and images are captured by a two-dimensional camera while controlling the operation of multiple AGVs to change the arrangement pattern of the multiple cardboard boxes. Learning data including the captured multiple images, the removal positions and postures of the cardboard boxes shown in each image, and the outer shape information of the cardboard boxes can be generated by the above-mentioned method. Machine learning is performed using the generated learning data to generate a learning model. For images of multiple cardboard boxes stacked on a pallet delivered in the receiving process captured by a two-dimensional camera, a robot device may remove the cardboard boxes at the removal positions and postures inferred using the learning model, place them on a conveyor, and flow. After that, post-processes such as unpacking, product check, and classification may be performed.

[0146] In each of the above-described controls, the order of steps can be changed as appropriate within the scope that does not change the functions and actions.

[0147] The above-mentioned embodiments can be combined as appropriate. In each of the above-mentioned drawings, the same or equivalent parts are given the same reference numerals. Note that the above-mentioned embodiments are examples and do not limit the invention. In addition, the embodiments include modifications of the embodiments shown in the claims. [Explanation of symbols]

[0148] 1,5 Learning data generation device 2,6 Robot control device 3. Robot 4 Hands 8,9 Robot Systems 10 Processing unit 11 Reception 12 Storage section 13a Placement pattern generation unit 14 Control section 14a Motion plan generation unit 14b Operation command generation section 15 Learning data generation unit 15b Judgment section 16 Image processing section 25 Conditions for generating training data 26 Training Data Generation Program 31 Transport vehicle 32 Visual Sensor 32a field of view 51 Machine Learning Department 54 Learning Department 55 Learning Model 56 Reasoning part 57 Training Data 58 images 59 Pick-up position and posture 60 Workpiece Outline 91 Work

Claims

1. A device for generating learning data for use in machine learning, comprising: a measuring device for measuring an area in which a plurality of workpieces are arranged and acquiring at least one of a two-dimensional image and a three-dimensional image; A moving device that moves at least one workpiece; A control unit for controlling an operation of the moving device; A learning data generating unit that generates learning data including an image acquired by the measuring device and workpiece removal position information for removing the workpiece, A learning data generation device that generates multiple learning data by repeating control of moving workpieces with the moving device so as to change the work placement pattern, measurement of placement areas of multiple workpieces with the measuring instrument, and generation of learning data by the learning data generation unit.

2. A placement pattern generating unit that generates a placement pattern of a plurality of workpieces, The learning data generating device according to claim 1 , wherein the arrangement pattern generating unit generates an arrangement pattern of the workpieces based on a predetermined learning data generating condition.

3. A reception unit for receiving information from an external source is provided, The learning data generation device of claim 2 , wherein the reception unit acquires learning data generation conditions including at least one of a target value for the number of learning data, a range of the number of work types, a range of work sizes, and work arrangement pattern conditions.

4. The learning data generating device according to claim 3 , wherein the conditions for the arrangement pattern of the workpieces include at least one of a range of the number of layers in which the plurality of workpieces are stacked and a condition for a gap between the workpieces.

5. The control unit includes: a motion plan generating unit that generates a motion plan for the moving device using a position and an attitude of a workpiece in the workpiece placement pattern generated by the placement pattern generating unit as a target value; and a motion command generating unit that generates a motion command for operating the moving device based on the generated motion plan; The learning data generating device according to claim 2 , wherein the movement device moves a workpiece in accordance with the generated motion command.

6. The learning data generating device according to claim 1 , wherein the moving device is configured to perform an operation for changing a three-dimensional position and orientation of a workpiece.

7. The measuring instrument includes a three-dimensional measuring instrument that performs three-dimensional measurement of a placement area of ​​a workpiece to obtain three-dimensional point cloud data; The learning data generating device according to claim 1 , wherein the learning data generating unit generates learning data including three-dimensional removal position information of a workpiece.

8. An image processing unit that processes an image of a workpiece acquired by the measuring instrument, the image processing unit outputs, as a processing result, workpiece pick-up position information for picking up the workpiece by a robot device including a robot and a hand; The learning data generating device according to claim 1 , wherein the learning data generating unit generates learning data including workpiece removal position information output from the image processing unit.

9. An image processing unit that processes an image of a workpiece acquired by the measuring instrument, The learning data generating device according to claim 5 , wherein the motion plan generating unit generates a motion plan based on a processing result by the image processing unit.

10. 10. The device for generating learning data according to claim 1, wherein the learning data generation unit includes a determination unit that determines whether the learning data is acceptable based on whether a number of pieces of learning data has reached a predetermined target value.

11. An image processing unit that processes an image of a workpiece acquired by the measuring instrument, 11. The device for generating learning data according to claim 1, wherein the learning data generation unit includes a determination unit that determines whether to store or discard the learning data based on a result of image processing by the image processing unit and a predetermined criterion for determining whether the learning data is acceptable.

12. The learning data generating device according to claim 1 , wherein the moving device includes a transport vehicle capable of moving on a floor surface with a workpiece placed thereon.

13. The learning data generating device according to claim 1 , wherein the moving device includes a robot device having a robot and a hand, and configured to hold a workpiece and move the workpiece.

14. The training data generating device according to claim 1 ; A learning unit that performs machine learning based on the learning data generated by the learning data generation unit and generates a learning model that estimates a removal position of the workpiece from an image of a placement area of ​​the workpiece; A machine learning device comprising: an inference unit that estimates a workpiece removal position from an image acquired by the measuring instrument based on a learning model generated by the learning unit.

15. A method for generating learning data for use in machine learning, comprising the steps of: a measuring step in which a measuring device measures an arrangement area of ​​a plurality of workpieces to obtain at least one of a two-dimensional image and a three-dimensional image; a moving step in which the moving device moves at least one workpiece to change the arrangement pattern of the workpieces; A learning data generating process for generating learning data including the image acquired in the measuring process and workpiece removal position information for removing the workpiece, A learning data generating method, comprising repeating the moving step, the measuring step, and the learning data generating step to generate a plurality of learning data.

16. A method for generating training data according to claim 15; A learning process of performing machine learning based on the learning data generated in the learning data generation process to generate a learning model that estimates a removal position of the workpiece from an image of a placement area of ​​the workpiece; A machine learning method comprising: an inference process of estimating a workpiece removal position from the image acquired in the measurement process based on the learning model generated in the learning process.

Images