Robot system, method of controlling robot system, method of manufacturing article, program, and recording medium

Abstract

To provide a robot system that reduces burden on an operator.SOLUTION: A robot system includes: a robot; a survey section which surveys a work region on a workpiece where work is performed and acquires survey data including information on the work region; and a control section which controls the robot to identify the work region on the basis of information on the work region, information which is associated with the work region and is on the work to be executed on the workpiece, and the survey data (S104 to S105) and to perform a work on the identified work region (S106).SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a robot system, a method for controlling a robot system, a method for manufacturing an article, a program, and a recording medium.

Background Art

[0002] For example, in a robot device such as an industrial robot installed in a factory or the like, there are those that perform operations such as assembling or attaching parts, applying an adhesive or paint, processing using a tool, etc. to a workpiece. Such operations can perform precise (high-precision) work regardless of the position and orientation of the workpiece by, for example, recognizing an image of the part of the workpiece where the operation is to be performed using a camera and controlling the position and orientation of the robot with respect to the recognized part. However, when such precise work is to be executed by a robot device, there is a problem that the burden of adjustment work for the operator to adjust the robot is large when, for example, the robot device is installed in a factory or when the work content or workpiece changes. For example, in a method using template matching in image recognition, in order to increase the matching accuracy of matching the template image and the captured image, the adjustment of the image processing process such as luminance correction and feature extraction is complicated, and it takes time and is burdensome for the operator to set the conditions.

[0003] Therefore, a method has been proposed in which an image of a workpiece is input to a trained learning machine to obtain two or more partial extraction images, blob analysis is performed on these partial extraction images to generate blob information, and the position and angle of the workpiece are calculated from the blob information (see Patent Document 1). Thus, the method of Patent Document 1 aims to reduce the load of the adjustment work in the above-described image processing process.

[0004] In addition, a system has been proposed that focuses on the function (affordance) of an object and generates a trained model for three-dimensionally recognizing a region indicating the function (see Non-Patent Document 1). The system of Non-Patent Document 1 uses the trained model to specify the working area of a three-dimensional robot manipulator and attempts to grip and transfer a workpiece.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, although the one in the above Patent Document 1 can cope with the measurement of the position and angle of a workpiece on a two-dimensional plane, there is a problem that it is difficult to calculate the three-dimensional position and posture information of the workpiece from blob information. Therefore, it is difficult to automatically generate the trajectory of the robot in the operation of the work, it takes time for the drive generation (teaching) of the robot, and the burden is imposed on the adjustment work of the robot by the operator.

[0007] In addition, although the one in the above Non-Patent Document 1 may be able to cope with rough operation work, it is difficult to realize what requires precise work (high recognition accuracy). That is, even if the one in the above Non-Patent Document 1 is adopted, for example, in precise assembly work, etc., it is necessary to set a precise operation (trajectory) of the robot arm with respect to the work area of the recognized workpiece, and an operator with specialized knowledge needs to perform the setting, and still the burden on the operator is large.

[0008] Therefore, an object of the present invention is to provide a robot system, a control method of the robot system, a manufacturing method of an article, a program, and a recording medium that can reduce the burden on an operator.

Means for Solving the Problems

[0009] One aspect of the present invention is a robot system comprising: a robot; a survey unit that surveys a work area where work is to be performed on a workpiece and acquires survey data including information on the work area; and a control unit that identifies the work area based on information on the work area, information related to the work area and associated with the work to be performed on the workpiece, and the survey data, and controls the robot to perform work on the identified work area.

[0010] One aspect of the present invention is a method for controlling a robot system comprising a robot, a survey unit that surveys a work area where work is to be performed on a workpiece, and a control unit. The method includes acquiring, by the survey unit, survey data including information on the work area, and identifying, by the control unit, the work area based on information on the work area, information related to the work area and associated with the work to be performed on the workpiece, and the survey data, and controlling the robot to perform work on the identified work area.

Advantages of the Invention

[0011] According to the present invention, the burden on the operator can be reduced.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Embodiments for Carrying Out the Invention

[0013] <First Embodiment> Hereinafter, the first embodiment for carrying out the present invention will be described with reference to FIGS. 1 to 14.

[0014] [Schematic Configuration of Robot System] First, the schematic configuration of the robot system according to the first embodiment will be described with reference to FIGS. 1, 2, and 3. FIG. 1 is a diagram showing the configuration of the robot system according to the first embodiment. FIG. 2 is a block diagram showing the configuration of the information processing apparatus according to the first embodiment. FIG. 3 is a block diagram showing the configuration of the robot controller according to the first embodiment.

[0015] The robot system 1 is an automatic assembly system that assembles, for example, a component 11 as an assembly work to a work 10 as an object to be assembled, and generally includes a robot device 100 and an information processing apparatus 501. The robot device 100 is fixedly supported on a gantry 13 and includes a robot arm (manipulator) 200 as a robot and a robot controller 201 that controls the robot arm 200.

[0016] In addition, the robot device 100 includes a robot hand 202 which is attached to the tip of the robot arm 200 and is an end effector for gripping (holding) the component 11. The robot hand 202 is not particularly limited in shape and structure as long as it can hold the component 11. For example, it may have a structure for adsorbing the component 11. Further, the robot hand 202 may be provided with a force sensor or the like as necessary.

[0017] Also, a work 10 is placed on a work table 12 installed on the gantry 13, and the robot device 100 includes a camera 300 as a detection unit or an imaging device disposed above the work table 12, that is, in the air above the work 10. The camera 300 captures an image of an imaging area (imaging range) including at least the work 10 and acquires it as image data of the actual image. This camera 300 may be a two-dimensional camera having a function of outputting two-dimensional image data, or may be a three-dimensional camera having a function of outputting three-dimensional image data such as a stereo camera. That is, the camera may be provided in the robot device 100 as a robot. In the present embodiment, the camera 300 is described as a fixed camera installed on the ceiling of a factory or the like, for example. However, as long as it can acquire an image of the imaging area including the work 10, an on-hand camera fixed to the robot hand 202 may also be used. The image data captured by the camera 300 is sent to the robot controller 201 and is processed for information as will be described in detail later. The information processing here refers to the robot controller 201 calculating command values (such as the trajectory of the robot arm) for robot control in order to assemble the component 11 to the work 10.

[0018] The robot system 1 configured as described above performs an assembly operation of assembling the component 11 grasped by the robot hand 202 of the robot device 100 into a hole portion, which is a work area of the work 10 described in detail later. In this way, the robot system 1 performs an assembly operation of assembling the component 11 to the work 10 using the robot device 100, thereby manufacturing the work 10 with the component 11 assembled as an article. In other words, it executes a manufacturing method of manufacturing an article in which the component 11 is assembled to the work 10 using the robot device 100.

[0019] (Configuration of Information Processing Device) Next, the configuration of the information processing device 501 will be described with reference to FIG. 2. As shown in FIG. 2, the information processing device 501 includes a CPU (Central Processing Unit) 502, which is an example of a processor. The CPU 502 is an example of a processing unit. Further, the information processing device 501 includes a ROM (Read Only Memory) 503, a RAM (Random Access Memory) 504, and an HDD (Hard Disk Drive) 505 as storage units. In addition, the information processing device 501 includes a recording disk drive 506, and a display 508, a keyboard 509, and a mouse 510 as input / output interfaces and as display devices. The CPU 502, the ROM 503, the RAM 504, the HDD 505, the recording disk drive 506, the display 508, the keyboard 509, and the mouse 510 are connected to each other communicably via a bus.

[0020] The ROM 503 stores the basic programs related to the operation of the computer. The RAM 504 is a storage device that temporarily stores various data such as the operation processing results of the CPU 502. The HDD 505 records the operation processing results of the CPU 502, various data acquired from the outside, and a program 507 for executing various processes described later. The program 507 is application software that enables the CPU 502 to perform various processes related to the preparatory process (Figure 4) described later. Therefore, the CPU 502 can execute various processes of the preparatory process described later by executing the program 507 recorded in the HDD 505. In addition, the HDD 505 is provided with an area for recording the learning model information 520 as model information obtained from the execution results of various processes of the preparatory process described later. The recording disk drive 506 can read various data and programs recorded on the recording disk 150.

[0021] Note that in this embodiment, the non-temporary recording medium readable by the computer is the HDD 505 and the program 507 is recorded in the HDD 505, but it is not limited thereto. The program 507 may be recorded on any recording medium as long as it is a non-temporary recording medium readable by the computer. As the recording medium for supplying the program 507 to the computer, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a magnetic tape, a non-volatile memory, etc. can be used.

[0022] In addition, a robot controller 201 is connected to the information processing device 501. As will be described in detail later, the information processing device 501 transmits the learning model information 520 to the robot controller 201 as the processing result of executing various processes of the preparatory process.

[0023] (Configuration of Robot Controller) Next, the configuration of the robot controller 201 will be described with reference to FIG. 3. As shown in FIG. 3, the robot controller 201 includes a CPU 204 which is an example of a processor. The CPU 204 is an example of a control unit. Also, the robot controller 201 includes a ROM 205, a RAM 206, and an HDD 207 as storage units. Further, the robot controller 201 includes a recording disk drive 208 and an interface 209 which is an input / output interface. The CPU 204, the ROM 205, the RAM 206, the HDD 207, the recording disk drive 208, and the interface 209 are connected to each other via a bus so as to be communicable with each other.

[0024] The ROM 205 stores a basic program related to the operation of the computer. The RAM 205 is a storage device that temporarily stores various data such as the arithmetic processing results of the CPU 204. The HDD 207 records the arithmetic processing results of the CPU 204 and various data acquired from the outside, and also stores a program 210 for causing the CPU 204 to execute various processes related to the actual machine processing (see FIG. 4) described later. The program 210 is application software that enables the CPU 204 to perform various processes of the actual machine processing described later. Therefore, the CPU 204 can execute control processing by executing the program 210 recorded in the HDD 207 and control the operation of the robot arm 200. Also, the HDD 207 is provided with an area for recording the learning model information 520 acquired by being transmitted from the information processing device 501. The recording disk drive 208 can read out various data, programs, etc. recorded on the recording disk 250.

[0025] In this embodiment, the non-temporary recording medium readable by a computer is the HDD 207, and the program 210 is recorded on the HDD 207, but the present invention is not limited thereto. The program 210 may be recorded on any recording medium as long as it is a non-temporary recording medium readable by a computer. As a recording medium for supplying the program 210 to a computer, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a magnetic tape, a non-volatile memory, etc. can be used.

[0026] Also, a camera 300, a robot arm 200, and the above-described information processing apparatus 501 are connected to the robot controller 201. As will be described in detail later, learning model information 520 as a processing result of executing various processes of pre-preparation processing is transmitted from the information processing apparatus 501 to the robot controller 201. Further, the camera 300 transmits the captured image data to the robot controller 201, and the image data is processed by the program 107. The processing result is output as a command value for robot control and transmitted to the robot arm 200.

[0027] Note that in this embodiment, the pre-preparation processing (see FIG. 4) is processed by the information processing apparatus 501 (CPU 502), and the actual machine processing (see FIG. 4) is processed and executed by the robot controller 201 (CPU 204), but the present invention is not limited thereto. These processes may be executed by one computer, that is, one CPU, or may be executed by three or more computers, that is, three or more CPUs. Further, when each process of the pre-preparation processing and the actual machine processing (see FIG. 4) is shared and processed by a plurality of computers, it does not matter which process is executed by which computer.

[0028] [Steps of the assembly work] Next, in the robot system 1 described above, the steps of the robot assembly work (control of the robot system) for realizing the assembly work as shown in FIG. 1 will be described with reference to FIGS. 4 to 14.

[0029] (Preparatory Process) First, the preparatory process executed by the above-described information processing apparatus 501 will be described with reference to FIGS. 4 to 11. FIG. 4 is a flowchart showing the steps of the assembly work by the robot system according to the first embodiment. FIG. 5 is a perspective view showing an example of the CAD model of the workpiece. FIG. 6 is a perspective view showing an example of the labeling portion of the workpiece. FIG. 7 is a perspective view showing an example of the label model obtained by modeling the labeling portion. FIG. 8 is an explanatory diagram for explaining an example of the label model information to which assembly information is added. FIG. 9 is a diagram showing a state in which the CAD model is imaged by a virtual camera in the virtual space. FIG. 10(a) is a diagram showing an example of the virtual CAD model image obtained by imaging the CAD model with the virtual camera. FIG. 10(b) is a diagram showing an example of the virtual region model image obtained by imaging the level model with the virtual camera. FIG. 11 is an explanatory diagram for explaining the learning process of learning the image features of the labeling portion.

[0030] As shown in FIG. 4, the preparatory process from step S101 to step S103 is a preparatory process before actually operating the robot device 100 (performing the actual assembly work). This preparatory process is a process of generating the learning model information 520 using a computer-aided design tool such as a CAD (Computer Aided Design) system. That is, as will be described in detail below, this preparatory process generates the learning model information 520 using the CAD data of the workpiece 10, which is the design information when the workpiece 10 is designed.

[0031] First, in step S101, the CPU 502 performs an operation called labeling (hereinafter referred to as "labeling operation") on the CAD model of the workpiece 10 on a computer-aided design support tool such as a CAD system. Here, the CAD model is a form of data representation on a computer-aided design support tool, and generally, formats such as STEP, IGES, and STL files are known. In the labeling operation, for the CAD model 20 corresponding to the workpiece 10 shown in FIG. 5, the working area where the assembling operation of assembling the component 11 in contact is performed is specified, and labeling is performed there. The label is something like an identification number indicating that it is the working area corresponding to the part where the component 11 is assembled on the CAD model 20. For example, data such as label number = 1 is added to the labeled working area. By processing in this way, as shown in FIG. 6, a labeling part 21 with a label attached to the working area is created. In this process, if there are multiple working areas where other components or workpieces come into contact with the workpiece 10, labels can be attached to the multiple working areas to provide multiple labeling parts, and these labeling parts can be distinguished, for example, as label numbers = 2, 3,....

[0032] Next, in step S102, the CPU 502 models the labeling part 21. The modeling of the labeling part 21 is, as shown in FIG. 7, modeling the working area labeled in step S101, for example, the labeling part 21 with data such as label number = 1 added, as a new three-dimensional model, the CAD model 22. Such a CAD model 22 can be expressed using the above-mentioned formats such as STEP, IGES, and STL, and constitutes information regarding the working area in the present embodiment.

[0033] Here, in the CAD model 22 of this labeled work area, as shown in FIG. 8, assembly information, which is work information regarding the work to be performed on the work area, is given (i.e., associated and stored in the HDD 505) as the work information regarding the assembly work. In order to give assembly information to the CAD model 22, first, the CAD model 22 is made to hold a reference coordinate system O at an arbitrary position. The coordinate system O has orthogonal axes x, y, z for expressing a three-dimensional Euclidean space and also has Rx, Ry, Rz corresponding to the rotation components around their respective axes. Thereby, in the six-axis coordinate system of the robot device 100, the position and orientation in the three-dimensional space can be expressed. Here, although the orthogonal coordinate space has been described as an example, other forms such as polar coordinate form or quaternion may be used as long as the coordinate system O can express the position and orientation in the three-dimensional space.

[0034] Next, table data TB showing assembly information as work information as shown in the table of FIG. 8 is created. The table data TB includes various information regarding the assembly work (operations, functions) such as information indicating the assembly direction at the time of assembly and the assembly stroke amount from the contact point until the assembly is completed. The table data TB also includes information such as the assembly phase angle indicating the assembly orientation, the insertion start position and the insertion completion position at the time of assembly, and the insertion force required for assembly. In this way, the CAD model 22 and the table data TB are held so as to correspond one-to-one (i.e., associated and stored in the HDD 505). If there are a plurality of labeled work areas on the workpiece 10 as described above, a plurality of CAD models corresponding thereto may be generated, and table data TB may be provided for each of those CAD models so as to correspond one-to-one.

[0035] Subsequently, in step S103, the CPU 502 performs learning of the image features of the labeling unit. In this step S103, as shown in FIG. 9, on the virtual space in a computer-aided design tool such as a CAD system, a virtual camera 301 corresponding to the camera 300 shown in FIG. 1 is constructed and prepared. Then, a process of imaging the three-dimensional CAD model 20 corresponding to the workpiece 10 as seen from the virtual camera 301 is performed.

[0036] Here, it is preferable to set the virtual camera 301 to the same settings as the camera 300 actually used in the robot system 1. For example, the settings such as the cell size of the image sensor, the number of pixels, the focal length regarding the lens, and the aperture correspond to this. By doing so, on the computer-aided design tool, an image (hereinafter referred to as "virtual image") 32 captured in the virtual space as shown in FIG. 10(a) can be obtained by the virtual camera 301. In the virtual image 32, since it is difficult to obtain the same shading and texture information of the workpiece 10 as in the actual shooting environment, these information are not necessarily required. The minimum necessary information is the information regarding the ridge lines indicating the shape of the workpiece 10. It is desirable that these ridge line information coincide with the ridge lines of the CAD model 20.

[0037] Next, on the virtual space of the computer-aided design tool, a three-dimensional CAD model 22 of the work area labeled so that the positional relationship matches the labeling unit 21 in the CAD model 20 of the workpiece 10 is arranged. Then, the CAD model 22 of the labeling unit 21 is imaged by the virtual camera 301 to obtain a virtual image 33 as shown in FIG. 10(b). This virtual image 33 becomes image data from which the ridge line information of the CAD model 22 of the labeling unit 21 can be obtained. And the image data of the virtual image 32 shown in FIG. 10(a) and the virtual image 33 shown in FIG. 10(b) are held so as to be paired (that is, associated and stored in the HDD 505).

[0038] Note that at least one pair of this image data is required to be acquired for the learning described later. However, the more virtual images with different imaging angles, brightness, etc., the better. To obtain a plurality of images, on the virtual space of the computer-aided design tool, the brightness of the virtual image or the texture of the workpiece 10 may be changed within a range where the ridge line information is not lost. Also, within the assumed range, the relative position between the virtual camera 301 and the CAD model 20 of the workpiece 10 or the CAD model 22 of the labeling unit 21 may be moved for imaging. The assumed range referred to here means the range of deviation that can occur in the relative positional relationship between the camera 300 and the workpiece 10 in the actual robot system 1.

[0039] Pairs of image data of a plurality of virtual images obtained by the above processing are used for the learning process as shown in FIG. 11. For example, learning is performed while associating the above table data TB (see FIG. 8) for each pair of virtual images with different imaging angles, brightness, etc. In this embodiment, in the learning process, a machine learning algorithm is used, and among machine learning, in particular, the "supervised learning" algorithm is used. Therefore, the image data of the plurality of virtual images obtained previously becomes the teacher data D1 used for "supervised learning". In the teacher data D1, a virtual image in which the CAD model 20 of the workpiece 10 is imaged is used as the input data D1A, and a virtual image in which the CAD model 22 of the labeling unit 21 is imaged is used as the output data D1B, and machine learning is performed so as to associate them. Thereby, learning model information 520 as a learned model is generated (generation step).

[0040] The machine learning algorithms used in this step S103 are, for example, semantic segmentation, instance segmentation, etc. These are a type of "supervised learning", and are algorithms that infer output values for each pixel of the input data D1A by performing machine learning based on the teacher data D1. If the learning progresses well, it becomes possible to obtain the ridge line information of the CAD model 22 of the labeling unit 21 from the input data D1A. Note that the algorithms used for machine learning are not limited to semantic segmentation and instance segmentation, and any other algorithms may be used as long as they have the function of extracting the above-described features. As described above, in step S103, the learned model information 520 obtained by learning is stored in, for example, the HDD 505 as the storage unit of the information processing apparatus 501. Then, this learned model information 520 is output in a form transferred to, for example, the HDD 207 as the storage unit of the robot controller 201, and is used for the actual machine processing described below.

[0041] (Actual Machine Processing) Next, the actual machine processing executed by the robot controller 201 will be described with reference to FIGS. 4, 12, 13, and 14. FIG. 12 is an explanatory diagram for explaining the inference process of inferring the labeling unit from the actual image data. FIG. 13 is an explanatory diagram for explaining the process of calculating the position of the labeling unit in the camera coordinate system (an example of the coordinate system at a predetermined part of the robot). FIG. 14 is an explanatory diagram for explaining the process of calculating the position of the labeling unit in the robot coordinate system.

[0042] As shown in FIG. 4, the actual machine processing of steps S104 to S106 is actual machine processing for operating the robot device 100 (performing actual assembly work). First, in step S104, the CPU 204 infers the labeling unit 21 from the actual image. Specifically, with the workpiece 10 placed on the workpiece table 12 (see FIG. 1), the camera 300 images the area (imaging range) including the workpiece 10. Here, the camera 300 functions as a search unit that searches for the work area where the work is performed on the workpiece 10 and acquires search data including information on the work area. The image data of the captured actual image as the search data is transferred to the robot controller 201, and inference processing by machine learning as shown in FIG. 12 is performed. When imaging the workpiece 10 on the workpiece table 12 with the camera 300, the camera 300 is moved to a position above the workpiece 10, and the posture of the camera 300 is controlled by the robot arm 200 so that the imaging direction of the camera 300 faces the workpiece 10. The trajectory of the robot arm 200 at this time is, for example, a trajectory previously taught by an operator using a teaching pendant or the like for the position and posture of the robot arm 200.

[0043] The input data shown in FIG. 12 is the image data of the actual image captured by the camera 300. The CPU 204 reads the above-described learned model information 520 and infers the output data by using the same machine learning algorithm as during learning. If it is the learned model information 520 of a well-trained learned model, the output data should be image data (hereinafter referred to as "inference image") corresponding to the ridge line information of the CAD model 22 of the labeling unit 21 on the workpiece 10.

[0044] In step S105, the CPU 204 performs model matching processing on the inference image obtained in step S104. For this matching processing, matching is performed using the CAD model 22 of the labeling unit 21 created in step S102. As a result, the position and orientation of the work area of the work 10 in the image data of the real image substantially captured by the camera 300 are specified. That is, in step S105, the work area is specified based on the learning model information 520, the image data, and the CAD model 22 of the labeling unit 21 (specification step).

[0045] Note that the image used for the matching process in step S105 may be the inference image obtained in step S104 as it is. However, for example, a portion corresponding to the region (matching region) obtained by inference from the captured image data is extracted, and the region other than that region is used as a mask region for mask processing, and matching processing may be performed on the image data obtained by mask processing the mask region. In short, the learning model information 520 and the image data obtained by mask processing the mask region may be subjected to matching processing, whereby the load of image processing can be reduced.

[0046] If the matching is successfully completed, as shown in FIG. 13, in combination with a known camera calibration technique, a vector Vc is obtained as the position information of the CAD model 22 of the labeling unit 21 in the three-dimensional space as seen from the camera w Note that Vc w represents a vector from the origin 319 of the camera coordinate system to the origin 23 of the coordinate system О serving as the reference of the CAD model of an arbitrary labeling unit.

[0047] In step S106, the CPU 204 performs a process of generating a trajectory for assembling the work 10 while the robot arm 200 holds (grips) the component 11. Specifically, first, as shown in FIG. 14, a vector Vc from the origin 220 of the coordinate system of the robot arm 200 to the origin 319 of the camera coordinate system rIt is obtained by using a known hand-eye calibration technique. Also, since the robot arm 200 holds the component 11 via the robot hand 202, the vector from the origin 220 of the coordinate system of the robot arm 200 to the reference position 24 of any component 11 is the vector Vr t × vector Vt w′ and is obtained thereby.

[0048] Here, the vector Vr t represents the vector from the origin 220 of the coordinate system of the robot arm 200 to the origin 221 of the coordinate system of the robot hand 202. The method for obtaining this vector Vr t may use the values of the encoders that detect the angles of the respective joints for the robot arm 200 to calculate its own position, or may perform position measurement from an image captured by, for example, a camera from the outside, and various known methods can be used. Also, the vector Vt w′ represents the vector from the origin 221 of the coordinate system of the robot hand 202 to the reference position 24 of any component 11. For the method of obtaining this vector Vt w′ too, various known methods such as a method of measuring from the outside and a method of mechanically positioning can be used.

[0049] In this way, an orbit for moving the component 11 to the labeling part 21 of the workpiece 10 by the robot arm 200, that is, an orbit Vw w′ until starting to assemble the component 11 to the workpiece 10 can be generated. Also, the orbit Vw w′ is not limited to a straight-line orbit, and as long as the start point and the end point coincide, arbitrary complementary processing such as spline interpolation may be added to the intermediate path, and the orbit can be freely determined.

[0050] Also, the CPU 204 reads the table data TB (see FIG. 8) indicating the assembly information included in the learning model information 520, and generates a trajectory from the position where the assembly of the component 11 into the workpiece 10 starts (insertion start) to the position where the assembly is completed (insertion completion). That is, the table data TB includes information such as the assembly direction, assembly stroke, assembly transfer angle, insertion start position, insertion completion position, insertion force, etc. as information regarding the work to be performed on the workpiece. Therefore, a trajectory for assembling the component 11 into the workpiece 10 is generated from these information starting from the assembly start position. And the trajectory Vw until the start of assembling the component 11 into the workpiece 10 obtained as described above w′ is added to the trajectory for assembling the component 11 into the workpiece 10, and thus the generation of the trajectory for controlling the robot arm 200 in the assembly operation is completed. Note that the information regarding the work to be performed on the workpiece only needs to include at least one of the component assembly direction, the component assembly stroke amount, the component assembly phase angle, the component insertion start position, the component insertion completion position, and the component insertion force.

[0051] When the trajectory for controlling the robot arm 200 in the assembly operation is generated in step S106 in this way, the CPU 204 outputs the trajectory as a command value to the robot arm 200 and drives the robot arm 200 to operate along the trajectory. As a result, based on the learning model information 520 (assembly information (see FIG. 8)), the component 11 is moved by the robot arm 200 to the position in the work area of the specified workpiece 10, and is assembled into the workpiece 10 while its position and orientation are controlled. That is, in step S106, the robot arm 200 is controlled to perform work in the work area specified in step S105 (work process).

[0052] [Summary of the First Embodiment] By performing the process of the assembly operation by the robot system 1 shown in FIG. 4 as described above, the component 11 can be automatically assembled by the robot device 100 into the work area of the workpiece 10 without imposing much load on the adjustment work of the operator.

[0053] Specifically, in this embodiment, in the steps of S101 to S102, the work 10 is modeled on the virtual space of the CAD system to generate the CAD model 20. Thereby, for example, unlike the prior art where the work 10 is imaged multiple times by the camera 300 while changing the imaging angle to generate a number of template images, or the adjustment work of the image processing process such as degree correction and feature extraction can be significantly reduced. Therefore, the burden of the adjustment work by the operator can be reduced.

[0054] Also, in this embodiment, not only the CAD model 20 obtained by modeling the entire work 10, but also the CAD model 22 of the labeling part 21 labeled as the work area is configured to be generated. Thereby, in the model matching process in S105, the amount of calculation can be significantly reduced compared to the case of performing the matching process with the CAD model 20 which is the entire work 10, and the speed can be increased. Further, in this model matching process, instead of matching the template image of the two-dimensional image with the real image, the CAD model 22 which is a three-dimensional model is matched with the real image. Thereby, it becomes possible to grasp the position and orientation of the work 10 in three dimensions from the learned model information 520.

[0055] Also, in this embodiment, in the step of S103, the image features of the labeling part 21 are configured to be learned. Thereby, for example, the number of times the CAD model 20 is imaged by the virtual camera 301 on the virtual space can be reduced compared to the case of preparing a number of template images. Therefore, the load of the preprocessing can be reduced, and the burden of the adjustment work by the operator can be reduced. Further, by generating the learned model information 520 which is the information of the learned model, the accuracy of the inference and model matching of the labeling part 21 in S104 to S105 can be improved.

[0056] Furthermore, in the present embodiment, the learning model information 520 is configured to store the table data TB as the assembly information obtained from the CAD data (design information) in association with the CAD model 22 of the labeling unit 21. Thereby, when generating the trajectory of the robot arm 200 in step S106, the table data TB associated with the CAD model 22 matched with the real image can be extracted, and the trajectory of the robot arm 200 with high accuracy can be automatically generated. Therefore, the operation of the assembly work by the robot device 100 can be accurately executed. Also, by using the assembly information obtained from the CAD data (design information) in this way, it becomes unnecessary for the operator to generate and prepare a large number of corresponding trajectories for each angle of the workpiece in advance, and the burden of the adjustment work by the operator can be reduced.

[0057] As described above, by performing the process of the assembly work by the robot system 1 according to the present embodiment, the burden of the adjustment work as the prior preparation can be reduced, and an automatic production system that performs the assembly work using the robot device 100 can be started up in a short time.

[0058] In the first embodiment, the CAD model 22 of the labeling unit 21 is generated and matched with the real image by the model matching process. However, the present invention is not limited to this, and the CAD model 20 of the workpiece 10 may be generated and matched with the real image.

[0059] Also, in the first embodiment, in order to enable identification of the modeled labeling unit 21 from the real image, learning using a machine learning algorithm is performed in steps S103 and S104. However, the present invention is not limited to this, and it may be configured to be identifiable using a method other than learning.

[0060] <Second Embodiment> Next, a second embodiment in which the first embodiment is partially modified will be described with reference to FIGS. 15 to 17. FIG. 15 is a diagram showing the configuration of a robot system according to the second embodiment. FIG. 16 is a flowchart showing the steps of an assembly operation by the robot device according to the second embodiment. FIG. 17 is an explanatory diagram for explaining the process of generating a three-dimensional point cloud image. In the description of the second embodiment, the same reference numerals are used for the same parts as in the first embodiment, and the description thereof is omitted.

[0061] [Configuration of Robot System According to Second Embodiment] In the robot system 1 according to the second embodiment, as shown in FIG. 15, in addition to a camera (hereinafter referred to as "first camera") 300, a second camera 320 as a search unit or an imaging unit is provided. Then, the first camera 300 and the second camera 320 constitute a stereo camera to perform three-dimensional measurement of the actual workpiece 10. In the robot system 1 shown in FIG. 15, an example is shown in which these first camera 300 and second camera 320 are fixed cameras, but it may also be an on-hand camera mounted on the robot hand 202. Further, the number of cameras configured as the imaging unit is not limited to two, and may be three or more. For example, it may be a stereo camera having both a fixed camera and an on-hand camera, and one or both of them are composed of two cameras.

[0062] [Steps of Assembly Operation According to Second Embodiment] Next, the process of the assembly work by the robot system 1 according to the second embodiment will be described. As shown in FIG. 16, steps S201, S202, and S203, which are preliminary preparation processes before actually operating the robot device 100, are the same as steps S101, S102, and S103 shown in FIG. 4 described above. However, in the learning in step S203, learning model information 520 may be generated from virtual images obtained by imaging the CAD model 22 in the respective virtual arrangements of the first camera 300 and the second camera 320. Also, common learning model information 520 may be generated. When generating common learning model information 520, the input data D1A will include image data obtained by imaging the CAD model 22 in the virtual arrangements of both the first camera 300 and the second camera 320.

[0063] Next, the actual machine processing in the process of the assembly work according to the second embodiment will be described. In this second embodiment, when the actual machine processing is started, first, in steps S204-1 and S204-2, the CPU 204 performs imaging with the first camera 300 and the second camera 320. Then, regarding the image data captured by the respective first camera 300 and second camera 320, inference by the labeling unit 21 is performed. At this time, as described above, the learning model information 520 used may be individual or common. In any case, the labeling unit 21 infers from the image data captured respectively.

[0064] Next, in step S205, the CPU 204 performs three-dimensional measurement. The three-dimensional measurement is made possible by performing known stereo calibration between the first camera 300 and the second camera 320 based on the principle of triangulation. That is, as shown in FIG. 17, based on the inference image 330 of the first camera 300 and the inference image 331 of the second camera 320, it is possible to obtain a three-dimensional point cloud image 340 by performing three-dimensional measurement using a known method such as block matching. Then, in step S206, model matching processing with the labeling unit 21 is performed on the three-dimensional point cloud image 340. Thereby, the position and orientation of the labeling unit 21 and the assembly information (refer to the table data TB in FIG. 8) can be obtained. After obtaining the position and orientation of the labeling unit 21 and the assembly information, the process of trajectory generation in step S207 is the same as the process of step S106 described above, and thereby the trajectory of the robot arm 200 can be generated.

[0065] Note that, when performing three-dimensional measurement in step S205, the inference image obtained in step S204 may be used as it is. However, for example, a portion corresponding to a region (matching region) obtained by inference from the captured image data is extracted, and a region other than that region is set as a mask region for mask processing, and three-dimensional measurement may be performed on the image data obtained by mask-processing the mask region. That is, a three-dimensional point cloud image 340 may be obtained from the image data obtained by mask-processing the mask region, and model matching processing with the labeling unit 21 may be performed. In short, in a broad sense, by performing matching processing on the learning model information 520 and the image data obtained by mask-processing the mask region, the load of image processing can be reduced.

[0066] Also, in the inference image 330 of the camera 300 and the inference image 331 of the camera 320, preprocessing such as removal of noise generated by inference, approximation of a straight line, and approximation of an ellipse may be performed before performing the matching processing.

[0067] [Summary of the Second Embodiment] As described above, in the process of the assembly work by the robot system 1 according to the second embodiment, the inference images 330 and 331 are acquired by the first camera 300 and the second camera 320 constituting the stereo camera. Then, by generating the three-dimensional point cloud image 340 from these inference images and performing the model matching process, the working area in the actual workpiece 10 can be accurately specified.

[0068] In addition, in this second embodiment, the other configurations, operations, and effects are the same as those in the first embodiment described above, and thus the description thereof is omitted.

[0069] <Third Embodiment> Next, a third embodiment in which the above first and second embodiments are partially modified will be described with reference to FIGS. 18 to 19. FIG. 18 is an explanatory diagram for explaining a process of defining a solid angle for a workpiece according to the third embodiment. FIG. 19 is an explanatory diagram for explaining a learning process of associating the image features of the labeling unit according to the third embodiment with the solid angle of the workpiece. In the description of this third embodiment as well, the same reference numerals are used for the same parts as those in the first and second embodiments described above, and the description thereof is omitted.

[0070] In this third embodiment, when learning the image features of the labeling unit 21 in step S103, in addition to the position of the ridge line in the CAD model 20 of the workpiece 10, the attitude information is also learned. Specifically, as shown in FIG. 18, on the virtual space, for an arbitrary reference vector VA provided in the CAD model 20, the direction in which an arbitrary normal line NL in the CAD model 20 is directed is defined by a solid angle α as the attitude information. That is, in a plurality of CAD models 20 with different imaging angles captured by the virtual camera 301 on the virtual space, the attitude is defined by the solid angle α, and it is included in the learning model information 520 as the attitude information associated with each CAD model 20.

[0071] That is, as shown in FIG. 19, learning is performed to associate the value of the solid angle α with the output data DIB. To perform such learning, if the information (posture information) of the solid angle α is tagged to the output data D1B, it is possible to perform learning using an algorithm such as the aforementioned instance segmentation. Here, if the learning progresses well, when the labeling unit 21 is inferred in step S104, the solid angle α is also inferred. Therefore, in the next step S105, when performing the model matching process, the risk of incorrect matching can be reduced, for example, by narrowing down the angle for matching.

[0072] In addition, the other configurations, operations, and effects in the third embodiment are the same as those in the first and second embodiments described above, and thus the description thereof is omitted.

[0073] <Fourth Embodiment> Next, a fourth embodiment in which the first to third embodiments are partially modified will be described. In the description of the fourth embodiment as well, the same reference numerals are used for the same parts as in the first to third embodiments, and the description thereof is omitted.

[0074] In the fourth embodiment, when generating the trajectory of the robot arm 200 in step S106, modeling is performed for components other than the labeling unit 21 so that a trajectory can be generated without interference between the robot arm 200 and the workpiece 10 itself or surrounding objects other than the workpiece 10. That is, for example, in step S102, from the CAD data, not only the labeling unit 21 (working area) which is the assembling part of the component 11, but also the workpiece 10 and surrounding objects arranged around the workpiece 10 are modeled to generate a surrounding model. Then, surrounding model information which is information regarding the surrounding model is generated. Note that for models other than the labeling unit 21, it is not necessary to associate table data TB as shown in FIG. 8. Other than this, the same procedure as the above-described assembling work process (see FIG. 4) can be performed. In this way, a surrounding model of surrounding objects other than the labeling unit 21 is generated, model matching processing is performed, a trajectory for avoiding interference with the surrounding objects (surrounding model) is determined, and the trajectory of the robot arm 200 is generated from the determined trajectory. Thereby, when generating the trajectory of the robot arm 200, a trajectory can be generated such that there is no interference between the robot arm 200 and the workpiece 10 or the surrounding objects of the workpiece 10.

[0075] Note that the other configurations, operations, and effects in the fourth embodiment are the same as those in the first to third embodiments, and thus the description thereof is omitted.

[0076] <Fifth Embodiment> Next, a fifth embodiment in which the first to fourth embodiments are partially modified will be described with reference to FIG. 20. FIG. 20 is a diagram showing an example of a GUI indicating the result of model matching according to the fifth embodiment. In the description of the fifth embodiment as well, the same reference numerals are used for the same parts as in the first to fourth embodiments, and the description thereof is omitted.

[0077] In the fifth embodiment, a GUI (Graphical User Interface) 130 that shows the user the progress of the assembly process shown in FIG. 4 is displayed on a display device such as a display 508 connected to, for example, an information processing apparatus 501. Regarding the process of generating this GUI 130, in this embodiment, the CPU 204 of the robot controller 201 performs the process, and an example will be described in which the process is transferred to the information processing apparatus 501 and displayed on the display 508. However, the present invention is not limited to this, and the GUI 130 may be generated by the CPU 204 and transferred and displayed on a display device directly connected to the robot controller 201, for example. Further, various data calculated by the CPU 204 of the robot controller 201 may be transmitted to the information processing apparatus 501, and the CPU 502 may generate the GUI 130 and display it on the display 508.

[0078] An example of the GUI 130 will be described with reference to FIG. 20. The GUI 130 has a main window 131 that displays the captured image and inference result captured in step S104 and the matching result in step S105. Further, the label number labeled in step S101 can be confirmed in the label information window 132, and the processing result corresponding to the label number selected here is displayed in the main window 131. Further, when the model matching process in step S105 is successfully completed, the reference coordinates of the CAD model 22 of the labeling unit 21 as shown in FIG. 8 are displayed as the detected coordinates in the label information window 132. Further, in the detection result window 134, characters such as "OK" indicating successful detection or characters such as "NG" in the case of detection failure are displayed. Further, in the assembly information window 133, table data TB indicating the assembly information corresponding to the label number of the labeling unit 21 as shown in FIG. 8 is displayed. By displaying the GUI 130 as described above, the user can determine whether the detection process of the workpiece 10 and its working area has been successfully performed.

[0079] In addition, since the other configurations, operations, and effects in the fifth embodiment are the same as those in the first to fourth embodiments described above, the description thereof will be omitted.

[0080] <Sixth Embodiment> Next, the sixth embodiment will be described with reference to FIGS. 21(a) to 22. FIGS. 21(a) and (b) are diagrams showing the position and orientation information of the CAD model 22 corresponding to the labeling unit 21 in the actual work 10 according to the sixth embodiment. FIG. 22 is a diagram showing the robot device 100 according to the sixth embodiment. In the description of the sixth embodiment, the same reference numerals are used for the same parts as in the above various embodiments, and the description thereof will be omitted. In this embodiment, the operation of the assembly work by the robot device 100 is performed without using the assembly information such as the table data TB described in the above embodiments as table information. The control flow of this embodiment basically operates according to the processing flow of FIG. 4, and the method for generating the correction trajectory in step S106 of FIG. 4 is different.

[0081] Execute the model matching in step S105 of the processing flow shown in FIG. 4, and as shown in FIG. 21, obtain the position and orientation information of the CAD model 22 corresponding to the labeling unit 21 in the actual work 10. This position and orientation information is obtained based on the coordinate system О. FIG. 21(a) shows the position and orientation information of the CAD model 22 corresponding to the labeling unit 21 in the actual work 10 based on the coordinate system О. FIG. 21(b) shows the work 10. Based on this information, the robot device 100 is moved to the assembly position. In this case, the method for generating the correction trajectory and the coordinate transformation method for moving the robot device 100 are as described with reference to FIG. 14. From here, when not using assembly information such as the table data TB, it is necessary to determine in advance the assembly direction, but for the assembly stroke, phase angle, insertion start position, and insertion completion position, they are detected by the force sensor 203 attached to the robot hand 202 as shown in FIG. 22.

[0082] The force sensor 203 can, for example, detect external forces and moments applied to the force sensor 203 from the outside in six axial directions, and it is assumed that the robot 200 can be instructed to perform a straight movement until a predetermined force is detected in each axial direction. Also, in some cases, known techniques such as admittance control and impedance control can be used based on the measured external force until the robot arm 200 completes the assembly operation.

[0083] Regarding the assembly direction, it is determined in advance in the program to operate the robot so as to move in a predetermined direction based on the position and orientation information of the acquired CAD model 22. For example, when the position and orientation information as shown in FIG. 21 is acquired based on the actual workpiece 10, as the predetermined direction, it is moved in the Z-axis vertical direction in the coordinate system О of the CAD model 22, that is, in the -Z-axis direction in the coordinate system О of the CAD model 22. The position and orientation information in FIG. 21 is assumed to be the same information as the position and orientation information shown in FIG. 8. The coordinate system О in FIG. 21 has a different notation from FIG. 8 for convenience of explanation. According to step S105 shown in FIG. 4, the position and orientation information based on the coordinate system О as shown in FIG. 21 can be acquired based on the actual workpiece 10, so the predetermined direction based on the coordinate system О is set in advance in the program as the assembly direction. By doing so, even if the position and orientation of the actual workpiece 10 as the target vary, the position and orientation information based on the coordinate system О of the CAD model 22 corresponding to the varying state can be acquired, so the robot can be moved in the direction in which the component 11 is assembled to the workpiece 10.

[0084] While moving the component 11 in the assembling direction, information regarding the force is acquired, and a predetermined value is set as a threshold value that can determine the completion of assembly. By detecting that the information regarding the force reaches the predetermined value, the completion of assembly can be detected, and the assembling operation can be completed. Further, when the accuracy of the clearance between the workpiece 10 and the component 11 is high, in order to align the phases of the workpiece 10 and the component 11, the component 11 moved by the robot arm 200 may be made to perform a probing operation while being in contact with the workpiece 10. The force information during the probing operation is detected, and when the force value reaches the predetermined value, it is determined that the phases are aligned, and by moving the component 11 in the assembling direction, it becomes possible to assemble workpieces with high clearance accuracy.

[0085] According to the present embodiment as described above, without using the table data TB, by acquiring the position and orientation information based on the coordinate system О of the actual workpiece 10, it becomes possible to assemble the component 11 to the workpiece 10. Thereby, in the assembling process using the robot device 100, the number of parameters to be set in advance can be reduced, and the burden of preliminary preparation can be further reduced. Further, an automatic production system that performs an assembling operation using the robot device 100 can be started up in a short time.

[0086] <Seventh Embodiment> Next, the seventh embodiment will be described with reference to FIGS. 23 to 25. FIG. 23 is a control block diagram for explaining control by the visual servo according to the seventh embodiment. FIGS. 24(a) and (b) are diagrams for explaining a method of generating image data corresponding to the target feature amount according to the seventh embodiment. FIG. 25 is a flowchart showing the steps of the assembling operation by the robot device according to the seventh embodiment. In the description of the seventh embodiment, the same reference numerals are used for the same parts as in the above various embodiments, and the description thereof is omitted. Further, in this embodiment, the operation of the assembling work by the robot device 100 is performed without using the assembling information such as the table data TB described in the above embodiments as table information. The control flow of this embodiment basically operates according to the processing flow of FIG. 4, and the method of generating the correction trajectory in step S106 of FIG. 4 is different. In this embodiment, the robot device 100 is controlled using a visual servo.

[0087] FIG. 23 is a control block diagram showing the outline of the control according to this embodiment. FIG. 23 illustrates the basic control block of a known visual servo. In this embodiment, as the target feature amount, the image data obtained by imaging the CAD model 22 of the labeling unit 21 by the virtual camera 301 in the virtual space is input. The method of generating the image data corresponding to this target feature amount will be described with reference to FIG. 24.

[0088] As shown in FIG. 24, in the case of using visual servo, it is assumed that the camera 300 is in the state of a moving camera mounted on the on-hand of the robot device 100. FIG. 24(a) shows a state in which such a configuration is represented in the virtual space. FIG. 24(a) shows a diagram in which a virtual camera 301 is arranged in the virtual space. That is, it shows a state in which the relative positional relationship among the origin 319 of the camera coordinate system, the reference position 24 of the component 11, and the origin 23 of the coordinate system О serving as the reference of the CAD model 22 corresponding to the labeling unit 21 is set as known information. In this state, the component 11 can be assembled to the work 10 based on the known relative positional relationship. In this state, a three-dimensional CAD model 22 of the work area labeled so as to match the positional relationship with the labeling unit 21 in the CAD model 20 showing the entire work 10 is arranged, and the CAD model 22 of the labeling unit 21 is imaged by the virtual camera 301. By doing so, image data showing the CAD model 22 in the virtual space as shown in FIG. 24(b) can be obtained. The image data obtained here becomes the target feature amount shown in FIG. 23.

[0089] Next, when actually operating the robot device 100, it is controlled as shown in the control flowchart of FIG. 25. The control flowchart shown in FIG. 25 is mainly executed by the CPU 502 and the CPU 204. As shown in FIG. 25, the processes of steps S301 to S303 are the same as the processes of steps S101 to S103 in FIG. 4 described above, but the processes after step S304 are different from the flow of FIG. 4.

[0090] In step S304, the CPU 204 captures an image of the workpiece 10 using the actual camera 300. The image data acquired by the actual camera 300 is, in step S305, inferred by the CPU 204 in the labeling unit, and based on the inference result, in step S306, the CPU 204 calculates a control amount. The calculation of the control amount in step S306 corresponds to the operation of calculating the difference between the target feature amount and the current feature amount in FIG. 23, and this result is sent to the feature-based controller shown in FIG. 23 to become the control amount for controlling the robot. Incidentally, as the control algorithm of the feature-based controller, various methods such as image Jacobian matrix calculation are known based on known feature point extraction methods.

[0091] Next, in step S307, the CPU 204 controls the robot device 100 so as to gradually approach the image data that becomes the target feature amount. In step S308, the CPU 204 determines whether or not the difference between the target feature amount and the current feature amount has achieved a predetermined target value (threshold value). This target value may be a predetermined value or may define a range. If it is Yes in step S308, it proceeds to step S309. If it is NO in step S308, it returns to just before step S304 and repeats the control of the robot by visual servo.

[0092] When it becomes YES in step S308, as shown in FIG. 24, the state of the actual robot device 100 becomes a state in which the component 11 can be assembled to the workpiece 10 based on the known information. Therefore, the CPU 204 moves the robot device 100 so that the reference position 24 coincides with the origin 23 based on the relative positional relationship between the origin 319, the reference position 24, and the origin 23, which are the known information. By doing so, the assembly operation is completed.

[0093] In this embodiment, regarding the feature point extraction method, a method based on image difference is shown. In this embodiment, as the target feature quantity, image data obtained by imaging the CAD model 22 of the labeling unit 21 with the virtual camera 301 is input, and as the current feature quantity, the inference result of the labeling unit is input, and the difference between the feature quantities is obtained. However, regarding the image difference calculation method in this case, various difference calculation methods for calculating feature points of interest from images such as SIFT (Scale-Invariant Feature Transformation) and AKAZE (Accelerated KAZE) and associating feature points with high similarity are known. Thus, an algorithm for robot control can be appropriately adopted from at least two or more pieces of image data.

[0094] According to the above embodiment, without using the table data TB, by acquiring the position and orientation information based on the coordinate system О of the actual workpiece 10, it becomes possible to assemble the component 11 to the workpiece 10. Also, in this embodiment, since the relative positional relationship that enables assembly by the operation of the robot device 100 is set in the virtual space, the burden of preliminary preparation can be further reduced. Thereby, an automatic production system that performs an assembly operation using the robot device 100 can be started up in a short time. In this embodiment, the above relative positional relationship is set in the virtual space, but of course, it may be set using the actual robot device 100. In this case, since the actual robot device 100 is used, it becomes possible to improve the accuracy of the assembly operation.

[0095] <Possibility of other embodiments> Note that in the above-described embodiment, one that generates a three-dimensional CAD model in the virtual space based on CAD data has been described, but it is not limited to this, and one that generates a two-dimensional model may also be used.

[0096] Also, in the above-described first to fifth embodiments, although the workpiece 10 and its work area (labeling unit 21) have been described as being modeled as CAD models 20 and CAD models 22 from design information such as CAD data, it is not limited to this. That is, the component 11 may also be modeled from design information such as CAD data and constructed as a CAD model. Thereby, it becomes possible to virtually assemble the CAD model 22 of the labeling unit 21 and the CAD model of the component 11 in a virtual space, and the trajectory of the robot arm 200 may be generated from the position and posture in the virtual assembly. Furthermore, only an object held by the robot arm 200 such as the component 11 (that is, the component 11 may be referred to as a workpiece) may be modeled. In this case, if the workpiece 10 placed on the workpiece table 12 or the like is arranged at a known position and in a known posture, the model of the component 11 can be model-matched and the trajectory can be generated.

[0097] Also, in the above-described embodiments, although the case where the workpiece is imaged by a camera to generate image data of a real image has been described, it is not limited to this. For example, any device that can explore the workpiece in the exploration direction and generate exploration data such as shape data including the work area of the workpiece, such as a tactile sensor, an ultrasonic sensor, a probe, etc., may be used.

[0098] Also, in the above-described embodiments, although the case where the component 11 is assembled to the work area of the workpiece 10 has been described as an example, it is not limited to this. For example, operations such as applying an adhesive, paint, oil, etc. to the work area (application area) of the workpiece may be performed. Also, for example, operations such as attaching components such as labels and seals to the work area (attachment area) of the workpiece may be performed. Also, for example, operations such as bringing tools such as drivers and cutters into contact with the work area (working area) of the workpiece may be performed.

[0099] Also, in the above-described embodiments, when modeling a workpiece or a work area (labeling section), an example was described in which a model is generated in a virtual space using CAD data. However, the present invention is not limited to this. For example, an operator or a designer may manually generate a virtual model such as a polygon model in the virtual space. Further, as design information, not only CAD data but also information in which the position and size of the workpiece are numerically given may be used.

[0100] Also, in the above-described embodiments, an example was described in which the trajectory of the robot arm 200 is generated in step S106 or step S207. However, the present invention is not limited to this. For example, when an operator has previously taught and generated a rough trajectory of the robot arm 200, a correction trajectory for correcting the trajectory created by the teaching may be generated in step S106 or step S207. That is, the trajectory generation in step S106 or step S207 may be either the generation of a new trajectory or the generation of a correction trajectory for correcting an existing trajectory.

[0101] Also, in the above-described embodiments, an example was described in which machine learning is performed on a plurality of images including the CAD model 22 of the labeling section 21 in step S103 or step S203. However, the present invention is not limited to this. That is, a plurality of virtual images of the CAD model 22 generated in step S102 or step S202 under different conditions (imaging angle, brightness, etc.) may be used as template images (target images). In this case, in step S104 or step S204-1, S204-2, the labeling section in the real image may be inferred from the template images, and methods such as template matching may be considered in step S105 or step S206.

[0102] In the above-described embodiment, the robot arm 200 of the robot device 100 has been described by taking as an example a six-axis articulated manipulator, but the present invention is not limited thereto. For example, a parallel link robot or a robot equipped with a mechanism that moves three-dimensionally in parallel may be used, that is, the robot may have any structure. Further, the present invention is applicable to a machine that can automatically perform operations of expansion and contraction, flexion and extension, vertical movement, horizontal movement, or turning, or a combination of these operations based on information stored in a storage device provided in a control device.

[0103] The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and causing one or more processors in a computer of the system or device to read and execute the program. Further, it can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0104] The present invention is not limited to the embodiments described above, and many modifications are possible within the technical idea of the present invention. Further, two or more of the above-described embodiments may be combined and implemented. In addition, the effects described in the embodiments are merely an enumeration of the most suitable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

[0105] <Summary of this embodiment> [Configuration 1] A robot, a search unit that searches for a work area for performing work on a workpiece and acquires search data including information on the work area, a control unit that identifies the work area based on information on the work area, information related to the work to be performed on the workpiece associated with the work area, and the search data, and controls the robot to perform work on the identified work area, characterized in that it is a robot system. [Configuration 2] The information regarding the work area is a model obtained by modeling the work area from the design information when designing the work. The robot system according to Configuration 1, characterized in that. [Configuration 3] The information regarding the work includes at least one of the assembly direction of the part, the assembly stroke amount of the part, the assembly phase angle of the part, the insertion start position of the part, the insertion completion position of the part, and the insertion force of the part. The robot system according to Configuration 1 or 2, characterized in that. [Configuration 4] The information regarding the work is set as table information. The robot system according to Configuration 3, characterized in that. [Configuration 5] The claim exploration data is data obtained from at least one of a camera, a tactile sensor, an ultrasonic sensor, and a probe. The robot system according to any one of Configurations 1 to 4, characterized in that. [Configuration 6] In the model, a coordinate system indicating the position and orientation of the model is set. Based on the model and the information regarding the work area obtained based on the exploration data, information regarding the position and orientation of the work area of the work defined by the coordinate system of the model is obtained. Based on the information regarding the position and orientation, the robot performs work on the work area of the work. The robot system according to any one of Configurations 2 to 5, characterized in that. [Configuration 7] The robot is provided with a sensor for acquiring information regarding force. Based on the information regarding the position and orientation and the information regarding the force applied to the robot, the robot performs work on the work area of the work. The robot system according to any one of Configurations 1 to 6, characterized in that. [Configuration 8] The exploration unit includes an imaging device provided on the robot. Based on the information regarding the position and orientation, the reference position of the imaging device, and the reference position of the component to be moved by the robot, the robot executes work in the work area of the workpiece. The robot system according to any one of Configurations 1 to 7, characterized in that. [Configuration 9] The exploration unit includes an imaging device. Based on the information regarding the position and orientation, the information regarding the work, the coordinate system at a predetermined part of the robot, the coordinate system in the imaging device, and the coordinate system of the robot, the trajectory of the robot for executing work on the workpiece is acquired. The robot system according to any one of Configurations 2 to 8, characterized in that. [Configuration 10] The exploration unit acquires the exploration data by exploring the work area in the exploration direction, acquires learning model information obtained by learning the characteristics of the work area when the model is explored from a plurality of angles, The control unit identifies the work area based on the learning model information and the exploration data. The robot system according to any one of Configurations 1 to 9, characterized in that. [Configuration 11] The exploration unit is an imaging device that images in the imaging direction and acquires image data including the characteristics of the work area. The learning model information is information obtained by learning the characteristics of the work area when the model is imaged from a plurality of angles. The control unit identifies the work area based on the learning model information and the image data acquired by the imaging device. The robot system according to Configuration 10, characterized in that. [Configuration 12] The control unit generates an inference image that infers the work area from the image data acquired by the imaging device based on the learning model information, and identifies the work area in the image data by performing matching processing on the learning model information and the inference image. The robot system according to Configuration 11, characterized by the above. [Configuration 13] The learning model information includes pose information regarding the pose of the model. The control unit identifies a matching area in the image data for matching the learning model information and the image data based on the pose information, and performs matching processing. The robot system according to any one of Configurations 1 to 12, characterized by the above. [Configuration 14] The imaging device has a plurality of cameras capable of three-dimensional measurement. The control unit identifies a matching area in the image data for matching the learning model information and the image data based on the image data respectively acquired by the plurality of cameras, and performs matching processing. The robot system according to Configuration 12, characterized by the above. [Configuration 15] The control unit masks an area other than the matching area in the image data, and performs matching processing on the learning model information and the masked image data. The robot system according to Configuration 13, characterized by the above. [Configuration 16] The learning model information includes work information regarding the work to be performed on the work area. The control unit controls the robot based on the work information. The robot system according to Configuration 10, characterized by the above. [Configuration 17] The work is an assembly work of assembling a part held by the robot in the area of the workpiece. The robot system according to Configuration 16, characterized by the above. [Configuration 18] When the control unit performs work in the work area, it controls the robot based on surrounding model information regarding a surrounding model obtained by modeling surrounding objects arranged around the workpiece and the exploration data. The robot system according to any one of Configurations 1 to 17, characterized in that. [Configuration 19] When the control unit performs work in the work area, it determines a trajectory for avoiding interference between the robot and the surrounding objects based on the surrounding model information and the exploration data, and controls the robot according to the determined trajectory. The robot system according to Configuration 18, characterized in that. [Configuration 20] When the control unit identifies the work area, it displays the identified work area on a display device. The robot system according to any one of Configurations 1 to 19, characterized in that. [Configuration 21] Comprising an information processing apparatus having a processing unit for acquiring the model. The robot system according to any one of Configurations 2 to 20, characterized in that. [Configuration 22] In a control method for a robot system including a robot, a exploration unit that explores a work area where work is performed on a workpiece, and a control unit, The exploration unit acquires exploration data including information on the work area. The control unit identifies the work area based on information regarding the work area, information related to the work area and associated with the work to be performed on the workpiece, and the exploration data, and controls the robot to perform work in the identified work area. A control method for a robot system, characterized in that. [Configuration 23] A method for manufacturing an article, characterized by manufacturing the article using the robot system according to any one of Configurations 1 to 22. [Configuration 24] A program for causing a computer to execute the control method of the robot system according to Configuration 22. [Configuration 25] A computer-readable recording medium storing the program described in Configuration 24.

Explanation of Signs

[0106] 1…Robot system / 10…Workpiece / 21…Labeling unit (working area) / 22…CAD model (model) / 100…Robot device / 200…Robot arm (robot) / 204…CPU (control unit) / 300…Camera (exploration unit, imaging unit) / 320…Camera (exploration unit, imaging unit) / 330…Inference image / 331…Inference image / 501…Information processing device / 502…CPU (processing unit) / 508…Display (display device) / 520…Learning model information (model information) / TB…Table data (working information) / α…Solid angle (posture information)

Claims

1. Robots and A search unit that searches a working area where a work is performed on a workpiece and acquires search data including information on the working area; and a control unit that identifies the working area based on information about the working area, information about an operation to be performed on the workpiece associated with the working area, and the exploration data, and controls the robot to perform the operation in the identified working area. A robot system comprising:

2. The information about the working area is a model obtained by modeling the working area from design information when the workpiece is designed.

2. The robot system according to claim 1 .

3. The information about the operation includes at least one of a part assembly direction, a part assembly stroke amount, a part assembly phase angle, a part insertion start position, a part insertion completion position, and a part insertion force.

2. The robot system according to claim 1 .

4. The information about the work is set as table information.

4. The robot system according to claim 3.

5. The exploration data is data obtained from at least one of a camera, a tactile sensor, an ultrasonic sensor, and a probe.

2. The robot system according to claim 1 .

6. A coordinate system indicating a position and orientation of the model is set for the model, Based on the model and the information about the working area acquired based on the search data, information about the position and orientation of the working area of ​​the workpiece, which is defined by a coordinate system of the model, is acquired; executing a task in the working area of ​​the workpiece by the robot based on the information regarding the position and orientation; 3. The robot system according to claim 2.

7. the robot includes a sensor for obtaining information regarding forces; executing a task in the working area of ​​the workpiece by the robot based on the information regarding the position and orientation and the information regarding a force acting on the robot; 7. The robot system according to claim 6.

8. The exploration unit includes an imaging device provided on the robot, performing a task in the working area of ​​the workpiece by the robot based on the information on the position and orientation, a reference position of the imaging device, and a reference position of a part to be moved by the robot; 7. The robot system according to claim 6.

9. The exploration unit includes an imaging device, acquiring a trajectory of the robot for performing the task on the workpiece based on information about the position and orientation, information about the task, a coordinate system of a predetermined portion of the robot, a coordinate system of an imaging device, and a coordinate system of the robot; 7. The robot system according to claim 6.

10. The exploration unit acquires the exploration data by exploring the working area in a exploration direction, Acquire learning model information acquired by learning the features of the working area when the model is explored from a plurality of angles; The control unit identifies the working area based on the learning model information and the exploration data.

3. The robot system according to claim 2.

11. The exploration unit is an imaging device that captures an image in an imaging direction and acquires image data including features of the work area, the learning model information is information acquired by learning features of the working area when the model is imaged from a plurality of angles, The control unit identifies the work area based on the learning model information and the image data acquired by the imaging device. The robot system according to claim 10 .

12. The control unit generates an inference image by inferring the working area from the image data acquired by the imaging device based on the learning model information, and identifies the working area in the image data by performing a matching process between the learning model information and the inference image. The robot system according to claim 11 .

13. The learning model information includes pose information regarding a pose of the model; The control unit identifies a matching area in the image data for matching the learning model information with the image data based on the posture information, and performs a matching process. The robot system according to claim 11 .

14. The imaging device has a plurality of cameras capable of three-dimensional measurement, The control unit identifies a matching area in the image data for matching the learning model information with the image data based on the image data acquired by each of the plurality of cameras, and performs a matching process. The robot system according to claim 12 .

15. The control unit performs a mask process on the image data other than the matching area, and performs a matching process on the learning model information and the masked image data. The robot system according to claim 13 .

16. The learning model information includes task information related to tasks to be performed on the task area, The control unit controls the robot based on the work information. The robot system according to claim 10 .

17. The work is an assembly work of assembling a part held by the robot in the area of ​​a workpiece.

17. The robot system according to claim 16.

18. The control unit controls the robot based on surrounding model information regarding a surrounding model that models surrounding objects arranged around the workpiece and the exploration data when performing work in the working area.

2. The robot system according to claim 1 .

19. the control unit, when performing work in the working area, determines a trajectory that avoids interference between the robot and the surrounding object based on the surrounding model information and the exploration data, and controls the robot according to the determined trajectory.

20. The robot system of claim 18.

20. When the control unit identifies the work area, the control unit displays the identified work area on a display device.

2. The robot system according to claim 1 .

21. An information processing device having a processing unit for acquiring the model, 3. The robot system according to claim 2.

22. A control method for a robot system including a robot, a detection unit that detects a working area where a task is performed on a workpiece, and a control unit, comprising: acquiring exploration data including information on the working area by the exploration unit; The control unit specifies the working area based on information about the working area, information about an operation to be performed on the workpiece that is associated with the working area, and the search data, and controls the robot to perform the operation in the specified working area. A method for controlling a robot system comprising:

23. A method for manufacturing an article, comprising the steps of: manufacturing an article by using the robot system according to claim 1;

24. A program for causing a computer to execute the control method for a robot system according to claim 22.

25. A computer-readable recording medium storing the program according to claim 24.

Images