Robot system, method for controlling a robot system, method for manufacturing articles, program, and recording medium

Abstract

To provide a robot system that reduces burden on an operator.SOLUTION: A robot system includes: a robot; a survey section which surveys a work region on a workpiece where work is performed and acquires survey data including information on the work region; and a control section which controls the robot to identify the work region on the basis of information on the work region, information which is associated with the work region and is on the work to be executed on the workpiece, and the survey data (S104 to S105) and to perform a work on the identified work region (S106).SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a robot system, a method for controlling a robot system, a method for manufacturing articles, a program, and a recording medium. [Background technology]

[0002] For example, in industrial robots installed in factories and other facilities, robotic devices perform tasks such as assembling and attaching parts, applying adhesives and paints, and processing with tools on workpieces. Such tasks can be performed precisely (with high accuracy) regardless of the workpiece's position or orientation by, for example, using a camera to recognize the part of the workpiece to be worked on and controlling the robot's position and orientation relative to the recognized part. However, when having robotic devices perform such precise tasks, there is a problem in that the adjustment work required by the operator to adjust the robot is significant, for example, when the robotic device is installed in the factory or when the work content or workpiece changes. For example, in image recognition methods that use template matching, adjusting the image processing process, such as brightness correction and feature extraction, to improve the accuracy of matching the template image with the captured image is complex, and setting the conditions takes time and places a burden on the operator.

[0003] Therefore, a method has been proposed in which an image of a workpiece is input into a trained learning machine to obtain two or more partial images, these partial images are subjected to blob analysis to generate blob information, and the position and angle of the workpiece are calculated from that blob information (see Patent Document 1). As a result, the method described in Patent Document 1 aims to reduce the workload of adjustment work in the image processing process described above.

[0004] Furthermore, a system has been proposed that focuses on the function (affordance) of an object and generates a trained model for recognizing the functional area in three dimensions (see Non-Patent Document 1). The system in Non-Patent Document 1 attempts to use the trained model to identify the working area of ​​a three-dimensional robot manipulator and perform gripping and transfer of workpieces. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2020-197983 [Non-Patent Document 1] Journal of the Robotics Society of Japan, Vol. 38, No. 6, pp. 525-529, 2020. "Functional Recognition of Objects and its Application to Robot Manipulation" (by Manabu Hashimoto). [Overview of the project] [Problems that the invention aims to solve]

[0006] However, while the above-mentioned Patent Document 1 is capable of measuring the position and angle of a workpiece on a two-dimensional plane, it has the problem that it is difficult to calculate the three-dimensional position and orientation information of the workpiece from blob information. As a result, it is difficult to automatically generate the robot's trajectory during work operations, the robot's drive generation (teaching) takes time, and the robot adjustment work by the operator is burdensome.

[0007] Furthermore, while the method described in Non-Patent Document 1 may be suitable for tasks requiring rough movements, it is difficult to implement for tasks requiring precise movements (high recognition accuracy). In other words, even if the method described in Non-Patent Document 1 is adopted, for example, intricate assembly tasks, it is necessary to set the precise movement (trajectory) of the robot arm relative to the work area of ​​the recognized workpiece, which requires an operator with specialized knowledge to perform the setting, thus still placing a heavy burden on the operator.

[0008] Therefore, the present invention aims to provide a robot system capable of reducing the burden on workers, a control method for the robot system, a method for manufacturing articles, a program, and a recording medium. [Means for solving the problem]

[0009] One aspect of the present invention comprises a robot, a search unit that searches for a work area on a workpiece and acquires search data including information about the work area, and a control unit that identifies the work area and controls the robot to perform work in the identified work area based on the information about the work area, information about the work associated with the work area and to be performed on the workpiece, and the search data. The robot system is characterized in that, the information regarding the work area is a model obtained by modeling the work area from design information when the workpiece was designed, the exploration unit is an imaging device that takes images in the imaging direction and obtains image data including the features of the work area as exploration data, and the control unit generates an inferred image that infers the work area from the image data obtained by the imaging device based on learned model information obtained by learning the features of the work area when the model is imaged from multiple angles, and identifies the work area in the image data by matching the learned model information and the inferred image.

[0010] One aspect of the present invention relates to a robot and a work area for performing work on a workpiece. imaging do Imaging device A control method for a robot system comprising a control unit, The imaging device The work area Features including image The control unit acquires the data and then Based on the learned model information obtained by learning the characteristics of the work area when the model obtained by modeling the work area from the design information when the work area was designed and the model is captured from multiple angles, an inferred image is generated in which the work area is inferred from the image data acquired by the imaging device, and the work area in the image data is identified by matching the learned model information and the inferred image. , information regarding the work to be performed on the workpiece Based This is a control method for a robot system, characterized by controlling the robot to perform work in the specified work area. [Effects of the Invention]

[0011] According to the present invention, the burden on workers can be reduced. [Brief explanation of the drawing]

[0012] [Figure 1] This is a diagram showing the configuration of the robot system according to the first embodiment. [Figure 2] This is a block diagram showing the configuration of the information processing device according to the first embodiment. [Figure 3] This is a block diagram showing the configuration of a robot controller according to the first embodiment. [Figure 4] This is a flowchart showing the assembly process using the robot system according to the first embodiment. [Figure 5] This is a perspective view showing an example of a CAD model of a workpiece. [Figure 6] This is a perspective view showing an example of the labeling section of a workpiece. [Figure 7] It is a perspective view showing an example of a label model obtained by modeling a labeling unit. [Figure 8] It is an explanatory diagram illustrating an example of label model information to which assembly information is added. [Figure 9] It is a diagram showing a state where a CAD model is imaged by a virtual camera in a virtual space. [Figure 10] (a) is a diagram showing an example of a virtual CAD model image obtained by imaging a CAD model with a virtual camera. (b) is a diagram showing an example of a virtual region model image obtained by imaging a label model with a virtual camera. [Figure 11] It is an explanatory diagram illustrating learning processing for learning image features of a labeling unit. [Figure 12] It is an explanatory diagram illustrating inference processing for inferring a labeling unit from actual image data. [Figure 13] It is an explanatory diagram illustrating processing for calculating the position of a labeling unit in a camera coordinate system. [Figure 14] It is an explanatory diagram illustrating processing for calculating the position of a labeling unit in a robot coordinate system. [Figure 15] It is a diagram showing a configuration of a robot system according to a second embodiment. [Figure 16] It is a flowchart showing steps of an assembling operation by a robot apparatus according to the second embodiment. [Figure 17] It is an explanatory diagram illustrating processing for generating a three-dimensional point cloud image. [Figure 18] It is an explanatory diagram illustrating processing for defining a solid angle on a work according to a third embodiment. [Figure 19] It is an explanatory diagram illustrating learning processing for learning image features of a labeling unit according to the third embodiment in association with a solid angle of a work. [Figure 20] It is a diagram showing an example of a GUI showing a result of model matching according to a fifth embodiment. [Figure 21] It is a diagram showing position and orientation information of a CAD model corresponding to a labeling unit on an actual work according to a sixth embodiment. [Figure 22] This figure shows the configuration of the robot device according to the sixth embodiment. [Figure 23] This is a control block diagram illustrating control using a visual servo according to the seventh embodiment. [Figure 24] This figure illustrates a method for generating image data corresponding to the target features according to the seventh embodiment. [Figure 25] This is a flowchart showing the assembly process using a robotic device according to the seventh embodiment. [Modes for carrying out the invention]

[0013] <First Embodiment> A first embodiment for carrying out the present invention will be described below with reference to Figures 1 to 14.

[0014] [Outline configuration of the robot system] First, the schematic configuration of the robot system according to this first embodiment will be described using Figures 1, 2, and 3. Figure 1 is a diagram showing the configuration of the robot system according to the first embodiment. Figure 2 is a block diagram showing the configuration of the information processing device according to the first embodiment. Figure 3 is a block diagram showing the configuration of the robot controller according to the first embodiment.

[0015] The robot system 1 is an automated assembly system that, for example, assembles a part 11, which is an assembly workpiece, onto a workpiece 10, which is a workpiece to be assembled. It is broadly composed of a robot device 100 and an information processing device 501. The robot device 100 is fixedly supported on a frame 13 and includes a robot arm (manipulator) 200 as a robot and a robot controller 201 that controls the robot arm 200.

[0016] Furthermore, the robot device 100 is equipped with a robot hand 202, which is an end effector attached to the tip of the robot arm 200 for gripping (holding) the part 11. The robot hand 202 has no particular restrictions on shape or structure as long as it can hold the part 11; for example, it may have a structure that sucks on the part 11. The robot hand 202 may also be equipped with force sensors or the like as needed.

[0017] Furthermore, the workpiece 10 is placed on the workpiece stand 12 installed on the frame 13, and the robot device 100 is equipped with a camera 300, which serves as a search unit or imaging device, positioned above the workpiece stand 12, that is, above the workpiece 10. The camera 300 captures an image of at least the imaging area (imaging range) including the workpiece 10 and acquires it as real image data. This camera 300 may be a two-dimensional camera that has the function of outputting two-dimensional image data, or it may be a three-dimensional camera that has the function of outputting three-dimensional image data, such as a stereo camera. In other words, the camera may be provided on the robot device 100 as a robot. In this embodiment, the camera 300 is described as a fixed camera installed on the ceiling of a factory, for example, but it may also be an on-hand camera fixed to the robot hand 202, as long as it can acquire an image of the imaging area including the workpiece 10. The image data captured by the camera 300 is sent to the robot controller 201 and processed as described in detail later. Here, information processing refers to the robot controller 201 calculating command values ​​(such as the robot arm's trajectory) for robot control in order to assemble the part 11 onto the workpiece 10.

[0018] The robot system 1 configured as described above performs an assembly operation in which the part 11 grasped by the robot hand 202 of the robot device 100 is assembled into the hole, which is the working area of ​​the workpiece 10, as will be described in more detail later. In this way, the robot system 1 uses the robot device 100 to perform the assembly operation of assembling the part 11 to the workpiece 10, thereby manufacturing the workpiece 10 with the part 11 assembled as an article. In other words, it performs a manufacturing method in which an article with the part 11 assembled to the workpiece 10 is manufactured using the robot device 100.

[0019] (Configuration of information processing device) Next, the configuration of the information processing device 501 will be explained using Figure 2. As shown in Figure 2, the information processing device 501 includes a CPU (Central Processing Unit) 502, which is an example of a processor. The CPU 502 is an example of a processing unit. The information processing device 501 also includes a ROM (Read Only Memory) 503, a RAM (Random Access Memory) 504, and an HDD (Hard Disk Drive) 505 as storage units. The information processing device 501 also includes a recording disk drive 506, and a display device that serves as an input / output interface: a display 508, a keyboard 509, and a mouse 510. The CPU 502, ROM 503, RAM 504, HDD 505, recording disk drive 506, display 508, keyboard 509, and mouse 510 are connected to each other by a bus so that they can communicate with one another.

[0020] ROM 503 stores the basic programs related to the operation of the computer. RAM 504 is a memory device that temporarily stores various data, such as the results of calculations performed by CPU 502. HDD 505 records the results of calculations performed by CPU 502 and various data acquired from external sources, as well as program 507 for executing various processes described later. Program 507 is application software that enables CPU 502 to perform various processes related to the pre-preparation process (Figure 4) described later. Therefore, CPU 502 can execute the various pre-preparation processes described later by executing program 507 recorded in HDD 505. HDD 505 also has an area for recording learning model information 520 as model information obtained from the execution results of the various pre-preparation processes described later. The recording disk drive 506 can read various data and programs recorded on the recording disk 150.

[0021] In this embodiment, the non-temporary recording medium readable by the computer is the HDD 505, and the program 507 is recorded on the HDD 505, but this is not the only possible representation. The program 507 may be recorded on any non-temporary recording medium readable by the computer. Examples of recording media that can be used to supply the program 507 to the computer include flexible disks, hard disks, optical disks, magneto-optical disks, magnetic tapes, non-volatile memory, and the like.

[0022] Furthermore, a robot controller 201 is connected to the information processing device 501. As will be explained in more detail later, the information processing device 501 transmits the learning model information 520 to the robot controller 201 as a result of performing various pre-preparation processes.

[0023] (Robot controller configuration) Next, the configuration of the robot controller 201 will be explained using Figure 3. As shown in Figure 3, the robot controller 201 includes a CPU 204, which is an example of a processor. The CPU 204 is an example of a control unit. The robot controller 201 also includes a ROM 205, RAM 206, and HDD 207 as storage units. Furthermore, the robot controller 201 includes a recording disk drive 208 and an input / output interface 209. The CPU 204, ROM 205, RAM 206, HDD 207, recording disk drive 208, and interface 209 are connected to each other by a bus so that they can communicate with one another.

[0024] ROM205 stores the basic programs related to the operation of the computer. RAM205 is a memory device that temporarily stores various data, such as the results of calculations performed by the CPU204. HDD207 records the results of calculations performed by the CPU204 and various data acquired from external sources, as well as a program 210 that causes the CPU204 to execute various processes related to the actual machine processing (see Figure 4) described later. Program 210 is application software that enables the CPU204 to perform various processes of the actual machine processing described later. Therefore, the CPU204 can execute control processing by executing the program 210 recorded in HDD207 and control the movement of the robot arm 200. In addition, HDD207 is provided with an area for recording learning model information 520 acquired by being transmitted from the information processing device 501. The recording disk drive 208 can read various data and programs recorded on the recording disk 250.

[0025] In this embodiment, the non-temporary recording medium readable by the computer is the HDD207, and the program 210 is recorded on the HDD207, but this is not the only possible representation. The program 210 may be recorded on any non-temporary recording medium readable by the computer. Examples of recording media that can be used to supply the program 210 to the computer include flexible disks, hard disks, optical disks, magneto-optical disks, magnetic tapes, non-volatile memory, and the like.

[0026] Furthermore, the robot controller 201 is connected to the camera 300, the robot arm 200, and the information processing device 501 mentioned above. As will be explained in more detail later, the robot controller 201 receives learning model information 520 from the information processing device 501 as the result of various pre-preparation processes. The camera 300 also transmits captured image data to the robot controller 201, where the image data is processed by program 107. The processing results are output as command values ​​for robot control and transmitted to the robot arm 200.

[0027] In this embodiment, the pre-preparation processing (see Figure 4) is processed by the information processing device 501 (CPU 502), and the actual machine processing (see Figure 4) is processed and executed by the robot controller 201 (CPU 204). However, the embodiment is not limited to this. These processes may be executed by one computer, i.e., one CPU, or by three or more computers, i.e., three or more CPUs. Furthermore, if the pre-preparation processing and the actual machine processing (see Figure 4) are divided among multiple computers, any process may be executed by any of the computers.

[0028] [Assembly process] Next, using the robot system 1 described above, the process of assembling the robot (control of the robot system) to achieve the assembly work shown in Figure 1 will be explained using Figures 4 to 14.

[0029] (Preparation process) First, the preparatory processing performed by the information processing device 501 described above will be explained using Figures 4 to 11. Figure 4 is a flowchart showing the assembly process by the robot system according to the first embodiment. Figure 5 is a perspective view showing an example of a CAD model of a workpiece. Figure 6 is a perspective view showing an example of a labeling section of a workpiece. Figure 7 is a perspective view showing an example of a label model that models the labeling section. Figure 8 is an explanatory diagram illustrating an example of label model information to which assembly information has been added. Figure 9 is a diagram showing the state in which a CAD model is imaged by a virtual camera in a virtual space. Figure 10(a) is a diagram showing an example of a virtual CAD model image obtained by imagering a CAD model with a virtual camera. Figure 10(b) is a diagram showing an example of a virtual region model image obtained by imagering a level model with a virtual camera. Figure 11 is an explanatory diagram illustrating the learning process for learning the image features of the labeling section.

[0030] As shown in Figure 4, the preparatory processes in steps S101 to S103 are preparatory processes before actually operating the robot device 100 (performing the actual assembly work). These preparatory processes generate learning model information 520 using computer design support tools such as a CAD (Computer Aided Design) system. In other words, as will be explained in detail below, these preparatory processes generate learning model information 520 using the CAD data of the workpiece 10, which is the design information when the workpiece 10 was designed.

[0031] First, in step S101, the CPU 502 performs a task called labeling (hereinafter referred to as "labeling work") on the CAD model of the workpiece 10 on a computer design support tool such as a CAD system. Here, a CAD model is a data representation format on a computer design support tool, and formats such as STEP, IGES, and STL files are generally known. In the labeling work, the CPU identifies the work area where the assembly work, in which the parts 11 come into contact and are assembled, is performed, on the CAD model 20 corresponding to the workpiece 10 shown in Figure 5, and labels it. A label is like an identification number on the CAD model 20 that indicates that it is the work area in which the parts 11 are assembled. For example, data such as label number = 1 is added to the labeled work area. By processing in this way, a labeling unit 21 with labels attached to the work area is created, as shown in Figure 6. Furthermore, in this process, if there are multiple work areas where the workpiece 10 comes into contact with other parts or workpieces, labels may be attached to multiple work areas to provide multiple labeling sections, and these labeling sections can be distinguished by label numbers such as 2, 3, ...

[0032] Next, in step S102, the CPU 502 models the labeling unit 21. As shown in Figure 7, the labeling unit 21, with the work area labeled in step S101, for example, data such as label number = 1 added, is modeled as a new three-dimensional model, the CAD model 22. Such a CAD model 22 can be expressed using formats such as STEP, IGES, and STL as described above, and constitutes information about the work area in this embodiment.

[0033] Here, the CAD model 22 of the labeled work area is assigned assembly information, which is information related to assembly work, as work information related to the work performed on the work area, as shown in Figure 8 (i.e., it is associated and stored in the HDD 505). In order to assign assembly information to the CAD model 22, first, the CAD model 22 is made to hold a reference coordinate system O at an arbitrary position. The coordinate system O has orthogonal axes x, y, z for representing three-dimensional Euclidean space, as well as rotational components Rx, Ry, Rz around each axis. This makes it possible to represent the position and orientation in three-dimensional space in the 6-axis coordinate system of the robot device 100. Although orthogonal coordinate space has been used as an example here, other forms such as polar coordinates or quaternions may be applied to the coordinate system O as long as they can represent the position and orientation in three-dimensional space.

[0034] Next, a table data TB is created that shows assembly information as work information, as shown in the table in Figure 8. The table data TB includes various information related to the assembly work (operation, function), such as information indicating the assembly direction during assembly, and the amount of assembly stroke from the contact point to the completion of assembly. The table data TB also includes information such as the assembly phase angle indicating the orientation of assembly, the insertion start position and insertion completion position during assembly, and the insertion force required for assembly. In this way, the CAD model 22 and the table data TB are maintained in a one-to-one correspondence (i.e., associated and stored in the HDD 505). If the workpiece 10 has multiple labeled work areas as described above, multiple CAD models corresponding to them may be generated, and a table data TB may be provided for each of these CAD models in a one-to-one correspondence.

[0035] Next, in step S103, the CPU 502 learns the image features of the labeling section. In this step S103, as shown in Figure 9, a virtual camera 301, corresponding to the camera 300 shown in Figure 1, is constructed and prepared in a virtual space in a computer design support tool such as a CAD system. Then, a process is performed to capture a three-dimensional CAD model 20 corresponding to the workpiece 10 as seen from the virtual camera 301.

[0036] Here, it is preferable to set the virtual camera 301 to the same settings as the camera 300 actually used in the robot system 1. For example, this includes settings such as the cell size of the image sensor, the number of pixels, and the focal length and aperture of the lens. In this way, the virtual camera 301 can acquire an image (hereinafter referred to as "virtual image") 32 captured in a virtual space as shown in Figure 10(a) on the computer design support tool. In the virtual image 32, it is difficult to obtain shading and texture information of the workpiece 10 that are exactly the same as in the actual environment, so this information is not necessarily required. The minimum required information is information about the edges that show the shape of the workpiece 10. It is desirable that this edge information is obtained to match the edges of the CAD model 20.

[0037] Next, in the virtual space of the computer design support tool, a three-dimensional CAD model 22 of the labeled work area is placed so that its positional relationship matches that of the labeling section 21 in the CAD model 20 of the workpiece 10. Then, the CAD model 22 of the labeling section 21 is captured by the virtual camera 301 to obtain a virtual image 33 as shown in Figure 10(b). This virtual image 33 is image data that allows the edge information of the CAD model 22 of the labeling section 21 to be acquired. The image data of the virtual image 32 shown in Figure 10(a) and the virtual image 33 shown in Figure 10(b) are then stored as a pair (i.e., associated and stored in the HDD 505).

[0038] Note that at least one pair of this image data must be acquired for the learning process described later, but it is preferable to have as many virtual images as possible with different imaging angles and brightness levels. To obtain multiple images, the brightness of the virtual images or the texture of the workpiece 10 may be changed in the virtual space of the computer design support tool, as long as the edge information is not lost. Alternatively, the relative position of the virtual camera 301 and the CAD model 20 of the workpiece 10 or the CAD model 22 of the labeling unit 21 may be moved within the expected range for imaging. The expected range referred to here means the range of deviation that may occur in the relative positional relationship between the camera 300 and the workpiece 10 in the actual robot system 1.

[0039] The pairs of image data of multiple virtual images obtained through the above process are used in the learning process shown in Figure 11. For example, learning is performed by associating the above table data TB (see Figure 8) with each pair of virtual images that have different imaging angles, brightness levels, etc. In this embodiment, a machine learning algorithm is used in the learning process, and in particular, a "supervised learning" algorithm is used. Therefore, the image data of the multiple virtual images obtained earlier becomes the training data D1 used for "supervised learning". In the training data D1, the virtual image in which the CAD model 20 of the workpiece 10 is captured is used as input data D1A, and the virtual image in which the CAD model 22 of the labeling unit 21 is captured is used as output data D1B, and machine learning is performed to associate them. This generates learning model information 520 as a trained model (generation process).

[0040] The machine learning algorithm used in step S103 is, for example, semantic segmentation or instance segmentation. These are types of "supervised learning" and are algorithms that infer output values ​​for each pixel of input data D1A by performing machine learning based on training data D1. If learning progresses well, it will be possible to obtain edge information of the CAD model 22 of the labeling unit 21 from the input data D1A. Note that the algorithm used for machine learning is not limited to semantic segmentation or instance segmentation; any algorithm that has the function of extracting the above-mentioned features may be used. In step S103, the learned model information 520 obtained through learning is stored in the storage unit of the information processing device 501, for example, HDD 505. This learned model information 520 is then output in the form of being transferred to the storage unit of the robot controller 201, for example, HDD 207, and used in the actual machine processing described next.

[0041] (Actual device processing) Next, the actual processing performed by the robot controller 201 will be explained using Figures 4, 12, 13, and 14. Figure 12 is an explanatory diagram illustrating the inference process for inferring the labeling unit from actual image data. Figure 13 is an explanatory diagram illustrating the process for calculating the position of the labeling unit in the camera coordinate system (an example of a coordinate system for a predetermined part of the robot). Figure 14 is an explanatory diagram illustrating the process for calculating the position of the labeling unit in the robot coordinate system.

[0042] As shown in Figure 4, the actual machine processing in steps S104 to S106 is actual machine processing that operates the robot device 100 (performs actual assembly work). First, in step S104, the CPU 204 infers the labeling unit 21 from the actual image. In detail, with the workpiece 10 placed on the workpiece stand 12 (see Figure 1), the camera 300 takes an image of the area (imaging range) including the workpiece 10. Here, the camera 300 functions as an exploration unit that explores the work area where work is performed on the workpiece 10 and acquires exploration data including information about the work area. The image data of the actual image captured as exploration data is transferred to the robot controller 201, where machine learning inference processing is performed as shown in Figure 12. When the camera 300 takes an image of the workpiece 10 on the workpiece stand 12, the camera 300 is moved to a position above the workpiece 10, and the posture of the camera 300 is controlled by the robot arm 200 so that the imaging direction of the camera 300 faces the workpiece 10. The trajectory of the robot arm 200 in this case is the trajectory that has been taught in advance by the operator, for example, using a teaching pendant or the like, to determine the position and orientation of the robot arm 200.

[0043] The input data shown in Figure 12 is image data of a real image captured by the camera 300. The CPU 204 reads the aforementioned learning model information 520 and infers the output data using the same machine learning algorithm as during training. If the learning model information 520 is for a well-trained model, the output data should be image data (hereinafter referred to as the "inferred image") corresponding to the edge information of the CAD model 22 of the labeling section 21 in the workpiece 10.

[0044] In step S105, the CPU 204 performs model matching on the inference image obtained in step S104. This matching process uses the CAD model 22 of the labeling unit 21 created in step S102. This effectively identifies the position and orientation of the work area of ​​the workpiece 10 in the image data of the actual image captured by the camera 300. In other words, in step S105, the work area is identified (identification step) based on the learning model information 520, the image data, and the CAD model 22 of the labeling unit 21.

[0045] In step S105, the image used for the matching process may be the inference image obtained in step S104. However, for example, a portion corresponding to the region obtained by inference (matching region) may be extracted from the captured image data, and the region other than that region may be masked to create a mask region. The matching process may then be performed on the image data with the mask region masked. In short, the learning model information 520 may be matched with the image data with the mask region masked, thereby reducing the image processing load.

[0046] Then, once the matching is successfully completed, as shown in Figure 13, in conjunction with known camera calibration techniques, the vector Vc represents the position information of the labeling unit 21 as seen from the camera, in the three-dimensional space of the CAD model 22. w It is possible to find Vc. w This shows a vector from the origin 319 of the camera coordinate system to the origin 23 of coordinate system O, which serves as the reference for the CAD model of an arbitrary labeling area.

[0047] In step S106, the CPU 204 performs a process to generate a trajectory for assembling the part 11 to the workpiece 10 while the robot arm 200 is holding (gripping) the part 11. Specifically, first, as shown in Figure 14, a vector Vc is generated from the origin 220 of the robot arm 200's coordinate system to the origin 319 of the camera coordinate system. ris obtained using a known hand-eye calibration technique. Further, since the robot arm 200 holds the component 11 via the robot hand 202, the vector from the coordinate system origin 220 of the robot arm 200 to the reference position 24 of the arbitrary component 11 is defined as vector Vr t × vector Vt w′ to obtain.

[0048] Here, the vector Vr t represents a vector from the coordinate system origin 220 of the robot arm 200 to the coordinate system origin 221 of the robot hand 202. For obtaining this vector Vr t , encoder values for detecting angles of respective joints that the robot arm 200 uses to calculate its own position may be used, or the position may be measured from an image captured externally by, for example, a camera, and various known methods can be used. Further, the vector Vt w′ represents a vector from the coordinate system origin 221 of the robot hand 202 to the reference position 24 of the arbitrary component 11. For obtaining this vector Vt w′ , various known methods such as an external measurement method and a mechanical positioning method can also be used.

[0049] In this way, a trajectory for moving the component 11 to the labeling portion 21 of the workpiece 10 by the robot arm 200, that is, the trajectory Vw until the component 11 starts to be assembled to the workpiece 10 w′ can be generated. Further, the trajectory Vw w′ is not limited to a linear trajectory, and as long as the start point and the end point match, any interpolation processing such as spline interpolation may be added to the intermediate path, and the trajectory may be freely determined.

[0050] Furthermore, the CPU 204 reads the table data TB (see Figure 8) which shows the assembly information included in the learning model information 520, and generates a trajectory from the position where assembly of part 11 to the workpiece 10 begins (insertion start) to the position where assembly is completed (insertion completion). That is, the table data TB contains information related to the work performed on the workpiece, such as assembly direction, assembly stroke, assembly transfer angle, insertion start position, insertion completion position, and insertion force. From this information, the CPU 204 generates a trajectory for assembling part 11 to the workpiece 10 from the assembly start position. Then, the trajectory Vw obtained as described above until assembly of part 11 to the workpiece 10 begins is generated. w′ Next, a trajectory for assembling part 11 onto workpiece 10 is added, and with that, the generation of the trajectory for controlling the robot arm 200 during the assembly operation is complete. Note that the information regarding the operation to be performed on the workpiece only needs to include at least one of the following: the assembly direction of the part, the assembly stroke amount of the part, the assembly phase angle of the part, the insertion start position of the part, the insertion completion position of the part, and the insertion force of the part.

[0051] In this way, once a trajectory for controlling the robot arm 200 during the assembly work is generated in step S106, the CPU 204 outputs that trajectory as a command value to the robot arm 200 and drives the robot arm 200 to operate along that trajectory. As a result, based on the learned model information 520 (assembly information (see Figure 8)), the robot arm 200 moves the part 11 to the position of the work area of ​​the identified workpiece 10, and assembles it to the workpiece 10 while controlling its position and orientation. In other words, in step S106, the robot arm 200 is controlled to perform work in the work area identified in step S105 (work process).

[0052] [Summary of the First Embodiment] As described above, by performing the assembly work process using the robot system 1 shown in Figure 4, the robot device 100 can automatically assemble the parts 11 into the work area of ​​the workpiece 10 without placing a heavy burden on the operator's adjustment work.

[0053] Specifically, in this embodiment, the process from steps S101 to S102 is configured to model the workpiece 10 in the virtual space of the CAD system and generate a CAD model 20. This significantly reduces the need for tasks such as the conventional method of generating numerous template images by repeatedly photographing the workpiece 10 with the camera 300 while changing the imaging angle, as well as the adjustment work for image processing processes such as angle correction and feature extraction. Consequently, the burden of adjustment work on the operator can be reduced.

[0054] Furthermore, in this embodiment, the system is configured to generate not only a CAD model 20 that models the entire workpiece 10, but also a CAD model 22 of the labeling section 21 that is labeled as a work area. This significantly reduces the amount of computation required in the model matching process in step S105 compared to matching with the CAD model 20, which represents the entire workpiece 10, thus enabling faster processing. In addition, this model matching process does not match a two-dimensional template image with the actual image, but rather matches the three-dimensional CAD model 22 with the actual image. This makes it possible to grasp the position and orientation of the workpiece 10 in three dimensions from the learned model information 520.

[0055] Furthermore, in this embodiment, the labeling unit 21 is configured to learn image features in step S103. This reduces the number of times the CAD model 20 is captured by the virtual camera 301 in the virtual space, compared to, for example, preparing a large number of template images. As a result, the load of pre-preparation processing can be reduced, and the burden of adjustment work by the operator can be reduced. In addition, by generating learning model information 520, which is information of the learned model, the accuracy of inference and model matching of the labeling unit 21 in steps S104 to S105 can be improved.

[0056] Furthermore, in this embodiment, the learning model information 520 is configured to store table data TB, which is assembly information obtained from CAD data (design information), in association with the CAD model 22 of the labeling unit 21. As a result, when generating the trajectory of the robot arm 200 in step S106, the table data TB associated with the CAD model 22 that matches the actual image can be extracted, and a highly accurate trajectory of the robot arm 200 can be automatically generated. Therefore, the assembly work by the robot device 100 can be executed with high accuracy. In addition, by using assembly information obtained from CAD data (design information) in this way, it becomes unnecessary for the operator to generate and prepare a large number of trajectories corresponding to each angle of the workpiece in advance, thereby reducing the burden of adjustment work on the operator.

[0057] As described above, by performing the assembly work process using the robot system 1 according to this embodiment, the burden of adjustment work as preparation can be reduced, and an automated production system that performs assembly work using the robot device 100 can be started up in a short time.

[0058] In this first embodiment, a CAD model 22 of the labeling unit 21 is generated and matched with the actual image using a model matching process. However, the invention is not limited to this; a CAD model 20 of the workpiece 10 is also generated and matched with the actual image.

[0059] Furthermore, in this first embodiment, in order to enable the identification of the labeling unit 21 modeled from the actual image, learning was performed using a machine learning algorithm in steps S103 and S104. However, the system is not limited to this, and it may be configured to be able to identify the labeling unit using methods other than learning.

[0060] <Second Embodiment> Next, a second embodiment, which is a modified version of the first embodiment described above, will be explained using Figures 15 to 17. Figure 15 is a diagram showing the configuration of the robot system according to the second embodiment. Figure 16 is a flowchart showing the assembly process by the robot device according to the second embodiment. Figure 17 is an explanatory diagram illustrating the process of generating a three-dimensional point cloud image. In this explanation of the second embodiment, the same reference numerals are used for the same parts as in the first embodiment, and their explanations are omitted.

[0061] [Configuration of the robot system according to the second embodiment] In the robot system 1 according to this second embodiment, as shown in Figure 15, in addition to the camera (hereinafter referred to as the "first camera") 300, a second camera 320 is provided as an exploration unit or imaging unit. These first camera 300 and second camera 320 constitute a stereo camera, which is used to measure the actual workpiece 10 in three dimensions. In the robot system 1 shown in Figure 15, the first camera 300 and second camera 320 are shown as fixed cameras as an example, but they may also be on-hand cameras mounted on the robot hand 202. Furthermore, the number of cameras configured as an imaging unit is not limited to two, but may be three or more. For example, a stereo camera may be provided with both a fixed camera and an on-hand camera, with one or both of them being two units.

[0062] [Assembly process according to the second embodiment] Next, the assembly process using the robot system 1 according to the second embodiment will be described. As shown in Figure 16, steps S201, S202, and S203, which are preparatory processes before actually operating the robot device 100, are the same as steps S101, S102, and S103 shown in Figure 4 above. However, in the learning in step S203, learning model information 520 may be generated from virtual images of the CAD model 22 captured in the virtual arrangements of the first camera 300 and the second camera 320. Alternatively, common learning model information 520 may be generated, and in the case of generating common learning model information 520, the input data D1A will include image data of the CAD model 22 captured in the virtual arrangements of both the first camera 300 and the second camera 320.

[0063] Next, the actual machine processing in the assembly process according to the second embodiment will be described. In this second embodiment, when the actual machine processing is started, first, in steps S204-1 and S204-2, the CPU 204 takes images with the first camera 300 and the second camera 320. Then, the labeling unit 21 is inferred from the image data taken by the first camera 300 and the second camera 320, respectively. At this time, the learning model information 520 used may be separate for each camera as described above, or it may be a common one. In any case, the labeling unit 21 is inferred from the image data taken by each camera.

[0064] Next, in step S205, the CPU 204 performs three-dimensional measurement. Three-dimensional measurement is made possible by the principle of triangulation by performing a known stereo calibration between the first camera 300 and the second camera 320. That is, as shown in Figure 17, a three-dimensional point cloud image 340 can be obtained by performing three-dimensional measurement using a known method such as block matching based on the inferred image 330 of the first camera 300 and the inferred image 331 of the second camera 320. Then, in step S206, model matching processing is performed on the three-dimensional point cloud image 340 with the labeling unit 21. This makes it possible to obtain the position and orientation of the labeling unit 21 and assembly information (see table data TB in Figure 8). After obtaining the position and orientation of the labeling unit 21 and the assembly information, the trajectory generation process in step S207 is the same as the process in step S106 described above, and this makes it possible to generate the trajectory of the robot arm 200.

[0065] In step S205, the image used for three-dimensional measurement may be the inference image obtained in step S204. However, for example, a portion corresponding to the region obtained by inference (matching region) may be extracted from the captured image data, and the region other than that region may be masked to create a mask region. Three-dimensional measurement may then be performed on the image data with the mask region masked. That is, a three-dimensional point cloud image 340 may be obtained from the image data with the mask region masked, and model-to-model matching processing may be performed with the labeling unit 21. In short, in a broad sense, the image processing load can be reduced by matching the learning model information 520 with the image data with the mask region masked.

[0066] Furthermore, the inference image 330 from camera 300 and the inference image 331 from camera 320 may be preprocessed by removing noise generated by inference, approximating straight lines, approximating ellipses, etc., before performing the matching process.

[0067] [Summary of the second embodiment] As described above, in the assembly process by the robot system 1 according to the second embodiment, inference images 330 and 331 are acquired by the first camera 300 and the second camera 320 which constitute the stereo camera. Then, by generating a three-dimensional point cloud image 340 from these inference images and performing model matching processing, the work area of ​​the actual workpiece 10 can be identified with high accuracy.

[0068] Furthermore, the configuration, operation, and effects of this second embodiment are the same as those of the first embodiment described above, so their explanation will be omitted.

[0069] <Third Embodiment> Next, a third embodiment, which is a modified version of the first and second embodiments described above, will be explained using Figures 18 to 19. Figure 18 is an explanatory diagram illustrating the process of defining a solid angle for a workpiece according to the third embodiment. Figure 19 is an explanatory diagram illustrating the learning process of learning the image features of the labeling unit according to the third embodiment in relation to the solid angle of the workpiece. In this explanation of the third embodiment, the same reference numerals will be used for the same parts as in the first and second embodiments, and their explanations will be omitted.

[0070] In this third embodiment, when learning the image features of the labeling unit 21 in step S103, in addition to the position of the edges in the CAD model 20 of the workpiece 10, orientation information is also learned. Specifically, as shown in Figure 18, in the virtual space, the direction in which an arbitrary normal vector NL in the CAD model 20 points with respect to an arbitrary reference vector VA set on the CAD model 20 is defined by the solid angle α as orientation information. In other words, for multiple CAD models 20 with different imaging angles captured by the virtual camera 301 in the virtual space, the orientation is defined by the solid angle α, and this is included in the learning model information 520 as orientation information associated with each CAD model 20.

[0071] Specifically, as shown in Figure 19, the model is trained to associate the value of the solid angle α with the output data DIB. To train the model in this way, the output data D1B is tagged with information about the solid angle α (pose information), and then it can be trained using algorithms such as the instance segmentation mentioned above. If the training proceeds well, the solid angle α will also be inferred when the labeling unit 21 is inferred in step S104. Therefore, in the next step S105, when performing model matching processing, the angle can be narrowed down for matching, thereby reducing the risk of mismatching.

[0072] Furthermore, the configuration, operation, and effects of this third embodiment are the same as those of the first and second embodiments described above, so their explanation will be omitted.

[0073] <Fourth Embodiment> Next, a fourth embodiment, which is a modified version of the first to third embodiments described above, will be explained. In this fourth embodiment, the same reference numerals as those used in the first to third embodiments will be used, and their explanations will be omitted.

[0074] In this fourth embodiment, when generating the trajectory of the robot arm 200 in step S106, the modeling is performed on parts other than the labeling section 21 so that a trajectory is generated in which the robot arm 200 does not interfere with the workpiece 10 itself or surrounding objects other than the workpiece 10. That is, for example, in step S102, a surrounding model is generated by modeling not only the labeling section 21 (work area), which is the assembly part of the part 11, but also the workpiece 10 and surrounding objects arranged around the workpiece 10 from the CAD data. Then, surrounding model information, which is information about that surrounding model, is generated. Note that it is not necessary to associate the table data TB shown in Figure 8 with the models other than the labeling section 21. Other than this, the same procedure as the assembly work process described above (see Figure 4) can be performed. In this way, a surrounding model of the surrounding objects other than the labeling section 21 is generated, a model matching process is performed to determine a trajectory that avoids interference with the surrounding objects (surrounding model), and the trajectory of the robot arm 200 is generated from the determined trajectory. This makes it possible to generate a trajectory for the robot arm 200 that does not interfere with the workpiece 10 or the objects surrounding the workpiece 10.

[0075] Furthermore, the configuration, operation, and effects of this fourth embodiment are the same as those of the first to third embodiments described above, so their explanation will be omitted.

[0076] <Fifth Embodiment> Next, a fifth embodiment, which is a modified version of the first to fourth embodiments described above, will be explained using Figure 20. Figure 20 is a diagram showing an example of a GUI that displays the results of model matching according to the fifth embodiment. In this description of the fifth embodiment, the same reference numerals as those used in the first to fourth embodiments will be used, and their explanations will be omitted.

[0077] In this fifth embodiment, the progress of the assembly work shown in Figure 4 is shown to the user via a GUI (Graphical User Interface) 130, which is displayed on a display device such as a display 508 connected to an information processing device 501. In this embodiment, the process of generating this GUI 130 is performed by the CPU 204 of the robot controller 201, and as an example, it is described as being transferred to the information processing device 501 and displayed on the display 508. However, it is not limited to this, and the GUI 130 may be generated by the CPU 204 and transferred to a display device directly connected to the robot controller 201 for display. Alternatively, various data calculated by the CPU 204 of the robot controller 201 may be sent to the information processing device 501, and the CPU 502 may generate the GUI 130 and display it on the display 508.

[0078] An example of GUI130 will be explained using Figure 20. GUI130 has a main window 131 that displays the captured image and inference results taken in step S104, and the matching results in step S105. The label number labeled in step S101 can be confirmed in the label information window 132, and the processing result corresponding to the selected label number is displayed in the main window 131. Furthermore, if the model matching process in step S105 is completed successfully, the reference coordinates of the CAD model 22 of the labeling unit 21 are displayed as detected coordinates in the label information window 132, as shown in Figure 8. In addition, the detection result window 134 displays text such as "OK" to indicate that detection was successful, or text such as "NG" if detection failed. Furthermore, the assembly information window 133 displays table data TB showing assembly information corresponding to the label number of the labeling unit 21, as shown in Figure 8. By displaying GUI130 in this way, the user can determine whether the detection process of the workpiece 10 or its work area was performed successfully.

[0079] Furthermore, the configuration, operation, and effects of this fifth embodiment are the same as those of the first to fourth embodiments described above, so their explanation will be omitted.

[0080] <Sixth Embodiment> Next, the sixth embodiment will be described using Figures 21(a) to 22. Figures 21(a) and (b) show the position and orientation information of the CAD model 22 corresponding to the labeling section 21 in the actual workpiece 10 according to the sixth embodiment. Figure 22 shows the robot device 100 according to the sixth embodiment. In this description of the sixth embodiment, the same reference numerals are used for parts as in the various embodiments described above, and their descriptions are omitted. In this embodiment, the assembly work by the robot device 100 is performed without using assembly information such as table data TB as table information, as described in the embodiments described above. The control flow of this embodiment basically operates according to the processing flow of Figure 4, but the method of generating the correction trajectory in step S106 of Figure 4 is different.

[0081] The model matching in step S105 of the processing flow shown in Figure 4 is performed to obtain positional orientation information of the CAD model 22 corresponding to the labeling section 21 on the actual workpiece 10, as shown in Figure 21. This positional orientation information is obtained based on coordinate system O. Figure 21(a) shows the positional orientation information of the CAD model 22 corresponding to the labeling section 21 on the actual workpiece 10 based on coordinate system O. Figure 21(b) shows the workpiece 10. Based on this information, the robot device 100 is moved to the assembly position. In this case, the method for generating the correction trajectory and the coordinate transformation method for moving the robot device 100 are as described using Figure 14. From here, if assembly information such as table data TB is not used, the assembly direction must be determined in advance, but the assembly stroke, phase angle, insertion start position, and insertion completion position are detected by a force sensor 203 attached to the robot hand 202 as shown in Figure 22.

[0082] The force sensor 203 can, for example, detect external forces and moments applied to it in six axial directions, and can instruct the robot 200 to move in a straight line until a predetermined force is detected in each axial direction. In addition, depending on the case, known techniques such as admittance control and impedance control can be used based on the measured external forces until the robot arm 200 completes the assembly operation.

[0083] The assembly direction is predetermined in the program to operate the robot so that it moves in a predetermined direction based on the acquired position and orientation information of the CAD model 22. For example, if position and orientation information as shown in Figure 21 is acquired based on the actual workpiece 10, the predetermined direction is to move in the vertical direction of the Z axis in the coordinate system O of the CAD model 22, that is, in the -Z axis direction in the coordinate system O of the CAD model 22. The position and orientation information in Figure 21 is assumed to be the same as the position and orientation information shown in Figure 8. For explanatory purposes, the notation of the coordinate system O in Figure 21 has been changed from that in Figure 8. By step S105 shown in Figure 4, position and orientation information based on the coordinate system O as shown in Figure 21 can be acquired based on the actual workpiece 10, so a predetermined direction based on that coordinate system O is set in the program in advance as the assembly direction. In this way, even if the position and orientation of the actual workpiece 10 varies, position and orientation information based on the coordinate system O of the CAD model 22 that corresponds to the varied state can be acquired, so the workpiece 10 can be moved in the direction in which the part 11 is assembled.

[0084] Then, information regarding the force while moving the part 11 in the assembly direction is acquired, and a predetermined value is set as a threshold for determining that assembly is complete. By detecting when the force information reaches the predetermined value, assembly completion is detected, and the assembly operation can be completed. In addition, if the clearance accuracy between the workpiece 10 and the part 11 is high, the robot arm 200 may perform a searching motion while bringing the part 11, which is being moved, into contact with the workpiece 10 in order to align the phases of the workpiece 10 and the part 11. By detecting the force information while performing the searching motion, and determining that the phases are aligned when the force value reaches the predetermined value, the part 11 is moved in the assembly direction, making it possible to assemble even workpieces with high clearance accuracy.

[0085] As described above, according to this embodiment, by acquiring position and orientation information based on the coordinate system O of the actual workpiece 10 without using table data TB, it becomes possible to assemble the part 11 onto the workpiece 10. This reduces the number of parameters that need to be set in advance in the assembly process using the robot device 100, further reducing the burden of preparation. In addition, an automated production system that performs assembly work using the robot device 100 can be started up in a short time.

[0086] <Seventh Embodiment> Next, the seventh embodiment will be described using Figures 23 to 25. Figure 23 is a control block diagram for explaining control by a visual servo according to the seventh embodiment. Figures 24(a) and (b) are diagrams for explaining the method of generating image data corresponding to the target feature quantity according to the seventh embodiment. Figure 25 is a flowchart of the assembly work process by the robot device according to the seventh embodiment. In this description of the seventh embodiment, the same reference numerals are used for parts that are the same as those in the various embodiments described above, and their explanations are omitted. In this embodiment, the robot device 100 performs assembly work without using assembly information such as table data TB as table information, as described in the embodiments described above. The control flow of this embodiment basically operates according to the processing flow of Figure 4, except that the method of generating the correction trajectory in step S106 of Figure 4 is different. In this embodiment, the robot device 100 is controlled using a visual servo.

[0087] Figure 23 is a control block diagram illustrating the control in this embodiment. Figure 23 shows the basic control block of a known visual servo. In this embodiment, image data captured by a virtual camera 301 in a virtual space of the CAD model 22 of the labeling unit 21 is input as the target feature. The method for generating the image data corresponding to this target feature will be explained using Figure 24.

[0088] As shown in Figure 24, when using a visual servo, it is assumed that the camera 300 is a mobile camera mounted on the on-hand of the robot device 100, and Figure 24(a) shows such a configuration represented in virtual space. Figure 24(a) shows a diagram in which a virtual camera 301 is placed in virtual space. That is, it shows a state in which the relative positional relationship between the origin 319 of the camera coordinate system, the reference position 24 of the part 11, and the origin 23 of the coordinate system O which is the reference for the CAD model 22 corresponding to the labeling section 21 is set as known information. In this state, the part 11 can be assembled to the workpiece 10 based on the known relative positional relationship. In this state, a three-dimensional CAD model 22 of the labeled work area is placed so that its positional relationship matches that of the labeling section 21 in the CAD model 20 representing the entire workpiece 10, and the CAD model 22 of the labeling section 21 is imaged by the virtual camera 301. By doing so, image data representing the CAD model 22 in the virtual space, as shown in Figure 24(b), can be obtained. The image data obtained here becomes the target feature set shown in Figure 23.

[0089] Next, when actually operating the robot device 100, it is controlled as shown in the control flowchart in Figure 25. The control flowchart in Figure 25 is mainly executed by CPU 502 and CPU 204. From Figure 25, the processing of steps S301 to S303 is the same as the processing of steps S101 to S103 in Figure 4, but the processing from step S304 onwards differs from the flow in Figure 4.

[0090] In step S304, the CPU 204 captures images of the workpiece 10 using the actual camera 300. In step S305, the CPU 204 performs inference on the labeling section using the image data acquired by the actual camera 300, and in step S306, the CPU 204 calculates a control variable based on the inference result. The calculation of the control variable in step S306 corresponds to the calculation of the difference between the target feature and the current feature in Figure 23, and this result is sent to the feature-based controller shown in Figure 23 to become the control variable for controlling the robot. Various methods are known as control algorithms for the feature-based controller, including image Jacobian matrix operations based on known feature point extraction methods.

[0091] Next, in step S307, the CPU 204 controls the robot device 100 so that it gradually approaches the image data that will become the target feature. In step S308, the CPU 204 determines whether the difference between the target feature and the current feature has reached a predetermined target value (threshold). This target value may be a predetermined value or a defined range. If the answer to step S308 is Yes, the process proceeds to step S309. If the answer to step S308 is No, the process returns to just before step S304, and the robot control is repeated using the visual servo.

[0092] If the answer to step S308 is YES, then, as shown in Figure 24, the actual state of the robot device 100 will be such that the part 11 can be assembled onto the workpiece 10, based on the known information. Therefore, the CPU 204 moves the robot device 100 so that the reference position 24 aligns with the origin 23, based on the known information of the relative positional relationship between the origin 319, the reference position 24, and the origin 23. In this way, the assembly operation is completed.

[0093] In this embodiment, the feature point extraction method is shown to be an image difference method. In this embodiment, image data captured by a virtual camera 301 of the CAD model 22 of the labeling unit 21 is input as the target feature quantity, and the inference result of the labeling unit is input as the current feature quantity, and the difference between each feature quantity is calculated. However, various difference calculation methods are known for calculating feature points of interest from an image and associating feature points with high similarity, such as SIFT (Scale-Invariant Feature Transformation) and AKAZE (Accelerated KAZE). Thus, an algorithm that performs robot control from at least two or more image data may be appropriately adopted.

[0094] As described above, according to this embodiment, it is possible to assemble the part 11 onto the workpiece 10 by acquiring position and orientation information based on the coordinate system O of the actual workpiece 10 without using table data TB. Furthermore, in this embodiment, the relative positional relationship that makes assembly possible by the operation of the robot device 100 is set in virtual space, so the burden of prior preparation can be further reduced. As a result, an automated production system that performs assembly work using the robot device 100 can be started up in a short time. In this embodiment, the relative positional relationship is set in virtual space, but it is of course also possible to set it using the actual robot device 100. In this case, since the actual robot device 100 is used, it is possible to improve the accuracy of the assembly operation.

[0095] <Possibility of other embodiments> In the embodiments described above, a system was described that generates a three-dimensional CAD model in a virtual space based on CAD data. However, the system is not limited to this, and a system that generates a two-dimensional model is also acceptable.

[0096] Furthermore, in the first to fifth embodiments described above, the workpiece 10 and its work area (labeling section 21) are modeled as CAD models 20 and 22 from design information such as CAD data, but the invention is not limited to this. That is, the part 11 may also be modeled from design information such as CAD data and constructed as a CAD model. This makes it possible to virtually assemble the CAD model 22 of the labeling section 21 and the CAD model of the part 11 in a virtual space, and the trajectory of the robot arm 200 may be generated from the position and orientation in this virtual assembly. Moreover, it is also possible to model only the object held by the robot arm 200, such as the part 11 (i.e., the part 11 may also be called a workpiece). In this case, if the workpiece 10 placed on the workpiece stand 12, etc., is positioned in a known position and orientation, the model of the part 11 can be model-matched and the trajectory can be generated.

[0097] Furthermore, although the above-described embodiment describes a system that captures an image of the workpiece using a camera and generates image data of the actual image, it is not limited to this. Any device that can explore the workpiece in the direction of exploration and generate exploration data such as shape data including the work area of ​​the workpiece is acceptable, such as a tactile sensor, ultrasonic sensor, or probe.

[0098] Furthermore, while the above-described embodiment explained an example in which the part 11 is assembled to the work area of ​​the workpiece 10, it is not limited to this. For example, it may involve applying adhesive, paint, oil, etc., to the work area (area to be applied) of the workpiece. It may also involve attaching parts such as labels or stickers to the work area (area to be attached) of the workpiece. It may also involve performing work by bringing a tool such as a screwdriver or cutter into contact with the work area (area to be worked on) of the workpiece.

[0099] Furthermore, while the above-described embodiment explains how to generate a model in virtual space using CAD data when modeling a workpiece or work area (labeling section), it is not limited to this. For example, an operator or designer may manually generate a virtual model, such as a polygon model, in virtual space. Also, the design information is not limited to CAD data; it may simply be information where the position and size of the workpiece are numerically given.

[0100] Furthermore, although the above-described embodiment describes a method in which the trajectory of the robot arm 200 is generated in step S106 or step S207, it is not limited to this. For example, if the operator has previously taught and generated a rough trajectory for the robot arm 200, a corrected trajectory that corrects the trajectory created by the teaching may be generated in step S106 or step S207. In other words, the trajectory generation in step S106 or step S207 can be either the generation of a new trajectory or the generation of a corrected trajectory that corrects an existing trajectory.

[0101] Furthermore, although the above-described embodiment explains how machine learning is performed from multiple images including the CAD model 22 of the labeling unit 21 in step S103 or step S203, the invention is not limited to this. That is, the CAD model 22 generated in step S102 or step S202 may be virtually captured multiple times under different conditions (such as imaging angle and brightness) and used as a template image (target image). In this case, in step S104 or steps S204-1, S204-2, the labeling unit in the actual image is inferred from the template image, and a method such as template matching is performed in step S105 or step S206 can be considered.

[0102] Furthermore, in the embodiments described above, the robot arm 200 of the robot device 100 was described as a 6-axis articulated manipulator as an example, but it is not limited to this. For example, it may be a parallel link robot or a robot equipped with a mechanism for three-dimensional translation, in other words, any robot structure is acceptable. The present invention is also applicable to machines that can automatically perform actions such as extension and retraction, bending and straightening, vertical movement, horizontal movement, or rotation, or combinations thereof, based on information stored in a memory device provided in the control device.

[0103] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0104] The present invention is not limited to the embodiments described above, and many modifications are possible within the technical concept of the present invention. Furthermore, two or more embodiments from the above-described embodiments may be combined and implemented. In addition, the effects described in the embodiments are merely a list of the most preferred effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

[0105] <Summary of this embodiment> [Configuration 1] Robots and, A search unit that searches for a work area in a workpiece and acquires search data including information about the work area, The system includes a control unit that identifies the work area and controls the robot to perform work on the identified work area based on information relating to the work area, information relating to the work area and to perform work on the workpiece, and the exploration data. A robotic system characterized by the following features. [Configuration 2] The information regarding the aforementioned work area is a model obtained by modeling the work area from the design information used when designing the workpiece. The robot system according to configuration 1, characterized by the features described above. [Configuration 3] The information relating to the aforementioned operation includes at least one of the following: the assembly direction of the part, the assembly stroke amount of the part, the assembly phase angle of the part, the starting position of insertion of the part, the completed position of insertion of the part, and the insertion force of the part. A robot system according to configuration 1 or 2, characterized by the above. [Structure 4] The information related to the aforementioned work is set as table information. The robot system according to configuration 3, characterized by the above. [Composition 5] The claimed exploration data is data obtained from at least one of a camera, a tactile sensor, an ultrasonic sensor, and a probe. A robot system according to any one of configurations 1 to 4 characterized by the above. [Composition 6] The aforementioned model has a coordinate system that indicates the position and orientation of the model. Based on the aforementioned model and the information regarding the work area obtained based on the exploration data, information regarding the position and orientation of the work area of ​​the workpiece, defined by the coordinate system of the aforementioned model, is obtained. Based on the position and orientation information, the robot performs work on the work area of ​​the workpiece. A robot system according to any one of configurations 2 to 5, characterized by the above. [Composition 7] The robot is equipped with sensors that acquire information about force, Based on the position and orientation information and the force applied to the robot, the robot performs work on the work area of ​​the workpiece. A robot system according to any one of configurations 1 to 6 characterized by the above. [Structure 8] The exploration unit is equipped with an imaging device on the robot, Based on the position and orientation information, the reference position of the imaging device, and the reference position of the part to be moved by the robot, the robot performs the operation on the work area of ​​the workpiece. A robot system according to any one of configurations 1 to 7 characterized by the above. [Composition 9] The aforementioned exploration unit is equipped with an imaging device, Based on the position and orientation information, the work information, the coordinate system at a predetermined part of the robot, the coordinate system of the imaging device, and the coordinate system of the robot, the trajectory of the robot for performing work on the workpiece is acquired. A robot system according to any one of configurations 2 to 8, characterized by the above. [Configuration 10] The exploration unit acquires the exploration data by directing the exploration direction toward the work area. When the aforementioned model is explored from multiple angles, the characteristics of the work area are learned and acquired, and then the learned model information is obtained. The control unit identifies the work area based on the learning model information and the exploration data. A robot system according to any one of configurations 1 to 9 characterized by the above. [Composition 11] The exploration unit is an imaging device that takes images in the imaging direction and acquires image data including the characteristics of the work area. The aforementioned learning model information is information obtained by learning the characteristics of the work area when the model is imaged from multiple angles. The control unit identifies the work area based on the learning model information and the image data acquired by the imaging device. The robot system according to configuration 10, characterized by the above. [Composition 12] The control unit generates an inferred image of the work area from image data acquired by the imaging device based on the learning model information, and identifies the work area in the image data by matching the learning model information with the inferred image. The robot system according to configuration 11, characterized by the features described above. [Composition 13] The learning model information includes posture information relating to the posture of the model, The control unit, based on the posture information, identifies a matching region in the image data that matches the learning model information with the image data, and performs a matching process. A robot system according to any one of configurations 1 to 12, characterized by the above. [Composition 14] The imaging device has multiple cameras capable of three-dimensional measurement, The control unit identifies a matching region in the image data that matches the learning model information with the image data, based on the image data acquired by each of the multiple cameras, and performs a matching process. The robot system according to configuration 12, characterized by the features described above. [Composition 15] The control unit masks the image data to remove areas other than the matching region, and performs a matching process between the learning model information and the masked image data. The robot system according to configuration 13, characterized by the above. [Composition 16] The learning model information includes work information relating to the work performed on the work area, The control unit controls the robot based on the work information. The robot system according to configuration 10, characterized by the above. [Composition 17] The aforementioned operation is an assembly operation in which a part held by the robot is attached to the aforementioned area of ​​the workpiece. The robot system according to configuration 16, characterized by the above. [Composition 18] When performing work in the work area, the control unit controls the robot based on surrounding model information relating to a surrounding model that models surrounding objects arranged around the workpiece, and the exploration data. A robot system according to any one of configurations 1 to 17, characterized by the above. [Composition 19] When performing work in the work area, the control unit determines a trajectory that avoids interference between the robot and surrounding objects based on the surrounding model information and the exploration data, and controls the robot according to the determined trajectory. The robot system according to configuration 18, characterized by the above. [Configuration 20] When the control unit identifies the work area, it displays the identified work area on the display device. A robot system according to any one of configurations 1 to 19, characterized by the above. [Composition 21] An information processing device having a processing unit for acquiring the aforementioned model, A robot system according to any one of configurations 2 to 20, characterized by the features described herein. [Composition 22] A control method for a robot system comprising a robot, a search unit for searching a work area on a workpiece, and a control unit, The exploration unit acquires exploration data including information about the work area. The control unit identifies the work area based on information relating to the work area, information relating to the work area and to be performed on the workpiece, and the exploration data, and controls the robot to perform the work in the identified work area. A method for controlling a robot system characterized by the following features. [Composition 23] A method for manufacturing articles, characterized by manufacturing articles using a robot system described in any one of configurations 1 to 22. [Composition 24] A program for causing a computer to execute the control method of the robot system described in Configuration 22. [Composition 25] A recording medium readable by a computer that stores the program described in configuration 24. [Explanation of symbols]

[0106] 1…Robot system / 10…Workpiece / 21…Labeling unit (work area) / 22…CAD model (model) / 100…Robot device / 200…Robot arm (robot) / 204…CPU (control unit) / 300…Camera (exploration unit, imaging unit) / 320…Camera (exploration unit, imaging unit) / 330…Inference image / 331…Inference image / 501…Information processing device / 502…CPU (processing unit) / 508…Display (display device) / 520…Learning model information (model information) / TB…Table data (work information) / α…Solid angle (pose information)

Claims

1. Robots and, A search unit that searches for a work area in a workpiece and acquires search data including information about the work area, The system includes a control unit that identifies the work area and controls the robot to perform work on the identified work area based on information relating to the work area, information relating to the work area and to perform work on the workpiece, and the exploration data, The information regarding the work area is a model obtained by modeling the work area from the design information used when designing the workpiece. The exploration unit is an imaging device that takes images in the imaging direction and acquires image data including the characteristics of the work area as exploration data. The control unit generates an inferred image of the work area from image data acquired by the imaging device, based on the learned model information obtained by learning the characteristics of the work area when the model is imaged from multiple angles, and identifies the work area in the image data by matching the learned model information with the inferred image. A robotic system characterized by the following features.

2. The information relating to the aforementioned operation includes at least one of the following: the assembly direction of the part, the assembly stroke amount of the part, the assembly phase angle of the part, the starting position of insertion of the part, the completed position of insertion of the part, and the insertion force of the part. The robot system according to feature 1.

3. The information related to the aforementioned work is set as table information. The robot system according to claim 2, characterized in that it is the same as described in claim 2.

4. The aforementioned model has a coordinate system that indicates the position and orientation of the model. Based on the aforementioned model and the information regarding the work area obtained based on the exploration data, information regarding the position and orientation of the work area of ​​the workpiece, defined by the coordinate system of the aforementioned model, is obtained. Based on the position and orientation information, the robot performs work on the work area of ​​the workpiece. The robot system according to feature 1.

5. The robot is equipped with sensors that acquire information about force, Based on the position and orientation information and the force applied to the robot, the robot performs work on the work area of ​​the workpiece. The robot system according to feature 4.

6. The imaging device is provided on the robot, Based on the position and orientation information, the reference position of the imaging device, and the reference position of the part to be moved by the robot, the robot performs the operation on the work area of ​​the workpiece. The robot system according to feature 4.

7. Based on the position and orientation information, the work information, the coordinate system at a predetermined part of the robot, the coordinate system of the imaging device, and the coordinate system of the robot, a trajectory of the robot for performing work on the workpiece is obtained. The robot system according to feature 4.

8. The learning model information includes posture information relating to the posture of the model, The control unit, based on the posture information, identifies a matching region in the image data that matches the learning model information with the image data, and performs a matching process. The robot system according to feature 1.

9. The imaging device has multiple cameras capable of three-dimensional measurement, The control unit identifies a matching region in the image data that matches the learning model information with the image data, based on the image data acquired by each of the multiple cameras, and performs a matching process. The robot system according to feature 1.

10. The control unit masks the image data to remove areas other than the matching region, and performs a matching process between the learning model information and the masked image data. The robot system according to feature 8.

11. The learning model information includes, as information relating to the work to be performed on the work, work information relating to the work performed on the work area, The control unit controls the robot based on the work information. The robot system according to feature 1.

12. The aforementioned operation is an assembly operation in which a part held by the robot is attached to the work area of ​​the workpiece. The robot system according to feature 11.

13. When performing work in the work area, the control unit controls the robot based on surrounding model information relating to a surrounding model that models surrounding objects arranged around the workpiece, and the exploration data. The robot system according to feature 1.

14. When performing work in the work area, the control unit determines a trajectory that avoids interference between the robot and surrounding objects based on the surrounding model information and the exploration data, and controls the robot according to the determined trajectory. The robot system according to claim 13, characterized in that it is the robot system according to claim 13.

15. When the control unit identifies the work area, it displays the identified work area on the display device. The robot system according to feature 1.

16. An information processing device having a processing unit for acquiring the aforementioned model, The robot system according to feature 1.

17. A control method for a robot system comprising a robot, an imaging device for imaging a work area on a workpiece, and a control unit, The imaging device acquires image data including the characteristics of the work area. The control unit generates an inferred image of the work area from image data acquired by the imaging device, based on learned model information obtained by learning the features of the work area when the model acquired by modeling the work area from design information when the work area was designed is imaged from multiple angles, and by matching the learned model information and the inferred image, it identifies the work area in the image data and controls the robot to perform work on the identified work area based on information about the work to be performed on the work area. A method for controlling a robot system characterized by the following features.

18. A method for manufacturing an article, characterized by manufacturing the article using the robot system described in claim 1.

19. A program for causing a computer to execute the control method for the robot system described in claim 17.

20. A recording medium readable by a computer storing the program described in claim 19.

Images