Robot system, method of controlling robot system, method of manufacturing product, program product, and recording medium

Abstract

The invention discloses a robot system, a method of controlling the robot system, a method of manufacturing a product, a program product, and a recording medium. A robot system includes: a robot; a search unit configured to search for a work area of a workpiece on which work is performed, and obtain search data including information on the work area; at least one processor; and at least one memory in communication with the at least one processor. At least one memory stores instructions, the instructions are to cause the at least one processor and the at least one memory to identify the work area and control the robot to cause the robot to perform the work on the identified work area based on the information related to the work area, the information about the work related to the work area and performed on the workpiece, and the search data.

Description

Technical Field

[0001] The present disclosure relates to a robot device, a robot system, an information processing device, a method of controlling a robot device, a method of controlling a robot system, a method of manufacturing a product, a program product, and a recording medium. Background Art

[0002] Robotic devices such as industrial robots deployed in factories, etc. perform various types of work, such as assembling or attaching components to workpieces, applying adhesives or coatings to workpieces, and machining workpieces by using tools. By using a camera, etc. and identifying the image of the part to be performed on the workpiece, and by controlling the position and posture of the robot relative to the identified part of the workpiece, these works can be performed precisely (accurately) on the workpiece, regardless of the position and posture of the workpiece. However, in the case where such precise work is performed by a robotic device, when the robotic device is installed in a factory or when the work or workpiece is changed, a heavy burden will be imposed on the worker to adjust the robotic device. For example, in a method using template matching in image recognition, adjustments such as correction of brightness and feature extraction of image processing processes are performed to improve the matching accuracy between the template image and the captured image. However, since the adjustment is complicated, the worker will spend time setting the conditions. Therefore, a burden will be imposed on the worker.

[0003] To this end, Japanese Patent Application Publication No. 2020-197983 proposes a technology for calculating the position and angle of a workpiece. In this technology, a captured image of a workpiece is input into a learned learner to obtain two or more partially extracted images. Blob analysis is performed on the partially extracted images to create blob information. The position and angle of the workpiece are calculated based on the blob information. Therefore, the technology proposed in Japanese Patent Application Publication No. 2020-197983 reduces the burden of the above-mentioned adjustment work performed for the image processing process.

[0004] In addition, in the research described in "Recognition of Function of Objects and its Application to Robot Manipulation" (Manabu Hashimoto, Journal of the Robotics Society of Japan, Vol. 38, No. 6, pp. 525-529, 2020), the function (affordance) of an object is focused on, and a system for creating a learned model that three-dimensionally recognizes an area representing the function is proposed. The system aims to recognize a three-dimensional working area for a robot manipulator by using the learned model, and enable the robot manipulator to hold and transfer workpieces. Summary of the invention

[0005] Therefore, embodiments of the present disclosure are intended to reduce the burden on workers.

[0006] According to a first aspect of the present disclosure, a robot system includes: a robot; a search unit, the search unit being configured to search a working area of ​​a workpiece where work is to be performed and obtain search data containing information about the working area; at least one processor; and at least one memory, the at least one memory communicating with the at least one processor, wherein the at least one memory stores instructions for enabling the at least one processor and the at least one memory to identify the working area based on information related to the working area, information about the work related to the working area and performed on the workpiece, and the search data, and to control the robot to perform the work on the identified working area.

[0007] According to a second aspect of the present disclosure, a method for controlling a robot system includes a robot, a search unit for detecting a working area of ​​a workpiece where work is to be performed, at least one processor, and at least one memory, the method comprising: obtaining, by the search unit, search data containing information about the working area; and identifying, by the at least one processor and the at least one memory, the working area based on information related to the working area, information about the work related to the working area and performed on the workpiece, and the search data, and controlling the robot to perform the work on the identified working area.

[0008] Further features of various embodiments will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a diagram illustrating the configuration of the robot system of the first embodiment.

[0010] Figure 2 is a block diagram illustrating the configuration of the information processing apparatus of the first embodiment.

[0011] Figure 3 is a block diagram illustrating the configuration of the robot controller of the first embodiment.

[0012] Figure 4 is a flowchart illustrating the processing of assembling work performed by the robot system of the first embodiment.

[0013] Figure 5 is a perspective view illustrating one example of a CAD model of a workpiece.

[0014] Figure 6 is a perspective view illustrating one example of a marked portion of a workpiece.

[0015] Figure 7 is a perspective view illustrating an example of a marker model in which a marker portion is modeled.

[0016] Figure 8 This is a diagram illustrating an example of marking model information given to assembly information.

[0017] Fig. 9 is a diagram illustrating a state in which an image of a CAD model is captured by a virtual camera in a virtual space.

[0018] Fig. 10A is a diagram illustrating one example of a virtual CAD model image obtained by capturing an image of the CAD model by a virtual camera.

[0019] Fig. 10B is a diagram illustrating one example of a virtual area model image obtained by capturing an image of a marker model by a virtual camera.

[0020] Fig.11 is a diagram illustrating a learning process for learning image features of a label portion.

[0021] Fig.12 is a diagram illustrating an inference process for inferring a labeled portion from actual image data.

[0022] Fig.13 is a diagram illustrating a process for calculating the position of a marking portion in a camera coordinate system.

[0023] Fig.14 is a diagram illustrating a process for calculating the position of a marking portion in a robot coordinate system.

[0024] Fig.15 is a diagram illustrating the configuration of a robot system of a second embodiment.

[0025] Fig.16 is a flowchart illustrating the processing of assembling work performed by the robot apparatus of the second embodiment.

[0026] Fig.17 is a diagram illustrating a process for creating a three-dimensional point group image.

[0027] Fig.18 is a diagram illustrating a process for defining a solid angle of a workpiece according to the third embodiment.

[0028] Fig.19 is a diagram illustrating a learning process of the third embodiment for learning image features while associating the image features of a marking portion with the solid angle of a workpiece.

[0029] Fig. 20is a diagram illustrating one example of a GUI showing the result of model matching according to the fifth embodiment.

[0030] Figure 21A-21B 1 is a diagram illustrating position and posture information of a CAD model 22 corresponding to a marked portion 21 of an actual workpiece 10 according to the sixth embodiment.

[0031] Fig. 22 is a diagram illustrating the configuration of the robot apparatus of the sixth embodiment.

[0032] Fig.23 : is a control block diagram for illustrating control performed by using visual servoing according to the seventh embodiment.

[0033] Figure 24A-24B is a diagram for illustrating a method of creating image data corresponding to a target feature according to the seventh embodiment.

[0034] Fig.25 : is a flowchart illustrating the process of assembling work performed by the robot apparatus according to the seventh embodiment. DETAILED DESCRIPTION

[0035] In Japanese Patent Application Publication No. 2020-197983, although the technology can measure the position and angle of the workpiece on a two-dimensional plane, it is difficult to calculate the three-dimensional position and posture information of the workpiece from the blob information. Therefore, it may be difficult to automatically create the trajectory of the robot for the movement of the work, so it may take time to create (teach) the trajectory of the robot. Therefore, it may bring a burden to the worker to adjust the robot.

[0036] Furthermore, although the technology described in the above-mentioned "Recognition of Function of Objects and its Application to Robot Manipulation" can be applied to the type of work in which the robot is roughly moved, it is difficult to be applied to the type of work that requires the robot to perform precise work (with high recognition accuracy). That is, even in the case of using the above-mentioned technology, in precise assembly work, etc., it is necessary to set the precise operation (trajectory) of the robot arm performed on the working area of ​​the workpiece recognized by the technology. Since this setting is required to be performed by a worker with professional knowledge, it will also impose a heavy burden on the worker in this case.

[0037] First embodiment

[0038] In the following, reference will be made to Figures 1 to 14 A first embodiment for carrying out the present disclosure is described.

[0039] Schematic configuration of the robot system

[0040] First, refer to Figure 1 , Figure 2 and Figure 3 A schematic configuration of the robot system of the first embodiment is described. Figure 1 is a diagram illustrating the configuration of the robot system of the first embodiment. Figure 2 is a block diagram illustrating the configuration of the information processing apparatus of the first embodiment. Figure 3 is a block diagram illustrating the configuration of the robot controller of the first embodiment.

[0041] The robot system 1 is an automatic assembly system that assembles, for example, a component 11 serving as an assembly workpiece to a workpiece 10 serving as an assembled workpiece. The robot system 1 mainly includes a robot device 100 and an information processing device 501. The robot device 100 is fixed to and supported by a support 13, and includes a robot arm (manipulator) 200 serving as a robot and a robot controller 201 that controls the robot arm 200.

[0042] Furthermore, the robot apparatus 100 includes a robot hand 202 attached to the distal end of the robot arm 200 and serving as an end effector that holds (grips) the component 11. The shape and structure of the robot hand 202 are not limited to a specific shape and structure as long as the robot hand 202 can hold the component 11. For example, the robot hand 202 may have a structure that applies suction to the component 11. In another case, the robot hand 202 may include a force sensor or the like as needed.

[0043] The workpiece 10 is placed on the workpiece support 12 disposed on the support 13. The robot device 100 includes a camera 300 used as a search unit or an image capture device and disposed above the workpiece support 12 or the workpiece 10. The camera 300 captures an image including at least an image capture area (image capture range) of the workpiece 10, and obtains the image as image data of the actual image. The camera 300 may be a two-dimensional camera having a function of outputting two-dimensional image data, or may be a three-dimensional camera having a function of outputting three-dimensional image data, such as a stereo camera. Note that in the present embodiment, the description will be made for the case where the camera 300 is a fixed camera disposed on the ceiling of a factory, for example. However, if a handheld camera can capture an image including an image capture area of ​​the workpiece 10, the camera 300 may be a handheld camera fixed to the robot hand 202. That is, the camera may be disposed on the robot device 100 used as a robot. The image data captured by the camera 300 is sent to the robot controller 201 and is subjected to information processing described in detail below. Information processing refers to the calculation of command values ​​(eg, the trajectory of the robot arm) by the robot controller 201 for controlling the robot to assemble the component 11 to the workpiece 10 .

[0044] The robot system 1 configured as described above performs an assembly work in which the component 11 held by the robot hand 202 of the robot device 100 is assembled to a hole portion of the workpiece 10 as a work area described in detail below. In this way, the robot system 1 manufactures the workpiece 10 (on which the component 11 is assembled) as a product by using the robot device 100 and performing the assembly work in which the component 11 is assembled to the workpiece 10. In other words, the robot system 1 uses the robot device 100 and performs a method of manufacturing a product in which the component 11 is assembled to the workpiece 10.

[0045] Configuration of information processing device

[0046] Next, we will refer to Figure 2 The configuration of the information processing device 501 is described below. Figure 2 As shown in , the information processing device 501 includes a central processing unit (CPU) 502 as an example of a processor. The CPU 502 is an example of a processing unit. In addition, the information processing device 501 includes a read-only memory (ROM) 503, a random access memory (RAM) 504, and a hard disk drive (HDD) 505 used as a storage unit. In addition, the information processing device 501 includes a recording disk drive 506, a display 508, a keyboard 509, and a mouse 510. The display 508 is used as a display device as an input / output interface. The CPU 502, the ROM 503, the RAM 504, the HDD 505, the recording disk drive 506, the display 508, the keyboard 509, and the mouse 510 are connected to each other via a bus to communicate.

[0047] The ROM 503 stores basic programs related to the operation of the computer. The RAM 504 is a storage device that temporarily stores various types of data such as the results of calculation processing performed by the CPU 502. The HDD 505 stores various types of data such as the results of calculation processing performed by the CPU 502 and data obtained from external devices, and a program 507 that causes the CPU 502 to perform various types of processing described below. The program 507 is a program that allows the CPU 502 to perform the following preparatory processing ( Figure 4 ) is an application software for various types of processing related to the prior preparation processing. Therefore, the CPU 502 can execute various types of processing of the prior preparation processing described below by executing the program 507 stored in the HDD 505. In addition, the HDD 505 includes an area storing the learning model information 520. The learning model information 520 is model information obtained from the execution results of various types of processing of the prior preparation processing described below. The recording disk drive 506 reads various types of data and programs stored in the recording disk 550.

[0048] In the present embodiment, HDD 505 is a computer-readable non-transitory recording medium and stores program 507. However, some embodiments of the present disclosure are not limited thereto. Program 507 may be stored in any recording medium as long as the recording medium is a computer-readable non-transitory recording medium. For example, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a magnetic tape, a non-volatile memory, etc. may be used as a recording medium for providing program 507 to a computer.

[0049] The information processing device 501 is connected to the robot controller 201. As described in detail below, the information processing device 501 transmits the learning model information 520 to the robot controller 201 as a processing result obtained by performing various types of processing of the advance preparation processing.

[0050] Robot controller configuration

[0051] Next, we will refer to Figure 3 The configuration of the robot controller 201 is described below. Figure 3 As shown in FIG. 2 , the robot controller 201 includes a CPU 204 as an example of a processor. The CPU 204 is an example of a processing unit. The robot controller 201 also includes a ROM 205, a RAM 206, and a HDD 207 used as a storage unit. The robot controller 201 also includes a recording disk drive 208 and an interface 209 as an input / output interface. The CPU 204, the ROM 205, the RAM 206, the HDD 207, the recording disk drive 208, and the interface 209 are communicatively connected to each other via a bus.

[0052] The ROM 205 stores basic programs related to the operation of the computer. The RAM 206 is a storage device that temporarily stores various types of data such as the results of calculation processing performed by the CPU 204. The HDD 207 stores various types of data such as the results of calculation processing performed by the CPU 204 and data obtained from external devices and enables the CPU 204 to perform actual machine processing described below (see Figure 4 ) is a program 210 for various types of processing related to the actual machine processing described below. The program 210 is application software that allows the CPU 204 to execute various types of processing related to the actual machine processing described below. Therefore, the CPU 204 can control the movement of the robot arm 200 by executing the control processing by executing the program 210 stored in the HDD 207. In addition, the HDD 207 includes an area in which the learning model information 520 sent from the above-mentioned information processing device 501 is stored. The recording disk drive 208 reads various types of data and programs stored in the recording disk 250.

[0053] In the present embodiment, HDD 207 is a computer-readable non-transitory recording medium and stores program 210. However, some embodiments of the present disclosure are not limited thereto. Program 210 may be stored in any recording medium as long as the recording medium is a computer-readable non-transitory recording medium. For example, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a magnetic tape, a non-volatile memory, etc. may be used as a recording medium for providing program 210 to a computer.

[0054] The robot controller 201 is connected to the camera 300, the robot arm 200, and the above-mentioned information processing device 501. As described in detail below, the robot controller 201 receives the learning model information 520 transmitted from the information processing device 501. The learning model information 520 is a processing result obtained by performing various types of processing for the preparatory processing. The camera 300 captures image data and transmits the image data to the robot controller 201. The image data is processed by the program 210. The processing result is output as a command value for controlling the robot and is transmitted to the robot arm 200.

[0055] Note that, although the present embodiment will focus on the preparatory processing (see Figure 4 ) and the actual machine processing is performed by the robot controller 201 (ie, CPU 204) (see Figure 4 ), but some embodiments of the present disclosure are not limited thereto. The preparatory processing and the actual machine processing may be performed by a single computer or a single CPU, or may be performed by three or more computers or three or more CPUs. Figure 4 ) is assigned to and executed by multiple computers, any processing can be executed by any computer.

[0056] Assembly work handling

[0057] Next, we will refer to Figures 4 to 14 The robot system 1 described above is used to perform assembly work (such as Figure 1 Processing of assembly work by a robot (shown in ) (i.e., control of the robot system).

[0058] Pre-processing

[0059] First, refer to Figures 4 to 11 The preliminary preparation process performed by the above-mentioned information processing apparatus 501 will be described. Figure 4 is a flowchart illustrating the processing of assembling work performed by the robot system of the first embodiment. Figure 5 is a perspective view illustrating one example of a CAD model of a workpiece. Figure 6is a perspective view illustrating one example of a marked portion of a workpiece. Figure 7 is a perspective view illustrating an example of a marker model in which a marker portion is modeled. Figure 8 This is a diagram illustrating an example of marking model information given to assembly information. Fig. 9 is a diagram illustrating a state in which an image of a CAD model is captured by a virtual camera in a virtual space. Fig. 10A is a diagram illustrating one example of a virtual CAD model image obtained by capturing an image of the CAD model by a virtual camera. Fig. 10B is a diagram illustrating one example of a virtual area model image obtained by capturing an image of a marker model by a virtual camera. Fig.11 is a diagram illustrating a learning process for learning image features of a label portion.

[0060] like Figure 4 As shown in , the preliminary preparation process performed in steps S101 to S103 is a process for preliminary preparation performed before the robot device 100 is actually operated (that is, before the actual assembly work is performed). The preliminary preparation process is a process for creating the learning model information 520 by using a computer-aided design tool such as a computer-aided design (CAD) system. That is, as described in detail below, the preliminary preparation process creates the learning model information 520 by using the CAD data of the workpiece 10 which is the design information used when the workpiece 10 is designed.

[0061] In step S101, the CPU 502 performs work (hereinafter referred to as marking work) on the CAD model of the workpiece 10 on a computer-aided design tool (such as a CAD system). The CAD model has a data representation format that the computer-aided design tool can handle. Generally speaking, some formats such as STEP files, IGES files, and STL files are known. In the marking work, the identification Figure 5 The work area of ​​the CAD model 20 shown in FIG. 1 and corresponding to the workpiece 10 is marked. The work area is an area where the component 11 is brought into contact with and assembled thereto. The mark is, for example, an identification number that identifies the work area of ​​the CAD model 20 corresponding to the portion to which the component 11 is assembled. For example, data indicating a mark number 1 is added to the marked work area. By performing the above-mentioned processing, as shown in FIG. Figure 6 As shown in , a marking portion 21 corresponding to the marked working area is created. Note that in this process, if the workpiece 10 has a plurality of working areas with which other components or workpieces are in contact, the plurality of working areas can be marked and a plurality of marking portions can be formed. The marking portions can be provided with marking numbers 2, 3, ... for distinguishing the marking portions from each other.

[0062] In step S102, the CPU 502 performs modeling of the marking portion 21. Figure 7 As shown in FIG. 1 , in the modeling of the marked portion 21, the work area marked in step S101—that is, the marked portion 21 to which data indicating, for example, the mark number 1 is added—is modeled as a CAD model 22 as a new three-dimensional model. The CAD model 22 can be expressed by the above-mentioned format, such as STEP, IGES, or STL; and represents information related to the work area in this embodiment.

[0063] like Figure 8 As shown in , the CAD model 22 of the marked work area is assigned assembly information as information about the assembly work (that is, the assembly information is associated with the CAD model 22 and stored in the HDD 505). The assembly information is information about the work to be performed on the work area. In order to assign the assembly information to the CAD model 22, the CAD model 22 has a coordinate system O deployed at a predetermined position and used as a reference. The coordinate system O has axes x, y and z that are orthogonal to each other and represent a three-dimensional Euclidean space, and rotation components Rx, Ry and Rz as components around each axis. Therefore, the position and posture in the three-dimensional space can be expressed in the six-axis coordinate system of the robot device 100. Note that although the Cartesian coordinate space is described as an example in the present embodiment, the coordinate system O may have another system, such as a polar coordinate system or a quaternion, as long as the position and posture in the three-dimensional space can be expressed in the coordinate system O.

[0064] Then, if Figure 8 As shown in the table of , tabular data TB representing assembly information (i.e., work information) is created. Tabular data TB contains various types of data about assembly work (operation, function), such as information indicating the assembly direction in which the component is assembled to the workpiece, and information about the assembly stroke from the contact point to the point where the assembly is completed. In addition, tabular data TB also contains information about the assembly phase angle indicating the orientation of the component assembled to the workpiece, information about the insertion start position and insertion completion position of the component assembled to the workpiece, and the insertion force required to assemble the component to the workpiece. In this way, the CAD model 22 and the tabular data TB are stored in association with each other one-to-one (i.e., the CAD model 22 and the tabular data TB are associated with each other and stored in the HDD 505). Note that if the workpiece 10 has a plurality of marked work areas as described above, a plurality of CAD models corresponding to the plurality of marked work areas can be created, and a corresponding tabular data TB can be provided one-to-one for each of the plurality of CAD models.

[0065] In step S103, the CPU 502 learns the image features of the marked portion. Fig. 9As shown in , a computer-aided design tool (such as a CAD system) is used to create and use Figure 1 The camera 300 shown in FIG. 1 corresponds to a virtual camera 301 . In addition, an image of the three-dimensional CAD model 20 corresponding to the workpiece 10 and observed from the virtual camera 301 is captured.

[0066] Preferably, the settings of the virtual camera 301 are made the same as those of the camera 300 actually used by the robot system 1. For example, the unit size of the image capture device, the number of pixels, the focal length of the lens, the aperture of the virtual camera 301, etc. are made the same as those of the camera 300. With the settings performed in this way, as Fig. 10A As shown in , a virtual camera 301 can capture an image 32 of a CAD model (hereinafter referred to as a virtual image) in a virtual space of a computer-aided design tool. Since it is difficult to obtain texture information and shadows of the workpiece 10 that are the same as the actual image, these types of information are not necessary. The minimum required information is information about the outline representing the shape of the workpiece 10. Preferably, the outline information is the same as the outline of the CAD model 20.

[0067] Then, the three-dimensional CAD model 22 of the work area is deployed in the virtual space of the computer-aided design tool. The work area is marked so that the position of the work area is equal to the position of the marked portion 21 of the CAD model 20 of the workpiece 10. After that, the image of the CAD model 22 of the marked portion 21 is captured by the virtual camera 301 so that the image is obtained as shown in FIG. Fig. 10B The virtual image 33 shown in FIG. The virtual image 33 is image data from which the contour information of the CAD model 22 of the marked portion 21 can be obtained. Then Fig. 10A The virtual image 32 and Fig. 10B The virtual image 33 shown in FIG505 is stored to form a pair (ie, the virtual image 32 and the virtual image 33 are associated with each other and stored in the HDD 505).

[0068] Note that, in order to perform the following learning, it is necessary to obtain at least one pair of image data. However, it may be more preferable if more virtual images with different image capture angles and different brightness levels are obtained. In order to obtain multiple virtual images, the brightness of the virtual image and / or the texture of the workpiece 10 may be changed unless the contour information is lost. In addition, when capturing an image of the CAD model 20, the relative position between the virtual camera 301 and the CAD model 20 of the workpiece 10, or the relative position between the virtual camera 301 and the CAD model 22 of the marking portion 21 may be changed within a possible range. The possible range is the range in which the positional relationship between the camera 300 and the workpiece 10 can be shifted from each other in the actual robot system 1.

[0069] The image data pairs of the plurality of virtual images obtained in the above process are used as follows. Fig.11 In the learning process shown in . For example, each of the virtual image pairs with different image capturing angles and different brightness levels is compared with the above-mentioned table data TB (see Figure 8 ) is associated. In the present embodiment, a machine learning algorithm is used in the learning process. In particular, among the machine learning algorithms, an algorithm of supervised learning is used. Therefore, image data of a plurality of virtual images obtained in advance is training data D1 for supervised learning. In the training data D1, a virtual image obtained by capturing an image of a CAD model 20 of the workpiece 10 is input data D1A, and a virtual image obtained by capturing an image of a CAD model 22 of a marking portion 21 is output data D1B. Machine learning is performed so that the input data D1A and the output data D1B are associated with each other. As a result, the learning model information 520 is created as a learned model (creation process).

[0070] The algorithm of machine learning used in step S103 may be semantic segmentation or instance segmentation. Each of semantic segmentation and instance segmentation is a type of supervised learning, and is an algorithm that infers the output value of each pixel of input data D1A by performing machine learning based on training data D1. If learning is performed well, the contour information of the CAD model 22 of the marked part 21 can be obtained from the input data D1A. Note that the algorithm used for machine learning is not limited to semantic segmentation or instance segmentation, and if other algorithms have the function of extracting the above-mentioned features, it may be another algorithm other than semantic segmentation and instance segmentation. In step S103, the learning model information 520 obtained by performing learning is stored in, for example, HDD 505 used as a storage unit of the information processing device 501. In addition, the learning model information 520 is output so as to be transmitted to, for example, HDD 207 used as a storage unit of the robot controller 201; and is used for the actual machine processing described below.

[0071] Actual machine processing

[0072] Next, we will refer to Figure 4 , Fig.12 , Fig.13 and Fig.14 The actual machine processing performed by the robot controller 201 is described. Fig.12 is a diagram illustrating an inference process for inferring a labeled portion from actual image data. Fig.13 is a diagram illustrating a process for calculating the position of a marking portion in a camera coordinate system. Fig.14 is a diagram illustrating a process for calculating the position of a marking portion in a robot coordinate system.

[0073] like Figure 4 As shown in FIG. 1 , the actual machine processing performed in steps S104 to S106 is processing for operating the robot device 100 (i.e., for performing actual assembly work). In step S104, the CPU 204 infers the marking portion 21 from the actual image. Specifically, in a state where the workpiece 10 is placed on the workpiece holder 12 (see FIG. 1 ), the marking portion 21 is inferred from the actual image. Figure 1 ), the camera 300 captures an image of an area (i.e., an image capturing range) containing the workpiece 10. Note that the camera 300 functions as a search unit that searches for a working area of ​​the workpiece 10 where work is performed and obtains search data containing information about the working area. Image data of an actual image captured as search data is transmitted to the robot controller 201, and as Fig.12 , is used for the inference processing performed in the machine learning. Note that, in the case where the image of the workpiece 10 placed on the workpiece holder 12 is captured by the camera 300, the camera 300 is moved to a position above the workpiece 10, and the posture of the camera 300 is controlled by the robot arm 200 so that the image capturing direction of the camera 300 faces the workpiece 10. For example, the trajectory of the robot arm 200 used in this case is a trajectory to which the worker has previously taught the posture and position of the robot arm 200 by using a teaching pendant or the like.

[0074] Fig.12 The input data shown in is the image data of the actual image captured by the camera 300. The CPU 204 infers the output data by reading the above-mentioned learning model information 520 and using the same machine learning algorithm as the algorithm used in learning. If the learning model information 520 is the learning model information of the learned model that has already been learned, the output data becomes the image data corresponding to the contour information of the CAD model 22 of the marking part 21 (hereinafter referred to as the inferred image).

[0075] In step S105, the CPU 204 performs a model matching process on the inferred image obtained in step S104. In the matching process, the CAD model 22 of the marked part 21 created in step S102 is used. In this way, the position and posture of the working area of ​​the workpiece 10 are substantially recognized in the image data of the actual image captured by the camera 300. That is, in step S105, the working area is recognized by using the learning model information 520, the image data, and the CAD model 22 of the marked part 21 (recognition process).

[0076] Note that the image used in the matching process in step S105 may be the inferred image itself obtained in step S104. However, a portion of the captured image data corresponding to the area obtained by performing the inference (i.e., the matching area) may be extracted, and then the area other than the above portion in the image data may be determined as a masked area that has been subjected to the masking process, and then the matching process may be performed on the image data whose masked area has been subjected to the masking process. In short, the matching process may be performed on the learning model information 520 and the image data whose masked area has been subjected to the masking process. In this way, the burden of image processing may be reduced.

[0077] like Fig.13 As shown in FIG, if the matching is done well, the vector Vc of the marked portion 21 observed from the camera can also be determined by using known camera calibration techniques. w . Vector Vc w is the position information of the CAD model 22 in the three-dimensional space. Note that the vector Vc w From the origin 319 of the camera coordinate system extends to the origin 23 of the coordinate system O which is used as a reference for the CAD model of any marked part.

[0078] In step S106, the CPU 204 creates a trajectory for the robot arm 200 to assemble the component 11 to the workpiece 10 in a state where the robot arm 200 holds (grasps) the component 11. Specifically, Fig.14 As shown in FIG. 1 , first, the CPU 204 determines the vector Vc from the origin 220 of the coordinate system of the robot arm 200 to the origin 319 of the camera coordinate system by using a known hand-eye calibration technique. r Furthermore, since the robot arm 200 holds the component 11 via the robot hand 202, the CPU 204 determines the vector from the origin 220 of the coordinate system of the robot arm 200 to the predetermined reference position 24 of the component 11 as the vector Vr t ×Vt w '.

[0079] Note that the vector Vr t The vector Vr extends from the origin 220 of the coordinate system of the robot arm 200 to the origin 221 of the coordinate system of the robot hand 202. t It can be calculated by using any of a variety of known methods. For example, the vector Vr t The vector Vr can be calculated by using the values ​​from the encoders that detect the angles of the corresponding joints. The values ​​from the encoders are used by the robot arm 200 to calculate the position of the robot arm 200. In another case, the vector Vr can be determined by measuring the position of the robot hand from an image captured by, for example, a camera deployed outside. t . Vector Vt w' Extends from the origin 221 of the coordinate system of the robot hand 202 to the predetermined reference position 24 of the component 11. The vector Vt w ' can also be calculated by using any of various known methods. That is, the vector Vt w ' can be determined by performing measurements from the outside, or can be located mechanically.

[0080] In this way, the CPU 204 creates a trajectory Vw for the robot arm 200. w ', to move the component 11 to the marked portion 21 of the workpiece 10. That is, the robot arm 200 moves the component 11 on the trajectory Vw w ' until the assembly 11 starts to be assembled to the workpiece 10. Trajectory Vw w 'It is not limited to a straight line trajectory, and may be any trajectory as long as the start point and the end point do not change. For example, the path between the start point and the end point may be subjected to any interpolation processing, such as spline interpolation.

[0081] In addition, based on the assembly information included in the above-mentioned learning model information 520 (see Figure 8 ), the CPU 204 creates a trajectory from the position where the assembly of the component 11 to the workpiece 10 is started (insertion is started) to the position where the assembly is completed (insertion is completed). The above-mentioned table data TB contains information about the assembly direction, assembly stroke, assembly phase angle, insertion start position, insertion completion position, insertion force, etc. as information about the work performed on the workpiece. Therefore, the CPU 204 uses this information and creates a trajectory for assembling the component 11 to the workpiece 10 from the assembly start position. Then, the CPU 204 adds the trajectory for assembling the component 11 to the workpiece 10 to the trajectory Vw determined as described above and used to move the component 11 until the assembly of the component 11 to the workpiece 10 is started. w ' to create a trajectory for controlling the robot arm 200 in the assembly work. Note that the information on the work performed on the workpiece only needs to include at least one of the assembly direction of the component, the assembly stroke for assembling the component, the assembly phase angle of the component, the insertion start position of the component, the insertion completion position of the component, and the insertion force of the component.

[0082] After the trajectory is created, in step S106, in order to control the robot arm 200 in the assembly work, the CPU 204 outputs the trajectory as a command value to the robot arm 200, and drives the robot arm 200 so that the robot arm 200 moves on the trajectory. In this way, based on the information contained in the learning model information 520 (see Figure 8), the robot arm 200 moves the component 11 to the identified working area of ​​the workpiece 10, and assembles the component 11 to the workpiece 10 while controlling the position and posture of the component 11. That is, in step S106, the robot arm 200 is controlled so that work (work processing) is performed on the working area identified in step S105.

[0083] Summary of the First Embodiment

[0084] As described above, by causing the robot system 1 to execute Figure 4 The process of the assembly work shown in can automatically assemble the component 11 to the working area of ​​the workpiece 10 by the robot device 100 without imposing a heavy burden of adjustment work on the worker.

[0085] Specifically, in the present embodiment, in the process of step S101 to step S102, the workpiece 10 is modeled as the CAD model 20 in the virtual space of the CAD system. This operation can significantly reduce the work conventionally performed for the advance preparation. For example, the work for generating many template images by using the camera 300 and capturing the image of the workpiece 10 while changing the image capture angle can be significantly reduced, and the adjustment work for the image processing process such as performing focus correction and extracting features can be significantly reduced. Therefore, the burden of the adjustment work on the worker can be reduced.

[0086] Furthermore, in the present embodiment, not only the CAD model 20 in which the entirety of the workpiece 10 is modeled, but also the CAD model 22 of the marked portion 21 marked as the work area is created. Therefore, in the model matching process in step S105, the amount of calculation can be significantly reduced and the speed can be increased compared to the case where the matching process is performed on the CAD model 20 as the model of the entirety of the workpiece 10. Furthermore, the model matching process is not performed on the two-dimensional template image and the actual image, but on the three-dimensional CAD model 22 and the actual image. Therefore, the position and posture of the workpiece 10 can be determined three-dimensionally from the learning model information 520.

[0087] Furthermore, in the present embodiment, the image features of the label portion 21 are learned in the process of step S103. By this operation, the number of images of the CAD model 20 captured by the virtual camera 301 in the virtual space can be reduced compared to the case where many template images are prepared. Therefore, the burden of the prior preparation process can be reduced, and the burden of the adjustment work performed by the worker can be reduced. Furthermore, since the learning model information 520 as information about the learned model is created, the accuracy of the inference and model matching of the label portion 21 performed in steps S104 to S105 can be improved.

[0088] Furthermore, in the present embodiment, the CAD model 22 and the tabular data TB are stored in the learning model information 520, so that the CAD model 22 of the marking part 21 is associated with the tabular data TB which is the assembly information obtained from the CAD data (design information). Therefore, in the case of creating the trajectory of the robot arm 200 in step S106, by extracting the tabular data TB associated with the CAD model 22 which has been subjected to the matching process together with the actual image, the trajectory of the robot arm 200 can be automatically created with high accuracy. As a result, the assembly work of the robot device 100 can be performed with high accuracy. Furthermore, since the assembly information obtained from the CAD data (design information) is used, the worker does not need to prepare in advance a plurality of trajectories created according to the angles of the workpiece. As a result, the burden of the adjustment work performed by the worker can be reduced.

[0089] As described above, by causing the robot system 1 of the present embodiment to execute the processing of the assembly work, the burden of the adjustment work as the advance preparation can be reduced, and an automatic production system that executes the assembly work by using the robot device 100 can be started in a short time.

[0090] Note that in the first embodiment, the case where the CAD model 22 of the marking portion 21 is created and matching is performed between the CAD model 22 and the actual image in the model matching process has been described. However, some embodiments of the present disclosure are not limited thereto. For example, the CAD model 20 of the workpiece 10 may be created, and matching may be performed between the CAD model 20 and the actual image.

[0091] In addition, in the first embodiment, in order to recognize the modeled marker part 21 from the actual image, learning is performed by using a machine learning algorithm in steps S103 and S104. However, some embodiments of the present disclosure are not limited thereto. For example, the marker part 21 may be recognized by using a method other than learning.

[0092] Second embodiment

[0093] Next, we will refer to Figures 15 to 17 A second embodiment is described. In the second embodiment, a part of the above-described first embodiment is changed. Fig.15 is a diagram illustrating the configuration of a robot system of a second embodiment. Fig.16 is a flowchart illustrating the processing of assembling work performed by the robot apparatus of the second embodiment. Fig.17 is a diagram illustrating a process for creating a three-dimensional point group image. Note that in the description of the second embodiment, the same components as those of the above-described first embodiment are given the same symbols, and description thereof will be omitted.

[0094] Configuration of the Robot System of the Second Embodiment

[0095] like Fig.15 As shown in FIG. 1 , the robot system 1 of the second embodiment includes a second camera 320 in addition to the camera 300 (hereinafter referred to as the first camera). The second camera 320 serves as a search unit or an image capture device. The first camera 300 and the second camera 320 constitute a stereo camera and measure the actual workpiece 10 three-dimensionally. Note that in FIG. Fig.15 In the robot system 1 shown in FIG. 2 , as an example, the first camera 300 and the second camera 320 are fixed cameras. However, the first camera 300 and the second camera 320 may be handheld cameras mounted on the robot hand 202. In addition, the number of cameras used as image capture devices is not limited to two, and may be three or more. For example, the robot system 1 may include both a fixed camera and a handheld camera, and one or both of the fixed camera and the handheld camera may be a stereo camera composed of two cameras.

[0096] Processing of Assembling Work of Second Embodiment

[0097] Next, the processing of the assembly work performed by the robot system 1 of the second embodiment will be described. Fig.16 As shown in FIG. 1 , steps S201, S202, and S203 as the preliminary preparation process performed before the robot device 100 is actually operated are similar to the steps S201, S202, and S203. Figure 4 The above-mentioned steps S101, S102 and S103 shown in are the same. However, in the learning in step S203, the learning model information 520 can be created from the virtual images of the CAD model 22 captured by the virtually arranged first camera 300 and the second camera 320. In another case, the common learning model information 520 can be created. In the case of creating the common learning model information 520, the input data D1A includes the image data of the CAD model 22 captured by both the virtually arranged first camera 300 and the second camera 320.

[0098] Next, the actual machine processing performed in the processing of the assembly work of the second embodiment will be described. In the second embodiment, after the actual machine processing starts, the CPU 204 causes the first camera 300 and the second camera 320 to capture images of the workpiece 10 in steps S204-1 and S204-2. Then, the CPU 204 infers the marking portion 21 from the image data captured by the first camera 300 and the image data captured by the second camera 320. In this case, as described above, the learning model information 520 dedicated to the image data captured by the first camera 300 and the learning model information 520 dedicated to the image data captured by the second camera 320 may be used, or the common learning model information 520 may be used. In any case, the marking portion 21 is inferred from the image data captured by the first camera and the image data captured by the second camera 320.

[0099] In step S205, the CPU 204 performs three-dimensional measurement. The three-dimensional measurement can be performed by performing a known stereo calibration on the first camera 300 and the second camera 320 using the triangulation principle. Fig.17 As shown in FIG. 2 , based on the inferred image 330 from the first camera 300 and the inferred image 331 from the second camera 320, a three-dimensional point group image 340 can be obtained by performing three-dimensional measurement using a known method such as block matching. In step S206, a model matching process is performed on the three-dimensional point group image 340 and the marker part 21. With this operation, the position and posture of the marker part 21 and the assembly information (see FIG. 204 ) can be obtained. Figure 8 After obtaining the position and posture of the marking part 21 and the assembly information, the CPU 204 creates a trajectory in step S207. The process of creating the trajectory is the same as the process of the above-mentioned step S106. In this way, the CPU 204 creates the trajectory of the robot arm 200.

[0100] Note that the image used to perform three-dimensional measurement in step S205 may be the inferred image itself obtained in step S204. However, a portion of the captured image data corresponding to the area obtained by performing the inference (i.e., the matching area) may be extracted, and then the area other than the above portion in the image data may be determined as the masked area subjected to the masking process, and then three-dimensional measurement may be performed on the image data on which the masked area has been subjected to the masking process. That is, the CPU 204 may obtain the three-dimensional point group image 340 from the image data on which the masked area has been subjected to the masking process, and may perform model matching processing on the three-dimensional point group image 340 and the marked portion 21. In short, broadly speaking, matching processing may be performed on the learning model information 520 and the image data on which the masked area has been subjected to the masking process. In this way, the burden of image processing may be reduced.

[0101] Furthermore, the matching process may be performed after performing preprocessing on the inferred image 330 from the camera 300 and the inferred image 331 from the camera 320. In the preprocessing, noise generated by performing the inference may be removed, and linear approximation or elliptical approximation may be performed.

[0102] Overview of Second Embodiment

[0103] As described above, in the processing of the assembly work performed by the robot system 1 of the second embodiment, the inferred image 330 and the inferred image 331 are obtained by the first camera 300 and the second camera 320 constituting the stereo camera. Then, the three-dimensional point group image 340 is created from the inferred image, and the model matching process is performed on the three-dimensional point group image 340. In this way, the working area of ​​the actual workpiece 10 can be recognized with high accuracy.

[0104] Note that since the other configuration, operation, and effects of the second embodiment are the same as those of the above-described first embodiment, description thereof will be omitted.

[0105] Third embodiment

[0106] Next, we will refer to Figure 18 to Figure 19 A third embodiment is described. In the third embodiment, a part of the first embodiment and the second embodiment described above is changed. Fig.18 is a diagram illustrating a process for defining a solid angle of a workpiece according to the third embodiment. Fig.19 1 is a diagram illustrating a learning process of the third embodiment for learning the image feature of the marking portion while associating the image feature with the solid angle of the workpiece. Note that, also in the description of the third embodiment, the same components as those of the first and second embodiments described above are given the same symbols, and their description will be omitted.

[0107] In the third embodiment, when the CPU 502 learns the image features of the marking portion 21 in the above-mentioned step S103, the CPU 502 also learns the posture information in addition to the position of the contour of the CAD model 20 of the workpiece 10. Specifically, Fig.18 As shown in , a solid angle α is defined as posture information in a virtual space relative to a reference vector VA set in the CAD model 20. The solid angle α indicates in which direction the normal line NL of the CAD model 20 faces. That is, the solid angle α is defined for the posture of each of a plurality of CAD models 20 whose images are captured in a virtual space by a virtual camera 301 and have different image capturing angles. The solid angle α is associated with each CAD model 20, and the solid angle α and the corresponding CAD model 20 are included in the learning model information 520 as posture information.

[0108] That is, Fig.19 As shown in , the CPU 502 learns the output data D1B while associating the output data D1B with the value of the solid angle α. In order to perform learning in this way, the CPU 502 marks the information of the solid angle α (posture information) for the output data D1B, and performs learning by using an algorithm (such as the above-mentioned instance segmentation). If the learning is performed well, the CPU 502 also infers the solid angle α when inferring the marked part 21 in step S104. Therefore, when the model matching process is performed in the next step S105, the CPU 502 can perform matching on an image with a limited angle, for example. Therefore, the risk of mismatching can be reduced.

[0109] Note that since other configurations, operations, and effects of the third embodiment are the same as those of the above-described first and second embodiments, descriptions thereof will be omitted.

[0110] Fourth embodiment

[0111] Next, a fourth embodiment will be described. In the fourth embodiment, a portion of the first to third embodiments described above is changed. Note that in the description of the fourth embodiment, components identical to those of the first to third embodiments described above are given the same symbols, and description thereof will be omitted.

[0112] In the fourth embodiment, in the case of creating the trajectory of the robot arm 200 in the above-mentioned step S106, objects other than the marking portion 21 are modeled for creating a trajectory in which the robot arm 200 does not interfere with the workpiece 10 itself and any surrounding objects other than the workpiece 10. That is, in step S102, as an example, a surrounding model is created from the CAD data by modeling not only the marking portion 21 (working area) which is a portion where the component 11 is assembled, but also surrounding objects disposed in and around the workpiece 10. Then, surrounding model information as information about the surrounding model is created. Note that the model other than the marking portion 21 may not be used in conjunction with the workpiece 10. Figure 8 The process other than the above process is related to the processing of the above-mentioned assembly work (see Figure 4 ). In this way, the CPU 502 creates a surrounding model of the surrounding objects other than the marking portion 21, performs a model matching process, determines a trajectory (surrounding model) in which the robot arm 200 does not interfere with the surrounding objects, and generates a trajectory of the robot arm 200 based on the trajectory determined by the CPU 502. In this way, when the CPU 502 creates the trajectory of the robot arm 200, the CPU 502 can create a trajectory of the robot arm 200 in which the robot arm 200 does not interfere with the workpiece 10 and objects around the workpiece 10.

[0113] Note that since other configurations, operations, and effects of the fourth embodiment are the same as those of the above-described first to third embodiments, descriptions thereof will be omitted.

[0114] Fifth embodiment

[0115] Next, we will refer to Fig. 20 A fifth embodiment is described. In the fifth embodiment, a part of the above-described first to fourth embodiments is changed. Fig. 20 1 is a diagram illustrating an example of a GUI showing the result of model matching of the fifth embodiment. Note that, also in the description of the fifth embodiment, the same components as those of the first to fourth embodiments described above are given the same symbols, and their description will be omitted.

[0116] In the fifth embodiment, Figure 4The progress in the process of the assembly work shown in is displayed as a graphical user interface (GUI) 130. For example, the GUI 130 can be displayed on a display device, such as a display 508 connected to the information processing device 501. In the present embodiment, as an example, a case where the CPU 204 of the robot controller 201 creates the GUI 130 and transmits the GUI 130 to the information processing device 501 and the display 508 displays the GUI 130 will be described. However, some embodiments of the present disclosure are not limited to this. For example, the CPU 204 can create the GUI 130, and can transmit the GUI 130 to a display device directly connected to the robot controller 201. In another case, the CPU 204 of the robot controller 201 can calculate various types of data and transmit the data to the information processing device 501, and the CPU 502 can create the GUI 130 and cause the display 508 to display the GUI 130.

[0117] Reference Fig. 20 An example of the GUI 130 is described. The GUI 130 includes a main window 131 that displays the image captured in the above step S104, the result of the inference performed in the above step S104, and the result of the matching performed in step S105. In addition, the tag number obtained by performing the tagging in step S101 can be checked in the tag information window 132. The processing result corresponding to the tag number selected in the tag information window 132 is displayed in the main window 131. If the model matching processing is performed well in step S105, then Figure 8 The reference coordinates of the CAD model 22 of the marking part 21 shown in FIG. 1 are displayed as the detection coordinates in the marking information window 132. In the detection result window 134, letters such as "OK" are displayed if the detection is successfully performed, or letters such as "NG" are displayed if the detection fails. In the assembly information window 133, letters such as "OK" are displayed if the detection fails. Figure 8 The tabular data TB shown in FIG. 1 and representing assembly information corresponding to the marking number of the marking portion 21. Since the GUI 130 is displayed as described above, the user can determine whether the workpiece 10 or the working area of ​​the workpiece 10 has been successfully inspected.

[0118] Note that since other configurations, operations, and effects of the fifth embodiment are the same as those of the above-described first to fourth embodiments, descriptions thereof will be omitted.

[0119] Sixth embodiment

[0120] Next, we will refer to FIG. 21A to FIG. 22 A sixth embodiment is described. Fig.21A and Fig.21B1 is a diagram illustrating position and posture information of a CAD model 22 corresponding to a marked portion 21 of an actual workpiece 10 according to the sixth embodiment. Fig. 22 1 is a diagram illustrating a robot apparatus 100 of a sixth embodiment. Note that in the description of the sixth embodiment, the same components as those of the various embodiments described above are given the same symbols, and descriptions thereof will be omitted. In this embodiment, the robot apparatus 100 performs assembly work without using any assembly information (such as the table data TB described in the above embodiments) as table information. The control flow of this embodiment is basically based on Figure 4 The processing flow shown in Figure 4 The method of creating the trajectory performed in step S106 is different.

[0121] CPU 204 executes Figure 4 The model matching in step S105 of the processing flow shown in FIG. Figure 21A-21B As shown in FIG. 1 , the position and posture information of the CAD model 22 corresponding to the marked portion 21 of the actual workpiece 10 is obtained. The CPU 204 obtains the position and posture information based on the coordinate system O. Fig.21A The position and posture information of the coordinate system O based on the CAD model 22 corresponding to the marked portion 21 of the actual workpiece 10 is illustrated. Fig.21B The workpiece 10 is shown. Based on this information, the CPU 204 moves the robot device 100 to the assembly position. In this case, the method of creating a trajectory for moving the robot device 100 and the coordinate transformation method are the same as those of the reference Fig.14 In the case where assembly information such as table data TB is not used, it is required to determine the assembly direction in advance. After the robot device 100 is moved to the assembly position, as shown in FIG. Fig. 22 As shown in , the assembly stroke, phase angle, insertion start position, and insertion completion position are detected by a force sensor 203 attached to the robot hand 202.

[0122] The force sensor 203 detects external forces or moments applied from the outside in six axial directions respectively; and allows the linear movement of the robot arm 200 until a predetermined level of force is sensed in each direction. In another case, a known technique such as admittance control or impedance control performed based on the measured external force may be used until the robot arm 200 completes the assembly operation.

[0123] The assembly direction is predetermined in a program for moving the robot so that the assembly can be moved in a specified direction based on the position and posture information of the obtained CAD model 22. For example, in the program for moving the robot so that the assembly can be moved in a specified direction based on the position and posture information of the obtained CAD model 22, the assembly direction is predetermined in a program for moving the robot so that the assembly can be moved in a specified direction based on the position and posture information of the obtained CAD model 22. Figure 21A-21BIn the case of the position and posture information shown, the specified direction is the vertical Z-axis direction in the coordinate system O of the CAD model 22. That is, the component moves in the -Z-axis direction in the coordinate system O of the CAD model 22. Figure 21A-21B The position and posture information shown is consistent with Figure 8 The position and posture information shown in are the same. However, for ease of description, Figure 21A-21B The coordinate system O shown in Figure 8 The coordinate system O shown in is expressed differently. Figure 4 In step S105 shown in FIG. 1 , based on the actual workpiece 10, the Figure 21A-21B , the position and posture information based on the coordinate system O is shown in , so the specified direction relative to the coordinate system O is set as the assembly direction in advance in the program. Therefore, even if the position and posture of the actual workpiece 10 to which the component is to be assembled changes, the position and posture information of the CAD model 22 corresponding to the workpiece 10 whose position and posture have changed can be obtained in the coordinate system O. As a result, the component 11 can be moved in the direction in which the component 11 is assembled to the workpiece 10.

[0124] The CPU 204 obtains information about the force applied to the force sensor 203 when the component 11 moves toward the assembly direction. In addition, the CPU 204 sets a predetermined value in advance as a threshold value for determining that the assembly is completed. If the CPU 204 detects that the information about the force has reached the predetermined value, the CPU 204 determines that the assembly is completed and completes the assembly operation. In the case where the gap between the workpiece 10 and the component 11 is formed with high accuracy, the position of the component 11 moved by the robot arm 200 can be adjusted in a state where the component 11 is in contact with the workpiece 10, so as to make the phase of the component 11 the same as the phase of the workpiece 10. In this case, the CPU 204 detects information about the force applied to the force sensor 203 when adjusting the position of the component 11. If the force value becomes equal to the predetermined value, the CPU 204 determines that the phase of the component 11 has become the same as the phase of the workpiece 10, and moves the component 11 toward the assembly direction. In this way, even if a gap is formed between the workpieces with high accuracy, one workpiece can be assembled to another workpiece.

[0125] As described above, in the present embodiment, the position and posture information is obtained in the coordinate system O of the actual workpiece 10 without using the table data TB, so that the component 11 can be assembled to the workpiece 10. In this configuration, the number of parameters set in advance in the assembly process using the robot device 100 can be reduced. As a result, the burden of prior preparation can be further reduced. In addition, the automatic production system that performs assembly work by using the robot device 100 can be started in a short time.

[0126] Seventh embodiment

[0127] Next, we will refer to Figure 23 to Figure 25 A seventh embodiment is described. Fig.23 : is a control block diagram for illustrating control performed by using visual servoing according to the seventh embodiment. Fig.24A and Fig. 24B is a diagram for illustrating a method of creating image data corresponding to a target feature according to the seventh embodiment. Fig.25 1 is a flowchart illustrating the processing of the assembly work performed by the robot apparatus of the seventh embodiment. Note that in the description of the seventh embodiment, the same components as those of the various embodiments described above are given the same symbols, and the description thereof will be omitted. In the present embodiment, the robot apparatus 100 performs the assembly work without using any assembly information (such as the table data TB described in the above embodiments) as table information. The control flow of the present embodiment is basically based on Figure 4 The processing flow shown in is executed, but Figure 4 In the present embodiment, the robot device 100 is controlled by using visual servoing.

[0128] Fig.23 is a control block diagram illustrating schematic control of this embodiment. Fig.23 3 is a diagram illustrating a basic control block diagram of known visual servoing. In this embodiment, image data of a CAD model 22 of a marking portion 21 captured by a virtual camera 301 in a virtual space is input as a target feature. Figure 24A-24B Describes a method for creating image data corresponding to a target feature.

[0129] When using visual servoing, such as Fig.24A As shown in , the camera 300 is used as a mobile camera mounted on a robot hand of the robot device 100 . Fig.24A The figure shows a state in which the camera 300 is used as a mobile camera mounted on a robot hand in a virtual space. Fig.24A In the example, the virtual camera 301 is deployed in the virtual space. That is, Fig.24A The diagram shows a state in which the relative positional relationship between the origin 319 of the camera coordinate system, the reference position 24 of the component 11, and the origin 23 of the coordinate system O used as a reference for the CAD model 22 corresponding to the marked portion 21 is set as known information. In this state, the component 11 can be assembled to the workpiece 10 based on the known relative positional relationship. In this state, the three-dimensional CAD model 22 of the work area marked so that the position of the work area is the same as the position of the marked portion 21 of the CAD model 20 representing the entire workpiece 10 is deployed, and an image of the CAD model 22 of the marked portion 21 is captured by the virtual camera 301. As a result, as shown in FIG. Fig. 24BAs shown in , image data representing the CAD model 22 in the virtual space can be obtained. The image data obtained in this way is Fig.23 The target features shown in .

[0130] In the case of actually operating the robot device 100, according to Fig.25 The robot device 100 is controlled by the control flowchart shown in FIG. Fig.25 The control flow chart shown in is mainly executed by CPU 502 or CPU 204. Fig.25 In the above, the processing in steps S301 to S303 is the same as that in steps S101 to S103, but the processing in and after step S304 is the same as that in step S306. Figure 4 The process is different.

[0131] In step S304, the CPU 204 causes the actual camera 300 to capture an image of the workpiece 10. In step S305, the CPU 204 infers the marked portion of the image data captured by the actual camera 300. In step S306, the CPU 204 calculates the control amount based on the result of the inference. The calculation of the control amount performed in step S306 is similar to the calculation for calculating the control amount of FIG. Fig.23 The result of the calculation is sent to the working correspondence of the difference between the target feature and the current feature shown in . Fig.23 The feature-based controller shown in is used as the control quantity for controlling the robot. Note that the control algorithm of the feature-based controller can be any of various known methods based on known feature extraction methods and including image Jacobian matrix calculation.

[0132] In step S307, the CPU 204 controls the robot device 100 so that the image data captured by the actual camera 300 gradually looks like the image data used as the target feature. In step S308, the CPU 204 determines whether the difference between the target feature and the current feature reaches a predetermined target value (threshold value). The target value may be a predetermined value, or may be a predetermined range. If the target value is reached (step S308: Yes), the CPU 204 proceeds to step S309. If the target value is not reached (step S308: No), the CPU 204 returns to the beginning of step S304 and repeats the control of the robot by using visual servoing.

[0133] If the target value is reached (step S308: Yes), then Figure 24A-24B As shown in , the actual robot apparatus 100 is in a state where the component 11 can be assembled to the workpiece 10 based on the known information. Therefore, the CPU 204 moves the robot apparatus 100 based on the relative positional relationship between the origin 319, the reference position 24, and the origin 23 so that the reference position 24 becomes the same as the origin 23. With this operation, the assembling work is completed.

[0134] Note that in the present embodiment, the feature extraction method is performed by executing a method for detecting the difference between images. In the present embodiment, the difference in features is determined by inputting the image data of the CAD model 22 of the marking part 21 captured by the virtual camera 301 as the target feature, and by inputting the result of the inference of the marking part as the current feature. However, the method of calculating the difference between the images may be any of various known methods for calculating the difference, such as scale-invariant feature transform (SIFT) or accelerated KAZE (AKAZE), which calculates the focus feature from the image and associates the feature of one image with the feature of another image (these features have a high similarity to each other). Therefore, an algorithm for performing robot control by using at least two or more pieces of image data can be appropriately used.

[0135] As described above, in the present embodiment, the position and posture information is obtained in the coordinate system O of the actual workpiece 10 without using the table data TB, so that the component 11 can be assembled to the workpiece 10. In addition, in the present embodiment, since a relative position relationship for allowing the robot device 100 to perform the assembly work is set in the virtual space, the burden of prior preparation can be further reduced. Therefore, an automatic production system that performs the assembly work by using the robot device 100 can be started in a short time. Note that, although the above-mentioned relative position relationship is set, in the present embodiment, the relative position relationship can be set by using the actual robot device 100 in the virtual space. In this case, since the actual robot device 100 is used, the accuracy of the assembly work can be improved.

[0136] Feasibility of other embodiments

[0137] Note that in the description of the first to fifth embodiments described above, a three-dimensional CAD model created in a virtual space based on CAD data has been described. However, some embodiments of the present disclosure are not limited thereto. For example, a two-dimensional model may be created.

[0138] In addition, although the first to fifth embodiments described above have been directed to the case where the workpiece 10 and the working area of ​​the workpiece 10 (i.e., the marking portion 21) are modeled as the CAD model 20 and the CAD model 22 by using design information (such as CAD data), some embodiments of the present disclosure are not limited thereto. That is, the component 11 may also be modeled as a CAD model by using design information (such as CAD data). In this case, since the CAD model of the component 11 can be virtually assembled to the CAD model 22 of the marking portion 21 in a virtual space, the trajectory of the robot arm 200 can be created by using the position and posture obtained in the virtual assembly. In another case, only an object such as the component 11 (the component 11 may be referred to as a workpiece) held by the robot arm 200 may be modeled. In this case, if the workpiece 10 placed on the workpiece holder 12 or the like is positioned at a known position and has a known posture, a trajectory may be created by performing model matching on the model of the component 11.

[0139] In addition, although the first to fifth embodiments described above have been directed to the case where image data of an actual image is created by causing a camera to capture an image of a workpiece, some embodiments of the present disclosure are not limited thereto. For example, another component such as a tactile sensor, an ultrasonic sensor, or a probe may be used as long as the component can search for the workpiece in a search direction and create search data (such as shape data including shape data of a working area of ​​the workpiece).

[0140] In addition, although the first to fifth embodiments described above have been described as an example for the case where the component 11 is assembled to the working area of ​​the workpiece 10, some embodiments of the present disclosure are not limited thereto. For example, during operation, adhesives, coatings, oils, etc. may be applied to the working area (i.e., the coating area) of the workpiece. In another case, during operation, components such as labels or seals may be pasted to the working area (i.e., the pasting area) of the workpiece. In another case, during operation, tools such as drivers or cutters may be pressed against the working area (i.e., the machining area) of the workpiece.

[0141] In addition, although the first to seventh embodiments described above have been directed to creating a model of a workpiece or a work area (i.e., a marking portion) in a virtual space by using CAD data, some embodiments of the present disclosure are not limited thereto. For example, a virtual model (such as a polygonal model) may be manually created in a virtual space by a worker or designer. In addition, the design information is not limited to CAD data. For example, the design information may be information that simply gives numerical values ​​of the position and size of the workpiece.

[0142] In addition, although the above-mentioned first to seventh embodiments have been described for the case where the trajectory of the robot arm 200 is created in step S106 or S207, some embodiments of the present disclosure are not limited thereto. For example, in the case where the worker creates a rough trajectory of the robot arm 200 in advance, a corrected trajectory in which the trajectory created by performing teaching is corrected may be created in step S106 or S207. That is, when creating a trajectory in step S106 or S207, a newly created trajectory may be created, or a corrected trajectory in which an existing trajectory is corrected may be created.

[0143] In addition, although the above-mentioned first to seventh embodiments have been described for the case where machine learning is performed by using multiple images of the CAD model 22 including the marked portion 21 in step S103 or S203, some embodiments of the present disclosure are not limited thereto. That is, multiple images of the CAD model 22 (created in step S102 or S202) virtually captured under different conditions (e.g., image capture angle and brightness) may be used as template images (target images). In this case, the marked portion of the actual image may be inferred from the template image in step S104 or steps S204-1 and S204-2, and template matching may be performed in step S105 or S206.

[0144] In addition, although the robot arm 200 of the robot device 100 is a six-axis articulated manipulator in the above-mentioned first to seventh embodiments, some embodiments of the present disclosure are not limited thereto. For example, the robot arm 200 of the robot device 100 may be a parallel link robot or a robot including a mechanism for three-dimensional translation. That is, the robot arm 200 of the robot device 100 may be a robot having any structure. In addition, some embodiments of the present disclosure may be applied to any machine that can automatically perform telescopic motion, bending and stretching motion, up-and-down motion, left-and-right motion, pivoting motion, or a combination thereof according to information data stored in a storage device of a control device.

[0145] The present disclosure may also be implemented by providing a program that performs one or more functions of the above-mentioned embodiments to a system or device via a network or storage medium and one or more processors included in a computer of the system or device read and execute the program. In addition, some embodiments of the present disclosure may also be implemented by using a circuit (such as an ASIC) that performs one or more functions.

[0146] Some embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications can be made within the scope of the technical concept of the present disclosure. In addition, two or more of the above-described multiple embodiments can be combined with each other and implemented. In addition, the effects described in the embodiments are only the most suitable effects produced by the present disclosure. Therefore, the effects of the present disclosure are not limited to the effects described in the embodiments.

[0147] The (one or more) embodiments of the present disclosure may also be implemented by a computer of a system or device that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more completely referred to as a "non-transitory computer-readable storage medium") to perform the functions of one or more of the above (one or more) embodiments and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing the functions of one or more of the above (one or more) embodiments, and by a method executed by a computer of a system or device, for example, by reading out and executing computer executable instructions from a storage medium to perform the functions of one or more of the above (one or more) embodiments and / or controlling one or more circuits to perform the functions of one or more of the above (one or more) embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of separate computers or separate processors to read out and execute computer executable instructions. Computer executable instructions may be provided to the computer, for example, from a network or a storage medium. The storage medium may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), a storage device of a distributed computing system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray disk (BD)) TM ), flash memory devices, memory cards, etc.

[0148] Other embodiments

[0149] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0150] Although the disclosure has described exemplary embodiments, it is to be understood that some embodiments are not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A robot system, comprising: robot; a search unit configured to search for a work area of ​​a workpiece that performs work and obtain search data including information about the work area; at least one processor; as well as At least one memory, the at least one memory communicating with the at least one processor, wherein the at least one memory stores instructions for causing the at least one processor and the at least one memory to identify a work area based on information related to the work area, information about work related to the work area and performed on a workpiece, and search data, and to control the robot to perform work on the identified work area. 2 . The robot system according to claim 1 , wherein the information related to the work area is a model obtained by modeling the work area based on design information for designing the workpiece.

3. A robot system according to claim 1 or 2, wherein the information about the work includes at least one of an assembly direction of the component, an assembly stroke for assembling the component, an assembly phase angle of the component, an insertion start position of the component, an insertion completion position of the component, and an insertion force for the component.

4. The robot system according to claim 3, wherein the information on the work is provided as table information.

5. The robotic system of claim 1, wherein the search data comprises data obtained from a camera, a tactile sensor, an ultrasonic sensor, or a probe.

6. The robot system according to claim 2, wherein a coordinate system indicating the position and posture of the model is set to the model, and wherein the robot system is configured to obtain information about a position and a posture of a work area of ​​the workpiece defined in a coordinate system of the model by using the model and information related to the work area obtained based on the search data, and Based on the information about the position and posture, the robot is caused to perform work on the working area of ​​the workpiece.

7. The robot system according to claim 6, wherein the robot comprises a sensor configured to obtain information about the force, and The robot system is configured to cause the robot to perform work on a work area of ​​a workpiece based on information about the position and posture and information about forces applied to the robot.

8. The robot system according to claim 6, wherein the search unit comprises an image capture device disposed on the robot, and The robot system is configured to cause the robot to perform work on a work area of ​​a workpiece based on information about the position and posture, a reference position of the image capture device, and a reference position of a component moved by the robot.

9. The robot system according to claim 6, wherein the search unit comprises an image capture device, and The robot system is configured to obtain a trajectory of the robot for causing the robot to perform work on a workpiece based on information about position and posture, information about work, a coordinate system of a predetermined part of the robot, a coordinate system of an image capture device, and a coordinate system of the robot.

10. The robot system according to claim 2, wherein the search unit is configured to obtain the search data by detecting the working area, wherein the search direction faces the working area, and The at least one memory also stores instructions for causing the at least one processor and the at least one memory to obtain learning model information having characteristics of a working area that has been learned when searching for models from directions at multiple angles, and to identify the working area based on the learning model information and the search data.

11. The robot system according to claim 10, wherein the search unit is an image capture device configured to capture an image in an image capture direction and obtain image data containing features of the work area, wherein the learning model information is learning model information of features of the working area that has been learned in a case where images of the model are captured from directions having a plurality of angles, and The at least one memory further stores instructions for causing the at least one processor and the at least one memory to identify a working area based on the learning model information and image data captured by the image capture device.

12. The robotic system of claim 11, wherein the at least one memory further stores instructions for causing the at least one processor and the at least one memory to: creating an inferred image by inferring a working area from image data captured by an image capture device based on the learned model information, and A working region of the image data is identified by performing a matching process of matching the model to the inferred image.

13. The robot system according to claim 12, wherein the learning model information includes posture information about the posture of the model, and The at least one memory further stores instructions for causing the at least one processor and the at least one memory to: identifying matching regions of the image data by performing matching of the model to the inferred image based on the pose information, and Perform matching processing.

14. The robotic system of claim 12, wherein the image capture device comprises a plurality of cameras configured to perform three-dimensional measurements, The at least one memory further stores instructions for causing the at least one processor and the at least one memory to: identifying matching regions of the image data by performing matching of a model with an inferred image based on image data captured by each of the plurality of cameras, and Perform matching processing.

15. The robotic system of claim 12, wherein the at least one memory further stores instructions for causing the at least one processor and the at least one memory to: performing masking on areas of the inferred image other than the matching area where the model is matched, and A matching process is performed to match the model with the inferred image on which the masking process has been performed.

16. The robot system according to claim 2, wherein the model has work information about work performed on the work area, and The at least one memory further stores instructions for causing the at least one processor and the at least one memory to control the robot based on the working information.

17. The robot system according to claim 16, wherein the work is an assembly work in which a component held by the robot is assembled to a work area of ​​the workpiece.

18. The robot system according to claim 1, wherein the at least one memory further stores instructions for causing the at least one processor and the at least one memory to control the robot based on surrounding model information about a surrounding model and search data when performing work on a work area, and surrounding objects deployed around a workpiece are modeled into the surrounding model.

19. The robot system according to claim 18, wherein the at least one memory further stores instructions for causing the at least one processor and the at least one memory to determine a trajectory in which the robot and surrounding objects do not interfere with each other based on surrounding model information and search data when performing work on a work area, and to control the robot according to the determined trajectory.

20. The robot system according to claim 1, wherein the at least one memory further stores instructions for causing the at least one processor and the at least one memory to cause a display device to display the identified work area when the work area is identified.

21. The robot system according to claim 2, further comprising an information processing device configured to obtain the model.

22. A method of controlling a robot system, the robot system comprising a robot, a search unit for detecting a working area of ​​a workpiece performing work, at least one processor, and at least one memory, the method comprising: obtaining, by the search unit, search data including information about the work area; as well as The at least one processor and the at least one memory identify the work area based on the information related to the work area, the information about the work related to the work area and performed on the workpiece, and the search data, and control the robot to perform the work on the identified work area.

23. A method comprising: A product is manufactured by using the robot system according to claim 1.

24. A computer executable program product which, when executed, causes a computer to perform the method according to claim 22.

25. A computer-readable non-transitory recording medium storing computer-executable instructions, the computer-executable instructions, when executed, causing a computer to perform a method, the method comprising: obtaining, by a search unit for detecting a working area of ​​a workpiece for performing work, search data containing information on the working area; as well as Based on the information related to the work area, the information about the work related to the work area and performed on the workpiece, and the search data, the work area is identified and the robot is controlled to perform the work on the identified work area.

Images