Simulation system and simulation method

Abstract

Teaching data created using 3D (Dimensional) simulation is corrected in accordance with deformation of a base or the like of a production line in an actual space. A simulation system is provided with a camera (111) attached to an articulated robot (100), and a simulation device. A camera (111) transmits an image of a landmark (150) disposed on a base (120) of a production line (10) in the real world to a simulation device. The simulation device analyzes a plurality of marks equally arranged on a landmark (150) in an image, estimates a deformation of the base (120) on the basis of the respective shapes and arrangement intervals of the plurality of marks observed from a camera (111), reproduces the deformation of the base (120) in a production line (10) reproduced in a 3D space on the basis of the result of the estimation, and reproduces the deformation of the base (120) in the 3D space on the basis of the result of the estimation. The teaching data during the processing of the workpiece (140) on the deformed base (120) by the multi-joint robot is corrected, and the correction result of the teaching data is fed back to a control device of the multi-joint robot.

Description

Technical Field

[0001] The present disclosure relates to a simulation system, and more particularly, to a technique for correcting teaching data of a robot. Background Art

[0002] In recent years, in order to teach an industrial robot, 3D (Dimensional) simulation technology has been used. Teaching based on 3D simulation does not require preparing an actual production line or the like for operation. Therefore, by using a 3D simulator, a user can easily and inexpensively perform teaching processing of a robot.

[0003] However, the positions and postures of objects in 3D simulation are ideal values. These values do not necessarily match the positions and postures of objects in the actual space. Therefore, the user must re-teach the robot by operating an actual production line. In particular, in the case of operating a robot to perform operations such as connector fitting operations and screw tightening operations, the user needs to repeatedly perform teaching of an actual device using the robot many times. Therefore, a technique for more easily and quickly completing re-teaching is desired. In addition, a technique of using certain markers to detect the position of an object to control a robot is known.

[0004] Regarding robot control using markers, for example, Japanese Unexamined Patent Application Publication No. 2017-083234 (Patent Document 1) discloses a three-dimensional shape measurement device "having a marker search unit that uses a search marker and calculates coordinates of a portion where a light cut line formed by line laser is formed on the surface of a measurement object" (see [Abstract]).

[0005] Prior Art Documents

[0006] Patent Documents

[0007] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2017-083234 Summary of the Invention

[0008] Problems to be Solved by the Invention

[0009] According to the technique disclosed in Patent Document 1, it may be possible to detect the shape of a workpiece in the actual space. However, the technique disclosed in Patent Document 1 cannot correct teaching data created using 3D simulation corresponding to deformation of a base or the like of a production line in the actual space. Therefore, a technique for correcting teaching data created using 3D simulation corresponding to deformation of a base or the like of a production line in the actual space is needed.

[0010] The present disclosure has been made in view of the above background, and an object of one aspect is to provide a technique for correcting teaching data created using 3D simulation corresponding to deformation of a base or the like of a production line in the actual space.

[0011] Means for Solving the Problem

[0012] According to an embodiment, a simulation system is provided, which includes: a camera mounted on a multi-joint robot; and a simulation device. The camera sends an image of a landmark disposed on a base of a production line in the real world to the simulation device. The simulation device analyzes a plurality of markers evenly disposed on the landmark in the image, estimates the deformation of the base based on the shape and arrangement interval of each of the plurality of markers observed from the camera, and reproduces the deformation of the base in a production line reproduced in a 3D (Dimensional) space, i.e., a three-dimensional space. In the 3D space, the teaching data in the machining process of the workpiece on the deformed base by the multi-joint robot is corrected, and the correction result of the teaching data is fed back to the control device of the multi-joint robot. According to the present disclosure, the simulation system can correct the teaching data of the multi-joint robot based on the deviation between the production line in the 3D space and the production line in the real world, and feed back the correction result to the control device.

[0013] In the above disclosure, correcting the teaching data includes: adjusting the torque of a power tool equipped on the multi-joint robot. According to the present disclosure, the simulation system can adjust the torque of the power tool based on the deviation between the production line in the 3D space and the production line in the real world.

[0014] In the above disclosure, correcting the teaching data includes: adjusting the force applied by the multi-joint robot to the workpiece during the fitting operation. According to the present disclosure, the simulation system can adjust the force applied by the multi-joint robot to the workpiece during the fitting operation based on the deviation between the production line in the 3D space and the production line in the real world.

[0015] In the above disclosure, reproducing the deformation of the base includes: reproducing the deformation of the workpiece on the base based on the deformation of the base. According to the present disclosure, the simulation system can reproduce the deformation of the workpiece based on the deformation of the base. In addition, the simulation system can correct the teaching data by simulating the machining process of the deformed workpiece.

[0016] In the above disclosure, the simulation device reads out the material information of the workpiece from a memory, and calculates the correction amount of the teaching data based on the deformation amount of the workpiece and the material information of the workpiece. According to the present disclosure, the simulation system can calculate the correction amount of the teaching data based on the deformation amount of the workpiece and the material information of the workpiece.

[0017] In the above disclosure, calculating the correction amount of the teaching data includes: using the finite element method to calculate the stress of the deformed part of the workpiece; and calculating the correction amount of the teaching data based on the stress. According to the present disclosure, the simulation system can use the finite element method to calculate the correction amount of the teaching data.

[0018] In the above disclosure, the landmarks include a plurality of landmarks that are dispersedly arranged on the base. Estimating the deformation of the base includes: estimating the deformation of each part of the base according to each of the plurality of landmarks. According to the present disclosure, the simulation system can use the plurality of landmarks to estimate the deformation of each part of the base.

[0019] According to another embodiment, a simulation method is provided. The simulation method has the following steps: capturing an image of a landmark disposed on a base of a production line in the real world through a camera mounted on a multi-joint robot; sending the image to a simulation device; analyzing a plurality of markers evenly disposed on the landmark in the image; estimating the deformation of the base based on the shape and arrangement interval of each of the plurality of markers observed from the camera; reproducing the deformation of the base in a production line reproduced in a 3D space based on the estimation result; correcting teaching data in the machining process of a workpiece on the deformed base by the multi-joint robot in the 3D space; and feeding back the correction result of the teaching data to a control device of the multi-joint robot. According to the present disclosure, the simulation system can correct the teaching data of the multi-joint robot based on the deviation between the production line in the 3D space and the production line in the real world, and feed back the correction result to the control device.

[0020] Effects of the Invention

[0021] According to an embodiment, it is possible to correct teaching data created using 3D simulation corresponding to the deformation of a base or the like of a production line in the actual space.

[0022] The above and other objects, features, aspects, and advantages of the present disclosure will become apparent from the following detailed description related to the present disclosure understood in association with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 FIG. is an example of a production line 10 to which the technology of the present disclosure can be applied.

[0024] Figure 2 FIG. is an example of the structure of the system 20 of the present embodiment.

[0025] Figure 3 FIG. is an example of the hardware structure of the information processing device 200.

[0026] Figure 4 FIG. is an example of the types of deformation of the base 120.

[0027] Figure 5 FIG. is an example of the process of calibrating the system 20.

[0028] Figure 6 FIG. is a specific example of the first calibration 510 of the camera 111.

[0029] Figure 7 This is an example of a diagram showing the process of detecting the position of the detection landmark 150.

[0030] Figure 8 This is an example of a diagram showing the process of detecting the posture of the detection landmark 150.

[0031] Figure 9 This is an example of a diagram showing the configuration of the landmark 150.

[0032] Figure 10 This is an example of a diagram showing the process of calculating the vertical stress applied to the workpiece 140.

[0033] Figure 11 This is a diagram showing a situation where the workpiece 140 deformed in a 3D simulation is processed (screw tightening). Detailed implementation manners

[0034] Hereinafter, embodiments of the technical idea of the present disclosure will be described with reference to the drawings. In the following description, the same reference numerals are assigned to the same components. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated.

[0035] <A. Application example>

[0036] Figure 1 This is an example of a diagram showing a production line 10 to which the technology of the present disclosure can be applied. Refer to Figure 1 , the structure of the production line 10, the problems of teaching using 3D simulation, and the technical overview of the present disclosure for solving the problems will be described.

[0037] (a. Structure of the production line)

[0038] The production line 10 includes an articulated robot 100, a base 120, and a fixture 130. In addition, the articulated robot 100 has an end effector 110 at its end. The end effector 110 includes a camera 111 and a tool 112.

[0039] The production line 10 is, for example, an assembly line for products. The workpiece is completed as a product by being processed on the production line 10. In addition, the products here include intermediate parts used in some other products. In this specification, a "workpiece" is an object to be processed. For example, the workpiece is a substrate, a plastic part, a metal part, or an object formed by combining them. In addition, "processing" of the workpiece includes any process such as screw tightening, fitting of parts to each other, welding, drilling, cutting, etc.

[0040] The multi-joint robot 100 is a robot driven by multiple servo motors and the like. The multi-joint robot 100 includes a vertical multi-joint robot, a horizontal multi-joint robot, a parallel link robot, and a robot combining these structures. The multi-joint robot 100 performs arbitrary processing such as screw fastening, fitting of components to each other, welding, hole opening, and cutting on the workpiece 140. The multi-joint robot 100 is controlled by a control device 220 (refer to Figure 2 ). According to a certain embodiment, the production line 10 has multiple multi-joint robots 100 for uses such as handling of each process and the workpiece 140.

[0041] The base 120 is a base for transferring or arranging the workpiece 140. When the workpiece 140 is subjected to the first processing, it moves on the base 120. Then, after the workpiece 140 is subjected to the second processing at the moving destination, it moves on the base 120 again. In this way, the production line 10 is configured to repeatedly execute the processing and moving processes of the workpiece 140. According to a certain embodiment, the production line 10 has one base 120. In this case, the workpiece 140 moves on one base 120. According to other embodiments, the production line 10 has multiple bases 120. In this case, whenever the processing of the workpiece 140 by each base 120 is completed, the workpiece 140 moves to the next base 120.

[0042] The fixture 130 fixes the workpiece 140. In addition, the fixture 130 is fixed to the base 120. The workpiece 140 is arranged on the fixture 130 during processing. As an example, the workpiece 140 is arranged on the fixture 130 by the multi-joint robot 100 for handling. The fixture 130 has, for example, a frame along the outer shape of the workpiece 140, or has a shaft inserted into a threaded hole provided in the workpiece 140. The fixture 130 fixes the workpiece 140 through these structures. According to a certain embodiment, the workpiece 140 can also be processed in a state of being directly arranged on the base 120.

[0043] The end effector 110 is a tool installed at the end of the multi-joint robot 100. The multi-joint robot 100 is configured to be able to load and unload the end effector 110. The user can install different end effectors on the multi-joint robot 100 for use according to the application. As an example, the end effector 110 includes an end effector for fitting processing, an end effector for screw fastening, an end effector for welding, and the like.

[0044] The camera 111 is installed on the multi-joint robot 100 or the end effector 110. The camera 111 photographs the landmark 150 on the base 120. The camera 111 sends the photographed image of the landmark 150 to the information processing device 200 via the control device 220 (refer to Figure 2)。The use of the landmark 150 will be described later. According to a certain embodiment, the camera 111 is a CCD (Charge Coupled Device) camera. According to other embodiments, the camera 111 is a CMOS (Complementary Metal Oxide Semiconductor) camera.

[0045] The tool 112 is a tool equipped on the end effector 110 for processing the workpiece 140. The tool 112 may include a manipulator for gripping a component that fits with the workpiece 140, a power tool for screw fastening, and tools for any other arbitrary processing.

[0046] (b. Problems with teaching using 3D simulation)

[0047] Next, the teaching of the multi-joint robot 100, the implementation method of teaching using 3D simulation, and the problems of teaching using 3D simulation will be described.

[0048] First, the teaching of the multi-joint robot 100 will be described. "Teaching" is an adjustment operation for making the multi-joint robot 100 perform a certain action. For example, assume a screw fastening operation on the workpiece 140. In this case, the multi-joint robot 100 needs to accurately move to the screw fastening position. In addition, the movement of the multi-joint robot 100 means moving the end of the tool 112 to the target position. In order for the user to make the multi-joint robot 100 perform the target action, the user needs to repeatedly change the commands sent to the multi-joint robot 100 to adjust the action of the multi-joint robot 100. In the teaching in the actual space, the user makes the multi-joint robot 100 actually move via the control device 220.

[0049] Next, the implementation method of teaching using 3D simulation will be described. The user first reproduces the actual space production line 10 in the simulator (3D simulator) 210 (refer to Figure 2 )). Then, the user sends commands to the multi-joint robot 100 via the virtual control device 220 within the 3D simulation. When the multi-joint robot 100 in the 3D space receives the command from the virtual control device 220, it acts in the same way as in the actual space. By moving the production line 10 and the multi-joint robot 100 constructed in the 3D space, the user can teach the multi-joint robot 100 before the construction of the formal production line 10.

[0050] Next, the problems of teaching using 3D simulation will be described. Each object included in the production line 10 reproduced in the 3D space assumes an ideal configuration and posture. However, each object included in the production line 10 constructed in the actual space may sometimes be deformed. For example, the base 120 may not be completely horizontal and may sometimes be slightly inclined. In addition, the base 120 may sometimes be slightly twisted or flexed. Such deformation (inclination, torsion, and / or flexure) of the base 120 sometimes has an adverse effect on the processing of the workpiece 140. Therefore, the user must adjust the teaching data created using the 3D simulator 210 in accordance with the deformation of the production line 10 (base 120) in the actual space.

[0051] (c. Solution)

[0052] To solve the above problems, the system (simulation system) 20 of the present embodiment uses the landmark 150. The landmark 150 is a plate on which marks are arranged at equal intervals on a grid. A "mark" is a hole or a sign that is provided at equal intervals on the grid of the landmark 150. In addition, as an example, the mark is circular.

[0053] The landmark 150 is arranged on the base 120. According to a certain embodiment, the landmark 150 is arranged near the workpiece 140 or the fixture 130. According to other embodiments, a plurality of landmarks 150 are dispersedly arranged on the base 120 at equal intervals. And, according to other embodiments, the entire surface of the base 120 is configured as the landmark 150. The camera 111 captures an image of the landmark 150 and sends the image of the landmark 150 to the control device 220. The control device 220 transmits the image to the information processing device 200. According to a certain embodiment, the camera 111 directly transmits the image to the information processing device 200.

[0054] The information processing device 200 (simulator 210) detects the deformation of the landmark 150 captured in the image. More specifically, the information processing device 200 (simulator 210) analyzes a plurality of marks that are evenly arranged on the landmark 150 in the image. Next, the information processing device 200 (simulator 210) estimates the deformation of the base 120 based on the shape and the arrangement interval of each of the plurality of marks observed from the camera 111. When the base 120 is in an ideal shape and posture, the shape and the arrangement interval of each of the plurality of marks observed from the camera 111 are not deformed.

[0055] The information processing device 200 (simulator 210) reproduces the deformation of the base 120 in the production line 10 reproduced in the 3D space based on the estimation result. In addition, the information processing device 200 (simulator 210) also reproduces the deformation of the workpiece 140 based on the deformation of the base 120. Then, the information processing device 200 (simulator 210) corrects the teaching data in the machining process of the workpiece 140 on the deformed base 120 by the multi-joint robot 100 in the 3D space. More specifically, the information processing device 200 (simulator 210) can correct the teaching data by simulating the machining process of the workpiece 140 on the fixture 130 provided on the reproduced deformed base 120. In addition, the information processing device 200 (simulator 210) feeds back the correction result of the teaching data to the control device 220 of the multi-joint robot 100. Thus, the teaching data of the multi-joint robot 100 is adjusted or corrected according to the deformation of the production line 10 (base 120) constructed in the actual space.

[0056] As an example, the information processing device 200 (simulator 210) estimates the deformation of the workpiece 140 arranged on the fixture 130 according to the deformation of the base 120. In addition, the information processing device 200 (simulator 210) calculates the correction amount of the teaching data of the multi-joint robot 100 based on the deformation amount and material information of the workpiece 140. If the information processing device 200 (simulator 210) obtains the deformation amount and material information of the workpiece 140, it can calculate the stress of the deformed part of the workpiece 140 using the finite element method or the like. The information processing device 200 (simulator 210) can calculate the correction amount of the teaching data of the multi-joint robot 100 based on the calculated stress.

[0057] <B. System Structure>

[0058] Next, with reference to Figure 2 and Figure 3 the structure of the system 20 of the present embodiment will be described. In this specification, "system" includes a structure composed of one or more devices, a server, a virtual machine or container constructed in a cloud environment, or a structure composed of at least a part of them. In addition, the device can include any mechanical device such as an information processing device 200 such as a personal computer, a workstation, a server device, a tablet computer, a smart phone, a control device 220 such as a PLC (Programmable Logic Controller), a multi-joint robot 100, etc., and can also be a combination thereof. In one aspect, the system 20 can also be connected to input / output devices such as a display and a keyboard for user use. In other aspects, the system 20 can also provide various functions to users as a cloud service or a web application via a network. In this case, the user can use the functions of the system 20 via a browser or client software installed on their own terminal.

[0059] Figure 2 This is a diagram showing an example of the structure of the system 20 of the present embodiment. The system 20 includes an information processing device 200, a control device 220, and an articulated robot 100. According to a certain embodiment, the system 20 includes the information processing device 200 and the control device 220, and does not include the articulated robot 100. According to other embodiments, the system 20 includes the information processing device 200, and does not include the control device 220 and the articulated robot 100. According to still another embodiment, the information processing device 200 and the control device 220 are configured as an integrated device.

[0060] The information processing device 200 is configured to be able to execute 3D simulation using the simulator 210. The simulator 210 has a function of reproducing the production line 10 in a 3D space and a function of reproducing a virtual control device 220. The user can operate the production line 10 in the 3D space in the same manner as in the actual space by causing the virtual control device 220 to execute a ladder program or the like. The information processing device 200 receives an image of the landmark 150 from the camera 111 via the control device 220. According to a certain embodiment, the simulator 210 can be implemented as software executed on the hardware of the information processing device 200 as shown. According to other embodiments, part or all of the functions of the simulator 210 can be implemented as hardware. Figure 3 shown.

[0061] The control device 220 controls the articulated robot 100 and any other mechanical devices. In addition, the control device 220 obtains signals from one or more sensors provided on the production line 10. As an example, the control device 220 is a PLC. As another example, the control device 220 is an integrated controller having a PLC and a robot control device. The control device 220 transmits the image received from the camera 111 to the information processing device 200.

[0062] Figure 3 This is a diagram showing an example of the hardware structure of the information processing device 200. According to a certain embodiment, the control device 220 has Figure 3 each of the structures shown as part of the hardware. The information processing device 200 includes a processor 301, a memory 302, a storage 303, an external device IF (Interface), an input IF 305, an output IF 306, and a communication IF 307. These respective structures are connected via a bus 308 so as to be able to communicate with each other.

[0063] The processor 301 can execute programs for implementing various functions of the information processing apparatus 200. The processor 301 is constituted by, for example, at least one integrated circuit. According to a certain embodiment, the information processing apparatus 200 may also include, for example, at least one CPU (Central Processing Unit), at least one GPU (Graphics Processing Unit), at least one FPGA (Field Programmable Gate Array), at least one ASIC (Application Specific Integrated Circuit), or a combination thereof, etc.

[0064] The memory 302 stores programs executed by the processor 301 and data referred to by the processor 301. In a certain aspect, the memory 302 can be implemented by a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory), etc.

[0065] The storage 303 is a non-volatile memory that stores programs executed by the processor 301 and data referred to by the processor 301. In this case, the processor 301 executes the program read from the storage 303 into the memory 302 and refers to the data read from the storage 303 into the memory 302. In a certain aspect, the storage 303 can be implemented by an HDD (Hard Disk Drive), an SSD (Solid State Drive), an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read Only Memory), or a flash memory, etc.

[0066] The external device IF 304 can be connected to any external device such as a printer, a scanner, and an external HDD. In a certain aspect, the external device IF 304 can be implemented by a USB (Universal Serial Bus) terminal, etc.

[0067] The input IF 305 can be connected to any input device such as a keyboard, a mouse, a touchpad, or a gamepad. In a certain aspect, the input IF 305 can be implemented by a USB terminal, a PS / 2 terminal, and a Bluetooth (registered trademark) module, etc.

[0068] The output IF 306 can be connected to any output device such as a cathode ray tube display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. In some aspects, the output IF 306 can be implemented by a USB terminal, a D-sub terminal, a DVI (Digital Visual Interface) terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal, a DisplayPort terminal, and the like.

[0069] The communication IF 307 is connected to other devices via a wired network or a wireless network. In some aspects, the communication IF 307 can be implemented by a wired LAN (Local Area Network) port, a Wi-Fi (registered trademark) (Wireless Fidelity) module, and the like. In other aspects, the communication IF 307 can use communication protocols such as TCP / IP (Transmission Control Protocol / Internet Protocol) and UDP (User Datagram Protocol) to send and receive data.

[0070] <C. Detection Method of Deformation of Base>

[0071] Next, with reference to Figures 4 to 9 , a method for the system 20 to detect the deformation of the base 120 will be described. The system 20 estimates the deformation of the workpiece 140 disposed on the base 120 or the jig 130 based on the deformation of the base 120.

[0072] Figure 4 is a diagram showing an example of the type of deformation of the base 120. The deformation of the base 120 is a change in the shape of the base 120. As an example, "deformation" includes flexure 410, torsion 420, a combination of flexure and tilt 430, a combination of torsion and tilt 440. In addition, the deformation includes a combination of flexure, torsion, and tilt.

[0073] Figure 5 is a diagram showing an example of the calibration process of the system 20. In order for the system 20 to accurately analyze the landmark 150, calibration of the camera 111 is required. The calibration includes a first calibration 510 that is the calibration of the camera 111 alone and a second calibration 520 that is the calibration of the camera 111 mounted on the multi-joint robot 100. After the first calibration 510 and the second calibration 520 are completed, the system 20 can perform the detection 530 of the position and posture of the landmark 150.

[0074] In the first calibration 510, the system 20 performs calibration of the single camera 111. This calibration is a process for defining the mutual positional relationship when the camera 111 captures the landmark 150. The positional relationship here includes at least one of the distance from the camera 111 to the landmark 150 and the orientation in which the camera 111 captures the landmark 150 (such as the direction based on the arrangement of the landmarks 150). According to an embodiment, the calibration of the camera 111 can also be performed in a state where the camera 111 is removed from the multi-joint robot 100. According to other embodiments, the calibration of the camera 111 can also be performed in a state where the camera 111 is installed on the multi-joint robot 100. Additionally, in other aspects, the calibration of the camera 111 can be performed only in the real space or in both the 3D space and the real space. According to an embodiment, the first calibration 510 is a calibration based on the method of Z. Zhang. Regarding the calibration based on the method of Z. Zhang, reference will be made to Figure 6 described later.

[0075] In the second calibration 520, the system 20 calibrates the camera 111 installed on the multi-joint robot 100. The system 20 performs calibration by using the camera 111 installed on the multi-joint robot 100 to capture the landmark 150. As an example, the system 20 pre-receives the input of the position information of the camera 111 in the robot coordinate system and the positional relationship of the landmark 150. These position information are ideal position information. The position of the camera 111 is, for example, the position of the camera 111 observed from the base (the reference point of the robot coordinate system) of the multi-joint robot 100. Additionally, the position of the landmark 150 is, for example, the position of the landmark 150 observed from the base (the reference point of the robot coordinate system) of the multi-joint robot 100. Next, the system 20 captures the landmark 150 by using the camera 111 installed on the multi-joint robot 100. The system 20 analyzes the obtained image and corrects the deviation between the ideal positional relationship of the camera 111 and the ideal landmark 150 and the actual positional relationship of the camera 111 and the ideal landmark 150.

[0076] In the detection 530 of the position and posture of the landmark 150, the system 20 extracts the difference between the ideal shape and posture of the landmark 150 observed from the camera 111 and the actual shape and posture of the landmark 150 observed from the camera 111. The ideal shape and posture of the landmark 150 observed from the camera 111 can also be said to be the distance and direction from the multi-joint robot 100 to the landmark 150 in the 3D space. Additionally, the actual shape and posture of the landmark 150 observed from the camera 111 can also be said to be the distance and direction from the multi-joint robot 100 to the landmark 150 in the real space. The system 20 estimates the deformation of the base 120 based on this difference.

[0077] Figure 6 This is a diagram showing a specific example of the first calibration 510 of the camera 111. Refer to Figure 6 and the calibration process based on the method of Z. Zhang will be described.

[0078] The system 20 pre-accepts the input of the design information (position of the marker) of the landmark 150. The design information of the landmark 150 includes constraint information such as the marker being configured in a planar shape. According to an embodiment, the design information of the landmark 150 further includes information on the shape of the marker and the setting interval of the marker. The system 20 analyzes the image and estimates the pose of the landmark 150 based on the above constraint information. In addition, the system 20 also calculates the distortion coefficient of the lens of the camera 111 at the same time.

[0079] As an example, the system 20 captures more than 10 images of the landmark 150. Thereby, the system 20 estimates the position and pose of the landmark 150 and the distortion of the lens. As an example, for the first calibration 510, the system 20 captures about 10 to 20 images of the landmark 150 under the following conditions.

[0080] The multiple captured images preferably satisfy the following conditions. As the first condition, the multiple captured images include images with different distances observed from the camera 111. For example, it is desirable that the multiple captured images include more than 3 images with different distances observed from the camera 111 within the distance range where the system 20 actually detects the landmark 150. As the second condition, the multiple captured images include an image of the landmark 150 facing the camera 111 directly. As the third condition, the multiple captured images include an image of the landmark 150 tilted when observed from the camera 111. For example, an image of the landmark 150 tilted when observed from the camera 111 is an image in which the landmark 150 is tilted in any one of the up, down, left, and right directions within the range where the shape of the marker on the landmark 150 does not deform. In addition, the multiple captured images preferably equally include images of the landmark 150 tilted in each of the up, down, left, and right directions. As the fourth condition, the multiple captured images include an image in which the landmark 150 is arranged at the center of the shooting range of the camera 111. As the fifth condition, the multiple captured images include an image in which the landmark 150 is arranged at the corner of the shooting range of the camera 111. As the sixth condition, the multiple captured images include, without deviation, the images captured under the first condition to the fifth condition.

[0081] In addition, the shooting environment of the landmark 150 preferably satisfies the following conditions. As the first condition, the shooting environment of the landmark 150 is an environment where no focus deviation occurs. As the second condition, in the shooting environment of the landmark 150, all the marks on the landmark 150 do not exceed the field of view of the camera 111. Sometimes, a part of the marks on the landmark 150 may exceed the field of view of the camera 111. As the third condition, the shooting environment of the landmark 150 is an environment where no halation is caused due to light reflection or the like.

[0082] Furthermore, the photographed landmark 150 preferably satisfies the following conditions. As the first condition, the landmark 150 has a size that occupies more than 1 / 4 of the field of view when observed from the camera 111. As the second condition, the number of columns (rows) of the marks provided on the landmark 150 is an odd number from 3 to 21.

[0083] There are two methods for shooting the landmark 150. The first shooting method 610 is a method of fixing the landmark 150 and moving the camera 111. The second shooting method 620 is a method of fixing the camera 111 and moving the landmark 150. The plurality of images 630 is a diagram showing an example of an image taken based on the first calibration 510.

[0084] Figure 7 It is a diagram showing an example of the process of detecting the position of the landmark 150. Figure 7 The process shown corresponds to the detection of the position of the landmark 150 in the detection 530 of the position and orientation of the landmark 150.

[0085] First, the system 20 moves the camera 111 to the shooting position of the landmark 150. As an example, the system 20 moves the camera 111 to a position where an image of the landmark 150 with a predetermined position and orientation can be obtained. The system 20 adjusts the position of the camera 111 based on the shape of each mark captured in the image and the distance between the marks. Then, the system 20 calculates the relative distance from the camera to the landmark 150. As an example, the system 20 calculates this relative distance based on the camera coordinate system.

[0086] Furthermore, the system 20 calculates the position of the landmark 150 in the robot coordinate system based on the relative distance from the camera 111 to the landmark 150. As an example, the system 20 calculates the coordinates of the camera 111 in the robot coordinate system based on the angles of the respective joints of the multi-joint robot 100 and the installation position of the camera 111. Then, the system 20 can calculate the position of the landmark 150 in the robot coordinate system by adding the relative distance from the camera to the landmark 150 to the coordinates of the camera 111 in the robot coordinate system.

[0087] Figure 8 It is a diagram showing an example of the process of detecting the orientation of the landmark 150. Figure 8The processing shown is equivalent to the detection of the posture of the landmark 150 in the detection 530 of the position and posture of the landmark 150.

[0088] The system 20 acquires an image from the camera 111 that has moved to a predetermined position and analyzes the image. The system 20 estimates the posture of the landmark 150 based on the relative distance between the camera 111 and the landmark 150 and the position deviation of the marker. The position deviation of the marker refers to the deviation between the ideal position of the marker and the position of the marker captured in the actually captured image. For example, when observing the landmark 150 from directly above, the markers are arranged at equal intervals. However, when the base 120 is deformed, the shape of each marker captured in the image and the interval between the markers also change. The system 20 uses a program such as Solve PnP to estimate the relative position / posture between the camera 111 and the landmark 150. Thereby, the system 20 can estimate the deformation of the base 120. Further, the system 20 can also estimate the deformation of the workpiece 140 disposed on the fixture 130 or the workpiece 140 directly disposed on the base 120 based on the deformation of the base 120.

[0089] According to an embodiment, the information processing device 200 (simulator 210) outputs commands for executing Figure 7 and Figure 8 processing to the control device 220 at any time. In this case, the control device 220 drives the multi-joint robot 100 in accordance with the received commands. According to other embodiments, the control device 220 drives the multi-joint robot 100 in accordance with a pre-installed program. In this case, the information processing device 200 (simulator 210) does not send commands to the control device 220, but receives an image and the position information of the camera 111, etc. from the control device 220.

[0090] Figure 9 is a diagram showing an example of the configuration of the landmark 150. As Figure 9 shown, the landmarks 150 may be dispersedly arranged on the base 120 at equal intervals. In this case, it can be said that the landmarks 150 detected by the system 20 include a plurality of landmarks 150 dispersedly arranged on the base. Further, it can be said that estimating the deformation of the base 120 includes estimating the deformation of each part of the base 120 based on each of the plurality of landmarks 150. The system 20 can estimate the deformation of each part of the base 120 based on the positions and postures of the dispersedly arranged landmarks 150. Further, the system 20 can estimate the overall deformation of the base 120 based on the deformation of each part of the base 120. According to an embodiment, the entire surface of the base 120 is configured as a landmark 150. According to other embodiments, the landmark 150 is only arranged near the workpiece 140 or the fixture 130.

[0091] <D. Method for correcting teaching data>

[0092] Next, referring to Figure 10 and Figure 11 , the process of correcting the taught data will be described. According to a certain embodiment, the correction of the taught data includes the correction of the position and posture of the multi-joint robot 100. According to other embodiments, the correction of the taught data includes the adjustment of the torque of the power tool included in the multi-joint robot 100. Further, according to other embodiments, the correction of the taught data includes the adjustment of the strength (the force applied by the multi-joint robot 100 to the workpiece 140 during the fitting operation) of pressing the fitting member (such as a plug) against the workpiece 140 in the fitting process. And, according to other embodiments, the correction of the taught data includes the correction of the position and posture of the multi-joint robot 100, and the adjustment of the torque of the dynamic tool or the adjustment of the strength of pressing the fitting member (such as a plug) against the workpiece 140.

[0093] Figure 10 is a diagram showing an example of the process of calculating the vertical stress applied to the workpiece 140. When the system 20 detects the deformation of the base 120, it reproduces the deformation of the base 120 in the 3D space. Moreover, when the system 20 performs machining on the workpiece 140 of the fixture 130 disposed on the deformed base 120 or the workpiece 140 directly disposed on the deformed base 120, it calculates the stress generated in the workpiece 140. For example, in the case of machining such as screw fastening, the system 20 calculates the stress generated in the workpiece 140 when the screw is fastened to the workpiece 140.

[0094] As an example, the system 20 calculates the stress generated in the workpiece 140 by the finite element method. In this case, the system 20 calculates the stress generated in the workpiece 140 by changing the shape of the workpiece 140 reproduced by the mesh structure in the 3D space. The shape change 1010 is the shape change when the screw 1000 is fastened to the non-deformed workpiece 140. The shape change 1020 is the shape change when the screw 1000 is fastened to the deformed workpiece 140 (for example, the inclined workpiece 140).

[0095] The system 20 can correct the taught data of the multi-joint robot 100 based on the stress generated in the workpiece 140 when machining the deformed workpiece 140 reproduced in the 3D space. More specifically, the system 20 adjusts the position and posture of the multi-joint robot 100 so that the stress generated in the workpiece 140 becomes equal to or less than a predetermined value. According to other embodiments, the system 20 adjusts the torque of the power tool included in the multi-joint robot 100 so that the stress generated in the workpiece 140 becomes equal to or less than a predetermined value. Further, according to other embodiments, the system 20 adjusts the force applied by the multi-joint robot 100 to the workpiece 140 during the fitting process so that the stress generated in the workpiece 140 becomes equal to or less than a predetermined value.

[0096] Figure 11 This is a diagram showing the case where the workpiece 140 deformed in the 3D simulation has been processed (screw fastening). In Figure 11 the example of, the fixture 130 is not depicted, but actually, the workpiece 140 is disposed on the fixture 130.

[0097] Based on the Figure 9 positions / poses of the respective landmarks 150 shown, the system 20 calculates the deformation of the base 120. Next, the system 20 reproduces the deformation of the base 120 in the 3D space as Figure 11 such. More specifically, the system 20 changes the positions and poses of the meshes of the deformed portions of the base 120 in the 3D space.

[0098] Similarly, the system 20 also changes the positions and poses of the meshes of the workpiece 140 on the base 120 (or on the fixture 130) in the 3D space. The system 20 processes (screw fastens, fits, etc.) the deformed workpiece 140 in the 3D space and calculates the stress generated in the workpiece 140. Next, the system 20 adjusts the position / pose of the multi-joint robot 100, the torque of the power tool, the force for pressing the workpiece 140 during fitting, etc., so that the stress generated in the workpiece 140 becomes below a certain value. The system 20 feeds back the corrected (adjusted) teaching data to the control device 220.

[0099] As described above, the system 20 of the present embodiment captures the landmark 150 with the camera 111 and analyzes the landmark 150 captured in the image, thereby detecting the deformation of the base 120 in the production line 10. In addition, the system 20 reproduces the deformation of the base 120 in the 3D space. In addition, the system 20 corrects the teaching data in the processing of the workpiece 140 on the deformed base 120 by the multi-joint robot 100 in the 3D space. Then, the system 20 feeds back the correction result of the teaching data to the control device 220 of the multi-joint robot 100.

[0100] <E. Supplementary Note>

[0101] As described above, the present embodiment includes the following disclosure.

[0102] [Structure 1]

[0103] A simulation system (20), comprising:

[0104] a camera (111) mounted on a multi-joint robot (100); and

[0105] a simulation device (200),

[0106] The camera (111) sends an image of a landmark (150) disposed on a base (120) of a production line (10) in the real world to the simulation device (200).

[0107] The simulation device (200) analyzes a plurality of markers disposed on the landmark (150) evenly arranged within the image, estimates deformation of the base (120) based on the shape and arrangement interval of each of the plurality of markers observed from the camera (111), and reproduces the deformation of the base (120) in the production line (10) reproduced in a 3D (Dimensional) space according to the estimation result. In the 3D space, teaching data in the machining process of a workpiece (140) by a multi-joint robot on the deformed base (120) is corrected, and the correction result of the teaching data is fed back to a control device of the multi-joint robot.

[0108] [Structure 2]

[0109] In the simulation system (20) described in Structure 1,

[0110] Correcting the teaching data includes: adjusting the torque of a power tool equipped on the multi-joint robot.

[0111] [Structure 3]

[0112] In the simulation system (20) described in Structure 1,

[0113] Correcting the teaching data includes: adjusting the force applied by the multi-joint robot to the workpiece (140) during a fitting operation.

[0114] [Structure 4]

[0115] In the simulation system (20) described in any one of Structures 1 to 3,

[0116] Reproducing the deformation of the base (120) includes: reproducing the deformation of the workpiece (140) on the base (120) based on the deformation of the base (120).

[0117] [Structure 5]

[0118] In the simulation system (20) described in Structure 4,

[0119] The simulation device (200) reads out material information of the workpiece (140) from a memory, and calculates a correction amount of the teaching data according to the deformation amount of the workpiece (140) and the material information of the workpiece (140).

[0120] [Structure 6]

[0121] In the simulation system (20) described in Structure 5,

[0122] Calculating the correction amount of the teaching data includes:

[0123] Calculating the stress of the deformed part of the workpiece (140) using the finite element method; and

[0124] Calculating the correction amount of the teaching data based on the stress.

[0125] [Structure 7]

[0126] In the simulation system (20) described in any one of Structures 1 to 3,

[0127] The landmark (150) includes a plurality of landmarks (150) distributed dispersedly on the base (120),

[0128] Estimating the deformation of the base (120) includes: estimating the deformation of each part of the base (120) according to each of the plurality of landmarks (150).

[0129] [Structure 8]

[0130] A simulation method having the following steps:

[0131] Taking an image of the landmark (150) disposed on the base (120) of the production line (10) in the real world through a camera (111) mounted on the multi-joint robot (100);

[0132] Sending the image to the simulation device 200;

[0133] Analyzing a plurality of markers evenly disposed on the landmark (150) within the image;

[0134] Estimating the deformation of the base (120) based on the shape and arrangement interval of each of the plurality of markers observed from the camera (111);

[0135] Reproducing the deformation of the base (120) in the production line (10) reproduced in the 3D space based on the estimation result;

[0136] In the 3D space, correcting the teaching data in the machining process of the workpiece (140) on the deformed base (120) by the multi-joint robot; and

[0137] Feeding back the correction result of the teaching data to the control device of the multi-joint robot.

[0138] The embodiments disclosed herein should be considered illustrative in all respects and not restrictive. The scope of the present disclosure is represented not by the above description but by the claims, and is intended to include the equivalents of the claims and all modifications within the scope. In addition, the disclosed content described in the embodiments and each modification can be implemented alone or in combination as long as possible.

[0139] Reference Numeral Explanation

[0140] 10: Production line; 20: System; 100: Multi-joint robot; 110: End effector; 111: Camera; 112: Tool; 120: Base; 130: Fixture; 140: Workpiece; 150: Landmark; 200: Information processing device; 210: Simulator; 220: Control device; 301: Processor; 302: Memory; 303: Storage; 304: External device IF; 305: Input IF; 306: Output IF; 307: Communication IF; 308: Bus; 410: Flexure; 420: Torsion; 430: Combination of flexure and tilt; 440: Combination of torsion and tilt; 510: First calibration; 520: Second calibration; 530: Detection of position and orientation of landmark 150; 610: First imaging method; 620: Second imaging method; 630: Image; 1000: Screw.

Claims

1. A simulation system, wherein, the simulation system includes: a camera installed on a robot; and a simulation device, the camera sends an image of a landmark on a base of a production line configured in the real world to the simulation device, the simulation device analyzes a plurality of markers on the landmark evenly arranged in the image, estimates the deformation of the base based on the shape and arrangement interval of each of the plurality of markers observed from the camera, and in the production line reproduced in a 3D space, i.e., a three-dimensional space, reproduces the deformation of the base, and in the 3D space, corrects the teaching data in the machining process of a workpiece on the deformed base by a multi-joint robot, and feeds back the correction result of the teaching data to a control device of the multi-joint robot.

2. The simulation system according to claim 1, wherein, correcting the teaching data includes: adjusting the torque of a power tool equipped on the multi-joint robot.

3. The simulation system according to claim 1, wherein, correcting the teaching data includes: adjusting the force applied to the workpiece by the multi-joint robot in a fitting operation.

4. The simulation system according to any one of claims 1 to 3, wherein, reproducing the deformation of the base includes: reproducing the deformation of the workpiece on the base based on the deformation of the base.

5. The simulation system according to claim 4, wherein, the simulation device reads the material information of the workpiece from a memory, and calculates the correction amount of the teaching data based on the deformation amount of the workpiece and the material information of the workpiece.

6. The simulation system according to claim 5, wherein, calculating the correction amount of the teaching data includes: using the finite element method to calculate the stress of the deformed part of the workpiece; and calculating the correction amount of the teaching data based on the stress.

7. The simulation system according to any one of claims 1 to 3, wherein, the landmark includes a plurality of landmarks dispersedly arranged on the base, estimating the deformation of the base includes: estimating the deformation of each part of the base according to each of the plurality of landmarks.

8. A simulation method, wherein, the simulation method has the following steps: taking an image of a landmark on a base of a production line configured in the real world through a camera installed on a multi-joint robot; sending the image to a simulation device; analyzing a plurality of markers on the landmark evenly arranged in the image; estimating the deformation of the base based on the shape and arrangement interval of each of the plurality of markers observed from the camera; reproducing the deformation of the base in the production line reproduced in a 3D space based on the estimation result; correcting the teaching data in the machining process of a workpiece on the deformed base by a multi-joint robot in the 3D space; and feeding back the correction result of the teaching data to a control device of the multi-joint robot.

Images