Robotic system

Abstract

The present invention provides a robot system that can accurately measure the positional relationship between an AR device and a marker regardless of the type of robot, and can use the AR device to relatively easily and accurately identify the position or orientation of the robot. The system comprises: a marker detection unit that simultaneously detects a reference marker and a robot coordinate system determination marker in a single detection operation; a robot system information receiving unit that receives information related to the robot system; a robot coordinate system determination unit that determines the robot's coordinate system based on the position and coordinate system information of the robot coordinate system determination marker; an AR device that displays information related to the robot system based on the robot's coordinate system; a coordinate system setting unit that sets the origin coordinate system by moving the robot to a specified position; and a coordinate system information sending unit that sends the coordinate system information set by the coordinate system setting unit to the AR device.

Description

Technical Field

[0001] The present invention relates to a robot system. Background Art

[0002] In robotic systems such as industrial robots, technologies using augmented reality (AR) are attracting attention and are being actively researched and developed.

[0003] In this type of robot system, for example, a system with the following structure is proposed: based on an image of a robot captured by a photographic device, a CG image of the robot is generated, and by the user touching the CG image displayed on the touch screen, the robot in the CG image on the touch screen is made to move virtually, and the actual robot is made to move following the movement of the virtual robot.

[0004] Furthermore, by superimposing the setting information set in the robot system on the image of the robot to generate a CG image of the setting information, the user can more intuitively understand the setting information. In addition, the CG image of the setting information can also be touched.

[0005] Here, in this robot system, it is important to correct the position of the robot's moving destination in a manner that follows the processing object based on the image of the processing object captured by the imaging device, and it is necessary to pre-set information related to the positions of the robot, the imaging device, and the processing object.

[0006] Information related to the positions of the robot, the photographic device, and the object to be processed includes, for example, (1) the origin and directions of the axes of the coordinate system serving as a reference for position correction of the robot, (2) the position of the robot serving as a reference for position correction of the robot (rotation angles or translation amounts of the joints of the robot, etc.), (3) the relative positional relationship between the robot and the photographic device, and (4) the relative positional relationship between the photographic device and the object to be processed.

[0007] In addition, as a method of using an AR device to identify the robot coordinate system, the following methods have been proposed: setting a mark at a specified position of the robot, and using the AR device to detect the mark to identify the robot coordinate system; using the AR device to identify the shape of the robot to identify the robot coordinate system; manually making the position and orientation of the AR graphic displayed at a certain position and orientation consistent with the actual position and orientation of the robot.

[0008] For example, Patent Document 1 discloses "a robot system comprising: a robot arm; a camera configured to photograph a workpiece; a calibration jig mounted on the front end of the robot arm and having a mark capable of image recognition; and a calibration device for deriving a correlation between camera coordinates and robot coordinates, the camera coordinates being coordinates within an image photographed by the camera, the robot coordinates being coordinates with the robot arm as a reference, the calibration device comprising: an arm control unit that controls the robot arm so as to move the mark to a plurality of photographic positions within a plane orthogonal to the optical axis of the camera while the mark is directed toward the camera; a camera coordinate acquisition unit that acquires the camera coordinates of the mark when the mark is at the photographic position; a posture information acquisition unit that acquires posture information of the robot when the mark is at the photographic position; and a correlation derivation unit that derives the correlation between the camera coordinates and the robot coordinates based on the camera coordinates and the posture information acquired by the camera coordinate acquisition unit and the posture information acquisition unit respectively."

[0009] Prior art literature

[0010] Patent Literature

[0011] Patent Document 1: Japanese Patent Application Laid-Open No. 2014-180720 Summary of the Invention

[0012] Problems to be solved by the invention

[0013] However, the method of using AR equipment (augmented reality display devices, etc.) to detect markers and identify the robot's coordinate system requires pre-determining the location of the markers according to the type of robot. Therefore, it places a heavy burden on the AR provider in terms of time and labor.

[0014] In the method of recognizing the shape of the robot using the AR device to recognize the robot coordinate system, it is also necessary to record (register) the robot model by type in advance, which increases the burden on the AR provider side.

[0015] The method of manually matching the position and orientation of an AR graphic displayed at a certain position and orientation with the actual position and orientation of a robot places a very heavy burden on the AR user.

[0016] Furthermore, if the AR device is moved away from the marker after recognizing the robot's coordinate system, the positional relationship between the AR device and the marker may not be accurately determined, resulting in a shift in the displayed AR graphics. AR devices use gyroscopes and other sensors to detect movement, but even when position and orientation are corrected based on these detections, display shifts can occur. This is particularly true during rotational movement.

[0017] Therefore, there is a strong demand for the development of a method that can accurately measure the positional relationship between the AR device and the marker regardless of the type of robot, and can recognize the position or orientation of the robot relatively easily and accurately using the AR device.

[0018] Solutions for solving problems

[0019] One embodiment of the robot system disclosed herein is configured to include: at least one or more reference markers, each of which is set at a specified position and has a reference feature point with clear coordinates; a robot coordinate system determination marker, which is set within the robot's motion range and has a determination feature point for determining the robot coordinate system; a marker detection unit, which simultaneously detects the robot coordinate system determination marker and at least one of the reference markers through a single detection operation; a robot system information receiving unit, which receives information related to the robot system; a robot coordinate system determination unit, which determines the robot's coordinate system based on at least the position of the robot coordinate system determination marker detected by the marker detection unit and the coordinate system information received by the robot system information receiving unit; an augmented reality display device, which is used to display the information related to the robot system received by the robot system information receiving unit based on the robot's coordinate system determined by the robot coordinate system determination unit; a coordinate system setting unit, which sets the origin coordinate system by moving the robot to a specified position; and a coordinate system information sending unit, which sends the coordinate system information set by the coordinate system setting unit to the augmented reality display device.

[0020] Effects of the Invention

[0021] In one embodiment of the robot system disclosed herein, the position and orientation of the robot can be relatively easily identified using an AR device (augmented reality display device), regardless of the robot type. Furthermore, by placing a reference marker within a range within which the reference marker can be detected within the AR device's range of motion, it is possible to suppress deviations in the display position of the AR graphic.

[0022] Therefore, according to one embodiment of the robot system disclosed herein, the positional relationship between the AR device and the robot coordinate system determination mark and the reference mark can be accurately measured regardless of the type of robot, and the position or orientation of the robot can be identified relatively easily and with high precision using the AR device. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a diagram showing a robot system of one embodiment.

[0024] Figure 2 This is a diagram showing a robot system of one embodiment.

[0025] Figure 3 This is a diagram used in explaining a method of suppressing positional deviation by detecting a plurality of markers (markers) in a robot system of one embodiment.

[0026] Figure 4 This is a diagram used to explain a method of detecting a marker (marker) and suppressing positional deviation in a robot system according to one embodiment. DETAILED DESCRIPTION

[0027] Below, refer to Figures 1 to 4 A robot system according to one embodiment will be described.

[0028] like Figure 1 As shown, the robot system 1 of this embodiment is constructed as follows: at least one or more reference marks (at least one or more reference feature points with clear coordinates) A, B... with clear coordinates are set within the range in which an AR device (augmented reality display device) 2 such as a head-mounted display or a projector is to be moved, and a robot coordinate system determination mark (a robot coordinate system determination mark with a determination feature point for determining the robot coordinate system) M for determining the robot coordinate system is set within the movement range of the robot 3. By using the AR device 2 to simultaneously detect the robot coordinate system determination mark M and at least two of the respective reference marks A, B..., the position or orientation of the coordinate system set on the robot coordinate system determination mark M relative to the reference marks A, B... is identified.

[0029] In addition, the structure of the augmented reality display device 2 does not need to be particularly limited as long as it includes at least an application for displaying a virtual model corresponding to a real device using an AR display unit (augmented reality display unit).

[0030] In this robot system 1, by setting the coordinate system of the robot 3 on the mark M for determining the robot coordinate system, the coordinate system of the robot 3 can be identified by using the AR device 2. For example, for the output line of the workpiece transported by the conveyor, the line on the production line in the factory where there are many obstacles and the AR device 2 needs to bypass the obstacles, etc., the position, orientation, etc. of the robot 3 can also be determined and displayed with high precision, thereby realizing a useful and better system.

[0031] More specifically, if Figure 1 As shown, the robot system 1 of this embodiment has at least two or more reference markers A, B, etc. with clear coordinates set within the moving range of the AR device 2 such as a head-mounted display and a projector, and at least one or more robot coordinate system determination markers M set within the movement range of the robot 3 for determining the robot coordinate system.

[0032] The robot system 1 of this embodiment is configured to include: a mark detection unit (marker detection unit) 4, which detects at least the positions of the reference marks A, B... and the robot coordinate system determination mark M set at specified positions; a robot system information receiving unit 5, which receives information related to the robot system 1; a robot coordinate system determination unit 6, which determines the coordinate system of the robot based on at least the positions of the reference marks A, B... and the robot coordinate system determination mark M detected by the mark detection unit 4 and the coordinate system information received by the robot system information receiving unit 5; a coordinate system setting unit 7, which sets the origin coordinate system by moving the robot 3 to a specified position; and a coordinate system information sending unit 8, which sends the coordinate system information set by the coordinate system setting unit 7 to the AR device 2, wherein the display unit (application) of the AR device 2 is used to display the information related to the robot system 1 received by the robot system information receiving unit 5 based on the coordinate system of the robot 3 determined by the robot coordinate system determination unit 6.

[0033] The robot coordinate system determination unit 6 is configured to determine the coordinate system of the robot 3 based on the coordinate system set by the coordinate system setting unit 7 on the robot coordinate system determination marker M. Furthermore, the posture of the robot coordinate system determination marker M in this embodiment is uniquely determined, and the robot coordinate system determination marker M is provided within the operating range of the robot 3 or on the robot itself.

[0034] The robot system 1 is configured to identify the position and orientation of the coordinate system set on the robot coordinate system determination mark M relative to the reference marks A, B... by simultaneously detecting the robot coordinate system determination mark M and at least two of the reference marks A, B... using the AR device 2.

[0035] <Example 1: Positioning Based on the Coordinate System Set on the Robot Coordinate System Determining Marker>

[0036] Here, an example of performing “position alignment based on the coordinate system set on the marker” in the robot system 1 of the present embodiment configured as described above will be described below.

[0037] like Figure 1 As shown in FIG, reference markers A and B are set in the AR graphic. In this case, reference markers A and B are set within a range that can be detected simultaneously by the AR device 2.

[0038] The AR device 2 simultaneously detects the reference mark A and the reference mark B. Thus, the AR device 2 recognizes the positions of the reference mark A and the reference mark B.

[0039] Next, a robot coordinate system determining marker M is set within the operating range of the robot 3 and within a range that can be detected simultaneously with any reference marker (here, the reference marker A) by the AR device 2 .

[0040] The robot 3 is moved to correct the origin, X-direction point, and Y-direction point of the robot coordinate system determination marker M using the tool of the robot 3 (origin coordinate system setting).

[0041] Next, the AR device 2 simultaneously detects the robot coordinate system determining marker M and the reference marker A. Thus, the AR device 2 recognizes the coordinate system on the robot coordinate system determining marker M and identifies the positional relationship between the reference marker A, the reference marker B, and the robot coordinate system determining marker M. Furthermore, the coordinate system on the robot coordinate system determining marker M is used to identify the coordinate system of the robot 3.

[0042] Then, the robot 3 transmits the robot coordinate system corresponding to the coordinate system on the robot coordinate system identification marker M to the AR device 2 (from the robot controller to the AR device controller). This allows the AR device 2 to recognize the coordinate system of the robot 3 .

[0043] Furthermore, as long as the reference marks A and B are detectable by the AR device 2 , it is possible to suppress a shift in the display position of the AR graphic.

[0044] Furthermore, after the coordinate system of the robot 3 is recognized by the AR device 2 , the robot coordinate system identification mark M may be removed from the installation location.

[0045] <Example 2: Positioning Based on the Coordinate System Set on the Robot Coordinate System Determining Marker Provided on the Robot Hand>

[0046] Next, the following describes how Figure 2 This is an example of performing “position alignment based on the coordinate system set on the mark (tool coordinate system mark) provided on the hand of the robot” as shown.

[0047] In the AR graphic, reference markers A and B are set. At this time, reference markers A and B are set within a range that can be simultaneously detected by the AR device 2 .

[0048] The AR device 2 simultaneously detects the reference mark A and the reference mark B. Thus, the AR device 2 recognizes the positions of the reference mark A and the reference mark B.

[0049] Next, a robot coordinate system determining marker M is set on the hand of the robot 3. A tool coordinate system is set for the set robot coordinate system determining marker M.

[0050] The AR device 2 simultaneously detects the robot coordinate system determining marker M and the reference marker A. Thus, the AR device 2 identifies the tool coordinate system on the robot coordinate system determining marker M and recognizes the positional relationship between the reference marker A, the reference marker B, and the robot coordinate system determining marker M. Furthermore, the tool coordinate system on the robot coordinate system determining marker M is used to identify the coordinate system of the robot 3.

[0051] Then, the robot 3 transmits the robot coordinate system corresponding to the coordinate system on the robot coordinate system identification marker M to the AR device 2 (from the robot controller to the AR device controller). This allows the AR device 2 to recognize the coordinate system of the robot 3 .

[0052] Furthermore, as long as the reference marks A and B are detectable by the AR device 2 , it is possible to suppress a shift in the display position of the AR graphic.

[0053] Furthermore, after the coordinate system of the robot 3 is recognized by the AR device 2 , the robot coordinate system identification mark M may be removed from the installation location.

[0054] Therefore, in the robot system 1 of this embodiment, the position and orientation of the robot 3 can be relatively easily identified using the AR device 2, regardless of the type of robot 3. Furthermore, by pre-setting reference markers A, B, etc. within the movement range of the AR device 2, it is possible to suppress deviations in the display position of the AR graphic as long as the reference markers A, B, etc. are within the detectable range.

[0055] Therefore, according to the robot system 1 of this embodiment, the positional relationship between the AR device 2 and the robot coordinate system determination marker M can be always accurately measured regardless of the type of robot 3, and the position or orientation of the robot 3 can be identified relatively easily and accurately using the AR device 2.

[0056] In addition, in the robot system 1 of this embodiment, Figure 3 When there are three or more reference marks A, B, C, D... as shown, the robot coordinate system can be identified based on any reference mark A, B, C, D... by appropriately detecting two of the reference marks A, B, C, D... at the same time each time.

[0057] And, as Figure 4 As shown, by detecting any of the reference marks A, B, C, D, etc. using the AR device 2, it is possible to suppress the deviation of the display of the AR graphic.

[0058] An embodiment of the robot system has been described above, but the robot system is not limited to the above-described embodiment and can be modified appropriately without departing from the spirit and scope of the invention.

[0059] Description of Reference Numerals

[0060] 1: Robot system; 2: AR device (augmented reality display device); 3: Robot; 4: Marker detection unit (marker detection unit); 5: Robot system information receiving unit; 6: Robot coordinate system determination unit; 7: Coordinate system setting unit; 8: Coordinate system information sending unit; A~D: Benchmark marks (benchmark marks); M: Robot coordinate system determination mark (robot coordinate system determination mark).

Claims

1. A robotic system comprising: One or more reference markers, each of which is provided at a predetermined position and has a reference feature point with clear coordinates; A robot coordinate system determination marker, which is provided within the robot's operating range and has a characteristic point for determining the robot coordinate system; a marker detection unit configured to simultaneously detect the robot coordinate system determination marker and at least one of the reference markers through a single detection operation; a robot system information receiving unit for receiving information related to the robot system; a robot coordinate system determination unit that determines the robot coordinate system based on at least the position of the robot coordinate system determination marker detected by the marker detection unit and the coordinate system information received by the robot system information reception unit; an augmented reality display device for displaying information related to the robot system received by the robot system information receiving unit based on the coordinate system of the robot determined by the robot coordinate system determining unit; a coordinate system setting unit that sets an origin coordinate system by moving the robot to a specified position; as well as A coordinate system information sending unit sends the coordinate system information set by the coordinate system setting unit to the augmented reality display device.

2. The robot system according to claim 1, wherein: The robot coordinate system determination unit determines the coordinate system of the robot based on the coordinate system set on the robot coordinate system determination marker by the coordinate system setting unit.

3. The robot system according to claim 1 or 2, wherein: The posture of the robot coordinate system determining marker is uniquely determined, and the robot coordinate system determining marker is provided within the operating range of the robot or on the robot itself.

Images