Display device and display program

Abstract

The present invention provides a display device and a display program that facilitate confirmation of a work site's layout and the robot's ability to perform a task before actually setting up the robot at the work site. The display device comprises an augmented reality display unit that displays a virtual robot operating according to a predetermined program along with objects in real space; a position detection unit that detects the position of the work object in real space by measuring the distance from the augmented reality display unit to the work object; and a control unit that, based on the position of the work object detected by the position detection unit, causes the virtual robot displayed on the augmented reality display unit to operate so as to perform a predetermined task on the work object.

Description

Technical Field

[0001] The present invention relates to a display device and a display program. Background Art

[0002] Conventionally, in systems that perform predetermined operations on workpieces transported by a transport device, when additional robots are deployed during peak hours, for example, it is sometimes necessary to confirm the layout of the work site after the robots are deployed. Furthermore, in order to assess the improvement in operations brought about by the additional robots, it is also sometimes necessary to confirm the processing capabilities of the robots once they are deployed at the work site.

[0003] In addition, Patent Document 1 discloses a display control device whose purpose is to present information that allows visual understanding of the relationship between the current position of an operated device and the target position to which the operated device is to be moved. The display control device comprises: an information processing unit that, based on three-dimensional data of a driven device and three-dimensional data of a structure, generates three-dimensional data of an aiming coordinate axis that passes through a control point set at the driven device and includes a first straight line within a virtual space defined by the three-dimensional data of the structure; and an output control unit that, based on the three-dimensional data of the driven device, the three-dimensional data of the structure, and the three-dimensional data of the aiming coordinate axis generated by the information processing unit, generates control information for displaying the aiming coordinate axis on an output device. The first straight line is a line segment extending from the intersection of the control point and the surface of the structure.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent No. 6385627 Summary of the Invention

[0007] Problems to be solved by the invention

[0008] To confirm the layout of the work site after the robot is added, it is necessary to review the work site layout by adding the robot to the existing system diagram and confirming the robot's operating range. This leads to the problem of taking a long time to actually install the robot at the work site and start working.

[0009] Furthermore, to confirm the robot's processing capabilities beforehand, the robot has traditionally been operated by simulating the workpiece flow in offline software to evaluate the robot's operational improvements. However, offline software struggles to accurately reproduce the actual workpiece flow, making it difficult to accurately evaluate the operational improvements.

[0010] Therefore, it is desired to be able to easily confirm the layout of the work site and the processing capability of the robot for the work before actually installing the robot at the work site.

[0011] Solutions for solving problems

[0012] One embodiment of the display device disclosed herein comprises: an augmented reality display unit, which displays a virtual robot that acts according to a prescribed program together with objects in real space; a position detection unit, which detects the position of the work object by measuring the distance from the augmented reality display unit to the work object in real space; and a control unit, which causes the virtual robot displayed on the augmented reality display unit to act based on the position of the work object detected by the position detection unit to perform a prescribed operation on the work object.

[0013] One mode of the display program disclosed herein enables a computer to function as a display device, wherein the display device comprises: an augmented reality display unit, which displays a virtual robot that acts according to a prescribed program together with objects in real space; a position detection unit, which measures the position of a work object from the augmented reality display unit; and a control unit, which causes the virtual robot displayed on the augmented reality display unit to act based on the position of the work object measured by the position detection unit to perform a prescribed work on the work object.

[0014] Effects of the Invention

[0015] According to one embodiment of the display device of the present disclosure, it is possible to easily check the layout of a work site and the processing capability of the robot for a work before actually installing the robot at the work site.

[0016] Furthermore, according to one aspect of the display program of the present disclosure, the layout of the work site and the processing capability of the robot for the work can be easily confirmed by a computer before the robot is actually installed at the work site. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a diagram showing a usage state of a display device according to one embodiment of the present disclosure.

[0018] Figure 2 This is a block diagram illustrating a configuration of a display device according to one embodiment of the present disclosure.

[0019] Figure 3 This is a diagram illustrating a state in which the moving speed of a work object is measured in a display device according to one embodiment of the present disclosure.

[0020] Figure 4 This is a diagram showing a state in which information indicating that a predetermined task has been completed for a task target is displayed on a display device according to one embodiment of the present disclosure.

[0021] Figure 5 This is a flowchart illustrating a display method in a display device according to one embodiment of the present disclosure.

[0022] Figure 6 This is a schematic diagram showing a display device according to another embodiment of the present disclosure.

[0023] Description of Reference Numerals

[0024] 1. 1A: Display device; 2: Augmented reality display unit; 3: Position detection unit; 4: Control unit; 5: Work object management unit; 10: Virtual robot; 20: Information indicating that the specified work for the work object has been completed; 100: Work site (object); 200: Transport device (object); 300: Work object (object); WK: Worker (object). DETAILED DESCRIPTION

[0025] Hereinafter, one embodiment of the present disclosure will be described with reference to the accompanying drawings. Figure 1 This is a diagram showing a usage state of a display device according to one embodiment of the present disclosure. Figure 2 This is a block diagram illustrating a configuration of a display device according to one embodiment of the present disclosure.

[0026] The display device 1 is a device that can confirm the layout of the work site 100 after the robot is installed and the processing capacity of the robot in advance when considering installing a new robot in the work site 100 in a factory, for example. Figure 1 The display device 1 shown is constituted by a tablet computer including a liquid crystal monitor screen as a display portion.

[0027] exist Figure 1 The work site 100 (object) in the real space shown is provided with a conveying device 200 (object). The conveying device 200 is moved along Figure 1 A plurality of work objects 300 (objects) are conveyed in one direction indicated by the hollow arrows in FIG. Two workers WK (objects) are shown at the work site 100 working on the work objects 300 conveyed by the conveying device 200 .

[0028] like Figure 2 As shown, the display device 1 includes an augmented reality display unit 2 , a position detection unit 3 , a control unit 4 , and a work object management unit 5 .

[0029] like Figure 1As shown, the augmented reality display unit 2 displays the virtual robot 10 along with objects in real space (in this embodiment, the work site 100, the transport device 200, the work object 300, and the worker WK). The augmented reality display unit 2 is typically composed of a computer monitor screen and displays the aforementioned objects in real space, captured by a camera provided on the display device 1, along with the virtual robot 10.

[0030] The virtual robot 10 is a virtual object displayed on the augmented reality display unit 2 according to a prescribed program pre-stored in the display device 1, and does not exist in the real space. However, the augmented reality display unit 2 displays the virtual robot 10 in three dimensions by overlapping it with objects in the real space, thereby displaying an augmented reality image as if the virtual robot 10 exists (is set) in the work site 100 in the real space. The virtual robot 10 can move on the augmented reality display unit 2 in the same manner as when the robot is actually set in the real space, performing prescribed operations based on the work program on the work object 300. The prescribed operations include all operations that the robot can perform on the work object 300, such as the removal operation of grasping the work object 300 and moving it from the conveying device 200 to other places, and the sticker affixing operation.

[0031] The augmented reality display unit 2 is configured to enable the virtual robot 10 to be positioned at any desired location. For example, if the augmented reality display unit 2 is configured as a touch panel, the operator of the display device 1 can touch the virtual robot 10 with a finger or a stylus and move it to any desired location within the augmented reality display unit 2. This allows the virtual robot 10 to be positioned at any desired location within the real space displayed on the augmented reality display unit 2. The display of the virtual robot 10 on the augmented reality display unit 2 is achieved through the functions of the control unit 4, which will be described later.

[0032] The position detection unit 3 has a function of detecting the position of the work object 300 from the augmented reality display unit 2. Specifically, the position detection unit 3 is configured to include an imaging device such as a two-dimensional camera capable of photographing the work object 300 and a distance image sensor. Figure 1 The example in which the position detection unit 3 is composed of a distance image sensor is shown, and the distance image sensor is provided on the surface of the display device 1 opposite to the surface on which the augmented reality display unit 2 is configured. However, the position detection unit 3 may also be a camera device and a distance image sensor provided independently of the display device 1. When a camera device such as a two-dimensional camera is used, the position of the work object 300 can be detected by machine learning of the appearance and position of the work object 300. Figure 1 As shown, the position detection unit 3 composed of the distance image sensor can also make the above-mentioned objects in the real space captured as Figure 1 The function shown is displayed on the augmented reality display unit 2. The position detection unit 3 measures the distance from the augmented reality display unit 2 to the work object 300 by performing image processing on the image of the work object 300 in real space according to a predetermined processing procedure. The position detection unit 3 detects the position of the work object 300 through the function of the work object management unit 5 described later.

[0033] The control unit 4 has the following functions: displaying the virtual robot 10 in three dimensions overlapping with the objects displayed in the real space on the augmented reality display unit 2, and causing the virtual robot 10 displayed on the augmented reality display unit 2 to move based on the position of the work object 300 detected by the position detection unit 3 to perform a prescribed work on the work object 300.

[0034] Specifically, the control unit 4 is composed of a CPU (Central Processing Unit) and other processing devices. In addition, the control unit 4 has auxiliary storage devices such as HDD (Hard Disk Drive) and SSD (Solid State Drive), and main storage devices such as RAM (Random Access Memory). The auxiliary storage devices store various programs for executing various functions of the display device 1, and the main storage devices are used to store data temporarily required for the execution of the programs by the processing device. In the control unit 4, the processing device reads various programs from the auxiliary storage device and expands the read programs in the main storage device while performing processing based on these various programs. The control unit 4 is configured to function as the display device 1 by controlling the various hardware (augmented reality display unit 2, position detection unit 3, etc.) connected to the control unit 4 based on its calculation results.

[0035] The control unit 4 adjusts the size of the virtual robot 10 displayed on the augmented reality display unit 2 in a manner that matches the size of the object in the real space displayed together with the virtual robot 10. Alternatively, the control unit 4 adjusts the size of the object in the real space displayed on the augmented reality display unit 2 in a manner that matches the size of the virtual robot 10 displayed on the augmented reality display unit 2. The size of the object in the real space is obtained based on the image of the object in the real space captured by the position detection unit 3 or other camera devices. Thus, the control unit 4 displays the virtual robot 10 overlapping with the object in the real space (the work site 100, the conveying device 200, the work object 300, and the worker WK) in the augmented reality display unit 2 in a manner that maintains the relative positional relationship in the real space with the objects in the real space.

[0036] The work object management unit 5 measures the current position of the work object 300 based on the position of the work object 300 detected by the position detection unit 3. Specifically, since the work object 300 on the conveying device 200 is continuously or intermittently conveyed at a constant speed by the conveying device 200, in order for the virtual robot 10 to perform a predetermined operation on the work object 300, it is necessary to constantly grasp the current position of the work object 300 on the augmented reality display unit 2 based on the moving speed of the work object 300 (the conveying speed of the conveying device 200) and confirm whether the work object 300 is within the working range of the virtual robot 10. Therefore, the work object management unit 5 uses the position detection unit 3 to continuously perform at least two position detections on any one of the multiple work objects 300 displayed on the augmented reality display unit 2 and conveyed by the conveying device 200, each considered to be the same object, and measures the moving speed of the work object 300 based on the detection results.

[0037] Figure 3 The figure shows a case where the moving speed of the work object 300 is measured on the display device 1. For example, when focusing on a work object 300 at the left end displayed on the conveying device 200 of the augmented reality display unit 2, the position of the work object 300 is detected twice continuously. In the first detection, the work object 300a ( Figure 3 ) is detected in the second detection, and the work object 300b (at Figure 3 ) is shown by a solid line. The work object management unit 5 measures the moving speed of the work object 300 by dividing the distance moved from the position of the work object 300a to the position of the work object 300b by the time interval from the first position detection to the second position detection. Based on the measured moving speed of the work object 300, the work object management unit 5 detects the current position of the work object 300 on the augmented reality display unit 2 at a predetermined control cycle. All or part of the functions of the work object management unit 5 may be included in the control unit 4.

[0038] Next, use Figure 5 The display method using the display device 1 is described in the flowchart shown in FIG. First, the operator starts the display device 1 and uses the position detection unit 3 which also functions as a camera device as shown in FIG. Figure 1 The work site 100 in the real space is captured as shown. As a result, the display device 1 displays the virtual robot 10 on the augmented reality display unit 2, superimposed on the captured images of the objects in the real space (the work site 100, the transport device 200, the work object 300, and the worker WK) (S1).

[0039] The position and posture of the virtual robot 10 are represented by a coordinate system that is identical to the coordinate system for the position of an object in real space obtained by the position detection unit 3. The operator touches and moves the virtual robot 10 with a finger or a stylus pen to place the virtual robot 10 at any desired location on the augmented reality display unit 2.

[0040] After the virtual robot 10 is positioned at an arbitrary position on the augmented reality display unit 2, the display device 1 uses the position detection unit 3 to detect the position of the work object 300 on the augmented reality display unit 2, for example, using an operator's input instruction to start the movement of the virtual robot 10 as a trigger. The display device 1 then measures the movement speed of the work object 300 based on this position and, based on this measurement result, continues to measure the current position of each work object 300 on the augmented reality display unit 2. Based on the measured current position of the work object 300, the display device 1 causes the virtual robot 10 displayed on the augmented reality display unit 2 to move and perform a predetermined work on the work object 300 (S2).

[0041] After the virtual robot 10 begins operating, the display device 1 monitors the control unit 4 to see whether the current position of the work object 300 has entered the operating range of the virtual robot 10. If the work object 300 on the augmented reality display unit 2 has entered the operating range of the virtual robot 10, the display device 1 causes the virtual robot 10 to operate while tracking the work object 300 based on the current position of the work object 300. This allows the virtual robot 10 to perform actions such as virtually grasping the work object 300 on the augmented reality display unit 2 and moving it to another location.

[0042] The virtual robot 10 determines that the work has been completed if the virtual robot 10 can move while tracking the current position of the work object 300 and virtually grasp the work object 300. Therefore, the display device 1 monitors the positional relationship between the virtual robot 10 (e.g., the position of the front end of the virtual robot 10) and the work object 300 on the augmented reality display unit 2 via the control unit 4 or the work object management unit 5 to detect whether the virtual robot 10 has completed the work on the work object 300 (S3).

[0043] In the above step S3, when the virtual robot 10 has completed the work on the work object 300 (“Yes” in step S3), Figure 4 As shown, the display device 1 displays a work completion mark 20 indicating that the work has been completed, superimposed on the work object 300 c where the work has been completed ( S4 ). Figure 4The work completion marker 20 is shown as a circle surrounding the work object 300c. However, as long as the completed work object 300c can be identified on the augmented reality display unit 2, the work completion marker 20 may be any marker. In step S3, if the virtual robot 10 has not completed the work on the work object 300 ("No" in step S3), the display device 1 does not display the work completion marker 20 for the work object 300 and the process proceeds to step S5.

[0044] The operation of the virtual robot 10 is continued while the work object 300 is being transported by the transport device 200 (S5). If the work object 300 being transported by the transport device 200 is present on the augmented reality display unit 2, or if there is no input instruction from the operator to terminate the operation of the virtual robot 10 ("No" in step S5), the process from step S3 onwards is repeated. If there is no work object 300 being transported by the transport device 200, or if there is an input instruction from the operator to terminate the operation of the virtual robot 10, the operation of the virtual robot 10 is stopped.

[0045] The display device 1 is not limited to being composed of a tablet computer. The display device 1 can be a computer having a display unit capable of displaying images, for example, a notebook computer, other portable terminals, etc. In addition, it can also be a tablet computer. Figure 6 1A, a head-mounted display integrally includes a position detection unit 3 and an augmented reality display unit 2. The control unit 4 and the work object management unit 5 may be built into the head-mounted display, or may be provided independently of the head-mounted display and connected to the head-mounted display in a communicative manner via a wired or wireless connection.

[0046] Furthermore, the augmented reality display unit 2, position detection unit 3, control unit 4, and work object management unit 5 in the display devices 1 and 1A can each be implemented by hardware, software, or a combination thereof. Furthermore, the display method performed by the coordinated operation of the augmented reality display unit 2, position detection unit 3, control unit 4, and work object management unit 5 can also be implemented by hardware, software, or a combination thereof. Here, implementation by software means implementation by a computer reading and executing a program.

[0047] The program is saved using various types of non-transitory computer-readable media and provided to the computer. Non-transitory computer-readable media include various types of tangible storage media. Non-transitory computer-readable media include magnetic recording media (e.g., floppy disks, magnetic tapes, hard disk drives), optical magnetic recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, RAMs (Random Access Memory)). In addition, the program can be provided to various types of transient computers. Transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transient computer-readable media provide the program to the computer via a wireless communication path or a wired communication path such as an electric wire and an optical fiber.

[0048] One embodiment of the display device 1 or 1A of the present disclosure described above produces the following effects.

[0049] The display device 1 or 1A of the present disclosure includes an augmented reality display unit 2 that displays a virtual robot 10 operating according to a predetermined program along with objects in real space (a work site 100, a transport device 200, a work object 300, and a worker WK); a position detection unit 3 that detects the position of the work object 300 by measuring the distance from the augmented reality display unit 2 to the work object 300 in real space; and a control unit 4 that causes the virtual robot 10 displayed on the augmented reality display unit 2 to operate so as to perform a predetermined operation on the work object 300 based on the position of the work object 300 detected by the position detection unit 3. This allows the layout of the robot installed in the work site 100 and the robot's operation to be confirmed on the augmented reality display unit 2. Therefore, there is no need to add the robot to an existing system diagram or simulate the workpiece flow in offline software to confirm the layout of the work site after the robot is installed. This allows for easy confirmation of the layout of the work site 100 and the robot's ability to handle the work before the robot is actually installed in the work site 100.

[0050] Furthermore, a work object management unit 5 is provided, which measures the current position of the work object 300 based on the position of the work object 300 detected by the position detection unit 3. This allows the relative positional relationship between the virtual robot 10 and the work object 300 on the augmented reality display unit 2 to be accurately grasped, thereby enabling the robot's ability to handle work to be more accurately determined.

[0051] The work object management unit 5 measures the moving speed of the work object 300 based on the results of at least two position detections of the work object 300, which is considered to be the same object, by the position detection unit 3, and measures the current position of the work object 300 based on the moving speed. In this way, the work object management unit 5 can accurately grasp the moving speed of the work object 300 without being affected by the conveying accuracy of the conveying device 200, and can measure the accurate current position of the work object 300.

[0052] When the current position of the work object 300 measured by the work object management unit 5 enters the working range of the virtual robot 10, the control unit 4 causes the virtual robot 10 to operate while tracking the work object 300 based on the current position of the work object 300. This allows the virtual robot 10 to operate on the work object 300 in the same manner as a real robot would. Consequently, the operator can more accurately grasp the real robot's handling capabilities for the work object 300 on the augmented reality display unit 2.

[0053] The augmented reality display unit 2 is configured to be able to place the virtual robot 10 at any position. Thus, the virtual robot 10 on the augmented reality display unit 2 can be placed in accordance with the placement of objects in the real space displayed on the augmented reality display unit 2 to match the actual robot's installation position.

[0054] The augmented reality display unit 2 displays information indicating that a predetermined work on the work object 300 has been completed, superimposed on the work object 300 . This allows the operator to easily check the status of the work performed on the work object 300 by the virtual robot 10 on the augmented reality display unit 2 .

[0055] Furthermore, the display program disclosed herein causes a computer to function as a display device 1 or 1A. The display device 1 or 1A includes: an augmented reality display unit 2 that displays a virtual robot 10 operating according to a predetermined program, along with objects in real space (a work site 100, a transport device 200, a work object 300, and a worker WK); a position detection unit 3 that detects the position of the work object 300 in real space by measuring the distance from the augmented reality display unit 2 to the work object 300; and a control unit 4 that causes the virtual robot 10 displayed on the augmented reality display unit 2 to operate, based on the position of the work object 300 detected by the position detection unit 3, so as to perform a predetermined task on the work object 300. Thus, by executing the display program on a computer equipped with the display device 1 or 1A including the augmented reality display unit 2, the position detection unit 3, and the control unit 4, the layout of the work site 100 and the robot's ability to handle tasks can be easily confirmed using the display device 1 or 1A before the robot is actually installed in the work site 100.

Claims

1. A display device comprising: an augmented reality display unit that virtually displays a virtual robot operating in accordance with a predetermined program together with objects at the real work site in order to confirm the layout of the work site and the robot's ability to handle the work before setting up the robot at the real work site; a position detection unit configured to detect a position of a work object from the augmented reality display unit; as well as a control unit that causes the virtual robot displayed on the augmented reality display unit to operate based on the position of the work object detected by the position detection unit to perform a predetermined work on the work object; The control unit adjusts the size of the virtual robot in a manner that matches the size of the object at the work site displayed on the augmented reality display unit, or adjusts the size of the object at the work site displayed on the augmented reality display unit in a manner that matches the size of the virtual robot.

2. The display device according to claim 1, wherein The device further includes a work object management unit configured to measure a current position of the work object based on the position of the work object detected by the position detection unit.

3. The display device according to claim 2, wherein: The work object management unit measures a moving speed of the work object based on the result of at least two position detections of the work object considered as the same object by the position detection unit, and measures a current position of the work object based on the moving speed.

4. The display device according to claim 2, wherein When the current position of the work object measured by the work object management unit enters the working range of the virtual robot, the control unit causes the virtual robot to operate while tracking the work object based on the current position of the work object.

5. The display device according to claim 3, wherein When the current position of the work object measured by the work object management unit enters the working range of the virtual robot, the control unit causes the virtual robot to operate while tracking the work object based on the current position of the work object.

6. The display device according to any one of claims 1 to 5, wherein: The augmented reality display unit is configured to be able to arrange the virtual robot at an arbitrary position.

7. The display device according to any one of claims 1 to 5, wherein: The augmented reality display unit displays information indicating that a predetermined task on the task object has been completed, superimposed on the task object.

8. The display device according to claim 6, wherein: The augmented reality display unit displays information indicating that a predetermined task on the task object has been completed, superimposed on the task object.

9. A display program product comprising a display program, wherein the display program causes a computer to function as a display device. The display device comprises: an augmented reality display unit that virtually displays a virtual robot operating in accordance with a predetermined program together with objects at the real work site in order to confirm the layout of the work site and the robot's ability to handle the work before setting up the robot at the real work site; a position detection unit configured to detect a position of a work object from the augmented reality display unit; as well as a control unit that causes the virtual robot displayed on the augmented reality display unit to operate based on the position of the work object detected by the position detection unit to perform a predetermined work on the work object; The control unit adjusts the size of the virtual robot in a manner that matches the size of the object at the work site displayed on the augmented reality display unit, or adjusts the size of the object at the work site displayed on the augmented reality display unit in a manner that matches the size of the virtual robot.

Images