Display system, display method, and program for remote operation

Abstract

To display on a head-mounted display a situation of a remote site and a state of an at-hand controller regardless of types of a work machine and a controller for operating the work machine.SOLUTION: A display system comprises: a stereoscopic camera composed of a right-eye camera and a left-eye camera and mounted on a work machine; an at-hand camera for imaging operation means for operating the work machine; an image composition unit for composing an image captured by the stereoscopic camera and an image captured by the at-hand camera and generating a right-eye image and a left-eye image; and a head-mounted display device for displaying the right-eye image and the left-eye image generated by the image composition unit. The display system further comprises a calibration unit for calculating a correction value at a three-dimensional coordinate axis of the right-eye camera or the left-eye camera on the basis of a reference marker imaged by the stereoscopic camara and image patterns respectively positioned on the right and left sides of the reference marker.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a display system for remotely operating a work machine on site, and in particular to a display system, display method, and program that displays images from a stereo camera attached to the work machine as a three-dimensional image on a head-mounted display device worn by the operator. [Background technology]

[0002] Conventionally, there have been proposed technologies for remotely operating a work machine at a distance. Patent Document 1 proposes a system in which an imaging device is attached to a shovel, which is a work machine, and the captured image is displayed on a head-mounted display worn by the operator, so that the operator can remotely control the shovel using a remote control device in order to enable the operator to see what he or she wants to see in a more intuitive way through an imaging device attached to the shovel to be remotely controlled.In this system, when a wearable sensor detects a predetermined movement of the operator's head, the system switches between distant imaging devices or rotates a pan head on which the imaging device is mounted, depending on the predetermined movement, and displays the desired image on the head-mounted display.

[0003] Furthermore, Patent Document 2 proposes a system in which panoramic images for the right and left eyes generated by multiple camera units installed on a distant work machine are cropped in accordance with the movement of the operator's line of sight, and the cropped images are transmitted to the operator's head-mounted display for display. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-226094 [Patent Document 2] Patent No. 6612865 Summary of the Invention [Problem to be solved by the invention]

[0005] In both of the technologies described in Patent Documents 1 and 2, the operator operates a controller (remote control device) at hand to operate the work machine while viewing images of the work site displayed on a head-mounted display, but wearing a head-mounted display prevents the operator from seeing the controller at hand. To address this issue, for example, a touch sensor could be provided on the controller to detect finger movements, thereby displaying the position of the operator's fingers on the controller on the head-mounted display, but different controllers are often used depending on the type of work machine, and developing a display program for the controller image each time is costly.

[0006] The present invention has been made in response to such conventional circumstances, and aims to provide a display system, display method, and program for remote operation that can display the situation of a distant work site and the status of a controller at hand on a head-mounted display, regardless of the type of work machine or the controller that operates it. [Means for solving the problem]

[0007] In order to achieve the above object, the display system of the present invention comprises: It has a right-eye camera and a left-eye camera. , written by a stereo camera provided on the industrial machine; a handheld camera that photographs an operating means for operating the work machine; an image synthesis unit that synthesizes an image captured by the stereo camera and an image captured by the handheld camera to generate an image for the right eye and an image for the left eye; a head-mounted display device that displays the right-eye image and the left-eye image generated by the image synthesis unit; and A display system for remotely operating a work machine, comprising: a calibration unit that calculates a correction value in a three-dimensional coordinate axis of the right-eye camera or the left-eye camera based on the reference marker photographed by the stereo camera and the image patterns located on the right and left sides of the reference marker, respectively; 、 As the image pattern, an object that exists near the intersection of the vertical center line of each of the images captured by the right-eye camera and the left-eye camera and the horizontal line of the reference marker is set. It is characterized by:

[0008] The display system of the present invention also includes: a stereo camera having a right-eye camera and a left-eye camera and installed on the work machine; a remote image calibration unit that calculates a correction value in a three-dimensional coordinate axis of the right-eye camera or the left-eye camera based on an image of a reference marker captured by the stereo camera; a handheld camera that photographs an operating means for operating the work machine; an image synthesis unit that synthesizes an image captured by the stereo camera and an image captured by the handheld camera to generate an image for the right eye and an image for the left eye; a head-mounted display device that displays the right-eye image and the left-eye image generated by the image synthesis unit; and A display system for remotely operating the work machine, comprising: A stereo camera is used as the handheld camera, Furthermore, a handheld image calibration unit that uses the operation means photographed by the handheld camera as a reference marker and calculates a correction value in a three-dimensional coordinate axis of the right-eye camera or the left-eye camera of the handheld camera based on an image of the reference marker. The present invention is characterized by the following features.

[0009] Preferably, the remote image calibration unit and the handy image calibration unit adjust the magnification so that the image size of the reference marker is the same for the right-eye camera and the left-eye camera, and calculate a correction value in the depth direction in three-dimensional coordinates based on the magnification. Here, the correction value is intended to include the magnification itself.

[0010] The system further includes a switching unit that selects an image captured by one of the stereo cameras provided on each of the plurality of work machines from the captured images sent from the stereo cameras, and the image synthesis unit generates the image for the right eye and the image for the left eye using the captured image selected by the switching unit and displays them on the head-mounted display device, thereby making it possible to remotely operate the plurality of work machines.

[0011] In addition, by providing multiple head-mounted display devices and displaying the right-eye image and left-eye image generated by the image synthesis unit on each of the head-mounted display devices, it becomes possible for multiple operators to monitor the situation at the site.

[0012] A display method according to the present invention is a display method for remotely operating a work machine using a head-mounted display device and an operation means, comprising: acquiring images from a stereo camera having a right-eye camera and a left-eye camera; acquiring a photographed image of a reference marker placed in front of the stereo camera; specifying image patterns located on the right and left sides of the reference marker in the captured image; correcting images from the stereo camera using the fiducial markers and the image pattern to generate right-eye and left-eye images; taking a photograph of a handheld operating means; a step of combining the captured image of the operating means at hand with each of the right-eye image and the left-eye image; outputting the combined right-eye image and left-eye image to the head-mounted display device; Including fruit, In the step of specifying the image pattern, an object that exists near the intersection of the vertical center line of each of the images captured by the right-eye camera and the left-eye camera and the horizontal line of the reference marker is set as the image pattern. It is characterized by:

[0013] Further, a program according to the present invention is a program that runs on a display system for remotely operating a work machine, acquiring images from a stereo camera having a right-eye camera and a left-eye camera; acquiring a captured image of a fiducial marker placed in front of a stereo camera; calculating a correction value based on the reference marker in the captured image and image patterns located to the right and left of the reference marker; correcting the images from the stereo camera using the correction values ​​to generate a right-eye image and a left-eye image; acquiring a photographed image of the operating means at hand; a step of combining the captured image of the operation means with each of the right-eye image and the left-eye image; outputting and displaying the combined right-eye image and left-eye image on a head-mounted display device; of Including, In the step of calculating the correction value, an object that exists near an intersection point between a vertical center line of each of the images captured by the right-eye camera and the left-eye camera and a horizontal line of a reference marker is set as the image pattern. It is characterized by:

[0014] In particular, the correction value for the image captured by the stereo camera is calculated by executing the following steps.

[0015] The method includes the steps of capturing an image of a reference marker placed in front of a stereo camera having a right-eye camera and a left-eye camera, detecting an image pattern in the captured image at a position in front of each of the right-eye camera and the left-eye camera, acquiring a correction magnification by enlarging or reducing the image size so that the sizes of the reference markers captured by the right-eye camera and the left-eye camera are the same, searching for the image pattern when the size of the reference markers is the same and determining horizontal and vertical movement amounts based on the image pattern, and calculating horizontal and vertical correction values ​​for each of the right-eye camera and the left-eye camera based on the movement amounts and the parallax distances between the left and right cameras, and correcting the images captured by the stereo camera using the calculated correction values ​​to generate right-eye and left-eye images. [Effects of the Invention]

[0016] As described above, according to the present invention, it is possible to display the situation of a remote work site and the status of a local controller on a head-mounted display, regardless of the type of work machine or the controller that operates it. In addition, because calibration between the right-eye camera and the left-eye camera of a stereo camera attached to a work machine in a remote location can be performed accurately and efficiently, this system can be applied to the remote control of a variety of work machines. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a schematic explanatory diagram of an overall configuration according to an embodiment of the present invention; [Figure 2] 1 is a configuration diagram of a display system 1 according to an embodiment of the present invention. [Figure 3] 1 is a functional block diagram of a display system 1 that is communicatively connected via a communication line 2. FIG. [Figure 4] FIG. 4 is an explanatory diagram of the processing procedure (step 1) of the calibration unit in FIG. [Figure 5] FIG. 4 is an explanatory diagram of the processing procedure (step 2) of the calibration unit in FIG. 3. [Figure 6] FIG. 4 is an explanatory diagram of the processing procedure (step 3) of the calibration unit in FIG. [Figure 7] FIG. 4 is an explanatory diagram of the processing procedure (step 4) of the calibration unit in FIG. [Figure 8] FIG. 10 is an explanatory diagram of another embodiment 1 of the present invention. [Figure 9] FIG. 10 is an explanatory diagram of another embodiment 2 of the present invention. [Figure 10] FIG. 10 is an explanatory diagram of another embodiment 3 of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0018] A first embodiment of a display system according to the present invention will be described below with reference to the drawings. 1 is a schematic explanatory diagram of the overall configuration according to this embodiment. Note that this embodiment will be described using a backhoe as an example of a work machine.

[0019] A stereo camera 10 is attached near the driver's seat of a remotely operated backhoe 3. The attachment position is preferably near the driver's line of sight, but is not limited to this and may be any convenient attachment position.

[0020] The images captured by the stereo camera 10 are sent to a monitoring device 20 in the office via a communication line 2. Incidentally, the monitoring device 20 can be realized by a general-purpose computer (hereinafter referred to as a "PC").

[0021] An operator in the office remotely controls the backhoe 3 using a controller 4. This controller 4 is photographed by a handheld camera 5 that captures the area around the operator's hands, and the photographed image is imported into a PC 20. The PC 20 is also connected to a head-mounted display (hereinafter referred to as "HMD") 6 worn by the operator, and the images photographed by the stereo camera 10 and the images photographed by the handheld camera 5 are combined and displayed on the HMD 6.

[0022] 2 is a diagram showing the configuration of a display system 1 according to this embodiment. The display system 1 is made up of a distant stereo camera 10, a handheld camera 5 in the office, a PC 20, and an HMD 6.

[0023] The feature of this embodiment is the so-called monitoring function, which adjusts the magnification and position of the right-eye and left-eye images captured by the stereo camera 10 as shown in Fig. 2, and then combines the adjusted image with the image of the controller captured by the handheld camera 5 to display and output on the HMD 6. The so-called control function, which remotely controls the backhoe 3 using the controller 4, can use any general-purpose technology, so a description thereof will be omitted.

[0024] As shown in FIG. 2, an image 60a of the site captured by the stereo camera 10 and an image 60b of the controller 4 captured by the handheld camera 5 are displayed on a display screen 60 of the HMD 6 in a composite form.

[0025] Next, the configuration and operation of the display system 1 according to this embodiment will be described in detail with reference to FIG. FIG. 3 is a functional block diagram of a display system 1 that is communicatively connected via a communication line 2. As shown in FIG. The stereo camera 10 mounted on the backhoe 3 at a remote location consists of a right-eye camera 10a and a left-eye camera 10b. Note that the wide viewing angle of the stereo camera makes it possible to mount multiple stereo cameras 10 on the backhoe 3 to generate panoramic images. That is, the system may be configured by mounting a stereo camera with a fisheye lens and a wide viewing angle only on the front side of the backhoe 3, or, if stereo cameras with a narrower viewing angle are used, they may be mounted in three locations: in front of the driver's seat, on the right, and on the left, and the operator's line of sight may be detected and the appropriate stereo camera may be selected sequentially, or the images from each stereo camera may be stitched together to generate a panoramic image.

[0026] 3, a stereo camera consisting of a right-eye camera 5a and a left-eye camera 5b can be used as the handheld camera 5 that captures the controller 4. Note that a camera included in the HMD 6 can also be used as the handheld camera 5.

[0027] The images from stereo camera 10 and handheld camera 5 are each captured by PC 20. PC 20 includes a remote image receiving unit 21 that receives image data sent from stereo camera 10, a calibration unit 22 that adjusts the magnification between the right-eye image and the left-eye image and the display position of the image data, a handheld image receiving unit 23 that inputs image data captured by handheld camera 5, a calibration unit 24 that adjusts the right-eye image and the left-eye image when handheld camera 5 is a stereo camera, an image synthesis unit 30 that synthesizes the image from stereo camera 10 and the image from handheld camera 5 in real time as images for the right eye and the left eye, respectively, based on the adjusted image data, and an image recording unit 40 that stores the synthesized image.

[0028] When stereo cameras 10 are attached to three locations in the driver's seat of the backhoe 3, for the front, right, and left, the image synthesis unit 30 includes a front right-eye image generation unit 31a, a front left-eye image generation unit 31b, a right right-eye image generation unit 32a, a right left-eye image generation unit 32b, a left right-eye image generation unit 33a, and a left left-eye image generation unit 33b. That is, a right-eye image and a left-eye image are generated for each stereo camera 10. Note that, if the stereo camera 10 is attached only to one location, for example, for the front, the image generation units 32a, 32b, 33a, and 33b are unnecessary. The image synthesis unit 30 also includes a handheld right-eye image generation unit 34a and a handheld left-eye image generation unit 34b.

[0029] The image synthesis unit 30 further has a right-eye image synthesis unit 35a that synthesizes an image from the stereo camera 10 and an image from the handheld camera 5 for the right-eye image, and a left-eye image synthesis unit 35b that synthesizes an image from the stereo camera 10 and an image from the handheld camera 5 for the left-eye image.

[0030] The right-eye image generated by the right-eye image synthesis unit 35a is output and displayed on the right-eye display device of the HMD 10, and the left-eye image generated by the left-eye image synthesis unit 35b is output and displayed on the left-eye display device of the HMD 10. The images generated by the right-eye image synthesis unit 35a and the left-eye image synthesis unit 35b are stored in the video recording unit 40.

[0031] The processing procedure of the calibration unit 22, which is one of the features of the display system 1 of this embodiment, will be described below.

[0032] (Step 1) Setting fiducial markers (Figure 4) First, at the work site, as shown in FIG. 4(a), a reference marker 80 is placed at the center of the image captured by the stereo camera 10. This reference marker can be formed, for example, from a panel with a specific shape or design. The reference marker 80 is then captured by two cameras 10a and 10b, one on the left and one on the right, that make up the stereo camera 10 installed in the driver's seat. Furthermore, any object present near the intersection (Ml, Mr) of the center line extension (vertical direction) of the image captured by each camera and the horizontal line (horizontal direction) of the reference marker in the image captured by each camera, as shown in FIG. 4(b), is set as image patterns 81a and 81b. These image patterns 81a and 81b can be manually specified in the captured image, or computer software can automatically recognize shapes that satisfy pre-registered conditions (color, shape, etc.).

[0033] (Step 2) Adjust the image size (Figure 5) Normally, when a reference marker 80 is photographed using the camera positional relationship shown in FIG. 5(a), the size of the reference marker in the image photographed by the camera closer to the reference marker 80 will be larger, as shown in FIG. 5(b).

[0034] Therefore, the calibration unit 22 enlarges or reduces the image size so that the sizes of the reference markers 80 captured by the cameras 10a and 10b are the same. For example, the image captured by one camera is used as a reference and the magnification of the other camera is enlarged or reduced so that the size (diameter, vertical or horizontal ratio, etc.) of the reference marker 80 in the image falls within a predetermined error range. This enlargement ratio is used as the correction value in the Z direction (depth direction). In the example of Figure 5(c), the image captured by the right camera is enlarged so that the size of the reference marker 80 in the image captured by the right camera is the same as that of the left camera.

[0035] Note that the correction value in the Z direction (depth direction) may be given as distance by calculating the distance to the reference marker 80 from each magnification. Specifically, assuming that the ratio of the size of the reference marker when the magnification is changed is equal to the ratio of the distance, this can be determined from the size of the reference marker in the image at the standard magnification, the distance at that time, and the size of the reference marker in the captured image when the magnification is changed. The correction value for the distance at this time, i.e., the difference between the distance when the magnification is changed and the distance at the standard magnification, becomes the correction value in the depth direction (Z axis direction).

[0036] (Step 3) Image pattern search (Figure 6) Next, image patterns 81a and 81b are searched again under conditions where the size of the reference marker 80 is the same in the left and right camera images. The image pattern at this time does not need to be the same as the image pattern searched for in step 1. Here, the X axis is taken horizontally and the Y axis is taken vertically in the captured image, and the amount of movement (X, Y) is determined so that the reference marker 80 in the images captured by each of cameras 10a and 10b is located on the center line. For example, if the position of the center line (vertical direction) in the captured image is set to coordinate X = 0, the X coordinate of the reference marker 80 after movement will be zero. In addition, the Y coordinate of the reference marker 80 must also be the same between the right and left cameras. However, for example, this may be the average value of the Y coordinate of the reference marker 80 in both the images captured by the left and right cameras shown in FIG. 6(b), which are images before movement, or it may be the same as the position of the reference marker 80 in one of the captured images.

[0037] By the above procedure, the amount of movement (Xl, Yl) is calculated for the image captured by the left camera, and the amount of movement (Xr, Yr) is calculated for the image captured by the right camera. The reference marker and image pattern after the movement are shown in Figure 6(c).

[0038] (Step 4) Calculation of correction values ​​(Figure 7) Next, for the state of Fig. 7(b) (state of Fig. 6(c)), the distance between the left and right cameras 10a, 10b, i.e., half the parallax distance between the cameras, is added to or subtracted from the movement amount calculated in step 3 above to finally calculate the correction values ​​in the X-axis and Y-axis directions. As a result, the X, Y correction value for the right camera is (Xr - parallax distance / 2, Yr), and the X, Y correction value for the left camera is (Xl + parallax distance / 2, Yl).

[0039] That is, since the amount of movement includes an error as shown in Fig. 6(b), the net error for the X coordinate is calculated by subtracting half the value of the parallax distance (assumed to be 4 cm in this embodiment) from the amount of movement in step 3. Then, these errors in the X and Y directions are stored as correction values.

[0040] The calibration unit 22 stores the correction magnification and the correction values ​​for the X-axis and Y-axis directions determined by the processing of steps 1 to 4 above in a memory unit of the PC 20. The front right-eye image generation unit 31a and the front left-eye image generation unit 31b of the PC 20 use these correction values ​​to correct the video data sent from the stereo camera 10 and generate corrected right-eye and left-eye images. By using these correction values ​​to correct the images captured by the stereo camera 10, calibration can be performed so that the position of the reference marker can be displayed directly in front of the HMD 6 even if the stereo camera 10 is attached to the work machine at a slight angle.

[0041] For the handheld camera 5, if a stereo camera is used, the correction value can be calculated using the same process as above. In this case, the controller can be used as a reference marker. If a stereo camera is not used for the handheld camera 5, an image captured by a single camera can be shifted left and right based on the parallax distance of the stereo camera 10 installed at a distance to generate a right-eye image for the handheld and a left-eye image for the handheld. In this case, the calibration unit 24 is not required. The parallax distance of the handheld camera 5 can be stored and made adjustable according to the operator's request. In this way, the image of the controller can be displayed at a depth position that suits the operator.

[0042] The above has mainly described the calibration process for the case where the stereo camera 10 is attached to the front of the backhoe 3. When the stereo cameras 10 are attached to the right and left sides of the backhoe 3, the same calibration process as above is performed to generate corrected images for each. In this case, the image from which camera is to be displayed on the HMD 6 can be determined by an angle sensor built into the HMD 6.

[0043] According to this embodiment, regardless of the type of work machine or the controller used to operate it, the situation at a distant work site and the status of the controller at hand can be displayed on the HMD and remotely operated. In particular, the operator can see the operation of the work machine according to the operation content and the situation at the work site in approximately the same field of view without having to move their line of sight significantly, thereby reducing the burden and fatigue on the operator.

[0044] Furthermore, since calibration between the right-eye camera and left-eye camera of a stereo camera attached to a work machine in a remote location can be performed accurately and efficiently, this system can be applied to the remote control of various work machines, not just backhoes, such as bulldozers and crane trucks.

[0045] The above embodiment can be realized by various modifications as follows. (Another Example 1) As shown in Fig. 8, a stereo camera 10 is provided in the driver's seat of the work machine 3, and a 360-degree camera is also provided. In this case, on the display screen of the HMD 6, an image 60a captured by the stereo camera 10 is fitted into an image 60c cut from the 360-degree camera, and an image captured by the handheld camera 5 is also fitted. This eliminates the need to record all on-site images as stereo images, and the capacity of the image recording unit 40 can be reduced.

[0046] (Another Example 2) 9, a switching unit 70 is provided in the PC 20 to switch between stereo images sent from a plurality of remote locations. In this case, calibration processing is performed in advance for each remote location, and the correction values ​​are stored in a storage unit of the PC 20. Then, using the correction values ​​corresponding to the remote location to which the switch is made, a right-eye image and a left-eye image are generated from the received images and stored in the image recording unit 40, while also being displayed on the HMD 6. This allows for efficient remote control.

[0047] (Another Example 3) As shown in Fig. 10, video is output from the PC 20 to multiple HMDs 6, allowing multiple people to monitor. This allows people other than the operator who operates the controller 4 to monitor the operation of the work machine in a remote location while watching the operation status of the controller, enabling safer work.

[0048] As described above, according to this embodiment, it is possible to display the situation at a remote work site and the status of the controller at hand on a head-mounted display, regardless of the type of work machine or the controller that operates it. Also, because calibration between the right-eye camera and left-eye camera of a stereo camera attached to a work machine at a remote location can be performed accurately and efficiently, this system can be applied to the remote control of a variety of work machines. [Explanation of symbols]

[0049] 1 Display System 2. Communication lines 3 Backhoe (work machine) 4 Controller 5. Handheld camera 5a Right eye camera 5b Left eye camera 10 Stereo Camera 10a Right eye camera 10b Left eye camera 11. 360-degree camera 20 Monitoring equipment 21 Remote video receiving unit 22 Calibration section 23 Handheld video receiver 24 Calibration section 30 Image synthesis unit 31a Front right eye image generation unit 31b Front left eye image generation section 32a Right eye image generation unit for right 32b Right-eye left-eye image generation unit 33a Right eye image generation unit for left 33b Left eye image generation unit for left 34a Right-eye image generation unit for handheld use 34b Left eye image generator for handheld use 35a Right eye image synthesis unit 35b Left eye image synthesis unit 40 Video Recording Section 60 display screen 60a Stereo camera image 60b Handheld camera footage 70 Switching section 80 fiducial marker 81a Right camera image pattern 81b Image pattern for left camera

Claims

1. a stereo camera having a right-eye camera and a left-eye camera and installed on the work machine; a handheld camera that photographs an operating means for operating the work machine; an image synthesis unit that synthesizes an image captured by the stereo camera and an image captured by the handheld camera to generate an image for the right eye and an image for the left eye; a head-mounted display device that displays the right-eye image and the left-eye image generated by the image synthesis unit; and A display system for remotely operating a work machine, comprising: a calibration unit that calculates a correction value in a three-dimensional coordinate axis of the right-eye camera or the left-eye camera based on the reference marker photographed by the stereo camera and image patterns located to the right and left of the reference marker, A display system characterized in that the image pattern is set to an object that exists near the intersection of the vertical center line of each of the camera images captured by the right-eye camera and the left-eye camera and the horizontal line of the reference marker.

2. A display method for remotely operating a work machine using a head-mounted display device and an operation means, comprising: acquiring images from a stereo camera having a right-eye camera and a left-eye camera; acquiring a photographed image of a reference marker placed in front of the stereo camera; specifying image patterns located on the right and left sides of the reference marker in the captured image; correcting images from the stereo camera using the fiducial markers and the image pattern to generate right-eye and left-eye images; taking a photograph of a handheld operating means; a step of synthesizing a captured image of the operation means with each of the right-eye image and the left-eye image; outputting the combined right-eye image and left-eye image to the head-mounted display device; Including, A display method characterized in that, in the step of specifying the image pattern, an object that exists near the intersection of the vertical center line of each of the images captured by the right-eye camera and the left-eye camera and the horizontal line of a reference marker is set as the image pattern.

3. A program that operates on a display system for remotely operating a work machine, acquiring images from a stereo camera having a right-eye camera and a left-eye camera; acquiring a captured image of a reference marker placed in front of the stereo camera; calculating a correction value based on the reference marker in the captured image and image patterns located to the right and left of the reference marker; correcting the images from the stereo camera using the correction values ​​to generate a right-eye image and a left-eye image; acquiring a photographed image of the operating means at hand; a step of synthesizing a captured image of the operation means with each of the right-eye image and the left-eye image; outputting and displaying the combined right-eye image and left-eye image on a head-mounted display device; Including, A computer-executable program characterized in that, in the step of calculating the correction value, an object that exists near the intersection of the vertical center line of each of the camera images captured by the right-eye camera and the left-eye camera and the horizontal line of a reference marker is set as the image pattern.

Images