Display system, display method, and program for remote control

Abstract

To display on a head-mounted display a situation of a remote site and a state of an at-hand controller regardless of types of a work machine and a controller for operating the work machine.SOLUTION: A display system comprises: a stereoscopic camera composed of a right-eye camera and a left-eye camera and mounted on a work machine; an at-hand camera for imaging operation means for operating the work machine; an image composition unit for composing an image captured by the stereoscopic camera and an image captured by the at-hand camera and generating a right-eye image and a left-eye image; and a head-mounted display device for displaying the right-eye image and the left-eye image generated by the image composition unit. The display system further comprises a calibration unit for calculating a correction value at a three-dimensional coordinate axis of the right-eye camera or the left-eye camera on the basis of a reference marker imaged by the stereoscopic camara and image patterns respectively positioned on the right and left sides of the reference marker.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a display system for remotely operating a work machine on site, and in particular to a display system, display method, and program for displaying images from a stereo camera attached to the work machine as a three-dimensional image on a head-mounted display device worn by an operator. [Background technology]

[0002] Conventionally, there have been proposed techniques for remotely operating a work machine located at a distance. For example, Patent Document 1 proposes a system in which an imaging device is attached to a shovel, which is a work machine, and the captured image is displayed on a head-mounted display worn by the operator, so that the operator can see what he or she wants to see in a more intuitive way through an imaging device attached to the shovel to be remotely operated. In this system, when a wearable sensor detects a predetermined movement of the operator's head, the system switches between distant imaging devices or rotates a camera platform carrying the imaging device according to the predetermined movement, and displays the desired image on the head-mounted display.

[0003] Furthermore, Patent Document 2 proposes a system in which panoramic images for the right and left eyes, generated by multiple camera units mounted on a distant work machine, are cropped in accordance with the movement of the operator's line of sight, and the cropped images are transmitted to the operator's head-mounted display for display. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2015-226094 A [Patent Document 2] Patent No. 6612865 Summary of the Invention [Problem to be solved by the invention]

[0005] In both of the technologies described in Patent Documents 1 and 2, the operator operates a controller (remote control device) at hand to operate the work machine while watching the image of the work site displayed on a head-mounted display, but wearing the head-mounted display means that the operator cannot see the controller at hand. To address this issue, for example, a touch sensor can be provided on the controller to detect finger movements, and the position of the operator's fingers on the controller can be displayed on the head-mounted display, but different controllers are often used depending on the type of work machine, and it is costly to develop a display program for the controller image each time.

[0006] The present invention has been made in response to such conventional circumstances, and has an object to provide a display system, display method, and program for remote operation that can display the situation of a distant work site and the status of a controller at hand on a head-mounted display regardless of the type of work machine or the controller that operates it. [Means for solving the problem]

[0007] In order to achieve the above object, the display system of the present invention comprises: 、 A stereo camera having a right-eye camera and a left-eye camera and provided on the work machine; a handheld camera for photographing an operating means for operating the work machine; an image synthesis unit that synthesizes an image captured by the stereo camera and an image captured by the handheld camera to generate an image for the right eye and an image for the left eye; a head mounted display device that displays the right eye image and the left eye image generated by the image synthesis unit; and A display system for remote operation of a work machine comprising: a calibration unit that calculates a correction value in a three-dimensional coordinate axis of the right-eye camera or the left-eye camera based on the reference marker photographed by the stereo camera and image patterns located to the right and left of the reference marker, respectively; The present invention is characterized by comprising:

[0008] The display system of the present invention further comprises: a stereo camera having a right-eye camera and a left-eye camera and provided on the work machine; a remote image calibration unit that calculates a correction value in a three-dimensional coordinate axis of the right-eye camera or the left-eye camera based on an image of a reference marker captured by the stereo camera; a handheld camera for photographing an operating means for operating the work machine; an image synthesis unit that synthesizes an image captured by the stereo camera and an image captured by the handheld camera to generate an image for the right eye and an image for the left eye; a head mounted display device that displays the right eye image and the left eye image generated by the image synthesis unit; and A display system for remotely operating the work machine, comprising: A stereo camera is used as the handheld camera, and a handheld image calibration unit that uses the operation means photographed by the handheld camera as a reference marker and calculates a correction value in a three-dimensional coordinate axis of the right-eye camera or the left-eye camera of the handheld camera based on an image of the reference marker. The present invention is characterized by comprising:

[0009] It is preferable that the remote image calibration unit and the hand image calibration unit adjust the magnification so that the image size of the reference marker is the same for the right-eye camera and the left-eye camera, and calculate a correction value in the depth direction in three-dimensional coordinates based on the magnification. Here, the correction value is intended to include the magnification itself.

[0010] The system further includes a switching unit that selects an image captured by one of the stereo cameras provided on each of the multiple work machines from images captured by the stereo cameras. The image synthesis unit generates an image for the right eye and an image for the left eye using the captured image selected by the switching unit and displays the images on the head-mounted display device, thereby making it possible to remotely operate the multiple work machines.

[0011] In addition, by providing multiple head-mounted display devices and displaying the right eye image and left eye image generated by the image synthesis unit on each of the head-mounted display devices, it becomes possible for multiple operators to monitor the situation at the site.

[0012] Further, a display method according to the present invention is a display method for remotely operating a work machine using a head-mounted display device and an operation means, comprising: acquiring an image from a stereo camera having a right-eye camera and a left-eye camera; acquiring a captured image of a fiducial marker placed in front of the stereo camera; specifying image patterns located to the right and left of the reference marker in the captured image; calibrating images from the stereo camera using the fiducial markers and the image pattern to generate right-eye and left-eye images; Taking an image of a control means at hand; a step of synthesizing the captured image of the operating means at hand with each of the right eye image and the left eye image; outputting the combined right-eye image and left-eye image on the head-mounted display device; The present invention is characterized by comprising:

[0013] Further, a program according to the present invention is a program that operates on a display system for remotely operating a work machine, Acquiring an image from a stereo camera having a right-eye camera and a left-eye camera; acquiring a captured image of a fiducial marker placed in front of a stereo camera; calculating a correction value based on a reference marker in the captured image and image patterns located to the right and left of the reference marker; correcting the image from the stereo camera using the correction value to generate a right-eye image and a left-eye image; A step of acquiring a photographed image of a control means at hand; a step of synthesizing the captured image of the operation means with each of the right eye image and the left eye image; outputting and displaying the combined right-eye image and left-eye image on a head-mounted display device; The present invention is characterized in that the above-mentioned is executed by a computer.

[0014] In particular, the correction value for the image captured by the stereo camera is calculated by executing the following steps.

[0015] The method includes the steps of: photographing a reference marker placed in front of a stereo camera having a right-eye camera and a left-eye camera; detecting an image pattern in front of each of the right-eye camera and the left-eye camera in the photographed image; acquiring a correction magnification by enlarging or reducing the image size so that the sizes of the reference markers photographed by the right-eye camera and the left-eye camera are the same; searching for the image pattern in a situation where the size of the reference marker is the same and determining horizontal and vertical movement amounts based on the image pattern; calculating horizontal and vertical correction values ​​for each of the right-eye camera and the left-eye camera based on the movement amounts and the parallax distances between the left and right cameras; and correcting the images photographed by the stereo camera using the calculated correction values ​​to generate right-eye and left-eye images. Effect of the Invention

[0016] As described above, according to the present invention, it is possible to display the situation of a distant work site and the state of a local controller on a head mounted display regardless of the type of work machine or the controller that operates it. Also, since calibration between the right eye camera and the left eye camera of a stereo camera attached to a work machine in a remote location can be performed accurately and efficiently, this system can be applied to the remote control of various work machines. [Brief description of the drawings]

[0017] [Figure 1] 1 is a schematic explanatory diagram of an overall configuration according to an embodiment of the present invention; [Diagram 2] 1 is a configuration diagram of a display system 1 according to an embodiment of the present invention. [Diagram 3] 1 is a functional block diagram of a display system 1 that is communicatively connected via a communication line 2. FIG. [Figure 4] FIG. 4 is an explanatory diagram of a processing procedure (step 1) of the calibration unit in FIG. [Diagram 5] FIG. 4 is an explanatory diagram of a processing procedure (step 2) of the calibration unit in FIG. [Figure 6] FIG. 4 is an explanatory diagram of a processing procedure (step 3) of the calibration unit in FIG. [Figure 7] FIG. 4 is an explanatory diagram of a processing procedure (step 4) of the calibration unit in FIG. [Figure 8] FIG. 4 is an explanatory diagram of another embodiment 1 of the present invention. [Figure 9] FIG. 11 is an explanatory diagram of another embodiment 2 of the present invention. [Figure 10] FIG. 11 is an explanatory diagram of another embodiment 3 of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] A first embodiment of a display system according to the present invention will be described below with reference to the drawings. 1 is a schematic explanatory diagram of the overall configuration according to this embodiment. Note that this embodiment will be described taking a backhoe as an example of a work machine.

[0019] A stereo camera 10 is attached near the driver's seat of a remotely operated backhoe 3. The attachment position is preferably near the driver's line of sight, but is not limited to this and may be any convenient position.

[0020] The images captured by the stereo camera 10 are sent to a monitoring device 20 in the office via a communication line 2. The monitoring device 20 can be realized by a general-purpose computer (hereinafter referred to as a "PC").

[0021] An operator in the office remotely controls the backhoe 3 using a controller 4. This controller 4 is photographed by a handheld camera 5 that captures the area around the operator's hands, and the photographed image is imported into a PC 20. The PC 20 is also connected to a head mounted display (hereinafter referred to as "HMD") 6 worn by the operator, and the image photographed by the stereo camera 10 and the image photographed by the handheld camera 5 are synthesized and displayed on the HMD 6.

[0022] 2 is a configuration diagram of a display system 1 according to this embodiment. The display system 1 is made up of a distant stereo camera 10, a handheld camera 5 in the office, a PC 20, and an HMD 6.

[0023] The feature of this embodiment is a so-called monitoring function that adjusts the magnification and position of the right-eye image and left-eye image captured by the stereo camera 10 as shown in Fig. 2, synthesizes the adjusted image with the image of the controller captured by the handheld camera 5, and displays it on the HMD 6. The so-called control function that remotely controls the backhoe 3 by the controller 4 can use any general-purpose technology, so a description thereof will be omitted.

[0024] As shown in FIG. 2, an image 60a of the site captured by the stereo camera 10 and an image 60b of the controller 4 captured by the handheld camera 5 are composited and displayed on a display screen 60 of the HMD 6.

[0025] Next, the configuration and operation of the display system 1 according to the present embodiment will be described in detail with reference to FIG. FIG. 3 is a functional block diagram of a display system 1 that is communicatively connected via a communication line 2. As shown in FIG. The stereo camera 10 mounted on the backhoe 3 at the remote location is composed of a right-eye camera 10a and a left-eye camera 10b. Note that the wide viewing angle of the stereo camera allows multiple stereo cameras 10 to be mounted on the backhoe 3 to generate panoramic images. That is, a stereo camera equipped with a fisheye lens and having a wide viewing angle may be mounted only on the front side of the backhoe 3 to configure the system, or when using stereo cameras with a narrower viewing angle, they may be mounted in three locations on the front, right, and left sides of the driver's seat, and the operator's line of sight may be detected and the appropriate stereo camera may be selected in sequence, or the images from each stereo camera may be joined together to generate a panoramic image.

[0026] 3, a stereo camera consisting of a right-eye camera 5a and a left-eye camera 5b can be used as the handheld camera 5 that captures the controller 4. Note that a camera included in the HMD 6 can also be used as the handheld camera 5.

[0027] The images from the stereo camera 10 and the handheld camera 5 are respectively captured by the PC 20. The PC 20 includes a remote image receiving unit 21 that receives image data sent from the stereo camera 10, a calibration unit 22 that adjusts the magnification between the right-eye image and the left-eye image and the display position in the image data, a handheld image receiving unit 23 that inputs image data captured by the handheld camera 5, a calibration unit 24 that adjusts between the right-eye image and the left-eye image when the handheld camera 5 is a stereo camera, an image synthesis unit 30 that synthesizes the image from the stereo camera 10 and the image from the handheld camera 5 as right-eye and left-eye images in real time based on the adjusted image data, and an image recording unit 40 that stores the synthesized image.

[0028] In the case where the stereo cameras 10 are attached to three locations, for the front, right, and left, in the driver's seat of the backhoe 3, the image synthesis unit 30 includes a front right-eye image generating unit 31a, a front left-eye image generating unit 31b, a right-eye image generating unit 32a, a right left-eye image generating unit 32b, and a left right-eye image generating unit 33a and a left left-eye image generating unit 33b. That is, a right-eye image and a left-eye image are generated for each stereo camera 10. Note that, when the stereo camera 10 is attached only to one location, for example, for the front, the image generating units 32a, 32b, 33a, and 33b are not necessary. The image synthesis unit 30 also includes a handy right-eye image generating unit 34a and a handy left-eye image generating unit 34b.

[0029] The image synthesis unit 30 further has a right-eye image synthesis unit 35a that synthesizes an image from the stereo camera 10 and an image from the handheld camera 5 for a right-eye image, and a left-eye image synthesis unit 35b that synthesizes an image from the stereo camera 10 and an image from the handheld camera 5 for a left-eye image.

[0030] The right-eye image generated by the right-eye image synthesis unit 35a is output and displayed on the right-eye display device of the HMD 10, and the left-eye image generated by the left-eye image synthesis unit 35b is output and displayed on the left-eye display device of the HMD 10. The images generated by the right-eye image synthesis unit 35a and the left-eye image synthesis unit 35b are stored in the video recording unit 40.

[0031] The following describes the processing procedure of the calibration unit 22, which is one of the features of the display system 1 of this embodiment.

[0032] (Step 1) Setting fiducial markers (Figure 4) First, at the work site, as shown in FIG. 4(a), a reference marker 80 is placed at a position to be the center of the screen of the stereo camera 10. For example, this reference marker can be formed of a panel with a specific shape or design. Then, the reference marker 80 is photographed by two cameras 10a and 10b, one on the left and one on the right, which constitute the stereo camera 10 provided in the driver's seat. In addition, any object existing near the intersection (Ml, Mr) of the extension of the center line (vertical direction) of each camera image and the horizontal line (horizontal direction) of the reference marker in the image captured by each camera shown in FIG. 4(b) is set as image patterns 81a and 81b. These image patterns 81a and 81b can be specified by a human body in the captured image, but a form that satisfies a condition (color, shape, etc.) registered in advance by computer software may be automatically recognized.

[0033] (Step 2) Adjust the image size (Figure 5) Normally, when a reference marker 80 is photographed with the camera positions shown in FIG. 5(a), the size of the reference marker in the image captured by the camera closer to the reference marker 80 will be larger, as shown in FIG. 5(b).

[0034] Therefore, the calibration unit 22 enlarges or reduces the image size so that the sizes of the reference marker 80 captured by the cameras 10a and 10b are the same. For example, the image of one camera is used as a reference and the magnification of the other camera is enlarged or reduced so that the size (diameter, vertical or horizontal ratio, etc.) of the reference marker 80 in the image falls within a predetermined error range. This enlargement ratio is used as the correction value in the Z direction (depth direction). In the example of Fig. 5(c), the image captured by the right camera is enlarged so that the size of the reference marker 80 in the image captured by the right camera is the same as that of the left camera.

[0035] It is also possible to provide a correction value in the Z direction (depth direction) as distance by calculating the distance to the reference marker 80 from each magnification. Specifically, assuming that the ratio of the size of the reference marker when the magnification is changed is equal to the ratio of the distance, the correction value can be obtained from the size of the reference marker in the image at the standard magnification, the distance at that time, and the size of the reference marker in the captured image at the time of the magnification change. The correction value of the distance at this time, i.e., the difference between the distance when the magnification is changed and the distance when the magnification is the standard magnification, becomes the correction value in the depth direction (Z axis direction).

[0036] (Step 3) Searching for image patterns (Figure 6) Next, image patterns 81a and 81b are searched again in a state where the size of the reference marker 80 is the same in the left and right camera images. The image pattern at this time does not need to be the same as the image pattern searched in step 1. Here, in the captured image, the X axis is taken in the horizontal direction and the Y axis is taken in the vertical direction, and the movement amount (X, Y) is determined so that the reference marker 80 in the captured image of each camera 10a and 10b is located on the center line. For example, if the position of the center line (vertical direction) in the captured image is coordinate X=0, the X coordinate of the reference marker 80 after the movement becomes zero. In addition, the Y coordinate of the reference marker 80 needs to be unified between the right camera and the left camera. For example, it may be the average value of the Y coordinate of the reference marker 80 in both the captured images of the left camera and the right camera shown in FIG. 6(b), which is the image before the movement, or it may be unified to the position of the reference marker 80 in one of the captured images.

[0037] By the above procedure, the amount of movement (Xl, Yl) is calculated for the image captured by the left camera, and the amount of movement (Xr, Yr) is calculated for the image captured by the right camera. The reference marker and image pattern after the movement are shown in Figure 6(c).

[0038] (Step 4) Calculating the correction value (Figure 7) Next, for the state of Fig. 7(b) (state of Fig. 6(c)), the distance between the left and right cameras 10a, 10b, i.e., half the parallax distance of the cameras, is added to or subtracted from the movement amount calculated in step 3 above to finally calculate the correction values ​​in the X-axis and Y-axis directions. As a result, the X,Y correction value of the right camera is (Xr-parallax distance / 2, Yr), and the X,Y correction value of the left camera is (Xl+parallax distance / 2, Yl).

[0039] That is, since the amount of movement includes an error as shown in Fig. 6(b), the net error for the X coordinate is calculated by subtracting half the parallax distance (assumed to be 4 cm in this embodiment) from the amount of movement in step 3. Then, these errors in the X and Y directions are stored as correction values.

[0040] The calibration unit 22 stores the correction magnification and the correction values ​​for the X-axis and Y-axis directions determined by the processing of steps 1 to 4 above in a memory unit of the PC 20. The front right-eye image generating unit 31a and the front left-eye image generating unit 31b of the PC 20 use these correction values ​​to correct the video data sent from the stereo camera 10 and generate corrected right-eye and left-eye images. By using these correction values ​​to correct the images captured by the stereo camera 10, it is possible to calibrate the HMD 6 so that the position of the reference marker can be displayed in front even if the stereo camera 10 is attached to the work machine with a slight tilt.

[0041] For the handheld camera 5, when a stereo camera is used, the correction value can be calculated by the same process as above. In this case, the controller can be used as a reference marker. If a stereo camera is not used for the handheld camera 5, an image taken by a single camera may be shifted left and right based on the parallax distance of the stereo camera 10 installed at a distance to generate a right-eye image for the hand and a left-eye image for the hand. In this case, the calibration unit 24 is not required. The parallax distance of the handheld camera 5 may be stored and the parallax distance may be adjustable according to the operator's request. In this way, the image of the controller can be displayed at a depth position that suits the operator.

[0042] The above has mainly described the calibration process for the case where the stereo camera 10 is attached to the front of the backhoe 3. When the stereo cameras 10 are attached to the right and left sides of the backhoe 3, the same calibration process as above is performed to generate corrected images. In this case, the image from which camera is to be displayed on the HMD 6 can be determined by an angle sensor built into the HMD 6.

[0043] According to this embodiment, regardless of the type of work machine or the controller that operates it, it is possible to display the situation of a distant work site and the state of the controller at hand on the HMD and perform remote operation. In particular, since the operation of the work machine according to the operation content and the situation of the work site can be recognized in approximately the same field of view without moving the line of sight significantly, the burden and fatigue of the operator can be reduced.

[0044] In addition, since calibration between the right-eye camera and left-eye camera of a stereo camera attached to a work machine in a remote location can be performed accurately and efficiently, this system can be applied to the remote control of various work machines, not limited to backhoes, such as bulldozers and cranes.

[0045] The above embodiment can be realized in various modifications as follows. (Another Example 1) 8, a stereo camera 10 is provided in the driver's seat of the work machine 3, and a 360-degree camera is also provided. In this case, on the display screen of the HMD 6, an image 60a captured by the stereo camera 10 is fitted into an image 60c cut from the 360-degree camera, and an image captured by the handheld camera 5 is also fitted. This eliminates the need to record all on-site images as stereo images, making it possible to reduce the capacity of the image recording unit 40.

[0046] (Another Example 2) 9, a switching unit 70 is provided in the PC 20 to switch between stereo images sent from a plurality of remote locations. In this case, a calibration process is performed in advance for each remote location, and the correction values ​​are stored in a storage unit of the PC 20. Then, using the correction value corresponding to the remote location to which the image has been switched, a right-eye image and a left-eye image are generated from the received image and stored in the image recording unit 40, while also being displayed on the HMD 6. This allows efficient remote control.

[0047] (Other Example 3) 10, video is output from the PC 20 to multiple HMDs 6 for monitoring by multiple people. This allows people other than the operator who operates the controller 4 to monitor the operation of a remote work machine while watching the operation status of the controller, enabling safer work.

[0048] As described above, according to this embodiment, it is possible to display the situation of a distant work site and the state of a local controller on a head mounted display, regardless of the type of work machine or the controller that operates it. Also, since calibration between the right eye camera and the left eye camera of a stereo camera attached to a work machine in a remote location can be performed accurately and efficiently, this system can be applied to the remote control of various work machines. [Explanation of symbols]

[0049] 1 Display System 2. Communication lines 3 Backhoe (work machine) 4. Controller 5. Handheld Camera 5a Right eye camera 5b Left eye camera 10 Stereo Camera 10a Right eye camera 10b Left eye camera 11. 360-degree camera 20 Monitoring equipment 21 Remote video receiving unit 22 Calibration section 23 Handheld video receiver 24 Calibration section 30 Image synthesis unit 31a Front right eye image generation unit 31b Front left eye image generation section 32a Right eye image generation unit for right 32b Right-eye left-eye image generator 33a Right eye image generation unit for left 33b Left eye image generation unit for left 34a Right-eye image generator for handheld 34b Left eye image generator for handheld 35a Right eye image synthesis unit 35b Left eye image synthesis unit 40 Video Recording Section 60 display screen 60a Stereo camera image 60b Handheld camera footage 70 Switching section 80 Reference Marker 81a Image pattern for right camera 81b Image pattern for left camera

Claims

1. A stereo camera having a right-eye camera and a left-eye camera and provided on the work machine; a handheld camera for photographing an operating means for operating the work machine; an image synthesis unit that synthesizes an image captured by the stereo camera and an image captured by the handheld camera to generate an image for the right eye and an image for the left eye; a head mounted display device that displays the right eye image and the left eye image generated by the image synthesis unit; and A display system for remote operation of a work machine comprising: A display system comprising a calibration unit that calculates a correction value in a three-dimensional coordinate axis of the right-eye camera or the left-eye camera based on a reference marker photographed by the stereo camera and image patterns located to the right and left of the reference marker, respectively.

2. The display system according to claim 1, characterized in that an object is set as the image pattern, the object being present near the intersection of the vertical center lines of the images captured by the right-eye camera and the left-eye camera and the horizontal line of the reference marker.

3. A display method for remotely operating a work machine using a head-mounted display device and an operation means, comprising: acquiring an image from a stereo camera having a right-eye camera and a left-eye camera; acquiring a captured image of a fiducial marker placed in front of the stereo camera; specifying image patterns located to the right and left of the reference marker in the captured image; calibrating images from the stereo camera using the fiducial markers and the image pattern to generate right-eye and left-eye images; Taking an image of a control means at hand; a step of synthesizing a captured image of the operation means with each of the right eye image and the left eye image; outputting the combined right-eye image and left-eye image on the head-mounted display device; A display method comprising:

4. A program that operates on a display system for remote operation of a work machine, Acquiring an image from a stereo camera having a right-eye camera and a left-eye camera; acquiring a captured image of a reference marker placed in front of the stereo camera; calculating a correction value based on a reference marker in the captured image and image patterns located to the right and left of the reference marker; correcting the image from the stereo camera using the correction value to generate a right-eye image and a left-eye image; A step of acquiring a photographed image of a control means at hand; a step of synthesizing a captured image of the operation means with each of the right eye image and the left eye image; outputting and displaying the combined right-eye image and left-eye image on a head-mounted display device; 4. A computer executable program comprising:

Images