Image display device and image display method

Abstract

To provide an image display device configured to reduce telexistence sickness caused by shifting between a direction of an operator and a direction of a camera mounted on a robot.SOLUTION: A synchronization controller 15 performs control so that operator motion data is transmitted to a robot when a synchronization switch 16 is set to ON to synchronize a rotational motion of an operator with a rotational motion of the robot. An image correction unit 18, when there is a difference between a first rotation angle of the operator and a second rotation angle of the robot, corrects a cutout range of an image so as to correspond to the first rotation angle. The synchronization controller 15 performs control so that the operator motion data is not transmitted to the robot when the synchronization switch 16 is set to OFF where a rotational motion of the operator is not synchronized with a rotational motion of the robot. The image correction unit 18 corrects the cutout range so as to correspond to the first rotation angle.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image display device and an image display method. [Background technology]

[0002] As described in Patent Document 1, a remote control system has been put into practical use in which a user wearing an image display device such as a head-mounted display controls a robot (a remotely controlled device) located in a remote location. The technology used in this type of remote control system for controlling a remotely located robot in real time is called telexistence. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6801136 Summary of the Invention [Problem to be solved by the invention]

[0004] When an operator wearing an image display device rotates their head, for example, 30 degrees, the robot also responds by rotating the movable part 30 degrees. However, in reality, depending on the precision of the driving parts, such as the motors that drive the movable part, it may not be possible to rotate the movable part exactly 30 degrees, resulting in a deviation in the rotation angle. This can cause telexistence sickness due to a discrepancy between the viewing angle perceived by the operator in response to the rotation angle and the viewing angle at which the camera mounted on the robot actually captures the robot's surroundings.

[0005] The present invention aims to provide an image display device and an image display method that can reduce telexistence sickness caused by a misalignment between the orientation of an operator wearing the image display device and the orientation of a camera mounted on a robot. [Means for solving the problem]

[0006] The present invention includes an operator motion data acquisition unit that acquires operator motion data indicating a first rotation angle when an operator operating a robot performs a rotational motion; a robot motion data acquisition unit that acquires robot motion data transmitted from the robot and indicating a second rotation angle when the robot performs a rotational motion in accordance with the operator motion data; an image data acquisition unit that acquires image data transmitted from the robot and generated by shooting with a 360-degree camera provided on the robot; an image correction unit that supplies image data of a cut-out range obtained by cutting out a range of an image to be displayed on a display panel viewed by the operator from the image data acquired by the image data acquisition unit to the display panel and corrects the cut-out range; an interlocking switch that sets whether or not to interlock the rotational motion of the operator and the rotational motion of the robot; and when the interlocking switch is set to on, which interlocks the rotational motion of the operator and the rotational motion of the robot, controls the operator motion data to be transmitted to the robot, and if there is a difference between the first rotation angle and the second rotation angle, corrects the cut-out range. , so as to shift the rotation angle by an angle difference between the first rotation angle and the second rotation angle. and a linkage control unit that, when the linkage switch is set to off so that the rotational movement of the operator and the rotational movement of the robot are not linked, controls the image correction unit so that the operator movement data is not transmitted to the robot, and controls the image correction unit to correct the cut-out range to be within a range corresponding to the first rotation angle.

[0007] The present invention acquires operator motion data indicating a first rotation angle when an operator operating a robot makes a rotational motion, acquires robot motion data transmitted from the robot indicating a second rotation angle when the robot makes a rotational motion in accordance with the operator motion data, acquires image data generated by photographing with a 360-degree camera provided on the robot and transmitted from the robot, supplies image data of a cut-out range obtained by cutting out a range of an image to be displayed on a display panel viewed by the operator from the acquired image data to the display panel, and controls the transmission of the operator motion data to the robot when the rotational motion of the operator and the rotational motion of the robot are set to be linked, and if there is a difference between the first rotation angle and the second rotation angle, controls the cut-out range to be displayed on a display panel viewed by the operator. , so as to shift the rotation angle by an angle difference between the first rotation angle and the second rotation angle. and when the setting is off, which does not link the rotational movement of the operator with the rotational movement of the robot, the operator movement data is controlled not to be transmitted to the robot, and the cut-out range is corrected to be a range corresponding to the first rotation angle. [Effects of the Invention]

[0008] The image display device and image display method of the present invention can reduce telexistence sickness caused by a misalignment between the orientation of an operator wearing the image display device and the orientation of a camera mounted on a robot. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a remote control system including an image display device and a remotely controlled device connected to each other via a network. [Figure 2] 1 is a block diagram illustrating an image display device according to an embodiment. [Figure 3] FIG. 1 is a block diagram showing a configuration example of a robot that is an example of a remotely controlled device. [Figure 4] 1 is a flowchart illustrating a general process executed by an image display device according to an embodiment. [Figure 5]5 is a flowchart showing a specific process of step S2 in FIG. 4. [Figure 6] 5 is a flowchart showing a specific process of step S3 in FIG. 4. [Figure 7A] FIG. 6 is a diagram showing a state in which the image cutout range is not corrected in the flowchart shown in FIG. 5. [Figure 7B] FIG. 6 is a diagram showing a state in which the image cutout range is corrected so as to be shifted in the direction of the operator's rotational movement in the flowchart shown in FIG. 5. [Figure 7C] FIG. 6 is a diagram showing a state in which the image cutout range is corrected so as to be shifted in the direction opposite to the direction of rotational movement of the operator in the flowchart shown in FIG. 5. [Figure 8] 10A and 10B are diagrams illustrating an example in which a range of an image having a viewing angle wider than the angle of view of a display panel is set as the image cropping range. [Figure 9] 9 is a diagram showing an example in which the size of the cropped area shown in FIG. 8 is compressed so that it fits within the angle of view of the display panel. DETAILED DESCRIPTION OF THE INVENTION

[0010] An image display device and an image display method according to an embodiment will be described below with reference to the accompanying drawings. First, a configuration example of a remote control system including an image display device and a remotely controlled device will be described with reference to FIG.

[0011] 1, an image display device 10 and a robot 30 are connected to each other via a network 20. The network 20 is typically the Internet. The image display device 10 and the robot 30 communicate with each other via communication units 11 and 31 provided in each device. The robot 30 is located in a remote location away from where the image display device 10 is located.

[0012] The image display device 10 is, for example, a head-mounted display that is worn on the head of the operator 100 so as to cover the eyes. The robot 30 is an example of a remotely controlled device whose movement is controlled in accordance with operator motion data indicating the movement of the image display device 10 or the operator 100 wearing the image display device 10. The robot 30 is equipped with a right-eye camera 36R and a left-eye camera 36L, which are 360-degree cameras equipped with fisheye lenses. The right-eye camera 36R and the left-eye camera 36L are only required to generate a panoramic image in at least the horizontal direction. The right-eye camera 36R and the left-eye camera 36L may also generate a panoramic image in both the horizontal and vertical directions (a celestial sphere image).

[0013] Fig. 2 shows a specific example of the configuration of the image display device 10. As shown in Fig. 2, in addition to a communication unit 11, the image display device 10 includes a motion sensor 12, an operator motion data acquisition unit 13, a robot motion data acquisition unit 14, an interlocking control unit 15, an interlocking switch 16, an image data acquisition unit 17, an image correction unit 18, and a display panel 19.

[0014] The motion sensor 12 includes at least a three-dimensional gyro sensor and detects the angular velocity of the image display device 10 (i.e., the head of the operator 100) when it makes a rotational motion. The operator motion data acquisition unit 13 acquires and temporarily stores operator motion data indicating the rotation angle of the image display device 10 based on the detected value of the angular velocity of the motion sensor 12. The communication unit 11 may transmit the operator motion data to the robot 30 via the network 20. When the communication unit 11 transmits the operator motion data to the robot 30, the movable part 34 (see FIG. 3) of the robot 30 makes a rotational motion according to the operator motion data.

[0015] The communication unit 11 receives robot motion data indicating the rotation angle when the movable part 34 of the robot 30 rotates in accordance with the operator motion data. The robot motion data acquisition unit 14 acquires the robot motion data received by the communication unit 11 and temporarily stores it.

[0016] The interlocking switch 16 is selectively set to on or off by the operator 100. When the interlocking switch 16 is set to on, the interlocking control unit 15 controls the operator motion data acquiring unit 13 to supply the operator motion data to the communication unit 11 in order to interlock the movement of the image display device 10 with the movement of the robot 30. When the interlocking switch 16 is set to off, the interlocking control unit 15 controls the operator motion data acquiring unit 13 not to supply the operator motion data to the communication unit 11 in order not to interlock the movement of the image display device 10 with the movement of the robot 30.

[0017] The communication unit 11 receives image data, which is a 360-degree panoramic image captured by the camera 36 (see FIG. 3) of the robot 30. The image data acquisition unit 17 acquires and temporarily stores the image data received by the communication unit 11. The camera 36 includes a right-eye camera 36R and a left-eye camera 36L, and the image data includes image data for the right eye and image data for the left eye.

[0018] The image correction unit 18 supplies image data of a cut-out range, which is obtained by cutting out the range of the image to be displayed on the display panel 19 viewed by the operator 100 from the image data acquired by the image data acquisition unit 17, to the display panel 19. At this time, the image correction unit 18 may correct the cut-out range. Because a panoramic image has a large amount of distortion, it is preferable that the image correction unit 18 corrects the image of the cut-out range so as to reduce distortion. The interlock control unit 15 corrects the cut-out range in a different way depending on whether the interlock switch 16 is set to on or off.

[0019] The display panel 19 displays an image based on the image data for the right eye of the cut-out range and an image based on the image data for the left eye of the cut-out range. The display panel 19 may be a liquid crystal panel or an organic EL panel.

[0020] Fig. 3 shows a specific example of the configuration of the robot 30. As shown in Fig. 3, the robot 30 includes a control unit 32, a drive unit 33, a movable unit 34, a motion sensor 35, and a video signal processing unit 37, in addition to a communication unit 31 and a camera 36. The motion sensor 35 and the camera 36 are provided on the movable unit 34, or are provided so as to move integrally with the movable unit 34.

[0021] The motion sensor 35 includes at least a three-dimensional gyro sensor and detects the angular velocity of the movable part 34 when it makes a rotational motion. The control unit 32 generates robot motion data indicating the rotation angle of the movable part 34 based on the angular velocity detected by the motion sensor 35, and supplies the robot motion data to the communication unit 31. The communication unit 31 transmits the robot motion data to the image display device 10 via the network 20. The control unit 32 can be configured with a microcomputer or a microprocessor.

[0022] The captured image signal output from the camera 36 is supplied to the video signal processing unit 37. The video signal processing unit 37 compresses and encodes the captured image signal, and supplies the compressed and encoded image data to the control unit 32. The video signal processing unit 37 may perform other video signal processing besides compression and encoding on the captured image signal. The control unit 32 supplies the image data to the communication unit 31. The communication unit 31 transmits the image data to the image display device 10 via the network 20.

[0023] Using the flowcharts shown in Figures 4 to 6, we will explain how the interlocking control unit 15 controls the correction of the cut-out range in the image correction unit 18 when the interlocking switch 16 is set to on and when it is set to off.

[0024] 4, in step S1, interlock control unit 15 determines whether interlock switch 16 is set to on. If interlock switch 16 is set to on (YES), in step S2, interlock control unit 15 and image correction unit 18 perform image correction when interlock switch 16 is on to correct the cut-out range. If interlock switch 16 is not set to on (NO), in step S3, interlock control unit 15 and image correction unit 18 perform image correction when interlock switch 16 is off to correct the cut-out range.

[0025] Following step S2 or S3, the process proceeds to step S4. In step S4, the image display device 10 determines whether or not the operation of the robot 30 is to be ended by a predetermined operation by the operator 100. If the operation of the robot 30 is not to be ended (NO), the image display device 10 repeats the processes of steps S1 to S4. If the operation of the robot 30 is to be ended (YES), the image display device 10 ends the process.

[0026] Fig. 5 shows a specific process of step S2 in Fig. 4, and Fig. 6 shows a specific process of step S3 in Fig. 4. First, a specific process of step S2 will be described. In Fig. 5, interlocking control unit 15 acquires operator motion data held in operator motion data acquisition unit 13 in step S21, and acquires robot motion data held in robot motion data acquisition unit 14 in step S22. Image correction unit 18 acquires image data held in image data acquisition unit 17 in step S23. Steps S21 to S23 may be performed in any order, and may be performed simultaneously.

[0027] As shown in FIG. 1, it is assumed that the rotation angle of the image display device 10 is θ1 (first rotation angle) and the rotation angle of the robot 30 is θ2 (second rotation angle). In step S24, the interlocking control unit 15 calculates an angle Δθ by subtracting the rotation angle θ2 of the robot 30 from the rotation angle θ1 of the image display device 10. The angle Δθ indicates the difference in angle between the rotation angle θ1 and the rotation angle θ2. In step S25, the interlocking control unit 15 determines whether the angle Δθ is 0. If the angle Δθ is 0 (YES), the process proceeds to step S4 in FIG. 4.

[0028] If the angle Δθ is not 0 in step S25 (NO), the interlocking control unit 15 determines in step S26 whether the angle Δθ is greater than 0. If the angle Δθ is greater than 0 (YES), the image correction unit 18 corrects the cut-out range of the image in step S27 so as to shift the cut-out range by the angle Δθ in the direction of rotational movement. If the angle Δθ is not greater than 0 (NO), the image correction unit 18 corrects the cut-out range of the image in step S28 so as to shift the cut-out range by the angle Δθ in the direction opposite to the direction of rotational movement. Following step S27 or S28, the process proceeds to step S4.

[0029] In Figures 7A to 7C, the rectangle indicated by the solid lines indicates the range of the image in the omnidirectional image that is input to the image correction unit 18. Figures 7A to 7C are common to image data for the right eye and image data for the left eye. The image correction unit 18 supplies image data of a cut-out range, which is obtained by cutting out the image range indicated by the dashed lines from the image range indicated by the solid lines, to the display panel 19. The area indicated by the dashed lines will be referred to as a cut-out area image. If the angle Δθ is 0, the image correction unit 18 cuts out the image in a state where the center of the image range indicated by the solid lines coincides with the center of the cut-out range, as shown in Figure 7A. Figure 7A means that the position of the preset cut-out area image is not corrected.

[0030] If the angle Δθ is greater than 0, it means that the rotation angle θ2 of the robot 30 is smaller than the rotation angle θ1, which is an appropriate rotation angle. Therefore, if the angle Δθ is greater than 0, the image correction unit 18 cuts out an image at a position obtained by moving the cut-out range by the angle Δθ in the direction of rotational movement (to the right in this case), as shown in FIG. 7B. If the angle Δθ is less than 0, it means that the rotation angle θ2 of the robot 30 is greater than the rotation angle θ1, which is an appropriate rotation angle. Therefore, if the angle Δθ is less than 0, the image correction unit 18 cuts out an image at a position obtained by moving the cut-out range by the angle Δθ in the direction opposite to the direction of rotational movement (to the left in this case), as shown in FIG. 7C.

[0031] As described above, when the interlocking switch 16 is set to ON, the interlocking control unit 15 controls the operator motion data acquiring unit 13 to transmit the operator motion data to the robot 30, and also controls the image correcting unit 18 as follows: The interlocking switch 16 is set to ON mainly when communication between the operator 100 and people around the robot 30 is required.

[0032] If there is a difference between the rotation angle θ1 of the image display device 10 indicated by the operator motion data and the rotation angle θ2 of the robot 30 indicated by the robot motion data, the image correction unit 18 corrects the cropped range of the omnidirectional image to be displayed on the display panel 19 to be within the range corresponding to the rotation angle θ1. By correcting the cropped range to be displayed on the display panel 19 to be within the range corresponding to the rotation angle θ1, it is possible to reduce telexistence sickness caused by a misalignment between the orientation of the operator 100 wearing the image display device 10 and the orientation of the camera 36 mounted on the robot 30.

[0033] Specific processing of step S3 shown in Fig. 6 will be described. In step S31, the interlocking control unit 15 acquires the operator motion data held in the operator motion data acquisition unit 13. In step S32, the image correction unit 18 acquires the image data held in the image data acquisition unit 17. The order of steps S31 and S32 may be reversed, or they may be performed simultaneously. In step S33, the image correction unit 18 corrects the cut-out range of the image so as to shift the cut-out range by the rotation angle θ1 in the direction of the rotational motion. Following step S33, the processing proceeds to step S4.

[0034] In this way, when the interlock switch 16 is set to OFF, the interlock control unit 15 controls the operator motion data acquisition unit 13 not to transmit the operator motion data to the robot 30, and also controls the image correction unit 18 as follows: The interlock switch 16 is set to OFF mainly when there is no need for communication between the operator 100 and people around the robot 30, and it is sufficient to acquire images taken by the camera 36.

[0035] Image correction unit 18 corrects the cropped range of the omnidirectional image to be displayed on display panel 19 to a range corresponding to rotation angle θ1. Because the movement of operator 100 and the movement of robot 30 are not linked and the cropped range to be displayed on display panel 19 is corrected to a range corresponding to the rotational movement of operator 100, telexistence sickness can be reduced.

[0036] The interlock switch 16 may be set to on automatically. The image display device 10, such as a head-mounted display, may be provided with a gaze detection function, and when the operator 100 continues to look (i.e., gaze) at a specific person for a predetermined period of time or more while the interlock switch 16 is off, the interlock control unit 15 may determine that the operator is attempting to communicate with that person. In this case, even if the interlock switch 16 is set to off, the interlock control unit 15 may set the interlock switch 16 to on (to a state equivalent to the state when the interlock switch 16 is set to on). Furthermore, when the operator 100 stops gazing after a predetermined period of time, the interlock control unit 15 may be configured to return the interlock switch 16 to off, and return the posture of the robot 30 to the posture before the interlock switch 16 was set to on.

[0037] As shown in Fig. 8, image correction unit 18 may use the image data acquired by image data acquisition unit 17 to set the cropping range to, for example, a 270-degree image range, which is a viewing angle wider than the angle of view of display panel 19. In this case, as shown in Fig. 9, image correction unit 18 compresses the size of the cropping range horizontally so that it fits within the angle of view of the display panel. For example, image correction unit 18 may divide the 270-degree range into three regions: a central 90-degree region and two 90-degree regions on the left and right, and apply low compression to the central 90-degree region and high compression to the two 90-degree region on the left and right. Low compression of the central 90-degree region includes a compression rate of 0.

[0038] If an image range with a wider viewing angle than the angle of view of the display panel 19 is displayed on the display panel 19, the operator 100 can easily obtain necessary visual information without turning his / her head. This reduces the likelihood of telexistence sickness occurring.

[0039] The present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the gist of the present invention. [Explanation of symbols]

[0040] 10 Image display device 11,31 Communications Department 12,35 Motion Sensor 13 Operator motion data acquisition unit 14 Robot motion data acquisition unit 15 Interlocking control unit 16 Interlocking switch 17 Image data acquisition unit 18 Image correction section 19 Display panel 20 Network 30 Robot 32 Control Unit 33 Drive unit 34 Moving parts 36 Camera 37 Video signal processing section

Claims

1. an operator motion data acquisition unit that acquires operator motion data indicating a first rotation angle when an operator who operates the robot performs a rotational motion; a robot motion data acquiring unit that acquires robot motion data transmitted from the robot and indicating a second rotation angle when the robot performs a rotational motion in accordance with the operator motion data; an image data acquisition unit that acquires image data generated by a 360-degree camera provided on the robot and transmitted from the robot; an image correction unit that supplies image data of a cut-out range obtained by cutting out a range of an image to be displayed on a display panel viewed by the operator from the image data acquired by the image data acquisition unit to the display panel, and corrects the cut-out range; an interlocking switch that sets whether or not the rotational movement of the operator and the rotational movement of the robot are to be interlocked; an interlocking control unit that, when the interlocking switch is set to ON, which interlocks the rotational movement of the operator with the rotational movement of the robot, controls the operator motion data to be transmitted to the robot, and, if there is a difference between the first rotation angle and the second rotation angle, corrects the cut-out range to be shifted by the angle difference between the first rotation angle and the second rotation angle, and, when the interlocking switch is set to OFF, which does not interlock the rotational movement of the operator with the rotational movement of the robot, controls the operator motion data not to be transmitted to the robot, and controls the image corrector to correct the cut-out range to be within a range corresponding to the first rotation angle; An image display device comprising:

2. 2. The image display device according to claim 1, wherein, when the interlocking switch is set to ON to interlock the movement of the operator with the movement of the robot, if the first rotation angle is greater than the second rotation angle, the interlocking control unit corrects the cut-out range to shift by the angle difference in the direction of the rotational movement of the operator, and if the first rotation angle is smaller than the second rotation angle, the interlocking control unit corrects the cut-out range to shift by the angle difference in the direction opposite to the direction of the rotational movement of the operator.

3. The image display device according to claim 1 or 2, wherein the interlocking control unit sets the interlocking switch to a state equivalent to being set on when the image displayed on the display panel is continuously viewed for a predetermined period of time or longer.

4. The image correction unit sets the cut-out range to a range of an image having a viewing angle wider than the angle of view of the display panel from the image data acquired by the image data acquisition unit, and compresses the size of the cut-out range so that it fits within the angle of view of the display panel.

5. acquiring operator motion data indicating a first rotation angle when an operator who operates the robot makes a rotational motion; acquiring robot motion data transmitted from the robot, the robot indicating a second rotation angle when the robot performs a rotational motion in response to the operator motion data; Acquire image data generated by a 360-degree camera provided on the robot and transmitted from the robot; supplying image data of a cut-out range obtained by cutting out a range of an image to be displayed on a display panel viewed by the operator from the acquired image data to the display panel; when the rotational movement of the operator and the rotational movement of the robot are set to be linked, control is performed to transmit the operator movement data to the robot, and if there is a difference between the first rotation angle and the second rotation angle, correct the cut-out range so as to be shifted by an angle difference between the first rotation angle and the second rotation angle; When the setting is OFF, which does not link the rotational movement of the operator with the rotational movement of the robot, the control is performed so that the operator movement data is not transmitted to the robot, and the cut-out range is corrected to be a range corresponding to the first rotation angle. Image display method.

Images