Remotely controlled device and image display device

Abstract

To provide a remotely controlled device in a remote control system comprising the remotely controlled device and an image display device for displaying a video signal transmitted from the remotely controlled device, the remotely controlled device being capable of mitigating video sickness or confusion of a user even when the remotely controlled device has moved regardless of the user's intention.SOLUTION: When a controlling unit 31 controls a drive unit 32 so as to move a movable unit 33 depending on a user's movement, a video signal processing unit 36 is controlled so as to make a rate of updating frame video of a video signal to be transmitted to an image display device 10 be a first rate. When the controlling unit 31 controls the drive unit 32 so as to move the movable unit 33 regardless of the user's movement, the video signal processing unit 36 is controlled so as to make the rate of updating frame video of the video signal to be transmitted to the image display device 10 be a second rate lower than the first rate.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a remotely controlled device and an image display device. [Background technology]

[0002] As described in Patent Document 1, a remote control system has been put into practical use in which a user wearing an image display device such as a head-mounted display controls a robot (a remotely controlled device) located in a remote location. The technology used in this type of remote control system for controlling a remotely located robot in real time is called telexistence. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6801136 Summary of the Invention [Problem to be solved by the invention]

[0004] A remotely controlled device such as a robot is basically configured to move in accordance with the user's movement. However, in order to avoid the remotely controlled device being damaged, for example, when some object hits the remotely controlled device, the remotely controlled device may take action to avoid danger. In this case, the video signal captured by the camera mounted on the remotely controlled device and transmitted to the image display device worn by the user moves suddenly and unintentionally, which may cause the user to experience motion sickness or become confused.

[0005] The present invention aims to provide a remotely controlled device, an image display device, and an image display control method in a remote control system comprising a remotely controlled device and an image display device that displays a video signal transmitted from the remotely controlled device, which can reduce visually induced motion sickness or confusion in a user even if the remotely controlled device moves regardless of the user's intentions. [Means for solving the problem]

[0006] The present invention provides a remotely controlled device comprising: a movable part; a drive part for driving the movable part; a network communication part for receiving motion data indicating a user's motion via a network and transmitting video data to an image display device worn by the user; a control part for controlling the drive part to move the movable part in accordance with the motion data; a camera attached to the movable part; and a video signal processing part for generating the video data based on a video signal captured by the camera, wherein the control part controls the video signal processing part to generate the video data that updates frame images at a first rate when controlling the drive part to move the movable part in accordance with the motion data, and to generate the video data that updates frame images at a second rate lower than the first rate when controlling the drive part to move the movable part independently of the motion data.

[0007] The present invention provides an image display device comprising: a motion sensor that detects user movement; a receiving unit that receives a video signal captured by a camera attached to a movable part of a remotely controlled device that is configured to move according to movement data indicating the user movement detected by the movement sensor, and additional data for identifying the start and end of a period for rotating the movable part; and an image display unit that displays an image of the video signal received by the receiving unit, wherein the additional data is superimposed on the video signal when the remotely controlled device moves the movable part regardless of the movement data, and when the additional data is not input, the image display unit displays an image of the video signal received by the receiving unit at a first rate that is a rate at which frame images are updated, and when the additional data is input, displays an image of the video signal received by the receiving unit at a second rate that is a rate at which frame images are updated that is lower than the first rate.

[0008] The present invention provides a video display control method in which motion data indicating a user's motion is transmitted to a remotely controlled device, the remotely controlled device moves a movable part in accordance with the received motion data, the remotely controlled device transmits a video signal captured by a camera attached to the movable part to an image display device worn by the user, an image display unit of the image display device displays an image of the video signal transmitted from the remotely controlled device, and when a control unit of the remotely controlled device moves the movable part in accordance with the motion data, the image display unit displays an image of the video signal having a frame image update rate of a first rate, and when the control unit moves the movable part regardless of the user's motion, the remotely controlled device transmits a video signal having a frame image update rate of a second rate lower than the first rate to the image display device, and the image display unit displays an image of the video signal having the second rate, or the remotely controlled device transmits a video signal having the first rate to the image display device, and the image display unit displays an image of the video signal having the second rate based on the video signal having the first rate. [Effects of the Invention]

[0009] According to the remotely controlled device, image display device, and video display control method of the present invention, in a remote control system comprising a remotely controlled device and an image display device that displays a video signal transmitted from the remotely controlled device, it is possible to reduce the user's visually induced motion sickness or confusion even if the remotely controlled device moves regardless of the user's intentions. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example of the configuration of a remote control system including the image display device of the first or second embodiment and the remotely controlled device of the first or second embodiment. [Figure 2A] 10 is a diagram looking down on the robot from directly above, showing the rotation of the robot's head 30H from direction D0 to direction D1. [Figure 2B]10 is a diagram looking down on the robot from directly above, showing the rotation of the robot's head 30H from direction D0 to direction D2. [Figure 3] 3 is a block diagram showing a specific configuration example of a video signal processing unit included in the remotely controlled device of the first embodiment. FIG. [Figure 4] 10A and 10B are diagrams showing examples of a video signal of 60 frames / second and a video signal in which the frame video update rate is 20 frames / second. [Figure 5] 10A and 10B are diagrams showing examples of a video signal of 60 frames / second, and video signals with frame image update rates of 20 frames / second and 10 frames / second. [Figure 6] FIG. 10 is a block diagram showing a specific example of the configuration of a video signal processing unit and a control unit included in a remotely controlled device of the second embodiment. [Figure 7] FIG. 10 is a block diagram showing a specific example of the configuration of an image display unit included in an image display device according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, a remotely controlled device, an image display device, and a video display control method according to each embodiment will be described with reference to the accompanying drawings. First, a configuration example of a remote control system including an image display device and a remotely controlled device will be described with reference to FIG.

[0012] In FIG. 1, an image display device 10 and a robot 30 are connected to each other via a network 20. The robot 30 is located at a remote location away from the location of the image display device 10. The image display device 10 is, for example, a head-mounted display. The robot 30 is an example of a remotely controlled device whose movement is controlled according to movement data indicating the movement of the image display device 10 or the user wearing the image display device 10. The network 20 is typically the Internet.

[0013] The image display device 10 includes an image display unit 11, a motion sensor 12, a speaker 13, and a network communication unit 14. The motion sensor 12 includes an acceleration sensor and a gyro sensor, and detects acceleration when the image display device 10 (i.e., the user) moves, and angular velocity indicating a change in rotation or orientation. The speaker 13 includes headphones or earphones.

[0014] When the user wearing the image display device 10 moves, the motion data indicating the user's motion detected by the motion sensor 12 is encoded and transmitted to the robot 30 via the network communication unit 14 and the network 20. The network communication unit 14 functions as a transmitting unit.

[0015] The robot 30 includes a control unit 31, a drive unit 32, a movable unit 33, a network communication unit 34, a camera 35, a video signal processing unit 36, microphones 37a to 37d, and distance measurement sensors 38a to 38d. The control unit 31 can be configured with a microcomputer or a microprocessor.

[0016] When the network communication unit 34 receives the motion data transmitted from the image display device 10, the network communication unit 34 supplies the motion data to the control unit 31. The network communication unit 34 functions as a receiving unit. The control unit 31 decodes the encoded motion data and controls the drive unit 32 to move the movable unit 33 in accordance with the motion data. The movable unit 33 is, for example, the head and legs of the robot 30. The drive unit 32 includes an actuator, which is at least one of an electric motor, an electromagnetic solenoid, a hydraulic cylinder, and a pneumatic cylinder, for rotating the head and moving the legs. The drive unit 32 drives the actuator based on the control of the control unit 31.

[0017] The movable part 33 of the robot 30 moves in accordance with the movement data transmitted from the image display device 10, so that the robot 30 moves so as to follow the movement of the user.

[0018] For example, a camera 35 is attached to the head 30H of the robot 30 (see FIG. 2A or 2B), and the camera 35 captures an image within a predetermined angle range around the robot 30. The camera 35 may be a stereo camera. A video signal consisting of digital data output from the camera 35 is supplied to a video signal processing unit 36. The video signal processing unit 36 ​​compresses and encodes the video signal, and supplies the compressed and encoded video data to the control unit 31. The network communication unit 34 transmits the video data supplied from the control unit 31 to the image display device 10 via the network 20. The network communication unit 34 functions as a transmission unit.

[0019] When the network communication unit 14 receives the video data transmitted from the robot 30, the network communication unit 14 supplies the video data to the image display unit 11. The network communication unit 14 functions as a receiving unit. The image display unit 11 displays an image based on a decoded video signal obtained by decoding the video data. The user visually recognizes the image displayed on the image display unit 11.

[0020] The robot 30 may encode the audio signals picked up by the microphones 37a to 37d and transmit the encoded audio data to the image display device 10. For example, the microphones 37a to 37d are arranged at 90-degree intervals in a line in the circumferential direction of the robot 30. When the image display device 10 receives the audio data, the speaker 13 outputs audio based on a decoded audio signal decoded by an audio decoding unit (not shown).

[0021] In this way, in a remote control system in which the image display device 10 and the robot 30 are connected via a network 20, the image display device 10 (user) controls the robot 30 in real time. The robot 30 moves according to movement data indicating the user's movement, and the image display device 10 receives images captured by the robot 30, or sounds picked up by the robot 30 in addition to the images.

[0022] To prevent the robot 30 from colliding with an object and breaking down, the robot 30 is configured to take action to avoid danger. The robot 30 is equipped with distance measurement sensors 38a to 38d. For example, the distance measurement sensors 38a to 38d are arranged at 90-degree intervals in a line around the circumference of the robot 30. The control unit 31 detects the presence or absence of an object approaching the robot 30 and the direction of the approach based on the sound picked up by the microphones 37a to 37d and the distance to the object measured by the distance measurement sensors 38a to 38d. The control unit 31 determines that an object is approaching based on the detection result.

[0023] The control unit 31 may detect the presence or absence of an object approaching the robot 30 and the direction of the approach based only on the distance to the object measured by the distance measuring sensors 38a to 38d.

[0024] As shown in FIG. 2A , the head 30H of the robot 30 faces in a direction D0, where the center of the camera 35's image capture range is aligned with the camera's 35's blind spot. The control unit 31 determines that an object is approaching from a direction Dx1, which is a blind spot of the camera 35. The direction from which the object is approaching is different from the direction from which the camera 35 is capturing the image. In this case, the control unit 31 controls the driving unit 32 to rotate the head 30H so that the center of the camera's image capture range is aligned with the camera's 35's image capture range in the direction D1, in order to confirm the object. Once the camera 35 captures the image of the object, the user can control the robot 30 to drive the legs to cause the robot 30 to take action to avoid danger. Alternatively, the control unit 31 may control the driving unit 32 to drive the legs to cause the robot 30 to take action to avoid danger.

[0025] 2B shows a case where the control unit 31 detects an approaching object from direction Dx2, which is in the blind spot of the camera 35 and is substantially behind direction D0. The direction from which the object is approaching in this case is a direction different from the direction from which the camera 35 is capturing images. In this case as well, the control unit 31 controls the drive unit 32 to rotate the head 30H so that the center of the capturing range of the camera 35 is in direction D2, in order to confirm the object.

[0026] In this way, when the robot 30 rotates its head 30H to take action to avoid danger, the video signal (video data) transmitted to the image display device 10 moves suddenly regardless of the user's intention, even if the user is standing still. This can cause the user to become visually ill or confused. Therefore, the image display device 10 and the robot 30 are configured as in the following first or second embodiment.

[0027] First Embodiment In the first embodiment, in order to reduce visually induced motion sickness and confusion in the user, the video signal processing unit 36 ​​in the robot 30 is configured as follows. As shown in Fig. 3, the video signal processing unit 36 ​​has a write control unit 361, a storage unit 362, a read control unit 363, a frame update control unit 364, and an encoding unit 365.

[0028] The write control unit 361 writes, for example, a 60 frame / second video signal output from the camera 35 into the storage unit 362. When the robot 30 is moving so as to follow the user's movement, the frame update control unit 364 controls the read control unit 363 to read the video signal stored in the storage unit 362 at 60 frames / second. Therefore, the read control unit 363 reads the video signal stored in the storage unit 362 at 60 frames / second. The encoding unit 365 compresses and encodes the read 60 frame / second video signal, and supplies the compressed and encoded video data to the control unit 31.

[0029] At this time, the frame image is updated at a first rate in the 60 frame / second video signal transmitted to the image display device 10. A frame image is an image in frame units.

[0030] The control unit 31 controls the driving unit 32 to rotate the head 30H as shown in FIG. 2A or 2B so that the robot 30 itself can avoid danger. At this time, the frame update control unit 364 controls the read control unit 363 to read the video signal stored in the storage unit 362 at a rate lower than 60 frames per second. For example, the frame update control unit 364 controls the read control unit 363 to read the video signal stored in the storage unit 362 at a rate equivalent to 20 frames per second. The encoding unit 365 compresses and encodes the read video signal equivalent to 20 frames per second, and supplies the compressed and encoded video data to the control unit 31.

[0031] At this time, the rate at which the frame images of the video signal transmitted to the image display device 10 are updated is a second rate that is lower than the first rate. The video signal at the second rate is generated based on the video signal at the first rate.

[0032] The control unit 31 can calculate the time required to rotate the head 30H so that the center of the imaging range of the camera 35 is in the direction D1 or D2. Therefore, the control unit 31 instructs the frame update control unit 364 to read at a rate lower than 60 frames per second from the time the head 30H starts to rotate until the center of the imaging range of the camera 35 is in the direction D1 or D2.

[0033] 4(a) shows frame images F1, F2, F3, F4... at 60 frames / second input to the video signal processing unit 36. The frame update control unit 364 controls the read control unit 363 to update the frame images that are read out once every three frames out of the frame images stored in the storage unit 362. As a result, as shown in FIG. 4(b), the frame image update rate is 20 frames / second, i.e., frame images F1, F1, F1, F4, F4, F4, F5.... The video signal processing unit 36 ​​outputs compression-encoded video data at substantially 20 frames / second.

[0034] 4(b) is a 60 frame / second video signal, but since the frame video update rate is 20 frames / second, it appears to be equivalent to a 20 frame / second video signal. Since video data equivalent to 20 frames / second is transmitted to the image display device 10, the video does not move suddenly, reducing the risk of the user experiencing motion sickness or confusion.

[0035] In (b) of Fig. 4, the frame update control unit 364 controls the reading of frame images from the storage unit 362 to reduce the rate at which the frame images are updated. As shown in (c) of Fig. 4, the video signal processing unit 36 ​​may actually control the frame rate to 20 frames per second. It is also possible to set the update rate of the frame images read from the storage unit 362 to 20 frames per second by thinning out the frame images written to the storage unit 362.

[0036] 2B, the angle by which head 30H is rotated is larger than that in FIG. 2A. Control unit 31 may increase the rotation speed as the angle by which head 30H is rotated increases. Correspondingly, control unit 31 may decrease the rate at which the frame images are updated as the rotation speed increases.

[0037] Figure 5(a) shows frame images F1, F2, F3, F4... output at 60 frames per second by the video signal processing unit 36. Figure 5(b) shows the rate at which the frame images are updated when the angle at which the head 30H is rotated is relatively small and the rotation speed is relatively slow, as shown in Figure 2A. As with Figure 4(b), the rate at which the frame images are updated is 20 frames per second.

[0038] Figure 5(c) shows the frame image update rate when the head 30H is rotated at a large angle and at a high rotation speed, as shown in Figure 2B. In Figure 5(c), the frame image update rate is set to 10 frames / second. Even if the control unit 31 rotates the head 30H quickly, image data equivalent to 10 frames / second is transmitted to the image display device 10, so the image does not move suddenly, and it is possible to reduce motion sickness and confusion experienced by the user.

[0039] It is preferable that the control unit 31 gradually increases the rate at which the frame images are updated immediately before the rotation of the head 30H is stopped, and gradually returns the rate to 60 frames per second.

[0040] Second Embodiment In the second embodiment, in order to reduce visually-induced motion sickness and confusion in the user, the video signal processing unit 36 ​​in the robot 30 and the image display unit 11 in the image display device 10 are configured as follows.

[0041] 6, the video signal processing unit 36 ​​has a write control unit 361, a storage unit 362, a read control unit 363, and an encoding unit 365. The control unit 31 has an additional data superimposing unit 311 that superimposes additional data on the compression-encoded video data output from the encoding unit 365. The video data output from the video signal processing unit 36 ​​is 60 frames per second. When the control unit 31 causes the robot 30 to rotate the head 30H in order to take action to avoid danger, the additional data superimposing unit 311 superimposes additional data on the video data to identify the period from the start to the end of the period during which the head 30H is rotated.

[0042] The additional data may include additional data indicating the start timing of the period during which head 30H is rotated and additional data indicating the end timing, or may include additional data indicating the start timing of the period during which head 30H is rotated and additional data indicating the rotation time. If control unit 31 increases the rotation speed as the angle by which head 30H is rotated increases, the additional data preferably includes additional data indicating at least two levels of rotation speed.

[0043] In this way, the robot 30 transmits the video data on which the additional data is superimposed to the image display device 10 when the robot 30 itself turns the head 30H to take action to avoid danger.

[0044] 7, the image display unit 11 has a decoding unit 111, a writing control unit 112, a storage unit 113, a reading control unit 114, a frame update control unit 115, and a display panel 116. The decoding unit 111 decodes the video data transmitted from the robot 30. The writing control unit 112 writes the decoded 60 frames / second video signal into the storage unit 113. The frame update control unit 115 receives the decoded additional data.

[0045] If no additional data is input, frame update control unit 115 controls read control unit 114 to read out the video signal stored in storage unit 113 at 60 frames per second. The 60 frames per second video signal supplied to display panel 116 has a first rate at which the frame video is updated.

[0046] When the additional data is input, the frame update control unit 115 controls the read control unit 114 to read out the video signal stored in the memory unit 113 from the start to the end of the period in which the head 30H is rotated so that the rate at which the frame video is updated is lower than 60 frames / second.

[0047] As in the first embodiment, the frame update control unit 115 controls the read control unit 114 to update the frame image that is read out once every three frames from among the frame images stored in the storage unit 113. As a result, the rate at which the frame images of the video signal displayed on the display panel 116 are updated is 20 frames per second, such as frame images F1, F1, F1, F4, F4, F4, F5, ... as in (b) of FIG.

[0048] The rate at which frame images of the video signal supplied to the display panel 116 are updated is a second rate that is lower than the first rate. The video signal at the second rate is generated based on the video signal at the first rate.

[0049] If the additional data includes information indicating the level of the rotation speed, the frame update control unit 115 may control the read control unit 114 so that the rate at which the frame images are updated decreases as the rotation speed increases.

[0050] In the second embodiment, the robot 30 transmits video data at 60 frames per second to the image display device 10. If the robot 30 itself rotates its head 30H to take action to avoid danger, a video signal with a frame video update rate of 20 frames per second or 10 frames per second is displayed on the display panel 116 of the image display device 10. Therefore, also in the second embodiment, the video does not move suddenly, and it is possible to reduce visually induced motion sickness and confusion for the user.

[0051] In the second embodiment as well, it is preferable to gradually increase the rate at which the frame images are updated immediately before the rotation of the head 30H is stopped, and gradually return it to 60 frames per second.

[0052] When the user's line of sight moves up and down as a result of moving their face up and down, instead of updating the frame video rate, image display unit 11 may display the image as follows: Image display unit 11 cuts out the area that overlaps in the vertical direction between the image before the movement and the image after the movement, and enlarges it as necessary, to display an image that is not blurred in the vertical direction.

[0053] The present invention is not limited to the first or second embodiment described above, and various modifications are possible without departing from the gist of the present invention. [Explanation of symbols]

[0054] 10 Image display device 11 Image display section 12 Motion Sensor 13 Speaker 14, 34 Network communication unit (transmitter, receiver) 20 Network 30 Robot (remotely controlled device) 31 Control Unit 32 Drive unit 33 Moving parts 35 Camera 36 Video signal processing section 37a~37d Microphones 38a~38d Distance measurement sensor 111 Decoding unit 112,361 Write control section 113,362 storage section 114,363 Read control section 115,364 Frame Update Control Unit 116 Display Panel 311 Additional data superimposition unit 365 Encoding section

Claims

1. A movable part; a drive unit that drives the movable unit; a network communication unit that transmits video data to an image display device worn by a user via a network; a control unit that controls the drive unit to rotate the movable unit; a camera attached to the movable part; a video signal processing unit that generates the video data based on a video signal captured by the camera; Equipped with the video signal processing unit generates the video data at a frame rate that varies depending on the rotation speed of the movable unit when the control unit controls the movable unit to rotate; Remotely controlled device.

2. the video signal processing unit generates the video data at a lower frame rate as the rotation speed of the movable part increases, The remotely controlled device according to claim 1 .

3. the control unit rotates the movable unit in accordance with the user's motion data, and controls the drive unit to move the movable unit independently of the motion data; the video signal processing unit generates the video data at different frame rates depending on the rotation speed of the movable unit when the control unit controls the movable unit to rotate independently of the motion data. The remotely controlled device according to claim 1 .

4. a receiving unit that receives a video signal captured by a camera attached to a movable part of the remote controlled device, and additional data for identifying the rotation speed of the movable part and the start and end of a period during which the movable part is rotated; an image display unit that displays an image of the video signal received by the receiving unit; Equipped with the image display unit, when the movable part is rotating, displays the video signal as an image with a different frame rate depending on the rotation speed of the movable part. Image display device.

Images