Image display device, image display method, and image display system

Abstract

To display an image capable of giving a user both presence about appreciation of an exhibition, an event, etc. and reality about movement between appreciation areas such as exhibition or event performance areas.SOLUTION: An image display device comprises a display unit and a display control unit which controls the display unit. The display control unit causes the display unit to switch and display a real time image in appreciation areas imaged by a camera mounted on a travel object travelling in the appreciation areas and a travel path image based on a recorded image captured in the past on a travel path between appreciation areas.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for displaying images. [Background technology]

[0002] When actually visiting an exhibition having a plurality of exhibition areas, a theme park having a plurality of event areas, or the like, there is a problem that moving between areas can be tiring.

[0003] To solve the above problems, it is conceivable that people could virtually participate in an exhibition or theme park by viewing real-time images of the area from their homes, satellite venues, or other remote locations.

[0004] In a video display system that allows real-time video within an area to be viewed from a remote location, a configuration is conceivable in which the real-time video within each area can be instantly switched, like a multiplex live broadcast relay in television broadcasting. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 5-96062 Summary of the Invention [Problem to be solved by the invention]

[0006] In a video display system that instantaneously switches between real-time images in each area, movement between areas appears as if it were instantaneous movement, and the user is unable to feel a sense of reality regarding movement between areas.

[0007] It should be noted that the virtual experience device disclosed in Patent Document 1 cannot give the user a sense of realism regarding the exhibition or event, since all the images shown to the user are recorded images.

[0008] In view of the above-mentioned problems, the present invention aims to provide a technology for displaying images that can give users both a sense of realism when viewing an exhibition or event, and a sense of reality when moving between viewing areas such as exhibition or event areas (areas similar to areas where exhibitions, events, etc. are viewed, experienced, etc. and enjoyed) [Means for solving the problem]

[0009] An exemplary video display device of the present invention includes a display unit and a display control unit that controls the display unit, and the display control unit switches between displaying real-time video of the viewing area captured by a camera mounted on a moving object moving within the viewing area and video of a moving path based on recorded video captured in the past along the moving path between the viewing areas. [Effects of the Invention]

[0010] According to exemplary embodiments of the present invention, a user can be given both a sense of realism regarding the viewing of an exhibition or event, etc., and a sense of reality regarding movement between viewing areas. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing a configuration of a video display system according to an embodiment. [Figure 2] 1 is a flowchart showing an example of the operation of a video display system according to an embodiment; [Figure 3] FIG. 10 is a diagram schematically illustrating an example of information linking a video display device and a robot. [Figure 4] FIG. 10 is a diagram showing an example of transition of an image displayed by a display unit; [Figure 5] A diagram showing the positional relationship between the viewpoint of the movement path video and the entrance to the second exhibition area. [Figure 6] A diagram showing the relative positions of the robot and the entrance to the second exhibition area. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.

[0013] <Video display system configuration> 1 is a block diagram showing the configuration of a video display system 1 according to an embodiment. The video display system 1 is a system used to allow a user to virtually participate in an exhibition having multiple exhibition areas, a theme park having multiple event areas, or the like.

[0014] Here, we will explain a case where the video display system 1 is used to allow a user to virtually participate in an exhibition having multiple exhibition areas. The exhibition areas may be set on a pavilion-by-pavilion basis, or on a section-by-section basis within a pavilion.

[0015] For the sake of simplicity, the exhibition will be assumed to have only a first exhibition area and a second exhibition area.

[0016] The video display system 1 includes a video display device 10, robots 20A and 20B, and a video providing device 30.

[0017] The video display device 10 includes a display control unit 11, a display unit 12, an input unit 13, a communication unit 14, a speaker 15, and a microphone 16. The video display device 10 is installed in a user's home, a satellite venue of an exhibition, or the like. Although only one video display device 10 is shown in FIG. 1, the video display system 1 includes multiple video display devices 10 so that multiple users can use the video display system 1 simultaneously.

[0018] The display control unit 11 controls the display unit 12 and causes the display unit 12 to display the video provided from the video providing device 30. The display control unit 11 includes, for example, a computer and various circuits having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), input / output ports, etc. In addition to controlling the display, the display control unit 11 also controls audio, controls communications, processes input data, etc.

[0019] A thin display panel such as a liquid crystal display panel or an organic EL (Electro Luminescence) display panel can be used as the display unit 12. If at least a part of the images provided by the image providing device 30 is VR (Virtual Reality) images, for example, VR glasses or VR goggles can be used as the display unit 12.

[0020] The input unit 13 inputs information related to remote control of the robots 20A and 20B, instruction information related to the movement path video, etc. As the input unit 13, for example, a keyboard, a pointing device, a touch panel, a joystick, etc. can be used.

[0021] The communication unit 14 performs data communication with the video providing device 30 via a network (not shown).

[0022] The speaker 15 reconstructs and outputs the sound collected by the microphone 25 of the robot 20A or 20B. The microphone 16 collects sounds around the image display device 10. The sound collected by the microphone 16 is reconstructed and output by the speaker 26 of the robot 20A or 20B. This makes it possible, for example, for a user using the image display device 10 to have a conversation with people in the first or second exhibition area. Note that the microphone 16 and the speaker 26 may not be provided so that the user using the image display device 10 can only hear sounds occurring in the first or second exhibition area. Alternatively, the speaker 15, microphone 16, microphone 25, and speaker 26 may not be provided so that the user using the image display device 10 can enjoy only the image.

[0023] Robot 20A is placed in the first exhibition area. Although only one robot 20A is shown in Fig. 1, video display system 1 includes multiple robots 20A so that multiple users can simultaneously view real-time video of their desired locations within the first exhibition area.

[0024] Robot 20B is placed in the second exhibition area. Although only one robot 20B is shown in Fig. 1, video display system 1 includes multiple robots 20B so that multiple users can simultaneously view real-time video of their desired locations in the second exhibition area.

[0025] Since the robots 20A and 20B have the same configuration, only the robot 20A will be described. Unlike this embodiment, the robots 20A and 20B may have slightly different configurations due to the addition or elimination of functions, functional sophistication, etc.

[0026] The robot 20A includes a control unit 21, a driving unit 22, a camera 23, a communication unit 24, a microphone 25, and a speaker 26. The robot 20A is a self-propelled robot. The robot 20A can be remotely controlled. The position of the robot 20A can be moved by remotely controlling the robot 20A, which makes it easy for the user to virtually move to a desired location within the first exhibition area.

[0027] The control unit 21 controls the entire robot 20A in accordance with remote control.

[0028] The driving unit 22 drives the wheels and the like under the control of the control unit 21, and changes the moving direction and speed of the robot 20A, the orientation of the robot 20A, and the like.

[0029] The camera 23 captures images. The camera 23 may be configured to be movable relative to the body of the robot 20A, so that the orientation of the camera 23 can be changed by remote control even when the robot 20A is stationary.

[0030] The communication unit 24 performs data communication with the video providing device 30 via a network (not shown).

[0031] The microphone 25 collects sounds around the robot 20A. The speaker 26 reconstructs and outputs sounds collected by the microphone 16 of the video display device 10. The robot 20A collects past images (described later) even while being remotely controlled. When collecting past images (described later), the robot 20A may also collect other operating states. In other words, the robot 20A transmits image data including position data, shooting method data, etc. as meta-information to the video providing device 30 as images for generating past images.

[0032] The video providing device 30 includes a control unit 31, a recording unit 32, and a communication unit 33. Note that the video providing device 30 may be, for example, a server device installed in a single location, or may be a distributed server device in which components are installed in multiple locations.

[0033] The control unit 31 controls the entire video providing device 30. The control unit 31 includes, for example, a computer having a CPU, ROM, RAM, HDD, input / output ports, and various circuits.

[0034] The recording unit 32 records video footage previously captured on the route between the first exhibition area and the second exhibition area.

[0035] The recording unit 32 also stores map information of the exhibition site (excluding satellite sites).

[0036] <Video display system operation> Next, a description will be given of the operation of the video display system 1. Fig. 2 is a flowchart showing an example of the operation of the video display system 1.

[0037] For example, the operation of the flowchart shown in Fig. 2 is started when the input unit 13 inputs information indicating the start of use of the system. The operation of the flowchart shown in Fig. 2 is ended when the input unit 13 inputs information indicating the end of use of the system.

[0038] First, in step S10, the image display device 10 selects an exhibition area for virtual entrance based on information input to the input unit 13.

[0039] In the next step S20, the video providing device 30 links the video display device 10 with the robot 20A or 20B, and records the linking information in the recording unit 32. The robot 20A or 20B to be linked to the video display device 10 is a robot (an available robot) that is not linked to another video display device 10 at the time of linking.

[0040] FIG. 3 is a diagram showing an example of linking information between a robot, a video display device, and another robot. A robot 20A with an ID of "111111" is linked to a video display device 10 with an ID of "KKKK." A robot 20B with an ID of "222222" is an available robot. A robot 20B with an ID of "333333" is linked to a video display device 10 with an ID of "BBBB." In the example shown in FIG. 3, the placement area is the "first exhibition area" or the "second exhibition area." However, since some users may want to actually move the robot along a movement path, the placement area may be either of the movement paths. Coordinates indicating the robot's current position include, for example, latitude and longitude information. The robot's current position data can be used for various control purposes, such as displaying the robot's current position on a map.

[0041] When the linking in step S20 is completed, the real-time video captured by the camera 23 of the robot 20A or 20B linked to the video display device 10 is relayed by the router function of the video providing device 30 and displayed on the display unit 12 of the video display device 10 (step S30). Note that if the linking between the robot 20A and the video display device 10 allows data for connection processing with the communication partner, such as each other's IP addresses, to be known, the real-time video can be transferred directly between the robot 20A and the video display device 10 without going through the video providing device 30.

[0042] During execution of step S30, the user can operate the input unit 13 to remotely control the robot 20A or 20B to which the image display device 10 is linked.

[0043] In the next step S40, the video providing device 30 determines whether or not the user will virtually exit the exhibition area that the user is virtually entering. For example, when the robot 20A or 20B linked to the video display device 10 is remotely controlled to move to the exit gate of the exhibition area that the user is virtually entering, the video providing device 30 may determine that the user will virtually exit the exhibition area that the user is virtually entering.

[0044] If it is determined in step S40 that the user will not pseudo-exit, the process continues with step S30. On the other hand, if it is determined in step S40 that the user will pseudo-exit, the process proceeds to step S50.

[0045] In step S50, the video providing device 30 releases the association between the video display device 10 and the robot 20A or 20B.

[0046] In the next step S60, the image display device 10 displays the travel route image provided by the image providing device 30 on the display unit 12.

[0047] Here, an outline of the travel route video will be explained.

[0048] The video providing device 30 generates a travel path video based on recorded video previously taken along the travel path between the first exhibition area and the second exhibition area. In other words, the video providing device 30 generates a travel path video by editing recorded video previously taken along the travel path between the first exhibition area and the second exhibition area. The video providing device 30 determines the content of the editing based on instruction information input to the input unit 13. Examples of instruction information input to the input unit 13 include a position along the travel path, a viewpoint of the video, and a viewing direction of the video.

[0049] If the moving speed of the camera that captured the recorded video is not constant, it is preferable that the video providing device 30 converts the recorded video into a video in which the camera moves at a constant speed (normalized video), and edits the normalized video to generate a video of the movement path. Note that in order for the video providing device 30 to create a normalized video, the moving speed of the camera needs to be included in the meta information of the recorded video.

[0050] The recorded video previously captured on the path of travel between the first exhibition area and the second exhibition area may be, for example, video captured by camera 23 after robot 20A or robot 20B is moved outside the exhibition area. This allows for a smooth connection between the real-time video within the exhibition area and the video of the path of travel generated by video providing device 30. Note that the recorded video previously captured on the path of travel between the first exhibition area and the second exhibition area may be, for example, video captured by a camera mounted on a smartphone, a drive recorder, or a fixed camera installed on the path of travel.

[0051] The video display device 10 varies the playback speed of the travel path video by transmitting instruction information regarding the playback speed input to the input unit 13 to the video providing device 30. For example, by increasing the playback speed of the travel path video, the time required for the simulated travel between the first exhibition area and the second exhibition area can be shortened.

[0052] In the next step S70, the image providing device 30 determines whether the user will virtually enter the exhibition area. For example, when the viewpoint of the movement path image reaches the entrance of the exhibition area, the image providing device 30 may determine that the user will virtually enter the exhibition area.

[0053] If it is determined in step S70 that the user will not enter the facility in a false manner, the process continues with step S60. On the other hand, if it is determined in step S70 that the user will enter the facility in a false manner, the process proceeds to step S20.

[0054] Through the above-described operation, the display control unit 11 of the video display device 10 switches between real-time video of the first exhibition area or the second exhibition area captured by the camera 23 and video of the travel route based on recorded video previously captured on the travel route between the first exhibition area and the second exhibition area, and displays the video on the display unit 12.

[0055] For example, if a user virtually looks around a first exhibition area, then virtually moves from the first exhibition area to a second exhibition area, and finally virtually looks around the second exhibition area, the images displayed by the display unit 12 will transition in the order of real-time image 41 within the first exhibition area, image 42 of the movement route, and real-time image 43 within the second exhibition area, as shown in Figure 4.

[0056] This video switching can give the user both a sense of realism about the exhibit and a sense of reality about movement between exhibit areas.

[0057] Furthermore, since the travel route video is generated based on recorded video, even if the travel route is congested in real time, the travel route video is not adversely affected.

[0058] In addition, the video of the route will allow the user to virtually move between the first and second exhibition areas, so it is expected that serendipity will occur regarding the scenery during the move, the stores along the route, and the products lined up in the stores along the route.

[0059] In step S30, real-time video captured by camera 23 is displayed on display unit 12, but there may be cases where it is desired to view the exhibit virtually from an angle that cannot be captured by camera 23 (for example, directly above the exhibit). Therefore, in step S30, the real-time video captured by camera 23 and an area image based on recorded video previously captured within the exhibition area by a camera different from camera 23 (for example, a fixed camera installed within the exhibition area, a camera mounted on a drone, etc.) may be switched and displayed on display unit 12. Display control unit 11 may switch between the real-time video and the area image based on instruction information input to input unit 13.

[0060] Furthermore, in step S60, the robot 20B that is not linked to another image display device 10 may be moved in advance (before the image of the movement path is switched to the real-time image of the second exhibition area) to a point where it is expected that the image will switch from the image of the movement path to the real-time image of the second exhibition area. This allows the image of the movement path to be switched to the real-time image of the second exhibition area without giving the user a sense of incongruity.

[0061] The image providing device 30 may predict the point at which the image of the travel route will switch to real-time image of the second exhibition area, for example, based on the distance between the viewpoint of the travel route image and the entrance to the second exhibition area. Specifically, when the distance between the viewpoint of the travel route image and the entrance to the second exhibition area approaches a certain distance L1, the image providing device 30 may predict that the entrance is the point at which the image of the travel route will switch to real-time image of the second exhibition area.

[0062] For example, as shown in Figure 5, if the second exhibition area 51 has a first entrance 52 and a second entrance 53, and the distance between the viewpoint 54 of the movement path image and the first entrance 52 is a certain distance L1, and the distance between the viewpoint 54 of the movement path image and the second entrance 53 is a distance L2 that is longer than the certain distance L1, the image providing device 30 predicts that the first entrance 52 is the point where the image switches from the movement path image to real-time image within the second exhibition area, and directs the robot 20B toward the first entrance 52.

[0063] As another method, in step S60, the video providing device 30 may set the point at which the video of the movement path is switched to the real-time video of the second exhibition area to the stopping position of the robot 20B. This makes it possible to switch from the video of the movement path to the real-time video of the second exhibition area without giving the user a sense of discomfort.

[0064] For example, as shown in Figure 6, if the second exhibition area 51 has a first entrance 52 and a second entrance 53 and the robot 20B is stopped at the second entrance 53, the image providing device 30 may forcibly set the movement path so that the viewpoint 54 of the movement path image moves to the second entrance 53, or may superimpose an arrow or the like on the movement path image so that the end point of the movement path is the second entrance 53 based on instruction information input by the user to the input unit 13, thereby guiding the end point of the movement path to the second entrance 53.

[0065] <Other> The above-described embodiments should be considered to be illustrative in all respects and not restrictive. The technical scope of the present invention is indicated by the claims, not by the description of the above-described embodiments, and should be understood to include all modifications that fall within the meaning and scope of the claims.

[0066] For example, in the above-described embodiment, the robot is a self-propelled robot, but it may also be a drone, etc. Also, instead of a robot, a vehicle that travels on an orbit may be used, and the vehicle that travels on an orbit may be equipped with a camera that can take 360-degree images, and a part of the image captured by the camera that can take 360-degree images may be cut out and used as real-time image, and the cut-out position may be remotely controlled.

[0067] Furthermore, the video display system may be configured so that a fee is required for simulated entry into each exhibition or event area, or so that a fee is required only for simulated entry into one exhibition, theme park, etc. [Explanation of symbols]

[0068] 1. Video display system 10. Video display device 11 Display control unit 12 Display section 13 Input section 14 Communications Department 15 Speaker 16. Mike 20A, 20B Robot 21 Control Unit 22 Drive unit 23 Camera 24 Communications Department 25. Mike 26 Speaker 30 Video providing equipment 31 Control Unit 32 Recording section 33 Communications Department 41 Real-time video of the first exhibition area 42 Travel route video 43 Real-time video of the second exhibition area 51 Second Exhibition Area 52 1st entrance 53 Second entrance 54 Travel route video

Claims

1. A display unit; a display control unit that controls the display unit, The display control unit of this video display device switches between real-time video of the viewing area captured by a camera mounted on a moving body moving within the viewing area and video of the travel path based on recorded video captured in the past along the travel path between the viewing areas, and displays this video on the display unit.

2. 2. The image display device according to claim 1, wherein the playback speed of the travel path image is variable.

3. The video display device described in claim 1 or claim 2, wherein the display control unit switches between the real-time video, the travel path video, and an area video based on recorded video previously taken within the viewing area by a camera other than the camera, and displays them on the display unit.

4. The image display device according to any one of claims 1 to 3, The moving body; a video providing device that provides the real-time video and the travel route video to the video display device; A video display system comprising:

5. The video display system according to claim 4 , wherein the moving body is a remotely controlled robot.

6. The image display system according to claim 5 , wherein the robot moves in advance to a point where the movement path image is expected to be switched to the real-time image.

7. The image display system according to claim 6 , wherein the point at which the moving path image is switched to the real-time image is set at a stopping position of the robot.

8. the moving object is capable of moving outside the viewing area, 8. The video display system according to claim 4, wherein the travel path video is generated based on recorded video previously captured by the camera on the travel path between the viewing areas.

9. This video display method switches between real-time video within a viewing area taken by a camera mounted on a moving body moving within the viewing area and video of a moving path based on recorded video taken in the past on the moving path between the viewing areas.

Images