Terminal devices, methods, and programs

Abstract

To record memories for activities on a VR space.SOLUTION: A terminal device comprises: a calculation unit for calculating a relative position between a display device that displays virtual reality to be used by a user and a terminal device that the user is actually holding in his / her hands; a display control unit for displaying a virtual terminal device on a virtual reality space according to control signals from the terminal device, and changing, on the virtual reality space displayed on the display device, a photographing range to be displayed on the virtual terminal device, based on the relative position; a video data acquisition unit for acquiring video data related to a video on a space in the photographing range that is changed by the display control unit on the virtual reality space; and a display unit for displaying information. The display control unit causes the display unit of the virtual terminal device to display a video related to the video data acquired by the video data acquisition unit, and the display unit displays information other than the video related to the video data.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a terminal device, a method, and a program.

Background Art

[0002] A technology that enables a plurality of users to experience a VR space that realizes virtual reality (VR) is known.

[0003] For example, Patent Document 1 describes a technology that enables a plurality of users to experience VR while moving the positions and orientations of their respective viewpoints within a VR space at the same time.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] It is assumed that various communications between a plurality of users are realized in a VR space. For example, in a VR space, a situation is assumed in which each user uses an avatar created individually to virtually go out to a display facility such as an art museum, a theme park, a trip, etc., or participate in a common game. Therefore, it is desired to record a scenery as a memory in the VR space or take a group photo of avatars by performing a predetermined operation in the real world.

[0006] An object of the present invention is to provide a terminal device, a method, and a program capable of recording memories of activities in a VR space.

Means for Solving the Problems

[0007] A terminal device according to one aspect of the present invention comprises: a display device that displays a virtual reality used by a user; a calculation unit that calculates the relative position of the terminal device actually held by the user; a display control unit that displays a virtual terminal device in the virtual reality space according to a control signal from the terminal device and changes the shooting range displayed on the virtual terminal device in the virtual reality space displayed on the display device based on the relative position; a video data acquisition unit that acquires video data relating to the video in the shooting range space changed by the display control unit in the virtual reality space; and a display unit that displays information. The display control unit causes the video related to the video data acquired by the video data acquisition unit to be displayed on the display unit of the virtual terminal device, and the display unit displays information other than video related to the video data.

[0008] A method according to one aspect of the present invention includes the steps of: calculating the relative position between a display device that displays a virtual reality used by a user and a terminal device actually held by the user; displaying a virtual terminal device in the virtual reality space in accordance with a control signal from the terminal device, and changing the shooting range displayed on the virtual terminal device in the virtual reality space displayed on the display device based on the relative position; acquiring video data relating to the image in the space of the changed shooting range in the virtual reality space; displaying the image relating to the acquired video data on the display unit of the virtual terminal device; and displaying non-video information relating to the video data on the display unit of the terminal device.

[0009] A program according to one aspect of the present invention causes a computer to perform the following steps: calculate the relative position between a display device that displays a virtual reality used by a user and a terminal device actually held by the user; display a virtual terminal device in the virtual reality space in accordance with a control signal from the terminal device, and change the shooting range displayed on the virtual terminal device in the virtual reality space displayed on the display device based on the relative position; acquire video data relating to the image in the space of the changed shooting range in the virtual reality space; display the image relating to the acquired video data on the display unit of the virtual terminal device; and display non-video information relating to the video data on the display unit of the terminal device. [Effects of the Invention]

[0010] According to the present invention, it is possible to record memories of activities in a VR space. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 is a diagram illustrating an example configuration of an information processing system according to an embodiment. [Figure 2] Figure 2 is a block diagram showing an example configuration of a terminal device according to an embodiment. [Figure 3] Figure 3 is a diagram illustrating the identifiers displayed on the display unit. [Figure 4] Figure 4 is a block diagram showing an example configuration of a display device according to the embodiment. [Figure 5] Figure 5 is a block diagram showing an example of a server configuration according to the embodiment. [Figure 6] Figure 6 is a diagram illustrating the method of using the terminal device and display device according to the embodiment. [Figure 7] Figure 7 is a diagram illustrating the method of using the terminal device and display device according to the embodiment. [Figure 8] Figure 8 is a flowchart showing an example of the processing flow of a terminal device according to this embodiment. [Figure 9]Figure 9 is a block diagram showing an example configuration of a terminal device according to the first modified example. [Figure 10] Figure 10 is a block diagram showing an example configuration of a terminal device according to the second modified example. [Figure 11] Figure 11 is a block diagram showing an example configuration of a display device according to the second modified example. [Modes for carrying out the invention]

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. Note that the present invention is not limited to these embodiments, and if there are multiple embodiments, they may be constructed by combining each embodiment. Furthermore, in the following embodiments, the same parts are denoted by the same reference numerals to omit redundant explanations. [Information Processing Systems] The information processing system according to the embodiment will be described using Figure 1. Figure 1 is a diagram illustrating an example of the configuration of the information processing system according to the embodiment.

[0013] As shown in Figure 1, the information processing system 1 comprises a terminal device 10, a display device 20, and a server 30. The terminal device 10, the display device 20, and the server 30 are communicated with each other via a network N. The information processing system 1 is a system that can record memories in a VR space. The terminal device 10 is a device that includes, for example, a smartphone or a tablet terminal. The display device 20 is a device that allows users to experience VR. The display device 20 is a device that includes, for example, an HMD (Head Mounted Display) worn on the user's head. The user can, for example, wear the display device 20 on their head and experience a theme park in a VR space using avatars with other users on the server 30. [Terminal device] The configuration of the terminal device according to the embodiment will be explained using Figure 2. Figure 2 is a block diagram showing an example of the configuration of the terminal device according to the embodiment.

[0014] As shown in FIG. 2, the terminal device 10 includes a display unit 12, an audio output unit 13, an operation unit 14, a storage unit 15, a communication unit 16, and a control unit 17. The display unit 12, the audio output unit 13, the operation unit 14, the storage unit 15, the communication unit 16, and the control unit 17 are respectively connected by a bus B1.

[0015] The imaging unit 11 captures various images around the terminal device 10. The imaging unit 11, for example, captures the display device 20. The imaging unit 11 includes, for example, an imaging element not shown and a circuit that generates video data based on the output of the imaging element. Examples of the imaging element include, but are not limited to, a CMOS (Complementary Metal Oxide Semiconductor) image sensor and a CCD (Charge Coupled Device).

[0016] The display unit 12 displays various information. The display unit 12, for example, displays various images. The display unit 12, for example, displays an identifier for calculating the relative position between the terminal device 10 and the display device 20. The identifier will be described later. The display unit 12 is, for example, composed of a display including a liquid crystal display (LCD) or an organic EL (Organic Electro-Luminescence) display.

[0017] The audio output unit 13 outputs various sounds. The audio output unit 13 can be realized, for example, by a speaker.

[0018] The operation unit 14 accepts various operations from the user for the terminal device 10. The operation unit 14 is composed of, for example, buttons, switches, and a touch panel. The operation unit 14 accepts, for example, operations to start or end communication with the display device 20. The operation unit 14 accepts, for example, an operation to capture and acquire the image displayed on the display device 20, and an operation to store the captured image in the storage unit 15.

[0019] The memory unit 15 stores various types of information. For example, the memory unit 15 stores video data of various objects such as scenery, buildings, and characters in the VR space displayed on the display device 20. The memory unit 15 may store video data rendered as two-dimensional images, or it may store three-dimensional objects and information that identifies the shooting range, or only information that identifies the shooting range. The memory unit 15 can be implemented as, for example, a semiconductor memory element such as RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or solid-state drive.

[0020] The communication unit 16 performs communication between the terminal device 10 and an external device. For example, the communication unit 16 performs communication with the display device 20.

[0021] The control unit 17 controls the operation of each part of the terminal device 10. The control unit 17 is implemented, for example, by a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing a program (for example, the program according to the present invention) stored in a memory unit (not shown) using RAM or the like as the working area. The control unit 17 may also be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). The control unit 17 may also be implemented by a combination of hardware and software.

[0022] The control unit 17 comprises a calculation unit 171, a video data acquisition unit 172, a display control unit 173, an output control unit 174, an operation control unit 175, an imaging control unit 176, a storage control unit 177, and a communication control unit 178.

[0023] The calculation unit 171 calculates the relative position between the terminal device 10 and the display device 20. The calculation unit 171 calculates the relative distance between the terminal device 10 and the display device 20 based on, for example, the size of the identifier displayed on the display unit 12, which has been captured by an external imaging device and acquired by the video data acquisition unit 172. The calculation unit 171 may calculate the relative distance between the terminal device 10 and the display device 20 based on, for example, a table (not shown) that associates the size of the identifier included in the captured video with the relative distance between the terminal device 10 and the display device 20. The calculation unit 171 calculates the relative inclination of the terminal device 10 with respect to the display device 20 based on, for example, the degree and direction of distortion of the identifier displayed on the display unit 12. The calculation unit 171 calculates the relative position between the terminal device 10 and the display device 20 based on, for example, the relative distance between the terminal device 10 and the display device 20 and the relative inclination of the terminal device 10 with respect to the display device 20.

[0024] Figure 3 is a diagram illustrating the identifier displayed on the display unit 12. As shown in Figure 3, the calculation unit 171 calculates the relative distance between the terminal device 10 and the display device 20 based on the size of the identifier 111 displayed on the display unit 12, for example, as captured by the imaging unit 21 of the display device 20. The calculation unit 171 calculates the relative tilt of the terminal device 10 with respect to the display device 20 based on the degree and direction of distortion of the identifier 111 displayed on the display unit 12, for example, as captured by the imaging unit 21 of the display device 20. The calculation unit 171 calculates the relative position between the terminal device 10 and the display device 20 based on the relative distance between the terminal device 10 and the display device 20 and the relative tilt of the terminal device 10 with respect to the display device 20. The calculation unit 171 may also calculate the relative position between the terminal device 10 and the display device 20 based on the size of the identifier 111 captured by another external camera, for example. The identifier 111 is displayed, for example, at the four corners of the display unit 12. Identifier 111 is, for example, a QR code (registered trademark), but is not limited to this. Identifier 111 may be displayed as a single instance in the central part of the display unit 12.

[0025] Furthermore, the calculation unit 171 may calculate the relative position between the terminal device 10 and the display device 20 based on the image data of the display device 20 captured by the imaging unit 11. The calculation unit 171 may, for example, determine the size of the display device 20 included in the image data using a well-known image recognition process, and calculate the relative position between the terminal device 10 and the display device 20 based on the determined size. In this case, the calculation unit 171 may calculate the relative distance between the terminal device 10 and the display device 20 based on, for example, a table that associates the size of the display device 20 with the relative distance between the terminal device 10 and the display device 20. The calculation unit 171 may, for example, calculate the relative tilt of the terminal device 10 with respect to the display device 20 based on the degree and direction of distortion of the display device 20.

[0026] Furthermore, the calculation unit 171 may calculate the relative position between the terminal device 10 and the display device 20 based on the captured data of an identifier, such as a QR code, provided on the housing of the display device 20, which is captured by the imaging unit 11. The calculation unit 171 may, for example, determine the size and distortion of the identifier included in the captured data using a well-known image recognition process, and based on the determined size, calculate the relative distance between the terminal device 10 and the display device 20, and the relative inclination of the terminal device 10 with respect to the display device 20. For example, the calculation unit 171 can calculate the relative position between the terminal device 10 and the display device 20 by determining the relative distance between the terminal device 10 and the display device 20, and the relative inclination of the terminal device 10 with respect to the display device 20, based on the determined size and distortion of the identifier. In this case, the calculation unit 171 can calculate the relative distance between the terminal device 10 and the display device 20 based on, for example, a table that associates the size of the identifier with the relative distance between the terminal device 10 and the display device 20. The calculation unit 171 can calculate the relative tilt of the terminal device 10 with respect to the display device 20 based, for example, on the degree and direction of distortion of the identifier.

[0027] The video data acquisition unit 172 acquires various types of video data. For example, the video data acquisition unit 172 acquires video data captured by a virtual terminal device in the VR space displayed on the display device 20 from the display device 20. For example, when the operation control unit 175 receives an operation signal from the operation unit 14 to capture video in the VR space displayed on the display device 20, the video data acquisition unit 172 acquires video data captured by a virtual terminal device in the VR space displayed on the display device 20. Capture by a virtual terminal device in the VR space refers to generating video by capturing a predetermined range of the VR space from the position of a virtual camera provided on the virtual terminal device in the VR space. The virtual terminal device in the VR space may have the same shape and size as the terminal device 10, or it may be different. The position of the virtual camera in the virtual terminal device may be the same as the position of the imaging unit 11 in the terminal device 10, or it may be different, if the terminal device 10 is equipped with an imaging unit 11. The characteristics of the virtual camera may match or differ from those of the imaging unit 11 if the terminal device 10 is equipped with an imaging unit 11. The characteristics of the virtual camera may be changed as appropriate by the user through operations in the VR space, such as wide-angle or telephoto (zoom in, zoom out). The user may be able to perform operations in the VR space by displaying an operation screen on the display unit of the virtual terminal device.

[0028] The display control unit 173 causes the display unit 12 to display various types of images. The display control unit 173, for example, causes the display unit 12 to display an identifier for calculating the relative position between the terminal device 10 and the display device 20. The display control unit 173 displays images on the display unit 12 related to the video data acquired by the video data acquisition unit 172 from the display device 20, which were captured in the VR space.

[0029] The display control unit 173 controls the shooting range displayed on the display unit of a virtual terminal device in a VR space, for example, when the virtual terminal device in the VR space displayed on the display device 20 takes pictures of various objects in the VR space. The display control unit 173 controls the shooting range displayed on the display unit of the virtual terminal device in the VR space based on the relative position between the real terminal device 10 and the display device 20. Specifically, the display control unit 173 controls the shooting range displayed on the display unit of the virtual terminal device in the VR space to change in response to changes in the relative position between the real terminal device 10 and the display device 20. The display control unit 173 outputs a control signal to the display device 20 via the communication unit 16 to change the shooting range displayed on the virtual terminal device in the VR space.

[0030] The display control unit 173 generates, for example, an avatar of the user using the terminal device 10 and the display device 20 to be displayed on the image shown on the display device 20. The display control unit 173 generates the avatar based, for example, on the facial image data of the user using the terminal device 10 and the display device 20. The display control unit 173 may also change each part that makes up the avatar's face based on operation information received from the user by the operation unit 14. The display control unit 173 outputs a control signal to the display device 20 via the communication unit 16 to display the generated avatar on the display device 20.

[0031] The output control unit 174 controls the audio output unit 13 to output audio.

[0032] The operation control unit 175 acquires operation signals related to operations input to the operation unit 14. The operation control unit 175 controls the operation of the terminal device 10 by outputting control signals related to the acquired operation signals. The operation control unit 175 acquires operation signals related to operations for capturing images in the VR space displayed on the display device 20, which are input to the operation unit 14.

[0033] The imaging control unit 176 controls the imaging unit 11. The imaging control unit 176 sets imaging conditions for the imaging unit 11 and causes the imaging unit 11 to perform imaging. For example, the imaging control unit 176 controls the imaging unit 11 to take an image of an identifier such as a QR code provided on the housing. Note that if the display unit 12 of the terminal device 10 displays an identifier such as a QR code, the imaging unit 11 and the imaging control unit 176 are not necessary.

[0034] The memory control unit 177 stores various types of data in the storage unit 15. For example, the memory control unit 177 stores video data related to the video captured in the VR space displayed on the display device 20, which is acquired by the video data acquisition unit 172, in the storage unit 15.

[0035] The communication control unit 178 controls the communication unit 16 to control communication between the terminal device 10 and an external device. For example, the communication control unit 178 controls the communication unit 16 to control communication between the terminal device 10 and the display device 20. [Display device] The configuration of the display device according to the embodiment will be explained using Figure 4. Figure 4 is a block diagram showing an example of the configuration of the display device according to the embodiment.

[0036] As shown in Figure 4, the display device 20 comprises an imaging unit 21, a display unit 22, an audio output unit 23, an operation unit 24, and a communication unit 25. The imaging unit 21, the display unit 22, the audio output unit 23, the operation unit 24, and the communication unit 25 are each connected by bus B3.

[0037] In this embodiment, the display device 20 is described as a device including an HMD worn on the user's head, but is not limited to this. The display device 20 may be a display device consisting of a liquid crystal display or an organic EL display, etc., which is installed on a desk in a room, for example.

[0038] The imaging unit 21 captures various images of the surrounding area of ​​the display device 20. For example, the imaging unit 21 captures the identifier 111 displayed on the display unit 12 of the terminal device 10. The imaging unit 21 includes an image sensor (not shown) and a circuit that generates video data based on the output of the image sensor. Examples of image sensors include, but are not limited to, CMOS image sensors and CCDs.

[0039] The display unit 22 displays various types of images. For example, the display unit 22 displays images in a VR space. The display unit 22 is implemented, for example, by an HMD (Head-Mounted Display).

[0040] The audio output unit 23 outputs various types of audio. The audio output unit 23 can be implemented, for example, by a speaker.

[0041] The operation unit 24 receives various operations from the user to the display device 20. The operation unit 24 is composed of, for example, buttons and switches. The operation unit 24 receives operations to start and end communication with the terminal device 10, for example.

[0042] The communication unit 25 performs communication between the display device 20 and external devices. The communication unit 25 performs communication with, for example, the terminal device 10. The communication unit 25 performs communication with, for example, the server 30.

[0043] The control unit 26 controls the operation of each part of the display device 20. The control unit 26 is implemented, for example, by a CPU or MPU, which executes a program stored in a memory unit (not shown) using RAM or the like as a working area. The control unit 26 may also be implemented by an integrated circuit such as an ASIC or FPGA. The control unit 26 may also be implemented by a combination of hardware and software.

[0044] The control unit 26 comprises an acquisition unit 261, a display control unit 262, an output control unit 263, an operation control unit 264, an imaging control unit 265, and a communication control unit 266. The acquisition unit 261, the display control unit 262, the output control unit 263, the operation control unit 264, the imaging control unit 265, and the communication control unit 266 are each connected by bus B4.

[0045] The acquisition unit 261 acquires various types of information. For example, the acquisition unit 261 acquires video data relating to the image of identifier 111 displayed on the display unit 12 of the terminal device 10, which has been captured by the imaging unit 21.

[0046] The display control unit 262 controls the video displayed on the display unit 22. For example, the display control unit 262 displays VR-related video on the display unit 22. For example, the display control unit 262 displays video on the display unit 22 that allows multiple users to experience a theme park or the like in a VR space. For example, the display control unit 262 changes the display range of the video displayed on the display unit 22 or displays an avatar on the video displayed on the display unit 22 according to a control signal received from the terminal device 10. For example, the display control unit 262 changes the shooting range displayed on the virtual terminal device in the VR space displayed on the display unit 22 according to a control signal received from the terminal device 10. The control signals that the display control unit 262 receives from the terminal device 10 are, for example, control signals generated based on an operation performed by a user on the operation unit 14 of the terminal device 10, on the operation screen displayed on the display unit of the virtual terminal device in the VR space. When the identifier 111 is displayed on the display unit 12 of the terminal device 10, the user is actually operating the terminal device 10 they are holding, but the identifier 111 is displayed on the display unit 12 and the operation screen is not displayed. On the other hand, the control signals generated based on the operation are transmitted from the terminal device 10 to the display device 20, and the display control unit 262 controls the operation screen displayed on the display unit of a virtual terminal device in the VR space corresponding to the terminal device 10 based on the control signals. The user perceives the behavior of the virtual operation screen visually from the display device 20 and perceives the actual operation through touch from the terminal device 10. The terminal device 10 may also generate vibrations using an actuator (not shown) to provide feedback to the user regarding the operation.

[0047] The output control unit 263 outputs various types of audio from the audio output unit 23. For example, the output control unit 263 outputs the audio of other users in the VR space. For example, the output control unit 263 causes the audio of other users experiencing a theme park together in the VR space to be output from the audio output unit 23.

[0048] The operation control unit 264 acquires operation signals related to operations input to the operation unit 24. The operation control unit 264, for example, outputs control signals related to the acquired operation signals to control the operation of the display device 20.

[0049] The imaging control unit 265 controls the imaging unit 21. The imaging control unit 265 sets imaging conditions for the imaging unit 21 and causes the imaging unit 21 to perform imaging.

[0050] The communication control unit 266 controls the communication unit 25 to control communication between the display device 20 and an external device. For example, the communication control unit 266 controls the communication unit 25 to control communication between the terminal device 10 and the display device 20. For example, the communication control unit 266 controls the communication unit 25 to control communication between the display device 20 and the server 30. [server] The server configuration according to this embodiment will be explained using Figure 5. Figure 5 is a block diagram showing an example of the server configuration according to this embodiment.

[0051] As shown in Figure 5, the server 30 comprises a communication unit 31, a storage unit 32, and a control unit 33.

[0052] The communication unit 31 performs communication between the server 30 and external devices. The communication unit 31 performs communication with, for example, the terminal device 10. The communication unit 31 performs communication with, for example, the display device 20.

[0053] The memory unit 32 stores various types of information. For example, the memory unit 32 stores map information in the VR space. The map information includes various types of information in the VR space, such as scenery, buildings, and characters. The memory unit 32 can be implemented using semiconductor memory elements such as RAM and flash memory, or storage devices such as hard disks and solid-state drives.

[0054] The control unit 33 controls the operation of each part of the server 30. The control unit 33 is implemented, for example, by a CPU or MPU, which executes a program stored in a memory unit (not shown) using RAM or the like as a working area. The control unit 33 may also be implemented by an integrated circuit such as an ASIC or FPGA. The control unit 33 may also be implemented by a combination of hardware and software.

[0055] The control unit 33 generates a VR space based on the map information stored in the memory unit 32. The control unit 33 generates a VR space in which users can experience, for example, a virtual theme park. By connecting the display device 20 and the server 30, users of the display device 20 can, for example, experience a virtual theme park using avatars together with multiple other users. [How to use terminal devices and display devices] The method of using the terminal device and display device according to the embodiment will be explained using Figures 6 and 7. Figures 6 and 7 are diagrams for explaining the method of using the terminal device and display device according to the embodiment.

[0056] As shown in Figure 6, the display device 20 is used, for example, by being worn on the head of user U. In other words, user U can use the display device 20 to experience a virtual theme park in VR space. For example, while using the display device 20, user U can operate the terminal device 10 to take pictures of scenery, buildings, characters, and other objects as memories of the theme park in VR space, or take pictures with multiple other users.

[0057] Figure 7 shows the captured data in the VR space. User U can, for example, operate the terminal device 10 to capture an image in the VR space that includes User U's avatar A1 and the avatars A2, A3, and A4 of other users using a virtual terminal device in the VR space. The captured image may be a still image or a moving image. The display range of the captured image changes depending on the relative position between the terminal device 10 and the display device 20. In other words, User U can change the capture range in the VR space by adjusting the relative position of the terminal device 10 and the display device 20 in real space. The capture range in the VR space is displayed on the display unit of the virtual terminal device in the VR space. By adjusting the relative position of the terminal device 10 and the display device 20 in real space, the capture range displayed on the display unit of the virtual terminal device can be adjusted. Specifically, user U can adjust the shooting range so that avatars A1, A2, A3, and A4 are all captured in a single photograph in the VR space by adjusting the relative positions of the terminal device 10 and the display device 20 in real space. [Processing of terminal devices] The processing flow of the terminal device according to the embodiment will be explained using Figure 8. Figure 8 is a flowchart showing an example of the processing flow of the terminal device according to the embodiment.

[0058] First, the operation control unit 175 determines, via the operation unit 14, whether the shooting mode of the virtual camera in the VR space is set to the shooting mode for shooting with the rear camera (step S10). If it is determined that the shooting mode for shooting with the rear camera is set (step S10; Yes), the process proceeds to step S12. If it is determined that the shooting mode for shooting with the rear camera is not set (step S10; No), the process proceeds to step S11.

[0059] If the result in step S10 is No, the operation control unit 175 determines via the operation unit 14 whether the shooting mode of the virtual camera in the VR space is set to the shooting mode for shooting with the front camera (step S11). If it is determined that the shooting mode for shooting with the front camera is set (step S11; Yes), the process proceeds to step S12. If it is determined that the shooting mode for shooting with the front camera is not set (step S11; No), the process proceeds to step S18.

[0060] If the answer is Yes in step S10 and Yes in step S11, the calculation unit 171 calculates the relative position between the terminal device 10 and the display device 20 (step S12). Specifically, the calculation unit 171 calculates the relative position between the terminal device 10 and the display device 20 based on the size of the identifier 111 displayed on the display unit 12 of the terminal device 10, which is captured by the imaging unit 21 of the display device 20. Then, the process proceeds to step S12.

[0061] The display control unit 173 recognizes the shooting range in the VR space displayed on the display device 20 based on the relative position calculated by the calculation unit 171 (step S13). Specifically, the display control unit 173 recognizes the shooting range displayed on the display unit of the virtual terminal device in the VR space. Then, the process proceeds to step S14.

[0062] The operation control unit 175 determines, via the operation unit 14, whether or not it has received an operation to photograph the shooting range in the VR space recognized in step S13 (step S14). If it is determined that an operation to photograph the shooting range in the VR space has been received (step S14; Yes), the process proceeds to step S15. If it is determined that an operation to photograph the shooting range in the VR space has not been received (step S14; No), the process proceeds to step S18.

[0063] If the result in step S14 is determined to be Yes, the video data acquisition unit 172 acquires shooting data related to the shooting range in the captured VR space from the display device 20 (step S15). Here, the display control unit 173 may display the video related to the shooting data in the VR space acquired by the video data acquisition unit 172 on the display unit 12. The shooting data may be a still image or a moving image. This allows the user to understand the video captured in the VR space. Then, the process proceeds to step S16.

[0064] The operation control unit 175 determines, via the operation unit 14, whether or not it has received an operation to save the image data acquired in step S15 (step S16). If it is determined that an operation to save the image data has been received (step S16; Yes), the process proceeds to step S17. If it is determined that an operation to save the image data has not been received (step S16; No), the process proceeds to step S18.

[0065] If the result in step S16 is determined to be Yes, the memory control unit 177 saves the image data acquired in step S15 to the memory unit 15 (step S17). Then, the process proceeds to step S18.

[0066] The control unit 17 determines whether or not to terminate the process (step S18). Specifically, the operation control unit 175 determines to terminate the process when it receives an operation to terminate shooting or an operation to turn off the power of the terminal device 10. If it is determined to terminate the process (step S18; Yes), the process shown in Figure 8 is terminated. If it is determined not to terminate the process (step S18; No), the process proceeds to step S10.

[0067] As described above, in this embodiment, the shooting range in the VR space displayed on the display device 20 is captured by operation using the terminal device 10, and the captured video data is saved in the terminal device 10. As a result, in this embodiment, memories of scenery in the VR space or virtual theme parks can be recorded in the terminal device 10, making it easy to share photos taken in the VR space on SNS (Social Network Service).

[0068] This embodiment allows users to experience a virtual theme park and communicate with other users using avatars in a VR space. Furthermore, this embodiment allows users to take group photos with their own avatar and other users' avatars in the same VR space. [First variation] A first modified example of this embodiment will be described using Figure 9. Figure 9 is a block diagram showing an example of the configuration of a terminal device according to the first modified example.

[0069] As shown in Figure 9, terminal device 10A differs from terminal device 10 shown in Figure 2 in that the control unit 17A includes an attitude detection unit 179.

[0070] The posture detection unit 179 detects the posture of the user experiencing VR using the display device 20. The posture detection unit 179 detects the posture of each part of the user, including the head, arms, and legs. In the first modified example, for example, the video data acquisition unit 172 acquires image data of the user from an external (not shown) imaging device. In this case, the posture detection unit 179 only needs to detect the user's posture based on the image data acquired by the video data acquisition unit 172. The posture detection unit 179 may also detect the user's posture using motion capture or other well-known techniques.

[0071] The display control unit 173 changes the posture of the user's avatar displayed on the display device 20 based on the detection result of the posture detection unit 179. For example, if the posture detection unit 179 detects that the user is raising their right hand, the display control unit 173 raises the right hand of the user's avatar. In other words, the display control unit 173 changes the posture of the avatar to match the user's posture detected by the posture detection unit 179.

[0072] As described above, the first modification of this embodiment changes the posture of the avatar in the VR space in real time to match the posture of the user in the real space. As a result, the first modification of the present invention improves usability because the posture of the avatar in the VR space can be easily changed. [Second variation] A second modified example of this embodiment will be described using Figure 10. Figure 10 is a block diagram showing an example of the configuration of a terminal device according to the second modified example.

[0073] As shown in Figure 10, terminal device 10B differs from terminal device 10 shown in Figure 2 in that it is equipped with a sensor 18.

[0074] Sensor 18 includes various types of sensors. Sensor 18 includes, for example, a sensor that detects the relative position between the terminal device 10B and the display device 20. Examples of sensors 18 include, but are not limited to, a laser radar (e.g., LIDAR: Laser Imaging Detection and Ranging), an infrared sensor including an infrared irradiator and a light receiving sensor, and a ToF (Time of Flight) sensor.

[0075] The calculation unit 171 of the control unit 17B calculates the relative position between the terminal device 10B and the display device 20 based on the detection result from the sensor 18. In other words, in the second modified example, the calculation unit 171 calculates the relative position between the terminal device 10B and the display device 20 using a spatial understanding means different from that of the imaging unit 11.

[0076] As described above, in the second modified example of this embodiment, the relative position between the terminal device 10B and the display device 20 is calculated based on the detection results of various sensors. This allows for increased design flexibility in the second modified example. [Third variation] A third modified example of this embodiment will be described using Figure 11. Figure 11 is a block diagram showing an example of the configuration of a display device according to the third modified example.

[0077] As shown in Figure 11, the display device 20A differs from the display device 20 shown in Figure 4 in that it includes a sensor 27 and the control unit 26A includes a calculation unit 267.

[0078] Sensor 27 includes various types of sensors. Sensor 27 includes, for example, a sensor that detects the relative position between the terminal device 10B and the display device 20. Examples of sensors 27 include, but are not limited to, a laser radar (e.g., LIDAR), an infrared sensor including an infrared irradiator and a light receiving sensor, and a ToF sensor.

[0079] The calculation unit 267 calculates the relative position between the terminal device 10 and the display device 20A based on the detection results from the sensor 27. In other words, the relative position between the terminal device 10 and the display device 20A may be calculated by the display device 20A rather than the terminal device 10. In this case, the display device 20A provides the terminal device 10 with information regarding the calculated relative position between the terminal device 10 and the display device 20A via the communication unit 25. The display control unit 173 then changes the shooting range displayed on the display unit of the virtual terminal device in the VR space displayed on the display device 20A, for example, based on the relative position between the terminal device 10 and the display device 20A calculated by the calculation unit 267.

[0080] As described above, in the third modified example of this embodiment, the relative position between the terminal device 10 and the display device 20A is calculated by the display device 20A based on the sensor detection results. This improves the design flexibility in the third modified example.

[0081] Although embodiments of the present invention have been described above, the present invention is not limited by the content of these embodiments. Furthermore, the aforementioned components include those that can be easily conceived by those skilled in the art, those that are substantially the same, and those that fall within the so-called equivalent range. Moreover, the aforementioned components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the embodiments described above. [Explanation of symbols]

[0082] 1. Information Processing System 10, 10A, 10B Terminal Devices 11,21 Imaging Unit 12,22 Display section 13,23 Audio output section 14,24 Operation section 15,32 Storage part 16, 25, 31 Communications Department 17, 17A, 17B, 26, 26A, 33 Control Unit 18,27 sensors 20 Display device 30 servers 171,267 Calculation Unit 172 Video data acquisition unit 173,262 Display Control Unit 174,263 Output control unit 175,264 Operation Control Unit 176,265 Imaging control unit 177 Memory Control Unit 178,266 Communication Control Unit 179 Posture detection unit 261 Acquisition Department

Claims

1. A display device that displays virtual reality used by the user, and a calculation unit that calculates the relative position between the display device and the terminal device actually held by the user, A display control unit that changes the shooting range displayed on the virtual terminal device in the virtual reality space shown on the display device based on the relative position, A video data acquisition unit acquires video data relating to the video in the virtual reality space that has been modified by the display control unit, A display unit that displays information, Equipped with, The display control unit causes the display unit of the virtual terminal device to display the video and operation screen related to the video data acquired by the video data acquisition unit. The terminal device is characterized in that the display unit of the terminal device does not display the operation screen.

2. A step of calculating the relative position between a display device that shows virtual reality used by a user and a terminal device that the user is actually holding, The steps include changing the shooting range displayed on the virtual terminal device in the virtual reality space shown on the display device based on the relative position, The steps include acquiring video data relating to the image in the space of the modified shooting range in the virtual reality space, The steps include: displaying the video and operation screen related to the acquired video data on the display unit of the virtual terminal device; The steps include: displaying information other than the aforementioned operation screen on the display unit of the terminal device; Methods that include...

3. A step of calculating the relative position between a display device that shows virtual reality used by a user and a terminal device that the user is actually holding, The steps include changing the shooting range displayed on the virtual terminal device in the virtual reality space shown on the display device based on the relative position, The steps include acquiring video data relating to the image in the space of the modified shooting range in the virtual reality space, The steps include: displaying the video and operation screen related to the acquired video data on the display unit of the virtual terminal device; The steps include: displaying information other than the aforementioned operation screen on the display unit of the terminal device; A program that causes a computer to execute something.

Images