Display control device, display control method, and display control program

Abstract

A display control device according to an embodiment of the present disclosure comprises: an acquisition unit for acquiring the position of a user in a virtual space, and content arrangement information relating to the arrangement of content provided in the virtual space, the content arrangement information including information relating to a first position set for viewing of the content and a first region associated with the first position; a determination unit for determining, on the basis of the acquired information, whether the user is positioned within the first region; and an adjustment unit for adjusting the relative position between the content and the user within the virtual space on the basis of the result of the determination.

Description

Display control device, display control method, and display control program

[0001] The present disclosure relates to a display control device, a display control method, and a display control program for controlling display processing in virtual reality.

[0002] Various technologies have been put to practical use for presenting users with three-dimensional virtual spaces (hereinafter simply referred to as "virtual spaces") constructed within computers or computer networks, such as those called metaverses. For example, active research and development is being conducted on the application of technologies related to VR (Virtual Reality) and AR (Augmented Reality) to the expression of virtual spaces.

[0003] As a technology related to visual representation of a virtual space, there is known a technology that improves user convenience by avoiding inconveniences associated with switching viewpoints when a user uses a device such as an HMD (Head Mounted Display) (for example, Patent Document 1).

[0004] Japanese Patent Application Laid-Open No. 2019-149122

[0005] However, there is still room for improvement in technologies that can improve user convenience and user experience in virtual spaces.

[0006] An advantage of a virtual space constructed in three dimensions is that a user can move freely within the virtual space. However, if a user moves freely within the virtual space, it becomes impossible for a content provider to predict where the user will move, and the content provider may not be able to provide the user with the video experience that the content provider desires. For example, if content such as live-action video is provided at a certain point within the virtual space, the user may not be able to view the video while maintaining an appropriate angle of view and distance, which may cause the user to feel uncomfortable.

[0007] Therefore, the present disclosure proposes a display control device, a display control method, and a display control program that can provide users with an experience intended by a provider in a virtual space.

[0008] In order to solve the above problem, one form of display control device according to the present disclosure includes an acquisition unit that acquires content placement information relating to a user's position in a virtual space and information regarding the placement of content provided in the virtual space, the content placement information including information regarding a first position set for viewing the content and a first area associated with the first position; a determination unit that determines whether the user is located within the first area based on the acquired information; and an adjustment unit that adjusts the relative position of the content and the user in the virtual space based on the result of the determination.

[0009] FIG. 1 is a diagram illustrating an overview of a display control system according to an embodiment. FIG. 1 is a diagram illustrating a virtual space according to an embodiment. FIG. 2 is a diagram illustrating a virtual space according to an embodiment. FIG. 3 is a diagram illustrating coordinate axes of the virtual space. FIG. 1 is a diagram illustrating a positional relationship in the virtual space. FIG. 2 is a diagram illustrating a positional relationship in the virtual space. FIG. 1 is a diagram illustrating an adjustment process according to an embodiment. FIG. 2 is a diagram illustrating an adjustment process according to an embodiment. A diagram illustrating an example of the configuration of a display control device according to an embodiment. A diagram illustrating an example of a data table related to a live-action video content storage unit. A diagram illustrating an example of a data table related to an audio content storage unit. A flowchart (1) illustrating the flow of display control processing according to an embodiment. A flowchart (2) illustrating the flow of display control processing according to an embodiment. A flowchart (3) illustrating the flow of display control processing according to an embodiment. A hardware configuration diagram illustrating an example of a computer that realizes the functions of a display control device.

[0010] Hereinafter, embodiments will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0011] The present disclosure will be described in the following order: 1. Embodiment 1-1. Overview of display control system according to embodiment 1-2. Overview of display control process according to embodiment 1-3. Configuration of display control device according to embodiment 1-4. Processing procedure according to embodiment 1-5. Modification 2. Other embodiments 3. Effects of display control device according to the present disclosure 4. Hardware configuration

[0012] (1. Embodiment) (1-1. Overview of Display Control System According to Embodiment) An example of a display control system according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an overview of a display control system 1 according to an embodiment.

[0013] A display control system 1 according to the embodiment includes an HMD 20 and a display control device 100. The display control system 1 is a so-called VR system that constructs a virtual space and reflects various actions of a user 10 wearing the HMD 20 in the virtual space.

[0014] The display control device 100 is, for example, an information processing terminal for implementing VR or AR technology. In the embodiment, the display control device 100 displays a virtual space constructed within a computer or network, such as a so-called metaverse, on a display unit (such as a display) such as an HMD 20. The display control device 100 also displays a user 10 (such as an avatar, which is a character that resembles the user) in the virtual space. The display control device 100 is realized by any information processing device, such as a personal computer (PC) or a cloud server.

[0015] The HMD 20 is an information processing device that allows the user 10 to experience the virtual space constructed by the display control device 100 and that handles input for reflecting the actions of the user 10 in the virtual space. For example, the HMD 20 has various functions for the user to use the virtual space, such as a self-posture estimation function, various motion sensors (acceleration sensor, gyro sensor, etc.), and a three-dimensional image display function.

[0016] The display control system 1 can be modified and expanded in various ways. For example, the user 10 may use a predetermined input device (such as a controller) that cooperates with the HMD 20 to input information into the virtual space. The display control device 100 may display the virtual space on a smartphone, tablet, or other device, rather than a display device with a dual-screen monitor capable of binocular stereoscopic vision, such as the HMD 20. The display control system 1 may use a 3D graphics engine on the cloud or a graphics engine built into the display control device 100 to display the virtual space. The display control system 1 may store data related to the virtual space in cloud storage or in a local device such as the display control device 100. The display control system 1 may also have a content execution function, such as a video content execution function, a game execution function, or the like, using a predetermined game engine.

[0017] The display control system 1 shown in Figure 1 allows a user 10 to move freely around the virtual space, interact with other users in the virtual space, watch content such as videos provided in the virtual space, and play games.

[0018] However, when a user 10 experiences a virtual space, compared to a real space (hereinafter referred to as "real space"), the user 10 may feel nauseous or have an uncomfortable experience due to sudden movements of the viewpoint and sound source. In particular, when content such as video is provided in a virtual space, the video and sound sources are located at predetermined positions in the virtual space, but the user 10 can move freely within the virtual space, so the user 10 may not be located in the position intended by the provider for optimal viewing. For example, when two-dimensional video content is provided at a certain point in the virtual space, unless the user views the video from an appropriate angle, the video may be distorted or the user may not understand what video is being displayed, resulting in a failure to provide the intended experience. Furthermore, when a sound source corresponding to the content is located at a certain point in the virtual space, the sound localization may suddenly change when the user moves beyond the sound source, causing the user to feel uncomfortable.

[0019] That is, in a virtual space where the user can move freely, it is impossible to predict where the user 10 will move, and therefore the content provider may not be able to provide the user 10 with the video experience that they desire.

[0020] Therefore, the display control device 100 according to the present disclosure achieves the above processing by the following configuration. First, the display control device 100 acquires information regarding the position of the user 10 in a virtual space and the placement of content provided in the virtual space (hereinafter referred to as "content placement information"). The content placement information includes information regarding a position set for viewing the content (hereinafter referred to as "optimal viewing position"). The optimal viewing position is a position in the virtual space that is preset by the content provider as the most suitable position for viewing the content. For example, if the content is two-dimensional live-action video content, the optimal viewing position is a position that is at an angle directly facing the content and a predetermined distance away from the content so that the image is displayed at the maximum angle of view relative to the user 10's field of view. Note that the optimal viewing position may be changed as appropriate depending on the performance (e.g., display angle of view) of the device, such as the HMD 20, used by the user 10. The content placement information also includes information regarding an area associated with the optimal viewing position (hereinafter referred to as "viewing area"). The viewing area is, for example, an area within a predetermined distance range centered on the optimum viewing position, and may have any shape.

[0021] The display control device 100 determines whether the user 10 is located within the viewing area based on the position of the user 10 and the content placement information. If the user 10 is located within the viewing area, the display control device 100 adjusts the relative position between the user 10 and the content. Specifically, the display control device 100 automatically adjusts the position of the user 10 to approach the optimal viewing position so that the user 10 can view the content in the optimal position. This allows the user 10 to move to the content viewing position without feeling uncomfortable, allowing the user 10 to enjoy the content without feeling sick or unnatural. In other words, the display control device 100 can naturally guide the user 10 to a position and angle of view where the video can be viewed directly in front of them, thereby providing the user with an experience in the virtual space that is exactly as intended by the provider.

[0022] (1-2. Overview of Display Control Processing According to Embodiment) The above display control processing will be described using FIG. 2 and subsequent figures. First, the flow of processing for constructing a virtual space in which content is placed at a predetermined location will be described using FIG. 2 and FIG. 3. FIG. 2 is a diagram (1) for explaining a virtual space according to an embodiment.

[0023] 2 shows a virtual space 30 that mimics a real-world concert hall. In the virtual space 30, the user 10 can experience content created by filming and recording a music concert held in the real-world concert hall.

[0024] In the example of FIG. 2 , a music concert held in advance in real space is filmed and recorded from multiple positions, and the filming and recording locations correspond to, for example, positions 40, 41, 42, 43, 44, 45, and 46 shown in FIG. 2 . Based on the data obtained from the filming and recording, a content provider creates content (a group of video and audio data) for the music concert. Specifically, the content provider installs cameras and microphones at positions 40 and the like and records the music concert. Note that positions 40 to 46 are merely examples, and filming and recording may be performed in real space at positions other than those shown at positions 40 to 46. Furthermore, the recorded data may be two-dimensional video captured with a normal camera, spherical video captured with a camera capable of capturing a 360-degree space, or stereo video captured with a twin-lens camera. Furthermore, the recorded data does not necessarily have to include video; it may be audio data only, picked up by microphones installed at predetermined positions.

[0025] Next, Fig. 3 shows how content is provided in the virtual space 30 based on data obtained from the real space. Fig. 3 is a diagram (2) for explaining the virtual space according to the embodiment.

[0026] FIG. 3 shows a state in which multiple contents are arranged in virtual space 30 corresponding to each viewpoint directed toward position 50 in the center of the stage. For example, at position 51 facing the audience seats from behind the stage, content 51A, which is an image corresponding to the line of sight, is arranged. Similarly, at position 52 facing the audience seats from diagonally behind the stage, content 52A, which is an image corresponding to the line of sight, is arranged. Similarly, at position 53 facing the stage from near the rear of the stage, content 53A, which is an image corresponding to the line of sight, is arranged. Similarly, at position 54 facing the stage from the right side of the audience seats, content 54A, which is an image corresponding to the line of sight, is arranged. Similarly, at position 55 facing the stage from the center of the audience seats, content 55A, which is an image corresponding to the line of sight, is arranged. Similarly, at position 56 facing the stage from the rear of the audience seats, content 56A, which is an image corresponding to the line of sight, is arranged. Similarly, at position 57 facing the stage from the very back of the audience seats, content 57A, which is an image corresponding to the line of sight, is arranged. Although not shown in the drawings, spherical content captured by a spherical camera may be placed at position 50.

[0027] For example, when user 10 moves near content 51A in virtual space 30, user 10 can see that content 51A is displayed as a two-dimensional live-action image. This allows user 10 to experience a sense of realism as if a music concert were actually being held in virtual space 30. Similarly, when user 10 moves near content 52A, user 10 can view content 52A as a two-dimensional live-action image, allowing user 10 to experience the music concert from a different angle than content 51A. Furthermore, when user 10 moves near position 50, user 10 can view spherical content in which user 10 stands in the center of the stage and relives the music concert from a 360-degree perspective.

[0028] In this way, the user 10 can view each piece of content at the position where the content is placed. For example, the position of each piece of content in the virtual space 30 is set according to the x, y, and z axes defined in the virtual space. Furthermore, if the content is a two-dimensional image, information indicating the angle of the image (for example, angle information consisting of components in the yaw direction, pitch direction, and roll direction) is set in the content.

[0029] As described above, each content is set with an optimal viewing position (position coordinates and direction) that is a viewing position suitable for each video. Figure 4 shows the relationship between the coordinate axes and the virtual space. Figure 4 is a diagram for explaining the coordinate axes of the virtual space.

[0030] 4, the virtual space 30 is given an x-axis in the horizontal direction of the space, a y-axis in the depth direction of the space, and a z-axis in the vertical direction of the space. For example, when content 60 is placed in the virtual space 30, coordinates on the xyz axes are given to the content 60. The content 60 is, for example, two-dimensional video content such as content 51A shown in FIG.

[0031] Next, the optimum viewing position and viewing area of ​​content will be described with reference to FIG. 5. FIG. 5 is a diagram (1) for explaining the positional relationship in a virtual space. As shown in FIG. 5, when content 60 is placed in a virtual space, an optimum viewing position 62 is determined by a coordinate position (x, y, z) in the virtual space 30 and a direction (θ, ω, ψ) relative to the content. In addition to the optimum viewing position 62, a viewing area 61 is set for the content 60. The viewing area 61 is, for example, a space defined with a predetermined size that includes the optimum viewing position 62.

[0032] In the example of Figure 5, the viewing area 61 is defined by the coordinates of eight endpoints (abcd-efgh), i.e., coordinate data indicating a closed space set to surround the optimal viewing position 62. Note that the size and shape of the viewing area 61 are not limited to the example shown in Figure 5, and any size and shape may be given. Furthermore, the viewing area 61 may not be defined by the content creator or the virtual space provider, but may be set automatically according to predetermined rules. For example, the viewing area 61 may be defined as a space within a predetermined distance from the optimal viewing position 62.

[0033] Next, Fig. 6 shows a plan view (viewed from the z-axis direction) of the viewing area 61 shown in Fig. 5. Fig. 6 is a diagram (2) for explaining the positional relationship in the virtual space. As shown in Fig. 6, the optimal viewing position 62 is a position directly facing the content 60, and the viewing direction is determined by the pitch, yaw, and roll values. Note that, if a position corresponding to the camera installation position (i.e., the shooting position) when the video corresponding to the content 60 was shot is recorded, the optimal viewing position 62 may be automatically set to the shooting position.

[0034] 5 and 6 , when user 10 views content 60 in virtual space 30, it is desirable for the user 10 to view the content 60 from an optimal viewing position 62. Therefore, when user 10 moves near optimal viewing position 62, i.e., into viewing area 61, display control device 100 automatically moves the user's position to optimal viewing position 62. This allows user 10 to view content provided in virtual space 30 from the optimal position intended by the content creator. Furthermore, display control device 100 automatically adjusts the field of view (viewing angle) along with the position of user 10 over a predetermined period of time, thereby preventing abrupt changes in viewpoint and the occurrence of 3D motion sickness, etc.

[0035] The adjustment process performed by the display control device 100 will be described with reference to Fig. 7. Fig. 7 is a diagram (1) for explaining the adjustment process according to the embodiment.

[0036] 7 shows content 60 arranged in virtual space 30, an optimal viewing position 62 corresponding to content 60, a viewing area 61, and a user position 65 which is the position of user 10 in virtual space 30. As shown in Fig. 6, display control device 100 determines whether user position 65 has entered viewing area 61 as a result of the movement of user 10 who moves freely in virtual space 30. For example, display control device 100 performs the determination process for each frame which is the drawing timing of the virtual space.

[0037] Then, when display control device 100 determines that user position 65 is included in viewing area 61 (step S10), it automatically moves user position 65 to optimal viewing position 62 and adsorbs user position 65 to optimal viewing position 62. At this time, display control device 100 also adjusts the viewing direction of user 10 (for example, the orientation of the body of user 10 in the virtual space) so that it coincides with optimal viewing position 62.

[0038] The process shown in Figure 6 can be expressed, for example, by the following equation, where the constants are the adsorption position (optimal viewing position 62) = CamA, the maximum speed of movement = Vmax, and the minimum speed = Vmin, and the variables for each frame are the current position of the user position 65 = Pos and the distance to the adsorption position = Dis.

[0039]

[0040] That is, when the user position 65 and the optimal viewing position 62 are farther apart than the maximum speed of movement, the display control device 100 reduces the distance between the user position 65 and the optimal viewing position 62 according to the maximum speed. Then, when the distance between the user position 65 and the optimal viewing position 62 becomes smaller than the minimum speed of movement, the display control device 100 determines that the user position 65 and the optimal viewing position 62 coincide, and completes adsorption. Furthermore, when automatically moving the user, the display control device 100 uses an algorithm (so-called adsorption) that gradually increases the moving speed as the user approaches the target optimal viewing position 62 and stops when the user arrives at the destination, thereby realizing natural and stress-free movement for the user. Note that adsorption is used as an example of an algorithm for automatic user movement, but automatic movement is not limited to this. For example, the display control device 100 can use any algorithm, such as simple uniform linear motion, as the algorithm for automatic movement.

[0041] Next, the adjustment of the audio position will be described with reference to Fig. 8. Fig. 8 is a diagram (2) for explaining the adjustment process according to the embodiment.

[0042] When audio content recorded simultaneously with the shooting of content or audio-only content is placed in virtual space 30, the audio is given sound source position coordinate data set for each audio along the x, y, and z axes defined in virtual space 30. For example, when multiple microphones are installed during video recording, a sound source position corresponding to the audio is given in virtual space 30 for each position corresponding to the installation position of the microphone in real space. Note that the audio data group may be virtual audio that reproduces audio at a virtual sound source position using three-dimensional audio processing.

[0043] Here, when the user 10 listens to sound from a point sound source while moving freely within the virtual space 30, the sound direction may suddenly change when the user moves near the sound source position, which may cause discomfort or annoyance.

[0044] Therefore, unlike in the case of video, the display control device 100 automatically moves the sound source for audio, rather than automatically moving the user 10. As a result, even if the user 10 moves freely, the audio is localized to the user 10, eliminating the possibility that the user 10 will feel uncomfortable.

[0045] This process will be described with reference to Fig. 8. Fig. 8 shows a user position 65, a sound source 71, and a viewing area 70 that is an area associated with the sound source 71. Note that the sound source 71 and the viewing area 70 may be accompanied by video content (e.g., content 60 shown in Fig. 6).

[0046] The display control device 100 determines whether the user position 65 has entered the viewing area 70 (step S20). If the display control device 100 determines that the user position 65 is included in the viewing area 70, the display control device 100 automatically moves the position of the sound source 71 to coincide with the user position 65.

[0047] Thereafter, the display control device 100 tracks the movement of the user position 65 (step S21). As long as the user position 65 is included in the viewing area 70, the display control device 100 moves the sound source 71 so that it coincides with the user position 65. Thereafter, if the user position 65 moves outside the viewing area 70, the display control device 100 returns the sound source 71 to its original position coordinates in the virtual space 30. At this time, the display control device 100 may immediately return the sound source 71 to its original position, or may adjust the sound source 71 so that it gradually returns to its original position from the position immediately after the user position 65 leaves the viewing area 70.

[0048] The movement of the sound source position can be rephrased as the volume perceived by the user 10. That is, the process shown in Fig. 7 is expressed by the following equation, for example, when the coordinate position corresponding to the viewing area 70 is set to Model(x, y, z), the volume at the sound source position is set to Svol, and the sound source position is set to Smin, and the current position of the user position 65 is set to Pos and the volume at the current position is set to Vol as variables for each frame.

[0049]

[0050] That is, when the user position 65 is included in the viewing area 70, the display control device 100 matches the volume at the sound source position with the volume at the user position 65, thereby making the user 10 perceive the sound source as being aligned with the user position 65. Furthermore, when the user position 65 is no longer included in the viewing area 70, the display control device 100 makes the user 10 perceive the sound from the sound source at a volume calculated based on distance attenuation. In this way, the display control device 100 uses an algorithm that gradually increases the volume from the sound source as the user 10 approaches the sound source, and matches the sound source position with the user position 65 once the user 10 has approached the sound source to a certain extent, thereby realizing natural and stress-free sound reproduction for the user 10.

[0051] The above-described sound localization processing is particularly effective in, for example, reproducing stereophonic sound. As an example, 3D sound technology (stereophonic sound technology) is known, which reproduces the sound heard in a real space at a user's location in the virtual space 30. In this case, one or more virtual sound sources are placed in the virtual space 30, and an audio signal at any point is synthesized using audio signals from these sound sources. The virtual sound source is represented as a single point in the virtual space 30, and the volume of the sound reproduced at the viewing position varies depending on the relative distance and relative direction from the sound source position. This does not pose a problem if the sound source position is a certain distance from the viewing position (i.e., the position of the user 10). However, if the user 10 is close to the sound source position or attempts to pass through the sound source position, the relative distance between the sound source and the user position becomes very small. In this case, the user 10 may perceive a very loud sound or experience a sudden change in the direction of the sound due to a sudden change in relative direction, which may cause great discomfort.

[0052] On the other hand, according to the display control device 100, when the user 10 approaches the sound source more than a certain distance (when the user 10 enters the viewing area), the sound source is localized to the user 10, thereby preventing the above-mentioned unnatural sound image localization from occurring.

[0053] As explained above using Figures 2 to 8, the display control device 100 automatically adjusts the user's position and the sound source position in the virtual space 30, allowing the user 10 to enjoy the content in the optimal state.

[0054] (1-3. Configuration of the display control device according to the embodiment) Next, a description will be given of the configuration of the display control device 100. Fig. 9 is a diagram showing an example of the configuration of the display control device 100 according to the embodiment.

[0055] 9 , the display control device 100 includes a communication unit 110, a storage unit 120, and a control unit 130. The display control device 100 may also include an input unit (such as a keyboard or a touch panel) that accepts various operations from a user 10 who uses the display control device 100.

[0056] The communication unit 110 is realized by, for example, a network interface card (NIC) or a network interface controller. The communication unit 110 is connected to a network N by wire or wirelessly, and transmits and receives information to and from the HMD 20 and the like via the network N. The network N is realized by, for example, a wireless communication standard or method such as Bluetooth (registered trademark), the Internet, Wi-Fi (registered trademark), UWB (Ultra Wide Band), or LPWA (Low Power Wide Area).

[0057] The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0058] The storage unit 120 stores various information related to the display control process according to the embodiment. In the embodiment, the storage unit 120 includes a live-action video content storage unit 121, an audio content storage unit 122, and a 3DCG content storage unit 123.

[0059] The live-action video content storage unit 121 stores information about live-action video content recorded in a real space corresponding to the virtual space 30. Various live-action video contents are arranged in the virtual space 30. By viewing the live-action video content within the virtual space 30, a user 10 who uses the virtual space 30 can experience a music concert or the like that is held in the real space as if it were being held in the virtual space 30.

[0060] An optimum viewing position and viewing area are set for each live-action video content. This will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of a data table 121A in the live-action video content storage unit 121.

[0061] In the example shown in Figure 10, data table 121A stores the optimal viewing position (cam(x, y, z)), the direction of the optimal viewing position (θ, ω, ψ), and the viewing area (coordinates abcd-efgh) for each live-action video content (video1, video2, etc.).

[0062] The audio content storage unit 122 stores information about audio content based on audio recorded together with video in a real space corresponding to the virtual space 30, virtually created audio, etc. Various audio contents are placed in the virtual space 30 in the same way as live-action video content.

[0063] A sound source position and a viewing area are set for each audio content. This will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of a data table 122A related to the audio content storage unit 122.

[0064] The example shown in Figure 11 shows that data table 122A stores audio content (audio1, etc.) in association with the sound source position (source(x, y, z)) and the viewing area (coordinates abcd-efgh).

[0065] The 3DCG content storage unit 123 stores various data for constructing the virtual space 30 .

[0066] The control unit 130 is realized by, for example, a central processing unit (CPU), a micro processing unit (MPU), a GPU, etc. executing a program (for example, a display control program according to the present disclosure) stored inside the display control device 100 using a RAM, etc. as a work area. The control unit 130 is a controller, and may be realized by, for example, an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0067] As shown in FIG. 9, the control unit 130 includes an acquisition unit 131, a determination unit 132, an adjustment unit 133, and an output control unit 134.

[0068] The acquisition unit 131 acquires various types of information. For example, the acquisition unit 131 acquires the position of the user in the virtual space 30. The acquisition unit 131 also acquires content placement information, which is information about the placement of content provided in the virtual space 30, including information about an optimal viewing position set for viewing the content and a viewing area associated with the optimal viewing position.

[0069] Furthermore, the acquisition unit 131 acquires, as content, information on video content or audio content captured or recorded in the real world corresponding to the virtual space 30. Specifically, the acquisition unit 131 acquires, as content, content placement information on two-dimensional video. At this time, the content placement information includes, for example, information on an optimal viewing position 62, a viewing area 61, and coordinates where the content 60 is placed in the virtual space 30, as shown in FIG. 5 . Furthermore, the acquisition unit 131 may acquire, as content, content placement information on spherical video.

[0070] Furthermore, if the content includes audio, the acquisition unit 131 may acquire content placement information including information on the sound source position set for the audio and the viewing area associated with the sound source position. Furthermore, if the content is stereophonic content whose sound source position includes three-dimensional information, the acquisition unit 131 acquires content placement information in which three-dimensional information (three-dimensional coordinate data) corresponding to the position where the sound source was recorded in the real world is associated as the sound source position. As shown in FIG. 8 , the content placement information for such stereophonic content includes information such as volume information of the sound source 71, the viewing area 70, and the coordinates where the sound source 71 is located in the virtual space 30.

[0071] The determination unit 132 determines whether or not the user 10 is located within the viewing area based on the information acquired by the acquisition unit 131. For example, when the virtual space 30 is three-dimensional, the determination unit 132 determines whether or not the position coordinates of the user 10 corresponding to the x, y, and z axes belong within the viewing area associated with each content.

[0072] The adjustment unit 133 adjusts the relative position between the content and the user in the virtual space 30 based on the result of the determination by the determination unit 132 .

[0073] For example, when it is determined that the user 10 is located within the viewing area, the adjustment unit 133 adjusts the relative position so that the optimal viewing position set for the two-dimensional video content is close to the position of the user 10. For example, the adjustment unit 133 adjusts the relative position so that the user 10 and the two-dimensional video are directly facing each other. Furthermore, the adjustment unit 133 may adjust the relative position so that the angle of view of the user 10 in the virtual space 30 is such that the two-dimensional video is displayed at its maximum.

[0074] Furthermore, when the content is a celestial sphere video, if it is determined that the user 10 is located within the viewing area, the adjustment unit 133 may adjust the relative position so that the position of the user is close to a position (optimal viewing position) corresponding to the position where the celestial sphere video is captured in the real world.

[0075] Furthermore, when the content is audio content, the adjustment unit 133 may adjust the localization of the audio in the virtual space 30 and the relative position of the user 10 .

[0076] Specifically, when it is determined that the user 10 is located within the viewing area, the adjustment unit 133 adjusts the localization of the sound so that it is closer to the user 10. Furthermore, when it is determined that the user 10 is located within the viewing area, the adjustment unit 133 may move the localization of the sound as the user 10 moves, and when the user 10 leaves the viewing area, the adjustment unit 133 may adjust the localization of the sound so that it is returned to its original position. Note that the adjustment of the localization of the sound may be performed by adjusting the volume of the sound source, for example, as shown in FIG. 8 .

[0077] The output control unit 134 controls the information (video, audio, etc.) output from the control unit 130 to be output to the HMD 20, etc. In other words, the output control unit 134 outputs a virtual space image rendered as video content to the HMD 20, which is a display. Note that an output destination device such as the HMD 20 may be another information device such as a smartphone, a video output device such as a television, a built-in display of the display control device 100, etc.

[0078] (1-4. Processing Procedure According to the Embodiment) Next, the processing procedure according to the embodiment will be described with reference to Fig. 12 and Fig. 14. Fig. 12 is a flowchart (1) showing the flow of display control processing according to the embodiment. Fig. 12 shows the processing flow when the content is a two-dimensional live-action video image.

[0079] 12, the display control device 100 reads content into the virtual space 30 (step S101). The display control device 100 also sets initial values ​​for the user 10 in the virtual space 30 (step S102).

[0080] Next, the display control device 100 determines whether the user's position has changed (step S103). If there has been a change (step S103; Yes), the display control device 100 calculates the destination of the user 10 (step S104). After calculating the destination, or if there has been no change in the user's position (step S103; No), the display control device 100 determines whether the user's position belongs to an area (viewing area) that is adjusted to automatically move the user's position (step S105).

[0081] If the user position is within the viewing area (step S105; Yes), the display control device 100 adjusts the user position and direction as shown in FIG. 7 (step S106).

[0082] After adjusting the user position, or after skipping step S106 if the user position is not within the viewing area (step S105; No), the display control device 100 renders the live-action 2D video image, which is the content (step S107). Furthermore, the display control device 100 renders the surrounding virtual 3D space (step S108). This completes the processing of one frame.

[0083] Next, an example in which the content is a spherical image will be shown using Fig. 13. Fig. 13 is a flowchart (2) showing the flow of the display control process according to the embodiment. Note that steps S201 to S205 are the same as steps S101 to S105 shown in Fig. 12, and therefore description thereof will be omitted.

[0084] If the user 10 is within the viewing area (step S205; Yes), the display control device 100 adjusts the user position to move to an optimal viewing position associated with the spherical video (for example, the shooting position of the spherical video) (step S206). Note that in the case of the spherical video, it is not necessary to identify the direction of the viewpoint at the optimal viewing position, and therefore, unlike step S106, the display control device 100 does not need to adjust the direction.

[0085] After the user position has been automatically moved, or if the user 10 is outside the viewing area (step S205; No), the display control device 100 renders the spherical video content (step S207). Furthermore, the display control device 100 renders the surrounding virtual 3D space (step S208). This causes the display control device 100 to complete processing for one frame.

[0086] Next, an example in which the content includes audio will be described with reference to Fig. 14. Fig. 14 is a flowchart (3) showing the flow of the display control process according to the embodiment. Note that steps S301 to S304 are the same as steps S101 to S104 shown in Fig. 12, and therefore description thereof will be omitted.

[0087] The display control device 100 determines whether the user is located within the area where the sound source position is to be adjusted (step S305). If the user 10 is located within the area (step S305; Yes), the display control device 100 automatically moves the sound source position toward the user's position (step S306). If the user 10 is located outside the area (step S305; No), the display control device 100 does not move the sound source but keeps it in its original position.

[0088] The display control device 100 then calculates the volume (volume perceived by the user 10) output from the sound source based on the position after the sound source has been moved if the sound source position has been moved, or based on the original position if the sound source has not been moved (step S307). The display control device 100 then plays back the 3D sound at the calculated volume (step S308). Furthermore, the display control device 100 renders the surrounding virtual 3D space (step S309). This completes the processing of one frame for the display control device 100.

[0089] (1-5. Modifications) The processing according to the above embodiment may be modified in various ways. For example, each device in Fig. 1 conceptually represents a function in the display control system 1, and may take various forms depending on the embodiment.

[0090] For example, in the above embodiment, an example was shown in which the display control device 100 performs processing in cooperation with the HMD 20. However, the display control device 100 may perform processing that is performed by the HMD 20 on its own device.

[0091] In this case, the display control device 100 may include sensors that detect various environmental information included in the HMD 20. For example, the sensors may include an outward-facing camera that captures images of the outside of the display control device 100, an inward-facing camera that captures images of the user, a microphone that can measure the user's voice and surrounding sounds, and the like. The sensors may also have a function for detecting various information related to the user's movements, such as the orientation, tilt, movement, and speed of the user's body. Specifically, the sensors detect information related to the user's movements, such as information related to the user's head and posture, the movements of the user's head and body (acceleration and angular velocity), the direction of the field of view, and the speed of viewpoint movement. For example, the sensors function as various motion sensors, such as a three-axis acceleration sensor, a gyro sensor, and a velocity sensor, to detect information related to the user's movements. More specifically, the sensors detect changes in at least one of the position and posture of the user's head by detecting the yaw, pitch, and roll components of the user's head movements.

[0092] In addition, in the embodiment, content that is captured or recorded in a real space is given as an example of content, but the content does not necessarily have to be live-action video captured in a real space, etc. For example, the content may be video that imitates live-action video, or creative content such as animation.

[0093] (2. Other Embodiments) The processing according to each of the above-described embodiments may be implemented in various different forms other than the above-described embodiments.

[0094] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. Furthermore, the information, including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings, can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0095] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0096] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0097] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0098] (3. Effects of the Display Control Device According to the Present Disclosure) As described above, the display control device according to the present disclosure (display control device 100 in the embodiment) includes an acquisition unit (acquisition unit 131 in the embodiment), a determination unit (determination unit 132 in the embodiment), and an adjustment unit (adjustment unit 133 in the embodiment). The acquisition unit acquires content placement information that is information about the position of the user in the virtual space and the placement of content provided in the virtual space, the content placement information including information about a first position (optimal viewing position in the embodiment) set for viewing the content and a first region (viewing region in the embodiment) associated with the first position. The determination unit determines whether the user is located within the first region based on the acquired information. The adjustment unit adjusts the relative position of the content and the user in the virtual space based on the result of the determination.

[0099] In this way, when content is provided in a sparsely populated space, the display control device according to the present disclosure automatically moves the user to a position suitable for viewing the content if the user is located in a predetermined area, thereby enabling the display control device to provide the user with the experience originally intended by the content provider.

[0100] The acquisition unit also acquires, as content, information about video content or audio content that has been captured or recorded in the real world corresponding to the virtual space.

[0101] In this way, the display control device provides the user with video content and audio content that have been created by shooting from a certain point in the real world within a virtual 3D space that imitates real space. For example, in an application in which live-action video is viewed within a virtual space, the display control device can allow the user to view the content from a position and direction that best matches the appearance in the virtual space with the live-action video, thereby further enhancing the user's sense of immersion.

[0102] The acquisition unit acquires content placement information related to the two-dimensional video as the content. When it is determined that the user is located within the first region, the adjustment unit adjusts the relative position so that the first position set in the two-dimensional video and the user's position are close to each other. For example, the adjustment unit adjusts the relative position so that the user and the two-dimensional video are directly facing each other. The adjustment unit may also adjust the relative position so that the angle of view of the user in the virtual space is such that the two-dimensional video is displayed at its maximum.

[0103] In this way, when the content is a two-dimensional video, the display control device automatically moves the user to the optimal viewing position, thereby preventing the user from feeling discomfort due to image distortion or a mismatch in size compared to the virtual space that serves as the background when the user views the video from a position that is shifted from the optimal viewing angle.

[0104] When it is determined that the user is located within the first area, the adjustment unit adjusts the relative position of the user so that the first position, which corresponds to the position where the omnidirectional image is captured in the real world, is close to the first position.

[0105] In this way, when the content is a celestial sphere video, the display control device automatically moves the user to the center of the celestial sphere (e.g., the shooting position), which is the optimal viewing position. This allows the display control device to prevent the user from feeling uncomfortable, such as distortion of the celestial sphere video or a missing part of the celestial sphere video, when the user views the video at a position shifted from the center of the celestial sphere. Furthermore, generally, a user who moves freely in a virtual space recognizes that a celestial sphere video is placed in the virtual space, but does not know where the optimal viewing position is. However, according to the display control device, when the user approaches the celestial sphere video to a certain extent, the display control device automatically moves the user to the optimal viewing position, allowing the user to easily experience a sense of immersion.

[0106] Furthermore, if the content includes audio, the acquisition unit acquires content placement information including information on a second position set for the audio (in the embodiment, the original audio source position defined for the audio source) and a second area (in the embodiment, the viewing area 70 related to the audio, etc.) associated with the second position. The determination unit determines whether the user is located within the second area. The adjustment unit adjusts the localization of the audio in the virtual space and the relative position of the user based on the determination result.

[0107] In this way, the display control device can adjust the relative relationship between the user and the sound source position not only for video but also for audio, allowing the user to experience audio playback as intended by the content provider.

[0108] Furthermore, when the adjustment unit determines that the user is located within the second area, it adjusts the localization of the sound to be closer to the user. When the adjustment unit determines that the user is located within the second area, it moves the localization of the sound as the user moves, and when the user leaves the second area, it adjusts the localization of the sound to return to the second position. Furthermore, the acquisition unit may acquire content placement information for the stereophonic content, in which three-dimensional information corresponding to a position where the sound was recorded in the real world is associated as the second position.

[0109] In this way, the display control device moves the position of the sound source placed in the virtual space according to the distance from the sound source, thereby preventing sudden changes in the relative position and direction between the user and the sound source, thereby realizing natural sound reproduction for the user.

[0110] (4. Hardware Configuration) Information devices such as the display control device 100 according to each of the above-described embodiments are realized by a computer 1000 configured as shown in FIG. 15, for example. The following description will be given using the display control device 100 as an example. FIG. 15 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the display control device 100. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0111] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0112] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0113] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records a display control program according to the present disclosure, which is an example of program data 1450.

[0114] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0115] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, and semiconductor memories.

[0116] For example, when the computer 1000 functions as the display control device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes a display control program loaded onto the RAM 1200, thereby realizing the functions of the control unit 130, etc. The display control program according to the present disclosure and data in the storage unit 120 are stored in the HDD 1400. The CPU 1100 reads and executes program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0117] The present technology can also be configured as follows. (1) A display control device comprising: an acquisition unit that acquires content placement information related to a user's position in a virtual space and information related to the placement of content provided in the virtual space, the content placement information including information on a first position set for viewing the content and a first area associated with the first position; a determination unit that determines whether the user is located within the first area based on the acquired information; and an adjustment unit that adjusts a relative position between the content and the user in the virtual space based on a result of the determination. (2) The display control device described in (1), wherein the acquisition unit acquires, as the content, information related to video content or audio content that is captured or recorded in a real world corresponding to the virtual space. (3) The display control device described in (2), wherein the acquisition unit acquires, as the content, the content placement information related to a two-dimensional image, and when it is determined that the user is located within the first area, the adjustment unit adjusts the relative position so that the first position set in the two-dimensional image and the position of the user are close to each other. (4) The display control device according to (3), wherein the adjustment unit adjusts the relative position so that the user and the two-dimensional image face each other. (5) The display control device according to (3) or (4), wherein the adjustment unit adjusts the relative position so that the two-dimensional image is displayed at a maximum angle with respect to a field of view of the user in the virtual space. (6) The display control device according to any one of (2) to (5), wherein the acquisition unit acquires the content placement information related to a celestial sphere image as the content, and when it is determined that the user is located within the first area, the adjustment unit adjusts the relative position so that the first position corresponding to a position where the celestial sphere image is captured in the real world is close to the position of the user.(7) The display control device according to any one of (2) to (6), wherein the acquisition unit, when the content includes audio, acquires the content placement information including information on a second position set for the audio and a second area associated with the second position, the determination unit determines whether the user is located within the second area, and the adjustment unit adjusts the localization of the audio and a relative position of the user in the virtual space based on a result of the determination. (8) The display control device according to (7), wherein, when it is determined that the user is located within the second area, the adjustment unit adjusts the localization of the audio to be closer to the user. (9) The display control device according to (8), wherein, when it is determined that the user is located within the second area, the adjustment unit moves the localization of the audio in accordance with movement of the user, and when the user leaves the second area, adjusts the localization of the audio to return to the second position. (10) The display control device according to any one of (7) to (9), wherein the acquisition unit acquires the content placement information related to stereophonic content, in which three-dimensional information corresponding to a position where audio was recorded in the real world is associated as the second position. (11) A display control method including: a computer acquiring content placement information related to a user's position in a virtual space and information about the placement of content provided in the virtual space, the content placement information including information about a first position set for viewing the content and a first area associated with the first position; determining whether the user is located within the first area based on the acquired information; and adjusting a relative position between the content and the user in the virtual space based on a result of the determination.(12) A display control program for causing a computer to function as a display control device having: an acquisition unit that acquires content placement information relating to a user's position in a virtual space and the placement of content provided in the virtual space, the content placement information including information relating to a first position set for viewing the content and a first area associated with the first position; a determination unit that determines whether the user is located within the first area based on the acquired information; and an adjustment unit that adjusts the relative position of the content and the user in the virtual space based on the result of the determination.

[0118] REFERENCE SIGNS LIST 10 User 20 HMD 100 Display control device 110 Communication unit 120 Storage unit 121 Live-action video content storage unit 122 Audio content storage unit 123 3DCG content storage unit 130 Control unit 131 Acquisition unit 132 Determination unit 133 Adjustment unit 134 Output control unit

Claims

1. A display control device comprising: an acquisition unit that acquires content placement information relating to a user's position in a virtual space and the placement of content provided in the virtual space, the content placement information including information regarding a first position set for viewing the content and a first area associated with the first position; a determination unit that determines whether the user is located within the first area based on the acquired information; and an adjustment unit that adjusts the relative position of the content and the user in the virtual space based on the result of the determination.

2. The display control device according to claim 1, wherein the acquisition unit acquires, as the content, information regarding video content or audio content captured or recorded in the real world corresponding to the virtual space.

3. The display control device described in claim 2, wherein the acquisition unit acquires the content placement information regarding a two-dimensional image as the content, and the adjustment unit adjusts the relative position when it is determined that a user is located within the first area so that the first position set in the two-dimensional image and the user's position are close to each other.

4. The display control device according to claim 3, wherein the adjustment unit adjusts the relative position so that the user and the two-dimensional image face each other directly.

5. The display control device according to claim 4, wherein the adjustment unit adjusts the relative position so that the angle of view at which the two-dimensional image is displayed is maximized with respect to the user's field of view in the virtual space.

6. The display control device according to claim 2, wherein the acquisition unit acquires the content placement information regarding a celestial sphere image as the content, and the adjustment unit, when it is determined that the user is located within the first area, adjusts the relative position so that the first position, which corresponds to the position at which the celestial sphere image was captured in the real world, and the position of the user are close to each other.

7. The display control device described in claim 2, wherein the acquisition unit acquires the content placement information including information on a second position set for the audio and a second area associated with the second position when the content includes audio, the determination unit determines whether the user is located within the second area, and the adjustment unit adjusts the position of the audio in the virtual space and the relative position of the user based on the result of the determination.

8. The display control device according to claim 7, wherein the adjustment unit adjusts the positioning of the sound so as to be closer to the user when it is determined that the user is located within the second area.

9. The display control device according to claim 8, wherein the adjustment unit, when it determines that the user is located within the second area, moves the audio localization in accordance with the movement of the user, and when the user leaves the second area, adjusts the audio localization to return it to the second position.

10. The display control device according to claim 7, wherein the acquisition unit acquires the content placement information relating to stereophonic content in which three-dimensional information corresponding to a position where sound was recorded in the real world is associated as the second position.

11. A display control method comprising: a computer acquiring content placement information relating to a user's position in a virtual space and the placement of content provided in the virtual space, the content placement information including a first position set for viewing the content and information regarding a first area associated with the first position; determining whether the user is located within the first area based on the acquired information; and adjusting the relative position of the content and the user in the virtual space based on the result of the determination.

12. A display control program for causing a computer to function as a display control device having: an acquisition unit that acquires content placement information relating to a user's position in a virtual space and the placement of content provided in the virtual space, the content placement information including information on a first position set for viewing the content and a first area associated with the first position; a judgment unit that judges whether the user is located within the first area based on the acquired information; and an adjustment unit that adjusts the relative position of the content and the user in the virtual space based on the result of the judgment.

Images