Information processing apparatus, method for controlling information processing apparatus, and storage medium

Abstract

To reduce stress of a user during XR video experience using an HMD.SOLUTION: According to an aspect of the present disclosure, there is provided an information processing apparatus that generates a virtual reality image to be stereoscopically displayed by a head-mounted display apparatus, the information processing apparatus including a first acquisition unit configured to acquire, as a material for generating the virtual reality image, data of a stereoscopic image capable of representing two images having a parallax therebetween and information regarding the parallax corresponding to the stereoscopic image, and a generation unit configured to generate the virtual reality image subjected to specific processing for allowing a user wearing the display apparatus to easily focus his / her eyes when a difference in parallax between different frames of the stereoscopic image is larger than a threshold value.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to image processing technology for XR video using a head-mounted display. [Background technology]

[0002] In recent years, video technology known as cross reality (XR), such as virtual reality (VR), which displays CG images in three dimensions to allow users to experience virtual spaces as if they were real, and mixed reality (MR), which blends the real world with the virtual world, has made remarkable progress. Patent Document 1 describes a technology for capturing a three-dimensional image (stereo image) by presenting a user with an area that is unlikely to fuse into a single image when viewed, and for acquiring a stereo image that allows comfortable stereoscopic viewing. Patent Document 2 also describes a technology for a VR system using an HMD, in which an information display object (UI) is positioned in the line of sight of the HMD wearer and the UI is displayed so as to follow changes in the wearer's line of sight. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-15620 A [Patent Document 2] JP 2018-45459 A Summary of the Invention [Problem to be solved by the invention]

[0004] The technology of Patent Document 1 above has a problem in that, when experiencing 3D images with an HMD, for example, when the scene switches from a distant image to a close-up image, the wearer's eyes' focus adjustment function cannot keep up with the sudden change in distance to the subject, causing eye strain. Also, the technology of Patent Document 2 above has a problem in that, when the wearer operates a UI to input commands while watching VR or MR images, for example, the UI may hide the content depending on the position where it is displayed, interfering with viewing the image. [Means for solving the problem]

[0005] The information processing device according to the present disclosure is an information processing device that generates a virtual reality image that is displayed in three dimensions by a head-mounted display device, and is characterized by having: a first acquisition means that acquires, as materials for generating the virtual reality image, data on a three-dimensional image that can represent two images with parallax from each other and information on the parallax corresponding to the three-dimensional image; and a generation means that generates a virtual reality image that has been subjected to specific processing to make it easier for a user wearing the display device to focus their eyes when the difference in parallax between different frames of the three-dimensional image is greater than a threshold value. [Effects of the Invention]

[0006] The technology disclosed herein can reduce stress experienced by a user while experiencing XR video using an HMD. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1A is a diagram showing an example of the configuration of an HMD video display system, and FIG. 1B is a diagram showing an example of the hardware configuration of an information processing device. [Figure 2] FIG. 2 is a functional block diagram showing the software configuration of the information processing device. [Figure 3] 10A and 10B are flowcharts showing the flow of processing when the information processing device according to the first embodiment generates a VR image. [Figure 4] 1A and 1B are diagrams illustrating the principle of stereoscopic vision. [Figure 5] (a) is a diagram showing how the wearer can observe a subject in three dimensions by viewing a VR image through an HMD, and (b) is a diagram showing the relationship between the raw data of the moving image and the VR image displayed on the HMD. [Figure 6] (a) is an explanatory diagram of a three-dimensional unit vector, and (b) is an explanatory diagram of a certain range. [Figure 7] 10A and 10B are block diagrams showing the internal configuration of an information processing device according to a modification of the first embodiment. [Figure 8] 10 is a flowchart showing the flow of operations performed by an information processing device according to Modification 1 of Embodiment 1 when generating a VR image. [Figure 9] 10 is a flowchart showing the flow of operations performed by the information processing device 0 according to the second modification of the first embodiment when generating a VR image. [Figure 10] FIG. 10 is a functional block diagram showing the software configuration of an information processing device according to a second embodiment. [Figure 11] 1A and 1B are explanatory diagrams of a three-dimensional coordinate system that represents the three-dimensional space in which the HMD wearer exists. [Figure 12] 10 is a flowchart showing a processing flow when an information processing device according to a second embodiment determines and displays a position of a UI. [Figure 13] 1A is a diagram illustrating the process of calculating the visual recognition area, and FIG. 1B is a diagram illustrating the process of determining the position of the UI based on the visual recognition area and the gaze area. [Figure 14] (a) is a diagram showing the HMD wearer watching video content, and (b) is a diagram showing the UI displayed. [Figure 15] 10A is a block diagram showing the internal configuration of an information processing device according to a first modification of the second embodiment, and FIG. 10B is a block diagram showing the internal configuration of an information processing device according to a second modification of the second embodiment. [Figure 16] 10 is a flowchart showing a processing flow when an information processing device according to a first modification of the second embodiment determines and displays a position of a UI. [Figure 17] FIG. 10 is a diagram for explaining the behavior of a UI in a first modified example of the second embodiment. [Figure 18]10 is a flowchart showing a processing flow when an information processing device according to a second modification of the second embodiment determines and displays the position of a UI. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the present invention, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present invention. Furthermore, the same components will be described with the same reference numerals.

[0009] [Embodiment 1] In this embodiment, we will explain how, when generating VR images to be displayed on an HMD (head-mounted display), the parallax in each frame to achieve stereoscopic vision is adjusted so that it is appropriate for the user (wearer) wearing the HMD.

[0010] <System configuration> Fig. 1(a) is a diagram showing an example of the configuration of a video display system using an HMD. The display system shown in Fig. 1(a) is composed of an information processing device 10 that controls the HMD 20, and the HMD 20, which is a head-mounted display device. In this embodiment, the information processing device 10 is described as being independent of the HMD 20 in a system configuration, but the HMD 20 may also be configured as an all-in-one HMD system that includes the information processing device 10 inside.

[0011] <Hardware configuration of information processing device> FIG. 1B is a diagram illustrating an example of the hardware configuration of an information processing device. In FIG. 1B, a CPU 101 uses a RAM 102 as a work memory, executes programs stored in a ROM 103 and a hard disk drive (HDD) 105, and controls the operation of each block (described later) via a system bus 112. An HDD interface (hereinafter, interface will be referred to as "I / F") 104 connects a secondary storage device such as the HDD 105 or an optical disk drive. The HDD I / F 104 is, for example, an I / F such as a serial ATA (SATA). The CPU 101 can read data from and write data to the HDD 105 via the HDD I / F 104. Furthermore, the CPU 101 can load data stored in the HDD 105 into the RAM 102, and conversely, can save data loaded in the RAM 102 to the HDD 105. The CPU 101 can then execute the data loaded in the RAM 102 as a program. The input I / F 106 can connect input devices such as a keyboard, a mouse, and an HMD controller. The input I / F 106 is, for example, a serial bus I / F such as USB or IEEE1394. The CPU 101 reads data from the input device 107 via the input I / F 106. The output I / F 108 connects the information processing device 10 to an HMD 20, which is an output device 109. The output I / F 108 is, for example, a video output I / F such as DVI or HDMI (registered trademark) and / or a serial bus I / F such as USB or IEEE1394. The CPU 101 can send data to the output device 109, such as the HMD 20, via the output I / F 108 to display a predetermined image. The CPU 101 also receives information such as the position and posture of the HMD 20 when the wearer is experiencing the image (hereinafter referred to as "HMD information") from the HMD 20. The HMD information may be input via a mouse, keyboard, camera, or the like. Although the information processing device 10 has other components in addition to those described above, they are not the focus of the present invention and therefore will not be described here.

[0012] <Software configuration of information processing device> 2 is a functional block diagram showing the software configuration (logical configuration) of the information processing device 10 according to this embodiment. The information processing device 10 according to this embodiment has a material data acquisition unit 201, an HMD information acquisition unit 202, a processing method determination unit 203, a VR image generation unit 204, and a display processing unit 205. Each unit will be described below.

[0013] The material data acquisition unit 201 acquires stereoscopic image data that has been captured and saved in advance from the secondary storage device 105, or acquires the stereoscopic image data directly in real time immediately after capture. The stereoscopic images of this embodiment are two equirectangular stereoscopic images (a pair of left and right images) with parallax between them, captured by a VR camera with a wide viewing angle of 180 degrees or 360 degrees. The data format of the stereoscopic images may be any format that can represent two images with parallax between them, such as a cube map format. The acquired stereoscopic image data is output to the VR image generation unit 204. The material data acquisition unit 201 also acquires information regarding parallax corresponding to the acquired stereoscopic images. This information regarding parallax (hereinafter referred to as "parallax data") is, for example, an image (parallax image) having the same resolution as the stereoscopic images, and each pixel value in the parallax image indicates the parallax value corresponding to each pixel of the left viewpoint in the stereoscopic images. The parallax images may be acquired as a set with the stereoscopic images, or may be derived from the stereoscopic images. Methods for deriving the parallax data from stereo images include, for example, a method of performing block matching between the left and right images that constitute the acquired stereo images to find corresponding regions, and a method using machine learning. Furthermore, for example, a stereoscopic video format called MV-HEVC includes depth data indicating the distance to the subject, and this depth data may be treated as parallax data. Note that the parallax data may be in any data format that allows the parallax value for each subject appearing in the stereo images to be acquired. For example, each pixel value of the parallax image may indicate a parallax value corresponding to the right viewpoint. Furthermore, the resolution of the parallax image does not necessarily have to be the same as the resolution of the stereo images, and may be lower than the resolution of the stereo images. The acquired parallax data is output to the processing method determination unit 203.

[0014] The HMD information acquisition unit 202 acquires, as HMD information, information indicating the position and orientation of the HMD 20 while the wearer is experiencing VR video. The HMD 20 has multiple RGB cameras (not shown) and an inertial measurement unit (IMU) (not shown) to achieve inside-out position tracking. The IMU detects three-dimensional inertial motion (translational and rotational motion in three orthogonal axes) and is composed of a gyro sensor that detects rotational motion and an acceleration sensor that detects translational motion. The IMU represents the orientation (gaze direction) of the wearer of the HMD 20 using a 3x3 rotation matrix in three-dimensional space, as well as roll, pitch, and yaw. Note that the orientation representation method is not limited to this, and other representation methods such as quaternions may also be used. The HMD 20 may also have an eye-tracking function that can track the wearer's eye movement and identify what the wearer is actually looking at. The acquired HMD information is output to the processing method determination unit 203 and the VR image generation unit 204.

[0015] The processing method determination unit 203 determines a processing method for generating a virtual reality image (VR image) to be displayed on the HMD 20 based on the input HMD information and parallax data. In this embodiment, based on the amount of change in parallax from the previous frame, it determines whether to generate the image by normal processing or by performing processing to match the line of sight with a smaller parallax. Information on the determined processing method is output to the VR image generation unit 204.

[0016] The VR image generation unit 204 generates a VR image for stereoscopic display based on the HMD information input from the HMD information acquisition unit 202 and the stereo image input from the material data acquisition unit 201, in accordance with the processing method determined by the processing method determination unit 203. Data of the generated VR image is output to the display processing unit 205.

[0017] The display processing unit 205 converts the VR image input from the VR image generation unit 204 into an image suitable for viewing on the HMD 20 and outputs it to the HMD 20. The conversion here includes color conversion processing suitable for the built-in display of the HMD 20, correction processing to correct distortion of the eyepiece lens of the HMD 20, and the like.

[0018] <Operation flow of information processing device> Next, the flow of processing when the information processing device 10 according to this embodiment generates a VR image will be described with reference to the flowchart of FIG. 3(a). The series of processes shown in the flowchart of FIG. 3(a) are realized by the CPU 101 expanding a program stored in the ROM 103 into the RAM 102 and executing the program. In this embodiment, a data set of stereo images of a moving image that has been captured and saved in advance and corresponding parallax images is read from the HDD 105, and processing is started in response to an instruction from the wearer to start viewing the VR video, and is executed frame by frame. Note that not all of the processing shown in the flowchart of FIG. 3(a) needs to be executed by the CPU 101; some or all of the processing may be executed by one or more processing circuits other than the CPU 101. Note that in the following description, the symbol "S" denotes a step.

[0019] In S301, the material data acquisition unit 201 acquires stereo images of frames of interest from a data set of input stereo images and corresponding parallax images. The stereo images acquired in this step and the parallax images acquired in S304 (described later) will now be described with reference to the accompanying drawings. As a premise, the principles of stereoscopic vision will be explained using FIGS. 4(a) and 4(b). The HMD 20 has lenses 401R / 401L and displays 402R / 402L placed in front of each eye. A user wearing the HMD 20 can perceive an image as a virtual image by viewing the displays 402R / 402L through the lenses 401R / 401L with each eye. By displaying different images with parallax on the displays 402R / 402L, the wearer can perceive the virtual image as a three-dimensional image due to binocular parallax. The position of this virtual image varies depending on the amount of parallax between the images displayed on the displays 402R / 402L. For example, as shown in FIG. 4(a), when the position of the same subject 404 is significantly different between the right-eye image 403R and the left-eye image 403L, resulting in a large parallax 405, the wearer perceives the subject 404 as being relatively close. On the other hand, as shown in FIG. 4(b), when the position of the same subject 406 is not significantly different between the right-eye image 403R and the left-eye image 403L, resulting in a small parallax 407, the wearer perceives the subject 406 as being relatively far away. As shown in FIG. 5(a), by viewing a VR image based on a stereo image captured by a VR camera on the HMD 20, the wearer can observe the subject in three dimensions. FIG. 5(b) illustrates the relationship between the material data of a moving image and the VR image displayed on the HMD 20. In FIG. 5(b), the lower side of the right-pointing arrow 500 representing the time axis in the center represents the material data, and the upper side represents the VR image. In this step, a stereo image 501 relating to the frame of interest (the nth frame in the figure) is displayed. n is acquired, and the corresponding parallax image 502 is acquired in step S304 (described later). n The parallax image 502 is obtained. nThe smaller the parallax, the darker the image, and the larger the parallax, the brighter the image. The stereo image data of the acquired frame of interest is output to the VR image generation unit 204. Note that the configuration shown in FIG. 4(a) is just an example, and a configuration consisting of one display and two lenses may also be used. Any configuration is acceptable as long as the HMD is capable of stereoscopic viewing.

[0020] In S302, the HMD information acquisition unit 202 acquires HMD information indicating the current position and posture of the HMD 20. The acquired HMD information is output to the processing method determination unit 203 and the VR image generation unit 204.

[0021] In S303, the next process to be executed is determined based on whether or not there is a frame immediately preceding the frame of interest. If there is a previous frame, S304 is executed. On the other hand, if the frame of interest is the first frame and there is no previous frame, S307 is executed next. In the example of FIG. 5(b) described above, since there is an (n-1)th frame for the nth frame, which is the frame of interest, S304 is executed.

[0022] In S304, the material data acquisition unit 201 acquires the parallax images of the frame of interest and the previous frame. In the example of FIG. 5(b), the parallax image 502 of the n-th frame is n and the disparity image 502 of the (n-1)th frame n―1 The acquired data of the parallax images of the frame of interest and the previous frame is output to the processing method determination unit 203.

[0023] In S305, the processing method determination unit 203 calculates the difference in the amount of parallax between the frame of interest and the previous frame based on the parallax images of the frame of interest and the previous frame acquired in S304. Specifically, first, the difference between the parallax image of the frame of interest and the parallax image of the previous frame is calculated for each corresponding pixel. Next, based on the posture information of the HMD 20 included in the HMD information, the area in the stereo image that the wearer is looking at (hereinafter referred to as the "observation area") is identified, and the average value of the difference values ​​of all pixels included in the area of ​​the parallax image corresponding to the observation area is calculated. Finally, the calculated average value is used as the difference in the amount of parallax between the two frames. Note that the method for calculating the difference in the amount of parallax is not limited to this. For example, the median or mode of all pixels in the area of ​​the parallax image that the wearer is looking at may be used, as long as it indicates a representative difference in the amount of parallax in the identified area.

[0024] In S306, the processing method determination unit 203 determines the type of VR image to generate for the frame of interest and the processing method based on the difference in the amount of parallax between the frame of interest and the previous frame calculated in S305. The next process to be executed is then assigned based on the determined result. In this embodiment, if the calculated difference in the amount of parallax is equal to or less than a predetermined threshold, the unit determines to generate the VR image using a normal processing method. If the calculated difference in the amount of parallax exceeds the threshold, the unit determines to generate the VR image using a processing method that involves adjusting the VR image to a line of sight that minimizes the difference in the amount of parallax. Once the processing method for generating the VR image is determined, the type of VR image is determined accordingly (i.e., whether to generate a normal VR image or to generate a VR image adjusted to a line of sight that minimizes the difference in the amount of parallax). Here, the threshold is set to, for example, 30% of the difference between the minimum and maximum parallax values ​​in the parallax image. This threshold may be set uniformly regardless of the type of stereo image, as long as it is an index suitable for determining whether the difference in the amount of parallax between the frame of interest and the previous frame is small. If the threshold processing results in a decision to generate a VR image aligned with a gaze direction that minimizes the difference in parallax, the processing method determination unit 203 further generates associated information indicating the gaze direction. This associated information is expressed as a three-dimensional unit vector in virtual space. This associated information is first obtained by acquiring a three-dimensional unit vector representing the direction in which the wearer of the HMD 20 is facing from the posture information included in the HMD information. FIG. 6(a) is an explanatory diagram of the three-dimensional unit vector, with the arrow 600 in the diagram indicating the direction in which the wearer is facing. Next, in the parallax image of the frame of interest, the average disparity value of all pixels within a certain range centered on the direction of the acquired three-dimensional unit vector is calculated, and the calculated average is used as the representative disparity value of the frame of interest. FIG. 6(b) is a diagram illustrating the certain range. In FIG. 6(b), it is assumed that the field of view viewable by the HMD 20 is 90 degrees, the wearer is facing the direction of the human subject in the stereo image 500, and can visually recognize the range 601 indicated by the dashed line in the figure. In this case, the range in which a person can receive information without moving their line of sight (for example, a range 602 with a viewing angle of 30 degrees) is the fixed range in this case. Then, from the disparity image of the next frame, a region having a disparity value close to the representative disparity value of the target frame is identified.Finally, a three-dimensional unit vector is calculated when the center of the identified area is set as the line of sight, and the calculated three-dimensional unit vector is used as associated information. This associated information is output to the VR image generation unit 204. In this way, if it is decided to generate a normal VR image, S307 is executed next, and if it is decided to generate a VR image aligned with the line of sight direction that reduces the difference in parallax amount, S308 is executed next.

[0025] In S307, the VR image generation unit 204 generates a VR image to be displayed on the HMD 20 using a normal processing method based on the stereo image of the frame of interest acquired in S301 and the HMD information acquired in S302. Specifically, first, the direction in which the wearer is facing in three-dimensional space is calculated as a three-dimensional unit vector from the posture information included in the HMD information. Then, based on the direction of the calculated three-dimensional unit vector, rendering is performed according to the display angle of view of the HMD 20. The display angle of view in this case is a fixed value that depends on the HMD 20 and is specified according to the field of view and panel resolution. Furthermore, the rendering is a process of generating a perspective projection image from equirectangular stereo images, and a normal three-dimensional rendering method may be used.

[0026] In S308, the VR image generating unit 204 generates a VR image based on the stereo image of the frame of interest acquired in S301 and the accompanying information generated in S306, using a processing method that involves processing to align the image with the line of sight direction that minimizes the difference in the amount of parallax. That is, based on the direction of the three-dimensional unit vector as accompanying information, rendering is performed according to the display angle of view of the HMD 20, and a perspective projection image is generated from the stereo image. Here, with reference to the above-mentioned FIG. 5(b), a specific example of the VR image generated in this step will be described. In FIG. 5(b), the images 503 arranged on the upper side across the right-pointing arrow 500 representing the time axis are VR images cut out from the wide-viewing-angle stereo image 501 in accordance with the position and posture of the wearer. For example, VR image 503 n-1 frame n The previous frame n-1 Stereo image 501 n-1The VR image corresponds to the area 504 indicated by the dashed line. n and 503 n’ frame n Stereo image 501 n The VR image is cut out according to the position and posture of the wearer. n When generating a VR image with the frame of interest, the position and posture of the wearer are n-1 In this case, the parallax image 502 n-1 Disparity and disparity image 502 in the corresponding region n If the difference between the parallax in the corresponding region and the parallax in the corresponding region is small, image generation is performed in S307 (Yes in S306). n-1 The color representing the amount of parallax in the corresponding area of ​​the disparity image 502 is gray. n The color representing the amount of parallax in the corresponding area is white, and the difference is not small. Therefore, image generation is performed in S308 (No in S306). Then, the area where the line of sight is shifted to the left so that the difference in the amount of parallax becomes small, that is, the area corresponding to the area 505 indicated by the dashed line and centered on the gray part in the parallax image 502n, is generated in the stereo image 501. n As a result, a VR image 503 is generated, in which a tree can be viewed stereoscopically at a position far from the wearer. n This eliminates the need for the wearer to refocus their eyes, reducing eye strain.

[0027] As shown in the flowchart of FIG. 3(b), instead of the above-mentioned S308, a process for reducing the visibility of the subject may be performed (S308'). The process for reducing the visibility may include adding blur, reducing the brightness value, reducing the contrast, reducing the color difference, etc. In this case, after performing rendering according to the posture information acquired in S302, a process for reducing the visibility may be additionally performed. VR image 503 in FIG. 5(b) n’ In this case, the line of sight is not changed, so the parallax image 502 nThe area centered on the white part in the stereo image 501 n A VR image is generated in which a person can be viewed stereoscopically from a position close to the wearer, but the outline of the person is blurred. The visibility reduction process may be performed on the entire region of the rendered perspective projection image, or only on regions with large differences in parallax based on parallax data. Furthermore, rendering may be performed to change the gaze direction to reduce the difference in parallax, and then a process to reduce the visibility of the subject may be performed additionally. Alternatively, either changing the gaze direction or reducing visibility may be selectively applied. When selectively applying this process, it may be determined prior to the start of execution of this flow, or it may be determined based on the content of the input stereo image. For example, if there is no region with the same amount of parallax between the target frame and the previous frame, visibility may be reduced, and if there is, the gaze direction may be changed. A VR image with reduced visibility can distract the wearer from subjects with large changes in parallax (≒ distance), eliminating the need for forced focusing and similarly reducing eye strain.

[0028] In S309, the display processing unit 205 performs necessary conversion processing on the VR image of the frame of interest generated in S308 for display on the HMD 20 and outputs the converted VR image to the HMD 20. The HMD 20 displays the converted VR image received from the information processing device 10 on the displays 402R / 402L.

[0029] In S310, it is determined whether there is a next frame. If there is a next frame, the process returns to S301 and continues with the next frame as the frame of interest. On the other hand, if there is no next frame, the process ends.

[0030] The above is the flow of operations when generating a VR image according to this embodiment. In this embodiment, the type of VR image to be generated (the processing method when generating a VR image) is determined based on the difference in the amount of parallax indicated by the parallax data of the previous and next frames. However, the same is applicable when depth data is used instead of parallax data. This is because there is a correlation between the two (larger parallax indicates a longer distance, and smaller parallax indicates a shorter distance). Therefore, when using depth data, control may be performed to generate a VR image aligned with the line of sight so that the change in distance is smaller when the change in distance to the subject indicated by the depth data is large. Furthermore, in this embodiment, an example has been described in which a VR image is generated and displayed based on a data set of specific stereo images and corresponding parallax images that have been captured and saved in advance, but this is not limiting. For example, stereo image data captured by a VR camera may be acquired in real time, and a VR image may be generated and displayed while deriving parallax images from the acquired stereo images in real time. In this case, the resolution of the parallax images may be reduced, for example, to generate and display the VR image while taking into consideration resources, image quality, and the like.

[0031] <Variation 1> Even if the difference in parallax between the previous and next frames is large, if the impact on the wearer's vision is low (for example, if the area with a large difference in parallax occupies a small proportion of the VR image, or if that area is blurred), the wearer will not find it difficult to focus. Therefore, if the difference in parallax is large, the impact on the wearer's vision is further calculated. Then, only if the impact on vision is high, a specific process (i.e., a process of changing the line of sight or reducing visibility) may be performed to make it easier for the wearer to focus, and a VR image may be generated.

[0032] FIG. 7(a) is a block diagram showing the internal configuration of the information processing device 10 according to this modification. The difference from the block diagram shown in FIG. 2 described above is that a visual influence calculation unit 701 is added. In this modification, the material data acquisition unit 201 outputs the acquired material data to the visual influence calculation unit 701 in addition to the VR image generation unit 204. The HMD information acquisition unit 202 outputs the acquired HMD information to the processing method determination unit 203, the VR image generation unit 204, and also to the visual influence calculation unit 701. The visual influence calculation unit 701 calculates the visual influence from the stereo image included in the material data acquired from the material data acquisition unit 201 and the HMD information acquired from the HMD information acquisition unit 202. A specific calculation method will be described later. The visual influence calculation unit 701 outputs the calculated visual influence to the processing method determination unit 203. The processing method determination unit 203 determines what kind of VR image to display on the HMD 20 based on the input HMD information, the parallax data included in the material data, and the visual influence. A specific method for determining this will be described later.

[0033] Fig. 8 is a flowchart showing the flow of operations performed when the information processing device 10 according to this modification generates a VR image. The following description will be given along the flowchart in Fig. 8, but parts that are the same as those in the flowchart in Fig. 3 according to the first embodiment described above will be assigned the same reference numerals and descriptions thereof will be omitted.

[0034] In step S801, if the calculated difference in the amount of parallax exceeds a predetermined threshold, the visual influence calculation unit 701 calculates the visual influence in the observation area within the stereo image. The visual influence is a scalar value between 0 and 1, with a higher value indicating a higher visual influence. To calculate the visual influence, the observation area within the stereo image is first identified based on the posture information included in the HMD information. Next, the area of ​​the subject within the identified observation area is detected. For example, semantic area segmentation of the observation area is performed using machine learning to detect the area of ​​the subject at its center. Alternatively, based on the parallax image, the center of a pixel group with the same amount of parallax may be detected as the area of ​​the subject. Note that the area of ​​the subject to be detected does not have to be the object located at the center of the observation area. For example, the area of ​​the subject may be detected by identifying what the wearer is looking at using gaze information from the eye tracking function described above. Once the area of ​​the subject has been detected, the visual impact is calculated by evaluating the subject from the viewpoint of increasing the visibility of the subject in the observation area, specifically, from the viewpoint of the size, blur, and brightness of the subject. Representative evaluation methods for each viewpoint are as follows:

[0035] <Size perspective> The visual impact in terms of size is calculated by dividing the total number of pixels in the area of ​​the detected subject by the total number of pixels in the observation area.

[0036] <Bokeh perspective> The degree of visual impact from the viewpoint of blur is calculated by calculating the amount of blur from the detected area of ​​the subject using a known blur detection method such as Fourier transform.

[0037] <Brightness perspective> The visual impact in terms of luminance is calculated by converting the pixel values ​​of the detected object area into luminance values, and then dividing the average value of the object area by the maximum pixel value of the image (255 for an 8-bit image).

[0038] As described above, the visual influence degree according to each viewpoint is calculated. The visual influence degree may be determined by selecting one of the following viewpoints as the visual influence degree value, or may be determined by combining a plurality of viewpoints as the visual influence degree value. When combining a plurality of viewpoints, any method may be used as long as it can determine a single visual influence degree value, such as taking the product of values ​​calculated by the calculation methods for each viewpoint.

[0039] In the next step S802, the processing method determination unit 203 determines to generate a normal VR image if the calculated visual influence is equal to or less than a predetermined threshold, and determines to generate a VR image aligned with a line of sight that reduces the difference in the amount of parallax if the calculated visual influence is greater than the threshold. In this way, if it is determined to generate a normal VR image, step S307 is executed next, and if it is determined to generate a VR image aligned with a line of sight that reduces the difference in the amount of parallax, step S308 is executed next. The subsequent processing is the same as in the above-described embodiment, and therefore description thereof will be omitted.

[0040] The above is the flow of operations according to this modification when generating a VR image by the information processing device 10. In this way, even if the difference in the amount of parallax between the previous and next frames is large, if the impact on the wearer's vision is small, a normal VR image may be generated.

[0041] <Variation 2> Even if the difference in the amount of parallax between previous and next frames is large, if the distance to the subject observed by the wearer (hereinafter referred to as the "observation distance") is not constant but fluctuating, the wearer is unlikely to feel difficulty in focusing. Therefore, when the difference in the amount of parallax is large, the change in the observation distance may be further checked, and depending on whether or not there is a change in the observation distance (or the degree of change), it may be determined whether to generate a normal VR image or a VR image with a changed line of sight (and / or reduced visibility).

[0042] FIG. 7(b) is a block diagram showing the internal configuration of the information processing device 10 according to this modification. This modification differs from the block diagram shown in FIG. 2 in that an observation distance analysis unit 702 is added. In this modification, the material data acquisition unit 201 outputs the acquired material data to the observation distance analysis unit 702 in addition to the VR image generation unit 204. The HMD information acquisition unit 202 outputs the acquired HMD information to the processing method determination unit 203 and the VR image generation unit 204. The observation distance analysis unit 702 analyzes the distance to the subject observed by the wearer, and outputs information indicating a change in the observation distance over time (the elapsed time since the current observation distance was reached) to the processing method determination unit 203. The processing method determination unit 203 determines a processing method for generating a VR image to be displayed on the HMD 20 based on the input HMD information, the parallax data included in the material data, and the information on the change in the observation distance over time. Specific methods for determining the method will be described later.

[0043] Fig. 9 is a flowchart showing the flow of operations performed by the information processing device 10 according to this modification when generating a VR image. The following description will be given along the flowchart in Fig. 9, but parts that are the same as those in the flowchart in Fig. 3 according to the first embodiment described above will be assigned the same reference numerals and descriptions thereof will be omitted.

[0044] In step S901, if the difference in the calculated parallax amount exceeds a predetermined threshold, the observation distance analysis unit 702 analyzes the change over time in the distance to the object observed by the wearer. Specifically, for example, the distance to the object shown in the input stereo image is calculated, and the elapsed time is reset when the distance changes due to a scene change, and the time from that point is measured. The elapsed time is then measured continuously unless the distance to the object changes by a certain amount or more. In this case, the method for calculating the distance to the object is not particularly limited. For example, the parallax amount of the object the wearer is focusing on can be calculated from a parallax image corresponding to the stereo image based on posture information included in the HMD information, and the parallax amount can be converted into distance using the principle of triangulation. Alternatively, the distance to the object can be calculated by calculating the convergence angle between the left and right eyes using the eye tracking function described above and converting the convergence angle into distance. Note that the timing for resetting the elapsed time is not limited to the above. For example, the elapsed time can be reset when a certain amount or more of change is detected in the wearer's posture information, and the time during which the same posture is maintained can be measured in the same manner. The information on the elapsed time during which no change in the observation distance is observed is output to the processing method determination unit 203 .

[0045] In S902, if the elapsed time at the current point exceeds a predetermined threshold (the variation in the observation distance is small), the processing method determination unit 203 determines to generate a VR image aligned with a line of sight that reduces the amount of parallax. On the other hand, if the elapsed time at the current point is within a predetermined threshold (the variation in the observation distance is large), the processing method determination unit 203 determines to generate a normal VR image. In this way, if it is determined to generate a normal VR image, S307 is executed next, and if it is determined to generate a VR image aligned with a line of sight that reduces the difference in the amount of parallax, S308 is executed next. The subsequent processing is the same as in the above-described embodiment, and therefore description thereof will be omitted.

[0046] The above is the flow of operations according to this modification when generating a VR image by the information processing device 10. In this way, even if there is a large difference in the amount of parallax between the previous and next frames, if the observation distance is fluctuating, a normal VR image may be generated.

[0047] As described above, according to this embodiment including each of the modified examples, when experiencing stereoscopic VR images using an HMD, the wearer can more easily focus on the subject, thereby reducing the occurrence of eye strain in the wearer.

[0048] [Embodiment 2] When watching videos or playing games using an HMD, it is necessary to display a window-shaped user interface (hereinafter simply referred to as "UI") on the display to present various information and allow the user to input commands during the experience. While an HMD can represent a wide virtual space, such as a 180-degree or 360-degree view, depending on the position at which the UI is displayed, the wearer may not be able to recognize the UI, or the UI may overlap with an object the wearer is gazing at, interfering with video viewing. Therefore, a mode in which the UI is displayed at an appropriate position when watching videos using an HMD will be described as a second embodiment. Note that a description of the system configuration and the hardware configuration of the information processing device, which are common to the first embodiment, will be omitted, and the following description will focus on the differences.

[0049] <Software configuration of information processing device> 10 is a functional block diagram showing the software configuration (logical configuration) of the information processing device 10 according to this embodiment. The information processing device 10 according to this embodiment has an HMD information acquisition unit 202, a visual recognition area calculation unit 1001, a gaze area calculation unit 1002, a UI position determination unit 1003, and a display processing unit 205. Each unit will be described below.

[0050] The HMD information acquisition unit 202 acquires HMD information indicating the position and posture of the HMD 20 while the wearer is experiencing video content. Furthermore, the HMD 20 of this embodiment has the aforementioned eye-tracking function, and the HMD information also includes gaze information that can identify what the wearer is looking at. Here, the HMD information of this embodiment will be reviewed. The gaze information included in the HMD information of this embodiment is information that can be derived from posture information of the HMD 20 and eye information of the wearer, and is data indicating the position in three-dimensional space where the wearer is gazing. This gaze information is expressed using coordinate values ​​in a three-dimensional coordinate system that represents the three-dimensional space in which the wearer of the HMD 20 exists. In this embodiment, as shown in FIG. 11( a), the three-dimensional coordinate system uses the wearer as the reference, with the x-axis representing the horizontal extent, the y-axis representing the vertical extent, and the z-axis representing the depth extent. This three-dimensional coordinate system is fixed regardless of the wearer's movement, and a reference state is required to define the coordinate system in the space in which the wearer is using the HMD 20. This reference state may be the state of the wearer when the HMD 20 starts operating or when the position / display reset function of the HMD 20 is executed. Of the two pieces of information required to derive gaze information, posture information is expressed as a 3 × 3 rotation matrix in the three-dimensional coordinate system x-y-z, which identifies the direction in which the wearer is currently facing. For example, the coordinate x' indicated by the dashed line in Figure 11(b) is identified as the horizontal direction, and z' is identified as the depth direction. Eye information, the other piece of information required to derive gaze information, is expressed as a direction vector in the three-dimensional coordinate system based on the position of the wearer's eyeballs. In Figure 11(b), arrow 1101 represents the direction vector of the left eye, and arrow 1102 represents the direction vector of the right eye. Based on these two direction vectors and the pupil distance, the three-dimensional coordinate value of the intersection 1103 of the two direction vectors on the coordinate axes x'-y'-z', which represents the direction in which the wearer is currently facing, represents the position in which the wearer is currently gazing. Furthermore, the rotation matrix is ​​used to convert the three-dimensional coordinate values ​​of the intersection 1103 into three-dimensional coordinate values ​​on the reference coordinate axes x-y-z. The converted three-dimensional coordinate values ​​become line-of-sight information that indicates the position in three-dimensional space at which the wearer is gazing.The information on the wearer's interpupillary distance may be, for example, the distance between the eyepieces stored in the HMD 20, the wearer's interpupillary distance included in the eyeball information, or a predetermined fixed value, as appropriate. Here, an example of deriving gaze information using posture information and eyeball information has been described. However, gaze information may also be derived using posture information alone. In this case, a predetermined fixed value d may be used as the depth direction of the wearer to convert the coordinate value (0,0,d) on the coordinate axes x'-y'-z' into a three-dimensional coordinate value on the reference coordinate axes x-y-z. Furthermore, gaze information may be derived taking into account changes in the wearer's position. In this case, a translation component from the reference state included in the posture information may be added to the coordinate value on the coordinate axes x'-y'-z'. Furthermore, the expression of posture information and eyeball information is not limited to the above example, and any expression may be used. The HMD information acquisition unit 202 outputs the acquired HMD information to the visual recognition area calculation unit 1001 and the gaze area calculation unit 1002.

[0051] The visual recognition area calculation unit 1001 calculates the visual recognition area of ​​the wearer based on the HMD information input from the HMD information acquisition unit 202. Here, the "visual recognition area" refers to the area in which the wearer can recognize objects when moving their gaze within a reasonable range without moving their head. This visual recognition area is expressed as a set of values: a center coordinate value on the reference coordinate axes x-y-z and a circle radius r. The center coordinate value is the gaze information itself. The radius r is calculated using z1 × tan(θ / 2) from a predetermined field of view θ and the distance z1 between the wearer's current position and the position in three-dimensional space to which the wearer is gazing. The distance z1 is calculated using the Euclidean distance in three-dimensional space from the posture information and gaze information. The predetermined field of view θ is determined based on human visual characteristics and is an angle (e.g., 30 degrees) corresponding to the range in which a person can move their gaze comfortably and receive information without moving their gaze. The value of the radius r may be different for each axis. In this case, the angle of view θ is calculated using different values ​​for the horizontal x-axis and the vertical y-axis. For the z-axis, the distance shifted by a predetermined parallax angle φ from the distance at which the wearer is gazing is calculated, and the difference between the two distances is used as the value of the radius r. The parallax angle φ is also determined based on human visual characteristics and is, for example, 5 degrees, an angle at which a person can comfortably diverge and converge their eyes. The method of expressing the visible area is not limited to the above example, and it may also be expressed as the range of each axis on the reference coordinate axes x-y-z. Information about the calculated visible area is output to the UI position determination unit 1003.

[0052] The gaze area calculation unit 1002 calculates the wearer's gaze area based on the HMD information acquired from the HMD information acquisition unit 202. Here, the "gaze area" refers to the area within the above-mentioned visual recognition area where the wearer can recognize objects contained within the area without moving their eyes or head. Like the visual recognition area, the gaze area is expressed as a set of values ​​consisting of a center coordinate value on the reference coordinate axes x-y-z and a circle radius, and the calculation method is also similar. However, the angle of view θ used in the calculation is smaller than that used in the calculation of the above-mentioned visual recognition area, such as 3 degrees, which is the highest resolution range of human vision. Like the visual recognition area, the gaze area can be expressed in various ways and is not particularly limited. Information about the calculated gaze area is output to the UI position determination unit 1003.

[0053] The UI position determination unit 1003 determines the position within the image displayed on the display of the HMD 20 at which to display the UI, based on the information about the visual recognition area acquired from the visual recognition area calculation unit 1001 and the information about the gaze area acquired from the gaze area calculation unit 1002. In this embodiment, the "position" here is expressed by the coordinate value of the upper left corner of the UI on the reference coordinate axes x-y-z, specifically, the coordinate value obtained by adding a three-dimensional vector v to the coordinate value of the center of the gaze area. The three-dimensional vector v in this case is obtained by multiplying a predetermined three-dimensional unit vector e by a coefficient α. The three-dimensional unit vector e is a vector that serves as the base for the position of the UI. The three-dimensional unit vector e can be arbitrarily determined according to the direction in which the UI is desired to be displayed. For example, if the UI is desired to be displayed to the right of the position where the wearer is gazing, the three-dimensional unit vector e = (1, 0, 0). The coefficient α is an adjustment coefficient for adjusting the position at which the UI is displayed to be within the visual recognition area but outside the gaze area. In the above example, a single radius value independent of the axis is used, but different radius values ​​may be used for each axis. In the above example, the position is expressed by the coordinate value of the top left corner of the UI, but the coordinate value of the center of the UI may be used, for example. When using other expressions such as the coordinate value of the center, it is necessary to appropriately adjust the coefficient β, the size of the UI, and the like so that the gaze area is not covered by the UI. The UI position determination unit 1003 outputs the determined position information of the UI to the display processing unit 205.

[0054] The display processing unit 205 performs the overall process of displaying video content such as VR video and MR video, and also performs the process of displaying a UI on the display in the HMD 20 based on the position information acquired from the UI position determination unit 1003. The overall process of displaying video content includes rendering according to the viewing angle of the HMD 20, color conversion suitable for the display of the HMD 20, correction processing to correct distortion of the eyepiece lens of the HMD 20, and the like.

[0055] <Operation flow of information processing device> Next, the flow of processing when the information processing device 10 according to this embodiment determines and displays the position of a UI will be described with reference to the flowchart of FIG. 12. The series of processes shown in the flowchart of FIG. 12 is realized by the CPU 101 loading a program stored in the ROM 103 into the RAM 102 and executing the program. This series of processes is started in response to a UI display instruction from the wearer who is watching video content, and is executed, for example, on a frame-by-frame basis. Note that not all of the processes shown in the flowchart of FIG. 12 need be executed by the CPU 101, and some or all of the processes may be executed by one or more processing circuits other than the CPU 101. Note that in the following description, the symbol "S" denotes a step.

[0056] In S1201, the HMD information acquisition unit 202 acquires the above-described HMD information of the current HMD 20. The acquired HMD information is output to the visual recognition area calculation unit 1001 and the gaze area calculation unit 1002.

[0057] In S1202, the visual recognition area calculation unit 1001 calculates the visual recognition area of ​​the wearer in the frame of interest. Fig. 13(a) is a diagram explaining the calculation process of the visual recognition area. In Fig. 13(a), a black circle 1301 indicates the position of the center coordinate value P = (px, py, pz) in the visual recognition area indicated by the line of sight information, and the distance z1 to the black circle 1301 is calculated by the following formula (1).

[0058]

number

[0059] The radius r of the circle in the visible area is calculated using the following equation (2).

[0060]

number

[0061] Now, let's assume that the distance z1 is 1000 mm and θ = 30 degrees. In this case, the radius r of the circle of the visible area is 267 mm according to the above formula (3), and this, together with the center coordinate value P = (px, py, pz), is obtained as information on the visible area.

[0062] In S1203, the gaze area calculation unit 1002 calculates the gaze area of ​​the wearer in the frame of interest. The calculation method is the same as that of the visual recognition area described above, except that the value of θ in the above formula (2) is changed. Now, assume that the distance z1 is 1000 mm and θ = 3 degrees. In this case, the radius r of the gaze recognition area circle is 26 mm according to the above formula (2), and the set of this and the center coordinate value P = (px, py, pz) is obtained as gaze area information.

[0063] In S1204, the UI position determination unit 1003 determines a position where the UI is to be displayed in an area within the visual recognition area but outside the gaze area. FIG. 13(b) is a diagram illustrating the process of determining the position of the UI based on the visual recognition area and the gaze area. A black circle 1301 indicates the position of the center coordinate value P = (px, py, pz) of the gaze area, and the position q where the UI is to be displayed is calculated as q = P + α × e = P + {r2 + (r1 - r2) × β} × e.

[0064] In the above formula, r1 represents the radius of the visible area, and r2 represents the radius of the gaze area. β is an arbitrary coefficient used to fine-tune the position where the UI is displayed in the area inside the visible area but outside the gaze area, and takes a value in the range of 0 to 1. For example, if β is set to 0.5, the UI display position will be the midpoint between the area inside the visible area but outside the gaze area. Since the position q is calculated for each of the xyz components, the above formula can be expressed as the following formula (3).

[0065]

number

[0066] Now, assume that the center coordinate value of the gaze area is (0,0,1000), the radius r of the visible area is 267 mm, and the radius r of the circle of the gaze area is 26 mm. In this case, α = {26 + (267 - 26) × 0.5}, and the center coordinate value of the position determined as the display position of the UI is (146,0,1000). In Figure 13(b), a white circle 1302 indicates the position of the center coordinate value q = (qx, qy, qz) at which the UI determined in this way will be displayed.

[0067] Here, the results of the processing in each step of S1203 to S1204 will be described using a specific example. FIG. 14(a) is a diagram showing a state in which a wearer 1401 of the HMD 20 is viewing video content. The wearer 1401 views different images 1402 with parallax displayed on two left and right displays 402R and 402L through lenses 401R and 401L (see FIG. 4(a) above), and the wearer 1401 can obtain a three-dimensional effect from the perceived virtual image. Note that the position of the virtual image at this time varies depending on the difference in parallax between the left and right images displayed on the displays 402R and 402L, and the wearer 1401 perceives the subject as being present at a distance corresponding to the parallax. Now, in image 1402, which represents an image for the left eye and an image for the right eye side by side, subjects 1403 and 1404 are displayed. When the field of view of the HMD 20 is 100 degrees, the field of view corresponding to each eye in image 1402 is also 100 degrees. In addition, in image 1402, only subject 1403 is shown in the image for the left eye, whereas two subjects 1403 and 1404 are shown in the image for the right eye. This indicates a situation in which the positions of the subjects are different between the left eye and the right eye, and the subject is included in the field of view of one eye but not the other. The wearer 1401 perceives the subject 1403 as being at a distance corresponding to the parallax in image 1402. In FIG. 14(a), image 1405 is obtained by superimposing, on the above-mentioned image 1402, a visible region 1406 of 1401 calculated in S1202 and a gaze region 1407 calculated in S1203. FIG. 14(b) is a diagram showing a state in which a UI is displayed in response to a UI display instruction from the wearer 1401 in the above-mentioned situation shown in FIG. 14(a). 14(b), as shown in image 1408, UI 1409 is displayed inside visual recognition area 1406 but outside fixation area 1407. In this case, if the UI overlaps with subject 1403 shown in fixation area 1407, the UI will be a nuisance, and if the UI is displayed outside visual recognition area 1406, wearer 1401 will not be able to immediately see the UI and will need to take action such as moving his or her head.In this embodiment, the UI is displayed inside the visual recognition area 1406 of the wearer 1401 and outside the gaze area 1407, so the UI does not get in the way while watching or the wearer cannot easily find the UI, improving operability for the wearer. Note that the UI to be displayed is not particularly limited, and can be applied to various things such as buttons for instructing playback / stop of video content, a display list of file names, and windows for inputting / changing various settings.

[0068] In S1205, the display processing unit 205 displays the UI at the position determined in S1204. In this case, the UI may be generated and displayed on a layer separate from the image of the video content, or may be displayed by being composited with the image of the video content. In addition, although the above description has been given of a case in which only one UI is displayed, this is not limiting, and multiple UIs may be displayed. When multiple UIs are displayed, all UIs may be displayed within the visible area but outside the gaze area, or only the main UI may be displayed within the visible area but outside the gaze area, and non-main UIs may be displayed outside the visible area.

[0069] In S1206, it is determined whether to end the display of the UI. If the purpose of the UI is achieved by, for example, inputting an instruction to stop the UI by the wearer or inputting a necessary command on the displayed UI, this processing ends. On the other hand, if the UI display is to continue, the processing returns to S1201 and continues.

[0070] The above is the flow of processing when determining and displaying the position of the UI according to this embodiment. By this processing, the position of the UI is updated frame by frame. As a result, for example, if the gaze direction of the wearer changes while the UI is being displayed, and the subject seen by the wearer changes, for example, from image 1408 to image 1410 in FIG. 14(b), the three-dimensional position of the UI also changes to follow the gaze direction. In other words, the three-dimensional display position of the UI changes from position 1411 to position 1412.

[0071] <Variation 1> In a situation where a UI is displayed while video content is being played, the line of sight (≒ gaze area) of the wearer of the HMD 20 may move back and forth between the video content and the UI. In such a situation, in the above-described embodiment, the position of the UI changes frequently, which may cause the UI to flicker and interfere with viewing of the video content. Therefore, as Modification 1, a mode will be described in which the position of the UI is not changed while the currently displayed UI is included in the visible area, and the position of the UI is changed only when the currently displayed UI is not included in the visible area.

[0072] FIG. 15(a) is a block diagram showing the internal configuration of information processing device 10 according to this modification. The difference from the block diagram shown in FIG. 10 described above is that a UI position update determination unit 1501 is added. In this modification, visual recognition area calculation unit 1001 outputs acquired visual recognition area information to UI position update determination unit 1501. Based on the visual recognition area calculated by visual recognition area calculation unit 1001 and the current position of the UI stored in RAM 102, UI position update determination unit 1501 compares the visual recognition area with the current position of the UI to determine whether to change the position of the UI. As described in the above embodiment, the visual recognition area and the position of the UI are expressed using common three-dimensional coordinate axes. If the coordinate values ​​of the UI are within the range of the visual recognition area, UI position update determination unit 1501 determines not to change the position, and if the coordinate values ​​of the UI are not within the range of the visual recognition area, UI position update determination unit 1501 determines to change the position of the UI. If it is determined that the visual recognition area should be changed, UI position update determination unit 1501 outputs information about the visual recognition area calculated by visual recognition area calculation unit 1001 to UI position determination unit 1004, and if it is determined that the visual recognition area should not be changed, outputs information about the current position of the UI read from RAM 102 to display processing unit 205. UI position determination unit 1003 outputs information about the determined display position of the UI to display processing unit 205 and also stores it in RAM 102.

[0073] Fig. 16 is a flowchart showing the flow of processing when the information processing device 10 according to this modification determines and displays the position of the UI. The following description will be given along the flowchart in Fig. 16, but parts that are the same as those in the flowchart in Fig. 12 according to the second embodiment described above will be assigned the same reference numerals and their description will be omitted.

[0074] After the calculation of the visible area (S1202) is completed, the next process to be executed is determined in S1601 depending on whether or not a UI is currently being displayed. If a UI is already being displayed, S1602 is executed next. On the other hand, if a UI is not being displayed, S1203 is executed next.

[0075] In S1602, the UI position update determination unit 1501 acquires information on the current three-dimensional position of the UI being displayed. Then, in the next S1603, the UI position update determination unit 1501 performs the above-mentioned determination process. Then, the next process to be executed is assigned according to the determination result. That is, if the UI is not included in the visual recognition area, it is determined that the UI position will be changed (Yes in S1603), and S1203 is then executed to determine a new UI position. On the other hand, if the UI is included in the visual recognition area, it is determined that the UI position will not be changed (No in S1603), and S1203 and S1204 are skipped, and S1205 is executed. As a result, the current UI position is maintained.

[0076] The above is the flow of processing when the information processing device 10 according to this modification determines and displays the position of the UI. Here, the behavior of the UI according to this modification will be specifically described with reference to FIG. 17 . In FIG. 17 , an image 1701 shows an image corresponding to one of the left or right eye displayed on the display of the HMD 20. In the image 1701, the visual recognition area of ​​the wearer 1700 is shown by a solid circle, and the gaze area is shown by a dashed circle, and the wearer 1700 is focusing on an object 1702. In FIG. 17 , of two image groups of three arranged vertically below the image 1701, the left image groups 1705, 1707, and 1709 correspond to the method of the embodiment, and the right image groups 1706, 1708, and 1710 correspond to the method of this modification.

[0077] First, assume that the wearer 1700 moves his / her gaze to the subject 1703 while the UI is being displayed within the visible area but outside the gaze area. In this case, according to the method of the above-described embodiment, the UI would be displayed at the position shown in image 1705. In contrast, according to this modified example, as shown in image 1706, the position of the UI does not change from the position before the gaze was changed (image 1701).

[0078] Next, if the wearer 1700 moves his / her gaze toward the UI while the UI remains displayed at the position shown in image 1701, then according to the method of the above-described embodiment, the UI will be displayed at the position shown in image 1707. In contrast, according to this modified example, as shown in image 1708, the position of the UI does not change from the position before the gaze was changed (image 1701).

[0079] Next, assume that wearer 1700 moves his / her gaze toward subject 1704 while the UI remains displayed at the position shown in image 1701. In this case, the currently displayed UI is outside the visible area. Therefore, whether the method of the above-described embodiment or the method of this modification is used, the position of the UI is changed to within the visible area but outside the gaze area. As is clear from a comparison of image 1709 and image 1710, the UI is displayed at the same position using either method.

[0080] As described above, according to this modification, when a UI is displayed while video content is being played, whether or not to change the position of the UI being displayed is controlled based on the relationship between the visible area and the display position of the UI, thereby suppressing UI flicker caused by changes in the wearer's line of sight.

[0081] <Variation 2> If the UI does not enter the gaze area even after a certain time has passed since the UI was displayed on the display, it is possible that the wearer has not noticed the UI. Therefore, a second modification will be described, which changes the display form of the UI so that the wearer can easily notice the UI.

[0082] FIG. 15(b) is a block diagram showing the internal configuration of the information processing device 10 according to this modification. The difference from the block diagram shown in FIG. 10 is the addition of a UI type setting unit 1502. In this modification, the gaze area calculation unit 1002 outputs information about the acquired visual recognition area to the UI position determination unit 1003 and the UI type setting unit 1502. The UI position determination unit 1003 outputs information about the determined UI display position to the display processing unit 205 and stores it in the RAM 102. The UI type setting unit 1502 sets the type of UI to be displayed according to the current situation. In this embodiment, there are three types of UI types that can be set: "normal," which is the standard display mode; "highlighted," which makes the UI more noticeable than the standard display mode; and "subdued," which makes the UI less noticeable than the standard display mode. Here, "highlighted" refers to, for example, slight changes in the size / position of the UI, blinking of the UI, etc.; and "subdued" refers to, for example, reducing the size of the UI, making the UI transparent, etc. The display modes for "highlighted" and "suppressed" are not limited to the above examples, and include changes in display mode according to the purpose. Specific setting methods will be described later. The UI type setting unit 1502 outputs the set UI type information to the display processing unit 205. The display processing unit 205 displays the UI based on the UI position information acquired from the UI position determination unit 1003 and the UI type information acquired from the UI type setting unit 1502. The display processing unit 205 also measures the elapsed time from the start of display of the current type of UI being displayed, and outputs this together with the type information to the UI type setting unit 1502. The elapsed time is reset when the UI type is changed or the UI display is stopped.

[0083] Fig. 18 is a flowchart showing the flow of processing when information processing device 10 according to this modification determines and displays the position of a UI. The following description will be given along the flowchart in Fig. 18, but parts that are the same as those in the flowchart in Fig. 12 according to the above-mentioned embodiment 2 and the flowchart in Fig. 16 according to the above-mentioned modification 1 will be assigned the same reference numerals and descriptions thereof will be omitted.

[0084] When the position of the UI is determined based on the visual recognition area and the gaze area (S1204), the next process to be executed is determined in S1601 depending on whether or not a UI is currently being displayed. If a UI is already being displayed, S1801 is executed next. On the other hand, if a UI is not being displayed, S1806 is executed next. S1801 to S1808 are executed by the UI type setting unit 1502.

[0085] In S1801, information on the position, type, and elapsed time since the start of display of the UI currently being displayed is acquired. The position of the UI currently being displayed is read and acquired from the RAM 102, and the type of the UI and elapsed time are acquired from the display processing unit 205.

[0086] In S1802, the number of times (watch count) that the wearer has looked at the displayed UI is counted. Specifically, it is determined whether the position of the displayed UI acquired in S1801 is included in the gaze area determined in S1204, and if it is included, the counter value is incremented (+1). This counter value is stored and updated in RAM 102, and is reset when the UI type is changed or when the display of the UI is stopped. As mentioned above, the gaze area and the position of the UI are expressed using a common three-dimensional coordinate axis, so by comparing the coordinate values ​​of the UI with the range of the gaze area, it is possible to determine whether the position of the displayed UI is included in the gaze area.

[0087] In S1803, the next process to be executed is determined based on whether the elapsed time acquired in S1801 is equal to or greater than a predetermined time. If it is equal to or greater than the predetermined time, S1804 is executed. On the other hand, if it is less than the predetermined time, S1806 is executed.

[0088] In S1804, the next process to be executed is determined based on the number of watches obtained in S1802. If the number of watches is 0, S1805 is executed next. On the other hand, if the number of watches is 1 or more, S1806 is executed next.

[0089] In S1805, the next process to be executed is determined depending on whether the current UI type acquired in S1801 is "emphasis." If the current UI type is "emphasis," S1807 is executed next. On the other hand, if the current UI type is not "emphasis," S1808 is executed next. Then, in S1806, the UI type is set to "normal," in S1807, the UI type is set to "subtle," and in S1808, the UI type is set to "emphasis."

[0090] The above is the flow of processing when the information processing device 10 according to this modification determines the position of the UI and displays it. Here, the behavior of the UI in this modification will be described using a specific example.

[0091] Case 1 First, suppose that after the wearer issues a command to display a UI and the UI is displayed, the UI remains outside the gaze area even after a certain time has passed. In this case, it is possible that the wearer is not noticing the UI being displayed. Therefore, control is performed to change the UI type from "normal" to "emphasis." That is, the results are Yes in S1803, Yes in S1804, and No in S1805, and the UI type is set to "emphasis" in S1808. In this way, display processing is performed to make the UI stand out so that the wearer can easily notice it.

[0092] <Case 2> Suppose that after the UI display is changed to be more prominent, the UI remains out of the gaze area even after a certain time has passed. In this case, the wearer may be aware of the UI being displayed but may think that it is not necessary at the moment. Therefore, control is performed to change the UI type from "highlighted" to "subdued." That is, the results are Yes in S1803, Yes in S1804, and Yes in S1805, and the UI type is set to "subdued" in S1807. In this way, display processing is performed to make the UI less prominent so that the wearer does not need to pay attention to the UI.

[0093] In the above-described S1804, the number of watches used as the basis for determination is set to 0, but it may be set to, for example, 1 or more. Also, instead of counting the number of watches, a binary flag indicating whether the wearer has seen or not seen may be used. Also, as a process when the UI type is set to "suppressed," the UI may be removed from the screen.

[0094] According to this modification, when a UI is displayed during playback of video content, the display mode of the UI is controlled based on the wearer's attitude toward the UI. This makes the UI easy to find when it is deemed necessary and inconspicuous when it is not, thereby improving user convenience.

[0095] As described above, according to this embodiment including the various modifications, it is possible to display the UI at an appropriate position when watching video using an HMD.

[0096] <Other Examples> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0097] The present disclosure also includes the following configurations and methods.

[0098] [Configuration 1] An information processing device that generates a virtual reality image that is displayed three-dimensionally by a head-mounted display device, a first acquisition means for acquiring, as materials for generating the virtual reality image, data of a three-dimensional image capable of expressing two images with a parallax therebetween and information on the parallax corresponding to the three-dimensional image; a generating means for generating a virtual reality image that has undergone specific processing to make it easier for a user wearing the display device to focus their eyes when a difference in parallax between different frames of the stereoscopic image is greater than a threshold value; An information processing device comprising:

[0099] [Configuration 2] 2. The information processing device according to configuration 1, wherein the difference in parallax between different frames of the stereoscopic image is a difference in parallax in an observation area viewed by the user in the stereoscopic image.

[0100] [Configuration 3] a second acquisition means for acquiring attitude information of the display device; a determination means for determining the observation area in the different frame of the stereoscopic image based on the posture information, and determining to generate a virtual reality image subjected to the determination processing when a difference in parallax in the observation area is greater than a threshold value; and The generating means generates a virtual reality image on which the specific processing has been performed based on the determination by the determining means. 3. The information processing device according to configuration 2.

[0101] [Configuration 4] 4. The information processing device according to any one of configurations 1 to 3, wherein the specific processing is processing for reducing visibility.

[0102] [Configuration 5] 5. The information processing device according to configuration 4, wherein the process of reducing visibility is a process of blurring an area in the stereoscopic image where the difference in parallax is greater than a threshold value.

[0103] [Configuration 6] 5. The information processing device according to configuration 4, wherein the process of reducing visibility is a process of reducing brightness.

[0104] [Configuration 7] 5. The information processing device according to configuration 4, wherein the process of reducing visibility is a process of reducing contrast.

[0105] [Configuration 8] 5. The information processing device according to configuration 4, wherein the process for reducing visibility is a process for reducing color difference.

[0106] [Configuration 9] 4. The information processing device according to any one of configurations 1 to 3, wherein the specifying process is a process of adjusting the gaze direction so that a difference in parallax between the different frames becomes smaller.

[0107] [Configuration 10] further comprising a calculation means for calculating a visual influence degree when the difference in parallax is greater than a threshold value; 4. The information processing apparatus according to configuration 3, wherein the determining means determines to generate a virtual reality image that has undergone the specific processing when the visual influence degree is greater than a threshold value.

[0108] [Configuration 11] The information processing device described in configuration 10, characterized in that the visual impact is calculated based on the viewpoint that the higher the visibility of the subject present in the observation area where the disparity difference is determined to be greater than a threshold, the larger the value of the visual impact.

[0109] [Configuration 12] 12. The information processing device according to configuration 11, wherein the visual impact is a value obtained by evaluating the subject in the stereoscopic image from the viewpoint of any one of size, blur, and brightness.

[0110] [Configuration 13] further comprising an analysis means for analyzing a change in the observation distance over time when the difference in parallax is greater than a threshold value; 4. The information processing device according to configuration 3, wherein the determining means determines to generate a virtual reality image that has undergone the specific processing when the change over time in the observation distance is smaller than a threshold value.

[0111] [Method 1] A control method for an information processing device that generates a virtual reality image that is displayed three-dimensionally by a head-mounted display device, comprising: an acquisition step of acquiring, as materials for generating the virtual reality image, data of a stereoscopic image capable of expressing two images with a parallax therebetween and information on the parallax corresponding to the stereoscopic image; a generating step of generating a virtual reality image that has been subjected to specific processing (to make it easier for a user wearing the display device to focus their eyes) when a difference in parallax between different frames of the stereoscopic image is greater than a threshold value; A control method comprising:

[0112] [Configuration 14] A program that causes a computer to function as each of the means of the information processing device according to any one of configurations 1 to 13.

[0113] [Configuration 15] An information processing device that controls a user interface for a user who wears a head-mounted display device and watches video, An information processing device characterized by having a display processing means for displaying the user interface inside the user's visual field in the video being viewed by the user, which is identified based on the user's gaze information, and outside the visual field in which the user is gazing, which is identified based on the user's gaze information.

[0114] [Configuration 16] the visible region is a region in which the user can recognize an object when moving their line of sight without moving their head, The gaze area is an area of ​​the visual recognition area in which the user can recognize an object without moving their head or line of sight. 16. The information processing device according to configuration 15.

[0115] [Configuration 17] an acquisition means for acquiring line-of-sight information of the user; a first area calculation means for calculating the visual recognition area based on the line-of-sight information; a second area calculation means for calculating the gaze area based on the line-of-sight information; a position determination means for determining a position of the user interface within the visual recognition area of ​​the video being viewed by the user, the visual recognition area being determined based on the user's line of sight information, and outside the user's line of sight area being determined based on the user's line of sight information, based on the visual recognition area and the gaze area; and 17. The information processing device according to configuration 16, wherein the display processing means displays the user interface at the position determined by the position determining means.

[0116] [Configuration 18] The position determining means determines the position using the following formula: q=P+α×e=P+{r2+(r1-r2)×β}×e In the above formula, q represents the determined position, P represents the center coordinate of the gaze area, α represents a coefficient for making the determined position inside the visual recognition area and outside the gaze area, e represents a three-dimensional unit vector, r1 represents the radius of the visual recognition area, r2 represents the radius of the gaze area, and β represents a coefficient for adjusting the determined position in the area inside the visual recognition area and outside the gaze area. 18. The information processing device according to configuration 17.

[0117] [Configuration 19] 20. The information processing device according to configuration 17 or 19, wherein the position determination means and the display processing means repeat the determination and the display on a frame-by-frame basis, respectively.

[0118] [Configuration 20] 21. The information processing device according to any one of configurations 17 to 20, wherein the position determination means does not perform the determination while the position of the user interface being displayed is included in the visible area.

[0119] [Configuration 21] The information processing device according to any one of configurations 15 to 20, characterized in that the display processing means changes the display form of the user interface when the user interface is not included in the gaze area even after a certain time has elapsed since display of the user interface began.

[0120] [Configuration 22] The information processing device described in configuration 21, characterized in that the display processing means changes the display form of the user interface to a display form that is more noticeable than the standard display form when the user interface is not included in the gaze area even after a certain time has elapsed since the display of the user interface began.

[0121] [Configuration 23] 23. The information processing device according to configuration 22, wherein the noticeable display mode is any one of a slight change in size of the user interface, a slight change in position of the user interface, and a blinking of the user interface.

[0122] [Configuration 25] The information processing device according to configuration 21 or 22, characterized in that the display processing means changes the display form of the user interface to a less conspicuous display form than the standard display form when the user interface is not included in the gaze area even after a certain period of time has elapsed since the display form of the user interface was changed to a more conspicuous display form.

[0123] [Configuration 26] 25. The information processing device according to configuration 24, wherein the inconspicuous display mode is any one of reducing the size of the user interface, making the user interface transparent, and stopping the display of the user interface.

[0124] [Method 2] A control method for an information processing device that controls a user interface for a user who wears a head-mounted display device and watches video, comprising: displaying the user interface inside a visual recognition area of ​​the user identified based on line-of-sight information of the user in the video being viewed by the user, and outside a gaze area of ​​the visual recognition area on which the user is gazing, identified based on the line-of-sight information of the user; A control method comprising:

[0125] [Configuration 27] A program that causes a computer to function as each of the means of the information processing device according to any one of configurations 15 to 26.

Claims

1. An information processing device that generates a virtual reality image that is displayed three-dimensionally by a head-mounted display device, a first acquisition means for acquiring, as materials for generating the virtual reality image, data of a three-dimensional image capable of expressing two images with a parallax therebetween and information on the parallax corresponding to the three-dimensional image; a generating means for generating a virtual reality image that has undergone specific processing to make it easier for a user wearing the display device to focus their eyes when a difference in parallax between different frames of the stereoscopic image is greater than a threshold value; An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein the difference in parallax between different frames of the stereoscopic image is a difference in parallax in an observation area viewed by the user in the stereoscopic image.

3. a second acquisition means for acquiring attitude information of the display device; a determination means for determining the observation area in the different frame of the stereoscopic image based on the posture information, and determining to generate a virtual reality image subjected to the determination processing when a difference in parallax in the observation area is greater than a threshold value; and The generating means generates a virtual reality image on which the specific processing has been performed based on the determination by the determining means.

3. The information processing apparatus according to claim 2, wherein:

4. 4. The information processing device according to claim 1, wherein the specific processing is processing for reducing visibility.

5. 5. The information processing apparatus according to claim 4, wherein the process of reducing visibility is a process of blurring an area in the stereoscopic image where a difference in parallax is greater than a threshold value.

6. 5. The information processing apparatus according to claim 4, wherein the process of reducing visibility is a process of reducing brightness.

7. 5. The information processing apparatus according to claim 4, wherein the process for reducing visibility is a process for reducing contrast.

8. 5. The information processing apparatus according to claim 4, wherein the process for reducing visibility is a process for reducing color difference.

9. The information processing apparatus according to claim 1 , wherein the specifying process is a process of adjusting the direction of the line of sight so that a difference in parallax between the different frames becomes small.

10. further comprising a calculation means for calculating a visual influence degree when the difference in parallax is greater than a threshold value; 4. The information processing apparatus according to claim 3, wherein the determining means determines to generate a virtual reality image that has undergone the specific processing when the visual influence degree is greater than a threshold value.

11. 11. The information processing device according to claim 10, wherein the visual influence degree is calculated based on the viewpoint that the higher the visibility of the subject present in the observation area where the difference in parallax is determined to be greater than a threshold value, the larger the visual influence degree value.

12. 12. The information processing apparatus according to claim 11, wherein the visual influence level is a value obtained by evaluating the subject in the stereoscopic image from the viewpoint of any one of size, blur amount, and brightness.

13. further comprising an analysis means for analyzing a change in the observation distance over time when the difference in parallax is greater than a threshold value; The information processing device according to claim 3, characterized in that the decision means decides to generate a virtual reality image that has undergone the specified processing when the elapsed time during which no change in the observation distance is observed exceeds a threshold value.

14. A control method for an information processing device that generates a virtual reality image that is displayed three-dimensionally by a head-mounted display device, comprising: an acquisition step of acquiring, as materials for generating the virtual reality image, data of a stereoscopic image capable of expressing two images with a parallax therebetween and information on the parallax corresponding to the stereoscopic image; a generating step of generating a virtual reality image that has been subjected to specific processing (to make it easier for a user wearing the display device to focus their eyes) when a difference in parallax between different frames of the stereoscopic image is greater than a threshold value; A control method comprising:

15. A program for causing a computer to execute the control method according to claim 14.

16. An information processing device that controls a user interface for a user who wears a head-mounted display device and watches video, An information processing device characterized by having a display processing means for displaying the user interface inside the user's visual field in the video being viewed by the user, which is identified based on the user's gaze information, and outside the visual field in which the user is gazing, which is identified based on the user's gaze information.

17. the visible region is a region in which the user can recognize an object when moving their line of sight without moving their head, The gaze area is an area of ​​the visual recognition area in which the user can recognize an object without moving their head or line of sight.

17. The information processing apparatus according to claim 16,

18. an acquisition means for acquiring line-of-sight information of the user; a first area calculation means for calculating the visual recognition area based on the line-of-sight information; a second area calculation means for calculating the gaze area based on the line-of-sight information; a position determination means for determining a position of the user interface within the visual recognition area of ​​the video being viewed by the user, the visual recognition area being determined based on the user's line of sight information, and outside the user's line of sight area being determined based on the user's line of sight information, based on the visual recognition area and the gaze area; and 18. The information processing apparatus according to claim 17, wherein said display processing means displays said user interface at the position determined by said position determining means.

19. The position determining means determines the position using the following formula: q=P+α×e=P+{r2+(r1-r2)×β}×e In the above formula, q represents a determined position, P represents the central coordinate of the gaze area, α represents a coefficient for making the determined position inside the visual recognition area and outside the gaze area, e represents a three-dimensional unit vector, r1 represents a radius of the visual recognition area, r2 represents a radius of the gaze area, and β represents a coefficient for adjusting the determined position in a region inside the visual recognition area and outside the gaze area.

19. The information processing apparatus according to claim 18,

20. 19. The information processing apparatus according to claim 18, wherein said position determining means and said display processing means respectively repeat said determination and said display on a frame-by-frame basis.

21. 19. The information processing apparatus according to claim 18, wherein said position determining means does not perform said determination while the position of said user interface being displayed is included in said visible area.

22. 17. The information processing device according to claim 16, wherein the display processing means changes the display form of the user interface when the user interface is not included in the gaze area even after a certain time has elapsed since display of the user interface started.

23. 23. The information processing device according to claim 22, wherein the display processing means changes the display form of the user interface to a display form that is more noticeable than a standard display form when the user interface is not included in the gaze area even after a certain time has elapsed since display of the user interface began.

24. 24. The information processing apparatus according to claim 23, wherein the noticeable display mode is any one of a slight change in size of the user interface, a slight change in position of the user interface, and a blinking of the user interface.

25. 23. The information processing device according to claim 22, wherein the display processing means changes the display form of the user interface to a less conspicuous display form than the standard display form if the user interface is not included in the gaze area even after a certain period of time has elapsed since the display form of the user interface was changed to a more conspicuous display form.

26. 26. The information processing apparatus according to claim 25, wherein the inconspicuous display mode is any one of reducing the size of the user interface, making the user interface transparent, and stopping the display of the user interface.

27. A control method for an information processing device that controls a user interface for a user who wears a head-mounted display device and watches video, comprising: displaying the user interface inside a visual recognition area of ​​the user identified based on line-of-sight information of the user in the video being viewed by the user, and outside a gaze area of ​​the visual recognition area on which the user is gazing, identified based on the line-of-sight information of the user; A control method comprising:

28. A program for causing a computer to execute the control method according to claim 27.

Images