Information processing apparatus

Abstract

To provide a technique capable of making a boundary portion inconspicuous by gradation processing and suppressing a change in image brightness due to gradation processing when the boundary portion is inconspicuous.SOLUTION: An information processing apparatus of the present invention comprises: first acquisition means for acquiring a first image that is an image for the left eye and a second image that is an image for the right eye and a portion of which corresponds to a portion of the first image; second acquiring means for acquiring information about an object that the user is viewing; and determining means for determining gradation processing to be performed on each of the first image and the second image. The determining means determines, when the area of the object in each of the first image and the second image does not fall within a specific range, the gradation processing with a higher intensity than when the area of the object in each of the first image and the second image falls within the specific range.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] Mixed Reality (MR) technology is known as a technology that seamlessly blends the real world and virtual world in real time. One type of MR technology is to capture the real space (subject) in front of the HMD (Head Mounted Display) with a video camera, and present (display) an MR image to the user, in which CG (Computer Graphics) is superimposed on the image of the real space. This type of HMD is called a video see-through HMD.

[0003] In an HMD, it is desirable for the user to be able to see a wide range (angle of view) of images so that they can feel highly realistic. A wide range of images can be seen by using an image for the left eye (left image) and an image for the right eye (right image) that meet the following conditions. The image area on the right side of the left image (the nose side of the user wearing the HMD) and the image area on the left side of the right image (the nose side of the user wearing the HMD) are corresponding overlapping areas. The image area on the left side of the left image (the left ear side of the user wearing the HMD) and the image area on the right side of the right image (the right ear side of the user wearing the HMD) are non-overlapping areas that do not correspond to any other area.

[0004] When left and right images that satisfy the above conditions are displayed, the non-overlapping portion of the left image is visible only to the left eye, while the overlapping portion is visible to the right eye as a black portion, such as a display frame. Similarly, the non-overlapping portion of the right image is visible only to the right eye, while the overlapping portion is visible to the left eye as a black portion. This causes binocular rivalry between the left and right eyes. This binocular rivalry makes the boundary between the non-overlapping portion and the overlapping portion stand out, preventing the user from viewing the image naturally.

[0005] Patent Document 1 discloses a technique for determining an area for gradation processing based on a group of brightness values ​​near the edge of an image. Patent Document 2 discloses a technique for performing gradation processing based on display time. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2017-215688 [Patent Document 2] Japanese Patent Application Laid-Open No. 2017-156513 Summary of the Invention [Problem to be solved by the invention]

[0007] However, boundaries are not always noticeable. In the techniques disclosed in Patent Documents 1 and 2, gradation processing is performed even when boundaries are not noticeable. When gradation processing is performed, the brightness of the image changes. Furthermore, it is undesirable for the brightness of the image to change due to unnecessary processing (for example, gradation processing when boundaries are not noticeable).

[0008] The present invention aims to provide a technology that can make boundaries less noticeable through gradation processing and can suppress changes in image brightness due to gradation processing when boundaries are not noticeable. [Means for solving the problem]

[0009] A first aspect of the present invention is an information processing device comprising a first acquisition means for acquiring a first image which is an image for the left eye and a second image which is an image for the right eye and part of which corresponds to part of the first image, a second acquisition means for acquiring information about an object which a user is looking at, and a determination means for determining gradation processing to be performed on each of the first image and the second image, wherein the determination means determines gradation processing of a higher intensity when the area of ​​the object in each of the first image and the second image does not fall within a specific range than when the area of ​​the object in each of the first image and the second image falls within the specific range.

[0010] A second aspect of the present invention is a control method for an information processing device, comprising a first acquisition step of acquiring a first image which is an image for the left eye and a second image which is an image for the right eye and part of which corresponds to part of the first image; a second acquisition step of acquiring information about an object which the user is looking at; and a determination step of determining gradation processing to be performed on each of the first image and the second image, wherein in the determination step, if the area of ​​the object in each of the first image and the second image does not fall within a specific range, a gradation processing of higher intensity is determined than when the area of ​​the object in each of the first image and the second image falls within the specific range.

[0011] A third aspect of the present invention is a program for causing a computer to function as each of the means of the information processing device. [Effects of the Invention]

[0012] According to the present invention, it is possible to make the boundary portion less noticeable by gradation processing, and to suppress the change in image brightness due to gradation processing when the boundary portion is not noticeable. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a schematic diagram illustrating an example of the configuration of a system according to a first embodiment. [Figure 2] 1 is a block diagram showing an example of the configuration of a system according to a first embodiment. [Figure 3] 3A to 3C are schematic diagrams showing examples of various images according to the first embodiment. [Figure 4] 3A to 3C are schematic diagrams showing examples of various images according to the first embodiment. [Figure 5] 4 is a flowchart showing an example of an operation according to the first embodiment. [Figure 6] 3A to 3C are schematic diagrams showing examples of various images according to the first embodiment. [Figure 7] 10 is a flowchart showing an example of an operation according to the second embodiment. [Figure 8] 10A to 10C are schematic diagrams showing examples of various images according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] (First embodiment) A first embodiment of the present invention will be described. Note that an example in which the present invention is applied to MR (Mixed Reality) technology, which presents (displays) to a user a composite image in which a CG object (virtual object) is superimposed on an image in real space, will be described, but the present invention is not limited to this. For example, the present invention may be applied to VR (Virtual Reality) technology, which presents (displays) to a user a composite image in which a virtual object is superimposed on an image in virtual space.

[0015] FIG. 1 is a schematic diagram showing an example of the configuration of a system according to the first embodiment. The system in FIG. 1 includes an HMD (Head Mounted Display) 100 and an information processing device 200. The HMD 100 is worn on the user's head and displays an image to enable the user to The HMD 100 provides the user with an MR experience. The information processing device 200 generates images for the MR experience. A cable 300 is connected to the HMD 100 and the information processing device 200, and the HMD 100 and the information processing device 200 can transmit and receive data to and from each other via the cable 300. Note that the transmitted and received data is not limited to image data, and for example, sensor data, audio data, or control data may be transmitted and received. Sensor data output from an acceleration sensor or angular velocity sensor provided in the HMD 100 may be transmitted from the HMD 100 to the information processing device 200. Control data for controlling the HMD 100 may be transmitted from the information processing device 200 to the HMD 100. Furthermore, although an example will be described in which wired communication is performed between the HMD 100 and the information processing device 200 using the cable 300, wireless communication may also be performed between the HMD 100 and the information processing device 200.

[0016] The HMD 100 captures images of real space with a camera facing forward, and displays MR images in which CG (CG images) are combined with images of the real space (camera images). The information processing device 200 acquires tracking data based on the camera images and sensor data, determines the drawing position of the CG based on the tracking data, and renders the CG. To determine the appropriate position and orientation of the CG, a marker 500 may be placed in real space (e.g., on the floor) and the position and orientation of the marker 500 may be detected from the camera images. This generates CG (world coordinate CG) corresponding to coordinate 1 (world coordinates) in a world coordinate system uniquely defined based on the marker 500, and generates MR images in which the world coordinate CG is combined with the camera images. Note that while the world coordinate 1 is shown at (near) the position of the marker 500 in FIG. 1, the world coordinate 1 may be distant from the marker 500.

[0017] The information processing device 200 may be a desktop computer such as a personal computer or a workstation, or may be a mobile terminal such as a smartphone or a wearable device. The information processing device 200 may be part of the HMD 100. The information processing device 200 may be divided into multiple devices. At least one of the multiple functions of the information processing device 200 may be executed on a cloud (server) or may be executed within the HMD 100. Similarly, the HMD 100 may be divided into multiple devices. At least one of the multiple functions of the HMD 100 may be executed on a cloud or may be executed within the information processing device 200. The processing of the present invention may be executed by the HMD 100, the information processing device 200, or another device.

[0018] FIG. 2 is a block diagram showing an example of the configuration of the system shown in FIG.

[0019] The HMD 100 includes imaging units 101L and 101R, display units 102L and 102R, gradation units 103L and 103R, gaze detection units 104L and 104R, an HMD position / posture detection unit 105, and an HMD control unit 110.

[0020] The HMD control unit 110 controls the entire HMD 100 .

[0021] The imaging unit 101L is a camera that faces forward so as to be positioned near the left eye of the user wearing the HMD 100, and captures a left camera image by capturing an image of real space. The imaging unit 101R is a camera that faces forward so as to be positioned near the right eye of the user wearing the HMD 100, and captures a right camera image by capturing an image of real space.

[0022] The HMD position / orientation detection unit 105 detects the position and orientation of the HMD 100. There is no particular limitation on the method for detecting the position and orientation of the HMD 100. For example, the HMD position / orientation detection unit 104 may include at least one of an acceleration sensor, an angular velocity sensor, and a geomagnetic sensor. The position and orientation of the HMD 100 may be interpreted as the position and orientation of the head of a user wearing the HMD 100.

[0023] The gaze detection unit 104L acquires left gaze information related to the gaze of the left eye of the user wearing the HMD 100. For example, the gaze detection unit 104L includes a camera that is provided facing backward so as to be positioned near the left eye of the user wearing the HMD 100, and acquires left gaze information using an image of the left eye acquired by the camera. The left gaze information includes, for example, information on the direction of the gaze of the left eye. The left gaze information may include information such as the position in three-dimensional space (for example, an MR space that combines real space and virtual space) where the gaze of the left eye is fixed (left gaze position) and the distance from the user (left eye) to the left gaze position (left gaze distance).

[0024] For example, the HMD control unit 110 acquires environmental distance information indicating the distribution of distances from the user (HMD 100) to each position in real space, and acquires object information indicating the position and size of a virtual object in the MR space. The gaze detection unit 104L detects, as the left gaze position, the position reached by the left eye's gaze in the MR space in which a virtual object is placed in real space, based on the left eye's gaze, environmental distance information, and object information. The gaze detection unit 104L then calculates the left gaze distance based on the left gaze position. The method of acquiring the environmental distance information and object information is not particularly limited. For example, the HMD control unit 110 acquires environmental distance information based on the left camera image and the right camera image, and acquires object information from the information processing device 200.

[0025] Similarly, the gaze detection unit 104R acquires right gaze information related to the gaze of the right eye of the user wearing the HMD 100. For example, the gaze detection unit 104R includes a camera facing backward so as to be positioned near the right eye of the user wearing the HMD 100, and acquires right gaze information using an image of the right eye acquired by the camera. The right gaze information includes, for example, information on the direction of the gaze of the right eye. The right gaze information may include information such as the position in three-dimensional space (for example, MR space) where the gaze of the right eye is fixed (right gaze position) and the distance from the user (right eye) to the right gaze position (right gaze distance).

[0026] The HMD control unit 110 outputs the left camera image, the right camera image, HMD position / posture information (detection results of the position and posture of the HMD 100), left gaze information, and right gaze information to the information processing device 200. Then, the HMD control unit 110 acquires a left composite image, a right composite image, left gradation processing data, and right gradation processing data. The left composite image is an image obtained by superimposing (compositing) an image of a virtual object on the left camera image. Instead of the left composite image, an image without a virtual object (for example, a left camera image) may be acquired. The right composite image is an image obtained by superimposing (compositing) an image of a virtual object on the right camera image. Instead of the right composite image, an image without a virtual object (for example, a right camera image) may be acquired. The left gradation processing data are parameters for gradation processing performed by the gradation unit 103L. The right gradation processing data are parameters for gradation processing performed by the gradation unit 103R.

[0027] The gradation section 103L performs gradation processing on the left composite image to generate a left display image. The gradation section 103R performs gradation processing on the right composite image to generate a right display image. The gradation processing will be described in detail later.

[0028] The display unit 102L displays a left display image. The user views the left display image displayed on the display unit 102L with his left eye. The display unit 102R displays a right display image. The user views the right display image displayed on the display unit 102R with his right eye. The method of displaying (presenting) the image is not particularly limited, and for example, the image may be displayed on a display panel such as a liquid crystal panel or an organic EL panel provided in a position facing the user's eyes, or ... display panel. Images may be projected directly onto the user's retina.

[0029] The information processing device 200 includes a computer control unit 201 , an object generation unit 202 , a content database (DB) 203 , an information processing unit 204 , and a parameter DB 205 .

[0030] The computer control unit 201 controls the entire information processing device 200. The computer control unit 201 acquires from the HMD control unit 110 the left camera image, the right camera image, HMD position / posture information, left gaze information, and right gaze information.

[0031] The content DB 203 stores data of virtual objects (for example, 3DCG data), etc. The object generation unit 202 reads data of virtual objects to be superimposed on real space (left camera image and right camera image) from the content DB 203 and outputs the data to the computer control unit 201.

[0032] The computer control unit 201 generates a left composite image and a right composite image by superimposing the image of the virtual object read by the object generation unit 202 onto the left camera image and the right camera image. At this time, the computer control unit 201 determines the position, orientation, and size of the virtual object in each of the left camera image and the right camera image based on the above-mentioned environment distance information, HMD position / orientation information, etc. This improves the realism of the virtual object that the user approaches. The computer control unit 201 can also determine the position and orientation of the virtual object in the MR space based on the environment distance information, HMD position / orientation information, etc. A method for acquiring the environment distance information is not particularly limited; for example, the computer control unit 201 acquires the environment distance information from the HMD 100.

[0033] The computer control unit 201 may perform image processing on the left camera image to reduce image distortion caused by aberrations in the optical system of the imaging unit 101L and the optical system of the display unit 102L. Similarly, the computer control unit 201 may perform image processing on the right camera image to reduce image distortion caused by aberrations in the optical system of the imaging unit 101R and the optical system of the display unit 102R.

[0034] The computer control unit 201 outputs the left composite image, the right composite image, the left gaze information, and the right gaze information to the information processing unit 204.

[0035] The information processing unit 204 acquires information about an object (gaze object) that the user is looking at based on the gaze information. The gaze object may be a virtual object or a real object (an object existing in real space). For example, the information processing unit 204 detects an object present at the left gaze position in the left composite image as the gaze object. Then, the information processing unit 204 acquires information about the position and size of the gaze object in the MR space based on the left composite image and the right composite image (a three-dimensional model formed by the left composite image and the right composite image). The information processing unit 204 may detect an object present at the right gaze position in the right composite image as the gaze object. The information processing unit 204 may detect an object present at the intersection of the left eye's gaze and the right eye's gaze in the three-dimensional model as the gaze object based on the left composite image, the right composite image, left gaze information, and right gaze information. Detection of the gaze object may be performed by an external device such as the HMD 100, and the information processing unit 204 may acquire information about the gaze object from the external device.

[0036] Furthermore, the information processing unit 204 determines the gradation processing to be performed on each of the left composite image and the right composite image. In the first embodiment, the parameter DB 205 stores the left gradation The information processing unit 204 stores left gradation processing data, right gradation processing data, etc. The information processing unit 204 reads out left gradation processing data and right gradation processing data from the parameter DB 205 based on the position and size of the gaze object in the MR space. The information processing unit 204 outputs the read out left gradation processing data and right gradation processing data to the computer control unit 201.

[0037] The computer control unit 201 outputs the left composite image, the right composite image, the left gradation processing data, and the right gradation processing data to the HMD 100.

[0038] An example of gradation processing will be described with reference to FIGS. 3(A) to 3(D).

[0039] 3A is a schematic diagram showing an example of a left display image and a right display image without gradation processing. Each of the left display image and the right display image has an overlapping portion, which is an image area between the left display image and the right display image that corresponds to each other, and a non-overlapping portion, which is an image area between the left display image and the right display image that does not correspond to each other. In FIG. 3A, the overlapping portion is a portion that can be viewed stereoscopically, and the non-overlapping portion is a portion that cannot be viewed stereoscopically, but this is not limited to this, and both the overlapping portion and the non-overlapping portion may be portions that cannot be viewed stereoscopically.

[0040] Fig. 3(B) is a schematic diagram showing an example of an image perceived by a user when viewing the left display image in Fig. 3(A) with the left eye and the right display image in Fig. 3(A) with the right eye. As shown in Figs. 3(A) and 3(B), viewing with both eyes allows a wider range to be observed in the horizontal direction (left-right direction) of the HMD 100 than viewing with one eye.

[0041] However, without gradation processing, black band-like noise may be visible at the boundary between the overlapping and non-overlapping areas, as shown in Figure 3(B). For example, the luminance contrast between the overlapping and non-overlapping areas changes suddenly, causing retinal rivalry and resulting in the appearance of black band-like noise.

[0042] Gradation processing is processing that reduces changes in brightness contrast between overlapping and non-overlapping portions. FIG. 3C is a schematic diagram showing an example of a left display image and a right display image when gradation processing is performed. In the left display image, gradation processing is performed to gradually change the brightness value from the right end of the overlapping portion toward the left side (inside), and in the right display image, gradation processing is performed to gradually change the brightness value from the left end of the overlapping portion toward the right side (inside). Note that, in the left display image, gradation processing may be performed to gradually change the brightness value from the left end of the overlapping portion toward the right side (inside), and in the right display image, gradation processing may be performed to gradually change the brightness value from the right end of the overlapping portion toward the left side. It is sufficient that one or more of these four gradation processing methods are performed.

[0043] Fig. 3(D) is a schematic diagram showing an example of an image perceived by a user when viewing the left display image in Fig. 3(C) with the left eye and the right display image in Fig. 3(C) with the right eye. As shown in Fig. 3(D), the gradation processing reduces the change in brightness contrast between the overlapping and non-overlapping areas, thereby suppressing the occurrence of retinal rivalry and the perception of black band-like noise.

[0044] Using Figures 4(A) to 4(E), we will explain how the gazed object appears and how conspicuous the black band (black band-like noise) is depending on the position and size of the gazed object. Here, to take into account the three-dimensional position of the gazed object, the user's observation distance and the horizontal position of the gazed object are used. The user's observation distance is the distance from the user to the position (gaze object) in the depth direction (front-to-back direction of the HMD 100) where the user is looking, and may be the left gaze distance or the right gaze distance. The information processing unit 204 determines whether the left gaze distance is the distance from the user to the position (gaze object) where the user is looking, in the depth direction (front-to-back direction of the HMD 100), and may be the left gaze distance or the right gaze distance. Based on the line information and right gaze information, the distance from the user to the intersection of the left eye's gaze and the right eye's gaze may be calculated as the observation distance. The information processing unit 204 may calculate the depth direction component of this distance as the observation distance. The observation distance information may or may not be part of the gaze object information described above. The horizontal position of the gaze object is the position of the gaze object in the horizontal direction of the HMD 100. Here, the width of the gaze object (the size of the gaze object in the horizontal direction of the HMD 100) is used as the size of the gaze object.

[0045] FIG. 4(A) is a schematic diagram showing an example of the range viewed by a user in an MR space. The solid line indicates the range of the left display image, and the dashed line indicates the range of the right display image. Area 401 is the overlapping portion of the left display image and the right display image, area 402 is the non-overlapping portion of the left display image, and area 403 is the non-overlapping portion of the right display image. The user can freely change the viewing distance when viewing the left display image and the right display image. Arrow 411 indicates a short viewing distance, and arrow 412 indicates a long viewing distance.

[0046] 4(B) is a schematic diagram showing an example in which a narrow gaze object 420 is present at a short observation distance 411. The gaze object 420 is contained within the overlapping portion 401 and is sufficiently far from the boundary between the overlapping portion 401 and the non-overlapping portions 402 and 403. Therefore, even without gradation processing, the black band is not noticeable. However, gradation processing makes the gaze object appear dark.

[0047] 4(C) is a schematic diagram showing an example in which a wide gaze object 430 is present at the same short observation distance 411 as in FIG. 4(B). The gaze object 430 does not fit within the overlapping portion 401, but straddles the boundary between the overlapping portion 401 and non-overlapping portions 402 and 403. For this reason, if gradation processing is not performed, the black bands will be noticeable.

[0048] 4(D) is a schematic diagram showing an example in which a gaze object 430 with the same large width as in FIG. 4(C) is present at a long observation distance 412. The gaze object 430 is contained within the overlapping portion 401 and is sufficiently far from the boundary between the overlapping portion 401 and the non-overlapping portions 402 and 403. Therefore, even without gradation processing, the black band is not noticeable. However, gradation processing makes the gaze object appear darker.

[0049] FIG. 4(E) is a schematic diagram showing an example in which a narrow-width gaze object 420 is present at a short observation distance 411, similar to FIG. 4(C). In FIG. 4(E), the horizontal position of the gaze object 420 is different from that in FIG. 4(C). The gaze object 420 does not fit within the overlapping portion 401, but straddles the boundary between the overlapping portion 401 and the non-overlapping portion 403. Therefore, without gradation processing, the black band (the black band corresponding to the boundary between the overlapping portion 401 and the non-overlapping portion 403) becomes conspicuous. Note that the gaze object 420 is sufficiently far from the boundary between the overlapping portion 401 and the non-overlapping portion 402, and the black band corresponding to the boundary between the overlapping portion 401 and the non-overlapping portion 402 is not conspicuous. Therefore, it is sufficient to perform gradation processing to make the black band corresponding to the boundary between the overlapping portion 401 and the non-overlapping portion 403 less noticeable, and it is not necessary to perform gradation processing to make the black band corresponding to the boundary between the overlapping portion 401 and the non-overlapping portion 402 less noticeable.

[0050] Therefore, in the first embodiment, when the area of ​​the gaze object does not fall within a specific range in each of the left and right composite images, a gradation process with a higher intensity is determined than when the area does not fall within a specific range. The specific range in the left composite image is a range in the overlapping portion of the left composite image where the distance from the left edge and the right edge is longer than a threshold. The specific range in the right composite image is a range in the overlapping portion of the right composite image where the distance from the left edge and the right edge is longer than a threshold.

[0051] FIG. 5 is a flowchart showing an example of the operation (operation for determining gradation processing) according to the first embodiment.

[0052] In S501, the information processing unit 204 acquires information about the user's viewing distance. As described above, the information processing unit 204 may use information about the left gaze distance acquired from the HMD 100 as information about the viewing distance, or may use information about the right gaze distance acquired from the HMD 100 as information about the viewing distance. The information processing unit 204 may calculate, as the viewing distance, the distance from the user to the intersection of the left eye's line of sight and the right eye's line of sight based on the left gaze information and the right gaze information. The information processing unit 204 may also calculate, as the viewing distance, the depth direction component of these distances.

[0053] In S502, the information processing unit 204 acquires information on the width and horizontal position of the gaze object in the MR space. As described above, the information processing unit 204 can acquire information on the width and horizontal position of the gaze object based on the left composite image, the right composite image, the left gaze position, the right gaze position, etc.

[0054] In S503, the information processing unit 204 determines whether the width of the gaze object acquired in S502 is equal to or less than a threshold, and if the width is greater than the threshold, determines whether the horizontal position of the gaze object is within an allowable range. If the width is greater than the threshold or the horizontal position is outside the allowable range, the area of ​​the gaze object does not fall within a specific range in each of the left and right composite images. If the width is equal to or less than the threshold or the horizontal position is within the allowable range, the area of ​​the gaze object falls within a specific range in each of the left and right composite images. If the width is equal to or less than the threshold or the horizontal position is within the allowable range, the process proceeds to S504; otherwise, the process proceeds to S505.

[0055] The threshold value of the width of the gaze object is determined according to the observation distance acquired in S501, and the allowable range of the horizontal position of the gaze object is determined according to the observation distance and the width of the gaze object. As the threshold value of the width of the gaze object, a plurality of values ​​corresponding to a plurality of observation distances may be stored in advance in a storage medium such as a memory (not shown) of the information processing device 200. As the allowable range of the horizontal position of the gaze object, a plurality of ranges corresponding to a plurality of combinations of the observation distance and the width of the gaze object may be stored in advance in the storage medium. The information processing unit 204 may determine the threshold value and the allowable range using a predetermined function.

[0056] Note that the method of determining whether the region of the gaze object in each of the left composite image and the right composite image is within a specific range is not limited to the above method. For example, the width of the region of the gaze object in the left composite image or the right composite image may be detected, and it may be determined whether the width is equal to or smaller than a threshold. Alternatively, the left and right edges of the region of the gaze object in the left composite image or the right composite image may be detected, and it may be determined whether the left and right edges are within an allowable range. In these cases, a fixed threshold and allowable range may be used regardless of the observation distance.

[0057] In S504, the information processing unit 204 reads out from the parameter DB 205 left gradation processing data and right gradation processing data that do not involve gradation processing.

[0058] In S505, the information processing unit 204 reads out, from the parameter DB 205, left gradation processing data and right gradation processing data for performing the gradation processing.

[0059] FIG. 6A is a schematic diagram showing an example of a left display image and a right display image. A gaze object 601 exists in each of the left display image and the right display image. In FIG. 6A, the area of ​​the gaze object 601 falls within a specific range in each of the left display image and the right display image. Therefore, no gradation processing is performed on either the left or right display image.

[0060] 6(B) is a schematic diagram showing an example of an image perceived by a user when viewing the left display image in FIG. 6(A) with the left eye and the right display image in FIG. 6(A) with the right eye. Because gradation processing is not performed, the perception of black bands is not suppressed. However, because the area of ​​the gaze object 601 falls within a specific range, the black bands are not noticeable. By not performing gradation processing, the gaze object 601 is prevented from appearing dark.

[0061] 6(C) is a schematic diagram showing an example of a left display image and a right display image. A gaze object 602 exists in both the left display image and the right display image. In FIG. 6(C), it is assumed that the area of ​​the gaze object 602 does not fall within a specific range in either the left display image or the right display image. Therefore, gradation processing is performed on both the left display image and the right display image.

[0062] Fig. 6(D) is a schematic diagram showing an example of an image perceived by a user when viewing the left display image of Fig. 6(C) with the left eye and the right display image of Fig. 6(C) with the right eye. By performing gradation processing, the perception of black bars is suppressed (the black bars are made less noticeable).

[0063] Although the example of switching between execution and non-execution of gradation processing has been described, this is not limiting. For example, left and right gradation processing data for performing gradation processing at a first intensity may be read in S504, and left and right gradation processing data for performing gradation processing at a second intensity higher than the first intensity may be read in S505. As the intensity of the gradation processing increases, for example, the area to be subjected to gradation processing may be enlarged, or the brightness range of the gradation applied by the gradation processing may be expanded. The brightness change pattern in the gradation may change depending on the intensity (e.g., a pattern in which brightness changes linearly (straight line) with respect to changes in horizontal position, or a pattern in which brightness changes nonlinearly (curvilinearly) with respect to changes in horizontal position). The intensity may be changed in more than two stages. For example, if the area of ​​the focused object is within the overlapping area, the closer the area of ​​the focused object is to the boundary between the overlapping area and the non-overlapping area, the stronger the intensity of the gradation processing may be. In this way, gradation processing may be determined in which at least one of the area to be subjected to gradation processing, the brightness range of the gradation, and the brightness change pattern of the gradation differs depending on the relationship between the area of ​​the focused object and the specific range.

[0064] As described above, according to the first embodiment, when the area of ​​the gaze object in each of the left and right composite images does not fall within a specific range, a gradation process with a higher intensity is determined than when this is not the case. As a result, the gradation process can make the boundary between the overlapping and non-overlapping areas less noticeable, and can suppress changes in image brightness due to the gradation process when the boundary is not noticeable.

[0065] (Second embodiment) A second embodiment of the present invention will be described. Note that, below, explanations of the same points as in the first embodiment (for example, the same configurations and processes as in the first embodiment) will be omitted, and only points different from the first embodiment will be described.

[0066] The brighter the image viewed by the user, the more noticeable the black band corresponding to the boundary between the overlapping and non-overlapping portions. Therefore, in the second embodiment, information on the brightness of at least one of the left composite image and the right composite image is acquired, and when the brightness is high, a narrower range is used as the specific range than when the brightness is low. By doing so, when the brightness is high, it becomes more difficult for the area of ​​the gaze object to fit into the specific range than when the brightness is low, and the gradation processing (high intensity) As a result, it is possible to prevent the black bars from becoming conspicuous due to the lack of gradation processing (at high intensity).

[0067] FIG. 7 is a flowchart showing an example of the operation (operation for determining gradation processing) according to the second embodiment.

[0068] In S701, the information processing unit 204 acquires luminance information (luminance information) of at least one of the left composite image and the right composite image. For example, based on the left composite image and the right composite image, the information processing unit 204 estimates a luminance distribution (luminance distribution of a perceived image having non-overlapping portions on the left and right of an overlapping portion) that makes the left composite image and the right composite image similar when viewed by a user. Then, based on the estimated luminance distribution, the information processing unit 204 calculates, as the luminance information, the ratio of regions (high luminance regions) whose luminance is equal to or greater than a threshold value relative to the entire perceived image.

[0069] The method of acquiring the luminance information is not limited to the above method. For example, information indicating the proportion of high-luminance areas in each of the overlapping portion and the non-overlapping portion may be acquired as the luminance information. In this case, the threshold for determining whether the luminance (luminance of the high-luminance area) is high may be different between the overlapping portion and the non-overlapping portion. The proportion of high-luminance areas in a specific area (for example, only one of the overlapping portion and the non-overlapping portion) may be acquired as the luminance information. The proportion of high-luminance areas in the left composite image (or the right composite image) in the entire left composite image (or the right composite image) may be acquired as the luminance information. Information indicating the representative luminance of the entire perceived image (average luminance, median luminance, poorest luminance, maximum luminance, minimum luminance, etc.), the representative luminance of the entire left composite image, the representative luminance of the right composite image, the representative luminance of a specific area, etc. may be acquired as the luminance information. The information processing unit 204 may acquire the luminance information from an external device such as the HMD 100.

[0070] In S702, the information processing unit 204 determines whether the value of the luminance information acquired in S701 is equal to or less than a threshold value. If the value of the luminance information is equal to or less than the threshold value (if the luminance is low), the process proceeds to S704; if not (if the luminance is high), the process proceeds to S703. The threshold value is stored in advance in a storage medium such as a memory (not shown) of the information processing device 200. Note that if the luminance information indicates multiple values, the process may proceed to S704 if all values ​​indicated by the luminance information are equal to or less than the threshold value, or may proceed to S703 if not. The process may proceed to S704 if at least N values ​​(N is an integer greater than or equal to 1) indicated by the luminance information are equal to or less than the threshold value, or may proceed to S703 if not.

[0071] In S703, the information processing unit 204 changes the correspondence relationship between the observation distance and the threshold value of the width of the gaze object so that each threshold value of the width of the gaze object increases. Also, the information processing unit 204 changes the correspondence relationship between the combination of the observation distance and the width of the gaze object and the allowable range so that each allowable range narrows. As a result, the above-mentioned specific range narrows.

[0072] S705 to S709 are the same as S501 to S505 in the first embodiment (FIG. 5).

[0073] Fig. 8(A) is a schematic diagram showing an example of a left display image and a right display image. In Fig. 8(A), a gaze object 801, which is the same as gaze object 601 in Fig. 6(A), exists in each of the left display image and the right display image, in the same arrangement as in Fig. 6(A). Because the left display image and the right display image (left composite image and right composite image) in Fig. 6(A) are dark, the black bands are not noticeable even without gradation processing. On the other hand, because the left display image and the right display image (left composite image and right composite image) in Fig. 8(A) are bright, the black bands are noticeable without gradation processing.

[0074] In the second embodiment, in the case of FIG. 6A, the left composite image and the right composite image are dark, so the specific range cannot be narrowed. As a result, as in the first embodiment, the area of ​​the gaze object 601 falls within the specific range in each of the left display image and the right display image, and the gradation processing is performed. 8A, the left and right composite images are bright, so the specific range is narrowed. As a result, the area of ​​the gaze object 801 in each of the left and right display images does not fit within the specific range, and gradation processing is performed.

[0075] Fig. 8(B) is a schematic diagram showing an example of an image perceived by a user when viewing the left display image in Fig. 8(A) with the left eye and the right display image in Fig. 8(A) with the right eye. By performing gradation processing, the perception of black bars is suppressed (the black bars are made less noticeable).

[0076] FIG. 8(C) is a schematic diagram showing an example of a left display image and a right display image. A gaze object 802 exists in each of the left display image and the right display image. In the case of FIG. 8(C), the left composite image and the right composite image are also bright, so the specific range is narrowed. However, gaze object 802 is smaller than gaze object 801 in FIG. 8(A), and is contained within the narrowed specific range. Therefore, gradation processing is not performed on either the left display image or the right display image.

[0077] 8(D) is a schematic diagram showing an example of an image perceived by a user when viewing the left display image in FIG. 8(C) with the left eye and the right display image in FIG. 8(C) with the right eye. Because gradation processing is not performed, the perception of black bands is not suppressed. However, because the area of ​​the gaze object 802 falls within a specific range, the black bands are not noticeable. By not performing gradation processing, the gaze object 802 is prevented from appearing dark.

[0078] Although the example in which the specific range is changed depending on the brightness of the image has been described, the present invention is not limited to this. For example, the strength of the gradation processing may be changed depending on the brightness of the image. When the brightness of the image is high, a gradation processing with a higher strength may be determined than when the brightness of the image is low.

[0079] As described above, according to the second embodiment, luminance information of at least one of the left composite image and the right composite image is acquired, and when the luminance is high, a narrower range is used as the specific range than when the luminance is low. By doing so, when the luminance is high, it becomes more difficult for the area of ​​the gaze object to fit into the specific range than when the luminance is low, and gradation processing becomes more likely to be performed (at high intensity). As a result, it is possible to prevent black bars from becoming conspicuous because gradation processing is not performed (at high intensity).

[0080] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0081] The above processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0082] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0083] (Other embodiments) The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program, or by a circuit that realizes one or more functions.

[0084] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a first acquisition means for acquiring a first image which is an image for the left eye and a second image which is an image for the right eye and a part of which corresponds to a part of the first image; a second acquiring means for acquiring information about an object that the user is viewing; a determination means for determining gradation processing to be performed on each of the first image and the second image; and When the area of ​​the object does not fall within a specific range in each of the first image and the second image, the determining means determines gradation processing with a higher intensity than when the area of ​​the object falls within the specific range in each of the first image and the second image. 1. An information processing device comprising: (Configuration 2) a third acquisition means for acquiring information about the user's line of sight; and 2. The information processing device according to configuration 1, wherein the second acquisition means acquires information about the object based on information about the line of sight. (Configuration 3) the second acquisition means acquires information about the position and size of the object in three-dimensional space based on the line-of-sight information; The determining means determines the gradation processing based on the position and size of the object. 3. The information processing device according to configuration 2. (Configuration 4) the specific range in the first image is a range in which a distance from a left end and a distance from a right end of a part of the first image corresponding to a part of the second image are longer than a threshold; The specific range in the second image is a range in which the distance from the left end and the distance from the right end of a part of the second image corresponding to a part of the first image are longer than the threshold. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 5) The corresponding portions of the first image and the second image are stereoscopically viewable portions. 5. The information processing device according to any one of configurations 1 to 4. (Configuration 6) The determining means At least one of the left and right edges of the part of the first image that corresponds to the part of the second image Gradation processing of the first image, gradually changing the brightness from the outside to the inside; and gradation processing of the second image that gradually changes brightness from at least one of the left end and the right end of a portion of the second image corresponding to the portion of the first image toward the inside. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) The determining means determining not to perform the gradation processing when the area of ​​the object in each of the first image and the second image is within the specific range; When the area of ​​the object in each of the first image and the second image does not fall within the specific range, it is determined that the gradation processing is to be performed. 7. The information processing device according to any one of configurations 1 to 6. (Configuration 8) The determining means determines, depending on the relationship between the area of ​​the object and the specific range, the area where the gradation processing is performed; The brightness range of the gradation to be applied in the gradation processing, and The luminance change pattern in the gradation At least one of the following determines the different gradation processing: 8. The information processing device according to any one of configurations 1 to 7. (Configuration 9) a fourth acquiring means for acquiring luminance information of at least one of the first image and the second image; and When the luminance is high, the determining means uses a narrower range as the specific range than when the luminance is low. 9. The information processing device according to any one of configurations 1 to 8. (Configuration 10) a processing means for performing the gradation processing on each of the first image and the second image at the intensity determined by the determining means; Further having 10. The information processing device according to any one of configurations 1 to 9. (method) a first acquisition step of acquiring a first image that is an image for the left eye and a second image that is an image for the right eye and a portion of which corresponds to a portion of the first image; a second acquisition step of acquiring information about the object the user is looking at; a determination step of determining gradation processing to be performed on each of the first image and the second image; and In the determining step, when the area of ​​the object does not fall within a specific range in each of the first image and the second image, a gradation process with a higher intensity is determined than when the area of ​​the object falls within the specific range in each of the first image and the second image. 2. A method for controlling an information processing apparatus comprising: (program) 11. A program for causing a computer to function as each means of the information processing device according to any one of configurations 1 to 10. [Explanation of symbols]

[0085] 200: Information processing device 201: Computer control unit 204: Information processing unit

Claims

1. a first acquisition means for acquiring a first image which is an image for the left eye and a second image which is an image for the right eye and a part of which corresponds to a part of the first image; a second acquisition means for acquiring information about an object that a user is viewing; a determination means for determining gradation processing to be performed on each of the first image and the second image; and When the area of ​​the object does not fall within a specific range in each of the first image and the second image, the determining means determines gradation processing with a higher intensity than when the area of ​​the object falls within the specific range in each of the first image and the second image.

1. An information processing device comprising:

2. a third acquisition means for acquiring information about the user's line of sight; and The information processing apparatus according to claim 1 , wherein the second acquisition means acquires the information about the object based on the information about the line of sight.

3. the second acquisition means acquires information about the position and size of the object in three-dimensional space based on the line-of-sight information; The determining means determines the gradation processing based on the position and size of the object.

3. The information processing apparatus according to claim 2, wherein:

4. the specific range in the first image is a range in which a distance from a left end and a distance from a right end of a part of the first image corresponding to a part of the second image are longer than a threshold; The specific range in the second image is a range in which the distance from the left end and the distance from the right end of a part of the second image corresponding to a part of the first image are longer than the threshold.

2. The information processing apparatus according to claim 1, wherein:

5. The part of the first image and the part of the second image that correspond to each other are parts that can be viewed stereoscopically.

2. The information processing apparatus according to claim 1, wherein:

6. The determining means gradation processing of the first image in which luminance is gradually changed from at least one of the left end and the right end of a portion of the first image corresponding to a portion of the second image toward the inside; and gradation processing of the second image that gradually changes brightness from at least one of the left end and the right end of a portion of the second image corresponding to the portion of the first image toward the inside.

2. The information processing apparatus according to claim 1, wherein:

7. The determining means determining not to perform the gradation processing when the area of ​​the object in each of the first image and the second image is within the specific range; When the area of ​​the object in each of the first image and the second image does not fall within the specific range, it is determined that the gradation processing is to be performed.

2. The information processing apparatus according to claim 1, wherein:

8. The determining means determines, depending on the relationship between the area of ​​the object and the specific range, the area where the gradation processing is performed; The brightness range of the gradation to be applied in the gradation processing, and The luminance change pattern in the gradation At least one of the following determines the different gradation processing:

2. The information processing apparatus according to claim 1, wherein:

9. a fourth acquiring means for acquiring information on the luminance of at least one of the first image and the second image; and When the luminance is high, the determining means uses a narrower range as the specific range than when the luminance is low.

2. The information processing apparatus according to claim 1, wherein:

10. a processing means for performing the gradation processing on each of the first image and the second image at the intensity determined by the determining means; Further having 2. The information processing apparatus according to claim 1, wherein:

11. a first acquisition step of acquiring a first image that is an image for the left eye and a second image that is an image for the right eye and a portion of which corresponds to a portion of the first image; a second acquisition step of acquiring information about an object that the user is viewing; a determining step of determining gradation processing to be performed on each of the first image and the second image; and In the determining step, when the area of ​​the object does not fall within a specific range in each of the first image and the second image, a gradation process with a higher intensity is determined than when the area of ​​the object falls within the specific range in each of the first image and the second image.

2. A method for controlling an information processing apparatus comprising:

12. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 10.

Images