Image processing apparatus

Abstract

To provide a technique capable of obtaining a suitable image (an image that can achieve natural observation over a wide range) even when performing image processing that reduces (suppresses) the delay in changing a display range.SOLUTION: An image processing apparatus of the present invention comprises: first acquisition means for acquiring a first image which is an image of a range based on at least one of a position and a posture of a user's head and a second image which is an image of a range based on at least one of a position and a posture of the head and a part of which corresponds to a part of the first image; second acquisition means for acquiring information regarding the movement of the head; first processing means for generating a third image by moving or transforming the first image and generating a fourth image by moving or transforming the second image based on the information; and second processing means for performing gradation processing on each of the third image and the fourth image. The second processing means performs the gradation processing based on the information.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an image processing device. [Background technology]

[0002] Mixed Reality (MR) technology is known as a technology that seamlessly blends the real world and the virtual world in real time. A known MR technology is a technology that captures the real space (subject) in front of the HMD (Head Mounted Display) with a video camera, and presents (displays) an MR image in which CG (Computer Graphics) is superimposed on the image of the real space to the user. This type of HMD is called a video see-through HMD.

[0003] In an HMD, it is desirable for a user to be able to see a wide range (angle of view) of images so that they can feel highly realistic. A wide range of images can be seen by using an image for the left eye (left image) and an image for the right eye (right image) that satisfy the following conditions. The image area on the right side of the left image (the nose side of the user wearing the HMD) and the image area on the left side of the right image (the nose side of the user wearing the HMD) correspond to each other and are overlapping areas. The image area on the left side of the left image (the left ear side of the user wearing the HMD) and the image area on the right side of the right image (the right ear side of the user wearing the HMD) are non-overlapping areas that do not correspond to each other.

[0004] When the left and right images that satisfy the above-mentioned conditions are displayed, the non-overlapping portion of the left image is only visible to the left eye, and the right eye sees the overlapping portion as a black portion such as a frame of the display. Similarly, the non-overlapping portion of the right image is only visible to the right eye, and the left eye sees the overlapping portion as a black portion. This causes binocular rivalry between the left and right eyes. The boundary between the non-overlapping portion and the overlapping portion stands out due to binocular rivalry, preventing the user from observing naturally.

[0005] Patent Document 1 discloses an HMD that reduces the brightness of an image from a predetermined position according to a predetermined decreasing curve so that black areas become less noticeable. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2003-337299 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, when changing the range of the image (display range) in response to the movement of the user's head, there is a delay in the change of the display range relative to the movement of the head, so image processing may be performed to move the image in a direction that counteracts the movement of the head in order to reduce (suppress) the delay. If this image processing is performed with the HMD disclosed in Patent Document 1, the boundary between the non-overlapping portion and the overlapping portion changes, so the boundary cannot be made inconspicuous and the user cannot observe naturally.

[0008] The present invention aims to provide a technology that can obtain a suitable image (an image that enables natural observation over a wide range) even when performing image processing that reduces (suppresses) the delay in changes in the display range. [Means for solving the problem]

[0009] A first aspect of the present invention is an image processing device comprising a first acquisition means for acquiring a first image, which is an image for the left eye and is an image of a range based on at least one of the position and posture of a user's head, and a second image, which is an image for the right eye and is an image of a range based on at least one of the position and posture of the head, a part of which corresponds to a part of the first image, a second acquisition means for acquiring information related to movement of the head, a first processing means for moving or deforming the first image based on the information to generate a third image and moving or deforming the second image to generate a fourth image, and a second processing means for performing gradation processing on each of the third image and the fourth image, wherein the second processing means performs the gradation processing based on the information.

[0010] A second aspect of the present invention is an image processing method comprising the steps of acquiring a first image, which is an image for the left eye and is an image of a range based on at least one of the position and posture of a user's head, and a second image, which is an image for the right eye and is an image of a range based on at least one of the position and posture of the head, a part of which corresponds to a part of the first image; acquiring information related to the movement of the head; moving or deforming the first image based on the information to generate a third image, and moving or deforming the second image to generate a fourth image; and performing gradation processing on each of the third image and the fourth image, wherein the gradation processing is performed based on the information.

[0011] A third aspect of the present invention is a program for causing a computer to function as each of the means of the image processing device described above. Effect of the Invention

[0012] According to the present invention, even when performing image processing that reduces (suppresses) the delay in the change of the display range, it is possible to obtain a suitable image (an image that allows natural observation over a wide range). [Brief description of the drawings]

[0013] [Figure 1] 1 is a schematic diagram showing an example of the configuration of a system according to a first embodiment. [Diagram 2] 1 is a block diagram showing an example of the configuration of a system according to a first embodiment. [Diagram 3] FIG. 4 is a schematic diagram for explaining gradation processing according to the first embodiment. [Figure 4] FIG. 2 is a schematic diagram for explaining a problem to be solved by the first embodiment. [Diagram 5] 4 is a flowchart showing an operation example according to the first embodiment. [Figure 6] 2A to 2C are schematic diagrams showing examples of various images according to the first embodiment. [Figure 7] FIG. 11 is a schematic diagram for explaining a problem to be solved by the second embodiment. [Figure 8] 13 is a flowchart showing an operation example according to the second embodiment. [Figure 9] 13A to 13C are schematic diagrams showing examples of various images according to the second embodiment. [Figure 10] FIG. 13 is a block diagram showing an example of the configuration of a system according to a third embodiment. [Figure 11] 13 is a flowchart showing an operation example according to the third embodiment. [Figure 12] 13A to 13C are schematic diagrams showing examples of various images according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] <<First embodiment>> A first embodiment of the present invention will be described below. An example in which the present invention is applied to MR (Mixed Reality) technology that presents (displays) to a user a composite image in which a CG object (virtual object) is superimposed on an image in real space, but the present invention is not limited to this. For example, the present invention may be applied to VR (Virtual Reality) technology that presents (displays) to a user a composite image in which a virtual object is superimposed on an image in virtual space.

[0015] <Configuration> FIG. 1 is a schematic diagram showing an example of the configuration of a system according to the first embodiment.

[0016] The system in FIG. 1 includes a head mounted display (HMD) 100 and an information processing device 200. The HMD 100 is worn on the user's head, and displays an image to provide the user with an MR experience. The information processing device 200 generates an image for the MR experience. A cable 300 is connected to the HMD 100 and the information processing device 200, and the HMD 100 and the information processing device 200 can transmit and receive data to and from each other via the cable 300. Note that the data transmitted and received is not limited to image data, and for example, sensor data, audio data, or control data may be transmitted and received. Sensor data output from an acceleration sensor or an angular velocity sensor provided in the HMD 100 may be transmitted from the HMD 100 to the information processing device 200. Control data for controlling the HMD 100 may be transmitted from the information processing device 200 to the HMD 100. An example in which wired communication is performed between the HMD 100 and the information processing device 200 using the cable 300 will be described, but wireless communication may also be performed between the HMD 100 and the information processing device 200.

[0017] The HMD 100 captures real space with a camera facing forward, and displays an MR image in which a real space image (camera image) is combined with a CG (CG image). The information processing device 200 determines a drawing position of the CG based on the camera image or tracking data, and performs rendering of the CG. In order to determine an appropriate position and orientation of the CG, a marker 500 may be placed in real space (e.g., a floor) and the position and orientation of the marker 500 may be detected from the camera image. This generates a CG (world coordinate CG) corresponding to coordinate 1 (world coordinate) of a world coordinate system uniquely determined based on the marker 500, and generates an MR image in which the world coordinate CG is combined with the camera image. Note that, although the world coordinate 1 is shown at (near) the position of the marker 500 in FIG. 1, the world coordinate 1 may be distant from the marker 500.

[0018] The information processing device 200 may be a stationary computer such as a personal computer or a workstation, or may be a mobile terminal such as a smartphone or a wearable device. The information processing device 200 may be a part of the HMD 100. The information processing device 200 may be divided into a plurality of devices. At least one of the plurality of functions of the information processing device 200 may be executed on a cloud (server) or may be executed in the HMD 100. Similarly, the HMD 100 may be divided into a plurality of devices. At least one of the plurality of functions of the HMD 100 may be executed on a cloud or may be executed in the information processing device 200. The image processing device according to the first embodiment may be provided in the HMD 100, in the information processing device 200, or in another device.

[0019] FIG. 2 is a block diagram showing an example of the configuration of the system shown in FIG.

[0020] The information processing device 200 has a CG position / posture determination unit 201, a rendering unit 202, a CG storage unit 203, a reprojection unit 204, a composition unit 205L, and a composition unit 205R.

[0021] The CG position / orientation determination unit 201 determines the position and orientation of the CG. For example, the CG position / orientation determination unit 201 detects the position and orientation of the marker 500 arranged in real space from a camera image (image of real space) obtained by a camera (imaging unit 101L or imaging unit 101R described later) of the HMD 100. Then, the CG position / orientation determination unit 201 detects the position and orientation of the HMD 100 in real space based on the position and orientation of the marker 500, and determines the position and orientation of the CG. At that time, the CG position / orientation determination unit 201 may detect world coordinates 1 of the CG based on the position and orientation of the marker 500, and convert the world coordinates 1 into camera coordinates (coordinates of a camera coordinate system based on the position and orientation of the camera). Since the marker 500 does not move in real space, the user wearing the HMD 100 Even if the user moves around, the relative positions of the HMD 100 and the marker 500 can be grasped and the world coordinates 1 can be converted into camera coordinates.

[0022] The method of determining the position and orientation of the CG is not limited to the above method. For example, the CG position / orientation determination unit 201 may detect feature points of an immovable object in real space, detect the position and orientation of the HMD 100 based on the feature points, and determine the position and orientation of the CG. The CG position / orientation determination unit 201 may detect feature points of the HMD 100 using an external camera, detect the position and orientation of the HMD 100 based on the feature points, and determine the position and orientation of the CG. The position and orientation of the CG can be determined using various tracking techniques for tracking the HMD 100. The CG position / orientation determination unit 201 may use an inside-out tracking technique or an outside-in tracking technique. The CG position / orientation determination unit 201 may determine the position and orientation of the CG based on the detection result (information on the position and orientation of the HMD 100) of the HMD position / orientation detection unit 104 described later.

[0023] The rendering unit 202 reads out CG (CG data) from the CG storage unit 203, and performs rendering of the read out CG in accordance with the position and orientation of the CG (CG camera coordinates) determined by the CG position / orientation determination unit 201. This generates a CG image. For example, the CG storage unit 203 stores a plurality of CGs, and the rendering unit 202 reads out from the CG storage unit 203 the CG that corresponds to world coordinates 1 detected by the CG position / orientation determination unit 201. The rendering method (algorithm, rendering engine) is not particularly limited, but for example, the rendering unit 202 performs polygon rendering (rendering that generates (draws) polygons one by one), which is a type of real-time rendering.

[0024] The rendering processing time depends on the performance of a GPU (not shown) provided in the information processing device 200 and the data size of the CG to be rendered. If the rendering processing time is long, the CG is displayed with a delay, and therefore when the user moves his / her head, the position change of the CG is delayed in relation to the head movement, causing the user to feel uncomfortable.

[0025] The reprojection unit 204 performs a CG reprojection process so as to reduce (suppress) the above-mentioned CG delay. The reprojection unit 204 acquires information on the position and orientation of the HMD 100 (HMD position / orientation information) from the HMD position / orientation detection unit 104 via the control unit 110 described later. By repeatedly acquiring the HMD position / orientation information, the HMD 100 can be tracked. The reprojection unit 204 performs a reprojection process to regenerate a CG image based on a plurality of HMD position / orientation information corresponding to a plurality of times respectively and the rendering processing time. For example, the reprojection unit 204 applies to the CG a change opposite to the change in the position and orientation of the HMD 100 during rendering (from the start of rendering to the end of rendering (start of reprojection)), and regenerates a CG image that matches the current position and orientation of the HMD 100.

[0026] In the first embodiment, a CG image for the left eye (left CG image) and a CG image for the right eye (right CG image) are generated. For example, when the left CG image is generated, the position and orientation of the marker 500 are detected from a camera image for the left eye (left camera image) obtained by the imaging unit 101L. When the right CG image is generated, the position and orientation of the marker 500 are detected from a camera image for the right eye (right camera image) obtained by the imaging unit 101R.

[0027] The synthesis unit 205L generates a synthetic image for the left eye (left synthetic image) by synthesizing the left camera image (image in real space) obtained by the imaging unit 101L with the left CG image regenerated by the reprojection unit 204. Then, the synthesis unit 205L transmits the left synthetic image (left synthetic image data) to the conversion unit 106L described later.

[0028] The synthesis unit 205R generates a synthesis image for the right eye (right synthesis image) by synthesizing the right camera image (image in real space) obtained by the imaging unit 101R with the right CG image regenerated by the reprojection unit 204. Then, the synthesis unit 205R transmits the right synthesis image (right synthesis image data) to the conversion unit 106R described later.

[0029] In the case of VR, the rendering unit 202 generates an image in which a virtual object is superimposed on a virtual space, so the imaging units 101R and 101L, the CG position / posture determination unit 201, and the synthesis units 205L and 205R do not need to be used. A process of detecting the position and posture of the HMD 100 is performed.

[0030] In the first embodiment, an image of a range based on the position and posture of the user's head is obtained as the composite image (left composite image and right composite image), but this is not limited to this. The range of the composite image may be a range based on at least one of the position and posture of the head. The range of the composite image may be a range based on the head position (a range not based on the posture of the head) or a range based on the posture of the head (a range not based on the position of the head).

[0031] The HMD 100 includes imaging units 101L and 101R, display units 102L and 102R, an HMD position / posture detection unit 104, conversion units 106L and 106R, gradation units 103L and 103R, and a control unit 110.

[0032] The imaging unit 101L captures a left camera image by capturing an image of the real space, and the imaging unit 101R captures a right camera image by capturing an image of the real space.

[0033] The HMD position / orientation detection unit 104 detects the position and orientation of the HMD 100. The HMD position / orientation detection unit 104 detects, for example, the rotation angle or tilt of the HMD 100 as the orientation of the HMD 100. The method of detecting the position and orientation of the HMD 100 is not particularly limited. For example, the HMD position / orientation detection unit 104 may detect the position and orientation of the HMD 100 using at least one of an acceleration sensor, an angular velocity sensor, and a geomagnetic sensor. The HMD position / orientation detection unit 104 may detect the position and orientation of the HMD 100 using the imaging unit 101L or the imaging unit 101R. The HMD position / orientation detection unit 104 may use an inside-out tracking technique or an outside-in tracking technique.

[0034] The control unit 110 controls the entire HMD 100. For example, the control unit 110 controls the processing of the conversion units 106L, 106R and the gradation units 103L, 103R by transmitting the detection result of the HMD position / orientation detection unit 104 to the conversion units 106L, 106R and the gradation units 103L, 103R. The control unit 110 also controls the processing of the reprojection unit 204 by transmitting the detection result of the HMD position / orientation detection unit 104 to the reprojection unit 204.

[0035] The video delay caused by the processing time of the rendering unit 202 is reduced by the reprojection process of the reprojection unit 204. However, video delay caused by the processing time of the CG position / orientation determination unit 201 and video delay caused by the processing time of the synthesis units 205L and 205R may also occur. Video delay caused by the processing time of the imaging units 101L and 101R and video delay caused by the processing time of the display units 102L and 102R may also occur. As various video delays exist, the video delay cannot be sufficiently reduced by the reprojection process alone.

[0036] The conversion unit 106L converts the HMD position / posture detection unit 104 into a Based on the detection result (HMD position / posture information) of the HMD position / posture detection unit 104, the conversion unit 106L moves or deforms the left composite image to generate a left converted image. For example, the conversion unit 106L moves the left composite image in a direction to cancel the head movement during the reprojection and composition period (from the end of rendering (start of reprojection) to the present (start of processing by the conversion unit 106L)). Similarly, the conversion unit 106R moves or deforms the right composite image to generate a right converted image based on the detection result (HMD position / posture information) of the HMD position / posture detection unit 104. In the first embodiment, it is assumed that the composite images (left composite image and right composite image) are moved. Note that the information used may be information related to the movement of the head, and does not have to be information indicating the position and posture of the HMD 100.

[0037] The gradation section 103L performs gradation processing on the left converted image to generate a left display image. The gradation section 103R performs gradation processing on the right converted image to generate a right display image. Different gradation processing is performed on the left converted image and the right converted image. The details of the gradation processing will be described later.

[0038] Display unit 102L displays a left display image. The user views the left display image displayed on display unit 102L with his left eye. Display unit 102R displays a right display image. The user views the right display image displayed on display unit 102R with his right eye. The method of displaying (presenting) the image is not particularly limited; for example, the image may be displayed on a display panel such as a liquid crystal panel or an organic EL panel arranged in a position facing the user's eyes, or the image may be projected directly onto the user's retina using a laser.

[0039] In the first embodiment, each of the left eye image and the right eye image has an overlapping portion which is an image area corresponding to each other between the images, and a non-overlapping portion which does not correspond to each other between the images. For example, each of the left composite image and the right composite image has an overlapping portion which is an image area corresponding to each other between the left composite image and the right composite image, and a non-overlapping portion which does not correspond to each other between the left composite image and the right composite image. Also, each of the left display image and the right display image has an overlapping portion which is an image area corresponding to each other between the left display image and the right display image, and a non-overlapping portion which does not correspond to each other between the left display image and the right display image. For example, the overlapping portion is a portion which can be viewed stereoscopically, and the non-overlapping portion is a portion which cannot be viewed stereoscopically. Since the processing of the conversion units 106L and 106R is performed, the overlapping portion between the left composite image and the right composite image is different from the overlapping portion between the left display image and the right display image. Similarly, the non-overlapping portion between the left composite image and the right composite image is different from the non-overlapping portion between the left display image and the right display image.

[0040] <Gradient processing> An example of the gradation process will be described with reference to FIGS. 3(A) to 3(D).

[0041] 3A is a schematic diagram showing an example of a left display image and a right display image without gradation processing. Each of the left display image and the right display image has an overlapping portion that is an image area between the left display image and the right display image that corresponds to each other, and a non-overlapping portion between the left display image and the right display image that does not correspond to each other.

[0042] Fig. 3(B) is a schematic diagram showing an example of an image perceived by a user when viewing the left display image in Fig. 3(A) with the left eye and the right display image in Fig. 3(A) with the right eye. As shown in Figs. 3(A) and 3(B), viewing with both eyes allows a wider range to be observed in the horizontal direction than viewing with one eye.

[0043] However, without gradation processing, black band-like noise may be seen at the boundary between the overlapping and non-overlapping areas, as shown in Fig. 3(B). For example, the luminance contrast between the overlapping and non-overlapping areas changes suddenly, causing retinal rivalry and resulting in the appearance of black band-like noise.

[0044] Gradation processing is a process that reduces the change in brightness contrast between overlapping and non-overlapping parts. Fig. 3(C) is a schematic diagram showing an example of a left display image and a right display image when gradation processing is performed. In the left display image, gradation processing is performed to gradually change the brightness value from the right end of the image to the left side, and in the right display image, gradation processing is performed to gradually change the brightness value from the left end of the image to the left side.

[0045] Fig. 3(D) is a schematic diagram showing an example of an image perceived by a user when the left display image in Fig. 3(C) is viewed with the left eye and the right display image in Fig. 3(C) is viewed with the right eye. As shown in Fig. 3(D), the gradation processing reduces the change in brightness contrast between the overlapping and non-overlapping parts, thereby suppressing the occurrence of retinal rivalry and the perception of black band-like noise.

[0046] <Challenges> When using the conversion units 106L and 106R, the conventional gradation processing may not provide a sufficient effect. This will be explained with reference to Figures 4(A) to 4(D). Here, consider the case where a user wearing the HMD 100 swings his head horizontally from right to left.

[0047] FIG. 4(A) is a schematic diagram showing an example of a left composite image and a right composite image, and FIG. 4(B) is a schematic diagram showing an example of a left converted image and a right converted image. Since the user shook his head horizontally from right to left, the conversion unit 106L obtains the left converted image of FIG. 4(B) by moving (shifting) the left composite image of FIG. 4(A) to the right by an amount of movement corresponding to the amount of head movement. Similarly, the conversion unit 106R obtains the right converted image of FIG. 4(B) by moving the right composite image of FIG. 4(A) to the right by an amount of movement corresponding to the amount of head movement. By moving the image in the opposite direction to the head movement direction, the video delay felt by the user can be reduced. There is no image to be displayed at the left end of the left converted image and the left end of the right converted image. Therefore, as shown in FIG. 4(B), the left end of the left converted image and the left end of the right converted image are black areas.

[0048] Fig. 4(C) is a schematic diagram showing an example of a left display image obtained by applying gradation processing to the left converted image of Fig. 4(B) and a right display image obtained by applying gradation processing to the right converted image of Fig. 4(B). As in Fig. 3(C), the left display image is subjected to gradation processing that gradually changes the luminance value from the right end of the image to the left, and the right display image is subjected to gradation processing that gradually changes the luminance value from the left end of the image to the left. In the right converted image, the left end of the image is a black area, so gradation processing is applied to the black area.

[0049] FIG. 4(D) is a schematic diagram showing an example of an image perceived by a user when the left display image in FIG. 4(C) is viewed with the left eye and the right display image in FIG. 4(C) is viewed with the right eye. Since the left converted image and the right converted image are shifted to the right by the conversion units 106L and 106R, the left boundary (left end of the overlapping portion) of the two boundaries between the overlapping portion and the non-overlapping portion is shifted to the right. Then, in the right converted image, the gradation process is not performed on the left boundary (left end of the overlapping portion), but the gradation process is performed on the black area at the left end of the image. As a result, as shown in FIG. 4(D), black band-like noise is seen at the left boundary.

[0050] Therefore, in the first embodiment, gradation processing is performed on both of the two boundaries between the overlapping and non-overlapping areas (both the left and right ends of the overlapping areas) based on HMD position / posture information (information regarding head movement).

[0051] <Operation> An operation example according to the first embodiment will be described with reference to Figures 5 and 6(A) to 6(D). Figure 5 is a flowchart showing an operation example according to the first embodiment, and Figures 6(A) to 6(D) are schematic diagrams showing examples of various images according to the first embodiment. The control unit 110 executes the process of Figure 5 every time the image to be displayed is updated.

[0052] In S501 , the control unit 110 acquires the latest (current) HMD position / posture information Q from the HMD position / posture detection unit 104 .

[0053] In S502, the control unit 110 acquires the held past HMD position / orientation information R. The HMD position / orientation information R is, for example, the HMD position / orientation information at the end of rendering (at the start of reprojection).

[0054] The control unit 110 can detect a change in the position and orientation of the HMD 100 during a specific period based on the change from the HMD position / orientation information R to the HMD position / orientation information Q. Then, the control unit 110 can determine the movement (movement direction and amount) of the image to reduce video delay based on the change in the position and orientation of the HMD 100.

[0055] In S503, the control unit 110 determines image shift values ​​SX and SY indicating the movement direction and movement amount of the image based on the change from the HMD position / posture information R to the HMD position / posture information Q. The positive and negative of the image shift value SX indicate the left / right movement direction, and the absolute value of the image shift value SX indicates the left / right movement amount. The positive and negative of the image shift value SY indicate the up / down movement direction, and the absolute value of the image shift value SY indicates the up / down movement amount. In the first embodiment, the image shift value SX is a positive value when the image is moved to the right (when it is determined that there has been a head movement to the left), and the image shift value SX is a negative value when the image is moved to the left (when it is determined that there has been a head movement to the right). In addition, the image shift value SY is a positive value when the image is moved down (when it is determined that there has been a head movement to the up), and the image shift value SY is a negative value when the image is moved up (when it is determined that there has been a head movement to the down).

[0056] In S504, the control unit 110 sets the image shift values ​​SX and SY in the conversion units 106L and 106R, respectively.

[0057] FIG. 6(A) is a schematic diagram showing an example of a left composite image and a right composite image, and FIG. 6(B) is a schematic diagram showing an example of a left converted image and a right converted image. Here, it is assumed that a user wearing the HMD 100 swings his head horizontally from right to left. Therefore, the control unit 110 sets the image shift value SX=positive value and the image shift value SY=0 for each of the conversion unit 106L and the conversion unit 106R. Then, the conversion unit 106L moves (shifts) the left composite image in FIG. 6(A) to the right by an amount of movement corresponding to the absolute value of the image shift value SX to obtain the left converted image in FIG. 6(B). Similarly, the conversion unit 106R moves the right composite image in FIG. 6(A) to the right by an amount of movement corresponding to the absolute value of the image shift value SX to obtain the right converted image in FIG. 6(B). No image to be displayed exists at the left end of the left converted image and the left end of the right converted image. Therefore, as shown in FIG. 6(B), the left end portion of the left converted image and the left end portion of the right converted image are both black areas.

[0058] In S505, the control unit 110 judges whether the image shift value SX is a positive value or not. If the image shift value SX is a positive value (if the image is moved to the right, and it is judged that there has been a head movement to the left), the process proceeds to S506, and if not, the process proceeds to S507.

[0059] In S506, the control unit 110 sets image shift values ​​SX and SY for the gradation portion 103R. In S507, the control unit 110 sets image shift values ​​SX and SY for the gradation portion 103L. Set image shift values ​​SX and SY for

[0060] In this way, when the image shift value SX is a positive value (when the left converted image and the right converted image are shifted to the right by the conversion units 106L and 106R and it is determined that there has been a head movement to the left), the gradation processing performed on the right converted image is changed. When the image shift value SX is a negative value (when the left converted image and the right converted image are shifted to the left by the conversion units 106L and 106R and it is determined that there has been a head movement to the right), the gradation processing performed on the left converted image is changed. In the first embodiment, when the image shift value SX is a positive value, the area in the right converted image where the gradation processing is performed is moved to the right, and when the image shift value SX is a negative value, the area in the left converted image where the gradation processing is performed is moved to the left.

[0061] Fig. 6(C) is a schematic diagram showing an example of a left display image obtained by applying gradation processing to the left converted image of Fig. 6(B) and a right display image obtained by applying gradation processing to the right converted image of Fig. 6(B). Since the image shift value SX is a positive value, the gradation processing of the gradation section 103L is not changed, and the area where the gradation processing of the gradation section 103R is performed is shifted to the right. Therefore, in the left display image, the gradation processing is performed on the right edge of the image, and in the right display image, the gradation processing is performed on the part adjacent to the right side of the black area, which is the left edge of the image.

[0062] In each of the left converted image and the left displayed image, the right edge of the image is the right edge of the two boundaries between the overlapping portion and the non-overlapping portion (the right edge of the overlapping portion). In each of the right converted image and the right displayed image, the above-mentioned portion adjacent to the right side of the black region, which is the left edge of the image, is the left edge of the two boundaries (the left edge of the overlapping portion). Therefore, in FIG. 6C, gradation processing is performed on both of the two boundaries between the overlapping portion and the non-overlapping portion (both the left edge and the right edge of the overlapping portion).

[0063] Each of the left converted image and the left display image includes a valid area (an area where the image to be displayed exists, an area other than the black area) and an invalid area (an area where the image to be displayed does not exist, a black area) according to the movement of the head. Similarly, each of the right converted image and the right display image includes a valid area and an invalid area according to the movement of the head. In the left converted image (left display image), the right end of the image may be interpreted as the end of the effective area of ​​the left converted image (left display image) on the side of the overlapping part (a part corresponding to a part of the right converted image (right display image)). In the right converted image (right display image), the above-mentioned part adjacent to the right side of the black area, which is the left end of the image, may be interpreted as the end of the effective area of ​​the right converted image (right display image) on the side of the overlapping part (a part corresponding to a part of the left converted image (left display image)). The gradation process performed on the left converted image may be interpreted as a gradation process that gradually changes the luminance value from the end of the effective area of ​​the left converted image on the side of the overlapping part (a part corresponding to a part of the right converted image) toward the inside of the effective area. The gradation processing performed on the right converted image may also be interpreted as gradation processing that gradually changes the brightness value from the end of the effective area of ​​the right converted image on the side of the overlapping portion (the portion corresponding to part of the left converted image) toward the inside of the effective area.

[0064] Fig. 6(D) is a schematic diagram showing an example of an image perceived by a user when the left display image in Fig. 6(C) is viewed with the left eye and the right display image in Fig. 6(C) is viewed with the right eye. By changing the area where gradation processing is performed according to the change in the overlapping portion, gradation processing is performed on both of the two boundaries between the overlapping portion and the non-overlapping portion (both the left and right ends of the overlapping portion). As a result, as shown in Fig. 6(D), the perception of black band-like noise can be suitably suppressed.

[0065] In addition, the change in brightness contrast at the right end of the superimposed portion is reduced for the left converted image. Although an example of performing gradation processing to reduce the change in brightness contrast at the left end of the superimposed portion on the right converted image has been described, the present invention is not limited to this. For example, as shown in FIG. 6(E), gradation processing to reduce the change in brightness contrast at the left end of the superimposed portion on the left converted image may be performed, and gradation processing to reduce the change in brightness contrast at the right end of the superimposed portion on the right converted image may be performed. In this case, if the image shift value SX is a positive value (if the left converted image and the right converted image are shifted to the right by the conversion units 106L and 106R and it is determined that there has been a head movement to the left), the gradation processing to be performed on the left converted image is changed. If the image shift value SX is a negative value (if the left converted image and the right converted image are shifted to the left by the conversion units 106L and 106R and it is determined that there has been a head movement to the right), the gradation processing to be performed on the left converted image is changed.

[0066] In S508, the control unit 110 holds the HMD position / posture information Q as past information.

[0067] As described above, according to the first embodiment, gradation processing is performed taking into account the movement of the head. This makes it possible to obtain a suitable image (an image that allows for natural observation of a wide range) even when the image is moved as image processing for reducing (suppressing) delays in changes in the display range.

[0068] <<Second embodiment>> The second embodiment of the present invention will be described below. Note that, in the following, the same points as the first embodiment (for example, the same configuration and processing as the first embodiment) will not be described, and only the points different from the first embodiment will be described.

[0069] In the first embodiment, an example in which an image is moved in the conversion unit (conversion unit 106L and conversion unit 106R, respectively) will be described. In the second embodiment, an example in which an image is deformed (projective transformation) in the conversion unit (conversion unit 106L and conversion unit 106R, respectively) will be described.

[0070] <Challenges> Problems with conventional gradation processing will be described with reference to Figures 7(A) to 7(D). Here, a case will be considered in which a user wearing the HMD 100 swings his / her head horizontally from right to left.

[0071] FIG. 7(A) is a schematic diagram showing an example of a left composite image and a right composite image, and FIG. 7(B) is a schematic diagram showing an example of a left transformed image and a right transformed image. Since the user shook his head horizontally from right to left, the transforming unit 106L obtains the left transformed image of FIG. 7(B) by transforming the left composite image of FIG. 7(A) with a transformation amount according to the amount of head movement. Similarly, the transforming unit 106R obtains the right transformed image of FIG. 7(B) by transforming the right composite image of FIG. 7(A) with a transformation amount according to the amount of head movement. A transformation (projective transformation) is performed to move the image to the right, move the left side of the image to the back, and move the right side of the image to the front. This makes it possible to realize a more natural image change. Since the projective transformation matrices are different between the transforming unit 106L and the transforming unit 106R, the black areas where no image is displayed are also different.

[0072] Fig. 7(C) is a schematic diagram showing an example of a left display image obtained by applying gradation processing to the left converted image of Fig. 7(B) and a right display image obtained by applying gradation processing to the right converted image of Fig. 7(B). In the left display image, gradation processing is performed to gradually change the luminance value from the right end of the image to the left side, and in the right display image, gradation processing is performed to gradually change the luminance value from the left end of the image to the left side.

[0073] Fig. 7(D) is a schematic diagram showing an example of an image perceived by a user when the left display image in Fig. 7(C) is viewed with the left eye and the right display image in Fig. 7(C) is viewed with the right eye. The overlapping portion changes due to the processing of the conversion units 106L and 106R, and the region where the brightness contrast changes suddenly changes, so that black band-like noise is seen at the boundary between the overlapping portion and the non-overlapping portion.

[0074] <Operation> An operation example according to the second embodiment will be described with reference to Figures 8 and 9(A) to 9(D). Figure 8 is a flowchart showing an operation example according to the second embodiment, and Figures 9(A) to 9(D) are schematic diagrams showing examples of various images according to the second embodiment. The control unit 110 executes the process of Figure 8 every time the image to be displayed is updated.

[0075] In S801, the control unit 110 acquires the latest (current) HMD position / posture information Q from the HMD position / posture detection unit 104.

[0076] In S802, the control unit 110 acquires the past HMD position / orientation information R that is held.

[0077] In S803, the control unit 110 determines the projective transformation matrices HL and HR based on the change from the HMD position / posture information R to the HMD position / posture information Q. The projective transformation matrix HL is a projective transformation matrix for transforming the left composite image, and the projective transformation matrix HR is a projective transformation matrix for transforming the right composite image. To perform projective transformation centered on the center of the user's entire visual field, two different projective transformation matrices are determined as the projective transformation matrix HL and the projective transformation matrix HR.

[0078] In S804, the control unit 110 sets the projective transformation matrix HL in the conversion unit 106L, and sets the projective transformation matrix HR in the conversion unit 106R.

[0079] Fig. 9(A) is a schematic diagram showing an example of a left composite image and a right composite image, and Fig. 9(B) is a schematic diagram showing an example of a left converted image and a right converted image. Here, it is assumed that a user wearing the HMD 100 swings his head horizontally from right to left. Therefore, the left converted image and the right converted image similar to those in Fig. 7(B) are obtained.

[0080] In S805, the control unit 110 determines the processing parameters TL of the gradation unit 103L based on the projective transformation matrix HL. For example, a black area in the image after projective transformation is detected based on the projective transformation matrix HL, and the processing parameters TL including the width of the black area are determined. In S806, the control unit 110 sets the processing parameters TL to the gradation unit 103L.

[0081] In S807, the control unit 110 determines the processing parameters TR of the gradation unit 103R based on the projective transformation matrix HR. In S808, the control unit 110 sets the processing parameters TR in the gradation unit 103R.

[0082] The processing parameters TL, TR of the gradation sections 103L, 103R may be processing parameters that change the area where the gradation processing is performed (e.g., position, horizontal size (width), vertical size (height), or shape). The processing parameters TL, TR may be processing parameters that indicate the strength of the gradation processing. The processing parameters TL, TR may be processing parameters that change both the area and the strength of the gradation processing.

[0083] FIG. 9C shows a left display image obtained by applying gradation processing to the left converted image of FIG. 9B. FIG. 9(B) is a schematic diagram showing an example of a right display image obtained by performing gradation processing on the right converted image. FIG. 9(D) is a schematic diagram showing an example of an image perceived by a user when the left display image of FIG. 9(C) is viewed with the left eye and the right display image of FIG. 9(C) is viewed with the right eye. In FIG. 9(C), the area to be subjected to gradation processing is changed according to the change in the overlapping area, so that gradation processing is suitably performed on both of the two boundary areas between the overlapping area and the non-overlapping area (both the left end and the right end of the overlapping area). It may be interpreted that gradation processing is performed on the boundary area between the valid area and the invalid area (black area). As a result, as shown in FIG. 9(D), the perception of black band-like noise can be suitably suppressed.

[0084] In S809, the control unit 110 holds the HMD position / posture information Q as past information.

[0085] As described above, in the second embodiment, gradation processing is performed in consideration of head movement. This makes it possible to obtain a suitable image (an image that allows natural observation over a wide range) even when image deformation (projective transformation) is performed as image processing to reduce (suppress) delays in changes to the display range.

[0086] <<Third embodiment>> The third embodiment of the present invention will be described below. Note that, in the following, the same points as the first embodiment (for example, the same configuration and processing as the first embodiment) will not be described, and only the points different from the first embodiment will be described. An example of moving an image as image processing for reducing video delay will be described, but image deformation (projective transformation) may also be performed.

[0087] In the first and second embodiments, examples have been described in which image processing for reducing image delay is performed only once after reprojection. In the third embodiment, an example will be described in which image processing for reducing image delay is performed twice after reprojection in order to further reduce image delay.

[0088] <Configuration> FIG. 10 is a schematic diagram showing an example of the configuration of a system according to the third embodiment.

[0089] The information processing device 200 has a CG position / posture determination unit 201, a rendering unit 202, a CG storage unit 203, and a reprojection unit 204. The reprojection unit 204 transmits the reprojected left CG image (left CG image data) to a first conversion unit 107L described later, and transmits the reprojected right CG image (right CG image data) to a first conversion unit 107R described later.

[0090] The HMD 100 has imaging units 101L and 101R, display units 102L and 102R, an HMD position / posture detection unit 104, first conversion units 107L and 107R, synthesis units 105L and 105R, second conversion units 108L and 108R, gradation units 103L and 103R, and a control unit 110.

[0091] The first conversion unit 107L moves or deforms the left CG image based on the detection result (HMD position / posture information) of the HMD position / posture detection unit 104 to generate a first left converted image so as to reduce video delay. For example, the first conversion unit 107L moves the left CG image in a direction that cancels out head movement during reprojection (from the end of rendering (the start of reprojection) to the end of reprojection (the start of processing by the first conversion unit 107L)). Similarly, the first conversion unit 107R moves or deforms the right CG image based on the detection result (HMD position / posture information) of the HMD position / posture detection unit 104 to generate a first right converted image.

[0092] The synthesis unit 105L synthesizes the left camera image (image in real space) obtained by the imaging unit 101L with the first left converted image obtained by the first conversion unit 107L, thereby obtaining a left-justified image. Similarly, the composition unit 105R generates a left composite image by combining the left camera image (image in real space) obtained by the imaging unit 101R with the first left converted image obtained by the first conversion unit 107R.

[0093] The second conversion unit 108L moves or deforms the left composite image based on the detection result (HMD position / posture information) of the HMD position / posture detection unit 104 to generate a second left converted image so as to reduce video delay. For example, the second conversion unit 108L moves the left composite image in a direction that cancels out the head movement during the processing period of the first conversion unit 107L and the synthesis unit 105L (from the end of reprojection (the start of processing by the first conversion unit 107L) to the present (the end of synthesis, the start of processing by the second conversion unit 108L)). Similarly, the second conversion unit 108R moves or deforms the right composite image based on the detection result (HMD position / posture information) of the HMD position / posture detection unit 104 to generate a second right converted image.

[0094] The gradation section 103L performs gradation processing on the second left converted image to generate a left display image, and the gradation section 103R performs gradation processing on the second right converted image to generate a right display image.

[0095] <Operation> An operation example according to the third embodiment will be described with reference to Figures 11 and 12(A) to 12(G). Figure 11 is a flowchart showing an operation example according to the third embodiment, and Figures 12(A) to 12(G) are schematic diagrams showing examples of various images according to the third embodiment. The control unit 110 executes the process of Figure 11 every time the image to be displayed is updated.

[0096] In S1101, the control unit 110 acquires the latest HMD position / orientation information Q1 from the HMD position / orientation detection unit 104 (at the end of reprojection, at the start of processing by the first conversion unit 107L).

[0097] In S1102, the control unit 110 acquires the past HMD position / orientation information R1 that is held. The HMD position / orientation information R1 is, for example, the HMD position / orientation information at the end of rendering (at the start of reprojection).

[0098] In S1103, the control unit 110 determines image shift values ​​SX1, SY1 indicating the movement direction and movement amount of the image based on the change from the HMD position / orientation information R1 to the HMD position / orientation information Q1.

[0099] In S1104, the control unit 110 sets image shift values ​​SX1 and SY1 in the first conversion unit 107L and the first conversion unit 107R, respectively. After that, the first conversion unit 107L, the first conversion unit 107R, the composition unit 105L, and the composition unit 105R perform processing.

[0100] FIG. 12(A) is a schematic diagram showing an example of the left CG image and the right CG image after reprojection, and FIG. 12(B) is a schematic diagram showing an example of the first left converted image and the first right converted image. Here, it is assumed that the user wearing the HMD 100 swings his head horizontally from right to left. Therefore, the control unit 110 sets the image shift value SX1=positive value and the image shift value SY1=0 for each of the first conversion unit 107L and the first conversion unit 107R. Then, the first conversion unit 107L moves (shifts) the left CG image in FIG. 12(A) to the right by an amount corresponding to the absolute value of the image shift value SX1, thereby obtaining the first left converted image in FIG. 12(B). Similarly, the first conversion unit 107R moves the right CG image in FIG. 12(A) to the right by an amount corresponding to the absolute value of the image shift value SX1, thereby obtaining the first right converted image in FIG. 12(B). There is no image to display at the left end of the first left converted image and the left end of the first right converted image, so as shown in FIG. 12B, the left end of the first left converted image and the left end of the first right converted image are black areas.

[0101] FIG. 12(C) is a schematic diagram showing an example of a left camera image (image in real space) obtained by the imaging unit 101L and a right camera image (image in real space) obtained by the imaging unit 101R. FIG. 12(D) is a schematic diagram showing an example of a left composite image obtained by the composition unit 105L and a right composite image obtained by the composition unit 105R. The left composite image in FIG. 12(D) is an image obtained by combining the first left converted image in FIG. 12(B) with the left camera image in FIG. 12(C), and the right composite image in FIG. 12(D) is an image obtained by combining the first right converted image in FIG. 12(B) with the right camera image in FIG. 12(C). By performing the processing of the first conversion units 107L and 107R, CG can be suitably combined (superimposed) on the camera images. CG cannot be drawn at the left end of the first left converted image and the left end of the first right converted image. Therefore, as shown in FIG. 12(B), the left end of the left composite image and the left end of the right composite image are each left as a black area.

[0102] In S1105, the control unit 110 holds the HMD position / orientation information Q1 as past HMD position / orientation information R2.

[0103] In S1106, the control unit 110 acquires the latest HMD position / orientation information Q2 from the HMD position / orientation detection unit 104 (at the end of synthesis, at the start of processing by the second conversion unit 108L).

[0104] In S1107, the control unit 110 determines image shift values ​​SX2, SY2 indicating the movement direction and movement amount of the image based on the change from the HMD position / orientation information R2 to the HMD position / orientation information Q2.

[0105] In S1108, the control unit 110 sets the image shift values ​​SX2 and SY2 in the second conversion units 108L and 108R, respectively. After that, the processes in the second conversion units 108L and 108R are performed.

[0106] FIG. 12(E) is a schematic diagram showing an example of the second left converted image and the second right converted image. As described above, it is assumed that the user wearing the HMD 100 swings his head horizontally from right to left. Therefore, the control unit 110 sets the image shift value SX2=positive value and the image shift value SY2=0 for each of the second conversion unit 108L and the second conversion unit 108R. Then, the second conversion unit 108L moves (shifts) the left composite image of FIG. 12(D) to the right by an amount of movement corresponding to the absolute value of the image shift value SX2 to obtain the second left converted image of FIG. 12(E). Similarly, the second conversion unit 108R moves the right composite image of FIG. 12(D) to the right by an amount of movement corresponding to the absolute value of the image shift value SX2 to obtain the second right converted image of FIG. 12(E). There is no image to be displayed at the left end of the second left converted image and the left end of the second right converted image. Therefore, as shown in Fig. 12(E), the left end of the second left converted image and the left end of the second right converted image are black areas. Because two right shifts are performed, the black areas in Fig. 12(E) are larger than the black areas in Figs. 12(B) and 12(D).

[0107] In S1109, the control unit 110 calculates image shift values ​​SX3 and SY3 (total image shift amount). The control unit 110 calculates image shift value SX3 by adding image shift value SX2 to image shift value SX1, and calculates image shift value SY3 by adding image shift value SY2 to image shift value SY1.

[0108] In S1110, the control unit 110 judges whether the image shift value SX3 is a positive value or not. If the image shift value SX3 is a positive value (if the image is moved to the right, and it is judged that there has been a head movement to the left), the process proceeds to S1111, and if not, the process proceeds to S1112.

[0109] In S1111, control unit 110 sets image shift values ​​SX3, SY3 for gradation portion 103R, and in S1112, control unit 110 sets image shift values ​​SX3, SY3 for gradation portion 103L.

[0110] Fig. 12(F) is a schematic diagram showing an example of a left display image obtained by applying gradation processing to the second left converted image in Fig. 12(E) and a right display image obtained by applying gradation processing to the second right converted image in Fig. 12(E). Because the image shift value SX3 is a positive value, the gradation processing of the gradation section 103L is not changed, and the area in which the gradation processing of the gradation section 103R is performed is shifted to the right. Therefore, in the left display image, the gradation processing is performed on the right edge of the image, and in the right display image, the gradation processing is performed on the part adjacent to the right side of the black area, which is the left edge of the image.

[0111] In each of the second left converted image and the left display image, the right edge of the image is the right edge (the right edge of the overlapping portion) of the two boundaries between the overlapping portion and the non-overlapping portion. In each of the second right converted image and the right display image, the above-mentioned portion adjacent to the right side of the black region, which is the left edge of the image, is the left edge (the left edge of the overlapping portion) of the two boundaries. Therefore, in FIG. 12(F), gradation processing is performed on both of the two boundaries between the overlapping portion and the non-overlapping portion (both the left edge and the right edge of the overlapping portion).

[0112] In S1113, the control unit 110 holds the HMD position / posture information Q2 as past information.

[0113] As described above, according to the third embodiment, gradation processing is performed based on two pieces of information related to head movement: information used in the first image processing to reduce image delay, and information used in the second image processing. This makes it possible to obtain a suitable image (an image that allows for natural observation over a wide range) even when image processing to reduce image delay is performed twice.

[0114] The above-described embodiment (including the modified examples) is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of the above-described embodiment within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiment.

[0115] <<Other embodiments>> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0116] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A first image, the first image being a left-eye image and having a range based on at least one of the position and orientation of the user's head; and a second image, which is an image for the right eye, an image of a range based on at least one of the position and the orientation of the head, and an image of which a part corresponds to a part of the first image; A first acquisition means for acquiring A second acquisition means for acquiring information regarding the movement of the head; a first processing means for generating a third image by moving or transforming the first image and generating a fourth image by moving or transforming the second image based on the information; A second processing step of performing gradation processing on each of the third image and the fourth image. Step by step having The second processing means performs the gradation processing based on the information. 13. An image processing device comprising: (Configuration 2) The mutually corresponding portions of the first image and the second image are stereoscopically viewable portions. 2. The image processing device according to claim 1, (Configuration 3) The second processing means changes an area where the gradation processing is performed based on the information. 3. The image processing device according to configuration 1 or 2. (Configuration 4) The second processing means performs the gradation processing on a boundary portion between a non-overlapping portion of the third image and the fourth image in an overlapping portion of the third image and the fourth image based on the information. 4. The image processing device according to any one of configurations 1 to 3. (Configuration 5) each of the third image and the fourth image includes a valid area and an invalid area according to the movement of the head; The second processing means performs the following processing based on the information: an end portion of an effective area of ​​the third image on a side of a portion corresponding to a part of the fourth image; The gradation processing is performed on an end portion of the effective area of ​​the fourth image on the side of a portion corresponding to a part of the third image. 5. The image processing device according to configuration 4. (Configuration 6) The second processing means performs the following processing based on the information: a gradation process for gradually changing a luminance value from an end of an effective area of ​​the third image on a side of a portion corresponding to a part of the fourth image toward an inside of the effective area; a gradation process for gradually changing a luminance value from an end of an effective area of ​​the fourth image on a side of a portion corresponding to a part of the third image toward an inside of the effective area; Do 6. The image processing device according to configuration 5. (Configuration 7) The second processing means performs the following processing based on the information: an area in which the gradation processing is performed; The strength of the gradation processing; Change at least one of the 7. The image processing device according to any one of configurations 1 to 6. (Configuration 8) The second processing means performs different gradation processing on the third image and the fourth image based on the information. 8. The image processing device according to any one of configurations 1 to 7. (Configuration 9) The second processing means performs the following processing based on the information: if it is determined that the head has moved to the right, modifying the gradation process performed on the third image; If it is determined that the head has moved to the left, the gradation processing performed on the fourth image is changed. 9. The image processing device according to any one of configurations 1 to 8. (Configuration 10) The second processing means performs the following processing based on the information: If it is determined that the head has moved to the right, the area in the third image where the gradation process is performed is moved to the left; If it is determined that the head has moved to the left, the area in the fourth image where the gradation process is performed is moved to the right. 10. The image processing device according to configuration 9. (Configuration 11) Further comprising a third acquisition means for acquiring an image of the real space, The first processing means is based on first information on the movement of the head acquired by the second acquisition means, moving or deforming the first image to generate a fifth image, and moving or deforming the second image to generate a sixth image; synthesizing the fifth image with the image in real space to generate a seventh image, and synthesizing the sixth image with the image in real space to generate an eighth image; based on second information on the movement of the head acquired by the second acquisition means, moving or deforming the seventh image to generate the third image, and moving or deforming the eighth image to generate the fourth image; The second processing means performs the gradation processing based on the first information and the second information. 11. The image processing device according to any one of configurations 1 to 10. (method) A first image, the first image being a left-eye image and having a range based on at least one of the position and orientation of the user's head; and a second image, which is an image for the right eye, an image of a range based on at least one of the position and the orientation of the head, and an image of which a part corresponds to a part of the first image; and obtaining obtaining information regarding the head movement; moving or transforming the first image to generate a third image and moving or transforming the second image to generate a fourth image based on the information; performing gradation processing on each of the third image and the fourth image; The gradation process is performed based on the information. 13. An image processing method comprising: (program) 12. A program for causing a computer to function as each of the means of the image processing device according to any one of configurations 1 to 11. [Explanation of symbols]

[0117] 100:HMD 101L, 101R: Imaging section 103L, 103R: Gradation section 104: HMD position / posture detection unit 106L, 106R: conversion unit 110: control unit

Claims

1. A first image, which is an image for the left eye and has a range based on at least one of the position and the orientation of the user's head; and a second image, which is an image for the right eye, an image of a range based on at least one of the position and the orientation of the head, and an image of which a part corresponds to a part of the first image; A first acquisition means for acquiring A second acquisition means for acquiring information regarding the movement of the head; a first processing means for generating a third image by moving or transforming the first image and generating a fourth image by moving or transforming the second image based on the information; a second processing means for performing gradation processing on each of the third image and the fourth image; having The second processing means performs the gradation processing based on the information.

13. An image processing device comprising:

2. The mutually corresponding portions of the first image and the second image are stereoscopically viewable portions.

2. The image processing device according to claim 1,

3. The second processing means changes an area where the gradation processing is performed based on the information.

2. The image processing device according to claim 1,

4. The second processing means performs the gradation processing on a boundary portion between a non-overlapping portion of the third image and the fourth image in an overlapping portion of the third image and the fourth image based on the information.

2. The image processing device according to claim 1,

5. each of the third image and the fourth image includes a valid area and an invalid area according to the movement of the head; The second processing means performs the following processing based on the information: an end portion of an effective area of ​​the third image on a side of a portion corresponding to a part of the fourth image; The gradation processing is performed on an end portion of the effective area of ​​the fourth image on the side of a portion corresponding to a part of the third image.

5. The image processing device according to claim 4.

6. The second processing means performs the following processing based on the information: a gradation process for gradually changing a luminance value from an end of an effective area of ​​the third image on a side of a portion corresponding to a part of the fourth image toward an inside of the effective area; a gradation process for gradually changing a luminance value from an end of an effective area of ​​the fourth image on a side of a portion corresponding to a part of the third image toward an inside of the effective area; Do 6. The image processing device according to claim 5,

7. The second processing means performs the following processing based on the information: an area in which the gradation processing is performed; The strength of the gradation processing; Change at least one of the 2. The image processing device according to claim 1,

8. The second processing means performs different gradation processing on the third image and the fourth image based on the information.

2. The image processing device according to claim 1,

9. The second processing means performs the following processing based on the information: if it is determined that the head has moved to the right, modifying the gradation process performed on the third image; If it is determined that the head has moved to the left, the gradation processing to be performed on the fourth image is changed.

2. The image processing device according to claim 1,

10. The second processing means performs the following processing based on the information: If it is determined that the head has moved to the right, the area in the third image where the gradation process is performed is moved to the left; If it is determined that the head has moved to the left, the area in the fourth image where the gradation process is performed is moved to the right.

10. The image processing device according to claim 9,

11. The present invention further includes a third acquisition means for acquiring an image of a real space, The first processing means is based on first information on the movement of the head acquired by the second acquisition means, moving or deforming the first image to generate a fifth image, and moving or deforming the second image to generate a sixth image; synthesizing the fifth image with the image in real space to generate a seventh image, and synthesizing the sixth image with the image in real space to generate an eighth image; based on second information on the movement of the head acquired by the second acquisition means, moving or deforming the seventh image to generate the third image, and moving or deforming the eighth image to generate the fourth image; The second processing means performs the gradation processing based on the first information and the second information.

2. The image processing device according to claim 1,

12. A first image, which is an image for the left eye and has a range based on at least one of the position and the orientation of the user's head; and a second image, which is an image for the right eye, an image of a range based on at least one of the position and the orientation of the head, and an image of which a part corresponds to a part of the first image; and obtaining obtaining information regarding the head movement; moving or transforming the first image to generate a third image and moving or transforming the second image to generate a fourth image based on the information; performing gradation processing on each of the third image and the fourth image; The gradation process is performed based on the information.

13. An image processing method comprising:

13. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 11.

Images