Image processing device, image processing method and program

Abstract

To reduce recognition burden on a viewer who views a stereo image and improve presence and immersion.SOLUTION: An image data acquisition unit acquires stereo image data from an input I / F or a secondary storage. A parallax data acquisition unit acquires parallax data corresponding to the stereo image data acquired from the image data acquisition unit. The line-of-sight information acquisition unit acquires information on the line of sight of a user who wears a head-mounted display from a line of sight detection device through a line of sight detection I / F. A processing area determination unit creates a sharpness map based on the parallax data acquired from the parallax data acquisition unit and the line of sight information acquired from the line-of-sight information acquisition unit. A sharpness changing unit performs image processing of changing sharpness on the stereo image data acquired from the image data acquisition unit based on the sharpness map acquired from the processing area determination unit. A display control unit causes the head-mounted display to display the stereo image whose sharpness is changed by the sharpness changing unit.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an image processing technique for generating stereo images. [Background technology]

[0002] In recent years, display devices capable of displaying stereo images composed of a left-eye image and a right-eye image with parallax between them have been sold as display devices used by viewers to enjoy three-dimensional images. One example of such a display device is a head-mounted display, which is worn on the head to view images.

[0003] Stereo images with parallax are generated using images captured by an imaging device equipped with multiple lenses or three-dimensional computer graphics (3DCG). Typically, the amount of blur of a subject in a stereo image is determined by the focal length set when the image is generated, as with general images. Meanwhile, humans perceive depth based on the amount of blur of the perceived image and the angle between the eyes (convergence angle) in addition to the difference in how the two eyes see (binocular parallax). Therefore, when viewing a stereo image, if the amount of blur or convergence angle of the object being gazed at differs from the real world, it can cause discomfort. This discomfort increases cognitive load and can lead to eye strain and a reduced sense of realism.

[0004] Patent Document 1 discloses a system that generates and displays stereo images with parallax from images captured by an imaging device controlled so that the focus is on the target that a user wearing a head-mounted display is gazing at. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2018-6914 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in addition to stereo images generated from images generated after a gaze target is determined, such as the stereo images described in Patent Document 1, there are also stereo images generated from images before a gaze target is determined. Since the focal length of an image cannot be changed after generation, the technology in Patent Document 1 cannot reduce the cognitive load on the viewer or improve the sense of realism or immersion for stereo images generated from images before a gaze target is determined. [Means for solving the problem]

[0007] The present invention is an image processing device, comprising: an image acquisition means for acquiring a stereo image consisting of a left-eye image and a right-eye image having a mutual parallax, the captured image having a predetermined focal length at the time of capture; and an image processing means for performing different image processing on a gaze area gazed upon by a viewer viewing the stereo image displayed on a display device and an area other than the gaze area, based on line-of-sight information of the viewer viewing the stereo image displayed on the display device, without changing the focal length of the stereo image. a display control means for causing a display means to display the stereo image whose sharpness has been changed by the image processing means; Equipped with The image processing means performs, as the different image processing, image processing to make the sharpness of the gaze area higher than that of areas other than the gaze area, or to make the saturation of the areas other than the gaze area lower than that of the gaze area; and when the stereo image acquired by the image acquisition means is a moving image, if a frame of the stereo image does not satisfy a predetermined condition, the display control means displays the frame on the display device without performing image processing by the image processing means, and the predetermined condition is that the magnitude of the difference between the disparity value in the gaze area of ​​a target frame that the display control means intends to display on the display device and the disparity value in the gaze area of ​​the most recent frame that the display control means has displayed on the display device is equal to or smaller than a predetermined threshold. It is characterized by: [Effects of the Invention]

[0008] The present invention can reduce the cognitive load on a viewer viewing a stereo image and improve the sense of realism and immersion. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing a hardware configuration of an image processing apparatus. [Figure 2] FIG. 2 is a block diagram showing the functional configuration of a parallax image generation process according to the first embodiment. [Figure 3] 4 is a flowchart showing a flow of generating parallax images according to the first embodiment. [Figure 4] 3A to 3C are diagrams illustrating the concept of parallax image generation processing according to the first embodiment. [Figure 5] 5A to 5C are diagrams for explaining a method of changing sharpness in the first embodiment. [Figure 6] 10A to 10C are diagrams illustrating the concept of parallax image generation processing according to the second embodiment. [Figure 7] FIG. 10 is a block diagram showing the functional configuration of a parallax image generation process according to a second embodiment. [Figure 8] 10 is a flowchart showing the flow of generating parallax images according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the present invention, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present invention. Note that the same components will be described with the same reference numerals.

[0011] <Embodiment 1> In embodiment 1, a method is described for converting an existing stereo image into a stereo image in which the area the user is gazing at is sharper than other areas, based on the gaze information of the user viewing the stereo image and the parallax data obtained from the stereo image.

[0012] FIG. 1 shows an example of the hardware configuration of an image processing apparatus according to this embodiment. In FIG. 1, a CPU 101 uses a RAM 102 as a work memory, executes programs stored in a ROM 103 and a hard disk drive (HDD) 105, and controls the operation of each block (described later) via a system bus 110. An HDD interface (hereinafter, interface will be referred to as "I / F") 104 connects a secondary storage device such as the HDD 105 or an optical disk drive. The HDD I / F 104 is, for example, an I / F such as a serial ATA (SATA). The CPU 101 can read data from and write data to the HDD 105 via the HDD I / F 104. Furthermore, the CPU 101 can load data stored in the HDD 105 into the RAM 102, and conversely, can save data loaded in the RAM 102 to the HDD 105. The CPU 101 can then execute the data loaded in the RAM 102 as a program.

[0013] The input I / F 106 connects to an input device 131 such as a keyboard, a mouse, a digital camera, or a scanner. The input I / F 106 can also connect a stereo camera provided in a head-mounted display as the input device 131. The input I / F 106 is, for example, a serial bus I / F such as USB or IEEE 1394. The CPU 101 can read data from the input device 131 via the input I / F 106. The output I / F 107 connects the image processing device 100 to a head-mounted display 132, which is an output device. The output I / F 107 is, for example, a video output I / F such as DVI or HDMI (registered trademark). The CPU 101 can send data to the head-mounted display 132 via the output I / F 107 and display a predetermined image on the head-mounted display 132. The orientation detection I / F 110 connects to an orientation detection device 133, such as an acceleration sensor or an angular velocity sensor. The posture detection I / F 108 is a serial bus I / F such as USB or IEEE1394. The posture detection device 133 is attached to the head-mounted display 132, and the CPU 101 can read posture information of the head-mounted display 132 from the posture detection device 133 via the posture detection I / F 108. The posture information can also be input via a mouse, keyboard, camera, or the like. The gaze detection I / F 109 connects a gaze detection device 134 such as an eye-gaze measuring device. The gaze detection I / F 109 is a serial bus I / F such as USB or IEEE1394. The gaze detection device 134 is attached inside the head-mounted display 132, and the CPU 101 can read gaze information of the user wearing the head-mounted display 132 from the gaze detection device 134 via the gaze detection I / F 109. The gaze information can also be input via a mouse, keyboard, camera, or the like. Note that the image processing device 100 includes other components, but these are not the focus of the present invention and will not be described here.

[0014] Here, an overview of a method for generating stereo image data for display performed in this embodiment will be described. Stereo images for display displayed on the head-mounted display 132 are generated from input stereo image data based on position and orientation information of the user wearing the head-mounted display 132. In this embodiment, stereo images for display are generated by performing a filter process on existing stereo image data to increase the sharpness of the user's gaze area relative to the sharpness of areas other than the gaze area. The filter process for changing the sharpness is performed appropriately according to the characteristics of the input stereo image. For example, if the input stereo image has uniformly high sharpness regardless of the area, a process for decreasing the sharpness of areas other than the user's gaze area is performed. As another example, if the sharpness of the user's gaze area is lower than the area with the highest sharpness in the input stereo image, a process for increasing the sharpness of the gaze area is performed, and a process for decreasing the sharpness of other areas is performed. Note that instead of changing the sharpness, a process for adding blur may be performed. In this case, the blurring process may be performed so that the blurring outside the gaze area is greater than that of the gaze area. Alternatively, the sharpening process and the blurring process may be combined, for example, by performing a process to increase the sharpness of the region of interest and a process to blur regions other than the region of interest.Furthermore, the contrast or saturation of regions other than the region of interest may be reduced.

[0015] The processing performed by the image processing device 100 of the first embodiment will be described below with reference to FIGS.

[0016] Fig. 2 is a block diagram showing the functional configuration of the image processing device 100. The image processing device 100 functions as each component shown in Fig. 2 and executes a series of processes shown in the flowchart of Fig. 3 by having the CPU 101 execute a program stored in the ROM 103 using the RAM 102 as a work memory. Note that it is not necessary for all of the processes shown below to be executed by the CPU 101, and the image processing device 100 may be configured so that part or all of the processes are executed by one or more processing circuits other than the CPU 101.

[0017] The image data acquisition unit 201 acquires stereo image data via the input I / F 106 or from the secondary storage device 105. The acquired stereo image data is stereo image data having parallax, consisting of image data for the left eye and image data for the right eye of the head-mounted display 132. This stereo image data is generated from wide-angle stereo images based on posture information acquired by the posture detection device 133. For example, from an ultra-wide-angle stereo image having omnidirectional images for the left and right eyes, image areas with a predetermined angle of view are cut out in the optical axis directions of two virtual imaging devices for the left and right eyes based on the posture information of the head-mounted display 132. Distortion correction is then performed on each of the cut-out images to generate images for the left eye and right eye of the stereo image. This allows a user wearing the head-mounted display 132 to view a three-dimensional image as if they were inside a sphere onto which the omnidirectional image is projected.

[0018] In this embodiment, the stereo image data will be described assuming that the left-eye image data and the right-eye image data are color image data having three RGB channels for each pixel. Note that the stereo image data is not limited to this, and the same can be applied to gray image data or moving image data having only one channel for each pixel. Furthermore, the image data acquisition unit 201 simultaneously acquires rendering information when the stereo image data is generated from wide-angle stereo images. The rendering information is information regarding the positions and orientations of virtual image capture devices for the left and right eyes used to generate the stereo image data, as well as the focal lengths of the virtual image capture devices.

[0019] In this embodiment, three-dimensional coordinate axes are defined, with the floor surface as the x- and y-axes and the height direction as the z-axis, as a coordinate system of a virtual space in which an object that reproduces the position and orientation of the head-mounted display 132 is placed. The rendering information has the positions of the virtual imaging devices for the left and right eyes as coordinate values ​​on these three-dimensional coordinate axes. Note that the method of defining the coordinate axes is not limited to this, and other methods of defining the coordinate axes, such as polar coordinates, may also be used.

[0020] The parallax data acquisition unit 202 acquires parallax data related to the stereo image data acquired by the image data acquisition unit 201. In this embodiment, the parallax data for the acquired stereo image data is calculated using a block matching method, which is a common method for acquiring parallax data from image data for the left eye and right eye included in the stereo image data. Note that the method for acquiring parallax data is not limited to this, and any known method for calculating parallax values ​​from stereo image data may be used. Furthermore, if the acquired stereo images are generated by rendering based on wide-angle stereo images and include parallax data, the parallax data may be acquired. The parallax data in this embodiment has the same number of pixels as the left-eye and right-eye images of the stereo image data, and each pixel holds a parallax value. Note that the format of the parallax data is not limited to this, and any format, such as distance data, may be used as long as it is possible to acquire a parallax value corresponding to each pixel of the stereo image data.

[0021] Note that in this embodiment, stereo image data is used that is generated by rendering based on wide-angle stereo image data so as to be linked to the position and posture of the head-mounted display 132, but the stereo image data is not limited to this. The stereo image data only needs to have image data for the left eye and image data for the right eye, and may be, for example, stereo image data that simply combines captured image data for the left eye and captured image data for the right eye that have parallax without performing rendering processing.

[0022] The gaze information acquisition unit 203 acquires gaze information of the user wearing the head mounted display 132 from the gaze detection device 134 via the gaze detection I / F 109. The gaze information in this embodiment is the position (xe) of each eye of the user wearing the head mounted display 132 in the above-mentioned three-dimensional space. i ,ye i ,ze i ) and its unit vector (vxe i ,vye i ,vzei ) (i=1, 2). The method of acquiring gaze information in this embodiment uses a known eye tracking technique, and acquires gaze information of the user's left eye and right eye using an imaging device attached inside the head mounted display 132. Furthermore, the gaze information is not limited to this, and various methods that can identify the position at which the user is gazing may be used, such as using the three-dimensional coordinate value of the head mounted display 132 at which the user is gazing as gaze information.

[0023] The processing area determination unit 204 determines a processing area in the stereo image data based on the parallax data and the line-of-sight information. In this embodiment, a sharpness map is created that has the same number of pixels as the left-eye and right-eye images of the stereo image data and that can hold a value between 0 and 1 for each pixel, and determines the pixel area of ​​the stereo image data that corresponds to pixels whose pixel value in the sharpness map is 1 as the processing area.

[0024] Here, a specific method for determining the processing area will be explained. As a preliminary step, all pixels included in the sharpness map are initialized to 0. First, the position (xt, yt, zt) of the virtual image that the user is gazing at is calculated from the gaze information acquired from the gaze information acquisition unit 203. The three-dimensional coordinate position of the target is calculated based on the eye position (xe i ,ye i ,ze i ) as the starting point, and the line of sight (vxe i ,vye i ,vze i ) as the intersection of the left and right vectors with the direction of the virtual image pickup device. Next, the distance Z from the virtual image pickup device to the target is calculated from the position (xt, yt, zt) of the gaze virtual image and the position of the virtual image pickup device in the rendering information acquired from the image data acquisition unit 201. In this embodiment, the distance Z is calculated as the distance between the center of gravity position (xc, yc, zc) of the positions of the left and right virtual image pickup devices and the position (xt, yt, zt) of the gaze virtual image.

[0025] 4 shows a diagram illustrating the relationship between the gaze target area on a stereo image, the position of the gaze target virtual image perceived by the user, and the user's line of sight in a head-mounted display 132 in which lenses 403 and displays 404 are placed in front of the left and right eyes, respectively. Distance Z to virtual images 410 and 420 that the user is gazing at is calculated as the distance to the intersection of the left and right lines of sight 401 and 402.

[0026] A user wearing the head-mounted display 132 perceives an image as a virtual image by viewing the display 404 through the lens 403. Images with parallax are displayed on the left and right displays 404, allowing the user to perceive images with parallax between the left and right eyes. This allows the user to perceive the virtual image with a three-dimensional effect based on the binocular parallax, convergence angle, and the like. The perceived depth of the virtual image changes depending on the magnitude of the parallax of the stereo image displayed on the display 404. For example, when the positions of an object 410 in a left-eye image 411 and a right-eye image 412 are significantly different, resulting in a large parallax 414 and a large convergence angle, as shown in FIG. 4(a), the user perceives the object 410 as being relatively close. On the other hand, when an object 420 is present in a similar position in a left-eye image 421 and a right-eye image 422, with a small parallax 424 and a small convergence angle, as shown in FIG. 4(b), the user perceives the object 420 as being relatively far away. In this embodiment, the distance Z to the gaze virtual image that the user is gazing at, calculated from the line of sight information as described below, is converted into a disparity value d in the stereo image, and the pixel area in the stereo image having the converted disparity value is determined to be the area corresponding to the gaze virtual image.

[0027] The calculated distance Z is converted into a parallax value d using a base line length T indicating the distance between the virtual image capture devices and a focal length f of the virtual image capture devices in the rendering information acquired from the image data acquisition unit 201. The base line length T can be calculated as the distance between coordinates in three-dimensional space from the positions of the left and right virtual image capture devices acquired from the rendering information. The parallax value d is calculated using d = f × T ÷ Z, which is the relationship between the parallax image and the distance.

[0028] Finally, in the disparity data acquired from the disparity data acquisition unit 202, all pixels that hold the calculated disparity value d are extracted, and the pixel value of the pixel in the sharpness map whose pixel position matches that of the extracted pixel is changed to 1. Similarly, when detecting pixels, it is not limited to the disparity value d, and taking into consideration accuracy errors in acquiring disparity values ​​by block matching and disparity fluctuations within the same subject due to the thickness of the object, it is also possible to set the pixel value to 1 for pixels within a predetermined range from the extracted pixel.

[0029] The sharpness modification unit 205 modifies the sharpness of pixel regions of the stereo image data whose pixel positions match pixels with a pixel value of 1 in the sharpness map so that the sharpness is higher than in other regions. The sharpness modification is performed appropriately according to the characteristics of the stereo image. For example, if the stereo image maintains uniformly high sharpness regardless of the region, a process to reduce the sharpness is applied to regions other than the determined region. As another example, if the sharpness of a determined region on the stereo image is lower than the region with the highest sharpness on the stereo image, a process to improve the sharpness is applied to the determined region, and a process to reduce the sharpness is applied to other regions.

[0030] The method of changing the sharpness will be specifically described with reference to Fig. 5. Note that the following processing is performed on each of the left-eye image data and right-eye image data of the stereo image data.

[0031] First, the sharpness modification unit 205 generates two pieces of image data for the stereo image data 501 based on the sharpness map 502. In the first piece of image data 505, only pixels corresponding to pixels having a pixel value of 1 in the sharpness map 502 have the same pixel values ​​as the stereo image data 501, and the other pixels have uniform pixel values. In the second piece of image data 506, only pixels corresponding to pixels having a pixel value of 0 in the sharpness map 502 have the same pixel values ​​as the stereo image data 501, and the other pixels have predetermined uniform pixel values. The uniform pixel values ​​in this case may be any value. Hereinafter, the image data 505 will be referred to as high-sharpness image data, and the image data 506 will be referred to as low-sharpness image data.

[0032] Next, the generated high-sharpness image data 505 and low-sharpness image data 506 are subjected to frequency filtering using different filters. A high-pass filter capable of emphasizing high-frequency regions is used for the high-sharpness image data 505. A low-pass filter capable of removing high-frequency components while leaving only low-frequency components is used for the low-sharpness image data 506. As a result, the high-sharpness image data 505 after filtering becomes a sharper image than before filtering, and the low-sharpness image data 506 after filtering becomes a blurred image than before filtering. The frequencies passed by each of the high-pass filter and the low-pass filter may be set arbitrarily, and may be set in any way as long as there is little or no overlap in the frequencies passed by the filters.

[0033] Finally, the filtered high-sharpness image data 505 and the filtered low-sharpness image data 506 are integrated. For the integration, pixels from the filtered high-sharpness image data 505 that correspond to pixels with a pixel value of 1 in the sharpness map 502 and pixels from the filtered low-sharpness image data 506 that correspond to pixels with a pixel value of 0 in the sharpness map 502 are used.

[0034] By the above processing, high-sharp stereo image data can be generated only in the area determined by the processing area determination unit 204 to be high-sharp.

[0035] In this embodiment, the sharpness is changed using frequency filtering, but this is not limiting and the sharpness may also be changed using spatial filtering or learning. For example, when spatial filtering is used, a sharpening filter is used for the high-sharpness image data 505 and an averaging filter or the like is used for the low-sharpness image data 506. The filter size and the number of times of filter processing may also be changed for each pixel. For example, if the degree of sharpness differs between regions in the low-sharpness image data 506, appropriate settings may be made, such as increasing the number of times of filter processing in regions with relatively high sharpness compared to regions with relatively low sharpness.

[0036] The display control unit 206 outputs the stereo image data, the sharpness of which has been changed and generated by the sharpness change unit 205, to the head-mounted display 132, and displays the stereo image on the head-mounted display 132.

[0037] The flow of processing performed by each component will be explained below.

[0038] In S301, the image data acquisition unit 201 acquires stereo image data via the input I / F 106 or from the secondary storage device 105, and outputs them to the disparity data acquisition unit 202, the processing area determination unit 204, and the sharpness change unit 205.

[0039] In S302 , the parallax data acquisition unit 202 acquires parallax data corresponding to the stereo image data acquired from the image data acquisition unit 201 , and outputs it to the processing region determination unit 204 .

[0040] In S303, the line-of-sight information acquisition unit 203 acquires line-of-sight information of the user wearing the head-mounted display 132 from the line-of-sight detection device 134 via the line-of-sight detection I / F 109, and outputs it to the processing area determination unit 204.

[0041] In S304, the processing area determination unit 204 generates a sharpness map based on the parallax data acquired from the parallax data acquisition unit 202 and the gaze information acquired from the gaze information acquisition unit 203, and outputs the generated sharpness map to the sharpness change unit 205.

[0042] In S305, the sharpness change unit 205 performs image processing to change the sharpness of the stereo image data acquired from the image data acquisition unit 201, based on the sharpness map acquired from the processing area determination unit 204. The sharpness change unit 205 outputs the stereo image data with the sharpness changed to the display control unit 206.

[0043] In S306, the display control unit 206 outputs the stereo image data whose sharpness has been changed by the sharpness change unit 205 to the head mounted display 132, and causes the head mounted display 132 to display the stereo image.

[0044] In S307, the display control unit 206 determines whether or not to display the next stereo image on the head mounted display 132. The next stereo image data corresponds to, for example, image data of the next frame in a moving image. If the next stereo image data exists (S307: YES), the process returns to S301 and the series of processes is repeated. If the next stereo image data does not exist (S307: NO), this process ends.

[0045] The above is the process executed by the image processing device 100 in this embodiment.

[0046] In this embodiment, the user's gaze area in the stereo image data is identified based on the distance to the gaze virtual image viewed by the user wearing the head-mounted display 132 and the parallax data of the stereo image data. Furthermore, filtering is performed so that the identified gaze area has high sharpness and other areas have lower sharpness than the gaze area. This allows the input stereo image data to be converted into stereo image data in which the area the user is gazing at is sharper than the other areas.

[0047] By displaying the stereo images converted in this manner on a display device such as a head-mounted display, it is possible to reduce fatigue caused by increased cognitive load, such as eye strain, which occurs due to a mismatch between the focal length and the convergence angle, and to improve the sense of realism. Note that the display device to which this embodiment can be applied is not limited to a head-mounted display, and may be any display device that can display stereo images individually to the left eye and the right eye.

[0048] <Embodiment 2> In the first embodiment, a process was described in which the sharpness of all acquired stereo image data is changed in accordance with the line-of-sight information of the user wearing the head-mounted display 132. In the second embodiment, when the acquired stereo image data is video data consisting of a plurality of frames, a process is added in which it is determined whether or not to change the sharpness in accordance with the difference in parallax data between a processing target frame for which the sharpness is to be changed and a detection target frame in which line-of-sight information is detected.

[0049] 6 is a diagram illustrating an outline of a method for determining whether or not to change sharpness, which is performed in this embodiment. In the first embodiment, the sharpness of the stereo image data is changed using gaze information acquired from the gaze information acquisition unit 203 for the stereo image being viewed by the user, and the image is displayed on the head-mounted display 132. At this time, if the stereo image is a video with a specific frame rate, there may be a large difference in object arrangement, etc., between the detection target frame in which the gaze information was detected by the user when the gaze information was acquired and the processing target frame in which the sharpness is to be changed.

[0050] 6, the sampling period 610 of the gaze detection unit 203, the frame rate 620 of the content, and the refresh rate 630 of the head-mounted display 132 are all different. The detection target frame displayed at display timing 631 and for which gaze information was detected is frame 621, and the detection target frame displayed at display timing 632 is a processing target frame in which the sharpness of frame 621 has been changed based on the gaze information. The gaze information used to change the sharpness uses data from detection timing 611, which is the most recent before display timing 632, or data before detection timing 611 depending on the processing time. In this case, the detection target frame in which gaze information was detected and the processing target frame in which sharpness is changed based on that gaze information match.

[0051] On the other hand, the detection target frame after sharpness change, which is displayed at display timing 633, is a frame in which the sharpness of frame 622 has been changed based on gaze information. The gaze information used for this change in sharpness uses data from detection timing 612, which is the most recent detection timing before display timing 633. At this time, the gaze information at detection timing 612 is data acquired while the user was looking at the detection target frame displayed at display timing 632, that is, frame 621. In this case, the detection target frame for which gaze information detection is performed and the processing target frame for which sharpness is changed based on that gaze information are different.

[0052] Although it depends on the relationship between the sampling frequency of the gaze detection unit 203, the frame rate of the content, and the refresh rate of the display, in many cases, a situation occurs in which the detection target frame for detecting gaze information differs from the processing target frame for sharpness modification. As a result, if a significant change occurs in the positioning of objects between those frames, performing sharpness modification processing without any change may result in high sharpness in areas that do not need to be sharpened, and conversely, in areas that should not be unsharp, becoming unsharp. This effect is particularly noticeable at times such as when a cut changes in content. Therefore, in the second embodiment, sharpness modification processing is performed only when certain conditions are met, thereby suppressing unnecessary sharpness modification processing.

[0053] As a specific process, in the second embodiment, in addition to the process of the first embodiment, disparity data is acquired in a detection target frame for detecting gaze information that is displayed at a display timing immediately before the processing target frame. Hereinafter, the disparity data in the processing target frame whose sharpness is to be changed is referred to as first disparity data, and the disparity data in the detection target frame that was displayed at the display timing immediately before that is referred to as second disparity data.

[0054] The determination process performed by the image processing device 100 of this embodiment will be described below. FIG. 7 is a block diagram showing an example of the functional configuration of the image processing device 100 of this embodiment. The image processing device 100 functions as each component shown in FIG. 7 and executes a series of processes shown in the flowchart of FIG. 8 by having the CPU 101 execute a program stored in the ROM 103 shown in FIG. 1 using the RAM 102 as a work memory. Note that it is not necessary for all of the processes shown below to be executed by the CPU 101, and the image processing device 100 may be configured so that part or all of the processes are executed by one or more processing circuits other than the CPU 101. The same components and processes as those of the first embodiment are denoted by the same reference numerals as those of the first embodiment, and descriptions thereof will be omitted.

[0055] In this embodiment, the disparity data acquisition unit 701 acquires, as second disparity data, disparity data of a detection target frame that was displayed immediately before and for which disparity information detection is to be performed, in addition to first disparity data of a processing target frame whose sharpness is to be changed. The second disparity data has the same data format as the first disparity data. The second disparity data may be stored in the RAM 102 or the ROM 103, and the disparity data acquisition unit 701 may acquire the second most recent data of the stored disparity data as the second disparity data. The disparity data acquisition unit 701 outputs the acquired first disparity data to the determination unit 702 in addition to the processing region determination unit 204, and outputs the second disparity data to the determination unit 702.

[0056] The determination unit 702 uses the first disparity data and the second disparity data to determine whether or not to perform sharpness modification processing on a processing target frame having the first disparity data. In this embodiment, the determination unit 702 compares the first disparity data and the second disparity data corresponding to pixel regions where the pixel value of the sharpness map obtained from the processing region determination unit 204 is 1. Specifically, the determination unit 702 obtains average disparity values ​​for pixel regions in the first and second disparity data corresponding to pixel regions where the pixel value of the sharpness map is 1. The sharpness modification determination unit 702 then determines to perform sharpness modification processing if the difference between the average disparity values ​​is equal to or smaller than a predetermined threshold, and determines not to perform sharpness modification processing if the difference is greater than the predetermined threshold.

[0057] The following describes the processes and configurations newly added in the second embodiment.

[0058] In S801, the disparity data acquisition unit 701 acquires second disparity data and outputs the acquired second disparity data to the determination unit 702.

[0059] In S802, the determination unit 702 determines whether to change the sharpness using the sharpness map acquired from the processing region determination unit 204, the first disparity data acquired from the disparity data acquisition unit 202, and the second disparity data acquired from the second disparity data acquisition unit 701. If the difference in disparity value is smaller than an arbitrarily determined threshold, it is determined that there is little change from the previous frame in the object the user is gazing at, and the process proceeds to S305. Conversely, if the difference in disparity value is equal to or greater than the arbitrarily determined threshold, it is determined that the sharpness is not to be changed, and the process proceeds to S307, where the determination unit 702 outputs the stereo image data acquired from the image data acquisition unit 201 to the display control unit 206.

[0060] In this embodiment, the average of the disparity values ​​is used to determine the difference between the first disparity data and the second disparity data, but this is not limiting and other statistical values ​​such as the median may be used. Also, instead of a statistical value, the disparity value of a representative pixel such as the pixel at the center of an area in the sharpness map where the pixel value is 1 may be used.

[0061] The above is the processing performed by the image processing device 100 of embodiment 2. In embodiment 2, a sharpness change determination unit is added to embodiment 1, and processing for determining whether or not to change the sharpness of stereo image data is added. This suppresses unnecessary sharpness change processing, and displays stereo image data that is more naturally focused on the object the user is gazing at in the time series direction, regardless of the content, thereby reducing eye strain for the user and improving the sense of realism.

[0062] <Other embodiments> The embodiments of the present invention are not limited to the above-described first and second embodiments, and various other embodiments are possible. For example, in the above-described first and second embodiments, the sharpness map generated by the processing region determination unit 204 is expressed as a binary value in which the pixel value of a pixel to be determined as high sharpness is 1 and the pixel value of other pixels is 0. However, the sharpness map may be expressed as a multi-value including a decimal point or a multi-value ranging from 0 to 255. For example, the sharpness map is set so that the sharpness is higher toward the center of the line of sight of the user wearing the head-mounted display 132 and decreases from the center to the periphery. Specifically, first, the processing region determination unit 204 calculates the coordinate position (xt, yt, zt) in a virtual three-dimensional space of the virtual image on which the user is gazing, based on the line-of-sight information acquired from the line-of-sight information acquisition unit 203. Then, the processing region determination unit 204 identifies the pixel position on which the user is gazing on stereo image data obtained by capturing an image using a virtual imaging device, based on the coordinate position (xt, yt, zt). Next, the pixel closest to the identified pixel position and having a disparity value d converted from the distance to the gaze virtual image is identified and set as the start pixel position in the disparity data of the stereo image data acquired from the disparity data acquisition unit 202. Then, the pixel value of the pixel on the sharpness map that is at the same position as the identified start pixel position is set to a maximum value such as 1 or 255. Finally, the sharpness map value is set so that the sharpness map value decreases linearly or nonlinearly as the pixel position moves away from the start pixel position.

[0063] Furthermore, in the above-described embodiment, stereo images captured by an imaging device equipped with multiple lenses or stereo images created as CG have been described, but the form of the stereo images is not limited to this. For example, the stereo images may be stereo images captured by a camera (a so-called outward-facing camera) provided in a head-mounted display that captures the direction in which the wearer of the head-mounted display is looking, and superimposed with a CG-rendered image.

[0064] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0065] 201 Image data acquisition unit 203 Gaze information acquisition unit 205 Sharpness change section

Claims

1. an image acquisition means for acquiring a stereo image consisting of a left-eye image and a right-eye image having a parallax therebetween, the image having a predetermined focal length at the time of image capture; an image processing means for performing different image processing on a gaze area gazed by a viewer viewing the stereo image displayed on a display device and an area other than the gaze area without changing the focal length of the stereo image, based on line-of-sight information of the viewer viewing the stereo image displayed on the display device; a display control means for displaying on a display device the stereo image whose sharpness has been changed by the image processing means; Equipped with the image processing means performs, as the different image processing, image processing to make the sharpness of the fixation area higher than that of an area other than the fixation area, or image processing to make the saturation of the area other than the fixation area lower than that of the fixation area, When the stereo image acquired by the image acquisition means is a moving image, if a frame of the stereo image does not satisfy a predetermined condition, the display control means displays the frame on the display device without performing image processing by the image processing means; The predetermined condition is that the magnitude of the difference between the disparity value in the gaze area of ​​the target frame that the display control means is to display on the display device and the disparity value in the gaze area of ​​the most recent frame that the display control means has displayed on the display device is equal to or smaller than a predetermined threshold value.

1. An image processing device comprising:

2. the image processing means applies a different frequency to the frequency filtering process applied to the region of interest and a different frequency to the frequency filtering process applied to the region other than the region of interest, as the different image processing; 2. The image processing device according to claim 1, wherein:

3. the image processing means, as the different image processing, makes the number of times of application of the filter processing applied to the fixation region different from the number of times of application of the filter processing applied to the region other than the fixation region; 2. The image processing device according to claim 1, wherein:

4. a calculation means for calculating a parallax value from the line-of-sight information; a disparity value acquisition means for acquiring a disparity value for each pixel of the stereo image; Furthermore, the image processing means determines, as the gaze area, pixels of the stereo images for which the disparity value acquired by the disparity value acquisition means is the same as the disparity value calculated by the calculation means; 4. The image processing device according to claim 1, wherein the image processing device is a computer.

5. the calculation means calculates a distance to a virtual image that the viewer is gazing at from the line-of-sight information, and calculates a parallax value from the distance.

5. The image processing device according to claim 4.

6. the disparity value acquisition means calculates a disparity value for each pixel of the stereo image from the left-eye image and the right-eye image of the stereo image; 6. The image processing device according to claim 4, wherein:

7. the image processing means performs a sharpening process or a blurring process as the image processing, 7. The image processing device according to claim 1, wherein the image processing device is a computer.

8. the image processing means performs a sharpening process on the gaze area and a blurring process on an area other than the gaze area, 8. The image processing device according to claim 1, wherein the image processing device is a computer.

9. The image processing device according to claim 1, characterized in that the display control means causes the display device to display the target frame before image processing by the image processing means if the difference is greater than the predetermined threshold, and causes the display device to display the target frame after image processing by the image processing means if the difference is equal to or less than the predetermined threshold.

10. 10. The image processing device according to claim 1, wherein the image processing means maximizes sharpness at the center of the fixation area and decreases sharpness from the center toward the periphery.

11. 11. The image processing device according to claim 1, wherein the display device is capable of projecting only the left-eye image onto the left eye of the viewer and projecting only the right-eye image onto the right eye of the viewer.

12. 12. The image processing device according to claim 1, wherein the display device is a head-mounted display.

13. an image acquisition step of acquiring a stereo image including a left-eye image and a right-eye image having parallax with respect to each other; an image processing step of performing different image processing on a gaze area of ​​the stereo image that is gazed upon by the viewer and an area other than the gaze area, based on line-of-sight information of the viewer viewing the stereo image displayed on a display device; a display control step of displaying on a display device the stereo image whose sharpness has been changed by the image processing step; Equipped with the image processing step performs, as the different image processing, image processing to make the sharpness of the fixation area higher than that of an area other than the fixation area, or image processing to make the saturation of the area other than the fixation area lower than that of the fixation area, When the stereo image acquired by the image acquisition step is a moving image, if a frame of the stereo image does not satisfy a predetermined condition, the display control step displays the frame on the display device without performing image processing by the image processing step; the predetermined condition is that the magnitude of the difference between the disparity value in the gaze area of ​​the target frame to be displayed on the display device in the display control step and the disparity value in the gaze area of ​​the most recent frame displayed on the display device in the display control step is equal to or less than a predetermined threshold value; An image processing method comprising:

14. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 12.

Images