Image processing device, image processing method and program

Abstract

Reduces image quality degradation caused by the screen door effect. [Solution] The image processing device has an image brightness acquisition means for acquiring image brightness information indicating the brightness of an image to be displayed on a specified display device where the screen door effect is visible, a correction means for correcting the image to be darker if the brightness of the image indicated by the acquired image brightness information is brighter than a specified brightness, and a display control means for displaying the corrected image on the specified display device.

Description

[Technical Field]

[0001] The present disclosure relates to a technique for viewing images on a head-mounted display. [Background technology]

[0002] There are head-mounted displays (hereinafter referred to as "HMDs") that are worn on the user's head and allow the user to view magnified images such as video through eyepieces, providing a greater sense of realism. However, if the display resolution is low, a phenomenon known as the screen door effect occurs, in which the non-emitting areas between the pixels of the display appear as a mesh pattern, and this may prevent the user (hereinafter referred to as the "wearer") from experiencing a high sense of realism.

[0003] With regard to reducing degradation in image quality of an image observed by a user, Patent Document 1 discloses a technique for adjusting color and contrast based on information relating to the amount of light obtained from a video signal. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-176773 Summary of the Invention [Problem to be solved by the invention]

[0005] Even if the technology of Patent Document 1 is adopted to reduce image quality degradation of images observed through an HMD, it can only reduce image quality degradation caused by the external environment, and cannot reduce image quality degradation caused by the screen door effect.

[0006] The present disclosure aims to provide a technique for reducing image quality degradation due to the screen door effect. [Means for solving the problem]

[0007] An image processing device according to one aspect of the present disclosure is characterized by having an image brightness acquisition means for acquiring image brightness information indicating the brightness of an image to be displayed on a specified display device on which the screen door effect is visible, a correction means for correcting the image to be darker when the brightness of the image indicated by the acquired image brightness information is brighter than a specified brightness, and a display control means for displaying the corrected image on the specified display device. [Effects of the Invention]

[0008] According to the technology of the present disclosure, it is possible to reduce image quality degradation caused by the screen door effect. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an outline of an HMD. [Figure 2] FIG. 1 is a diagram illustrating a hardware configuration of an image processing apparatus. [Figure 3] FIG. 1 is a diagram showing the screen door effect of an HMD. [Figure 4] This is a spectral distribution diagram of the screen door effect of an HMD. [Figure 5] FIG. 10 is a diagram showing an example of a VTF curve relating to the visibility of the screen door effect of an HMD. [Figure 6] FIG. 2 is a diagram illustrating a functional configuration of the image processing apparatus. [Figure 7] FIG. 10 is a diagram showing the difference in visibility of the screen door effect due to differences in brightness. [Figure 8] FIG. 10 is a diagram relating to determination of a correction amount. [Figure 9] 10 is a flowchart showing the flow of processing executed by the image processing apparatus. [Figure 10] FIG. 2 is a diagram illustrating a functional configuration of the image processing apparatus. [Figure 11] FIG. 10 is a diagram illustrating an example of correction processing. [Figure 12] FIG. 10 is a diagram for explaining correction using information on changes over time. [Figure 13]10 is a flowchart showing the flow of processing executed by the image processing apparatus. [Figure 14] FIG. 2 is a diagram illustrating a functional configuration of the image processing apparatus. [Figure 15] 10A and 10B are schematic diagrams illustrating differences in visibility of the screen door effect depending on the subject. [Figure 16] 10 is a flowchart showing the flow of processing executed by the image processing apparatus. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the technology of the present disclosure according to the claims. Not all combinations of features described in the present embodiments are necessarily essential as solutions to the technology of the present disclosure, and multiple features may be combined arbitrarily. Identical configurations will be described with the same reference numerals. In this specification, brightness will be referred to as luminance. Furthermore, each process (step) in a flowchart will be denoted with an "S" at the beginning.

[0011] <<Embodiment 1>> In this embodiment, a mode will be described in which processing is performed to make the mesh pattern caused by the screen door effect less visible to the wearer of the HMD, which is a predetermined display device.

[0012] 1 is a schematic diagram showing an overview of an HMD. The HMD 100 is a head-mounted display device worn on the user's head, and includes a display unit main body 110 and a wearing tool main body 120. The display unit main body 110 includes a housing 111, a left eyepiece 112L, a right eyepiece 112R, a left eye display 113L, and a right eye display 113R. The wearing tool main body 120 includes a strap 121 and a strap length adjustment unit 122.

[0013] The strap 121 is a band strap for attaching the display unit main body 110 of the HMD 100 to the head of a user who is observing an image displayed on the HMD 100. Both ends of the strap 121 are attached to the side surfaces of the display unit main body 110 of the HMD 100 so as to be swingable around the attachment points of both ends.

[0014] The strap length adjustment unit 122 is a device for adjusting the length of the strap 121. For example, by adjusting the length of the strap 121 with the strap length adjustment unit 122, the HMD 100 can be worn on the user's head so that the eyepieces 112L and 112R for the left and right eyes are positioned corresponding to the left and right eyes of the user.

[0015] The left-eye and right-eye eyepieces 112L and 112R are provided at a distance from each other on the rear surface of the housing 111. The left-eye display 113L is arranged facing the left-eye eyepiece 112L, and the right-eye display 113R is arranged facing the right-eye eyepiece 112R. Images input to the HMD 100 are composed of, for example, a left-side image displayed on the left-eye display 113L and a right-side image displayed on the right-eye display 113R. Since the left-eye and right-eye displays 113L and 113R display images, they can be said to correspond to the display units of a predetermined display device. The wearer observes the left-side image through the left-eye eyepiece 112L with their left eye and the right-side image through the right-eye eyepiece 112R with their right eye. The left-eye and right-eye displays 113L and 113R may be configured as independent displays, or may be configured as a single display that displays the left image on the left side and the right image on the right side.

[0016] The HMD 100 also functions as an image processing device, and performs various types of image processing such as correction processing.

[0017] (Hardware configuration of image processing device) 2 is a diagram showing an example of the hardware configuration of an image processing device. The image processing device 200 is configured as an application example of the HMD 100, detects the positions of the left and right eyes of the wearer when wearing the HMD 100, and changes the sharpness of the image based on the detected eye positions.

[0018] The image processing device 200 includes a CPU 201 , a RAM 202 , a ROM 203 , a VC (video card) 204 , a SATA (serial ATA) I / F 205 , a general-purpose I / F 206 , a NIC (network interface card) 207 , and a system bus 208 .

[0019] The CPU 201 uses the RAM 202 as a work memory to execute an OS (operating system) and various programs stored in the ROM 203, HDD (hard disk drive) 210, etc. The CPU 201 also controls each component via a system bus 208. Note that the processing of the flowcharts described below is performed by the CPU 201 after program code stored in the ROM 203, HDD 210, etc. is loaded into the RAM 202. A display device 211 such as a display is connected to the VC 204. The display device 211 corresponds to the left-eye display 113L and right-eye display 113R of the HMD 100. An input device 231 such as a mouse and keyboard and an imaging device 232 are connected to the general-purpose I / F 206 via the system bus 208. A general-purpose drive 222 that reads and writes data from the HDD 221 and various recording media is connected to the SATA I / F 205 via the system bus 208. The NIC 207 inputs and outputs information to and from external devices. The CPU 201 uses various recording media mounted on the HDD 221 or the general-purpose drive 222 as storage locations for various data. The CPU 201 displays a GUI (Graphical User Interface) provided by a program on the display device 211, and receives inputs such as user instructions via the input device 231.

[0020] (Screen door effect) In this embodiment, correction is performed when the wearer can see the screen door effect. First, a situation in which the screen door effect is visible will be described using FIG. 3. FIG. 3 is a diagram for explaining the screen door effect. FIG. 3(a) schematically shows an input image, and FIG. 3(b) schematically shows how the input image shown in FIG. 3(a) appears when viewed by a wearer through an eyepiece lens, as displayed on the display of an HMD. Note that the input image 301 shown in FIG. 3(a) is a uniformly white image input to the HMD and displayed on the left-eye or right-eye display 113L or 113R. Compare FIG. 3(a) with FIG. 3(b). Although the input image 301 does not have a mesh pattern, the wearer sees a mesh pattern in image 302, as shown in FIG. 3(b), because the eyepiece lens magnifies the pixels of the display and the non-light-emitting areas between the pixels. This is the phenomenon known as the screen door effect.

[0021] (power spectrum distribution) The screen door effect occurs at the pixel period of the display, and therefore appears as a peak component in spatial frequency. The graph in Figure 4 shows an example of the spectral distribution of the HMD display as seen by the wearer shown in Figure 3(b). In Figure 4, the horizontal axis represents spatial frequency (cycles / degree). In Figure 4, the vertical axis represents the frequency characteristics obtained by performing a two-dimensional Fourier transform on the image shown in Figure 3(b) and averaging the pixel values ​​of pixels equidistant from the center of the image, resulting in a power spectrum image. Figure 4 shows a peak component at 20 cycles / degree, which was confirmed to be the mesh pattern visible as the screen door effect in Figure 3(b).

[0022] (Visual sensitivity curve) FIG. 5 is a diagram showing an example of a human visual sensitivity curve (hereinafter referred to as a VTF (Visual Transfer Function) curve). In FIG. 5, the horizontal axis represents spatial frequency (cycles / degree), and the vertical axis represents visual sensitivity expressed as a numerical value between 0.0 and 1.0. The VTF curve shown in FIG. 5 shows the relationship between spatial frequency and human visual sensitivity. The higher the value on the vertical axis, the higher the human visual sensitivity. In the VTF curve shown in FIG. 5, the peak frequency of visual sensitivity varies depending on the distance from the visual object. 500 mm was set as an example of the distance between the wearer's eyes and the virtual image of the subject when viewed with an HMD. The distance is not limited to 500 mm, and may be set to a distance shorter or longer than 500 mm.

[0023] Figure 5 shows that visual sensitivity reaches its maximum at a frequency of 5 cycles / degree. As the spatial frequency increases, visual sensitivity decreases. This confirms that the higher the spatial frequency, the lower the human visual sensitivity. Figure 5 also confirms that when the spatial frequency of the screen-door effect is 20 cycles / degree, the VTF curve value is approximately 0.2. Since the spatial frequency that exhibits the screen-door effect is a frequency band where visual sensitivity remains, this indicates that the screen-door effect is visible. Note that a visual sensitivity of 0 indicates that the screen-door effect is not visible. Figure 5 also confirms that the spatial frequency at which visual sensitivity is 0 is 43 cycles / degree. Therefore, the screen-door effect is not visible when the spatial frequency is greater than 43 cycles / degree, but is visible when the spatial frequency is below a certain threshold of 43 cycles / degree.

[0024] As a means for determining whether the screen door effect is visible based on the HMD, for example, the angle of view per pixel can be calculated from the angle of view and display resolution of the left-eye or right-eye display 113L or 113R of the HMD. For example, if the angle of view of the left-eye display 113L of the HMD is 90 × 90° and the display resolution is 1800 × 1800 pixels, the angle of view per pixel, which is the angle per pixel, is 0.05°. In this case, there are 20 cycles per degree, so this can be expressed as 20 cycles / degree. Alternatively, the screen door effect may be determined based on an image captured by an imaging device through the HMD's eyepiece. This can also be said to determine whether the screen door effect is visible based on the physical quantity of the spacing between non-light-emitting portions of the display. For example, an imaging device is placed directly facing the left-eye or right-eye eyepiece 112L or 112R at a position that mimics the user's eyes observing the image. If the viewing angle of the imaging device is 50 x 50 degrees and 20 cycles of screen door are observed per degree in the horizontal and vertical directions, the spatial frequency of the screen door effect can be said to be 20 cycles / degree. Also, whether or not the screen door effect is visible can be determined based on the area ratio of all pixels to one pixel on the HMD display. Note that the methods for determining whether or not the screen door effect is visible are not limited to these methods.

[0025] From the above, if it is determined that the screen door effect is visible, processing is performed in the following functional units.

[0026] (Functional configuration of image processing device) 6 is a block diagram showing an example of the functional configuration of the image processing device 200 of this embodiment. Note that the image displayed on the HMD 100, which also functions as an image processing device, may be an image that can be displayed on the left-eye or right-eye display 113L or 113R. For example, the image may be an image captured by an imaging device mounted on the HMD 100, or an image (moving image) or still image stored in the HDD 221 of the HMD 100. The image processing device 200 includes an image luminance acquisition unit 601, a correction unit 602, and a display control unit 603.

[0027] The image luminance acquisition unit 601 converts the RGB color signal values ​​of each pixel of an image input to the image processing device 200 and displayed on the HMD 100 into luminance values ​​to acquire the luminance values ​​of the image. The luminance value of an image indicates the brightness of the image, and the image luminance acquisition unit 601 can also be said to acquire image brightness information indicating the brightness of the image. Specifically, the image luminance acquisition unit 601 acquires all frames constituting the image from the HDD 221 or the imaging device 232 of the image processing device 200 and acquires the luminance values ​​of the image. The luminance values ​​may be acquired from all acquired frames or from frames at a specific interval among all acquired frames. The gamma RGB color signal values ​​of the display are then converted into a luminance value Y using the following formula, and the luminance value resulting from the conversion is acquired:

[0028] Y=0.299R+0.587G+0.114B If the RGB color signal values ​​of the image displayed on the HMD 100 are not linear but have undergone gamma conversion, the RGB color signal values ​​are converted to linear values ​​using inverse gamma conversion, and then converted to luminance values ​​using the above conversion formula. Furthermore, the image luminance acquisition unit 601 may acquire the luminance value of each pixel as the luminance value of the image, or may acquire the average luminance value of an arbitrary region. The luminance values ​​of the image acquired by the image luminance acquisition unit 601 are output to the correction unit 602.

[0029] Correction unit 602 performs a luminance value correction process using as input the luminance values ​​acquired by video luminance acquisition unit 601 and all frames constituting the video acquired from video luminance acquisition unit 601. The results of the luminance value correction process are output to display control unit 603.

[0030] (Difference in visibility of the screen door effect due to differences in brightness) Here, an example of one frame constituting an image is shown in FIG. 7 , and how the screen-door effect appears depending on the brightness of one frame is explained. FIG. 7 illustrates the difference in visibility of the screen-door effect depending on the brightness. FIG. 7( a) shows how the display appears to the wearer when one uniformly black frame is displayed on the display of the HMD 100, and FIG. 7( b) shows how the display appears to the wearer when one uniformly white frame is displayed on the display of the HMD 100. Because the screen-door effect is a black mesh pattern, it was confirmed that the following relationship exists between the frames constituting the image and the screen-door effect. As shown in FIG. 7( a), it was confirmed that the screen-door effect is difficult to see in the dark frame 701 constituting the image. On the other hand, as shown in FIG. 7( b), it was confirmed that the screen-door effect is easily seen and noticeable in the bright frame 702 constituting the image. Therefore, since the screen-door effect is more noticeable when the frame is high in brightness, a correction process is performed to reduce the brightness value on the high-brightness side compared to when the frame is low in brightness.

[0031] (Brightness value correction) 8(a) and 8(b) are diagrams showing examples of the amount of correction performed by the correction unit 602 on frames whose luminance values ​​acquired by the image luminance acquisition unit 601 are expressed in the range of 0 to 255. In FIGS. 8(a) and 8(b), the horizontal axis represents the luminance value of the image acquired by the image luminance acquisition unit 601, and the vertical axis represents the luminance value of the image after correction. Note that FIG. 8(a) shows a case where, when the viewing angle is 110° and the high spatial frequency is assumed to be 4K, the spatial frequency is approximately 34 cycles / degree. FIG. 8(b) shows a case where, when the viewing angle is 110° and the low spatial frequency is assumed to be 2K, the spatial frequency is approximately 19 cycles / degree. In other words, FIGS. 8(a) and 8(b) each show that an image is corrected with a correction amount determined from the spacing between non-emitting portions of pixels on the display and the luminance value indicated as brightness information.

[0032] 8 performs linear conversion up to a given brightness value, and corrects brightness values ​​above that value so that they become darker. Note that the correction value may be set arbitrarily depending on how noticeable the screen door effect is.

[0033] For example, Figure 8(a) shows a correction table 810 that is used when the spatial frequency of the screen door effect is high, and that is linear 811 up to a luminance value of 180, but becomes nonlinear 812 when the luminance value exceeds 180. It can also be said that correction table 810 indicates that the image is corrected to be darker when the image brightness indicated by the image brightness information is within the brighter 30% of the range of image brightness indicated by the image brightness information. When the spatial frequency of the screen door effect is high, that is, when the period of the non-emitting portions between display pixels is small, the screen door effect is less noticeable and the amount of image degradation is smaller. Therefore, the amount of correction is smaller.

[0034] FIG. 8(b) shows a correction table 820 used when the spatial frequency of the screen door effect is small. The correction table 820 is linear 821 up to a luminance value of 130, but becomes nonlinear 822 when the luminance value exceeds 130. Correction table 820 can also be said to indicate that the image is corrected to be darker when the image brightness indicated by the image brightness information is within the brighter 50% of the range of image brightness indicated by the image brightness information. When the spatial frequency of the screen door effect is small, that is, when the period of non-emitting areas between display pixels is large, the screen door effect becomes more noticeable, resulting in greater image quality degradation. Therefore, the amount of correction is greater than in the case of FIG. 8(a).

[0035] From Figures 8(a) and 8(b), it can be said that when the spatial frequency of the screen door effect is relatively small, the image is corrected to be darker as the image brightness indicated by the image brightness information becomes brighter, compared to when the spatial frequency of the screen door effect is relatively large.

[0036] As described above, the correction unit 602 uses two types of correction tables 810 and 820 that are prepared in advance and correspond to the magnitude of the spatial frequency of the screen door effect, but this is not limiting. For example, three types of correction tables that are prepared in advance and correspond to the magnitude of the spatial frequency of the screen door effect may be used.

[0037] The display control unit 603 displays the corrected images obtained by correcting the luminance values ​​of the images in the correction unit 602 on the left-eye and right-eye displays 113L and 113R.

[0038] (Processing performed by the image processing device) Fig. 9 is a flowchart showing the flow of processing executed by the image processing device 200. The CPU 201 reads out a program for realizing the flowchart shown in Fig. 9 stored in the ROM 203 or the HDD 221, and executes it using the RAM 202 as a work area. As a result, the CPU 201 fulfills the roles of the respective functional components shown in Fig. 6.

[0039] In S901, the image luminance acquisition unit 601 acquires luminance values ​​of an image obtained by converting RGB color signal values ​​of each pixel of an image input to the image processing device 200 and displayed on the HMD 100 from the HDD 221 or the imaging device 232. The acquired luminance values ​​of the image are output to the correction unit 602.

[0040] In S902, correction unit 602 performs a luminance value correction process using as input the luminance values ​​of the video acquired by video luminance acquisition unit 601 in S901 and all frames constituting the video acquired from video luminance acquisition unit 601. The results obtained from the luminance value correction process are output to display control unit 603.

[0041] In S903, the display control unit 603 displays the images obtained as a result of the correction process of the luminance values ​​performed by the correction unit 602 in S902 on the left-eye and right-eye displays 113L and 113R.

[0042] As described above, in this embodiment, the brightness values ​​of an image displayed on an HMD that allows the screen door effect to be visible are acquired, and correction is performed to darken the brightness values ​​at which the screen door effect is noticeable. By displaying the image after such correction on the HMD, the screen door effect caused by the display pixels and the non-light-emitting portions between pixels can be made less noticeable when the wearer views the image displayed on the HMD. This makes it difficult for the wearer to see the mesh pattern caused by the screen door effect, preventing a decrease in realism and discomfort for the wearer.

[0043] <<Embodiment 2>> In this embodiment, information on the change in brightness over time is acquired, and an example of changing the correction amount when an image whose brightness value is within a predetermined range continues to be displayed will be described. In this embodiment, the differences from the first embodiment will be mainly described.

[0044] (visual characteristics) In general, human vision has a visual characteristic called adaptation, which allows the eyes to unconsciously adjust pupil diameter according to the amount of ambient light, thereby adjusting the sensitivity of the retina to an appropriate state. Adjusting pupil diameter and retinal sensitivity takes a certain amount of time, but once this adjustment is complete, it becomes possible to detect subtle differences in brightness that were previously invisible. For example, if you are in a dimly lit room, you may initially be unable to recognize the shapes of objects in the dark, but after a certain amount of time, your eyes will adapt and the shapes of the objects in the room will become visible. Given this visual characteristic, if you observe images with brightness values ​​within a certain range for a certain period of time using an HMD, you will be able to detect subtle differences in brightness as your eyes adapt, which may make the screen door effect more noticeable.

[0045] In this embodiment, in addition to the correction in the first embodiment, a change in luminance value over time is acquired, and the luminance value is corrected when an image in which the brightness is within a predetermined range is observed for a certain period of time. In this embodiment, a method is described in which information on the change in luminance value of an image over time is acquired, and the amount of correction of the luminance value is changed based on the acquired information. Note that the hardware configuration of the image processing device in this embodiment is the same as in the first embodiment, and therefore a detailed description thereof will be omitted.

[0046] (Functional configuration of image processing device) 10 is a block diagram showing an example of the functional configuration of an image processing device 1000 according to this embodiment. Similar to the image processing device 600, the image processing device 1000 includes an image luminance acquisition unit 601, a correction unit 602, and a display control unit 603, and further includes a time information acquisition unit 1001 and a correction amount change unit 1002.

[0047] The image luminance acquisition unit 601, correction unit 602, and display control unit 603 of the image processing device 1000 perform the same processes as those of the image processing device 600 of embodiment 1, and therefore detailed description thereof will be omitted. In this embodiment, a time information acquisition unit 1001 and a correction amount change unit 1002, which are newly added compared to embodiment 1, will be described.

[0048] The time information acquisition unit 1001 acquires information on changes in luminance values ​​over time of an image displayed on the HMD 100. When the image displayed on the HMD 100 is a moving image, the information on changes in luminance values ​​refers to time information on changes in luminance values ​​acquired from previous and subsequent frames and frames acquired at specific intervals. When the displayed image is a still image, the information on changes in luminance values ​​refers to time information on the continuous display of the same image. For example, if the luminance values ​​remain unchanged within a predetermined range in all frames of a 10-second moving image and this state continues, information on changes in luminance values ​​over time indicating that "the luminance values ​​have remained within the predetermined range for 10 seconds" is acquired. Note that the information on changes in luminance values ​​over time may be acquired for each pixel or for a specific region. Examples of specific regions include, but are not limited to, each region divided into frames constituting the image, or a region containing a specific object and the rest of the frame.

[0049] The correction amount change unit 1002 uses information on the change in luminance value over time acquired by the time information acquisition unit 1001 to change the amount of correction according to eye adaptation for the image for which the luminance value has been corrected by the correction unit 602. For example, let us assume that the adaptation time of the human eye is 2 minutes, and the wearer watches an image for 180 seconds continuously in which the luminance value falls within a predetermined range. The luminance value falling within the predetermined range is assumed to be a magnitude in which the human eye cannot detect the change in luminance. When expressed in terms of lightness, ΔL *Assume that the luminance value is approximately 1.0. In the same image, FIG. 11(a) shows the luminance values ​​corrected by the correction unit 602, and FIG. 11(b) shows the luminance values ​​after the correction amount change unit 1002 has changed the correction amount for the luminance values ​​corrected by the correction unit 602. In FIGS. 11(a) and 11(b), the horizontal axis represents time (seconds) and the vertical axis represents the luminance value. In FIG. 11(a), the magnitude of the luminance value remains unchanged from 0 to 180 seconds and falls within a predetermined range. In contrast, when the correction amount change unit 1002 changes the correction amount, as shown in FIG. 11(b), the correction amount is changed over 120 seconds (2 minutes), which is the eye's adaptation time, and the luminance value is gradually reduced. When the luminance value remains almost unchanged and falls within the predetermined range for a certain period of time or more, the correction amount can be changed to make the screen door effect less noticeable according to the eye's adaptation. For example, if the brightness value remains unchanged and within a predetermined range for 90 seconds, the brightness value is corrected to a value corresponding to 90 seconds. If the brightness value remains unchanged and within a predetermined range for 200 seconds, the brightness value is corrected to a value corresponding to 180 seconds. In other words, if the brightness value is greater than 180 seconds, the brightness value is corrected to the same value as for 180 seconds.

[0050] (Correction of brightness values ​​using information on changes over time) Fig. 12 is a diagram for explaining correction using information on changes over time. Fig. 12(a) shows a case where the correction amount is not changed, and Fig. 12(b) shows a case where the correction amount is changed.

[0051] When a tunnel image switches from a dark image to a bright image, t seconds pass from the state immediately after the switch, and the screen door effect becomes more noticeable over time.

[0052] In Figure 12(a), it was confirmed that t seconds have passed since image 1201, which is the state immediately after the switch, and image 1202, in which the screen door effect becomes more noticeable over time, is obtained. From this image, information on the change in luminance over time is obtained, and t seconds have passed since the state immediately after the switch from a dark image to a bright image, at the timing when the eyes have adapted, the amount of correction is changed using the information on the change in luminance over time. It was confirmed that image 1212, in which the amount of correction has been changed in this way, has a less noticeable screen door effect than image 1202 shown in Figure 12(a).

[0053] The region for which the correction amount change unit 1002 changes the luminance value of the image may be a specific region within the entire region where the luminance value continues to fall within a predetermined range, or the entire region. Examples of specific regions include, but are not limited to, a region that is brighter than its surroundings, each region into which the image is divided, a region including a specific object and the rest of the region, etc. The eye adaptation time is also not limited to this. As described above, the image for which the correction amount change unit 1002 has changed the correction amount is output to the display control unit 603.

[0054] (Processing performed by the image processing device) Fig. 13 is a flowchart showing the flow of processing executed by the image processing device 1000. In this embodiment, the CPU 201 reads out a program that realizes the flowchart shown in Fig. 13 stored in the ROM 203 or the HDD 221, and executes it using the RAM 202 as a work area. This causes the CPU 201 to fulfill the roles of the various functional components shown in Fig. 10. The processing of S901, S902, and S903 shown in Fig. 13 is the same as in the first embodiment, and therefore detailed description thereof will be omitted. When the processing of S901 in the flow shown in Fig. 13 is completed, the processing proceeds to S1301.

[0055] In S1301, the time information acquisition unit 1001 acquires information on changes in luminance values ​​over time from the image. The information on changes in luminance values ​​over time may be acquired from luminance value information acquired by the video luminance acquisition unit 601. The information on changes in luminance values ​​over time acquired by the time information acquisition unit 1001 is output to the correction unit 602. When the processing of S1301 is completed, the processing proceeds to S902. The video corrected in S902 is output to the correction amount change unit 1002. When the processing of S902 is completed, the processing proceeds to S1302.

[0056] In S1302, if the video corrected by the correction unit 602 in S902 continues for a certain period of time with almost no change in luminance value, the correction amount change unit 1002 changes the correction amount according to eye adaptation using information on the change in luminance value over time acquired by the time information acquisition unit 1001. Specifically, for example, assume that processing is performed on a 300-second video. The video is of a vehicle traveling through a tunnel, and the first 90 seconds of the 300 seconds are filled with frames showing dark luminance values ​​inside the tunnel. After 100 seconds, a series of bright frames of the vehicle exiting the tunnel continues for 200 seconds. Within this video, the time information acquisition unit 1001 acquires the above-mentioned time change information. The acquired time change information is sent to the correction amount change unit 1002 along with the video. The correction amount change unit 1002 performs correction on the video. The correction is not performed on the first 100 seconds of the video inside the tunnel. When the luminance changes after 100 seconds, the correction amount is changed according to eye adaptation. If the eye adaptation time is 120 seconds, the correction is performed so that the luminance value is gradually reduced over the 120 seconds. This correction is maintained for the remaining time. After the process of S1302 is completed, the process proceeds to S903. In S903, the display control unit 603 of this embodiment displays the image with the changed correction amount. After the process of S903 is completed, the flow shown in FIG. 13 ends.

[0057] As described above, in this embodiment, the change in luminance value over time is acquired, and if the image shows almost no change in luminance value for a certain period of time, the luminance value is re-corrected to a value that corresponds to the human visual characteristics. This makes it possible to reduce the visibility of the screen door effect caused by adaptation, which is a human visual characteristic.

[0058] <<Embodiment 3>> In this embodiment, a frequency analysis is performed on an image, and the strength (correction amount) of correction is changed for each region according to the frequency analysis results. Specifically, for an image determined to have a frequency at which the screen door effect is not noticeable based on the frequency analysis results, the correction amount set before the analysis is reduced. This change in correction strength to reduce the correction amount is performed by determining a correction amount that matches the frequency characteristics for each region constituting the image. In this embodiment, the correction amount is determined based on the frequency characteristics in addition to the luminance value of the image, so more appropriate correction can be performed on the image, thereby improving image quality. Note that this embodiment will mainly describe the differences from embodiment 1. Since the hardware configuration of the image processing device of this embodiment is the same as that of embodiment 1, a detailed description thereof will be omitted.

[0059] (Functional configuration of image processing device) 14 is a block diagram showing an example of the functional configuration of an image processing device 1400 according to this embodiment. Similar to the image processing device 600, the image processing device 1400 includes a video luminance acquisition unit 601, a correction unit 602, and a display control unit 603, and further includes a frequency analysis unit 1401.

[0060] The image luminance acquisition unit 601 and display control unit 603 included in the image processing device 1400 perform the same processing as in the image processing device 600 of the first embodiment, and therefore detailed description thereof will be omitted. In this embodiment, a frequency analysis unit 1401 that is newly added compared to the first embodiment will be described. Also, the correction unit 602 included in the image processing device 1400 performs processing that is partially different from that in the first embodiment.

[0061] The frequency analysis unit 1401 performs frequency analysis on the video acquired by the video brightness acquisition unit 601. Because the screen door effect occurs at the pixel cycle of the display, areas with high-frequency components in an image input to and displayed on an HMD are masked by the frequency of the image and are therefore less noticeable. On the other hand, areas with low-frequency components are more noticeable due to the screen door effect, a phenomenon in which non-light-emitting areas between pixels are visible. Utilizing this characteristic, the frequency analysis is performed on the video acquired by the video brightness acquisition unit 601, and the strength (amount of correction) of correction is changed according to the frequency of the video. The frequency analysis may be performed by detecting objects from the video and performing it for each detected object, or by dividing the frames constituting the video into regions and performing it for each divided region. The frequency analysis may be performed on each frame, or on frames acquired at specific intervals, such as every 10 frames.

[0062] (Screen door effect visibility difference) FIG. 15 is a schematic diagram showing differences in visibility of the screen door effect depending on the subject. FIG. 15 illustrates an example of an image in which a tree and the sky are the subjects (objects). When a tree 1502 and a blue sky 1503 are used as subjects and frequency analysis is performed on each of the tree 1502 and the blue sky 1503, the spatial frequency of the tree 1502 is high because the leaves of the trees overlap in a complex manner. For example, a high frequency refers to when the luminance frequency in the image is 35 cycles / degree. Therefore, the screen door effect is not noticeable in high-frequency regions.

[0063] In contrast, the blue sky 1503 is a uniform image, and therefore the spatial frequency within the region is low. For example, low frequency refers to when the luminance frequency within the image is 0.5 cycles / degree. Therefore, in low-frequency regions, the screen door effect becomes noticeable. The results of the analysis of these spatial frequencies are passed to the correction unit 602.

[0064] The correction unit 602 of the image processing device 1400 performs correction on the luminance values ​​acquired by the image luminance acquisition unit 601 while varying the correction strength based on the frequency analysis results for different regions within the same frame obtained by the frequency analysis unit 1401. For example, since the screen door effect is not noticeable in high-frequency regions such as trees 1502, a table with a small correction amount as shown in FIG. 8(a) is used. On the other hand, since the screen door effect is noticeable in low-frequency regions such as blue sky 1503, a table with a large correction amount as shown in FIG. 8(b) is used. It can be said that the correction unit 602 of the image processing device 1400 corrects the image so that regions where the analysis results are low-frequency are darker than regions where the analysis results are high-frequency. The image to which the correction amount corresponding to the frequency analysis results has been applied for each region is sent to the display control unit 603.

[0065] (Processing performed by the image processing device) Fig. 16 is a flowchart showing the flow of processing executed by the image processing device 1400. In this embodiment, the CPU 201 reads out a program that realizes the flowchart shown in Fig. 16 stored in the ROM 203 or the HDD 221, and executes it using the RAM 202 as a work area. This causes the CPU 201 to fulfill the roles of the various functional components shown in Fig. 14. The processing of S901, S902, and S903 shown in Fig. 16 is the same as in the first embodiment, and therefore detailed description thereof will be omitted. When the processing of S901 in the flow shown in Fig. 16 is completed, the processing proceeds to S1601.

[0066] In S1601, frequency analysis unit 1401 performs frequency analysis on the video acquired from video brightness acquisition unit 601. The frequency analysis result is output to correction unit 602. When the process of S1601 ends, the process proceeds to S902.

[0067] In S902, the correction unit 602 receives the frequency analysis result from the frequency analysis unit 1401 and the video acquired from the video brightness acquisition unit 601 as input, and performs correction while changing the intensity for regions with different frequencies within the same frame. The correction result is output to the display control unit 603. When the processing of S902 is completed, the process proceeds to S903.

[0068] In S903, the display control unit 603 of the image processing device 1400 displays the image corrected in S902. When the process of S903 ends, the flow shown in FIG.

[0069] An example of an image input in S1601 will be described with reference to FIG. 15. For example, in a high frequency region such as a tree 1502 shown in FIG. 15, the screen door effect is not noticeable, so a table with a small amount of correction such as that shown in FIG. 8(a) is used. On the other hand, in a low frequency region such as a blue sky 1503 shown in FIG. 15, the screen door effect is noticeable, so a table with a large amount of correction such as that shown in FIG. 8(b) is used. The images to which different amounts of correction have been applied depending on the region are sent to the display control unit 603.

[0070] As described above, in this embodiment, the frequency of an image is analyzed, and the correction amount is reduced in areas of the image that have high-frequency components, and increased in areas that have low-frequency components, thereby changing the strength (amount) of correction according to the magnitude of the frequency. By correcting the image in this way, it is possible to perform more appropriate correction, making the screen door effect even less noticeable and improving the image quality.

[0071] The above describes the case where the brightness value of the image is used, but this is not limited to this, and instead of the brightness value of the image, brightness expressed by at least one numerical value from among brightness, luminous intensity, luminous flux, and illuminance may be used.

[0072] <Other embodiments> The present disclosure can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. The program may also be provided by recording it on a computer-readable storage medium.

[0073] The disclosure of this embodiment includes the following configuration examples. (Configuration 1) an image brightness acquisition means for acquiring image brightness information indicating the brightness of an image to be displayed on a predetermined display device in which the screen door effect is visually recognized; a correction means for correcting the image so as to darken it when the brightness of the image indicated by the acquired image brightness information is brighter than a predetermined brightness; a display control means for displaying the corrected image on the predetermined display device; 1. An image processing device comprising: (Configuration 2) The correction means corrects the image so that the brighter the image indicated by the image brightness information, the darker the image becomes. 2. The image processing device according to configuration 1, (Configuration 3) The correction means When the spatial frequency of the screen door effect is relatively large, the image is corrected to be darker when the brightness of the image indicated by the image brightness information is within a range of 30% on the bright side of the brightness range of the image indicated by the image brightness information. 3. The image processing device according to configuration 1 or 2. (Configuration 4) The correction means When the spatial frequency of the screen door effect is relatively small, the image is corrected to be darker when the brightness of the image indicated by the image brightness information is within a range of 50% on the bright side of the brightness range of the image indicated by the image brightness information. 3. The image processing device according to configuration 1 or 2. (Configuration 5) The correction means When the spatial frequency of the screen door effect is relatively small, the image is corrected so that the image is darker as the brightness of the image indicated by the image brightness information increases, compared to when the spatial frequency of the screen door effect is relatively large. 5. The image processing device according to any one of configurations 1 to 4. (Configuration 6) 6. The image processing device according to any one of configurations 1 to 5, wherein the image brightness information is a luminance value converted from an RGB color signal value of each pixel of the image. (Configuration 7) time information acquiring means for acquiring time information indicating a time period during which the brightness is within a predetermined range; a change means for changing the brightness of the image corrected by the correction means so as to become darker in accordance with the length of time indicated by the acquired time information; and 7. The image processing device according to any one of configurations 1 to 6, wherein the display control means displays the image whose brightness has been changed by the change means. (Configuration 8) further comprising an analysis means for performing a frequency analysis on the image; The correction means corrects the image based on the acquired image brightness information and the analysis result by the analysis means. 8. The image processing device according to any one of configurations 1 to 7, (Configuration 9) The correction means corrects the image so that a region where the analysis result by the analysis means is a low frequency is darker than a region where the analysis result by the analysis means is a high frequency. 9. The image processing device according to configuration 8, (Configuration 10) The correction means corrects the image with a correction amount determined from the interval between non-light-emitting portions of the pixels of the display unit and the brightness information. 10. The image processing device according to any one of configurations 1 to 9, (Configuration 11) In the display unit, the period of non-light-emitting portions between pixels is equal to or less than a specific threshold value. 11. The image processing device according to any one of configurations 1 to 10. (Configuration 12) The specific threshold value is calculated from any one of an angle per pixel calculated from the angle of view of the display unit and the resolution of the display unit, an area ratio of one pixel to all pixels in the display unit, and a physical quantity of the interval between the non-light-emitting portions in the display unit. 12. The image processing device according to configuration 11. (Configuration 13) The specific threshold is 43 cycles / degree or less. 13. The image processing device according to configuration 11 or 12. (Configuration 14) The brightness is expressed by at least one value of brightness, luminous intensity, brightness, luminous flux, and illuminance. 14. The image processing device according to any one of configurations 1 to 13, (Configuration 15) The predetermined display device is a head-mounted display that is worn on the user's head. 15. The image processing device according to any one of configurations 1 to 14. (Configuration 16) an image brightness acquisition step of acquiring image brightness information indicating the brightness of an image to be displayed on a predetermined display device in which the screen door effect is visually recognized; a correction step of correcting the image so as to darken it when the brightness of the image indicated by the acquired image brightness information is brighter than a predetermined brightness; a display control step of displaying the corrected image on the predetermined display device; An image processing method comprising: (Configuration 17) 17. A program for causing a computer to execute the image processing method according to claim 16.

Claims

1. an image brightness acquisition means for acquiring image brightness information indicating the brightness of an image displayed on a predetermined display device in which the screen door effect is visually recognized; a correction means for correcting the image so as to darken it when the brightness of the image indicated by the acquired image brightness information is brighter than a predetermined brightness; a display control means for displaying the corrected image on a display unit of the predetermined display device; 1. An image processing device comprising:

2. The correction means corrects the image so that the brighter the image indicated by the image brightness information, the darker the image becomes.

2. The image processing device according to claim 1, wherein:

3. The correction means When the spatial frequency of the screen door effect is relatively large, the image is corrected to be darker when the brightness of the image indicated by the image brightness information is within a range of 30% on the bright side of the brightness range of the image indicated by the image brightness information.

2. The image processing device according to claim 1, wherein:

4. The correction means When the spatial frequency of the screen door effect is relatively small, the image is corrected to be darker when the brightness of the image indicated by the image brightness information is within a range of 50% on the bright side of the brightness range of the image indicated by the image brightness information.

2. The image processing device according to claim 1, wherein:

5. The correction means When the spatial frequency of the screen door effect is relatively small, the image is corrected so that the image is darker as the brightness of the image indicated by the image brightness information increases, compared to when the spatial frequency of the screen door effect is relatively large.

2. The image processing device according to claim 1, wherein:

6. 2. The image processing apparatus according to claim 1, wherein the image brightness information is a luminance value converted from an RGB color signal value of each pixel of the image.

7. time information acquiring means for acquiring time information indicating a time period during which the brightness is within a predetermined range; a change means for changing the brightness of the image corrected by the correction means so as to become darker in accordance with the length of time indicated by the acquired time information; and 2. The image processing apparatus according to claim 1, wherein said display control means displays the image whose brightness has been changed by said change means.

8. further comprising an analysis means for performing a frequency analysis on the image; The correction means corrects the image based on the acquired image brightness information and the analysis result by the analysis means.

2. The image processing device according to claim 1, wherein:

9. The correction means corrects the image so that a region where the analysis result by the analysis means is a low frequency is darker than a region where the analysis result by the analysis means is a high frequency.

9. The image processing device according to claim 8,

10. The correction means corrects the image with a correction amount determined from the interval between non-light-emitting portions of the pixels of the display unit and the brightness information.

2. The image processing device according to claim 1, wherein:

11. In the display unit, the period of non-light-emitting portions between pixels is equal to or less than a specific threshold value.

2. The image processing device according to claim 1, wherein:

12. The specific threshold value is calculated from any one of an angle per pixel calculated from the angle of view of the display unit and the resolution of the display unit, an area ratio of one pixel to all pixels in the display unit, and a physical quantity of the interval between the non-light-emitting portions in the display unit.

12. The image processing device according to claim 11.

13. The specific threshold is 43 cycles / degree 12. The image processing device according to claim 11.

14. The brightness is expressed by at least one value of brightness, luminous intensity, brightness, luminous flux, and illuminance.

2. The image processing device according to claim 1, wherein:

15. The predetermined display device is a head-mounted display that is worn on the user's head.

2. The image processing device according to claim 1, wherein:

16. an image brightness acquisition step of acquiring image brightness information indicating the brightness of an image to be displayed on a predetermined display device in which the screen door effect is visually recognized; a correction step of correcting the image so as to darken it when the brightness of the image indicated by the acquired image brightness information is brighter than a predetermined brightness; a display control step of displaying the corrected image on the predetermined display device; An image processing method comprising:

17. A program for causing a computer to execute the image processing method according to claim 16.

Images