Image processing apparatus and method, and program

Abstract

The present technology relates to an image processing device and method and a program which make it possible to express light in an image naturally in accordance with a viewpoint. The image processing device calculates information indicating a change in a light source region between an input image and a viewpoint conversion image obtained by performing viewpoint conversion on the input image on the basis of a designated viewpoint, and causes the expression of light in the viewpoint conversion image to be changed on the basis of the calculated information indicating the change in the light source region. The present technology may be applied to an image display system that generates a pseudostereoscopic image having a motion parallax from a single image.

Description

Technical Field

[0001] The present technology relates to an image processing apparatus and method, and a program, and particularly relates to an image processing apparatus and method, and a program that can naturally express light in an image according to the viewpoint.

Background Art

[0002] When generating a pseudo-stereoscopic image with motion parallax based on a plurality of viewpoint images, the expression of light in which the area of the light source region in the image changes due to the change in the positional relationship between the light source and the object caused by the change in the viewpoint is realized by interpolating adjacent viewpoint images (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the technology of Patent Document 1, the above-described light expression can be performed because the technology of Patent Document 1 is a technology in a virtual three-dimensional space premised on CG (Computer Graphics). That is, in the virtual three-dimensional space, since the positional relationship between the light source and the object is known, the above-described light expression can be performed.

[0005] On the other hand, when generating a pseudo-stereoscopic image based on a single image captured in the real space, since the positional relationship between the light source and the object is unknown, it has been difficult to perform the above-described light expression.

[0006] The present technology has been made in view of such a situation, and enables natural expression of light in an image according to the viewpoint.

Means for Solving the Problem

[0007] An image processing apparatus according to one aspect of the present technology includes a light source change calculation unit that calculates information indicating a change in a light source region between an input image and a viewpoint-converted image obtained by viewpoint-converting the input image based on a specified viewpoint, and a light expression control unit that changes the expression of light in the viewpoint-converted image based on the information indicating the change in the light source region calculated by the light source change calculation unit.

[0008] In one aspect of the present technology, information indicating a change in a light source region between an input image and a viewpoint-converted image obtained by viewpoint-converting the input image based on a specified viewpoint is calculated, and based on the calculated information indicating the change in the light source region, the expression of light in the viewpoint-converted image is changed.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments for carrying out the present technology will be described. The description will be made in the following order. 1. Outline of the present technology 2. First Embodiment (TV (Televison) Device with a Camera) 3. Second Embodiment (Light Field Display Device) 4. Others

[0011] <1. Outline of the present technology> (Configuration Example of Image Display System) FIG. 1 is a diagram showing the configuration of an embodiment of an image display system to which the present technology is applied.

[0012] The image display system 1 in FIG. 1 is a system that generates and displays a pseudo-stereoscopic image with motion parallax from a single input image based on line-of-sight information. The pseudo-stereoscopic image consists of a plurality of viewpoint-converted images that are images obtained by performing a plurality of viewpoint conversions on the input image.

[0013] The image display system 1 is composed of, for example, a TV device with a camera capable of acquiring the line-of-sight information of the viewer, or a Light Field Display device that displays images for N viewpoints.

[0014] In the image display system 1, for example, an input image 3E is input when viewed from a viewpoint at a central position facing the display.

[0015] The image display system 1 generates, for example, a viewpoint-converted image 3L or a viewpoint-converted image 3R from a single input image 3E based on the line-of-sight information at the viewing position 2L or position 2R. The image display system 1 displays the generated viewpoint-converted image 3L or viewpoint-converted image 3R on the display.

[0016] At this time, the image display system 1 estimates the light source region in the viewpoint-converted image 3L / 3R based on the line-of-sight information at the viewing position 2L or position 2R and the region corresponding to the light source in the input image 3E (hereinafter referred to as the light source region). Then, the image display system 1 controls the expression of light by calculating information indicating the change in the light source region between the input image 3E and the viewpoint-converted image 3L / 3R.

[0017] FIG. 2 is a diagram showing an example of the effect of this technology.

[0018] In FIG. 2, for example, an input image is shown in which light is irradiated from a light source on the stage in the back of the figure toward the audience in the front of the figure. In the input image, since the stage is located in the back and the audience is located in the front, depending on the position of the hand held up by the audience, the light source and the light irradiated from the light source appear and disappear.

[0019] When the user's viewpoint moves and the hand moves in the direction indicated by the arrow in the input image of FIG. 2 and the "light source" is blocked, due to the control of the light expression of this technology, the viewpoint-converted image is appropriately darkened. Conversely, when the hand moves and the "light source" appears, due to the control of the light expression of this technology, the viewpoint-converted image is appropriately brightened.

[0020] As described above, according to this technology, the light in the image can be naturally expressed according to the viewpoint. As a result, the viewer can obtain a high sense of presence.

[0021] <2. First Embodiment (TV Device with Camera)> (Configuration Example of TV Device with Camera) Figure 3 is a block diagram showing the configuration of a TV device 11 with a camera, which is the first embodiment of the image display system of the present technology.

[0022] The TV device 11 in Figure 3 is composed of an imaging unit 21, a signal processing unit 22, and a display unit 23.

[0023] The imaging unit 21 is provided, for example, above the display unit 23 to image the viewer. The imaging unit 21 is composed of a camera having an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging unit 21 outputs the captured image of the viewer to the signal processing unit 22.

[0024] The signal processing unit 22 consists of a viewpoint information generation unit 31 and an image processing unit 32.

[0025] The viewpoint information generation unit 31 generates the viewpoint information of the viewer based on the image of the viewer captured by the imaging unit 21. The viewpoint information generation unit 31 outputs the generated viewpoint information to the image processing unit 32.

[0026] The image processing unit 32 generates a viewpoint conversion image from the input image input from, for example, a tuner (not shown) in the previous stage based on the viewpoint information supplied from the viewpoint information generation unit 31. The image processing unit 32 outputs the generated viewpoint conversion image to the display unit 23. The input image is basically an image of one viewpoint, but a plurality of viewpoint images may be used.

[0027] At this time, the image processing unit 32 estimates the light source region of the input image, and based on the viewpoint information supplied from the viewpoint information generation unit 31, generates information indicating the light source region of the viewpoint-converted image (hereinafter referred to as viewpoint-converted light source region information) from the information indicating the light source region of the input image (hereinafter referred to as input light source region information). The image processing unit 32 controls the expression of light in the viewpoint-converted image by calculating information indicating the change in the light source region between the input image and the viewpoint-converted image based on the input light source region information and the viewpoint-converted light source information.

[0028] The display unit 23 is composed of an LCD (liquid crystal display), an organic EL (OLED: Organic Light Emitting Diode), or the like. The display unit 23 displays the image supplied from the image processing unit 32.

[0029] (Configuration example of the image processing unit) FIG. 4 is a block diagram showing a configuration example of the image processing unit 32.

[0030] In FIG. 4, the image processing unit 32 is composed of a monocular depth estimation unit 51, a light source estimation unit 52, a viewpoint conversion unit 53, a light source change calculation unit 54, and a light expression control unit 55.

[0031] The input image is input to the monocular depth estimation unit 51, the light source estimation unit 52, the viewpoint conversion unit 53, and the light expression control unit 55. Further, the viewpoint information supplied from the viewpoint information generation unit 31 is input to the viewpoint conversion unit 53.

[0032] The monocular depth estimation unit 51 estimates the depth for each pixel based on the input image. The method of depth estimation is not limited. Representative monocular depth estimation methods include, for example, a high-speed DNN (Deep Neural Network)-based monocular depth estimation method considering real-time processing, and a monocular depth estimation method that enables depth learning from two-viewpoint images without teacher data using machine learning. The monocular depth estimation unit 51 outputs the estimated depth of each pixel to the viewpoint conversion unit 53.

[0033] The light source estimation unit 52 estimates the light source region in the input image.

[0034] The light source is the object itself that emits light, and the emitted light is not included in the light source. The method for estimating the light source region is not particularly limited. The method for estimating the light source region may be a model-based method or a DNN-based method.

[0035] The light source region estimated in the input image is labeled in an appropriate format. For example, for each image, a label representing the "probability of being a light source" from 0.0 to 1.0 is attached. Also, for example, a label representing "whether it is a light source" in a binary manner with 0 or 1 is attached. When there are multiple light source regions, an ID may be assigned to each light source region, and labels may be attached to each light source region.

[0036] The light source estimation unit 52 outputs the image labeled in an appropriate format for the input image as a light source label image, which is information indicating the light source region, to the viewpoint conversion unit 53 and the light source change calculation unit 54.

[0037] Note that in the light source label image, since the information of the pixels to which no label is assigned is unnecessary, the light source label image does not have to be in a general image format.

[0038] The viewpoint conversion unit 53 generates a viewpoint conversion image from the input image based on the viewpoint information supplied from the viewpoint information generation unit 31 and the depth estimated by the monocular depth estimation unit 51. The viewpoint conversion image is an image of the subject in the input image seen from the viewpoint specified in the viewpoint information supplied from the viewpoint information generation unit 31.

[0039] The method for performing the viewpoint conversion is not limited. The method for performing the viewpoint conversion may be a model-based method or a DNN-based method. Also, when using a DNN-based method, the light source label image of the input image may be used as the input to the DNN to perform the viewpoint conversion.

[0040] In addition, the viewpoint conversion unit 53 generates a light source label image indicating the light source region of the viewpoint conversion image (hereinafter referred to as the viewpoint conversion light source label image) from the light source label image indicating the light source region of the input image (hereinafter referred to as the input light source label image) supplied from the light source estimation unit 52, based on the viewpoint information supplied from the viewpoint information generation unit 31 and the depth estimated by the monocular depth estimation unit 51.

[0041] The viewpoint conversion image and the viewpoint conversion light source label image generated by the viewpoint conversion unit 53 are output to the light source change calculation unit 54.

[0042] The light source change calculation unit 54 calculates information indicating the change in the light source region between the input image and the viewpoint conversion image, based on the input light source label image supplied from the light source estimation unit 52 and the viewpoint conversion light source label image supplied from the viewpoint conversion unit 53. The light source change calculation unit 54 outputs the calculated information indicating the change in the light source region to the light expression control unit 55.

[0043] Specifically, the light source change calculation unit 54 calculates a light source area change feature amount representing the change in the area of the light source region between the input image and the viewpoint conversion image, and outputs it to the light expression control unit 55. In addition, the light source change calculation unit 54 generates a light source change image representing the change in the light source region between the input image and the viewpoint conversion image, and outputs it to the light expression control unit 55.

[0044] The light expression control unit 55 controls the expression of light in the viewpoint conversion image, based on at least one of the light source area change feature amount and the light source change image supplied from the light source change calculation unit 54, and the viewpoint conversion light source label image supplied from the viewpoint conversion unit 53. The viewpoint conversion image in which the expression of light is controlled by the light expression control unit 55 is output to the display unit 23.

[0045] Specifically, in the viewpoint conversion image, the light expression control unit 55 causes the histogram of the "region other than the light source region" to be converted according to the increase or decrease in the area of the light source region, and brightens or darkens the region of the "region other than the light source region". A histogram is a graph representing the distribution of the luminance values of the pixels in an image.

[0046] For example, in the real world, if the area of the light source decreases due to occlusion, the amount of intraocular scattered light generated in the human eye decreases, and the overall amount of light decreases and is perceived. Therefore, when the area of the light source region in the image decreases, a more natural light expression can be realized by darkening the region outside the light source region.

[0047] In addition, by using the light source change image, it becomes possible to more naturally express the change in light accompanying the change in viewpoint. For example, when light rays radiate radially from a light source, by learning the correspondence between the radial light, the occluder, and the light source through machine learning, it becomes possible to express light rays according to the change in the shape of the light source and to express a halo around the occluder.

[0048] Next, the light source change calculation process will be described.

[0049] (Details of the light source change calculation process) FIG. 5 is a diagram showing an example of an image used in the light source change calculation process.

[0050] In FIG. 5, from the left, an input image and an input light source label image are shown, and a viewpoint conversion image and a viewpoint conversion light source label image are shown.

[0051] In the input image, an occluder exists at a position away from the light source on the right side of the light source. In the viewpoint conversion image, the occluder overlaps the light source and shields a part of the light source.

[0052] Depending on how the light expression is controlled in the light expression control unit 55 described later, the type of information indicating the required change in the light source region differs. For example, the information indicating the change in the light source region includes the light source area change feature amount and the light source change image described above. Here, the details of the light source area change feature amount and the light source change image will be described.

[0053] (Light source area change feature amount) The light source change calculation unit 54 obtains the integrated value of the pixel values in the entire image area of the light source label image, and the difference L diff and ratio Lrate Output the like to the light expression control unit 55 as a light source area change feature amount. L diff L is expressed as the following formula (1). L rate L' is expressed as the following formula (2). [Number]

[0054] Here, L is the light source area feature amount of the input image, and L' is the light source area feature amount of the viewpoint-transformed image.

[0055] x i represents the pixel value vector at the position i of the input image. The pixel value vector is a set of RGB pixel values. L i represents the value at the position i of the input light source label image. x' i represents the pixel value vector at the position i of the viewpoint-transformed image. L' i represents the value at the position i of the viewpoint-transformed light source label image. f(·) represents an arbitrary function for setting the weight by the pixel value.

[0056] Here, when the dimension of x (for example, 3 dimensions for RGB) is degenerate to a one-dimensional weight such as luminance by f(·), L also becomes one-dimensional and a scalar value. When the dimension of x is not degenerate to a one-dimensional weight such as luminance by f(·), L is the number of dimensions that x has.

[0057] Note that as formula (1), an example of the definition of L diff is shown, and as formula (2), an example of the definition of L rate is shown. However, especially when x is represented as a vector, it is necessary to adopt an appropriate definition depending on what color space is being handled.

[0058] In formulas (1) and (2), a three-dimensional RGB space is assumed. However, the information indicating the change in the light source area may be a quantitative index having meanings such as "difference" and "ratio" regarding the increase or decrease of the light source area. Further, in addition to outputting formula (1) or formula (2) as the light source area change feature amount, for example, a pair of L and L' may be output as the light source area change feature amount.

[0059] Also, when obtaining the integrated value for L and L', for example, the integrated value may be obtained by weighting with the intensity of the light source with respect to luminance or the like.

[0060] (Light source change image) The light source change calculation unit 54 aligns the input light source label image with the viewpoint-transformed light source label image, performs projection, and takes the difference. As a result, a light source change image representing the change (increase or decrease) of the light source area is generated as shown by the arrow in FIG. 5.

[0061] In addition, when there are a plurality of light source areas, an ID may be assigned to each light source area, and the light source area change feature amount, the light source change image, etc. may be obtained for each light source area.

[0062] (Configuration example of the light expression control unit) FIG. 6 is a block diagram showing a configuration example of the light expression control unit.

[0063] In FIG. 6, the light expression control unit 55 is composed of a control signal generation unit 61 and a histogram conversion unit 62.

[0064] Here, the intraocular scattered light generated in a human eye increases or decreases according to the increase or decrease of the shielding area of the light source. However, the intensity of the light of the light source itself does not change. Since the intensity of the light source is usually sufficiently strong with respect to the scattered light, the light in the image can be expressed more naturally without performing histogram conversion on the light source area.

[0065] Therefore, based on the viewpoint conversion light source label image, the control signal generation unit 61 generates a control signal such as a mask signal to exclude the light source area from the histogram conversion. The control signal generation unit 61 outputs the generated control signal to the histogram conversion unit 62. Note that histogram conversion is a process of converting a histogram, which is a graph representing the distribution of luminance values.

[0066] Based on the control signal supplied from the control signal generation unit 61 and in accordance with the light source area change feature amount supplied from the light source change calculation unit 54, the histogram conversion unit 62 performs histogram conversion on the viewpoint conversion image.

[0067] Histogram conversion is performed by a general method such as using a tone curve or gamma conversion. The histogram conversion unit 62 performs histogram conversion by changing the tone curve of the viewpoint conversion image according to the light source area change feature amount, and outputs it as an output image.

[0068] Here, when the light source area change feature amount has a color (that is, a vector representation such as three-dimensional is made), the histogram conversion unit 62 changes the tone curve according to the color. Also, when there are a plurality of light source areas, the histogram conversion unit 62 changes the tone curve in consideration of which light source area's shielding area is changing.

[0069] Note that although an example of performing histogram conversion has been described in the light expression control unit 55, the light expression method is not limited to histogram conversion. For example, more advanced image conversion processing such as removing or adding intraocular scattered light generated in a person's eye may be performed. In this case, the light in the image can be expressed more naturally.

[0070] (Operation of the TV device with a camera) FIG. 7 is a flowchart for explaining the image processing of the image processing unit 32.

[0071] In step S11, the monocular depth estimation unit 51 performs monocular depth estimation. Specifically, the monocular depth estimation unit 51 estimates the depth for each pixel based on the input image. The monocular depth estimation unit 51 outputs the estimated depth of each pixel to the viewpoint conversion unit 53.

[0072] In step S12, the light source estimation unit 52 performs light source estimation in the input image. The light source estimation unit 52 generates an input light source label image by labeling the light source region estimated as the light source in an appropriate format, and outputs it to the viewpoint conversion unit 53 and the light source change calculation unit 54.

[0073] In step S13, the viewpoint conversion unit 53 performs viewpoint conversion. Specifically, the viewpoint conversion unit 53 generates a viewpoint-converted image from the input image based on the viewpoint information supplied from the viewpoint information generation unit 31 and the depth estimated by the monocular depth estimation unit 51. Also, the viewpoint conversion unit 53 generates a viewpoint-converted light source label image from the input light source label image supplied from the light source estimation unit 52 based on the viewpoint information supplied from the viewpoint information generation unit 31 and the depth estimated by the monocular depth estimation unit 51.

[0074] In step S14, the light source change calculation unit 54 calculates information indicating the change in the light source region. Specifically, the light source change calculation unit 54 calculates information indicating the change in the light source region between the input image and the viewpoint-converted image based on the input light source label image supplied from the light source estimation unit 52 and the viewpoint-converted light source label image supplied from the viewpoint conversion unit 53.

[0075] In step S15, the light expression control unit 55 performs light expression control based on the information indicating the change in the light source region calculated by the light source change calculation unit 54. The details of the light expression control will be described later with reference to FIG. 8. By step S15, the light expression in the viewpoint-converted image is controlled, and the light is expressed more naturally.

[0076] After step S15, the viewpoint conversion image with the light expression controlled is output to the display unit 23 as an output image. The display unit 23 displays the viewpoint conversion image.

[0077] (Light expression control process) FIG. 8 is a flowchart for explaining the light expression control process in step S15 of FIG. 7.

[0078] In step S31, the control signal generation unit 61 generates a control signal such as a mask signal based on the viewpoint conversion light source label image. The control signal generation unit 61 outputs the generated control signal to the histogram conversion unit 62.

[0079] In step S32, the histogram conversion unit 62 performs histogram conversion on the viewpoint conversion image according to the light source area change feature amount supplied from the light source change calculation unit 54 based on the control signal supplied from the control signal generation unit 61, and outputs it as an output image. The histogram conversion unit 62 changes, for example, the tone curve of the viewpoint conversion image according to the light source area change feature amount.

[0080] As described above, the light expression in the viewpoint conversion image is controlled, and a viewpoint conversion image that more naturally expresses the change in light accompanying the change in viewpoint is displayed.

[0081] (Another example of light expression control) Referring to FIG. 9, another example of light expression control will be described.

[0082] In FIG. 9, for example, an input image in which radial rays of light emitted from the sun as a light source are pouring through the gaps in the trees is shown. These rays of light are called light streaks or rays of light.

[0083] In FIG. 9, region A and region B are shown. Region A is a region where radial rays of light are emitted from the light source. Region A is a region where it is estimated that the change in light is large due to viewpoint conversion. Region B is a region where it is estimated that the change in light is small due to viewpoint conversion.

[0084] As described above, when light rays radiate radially from a light source in the input image, the light expression control unit 55 uses the light source change image to machine-learn the correspondence relationships among the radial light, the shielding object, and the light source. As a result, when executing the light expression control, the light expression control unit 55 can appropriately represent the light rays (light streaks and rays) when the shape of the light source changes.

[0085] (Configuration example of the light expression control unit during learning) FIG. 10 is a block diagram showing a configuration example of the light expression control unit during the above-described learning. Note that the light expression control unit during learning is realized by being expanded in a RAM (Random Access Memory) or the like by a CPU (Central Processing Unit) such as a personal computer.

[0086] The light expression control unit 55 in FIG. 10 is composed of a viewpoint conversion unit 71 and a light expression conversion unit 72.

[0087] The viewpoint conversion unit 71 performs a viewpoint conversion on the input image by the method used by the viewpoint conversion unit 53 in FIG. 4, and outputs the viewpoint conversion image to the light expression conversion unit 72.

[0088] For the learning of the light expression conversion unit 72, a plurality of viewpoint images obtained using a Light Field technique or the like are prepared. The viewpoint images may be created by CG or the like.

[0089] In the light expression conversion unit 72, one viewpoint image is defined as a student image, and the viewpoint image corresponding to the viewpoint of the viewpoint conversion destination of the student image (seen from the viewpoint of the viewpoint conversion destination of the student image) is defined as a teacher image. Note that the learning set is a group of images of pairs of teacher images and student images. It is desirable that the learning set richly contains images accompanied by a change in the area of the light source when performing a viewpoint conversion from the student image to the teacher image.

[0090] The light expression conversion unit 72 uses machine learning such as DNN to learn how to transform the viewpoint conversion image supplied from the viewpoint conversion unit 71 and the light source change image supplied from the light source change calculation unit 54 as student images so as to approach the teacher image.

[0091] During this learning, even if the input image itself is not given to the light expression conversion unit 72, it is possible to specify how the light source has changed due to the viewpoint conversion by the light source change image. Thereby, it is possible to learn how to change the light expression in association with the change of the light source.

[0092] In addition, when the input image before the viewpoint conversion is added to the student image, the light expression conversion unit 72 can learn the conversion method while considering more "how the conversion is performed by the viewpoint conversion unit 71".

[0093] Also, by performing the above-described learning process while switching the teacher image among a plurality of viewpoints, in the viewpoint conversion image, since the light rays (light strips and glows) change as the shape of the light source region changes, it is expected to learn the change of the light rays.

[0094] (Configuration example of the light expression control unit at the time of execution) FIG. 11 is a block diagram showing a configuration example of the light expression control unit when performing light expression control after learning.

[0095] In FIG. 11, the light expression control unit 55 is composed of the light expression conversion unit 72 of FIG. 10.

[0096] An input image, a viewpoint conversion image supplied from the viewpoint conversion unit 53, and a light source change image supplied from the light source change calculation unit 54 are input to the light expression conversion unit 72. Note that, as described above, the input image is used to consider more "how the conversion is performed by the previous viewpoint conversion unit 53" during learning, and is not necessarily essential.

[0097] The light expression conversion unit 72 changes the light rays (light streaks and glows) in the viewpoint conversion image based on the light source change image and the learned correspondence relationships of the radial light, occluders, and light sources, and outputs it as an output image.

[0098] (Another example of light expression control processing) FIG. 12 is a flowchart for explaining another example of the light expression control processing in step S15 of FIG. 7.

[0099] In step S61, the light expression conversion unit 72 converts the light expression in the viewpoint conversion image. Specifically, the light expression conversion unit 72 changes the light rays (light streaks and glows) in the viewpoint conversion image based on the light source change image and the learned correspondence relationships of the radial light, occluders, and light sources, and outputs it as an output image.

[0100] As described above, the expression of the light rays in the viewpoint conversion image is controlled, and a viewpoint conversion image that more naturally represents the change in light accompanying the change in viewpoint is displayed.

[0101] (Another configuration example of the light expression control unit) FIG. 13 is a block diagram showing another configuration example of the light expression control unit.

[0102] In FIG. 13, the light expression control unit 55 is configured by combining each part of the light expression control unit 55 in FIGS. 6 and 11.

[0103] That is, the light expression control unit 55 is composed of the light expression conversion unit 72 in FIG. 11, the control signal generation unit 61 in FIG. 6, and the histogram conversion unit 62 in FIG. 6.

[0104] The light expression conversion unit 72 uses the viewpoint conversion image supplied from the viewpoint conversion unit 53, the light source change image supplied from the light source change calculation unit 54, and the learned correspondence relationships of the radial light, occluders, and light sources to change the light rays (light streaks and glows) in the viewpoint conversion image and convert the light expression. The light expression conversion unit 72 outputs the viewpoint conversion image with the converted light expression to the histogram conversion unit 62.

[0105] Based on the viewpoint conversion light source label image corresponding to the viewpoint conversion image supplied to the light expression conversion unit 72, the control signal generation unit 61 generates a control signal such as a mask signal in order to exclude the light source region from the histogram conversion. The control signal generation unit 61 outputs the generated control signal to the histogram conversion unit 62.

[0106] Based on the control signal supplied from the control signal generation unit 61 and according to the light source area change feature amount supplied from the light source change calculation unit 54, the histogram conversion unit 62 performs histogram conversion on the viewpoint conversion image whose light expression has been converted by the light expression conversion unit 72, and outputs it as an output image.

[0107] (Another example of light expression control processing) FIG. 14 is a flowchart for explaining another example of the light expression control processing in step S15 of FIG. 7.

[0108] In step S71, the light expression conversion unit 72 converts the light expression in the viewpoint conversion image in the same manner as the process of step S61 in FIG. 12.

[0109] In step S72, based on the viewpoint conversion light source label image corresponding to the viewpoint conversion image supplied to the light expression conversion unit 72, the control signal generation unit 61 generates a control signal such as a mask signal. The control signal generation unit 61 outputs the generated control signal to the histogram conversion unit 62.

[0110] In step S73, based on the control signal supplied from the control signal generation unit 61 and according to the light source area change feature amount supplied from the light source change calculation unit 54, the histogram conversion unit 62 performs histogram conversion on the viewpoint conversion image whose light expression has been converted by the light expression conversion unit 72. The viewpoint conversion image on which the histogram conversion has been performed is output as an output image.

[0111] As described above, the expression of light rays in the viewpoint-converted image is converted, the brightness and darkness of light are controlled, and a viewpoint-converted image that more naturally expresses the change in light accompanying the change in viewpoint is displayed.

[0112] Note that the conversion of the expression of light described above in step S71 of FIG. 14 may be performed after histogram conversion.

[0113] <3. Second Embodiment (Light Field Display Device)> (Configuration Example of Light Field Display Device) FIG. 15 is a block diagram showing the configuration of a Light Field Display device which is a second embodiment of the image display system of the present technology.

[0114] The Light Field Display device 101 in FIG. 15 generates viewpoint-converted images for N viewpoints from an input image, and displays the generated viewpoint-converted images for N viewpoints.

[0115] The Light Field Display device 101 is composed of a signal processing unit 111 and a Light Field Display 112.

[0116] The signal processing unit 111 is composed of a control unit 121 and image processing units 122-1 to 122-N.

[0117] The control unit 121 outputs viewpoint information 1 to N to the image processing units 122-1 to 122-N. Each of the viewpoint information 1 to N corresponds to the viewpoint-converted images for N viewpoints output by the Light Field Display 112. The viewpoint information 1 to N is fixed for each Light Field Display device.

[0118] The image processing units 122-1 to 122-N are configured in the same manner as the image processing unit 32 in FIG. 3. The image processing units 122-1 to 122-N generate a viewpoint-converted image from an input image supplied from, for example, a tuner (not shown) in a previous stage, based on each viewpoint information 1 to N supplied from the control unit 121. The image processing units 122-1 to 122-N output the generated viewpoint-converted image to the Light Field Display 112.

[0119] At this time, the image processing units 122-1 to 122-N, similar to the image processing unit 32, convert the input image into a viewpoint-converted image and control the expression of light in the viewpoint-converted image.

[0120] The Light Field Display 112 is composed of an LCD, an organic EL, etc. The Light Field Display 112 has display areas 131-1 to 131-N for displaying the viewpoint-converted images of viewpoints 1 to N. The Light Field Display 112 displays the viewpoint-converted images of viewpoints 1 to N supplied from the image processing units 122-1 to 122-N as output images 1 to N in the corresponding display areas 131-1 to 131-N.

[0121] Note that the operations of the image processing units 122-1 to 122-N in the Light Field Display device 101 are the same as the operations of the image processing unit 32 in FIG. 3 described above with reference to FIG. 7, so the description thereof is omitted.

[0122] <4. Others> (Effect) As described above, in the present technology, information indicating a change in the light source area between the input image and the viewpoint-converted image obtained by viewpoint-converting the input image based on the specified viewpoint is calculated, and based on the calculated information indicating the change in the light source area, the expression of light in the viewpoint-converted image is changed.

[0123] Therefore, according to the present technology, when the area of the light source occupying the image changes as a result of viewpoint conversion, the light in the image can be naturally expressed according to the viewpoint.

[0124] (Configuration example of a computer) The above series of processes can be executed either by hardware or by software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer in which the program is incorporated in dedicated hardware, or a general-purpose personal computer or the like.

[0125] FIG. 16 is a block diagram showing a configuration example of the hardware of a computer that executes the above series of processes by a program.

[0126] The CPU 301, ROM (Read Only Memory) 302, and RAM 303 are interconnected by a bus 304.

[0127] An input / output interface 305 is further connected to the bus 304. An input unit 306 composed of a keyboard, a mouse, etc., and an output unit 307 composed of a display, a speaker, etc. are connected to the input / output interface 305. Also, a storage unit 308 composed of a hard disk, a non-volatile memory, etc., a communication unit 309 composed of a network interface, etc., and a drive 310 for driving a removable medium 311 are connected to the input / output interface 305.

[0128] In the computer configured as described above, the CPU 301 loads and executes, for example, a program stored in the storage unit 308 into the RAM 303 via the input / output interface 305 and the bus 304, whereby the above series of processes is performed.

[0129] The program executed by the CPU 301 is recorded, for example, on the removable medium 311, or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and installed in the storage unit 308.

[0130] Note that the program executed by the computer may be a program whose processing is performed in time series according to the order described in this specification, or a program whose processing is performed in parallel or at a necessary timing such as when a call is made.

[0131] Note that in this specification, the system means a collection of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and one device in which a plurality of modules are housed in one housing are both systems.

[0132] Also, the effects described in this specification are merely examples and are not limiting, and there may be other effects.

[0133] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0134] For example, the present technology can adopt a configuration of cloud computing in which one function is shared and jointly processed by a plurality of devices via a network.

[0135] Also, each step described in the above flowchart can be executed by one device or can be shared and executed by a plurality of devices.

[0136] Furthermore, when a plurality of processes are included in one step, the plurality of processes included in that one step can be executed by one device or can be shared and executed by a plurality of devices.

[0137] <Example of configuration combination> The present technology can also adopt the following configuration. (1) A light source change calculation unit that calculates information indicating a change in a light source region between an input image and a perspective-transformed image obtained by perspective-transforming the input image based on a specified perspective; A light expression control unit that changes the expression of light in the perspective-transformed image based on the information indicating the change in the light source region calculated by the light source change calculation unit; An image processing apparatus comprising the above. (2) The image processing apparatus according to (1), further comprising a perspective transformation unit that generates the perspective-transformed image based on the specified perspective. The image processing apparatus according to (1) above. (3) The image processing apparatus further comprises a light source estimation unit that estimates the light source region in the input image and generates input light source region information indicating the light source region in the input image. The perspective transformation unit generates perspective-transformed light source region information indicating the light source region in the perspective-transformed image. The light source change calculation unit calculates information indicating the change in the light source region based on the input light source region information and the perspective-transformed light source region information. The perspective transformation unit generates perspective-transformed light source region information indicating the light source region in the perspective-transformed image. The light source change calculation unit calculates information indicating the change in the light source region based on the input light source region information and the perspective-transformed light source region information. The image processing apparatus according to (2) above. (4) The light source change calculation unit calculates a light source area change feature amount indicating a change in the area of the light source region based on the input light source region information and the perspective-transformed light source region information. The light expression control unit changes the expression of light in the perspective-transformed image based on the light source area change feature amount calculated by the light source change calculation unit and the perspective-transformed light source region information. The light expression control unit changes the expression of light in the perspective-transformed image based on the light source area change feature amount calculated by the light source change calculation unit and the perspective-transformed light source region information. The image processing apparatus according to (3) above. (5) The light expression control unit changes the histogram of a region other than the light source region of the perspective-transformed image based on the change in the area of the light source region. The image processing apparatus according to (4) above. (6) The light source change calculation unit generates a light source change image indicating a change in the shape of the light source region in the image based on the input light source region information and the perspective-transformed light source region information. The light expression control unit changes the expression of light in the viewpoint conversion image based on the light source change image generated by the light source change calculation unit. The image processing apparatus according to (3) above. (7) The light expression control unit changes the expression of light in the viewpoint conversion image based on the light source change image and the input image generated by the light source change calculation unit. The image processing apparatus according to (6) above. (8) The light expression control unit changes the light streaks and light rays in the viewpoint conversion image based on the light source change image generated by the light source change calculation unit. The image processing apparatus according to (6) above. (9) The light source estimation unit generates the input light source region information by attaching a label indicating the estimated light source region to the input image. The image processing apparatus according to (3) above. (10) The image processing apparatus further includes a depth estimation unit that estimates the depth of the pixels constituting the input image. The viewpoint conversion unit generates the viewpoint conversion image and the viewpoint conversion light source region information based on the depth estimated by the depth estimation unit. The image processing apparatus according to (3) above. (11) The input image is a single image. The image processing apparatus according to any one of (1) to (10) above. (12) An image processing apparatus calculates information indicating a change in the light source region between an input image and a viewpoint conversion image obtained by viewpoint-converting the input image based on a specified viewpoint, and changes the expression of light in the viewpoint conversion image based on the calculated information indicating the change in the light source region. An image processing method. (13) a light source change calculation unit that calculates information indicating a change in the light source region between an input image and a viewpoint conversion image obtained by viewpoint-converting the input image based on a specified viewpoint, Based on the information indicating the change in the light source area calculated by the light source change calculation unit, a light expression control unit that changes the expression of light in the viewpoint conversion image, and A program for causing a computer to function.

Explanation of Signs

[0138] 1 Image display system, 11 TV device with camera, 21 Imaging unit, 22 Signal processing unit, 23 Display unit, 31 Viewpoint information generation unit, 32 Image processing unit, 51 Monocular depth estimation unit, 52 Light source estimation unit, 53 Viewpoint conversion unit, 54 Light source change calculation unit, 55 Light expression control unit, 61 Control signal generation unit, 62 Histogram conversion unit, 71 Viewpoint conversion unit, 72 Light expression conversion unit, 101 Light Field Display device, 111 Signal processing unit, 112 Light Field Display, 121 Control unit, 122-1 to 122-N Image processing unit, 131-1 to 131-N Display area

Claims

Claim 1: A light source estimation unit that estimates a light source region in a captured image and generates captured image light source region information indicating the light source region in the captured image; A viewpoint conversion unit that generates viewpoint conversion light source region information indicating the light source region in a viewpoint-converted image obtained by viewpoint-converting the captured image based on a specified viewpoint; A light source change calculation unit that calculates a light source area change feature amount indicating a change in the area of the light source region between the captured image and the viewpoint-converted image based on the captured image light source region information and the viewpoint conversion light source region information; A light expression control unit that changes the expression of light in the viewpoint-converted image based on the light source area change feature amount calculated by the light source change calculation unit and the viewpoint conversion light source region information An image processing apparatus comprising the above. Claim 2: The viewpoint conversion unit generates the viewpoint-converted image based on the specified viewpoint The image processing apparatus according to claim 1. Claim 3: Further comprising a depth estimation unit that estimates the depth of pixels constituting the captured image, The viewpoint conversion unit generates the viewpoint-converted image based on the depth estimated by the depth estimation unit The image processing apparatus according to claim 2. Claim 4 The light expression control unit changes the histogram of a region other than the light source region of the viewpoint-converted image based on a change in the area of the light source region. The image processing apparatus according to claim 1. Claim 5 The light source change calculation unit also generates a light source change image indicating a change in the shape of the light source region in the image based on the captured image light source region information and the viewpoint conversion light source region information, The light expression control unit changes the expression of light in the viewpoint-converted image also based on the light source change image generated by the light source change calculation unit. The image processing apparatus according to claim 1. Claim 6 The light expression control unit changes the expression of light in the viewpoint-converted image also based on the light source change image generated by the light source change calculation unit. The image processing apparatus according to claim 5. Claim 7 The light expression control unit changes light streaks and light rays in the viewpoint-converted image based on the light source change image generated by the light source change calculation unit. The image processing apparatus according to claim 5. Claim 8 The light source estimation unit generates the captured image light source region information by attaching a label indicating the estimated light source region to the captured image. The image processing apparatus according to claim 1. Claim 9 further comprising a depth estimation unit that estimates the depth of pixels constituting the captured image; the viewpoint conversion unit generates the viewpoint conversion light source region information based on the depth estimated by the depth estimation unit The image processing apparatus according to claim 1.

10. the captured image is a single image The image processing apparatus according to claim 1.

11. An image processing apparatus estimates a light source region in a captured image and generates captured image light source region information indicating the light source region in the captured image; generates viewpoint conversion light source region information indicating the light source region in a viewpoint-converted image obtained by viewpoint-converting the captured image based on a specified viewpoint; calculates a light source area change feature amount indicating a change in the area of the light source region between the captured image and the viewpoint-converted image based on the captured image light source region information and the viewpoint conversion light source region information; changes the expression of light in the viewpoint-converted image based on the light source area change feature amount and the viewpoint conversion light source region information An image processing method.

12. A light source estimation unit that estimates a light source region in a captured image and generates captured image light source region information indicating the light source region in the captured image; a viewpoint conversion unit that generates viewpoint conversion light source region information indicating the light source region in a viewpoint-converted image obtained by viewpoint-converting the captured image based on a specified viewpoint; a light source change calculation unit that calculates a light source area change feature amount indicating a change in the area of the light source region between the captured image and the viewpoint-converted image based on the captured image light source region information and the viewpoint conversion light source region information; a light expression control unit that changes the expression of light in the viewpoint-converted image based on the light source area change feature amount calculated by the light source change calculation unit and the viewpoint conversion light source region information A program for causing a computer to function.

Images