Image processing apparatus, method, and program

Abstract

An image processing apparatus according to an embodiment includes: a depth estimation unit that estimates the depth of an image; a depth optimization unit that outputs a depth map of the image by applying, to the depth estimated by the depth estimation unit, a depth optimization function used for mapping the depth; and a setting unit that performs histogram analysis of the depth map output by the depth optimization unit and sets depth values in a display screen of the image on the basis of the distribution of the depth values indicated by the result of the analysis.

Description

Technical Field

[0001] Embodiments of the present invention relate to an image processing apparatus, method, and program.

Background Art

[0002] To generate a three-dimensional (3D) image from a two-dimensional (2D) image, a depth map representing depth information of the image is used. A depth map is data generated by mapping distance information (depth information) from a user's viewpoint for each pixel of the image. A depth map can be represented in grayscale. In the case of a general 8-bit gradation (0 to 255), the farthest depth is represented by the minimum value of 0 (black), and the nearest depth is represented by the maximum value of 255 (white).

[0003] Also, when a depth map is generated from a 2D monocular image or a stereo image (two images with parallax taken from the positions of the left and right eyes respectively), data may be biased in a certain depth range, and the available value range of the bit gradation may not be used effectively. Remapping such a depth map so as to use the value range to the maximum extent is called optimizing the depth.

[0004] For example, in Non-Patent Document 1, based on the fact that the range where the user can effectively feel the parallax is around the display surface, the position where the object of interest exists is set as the display surface, and a method of processing the range from the 5th percentile to the 95th percentile of the value range of the depth range and non-linearly remapping the depth information is described.

[0005] The function used when the depth is remapped as described above is called a depth optimization function. When depth optimization processing is performed for each of different images, the depth optimization function differs depending on the depth distribution within the image.

[0006] Also, Non-Patent Document 2 describes a method of deriving a disparity layer from the analysis result of the histogram of disparity and optimizing for a certain range of depths in the layer where the object of interest exists. Thereby, the depth of details in the object of interest can be sufficiently represented.

Prior Art Documents

Non-Patent Documents

[0007]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0008] When producing a 3D image, in order to express the sense of depth of the image, it is important to determine which part of the image will be the display surface without a popping-out feeling, and how much sense of depth should be set in front of and behind this display surface.

[0009] When the above technology for optimizing depth is directly applied to a moving image (video), the setting of the display surface is not considered. For example, when the depth is expressed in 256 levels, a fixed central value (the value at the 128th level) is set as the display surface, so there is a possibility that an appropriate three-dimensional effect cannot be obtained for the user. That is, as a result of the depth optimization process, when there is an object (such as a person) with a depth value of 128, this object does not feel a three-dimensional effect.

[0010] Specifically, when 3D representation is performed, an object located in front of the display surface is represented as popping out from the display surface, and an object located behind the display surface is represented as receding. Also, when the display surface is set at the center point of the depth value, an object (such as a person) that happens to be set at the corresponding depth does not have a sense of depth expressed.

[0011] This invention is made by paying attention to the above circumstances, and its object is to provide an image processing apparatus, method, and program that can appropriately set the depth value corresponding to the display screen in an image where depth is expressed.

Means for Solving the Problem

[0012] An image processing apparatus according to an aspect of the present invention includes a depth estimation unit that estimates the depth of an image, and a depth optimization processing unit that applies a depth optimization function used for mapping the depth to the depth estimated by the depth estimation unit to output a depth map of the image, and a setting unit that performs a histogram analysis of the depth map output by the depth optimization processing unit and sets the value of the depth in the display screen of the image based on the distribution of the depth values indicated by the result of this analysis.

[0013] An image processing method according to an aspect of the present invention is a method performed by an image processing apparatus, and includes a depth estimation unit that estimates the depth of an image, a depth optimization processing unit that applies a depth optimization function used for mapping the depth to the depth estimated by the depth estimation unit to output a depth map of the image, and a setting unit that performs a histogram analysis of the depth map output by the depth optimization processing unit and sets the value of the depth in the display screen of the image based on the distribution of the depth values indicated by the result of this analysis.

Effects of the Invention

[0014] According to the present invention, it is possible to appropriately set the depth value corresponding to the display screen in an image in which the depth is expressed.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing an application example of a depth map generation device according to an embodiment of the present invention. As shown in FIG. 1, a depth map generation device 100, which is an image processing device according to an embodiment of the present invention, includes a depth estimation unit 11, a depth optimization processing unit 12, an inter-frame difference calculation unit 13, a display surface information determination unit 14, and a depth map correction unit 15.

[0017] FIG. 2 is a flowchart showing an example of the processing operation by the depth map generation device according to an embodiment of the present invention. In this embodiment, a moving image composed of a plurality of frames continuous in time series will be described, but a still image can also be the target as long as it is an image including depth information.

[0018] The inter-frame difference calculation unit 13 inputs image information, which is a moving image from the outside, for example, a monocular image or a stereo image, and calculates an inter-frame difference, which is difference information between a frame to be processed (sometimes referred to as a subsequent frame) in this image information and a frame that is continuous in time series with respect to the frame and is the previous frame in time series (sometimes referred to as a previous frame) (S11).

[0019] The depth estimation unit 11 inputs the above image information, estimates the depth information of each frame of this image information, and outputs the estimated depth information to the depth optimization processing unit 12 (S12). Note that instead of the estimation process by the depth estimation unit 11, a depth camera image, which is an image associated with a time stamp, may be used as the above depth information (see (A) in FIG. 1).

[0020] Based on the depth information from the depth estimation unit 11 and a predetermined depth optimization function, the depth optimization processing unit 12 performs depth optimization processing to map the depth information corresponding to each frame to each frame, and outputs the depth map related to each frame to the display surface information determination unit 14 and the depth map correction unit 15 (S13).

[0021] The display surface information determination unit 14 analyzes the depth map from the depth optimization processing unit 12, and based on the result of this analysis and the inter-frame difference from the inter-frame difference calculation unit 13, calculates a display surface determination index value (hereinafter, may be simply referred to as an index value) for the depth value indicated by the depth map. Among each depth value, the depth value indicated by the lowest index value is determined as the display surface, and the display surface information indicated by this depth value is output to the depth map correction unit 15 (S14).

[0022] Based on the depth map from the depth optimization processing unit 12 and the display surface information from the display surface information determination unit 14, the depth map correction unit 15 corrects the depth map so that the depth value corresponding to the display surface becomes a predetermined depth value, and outputs it as the final depth map (S15).

[0023] Next, the details of the processing of each part of the depth map generation device 100 will be described. FIG. 3 is a diagram showing an example of processing related to the setting of a display surface by a depth map generation device according to an embodiment of the present invention. The display surface information determination unit 14 of the depth map generation device 100 has a function of analyzing the depth map (FIG. 3(a)) and setting the display surface so that the value estimated to have a subject does not correspond to the display surface.

[0024] The display surface information determination unit 14 controls such that depth values with a high possibility of the presence of a subject to be gazed at are not set on the display surface based on information related to the depth map. Specifically, the display surface information determination unit 14 calculates a display surface determination index value (hereinafter referred to as the index value) for each depth value such that depth values with a high possibility of the presence of a subject have a high index value and depth values with a low possibility have a low index value, and sets, for example, a relatively low depth value on the display surface when this index value is low.

[0025] The display surface information determination unit 14 performs histogram analysis of the depth map, and sets a high value for the index value related to the depth value with a high frequency among the depth values indicated by this analysis result ((b) in FIG. 3).

[0026] At this time, when setting the index value, the display surface information determination unit 14 may set a high value for the index value with respect to a certain width centered on the peak value instead of the ratio occupied by simple depth values. Also, when there is a result of performing segmentation processing or tracking processing on the material image associated with the depth map, and when there is an object (for example, a person, etc.) that tends to be gazed at, the display surface information determination unit 14 may set a high index value for the depth value associated with the region of this object ((c) in FIG. 3).

[0027] Furthermore, when the depth map of the processing target has an extended range of depth values due to depth optimization processing, the display surface information determination unit 14 may consider the influence of the change in depth value, and set a high index value when the amount of change in depth value is large and a low index value when the amount of change is small ((d) in FIG. 3).

[0028] When a display surface is set for each of consecutive frames in a video, if the display surface fluctuates rapidly, it may cause discomfort when viewing a 3D video. Therefore, the display surface information determination unit 14 corrects the index value by adding a value corresponding to the difference from the depth value set for the display surface of the previous frame to the index value of each depth value shown above (Fig. 3(e)), and can set the depth value that is the source of the minimum corrected index value for the display surface. The index values related to the more frequently used depth values described above may be corrected by adding the values shown in Figs. 3(c) to (e) together. Also, it is not necessary that all the values from (c) to (e) be added, and they can be arbitrarily selected and corrected as described above.

[0029] As described above, by setting the display surface according to the index value, it is possible to avoid setting the object that is the fixation point for the display surface.

[0030] However, when a scene change occurs between consecutive frames, since there is no problem even if the display surface fluctuates, when a scene change is detected, the display surface information determination unit 14 does not have to consider the index value corresponding to the difference from the display surface of the previous frame. Examples of methods for detecting a scene change include methods such as the inter-frame difference calculation unit 13 obtaining the inter-frame difference and detecting that a scene change has occurred when this difference is a value equal to or greater than a certain value.

[0031] As another example, when the depth value of the entire subject fluctuates according to the movement of the camera, such as a zoom operation or camera work, etc., it is possible that the user's discomfort may be reduced if the display surface fluctuates seamlessly. For this reason, the depth map correction unit 15 of the depth map generation device 100 can correct the index value based on the fluctuation of the camera parameters.

[0032] The depth map correction unit 15 corrects the depth map by using the information on the display surface set as described above. Specifically, the depth map correction unit 15 corrects each depth value of the depth map so that it becomes the depth value specified by the display surface.

[0033] For example, when the depth value set on the display surface is 120 and the depth value is to be changed to 128 after the display surface is corrected, the depth map correction unit 15 extends the depth values in the range from 0 to 120 to the range from 0 to 128, and compresses the depth values in the range from 121 to 256 to the range from 129 to 256.

[0034] FIG. 4 is a block diagram showing an example of the hardware configuration of a depth map generation device according to an embodiment of the present invention. In the example shown in FIG. 4, the depth map generation device 100 according to the above embodiment is constituted by, for example, a server computer or a personal computer, and has a hardware processor 111A such as a CPU (Central Processing Unit). Then, a program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to the hardware processor 111A via a bus 120.

[0035] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables information to be transmitted and received to and from a communication network NW. As the wireless interface, for example, an interface adopting a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0036] Connected to the input / output interface 113 are an input device 200 and an output device 300 that are attached to the depth map generation device 100 and used by users or the like. The input / output interface 113 captures operation data input by users or the like through an input device 200 such as a keyboard, a touch panel, a touchpad, or a mouse, and performs a process of outputting and displaying the output data to an output device 300 including a display device using liquid crystal or organic EL (Electro Luminescence). Note that devices built into the depth map generation device 100 may be used for the input device 200 and the output device 300, or input devices and output devices of other information terminals capable of communicating with the depth map generation device 100 via the network NW may be used.

[0037] The program memory 111B is a non-temporary tangible storage medium, for example, a combination of a non-volatile memory such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) that can be written to and read from at any time and a non-volatile memory such as a ROM (Read Only Memory), and stores programs necessary for executing various control processes according to one embodiment.

[0038] The data memory 112 is a tangible storage medium, for example, a combination of the above non-volatile memory and a volatile memory such as a RAM (Random Access Memory), and is used to store various data acquired and created during the process of performing various processes.

[0039] The depth map generation device 100 according to an embodiment of the present invention can be configured as a data processing device having a depth estimation unit 11, a depth optimization processing unit 12, an inter-frame difference calculation unit 13, a display surface information determination unit 14, and a depth map correction unit 15 shown in FIG. 1 as processing functional units by software.

[0040] Each information storage unit used as a working memory or the like by each part of the depth map generation device 100 can be configured by using the data memory 112 shown in FIG. 4. However, these configured storage areas are not essential components within the depth map generation device 100. For example, they may be areas provided in an external storage medium such as a USB (Universal Serial Bus) memory or a storage device such as a database server arranged in the cloud.

[0041] The processing functional units in each of the above-mentioned depth estimation unit 11, depth optimization processing unit 12, inter-frame difference calculation unit 13, display surface information determination unit 14, and depth map correction unit 15 can all be realized by causing the hardware processor 111A to read and execute the program stored in the program memory 111B. Note that some or all of these processing functional units may be realized in other various forms including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0042] In this embodiment, in order to represent an effective 3D image, attention is paid to the fact that there is a relationship between the depth at which the fixation point exists in the original image and the depth set on the display surface, and by preventing the object that is the fixation point from being arranged on the display surface, it is possible to realize the automatic generation of 3D content that is more likely to obtain a stereoscopic effect.

[0043] Also, the methods described in each embodiment can be stored as a program (software means) to be executed by a computer on a recording medium such as a magnetic disk (e.g., a floppy disk, a hard disk, etc.), an optical disc (e.g., a CD-ROM, a DVD, an MO, etc.), a semiconductor memory (e.g., a ROM, a RAM, a flash memory, etc.), and can be transmitted and distributed via a communication medium. Note that the program stored on the medium side includes a setting program for configuring software means (including not only an execution program but also a table and a data structure) to be executed by a computer in the computer. The computer that realizes this apparatus reads the program recorded on the recording medium, and in some cases, constructs software means by the setting program, and executes the above-described processing by being controlled by this software means. Note that the recording medium referred to in this specification includes not only a medium for distribution but also a storage medium such as a magnetic disk or a semiconductor memory provided inside a computer or in a device connected via a network.

[0044] Note that the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the gist thereof at the implementation stage. Also, the embodiments may be combined as appropriate, and in that case, the combined effects can be obtained. Further, the above-described embodiments include various inventions, and various inventions can be extracted by combinations selected from a plurality of disclosed constituent elements. For example, even if some constituent elements are deleted from all the constituent elements shown in the embodiments and the problem can be solved and the effects can be obtained, the configuration from which these constituent elements are deleted can be extracted as an invention.

Description of Reference Numerals

[0045] 100... Depth map generation device 11... Depth estimation unit 12... Depth optimization processing unit 13... Inter-frame difference calculation unit 14... Display surface information determination unit 15… Depth map correction unit

Claims

1. A depth estimation unit that estimates the depth of an image, A depth optimization processing unit that outputs a depth map of the image by applying a depth optimization function used for mapping the depth to the depth estimated by the depth estimation unit, A setting unit that performs histogram analysis on the depth map output by the depth optimization processing unit and sets the depth value on the display screen of the image based on the distribution of the depth values indicated by the result of this analysis, An image processing apparatus comprising the above.

2. The setting unit calculates an index value for each of the depth values indicated by the distribution, and sets the depth value used for calculating the lowest index value as the depth value on the display screen of the image, The image processing apparatus according to claim 1.

3. The setting unit corrects the index value calculated for each of the depth values indicated by the distribution according to the value calculated for the depth value related to the area of the object of attention among the depth values indicated by the distribution, The image processing apparatus according to claim 2.

4. The setting unit corrects the index value calculated for each of the depth values indicated by the distribution according to the value calculated for the depth value to which the depth optimization function by the depth optimization processing unit is applied among the depth values indicated by the distribution, The image processing apparatus according to claim 2.

5. The image is a moving image composed of a plurality of frames, The image further comprises a difference calculation unit that calculates the size of the difference region between one frame along the time series in the image and a frame at a timing after the timing of the one frame, The setting unit Correcting the index values calculated for each of the depth values indicated by the distribution according to the magnitude of the difference calculated by the difference calculation unit. The image processing apparatus according to claim 2.

6. A method performed by an image processing apparatus, A depth estimation unit that estimates the depth of an image, A depth optimization processing unit that outputs a depth map of the image by applying a depth optimization function used for mapping the depth to the depth estimated by the depth estimation unit, Performing a histogram analysis of the depth map output by the depth optimization processing unit, and setting the depth value in the display screen of the image based on the distribution of the depth values indicated by the result of this analysis. An image processing method comprising:

7. The setting unit Calculates an index value for each of the depth values indicated by the distribution, and sets the depth value used for calculating the lowest index value as the depth value in the display screen of the image. The image processing method according to claim 6.

8. An image processing program that causes a processor to function as each of the units of the image processing apparatus according to any one of claims 1 to 5.

Images