DATA PROCESSING APPARATUS, DATA PROCESSING METHOD, AND PROGRAM

Abstract

A data processing device according to an embodiment of the present invention processes a depth map obtained by mapping a depth value for each pixel of an image displayed on a display. This data processing device is provided with a region setting unit, a histogram generation unit, a mapping function generation unit, a first correction processing unit, and a second correction processing unit. The region setting unit sets, in a depth map being processed, a contact region that contacts the display surface. The histogram generation unit generates a histogram obtained by associating, with each of a plurality of classes for dividing the range of depth values, the number of pixels belonging to each class, for a non-contact region that is a region other than the contact region in the depth map. The mapping function generation unit clusters the histogram into a plurality of depth layers and generates a mapping function for converting the depth values in the non-contact region into values based on clustering. The first correction processing unit corrects the depth values in the non-contact region using the mapping function. The second correction processing unit corrects the depth values in the contact region and continuously connects the depth values in the contact region to the corrected depth values in the non-contact region.

Description

[Technical field]

[0001] SUMMARY OF THE DISCLOSURE The present invention relates to a technique for generating depth maps for use in generating three-dimensional (3D) video. [Background technology]

[0002] To generate a 3D image, a depth map is used to represent the depth information of the image. A depth map is data generated by mapping distance information (depth information) from the viewpoint to each pixel of an image. A depth map can be expressed in grayscale. In a typical 8-bit gradation (0-255), the deepest part is represented by the minimum value of 0 (black) and the closest part is represented by the maximum value of 255 (white). Hereinafter, depth information will be referred to as depth value.

[0003] A method has been proposed in which a depth map is used to remap a wide depth range to a notable subject in a frame, thereby emphasizing the three-dimensional effect of the notable subject (see, for example, Non-Patent Document 1). Non-Patent Document 1 discloses a method of correcting the inversion of a mapping function generated when extending the depth value of a notable subject by using a non-linear least squares method.

[0004] Also, a patent application has been filed for a method of identifying an area in a mapping function where pre-decompression depth values ​​and post-decompression depth values ​​have a negative correlation, and correcting the area so that there is a positive correlation only in the identified area. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Sangwoo Lee, Younghui Kim, Jungjin Lee, Kyehyun Kim, Kyunghan Lee and Junyoung Noh, “Depth manipulation using disparity histogram analysis for stereoscopic 3D”, The Visual Computer 30(4):455-465, April2014. Summary of the Invention [Problem to be solved by the invention]

[0006] Generally, there is less surrounding image information at the top, bottom, left, and right edges of a frame. For this reason, if a depth map is created based on depth information estimated from monocular 2D images or binocular stereo images, the accuracy of the depth information estimation may deteriorate. Even if a mapping function based on low-precision depth information is used, it is difficult to effectively express depth because it is affected by the data at the edge of the frame. If a 3D image is generated using inaccurate depth values ​​at the edge of the screen, noise that was not present in the original image may occur. In addition, the sense of incongruity between the edge of the screen and the display surface becomes more noticeable, which may cause eye fatigue for the viewer.

[0007] The present invention has been made in light of the above-mentioned circumstances, and aims to provide a technique for reducing the sense of incongruity between the screen edge and the display surface, and for realizing a more natural expression of depth. [Means for solving the problem]

[0008] A data processing device according to an embodiment of the present invention processes a depth map in which a depth value is mapped for each pixel of an image displayed on a display. The data processing device includes a region setting unit, a histogram generating unit, a mapping function generating unit, a first correction processing unit, and a second correction processing unit. The region setting unit sets an abutment region that abuts a display surface in a depth map to be processed. The histogram generating unit generates a histogram for a non-abutment region, which is a region other than the abutment region of the depth map, in which the number of pixels belonging to each class is associated with each of a plurality of classes that divide a range of depth values. The mapping function generating unit clusters the histogram into a plurality of depth layers, and generates a mapping function that converts the depth value of the non-abutment region into a value based on the clustering. The first correction processing unit corrects the depth value of the non-abutment region by the mapping function. The second correction processing unit corrects the depth value of the abutment region and continuously connects the depth value of the abutment region to the depth value of the corrected non-abutment region. Effect of the Invention

[0009] According to one aspect of the present invention, it is possible to reduce the sense of incongruity between the screen edge and the display surface, and to realize a more natural expression of depth. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of a data processing device according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of an outer frame region of an image. [Diagram 3] FIG. 3 is a diagram for explaining correction of the depth value of the outer frame region. [Figure 4] FIG. 4 is a flowchart showing an example of a processing procedure of the data processing device 1. [Diagram 5] FIG. 5 is a diagram showing a state in which an object labeled in the fixation area protrudes into the outer frame area. [Figure 6] FIG. 6 is a diagram showing an example of a modified central region. [Figure 7]FIG. 7 is a functional block diagram showing an example of a computer capable of implementing the functions of the data processing device 1. As shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a block diagram showing an example of a data processing device according to an embodiment. The data processing device shown in Fig. 1 is a computer, and has a function of correcting and optimizing depth values ​​of a depth map of a three-dimensional image frame, for example. The data processing device of the embodiment can be combined with a display that displays an image, and processes the depth map of the image displayed on the display. The data processing device 1 shown in FIG. 1 includes a region setting unit 11, a histogram generating unit 12, a mapping function generating unit 13, a correction processing unit 14, and an optimization processing unit 15.

[0012] The region setting unit 11 sets a contact region that contacts the display surface in the depth map to be processed. That is, the region setting unit 11 sets regions (indicated by hatching) of a certain width on the top, bottom, left and right of the image as shown in Fig. 2. In the following description, this region is referred to as an outer frame region.

[0013] In Fig. 2, w denotes the width of the outer frame region that abuts against the display surface 2. w can be set arbitrarily according to the size of the image; for example, the left and right outer frame regions can be set to a few % of the horizontal size of the image, and the top and bottom outer frame regions can be set to a few % of the vertical size of the image. Also, it is not essential to provide frames on all four sides; only the horizontal frame, only the vertical frame, or only the left frame can be set arbitrarily according to the content of the image. Information on the set outer frame region and information on the region other than the outer frame region (non-abutting region: hereinafter referred to as the central region) are output to the histogram generating unit 12 and the correction processing unit 14.

[0014] The histogram generating unit 12 generates a histogram for the central region, which is a region other than the outer frame region of the depth map, in which the number of pixels belonging to each class is associated with a plurality of classes that divide the range of depth values. That is, the histogram generating unit 12 generates a histogram for the central region in which the number of pixels belonging to each class is associated with a plurality of classes that divide the depth values. The generated histogram is output to the mapping function generating unit 13 and is referred to when generating the mapping function.

[0015] The mapping function generating unit 13 generates a mapping function that clusters the histogram into multiple depth layers and converts the depth values ​​in the central region into values ​​based on the clustering. That is, the mapping function generating unit 13 clusters the histogram into multiple depth layers having a Gaussian distribution shape. The mapping function generating unit 13 then generates a mapping function that converts the values ​​of the pre-correction depth map into the values ​​of the corrected depth map by expanding the width of the classes belonging to a predetermined depth layer. The histogram and the mapping function can be generated, for example, using the method described in PCT / JP2020 / 028527. The mapping function is also sometimes called a depth compression function.

[0016] The optimization processing unit 15 as a first correction processing unit corrects the depth value of the central region using the above mapping function. That is, the optimization processing unit 15 optimizes the depth value of the central region according to the mapping function generated by the mapping function generating unit 13.

[0017] The correction processing unit 14 as the second correction processing unit corrects the depth value of the outer frame region so as to be continuously connected to the depth value of the central region that has been separately corrected. That is, the correction processing unit 14 corrects the depth value for the region set as the outer frame region by the region setting unit 11, and outputs the corrected depth value to the optimization processing unit 15. Fig. 3 is a diagram for explaining correction of depth values ​​of the outer frame region. Fig. 3 is an enlarged view of the left end of the outer frame region shown in Fig. 2, for example. The hatched portion is the outer frame region set by region setting unit 11, and has a region width of 10 pixels. In the correction process, first, the depth value at the edge of the screen (0th column) is replaced with a depth value that corresponds to the display surface.

[0018] For example, in an 8-bit depth map, the depth value that is the display surface is generally the central value, so the correction value is 128. The depth value is corrected by linearly interpolating between this value and the depth value (200) outside the area (10th column). Linear interpolation is similarly performed for the right, top, and bottom sides of the image, from the edge of the screen toward the center of the image.

[0019] Fig. 4 is a flowchart showing an example of a processing procedure of the data processing device 1. In Fig. 4, the data processing device 1 sets an outer frame region and a central region in a depth map to be processed (step S1). Next, the data processing device 1 generates a histogram for the central region in which the number of pixels belonging to each of a plurality of classes that divide the range of depth values ​​is associated with each class (step S2).

[0020] Next, the data processing device 1 clusters the generated histogram into a plurality of depth layers, and generates a mapping function for converting the depth values ​​of the central region into values ​​based on the clustering (step S3).

[0021] Next, the data processing device 1 corrects the depth value of the central region using a mapping function (step S4). Furthermore, the data processing device 1 corrects the depth value of the outer frame region and continuously connects the depth value of the outer frame region to the corrected depth value of the central region (step S5).

[0022] As described above, the data processing device 1 according to one embodiment of the present invention determines an outer frame region of an image for a depth map to be processed by the region setting unit 11. The histogram generating unit 12 generates a histogram for the central region excluding the outer frame region, in which the number of pixels belonging to each class is associated with each of a plurality of classes that divide the range of depth values. The mapping function generating unit 13 clusters the histogram into a plurality of depth layers, and generates a mapping function that converts pre-optimization depth values ​​in the depth map to post-optimization depth values. The optimization processing unit 15 corrects the depth values ​​of the central region using the mapping function. The correction processing unit 14 corrects the depth values ​​of the outer frame region so as to be continuously connected to the depth values ​​of the central region.

[0023] That is, in the embodiment, a certain area (outer frame area) is set from the screen edge of the image, and a depth map is generated based on a mapping function calculated based on the central area excluding the outer frame area. In this way, the depth value of the depth map before processing can be optimized, which reduces the sense of incongruity of 3D in the outer frame area of ​​the image and realizes highly accurate depth expression for the entire screen. That is, according to the embodiment, it is possible to provide a data processing device, a data processing method, and a program that reduce the sense of incongruity between the screen edge and the display surface and realize a more natural depth expression.

[0024] It should be noted that the present invention is not limited to the above embodiment. For example, the shape of the central region is not limited to a rectangle. For example, when an image includes a gaze object and the object protrudes into the outer frame region, the shape of the central region is deformed to trace the contour of the object. In addition, when the image itself is not rectangular, such as a circle or an ellipse, the shape of the central region excluding the outer frame region may not be rectangular.

[0025] Fig. 5 is a diagram showing a state in which an object labeled in a fixation region protrudes into an outer frame region. As shown in Fig. 5, when an object labeled with a label corresponding to the fixation region, such as a person, a car, or an animal, protrudes into the outer frame region, the region setting unit 11 sets the part of the object protruding into the outer frame region as the central region. As shown in Fig. 6, the central region is expanded by the amount of the protruding object contour. Then, the data processing device 1 generates a histogram for this region (step S2 in Fig. 4), clusters the histogram and generates a mapping function (step S3), and corrects the depth value using the mapping function (step S4). On the other hand, the outer frame region is reduced by only the contour portion of the object, and the depth value of this region is continuously connected to the depth value of the corrected central region (step S5).

[0026] That is, for example, when segmentation information is input from an external processing block, if there is an object in the outer frame area that is labeled as a gaze area, such as a person or a car, the depth value of that area is not corrected. This is to avoid the discomfort of 3D being noticeable by correcting the depth of the gaze object.

[0027] The data processing device 1 may also perform a depth value correction process only when the original image of the target location is flat. That is, the region setting unit 11 sets an outer frame region in a region where the variance value of pixel values ​​is lower than a predetermined threshold value. That is, if the original image is too fine, processing may not be performed. The fineness of an image can be determined, for example, by the maximum / minimum value of pixel values ​​or the variance value.

[0028] In other words, if the luminance variance of the original image in the outer frame region (for example, the 10 pixels from column 0 to column 9 in Figure 3) is equal to or greater than a preset threshold, the depth value is not corrected. This is because areas with fine high-frequency components in the original image tend to become areas of interest, and correcting the depth can actually make the 3D look unnatural.

[0029] The above-described area setting unit 11, histogram generating unit 12, mapping function generating unit 13, correction processing unit 14, and optimization processing unit 15 can be implemented, for example, by a computer including a CPU (Central Processing Unit) 31, a GPU (Graphic Processing Unit) 32, a memory 33, a storage 34, a communication device 35, an input device 36, and an output device 37 as shown in Fig. 7. In this computer, the area setting unit 11, the histogram generating unit 12, the mapping function generating unit 13, the correction processing unit 14, and the optimization processing unit 15 are realized by the CPU 31 and the GPU 32 executing a program loaded into the memory 33. The program can be recorded on a computer-readable recording medium such as an optical disc or a semiconductor memory, or can be distributed via a network.

[0030] It should be noted that the present invention is not limited to the above embodiment. For example, the depth value has been described as 8 bits and 256 gradations, but it is of course not limited to this value. Furthermore, the flow of each process described above is not limited to the procedure described above, and the order of some steps may be changed, or some steps may be performed simultaneously in parallel. Also, the series of processes described above do not need to be performed consecutively in time, and each step may be performed at any timing.

[0031] The program for implementing the above-mentioned processing may be provided by being stored in a computer-readable recording medium (or storage medium). The program is stored in the recording medium as a file in an installable format or an executable format. Examples of recording media include magnetic disks, optical disks (CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), magneto-optical disks (MO, etc.), and semiconductor memories. The program for implementing the above-mentioned processing may also be stored on a computer (server) connected to a network such as the Internet, and downloaded to a computer (client) via the network.

[0032] In addition, the histogram clustering method, the depth layer setting method, the mapping function generation method, and the like can also be modified in various ways without departing from the spirit and scope of the present invention.

[0033] The data processing device according to the embodiment can be constructed by configuring the operation of each component as a program, and installed and executed on a computer used as the data processing device, or distributed via a network. The present invention is not limited to the above-mentioned embodiment, and various modifications and applications are possible.

[0034] In short, this invention is not limited to the above-mentioned embodiment, and can be modified in various ways in the implementation stage without departing from the gist of the invention. In addition, each embodiment may be implemented in appropriate combination, in which case the combined effect can be obtained. Furthermore, the above-mentioned embodiment includes various inventions, and various inventions can be extracted by combinations selected from the multiple constituent elements disclosed. For example, if the problem can be solved and the effect can be obtained even if some constituent elements are deleted from all the constituent elements shown in the embodiment, the configuration from which the constituent elements are deleted can be extracted as an invention. [Explanation of symbols]

[0035] 1...Data processing device 2. Display surface 11...Area setting section 12...Histogram generation section 13...Mapping function generator 14...Correction processing section 15...Optimization processing section 31…CPU 33…Memory 34…Storage 35...Communication equipment 36...Input device 37...Output device.

Claims

1. A data processing device that processes a depth map in which a depth value is mapped for each pixel of an image to be displayed on a display, a region setting unit that sets a contact region in a region of a depth map to be processed that contacts a display surface and has a variance value of pixel values ​​lower than a predetermined threshold; a histogram generating unit that generates a histogram for a non-contact region of the depth map other than the contact region, the histogram including a number of pixels belonging to each of a plurality of classes that divide a range of depth values; a mapping function generating unit that generates a mapping function that converts the depth values ​​of the non-contact regions into values ​​based on the clustering by clustering the histogram into a plurality of depth layers; a first correction processing unit that corrects the depth value of the non-contact region using the mapping function; A data processing device comprising: a second correction processing unit that replaces the depth value of the screen edge with the depth value of the display surface, corrects the depth value of the abutment area, and continuously connects the depth value of the abutment area to the depth value of the corrected non-abutment area.

2. The data processing device according to claim 1 , wherein the region setting section sets a frame-shaped region having a width corresponding to a size of the image as the contact region.

3. A data processing device as described in claim 1, wherein, when an object assigned a label corresponding to a gaze area extends from the non-contact area into the contact area, the area setting unit also sets the portion of the object that extends into the contact area as the non-contact area.

4. A data processing method for processing, by a computer, a depth map in which a depth value is mapped for each pixel of an image displayed on a display, comprising: The computer sets a contact area in a depth map to be processed in a region that contacts a display surface and has a variance value of pixel values ​​lower than a predetermined threshold; The computer generates a histogram for a non-contact region of the depth map other than the contact region, in which the number of pixels belonging to each of a plurality of classes that divide a range of depth values ​​is associated with each of the classes; the computer clustering the histograms into a plurality of depth layers and generating a mapping function that converts depth values ​​of the non-contact regions to values ​​based on the clustering; the computer corrects the depth value of the non-contact region using the mapping function; A data processing method comprising the computer replacing depth values ​​of the screen edges with depth values ​​of the display surface, correcting the depth values ​​of the abutment area, and continuously connecting the depth values ​​of the abutment area to the corrected depth values ​​of the non-abutment area.

5. A program for causing a computer to function as the area setting unit, the histogram generation unit, the mapping function generation unit, the first correction processing unit, and the second correction processing unit of a data processing device described in any one of claims 1 to 3.

Images