Image processing apparatus, image processing method, and computer program

Abstract

To provide an image processing apparatus, method and program for reducing unnecessary unevenness while generating unevenness of a desired part when generating a three dimensional image.SOLUTION: According to an aspect of the present invention, there is provided an image processing device including an image processing unit configured to generate 3D information of an object based on a plurality of pieces of depth information corresponding to a plurality of regions of an image, and a smoothing unit configured to perform smoothing processing on the plurality of pieces of depth information corresponding to the plurality of regions of the image to generate 3D information of the object based on the plurality of pieces of depth information and the image, wherein the smoothing processing includes: 3D.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, a computer program, and the like. [Background technology]

[0002] For example, Patent Document 1 describes an image processing device that can generate a three-dimensional image by simultaneously acquiring distance information when capturing a single still image and processing the image based on the distance information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2024-8596 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in such technology, a smoothing (median) filter is generally applied to remove noise from the distance data. However, if the number of filter taps is increased, the unevenness of organs (eyes, nose, mouth), accessories, and hair cannot be reproduced. Conversely, if the number of filter taps is reduced, noise tends to remain in skin with low texture, which is an issue.

[0005] An object of the present invention is to provide an image processing device that can reduce unnecessary unevenness while producing unevenness in desired areas when generating a three-dimensional image. [Means for solving the problem]

[0006] An image processing device according to an embodiment of the present invention includes: an area division means for classifying and dividing an image into a plurality of areas; a filter processing means for performing a filter process with different characteristics according to the classification on each of the distance information corresponding to the plurality of regions of the image; a synthesis means for synthesizing the distance information that has been subjected to the filtering process; a 3D data generation processing means for generating 3D data of the subject based on the distance information and the image synthesized by the synthesis means; 1. An image processing device comprising: [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an image processing device and the like that can reduce unnecessary unevenness while producing unevenness in desired areas when generating a three-dimensional image. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a functional block diagram showing an example of the configuration of an imaging device 100 according to a first embodiment of the present invention. [Figure 2] 1A and 1B are diagrams illustrating a detailed configuration of the image sensor 2 included in the image capturing device 100 according to the first embodiment of the present invention. [Figure 3] (A) is a schematic diagram showing the exit pupil 304 of the optical system 1 and the light beam received by the first photoelectric conversion unit 215 of a pixel in the image sensor 2, and (B) is a schematic diagram similarly showing the light beam received by the second photoelectric conversion unit 216. [Figure 4] 1A to 1C are schematic diagrams showing the relationship between an image sensor 2 and an optical system 1 of an image pickup device 100 according to a first embodiment of the present invention. [Figure 5] 5 is a flowchart showing an example of a 3D data generation process performed by the image processing unit 3 of the imaging device 100 according to the first embodiment of the present invention. [Figure 6] (A) is a diagram showing an example of an image obtained in step S20, (B) is a diagram showing an example of a distance map, (C) is a diagram showing an example of a mesh image generated based on point cloud data, and (D) is a diagram showing an example of a texture image generated based on the mesh image of Figure 4(C). [Figure 7]10 is a flowchart illustrating an example of a smoothing process in step S505. [Figure 8] 8A is a flowchart showing an example of the smoothing process in step S505 according to the first embodiment of the present invention, and FIG. 8B is a diagram for explaining an example of the type of region in step S801 in FIG. 8A. FIG. 8C is a diagram for explaining an example of changing the smoothing filter size in step S802 in FIG. [Figure 9] 10 is a flowchart showing an example of the smoothing process in step S505 according to the second embodiment of the present invention. [Figure 10] 11 is a flowchart showing an example of the smoothing process in step S505 according to the third embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each drawing, the same members or elements are designated by the same reference numerals, and duplicate descriptions will be omitted or simplified.

[0010] <Embodiment 1> Fig. 1 is a diagram showing an example of the configuration of an image capture device 100 according to a first embodiment of the present invention. Note that some of the functional blocks shown in Fig. 1 are realized by causing a CPU or the like serving as a computer (not shown) included in the image capture device 100 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0011] However, some or all of these functions may be implemented by hardware. Examples of such hardware include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs). Furthermore, the functional blocks shown in Figure 1 do not have to be housed in the same housing, and may be configured as separate devices connected to each other via signal paths.

[0012] The imaging device 100 is applicable to digital still cameras, digital video cameras, in-vehicle cameras, surveillance cameras, smartphones, etc. The imaging device 100 includes an optical system 1, an imaging element 2, an image processing unit 3, a compression / decompression unit 4, a control unit 5, an operation unit 6, an image display unit 7, and an image recording unit 8. The imaging device 100 in this embodiment functions as an image processing device.

[0013] The optical system 1 includes a lens, a lens driving mechanism, a mechanical shutter mechanism, an aperture mechanism, etc. Among these, the movable parts are driven based on control signals from the control unit 5.

[0014] The imaging element 2 is, for example, an XY address type CMOS (Complementary Metal Oxide Semiconductor) image sensor, and performs imaging operations in response to control signals from the control unit 5. Furthermore, the imaging signal is digitized by an AD conversion circuit included in the imaging element 2 and output to the image processing unit 3 as an image signal.

[0015] In the image sensor 2 of this embodiment, for example, a first photoelectric conversion unit and a second photoelectric conversion unit are arranged side by side in each pixel. A common microlens is disposed on the light incident surface of the first photoelectric conversion unit and the second photoelectric conversion unit. As a result, light from different exit pupils of the photographing lens included in the optical system 1 is incident on the first photoelectric conversion unit and the second photoelectric conversion unit, respectively.

[0016] Therefore, there is parallax between the first image signal obtained from the group of first photoelectric conversion units of the plurality of pixels and the second image signal obtained from the group of second photoelectric conversion units of the plurality of pixels. Note that the image sensor 2 can read out a signal obtained by adding together the signals of the first photoelectric conversion unit and the second photoelectric conversion unit for each pixel as image data for display.

[0017] Alternatively, the image sensor 2 may be configured to separately output the first image signal and the second image signal, or to separately read out the image data added for each pixel and the first image signal, thereby allowing the downstream image processor 3 to calculate the second image signal by subtracting the first image signal from the added image data.

[0018] The image processing unit 3 generates a distance image (distance map) by calculating distance information to the subject based on the correlation distance (phase difference) between the first image signal and the second image signal obtained from the image sensor 2. Furthermore, as described below, a stereoscopic image (3D data) is generated based on the image signal and the distance image (distance map). A detailed configuration example of the image sensor 2 and a method for calculating distance information will be described later.

[0019] Under the control of the control unit 5, the image processing unit 3 also performs image processing such as noise correction and white balance processing on the digitized image signal input from the image sensor 2. The image processing unit 3 also generates a control signal for controlling the focus lens of the optical system 1 based on the distance information described above, and generates control signals for controlling the accumulation time and aperture of the image sensor based on the luminance information of the image signal.

[0020] The image signals and control information that have been subjected to image processing in the image processing unit 3 are output to the control unit 5. Note that at least a part of the image processing for generating a stereoscopic image may be performed in an external image processing device separate from the imaging device 100.

[0021] The compression / decompression unit 4 operates under the control of the control unit 5, and performs compression / encoding processing of image signals and decompression / decoding processing of encoded data of still images. It may also perform compression / encoding / decompression / decoding processing of moving images.

[0022] The control unit 5 is, for example, a microcontroller including a central processing unit (CPU), a read-only memory (ROM), a random access memory (RAM), and the like.

[0023] The CPU serving as a computer of the control unit 5 executes a computer program stored in a storage medium such as a ROM to comprehensively control each unit of the entire imaging device 100. The operation unit 6 is composed of various operation members such as a shutter release button, and outputs a control signal to the control unit 5 in response to an input operation by the user.

[0024] Examples of input operations by the user include setting the recording mode for still images, video, etc., and exposure control (aperture, accumulation time of the image sensor, ISO sensitivity).

[0025] The image display unit 7 supplies an image signal to a display device such as an LCD (Liquid Crystal Display) to display an image. The image recording unit 8 is connected to, for example, a portable recording medium and stores compressed and encoded image data files.

[0026] Note that a distance image (distance map) may be further linked to the image data file and recorded in the image recording unit 8. Alternatively, the first image signal and the second image signal may be recorded as an image data file in the image recording unit 8. Alternatively, the image recording unit 8 may record the display image data and the first image signal that are added for each pixel, so that the second image signal can be calculated later.

[0027] By doing as described above, it is possible to generate a three-dimensional image by reading the image data file, distance image (distance map), etc. from the image recording unit 8 at any timing after shooting. Note that the imaging device 100 may have a communication unit, and for example, it can transmit the image data and distance image (distance map), etc. recorded in the image recording unit 8 to an external image processing device. Therefore, 3D data (three-dimensional image) can be generated in the external image processing device.

[0028] Here, the optical system 1 is a photographing lens included in the imaging device 100, and forms an optical image of a subject on the imaging surface of the imaging element 2. The optical system 1 is made up of multiple lenses (not shown) arranged on an optical axis 303, and has an exit pupil 304 at a position a predetermined distance away from the imaging element 2.

[0029] In this specification, the direction parallel to the optical axis 303 is defined as the z direction or depth direction, the direction perpendicular to the optical axis 303 and parallel to the horizontal scanning direction of the image signal of the imaging element 2 is defined as the x direction, and the direction parallel to the vertical scanning direction of the image signal is defined as the y direction, or axes are set.

[0030] In this embodiment, the image sensor 2 is configured to be able to acquire a group of images used for distance measurement using an image plane phase difference distance measurement method.

[0031] 2A and 2B are diagrams illustrating a detailed configuration of the image sensor 2 included in the image capturing device 100 according to the first embodiment of the present invention. As shown in Fig. 2A, the image sensor 2 is configured by connecting and arranging a plurality of pixel groups 201 in 2 rows and 2 columns, to which different color filters are applied.

[0032] As shown in the enlarged view, a pixel group 201 consisting of four pixels has red (R), green (G), and blue (B) color filters arranged therein, and each pixel outputs an image signal indicating color information of R, G, or B. Note that, as an example, in this embodiment, the color filters are described as having a Bayer array as shown in the drawing, but the color filter array is not limited to this.

[0033] Fig. 2(B) is a diagram showing an example of a cross section taken along line II' in Fig. 2(A). In order to realize a ranging function using an imaging surface phase difference ranging method, the image sensor 2 of this embodiment has a single pixel in which multiple (for example, two) photoelectric conversion units are arranged side by side in the horizontal scanning direction (x direction) of the image sensor 2, as shown in Fig. 2(B).

[0034] That is, as shown in Figure 2(B), each pixel of the imaging element 2 is composed of a light-guiding layer 213 including a microlens 211 and a color filter 212, and a light-receiving layer 214 including a first photoelectric conversion section 215 and a second photoelectric conversion section 216.

[0035] In the light guide layer 213, the microlenses 211 are configured to efficiently guide the light beams incident on the pixels to the first photoelectric conversion section 215 and the second photoelectric conversion section 216. The color filters 212 transmit only light in any one of the wavelength bands of R, G, and B described above, and guide the light to the first photoelectric conversion section 215 and the second photoelectric conversion section 216.

[0036] The light receiving layer 214 is provided with a first photoelectric conversion unit 215 and a second photoelectric conversion unit 216 that convert the received light into an analog image signal, and the two types of signals output from these two photoelectric conversion units are used for distance measurement.

[0037] That is, each pixel of the image sensor 2 has two photoelectric conversion units similarly arranged in the horizontal scanning direction, and a first image signal made up of signals output from the first group of photoelectric conversion units 215 of all pixels and a second image signal made up of signals output from the second group of photoelectric conversion units 216 are used.

[0038] The first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 each partially receive the light beam that enters the pixel through the microlens 211, and therefore each photoelectric conversion unit receives the light beam that has passed through a different pupil region of the exit pupil of the optical system 1.

[0039] That is, the image sensor 2 is capable of capturing two image signals having parallax that have passed through different pupil regions of the optical system 1. Here, the photoelectric conversion signals from the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 at each pixel are combined for each pixel and can be used as an image signal for display.

[0040] In addition, the imaging element 2 of this embodiment is capable of separately outputting an image signal for display (an image signal obtained by adding together the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 for each pixel) and an image signal for ranging (at least one of the first image signal and the second image signal).

[0041] In this embodiment, an example will be described in which all pixels of the image sensor 2 are provided with two photoelectric conversion units and configured to be able to output high-density depth information, but this is not limiting. For example, the number of photoelectric conversion units provided in each pixel may be three or more, and only some of the pixels of the image sensor 2 may have multiple photoelectric conversion units.

[0042] Next, the principle of measuring the subject distance based on the first image signal and the second image signal output from the group of first photoelectric conversion units 215 and the group of second photoelectric conversion units 216, respectively, will be explained with reference to Figures 3 and 4.

[0043] Fig. 3(A) is a schematic diagram showing the exit pupil 304 of the optical system 1 and the light beam received by the first photoelectric conversion unit 215 of a pixel in the image sensor 2. Fig. 3(B) is a similar schematic diagram showing the light beam received by the second photoelectric conversion unit 216.

[0044] 3(A) and 3(B) is arranged so that the exit pupil 304 and the light receiving layer 214 are optically conjugate with each other. A light beam that has passed through the exit pupil 304 of the optical system 1 is collected by the microlens 211 and guided to the first photoelectric conversion unit 215 or the second photoelectric conversion unit 216.

[0045] 3A and 3B, the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 mainly receive light beams that have passed through different pupil regions. That is, the first photoelectric conversion unit 215 receives a light beam that has passed through a first pupil region 301 of the exit pupil 304, and the second photoelectric conversion unit 216 receives a light beam that has passed through a second pupil region 302 of the exit pupil 304.

[0046] The plurality of first photoelectric conversion units 215 included in the image sensor 2 mainly receive the light beam that has passed through the first pupil region 301 and output a first image signal. At the same time, the plurality of second photoelectric conversion units 216 included in the image sensor 2 mainly receive the light beam that has passed through the second pupil region 302 and output a second image signal.

[0047] From the first image signal, it is possible to obtain the intensity distribution of the image formed on the image sensor 2 by the light beam that has passed through the first pupil region 301. Also, from the second image signal, it is possible to obtain the intensity distribution of the image formed on the image sensor 2 by the light beam that has passed through the second pupil region 302.

[0048] The relative positional deviation amount (so-called phase difference or parallax amount) between the first image signal and the second image signal is a value according to the defocus amount. The relationship between the parallax amount and the defocus amount will be described with reference to FIGS. 4(A) to 4(C).

[0049] 4A to 4C are schematic diagrams showing the relationship between the image sensor 2 and the optical system 1 according to the first embodiment. In the drawings, reference numeral 401 denotes a first light flux passing through the first pupil region 301, and reference numeral 402 denotes a second light flux passing through the second pupil region 302.

[0050] 4A shows a state during focusing, in which a first light beam 401 and a second light beam 402 converge on the image sensor 2. At this time, the amount of parallax between the first image signal formed by the first light beam 401 and the second image signal formed by the second light beam 402 is zero.

[0051] 4B shows a state where the image side is defocused in the negative direction of the z axis, in which the amount of parallax between the first image signal formed by the first light beam 401 and the second image signal formed by the second light beam 402 has a negative value.

[0052] 4C shows a state where the image side is defocused in the positive direction of the z axis. In this state, the parallax amount between the first image signal formed by the first light beam 401 and the second image signal formed by the second light beam 402 has a positive value.

[0053] A comparison of Figures 4(B) and 4(C) reveals that the direction of the positional shift reverses depending on whether the defocus amount is positive or negative. It also reveals that the positional shift occurs according to the imaging relationship (geometric relationship) of the optical system 1, depending on the defocus amount. The amount of parallax, which is the positional shift between the first image signal and the second image signal, can be detected by region-based matching processing.

[0054] 5 is a flowchart showing an example of 3D data generation processing by the image processing unit 3 of the imaging device 100 according to the first embodiment of the present invention. Note that the operations of the steps in the flowchart of FIG. 5 and other flowcharts in the following description are sequentially performed by a CPU or the like serving as a computer in the control unit 5 executing a computer program stored in memory.

[0055] The processing flow of FIG. 5 starts when an instruction for generating a stereoscopic image is input via the operation unit 6, for example.

[0056] In step S501, an image signal is acquired from the image sensor 2 or the image recording unit 8. Here, the image signal refers to, for example, an image signal for display and at least one of a first image signal and a second image signal.

[0057] That is, it is assumed here that the imaging element is configured to read out an image signal for display and one of the first and second image signals, and that the image recording unit 8 records the image signal for display and one of the first and second image signals.

[0058] Then, in step S501, the second image signal is obtained by subtracting, for example, the first image signal from the image signal for display. In this way, in step S501, the first image signal and the second image signal are finally obtained.

[0059] The first and second image signals may be separately read out from the imaging element or from the image recording unit, thereby obtaining the first and second image signals in step S501.

[0060] Next, in step S502, the image processing unit 3 calculates the amount of parallax between the first image signal and the second image signal based on these images. Specifically, the image processing unit 3 sets, for example, within the first image signal, a point of interest corresponding to the representative pixel information and a matching region centered on the point of interest.

[0061] The matching area may be a rectangular area, such as a square area with a predetermined side length and centered on the point of interest. Next, the image processing unit 3 sets a reference point in the second image signal and sets a reference area centered on the reference point.

[0062] The reference area has the same size and shape as the above-mentioned matching area. The image processing unit 3 sequentially moves the reference point to derive the correlation between the image included in the matching area of ​​the first image signal and the image included in the reference area of ​​the second image signal, and identifies the reference point with the highest correlation as the corresponding point corresponding to the point of interest in the second image signal. The relative positional deviation between the corresponding point identified in this way and the point of interest is the parallax amount at the point of interest.

[0063] In step S502, the image processing unit 3 calculates the amount of parallax while sequentially changing the attention point in accordance with the representative pixel information in this way, thereby deriving the amount of parallax at a plurality of pixel positions determined by the representative pixel information.

[0064] Next, in step S503, the defocus amount is calculated based on the parallax amount calculated in step S502. That is, the parallax amount is converted into the defocus amount, which is the distance from the image sensor 2 to the focal point of the optical system 1, using a predetermined conversion coefficient.

[0065] That is, when a predetermined conversion coefficient K and a defocus amount ΔL are used, the parallax amount is ΔL=K×d can be converted into the defocus amount by

[0066] Furthermore, in step S504, a distance map is calculated. That is, the defocus amount ΔL is calculated using the lens formula in geometrical optics: 1 / A+1 / B=1 / F is used to convert it into the subject distance for each pixel.

[0067] Here, A is the distance from the object plane of the object to the principal point of optical system 1 (object distance), B is the distance from the principal point of optical system 1 to the image plane, and F is the focal length of optical system 1. That is, in the above lens formula, the value of B can be derived from the defocus amount ΔL, so the object distance A from optical system 1 to the object plane can be derived based on the focal length setting at the time of image capture.

[0068] In this way, the image processing unit 3 generates two-dimensional information (distance map) in which the subject distance derived in step S504 is used as a pixel value, and stores the information in the image recording unit 8 or memory in the control unit 5. Here, steps S501 to S504 function as an information acquisition step (information acquisition means) for acquiring the image and distance information (distance map).

[0069] The information acquiring step (information acquiring means) may acquire an image and distance information (distance map) from an imaging element, or may acquire an image and distance map from an image recording means.

[0070] Thereafter, in step S505, a smoothing process is performed to reduce noise in the distance map. The smoothing process in step S505 will be described in detail later. Next, in steps S506 to S508, a 3D data generation process is performed.

[0071] That is, a process for generating a three-dimensional image is performed based on image data and a distance image (distance map), etc. Note that 3D data in this embodiment means a three-dimensional image from the front up to a predetermined rotation angle range.

[0072] Fig. 6(A) is a diagram showing an example of an image obtained in step S501, and Fig. 6(B) is a diagram showing an example of a distance map obtained in step S504. In Fig. 6(B), higher density indicates greater distance.

[0073] In step S506 of Fig. 5, point cloud conversion is performed based on the distance map data to obtain point cloud data. Further, in step S507, a mesh image is generated based on the point cloud data. Fig. 6(C) is a diagram showing an example of a mesh image generated based on the point cloud data.

[0074] In step S508, a texture image is generated by associating the position of the image acquired in step S501 with the position of the mesh image generated in step S507, thereby generating 3D data.

[0075] Here, steps S506 to S508 function as a 3D data generation processing step (3D data generation processing) that generates 3D data of the subject based on the distance map and the image.

[0076] Fig. 6(D) is a diagram showing an example of a texture image generated based on the mesh image of Fig. 6(C). This texture image is output as 3D data. When the processing of step S508 is completed, the processing flow of Fig. 5 ends, and the 3D data generated in step S508 is recorded, for example, in the image recording unit 8.

[0077] 7 is a flowchart illustrating an example of the smoothing process in step S505. In step S701, a median filter process with, for example, N1 taps is performed, thereby reducing errors in the distance map calculated in step S504.

[0078] Next, in step S702, an interpolation process for low-reliability distances is performed, that is, in the distance map in which noise has been reduced in step S701, low-reliability distance data is interpolated using distance data around it.

[0079] Then, after step S703, median processing is performed again, for example, with the number of taps being N1. This is processing to reduce the difference between the area of ​​low-reliability distance data and the other areas due to the interpolation processing in step S702.

[0080] In this way, by performing the processes of steps S701 to S703, noise in the distance data in the distance map can be reduced, low-reliability areas can be interpolated with surrounding distance data, and the resulting step can be reduced.

[0081] 7, if the median filter is strengthened in step S701 or step S703 (for example, by increasing the number of taps to N3 (N3>N1)), the unevenness of organs, accessories, and hair may not be reproduced. On the other hand, if the filter is weakened (for example, by reducing the number of taps), noise is more likely to remain in skin with low texture, and the skin around the bangs or glasses may stand out.

[0082] Fig. 8(A) is a flowchart showing an example of the smoothing process in step S505 according to the first embodiment of the present invention, (B) is a diagram for explaining an example of the type of region in step S801 in Fig. 8(A), and (C) is a diagram for explaining an example of changing the smoothing filter size in step S802 in Fig. 8(A).

[0083] In the example of Fig. 8(A), the filter processing means switches the number of filter taps to perform filter processing with different characteristics for each piece of distance information (area of ​​the distance map) corresponding to the classified image area. In the example of Fig. 8(A), an example of processing using a smoothing filter (median filter) as the filter processing will be described, but processing using a filter with other frequency characteristics, such as a band-pass filter, may also be used.

[0084] In step S801 of Fig. 8A, a region segmentation process is performed to divide the image of the subject into a plurality of regions. Here, each region is semantically segmented based on a model that has been machine-learned in advance.

[0085] That is, in this embodiment, the region dividing means divides the image into a plurality of regions by semantic segmentation. However, the method is not limited to semantic segmentation, and for example, regions of skin or hair may be divided according to the area and color of the face region. That is, for example, a region whose hue is close to the color that occupies a large area in the face region may be classified as skin.

[0086] In this embodiment, as shown in Fig. 8(B), each region is divided into organs such as eyes, nose, and mouth, and other parts such as hair, accessories, etc. For example, the image may be divided into organs and parts other than the organs.

[0087] In this embodiment, organs include, for example, eyes, nose, or mouth. Other than organs, they include skin, hair, or accessories. Accessories include, for example, glasses or earrings. Step S801 functions as a region division step (region division means) that classifies and divides the image into multiple regions.

[0088] Next, in step S802, a smoothing process is performed on the distance map using a smoothing filter. At this time, the size (number of taps) of the smoothing filter for the corresponding distance map region is changed for each region of the image as described above. That is, in step 802, the size (number of taps) of the smoothing filter for the corresponding distance map region is set to be different between, for example, an organ and a non-organ.

[0089] Specifically, for example, the size (number of taps) of the smoothing filter in the area of ​​the distance map corresponding to an organ is made smaller than the size (number of taps) of the smoothing filter in the area of ​​the distance map corresponding to a non-organ.

[0090] Next, in step S803, the distance information (divided regions of the distance map) that have been subjected to different smoothing filter processes are synthesized. Here, step S803 functions as a synthesis step (synthesis means) that synthesizes the distance information (distance map) that have been subjected to filter processes with different characteristics.

[0091] In step S802 of this embodiment, the size (number of taps) of the smoothing filter is changed between the area of ​​the distance map corresponding to an organ and the area of ​​the distance map corresponding to a non-organ. However, different sizes (number of taps) of the smoothing filter may be set for the area of ​​the distance map corresponding to each organ, such as the eyes, nose, or mouth, or for each of the hair, glasses, etc.

[0092] In this embodiment, the image is divided into multiple regions based on the results of image recognition, and the size (number of taps, etc.) of the smoothing filter in the distance information (distance map region) corresponding to each image region is changed for each region.

[0093] After the synthesis process in step S803, the process proceeds to step S506. The subsequent steps S506 to S508 function as a 3D data generation processing step (3D data generation processing means) for generating 3D data of the subject based on the distance information (distance map) synthesized in step S803 and the image.

[0094] As described above, in this embodiment, when generating 3D data based on an image and a distance map, it is possible to enhance the unevenness of the 3D data of a desired subject area and reduce the unevenness of the 3D data of an unnecessary subject area.

[0095] In addition, step S802 functions as a filter processing step (filter processing means) that performs filter processing with different characteristics according to classification on each distance information (area of the distance map) corresponding to a plurality of areas of the image.

[0096] In addition, the user may be able to arbitrarily set the size (number of taps, etc.) of the smoothing filter for each semantically segmented area. Alternatively, the size (number of taps, etc.) of the smoothing filter for each semantically segmented area may be automatically set based on machine learning.

[0097] <Embodiment 2> FIG. 9 is a flowchart showing an example of the smoothing process of step S505 according to Embodiment 2 of the present invention. In Embodiment 2, the order of processing in the smoothing process of step S505 is different from that in Embodiment 1.

[0098] As shown in FIG. 9, in step S901, noise in the distance map is reduced by performing median filter processing with the number of taps N2 (N2 < N1). Next, in step S902, low-confidence distance data is interpolated with surrounding distance data. Then, in step S903, an area division process for dividing the image of the subject into a plurality of areas is performed.

[0099] Also in Embodiment 2, each region is semantically segmented (semantic segmentation) based on a pre-machine learned model. In Embodiment 2, additionally, each region is divided into, for example, organs such as eyes, nose, and mouth, and non-organs (e.g., skin, hair, glasses, etc.).

[0100] Then, in step S904, for example, for organs such as eyes, nose, and mouth, a weak median filter process with a tap number of N2 (N2 < N1) is executed. Also, in step S905, for example, for non-organs (e.g., skin, hair, glasses, etc.), a strong median filter process with a tap number of N3 (N3 > N1) is executed.

[0101] Then, the organ region weakly smoothed in step S904 and the non-organ region strongly smoothed in step S905 are combined in step S906. After that, the smoothing process in FIG. 9 is completed, and the process proceeds to the point cloud generation process in step S506.

[0102] <Embodiment 3> FIG. 10 is a flowchart showing an example of the smoothing process in step S505 according to Embodiment 3 of the present invention. In this embodiment, different median filter processes are performed for each of the organs, non-organs, hair, and glasses.

[0103] In step S1001, distance data with low reliability is interpolated with surrounding distance data, and in step S1002, a region division process for dividing the image of the subject into a plurality of regions is performed. Also in Embodiment 3, each region is semantically segmented (semantic segmentation) based on a pre-machine learned model.

[0104] In Embodiment 3, additionally, each region is divided into, for example, organs such as eyes, nose, and mouth, non-organs (e.g., skin), hair, and glasses. Then, in step S1003, a weak median filter process is performed on the organs, and in step S1004, a good median filter process is performed on non-organs (e.g., skin).

[0105] In step S1005, the hair is subjected to a median filter process having different characteristics from the other median filters, for example, a relatively weak median filter, and in step S1006, the glasses are subjected to a median filter process having different characteristics from the other median filters, for example, a relatively strong median filter.

[0106] For example, in the third embodiment, the number of taps of the median filter may be skin>hair>organs>glasses. In this way, it is desirable to make the size of the filter processing for organs smaller than the size of the filter processing for skin, for example.

[0107] In step S1007, the processing results of steps S1003 to S1006 are replaced and combined. That is, the processing results of steps S1003 and S1004 are combined, and the combined result is replaced, for example, with the processing results of steps S1005 and S1006. Thereafter, the processing flow in FIG. 10 ends, and the process proceeds to step S506 in FIG. 5, where processing for 3D conversion is performed.

[0108] The compositing process in step S1007 may be either additive or substitution. In this way, in the third embodiment, median filter processing with different characteristics is performed on hair, glasses, etc., so that smoothing of each region in the 3D data can be further optimized.

[0109] In the above embodiment, an example of a distance map generated using a phase difference detection CMOS image sensor has been described, but the present invention is not limited to this. The distance map may be generated using, for example, a stereo camera or a TOF (Time Of Flight) sensor. Furthermore, machine learning (deep learning) or the like may be used to generate the distance map.

[0110] Furthermore, in the above embodiment, an example has been described in which a median filter is used for smoothing, but the filter for smoothing does not have to be a median filter, and may be a filter having frequency characteristics such as a low-pass filter or a band-pass filter.

[0111] Although the example of changing the characteristics of the smoothing filter by changing the number of taps has been described, the present invention is not limited to changing the number of taps. For example, desired smoothing characteristics may be achieved by switching between multiple filters having different frequency characteristics or by changing the way they are combined.

[0112] Furthermore, as a smoothing filter, for example, the frame of an accessory (such as glasses) has a small number of distance measurement points, so instead of smoothing, linear interpolation or substitution using representative values ​​may be performed. Furthermore, distance measurement values ​​in the region of the division boundary have a large distance error (when they extend beyond the window, etc.), so they may not be used in the filter processing or their weight may be reduced.

[0113] Furthermore, since the shapes of noses, eyeglasses, and the like are somewhat fixed, filtering such as smoothing may be performed based on predetermined average model information (distance information). Furthermore, whether or not filtering such as smoothing according to the first or second embodiment is performed may be controlled according to the shooting distance or the size of the face. That is, for example, if the shooting distance to the subject is equal to or less than a predetermined value or if the face size is equal to or greater than a predetermined value, filtering with different characteristics according to the classification may not be performed.

[0114] Furthermore, the filter characteristics (number of taps, etc.) may be changed for the same organ depending on the direction of the face, because, for example, the size and shape of the left and right eyes may differ when viewed from an oblique angle.

[0115] The present invention has been described above in detail based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible based on the spirit of the present invention, and these are not excluded from the scope of the present invention.

[0116] The present invention also includes those that realize the functions of the above embodiments using, for example, at least one processor such as a CPU, memory, or circuit (for example, ASIC). Also, multiple processors may be used to perform distributed processing.

[0117] In order to realize part or all of the control in the above-described embodiments, a computer program that realizes the functions of the above-described embodiments may be supplied to an image processing device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the image processing device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. The present invention also includes the following combinations.

[0118] (Configuration 1) An image processing device characterized by having an area division means for classifying and dividing an image into a plurality of areas, a filter processing means for performing filter processing with different characteristics according to the classification on each distance information corresponding to the plurality of areas of the image, a synthesis means for synthesizing the distance information after the filter processing, and a 3D data generation processing means for generating 3D data of a subject based on the distance information synthesized by the synthesis means and the image.

[0119] (Configuration 2) The image processing device according to Configuration 1, further comprising information acquisition means for acquiring the image and the distance information.

[0120] (Configuration 3) The image processing device according to Configuration 2, wherein the information acquisition means acquires the image and the distance information from an imaging element.

[0121] (Configuration 4) The image processing device according to configuration 2 or 3, wherein the information acquisition means acquires the image and the distance information from an image recording means.

[0122] (Configuration 5) The image processing device according to any one of configurations 1 to 4, wherein the filter processing means performs the filter processing with different characteristics on each piece of distance information by switching the number of taps of the filter.

[0123] (Configuration 6) The image processing device according to any one of configurations 1 to 5, wherein the filter processing is processing using a smoothing filter.

[0124] (Configuration 7) The image processing device according to configuration 6, wherein the filtering process is a process using a median filter.

[0125] (Configuration 8) The image processing device according to any one of configurations 1 to 7, wherein the region dividing means divides the image by classifying it into organs and non-organs.

[0126] (Configuration 9) The image processing device according to configuration 8, wherein the organs include any one of the eyes, nose, and mouth.

[0127] (Configuration 10) The image processing device according to either of configurations 8 or 9, characterized in that the object other than the organs includes any one of skin, hair, and accessories.

[0128] (Configuration 11) The image processing device according to configuration 10, wherein the accessory includes glasses or earrings.

[0129] (Configuration 12) The image processing device according to any one of configurations 1 to 11, wherein the region dividing means divides the image into a plurality of regions by semantic segmentation.

[0130] (Configuration 13) The image processing device according to any one of configurations 1 to 12, wherein the region dividing means divides the skin or hair region according to the area and color of the face region.

[0131] (Configuration 14) The image processing device according to any one of configurations 1 to 13, wherein the filtering means makes the size of the filtering for the organs smaller than the size of the filtering for the skin.

[0132] (Configuration 15) The image processing device according to any one of configurations 1 to 14, wherein the filter processing means performs linear interpolation or substitution with a representative value for an accessory.

[0133] (Configuration 16) The image processing device according to any one of configurations 1 to 15, wherein the filtering means does not use or reduces the weight of distance measurement values ​​in the region of the division boundary in the filtering process.

[0134] (Configuration 17) The image processing device according to any one of configurations 1 to 16, wherein the filtering means performs filtering on the nose or the eyeglasses based on predetermined model information.

[0135] (Configuration 18) The image processing device according to any one of configurations 1 to 17, wherein the filter processing means controls whether or not to perform the filter processing depending on the shooting distance or the size of the face.

[0136] (Configuration 19) The image processing device according to any one of configurations 1 to 18, wherein the filter processing means changes the filter characteristics even for the same organ depending on the direction of the face.

[0137] (Method) An image processing method characterized by having an area division step of classifying and dividing an image into a plurality of areas; a filtering processing step of performing filtering processing with different characteristics according to the classification on each distance information corresponding to the plurality of areas of the image; a synthesis step of synthesizing the distance information after the filtering processing; and a 3D data generation processing step of generating 3D data of a subject based on the distance information synthesized by the synthesis step and the image.

[0138] (Program) A computer program for controlling each means of the image processing device according to any one of configurations 1 to 19 by a computer. [Explanation of symbols]

[0139] 100: Imaging device 1:Optical system 2: Image sensor 3: Image processing section 5: Control unit 6:Operation unit 7: Image display section 8: Image recording unit

Claims

1. an area division means for classifying and dividing an image into a plurality of areas; a filter processing means for performing a filter process with different characteristics according to the classification on each of the distance information corresponding to the plurality of regions of the image; a synthesis means for synthesizing the distance information that has been subjected to the filtering process; a 3D data generation processing means for generating 3D data of the subject based on the distance information and the image synthesized by the synthesis means; 1. An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, further comprising information acquisition means for acquiring the image and the distance information.

3. 3. The image processing apparatus according to claim 2, wherein the information acquisition means acquires the image and the distance information from an image sensor.

4. 3. The image processing apparatus according to claim 2, wherein the information acquisition means acquires the image and the distance information from an image recording means.

5. 2. The image processing apparatus according to claim 1, wherein the filter processing means performs the filter processing with different characteristics on each piece of distance information by switching the number of taps of a filter.

6. 2. The image processing apparatus according to claim 1, wherein the filtering process is a process using a smoothing filter.

7. 7. The image processing apparatus according to claim 6, wherein the filtering process is a process using a median filter.

8. 2. The image processing apparatus according to claim 1, wherein said region dividing means divides said image into areas classified into organs and areas other than said organs.

9. 9. The image processing device according to claim 8, wherein the organs include any one of an eye, a nose, and a mouth.

10. 9. The image processing device according to claim 8, wherein the object other than the organ includes any one of skin, hair, and accessories.

11. The image processing device according to claim 10, wherein the accessory includes glasses or earrings.

12. The image processing apparatus according to claim 1 , wherein the region dividing means divides the image into a plurality of regions by semantic segmentation.

13. 2. The image processing apparatus according to claim 1, wherein the area dividing means divides the face area into skin or hair areas according to the area and color of the face area.

14. 2. The image processing apparatus according to claim 1, wherein the filtering means makes the size of the filtering process for the organ smaller than the size of the filtering process for the skin.

15. 2. The image processing apparatus according to claim 1, wherein the filter processing means performs linear interpolation or substitution using a representative value for an accessory.

16. 2. The image processing apparatus according to claim 1, wherein the filtering means does not use or reduces the weight of the distance measurement values ​​in the region of the division boundary in the filtering process.

17. 2. The image processing device according to claim 1, wherein the filtering means performs filtering on the nose or the eyeglasses based on predetermined model information.

18. 2. The image processing device according to claim 1, wherein the filter processing means controls whether or not to perform the filter processing depending on a photographing distance or a size of a face.

19. 2. The image processing device according to claim 1, wherein said filter processing means changes filter characteristics even for the same organ depending on the face orientation.

20. A region segmentation step of classifying and segmenting the image into a plurality of regions; a filtering step of performing filtering processing with different characteristics according to the classification on each of the distance information corresponding to the plurality of regions of the image; a combining step of combining the filtered distance information; a 3D data generation processing step of generating 3D data of the subject based on the distance information and the image synthesized in the synthesis step; An image processing method comprising:

21. A computer program for controlling each means of the image processing apparatus according to any one of claims 1 to 19 by a computer.

Images