Information processing device, imaging device, control method, and program

Abstract

Generate 3D shape data that provides a suitable viewing experience. [Solution] The information processing device is an information processing device having a generation means for generating three-dimensional shape data of an object based on a distance image, and includes a identification means for identifying an edge region in which edges relating to the object are distributed in the distance image, a determination means for determining whether or not there are irregularities that do not meet the tolerance conditions in the ridge lines corresponding to the edge region in the three-dimensional shape data generated by the generation means, and a correction means for executing a process related to correcting the shape of the ridge lines when the determination means determines that there are irregularities that do not meet the tolerance conditions.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an imaging apparatus, a control method, and a program, and particularly to a technique for generating a three-dimensional shape based on a distance image.

Background Art

[0002] There is a technique for generating three-dimensional shape data of a subject based on a captured image and distance information corresponding to the captured image (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in the technique described in Patent Document 1, since the distance information indicates the distance from the imaging device at the time of capturing the captured image, three-dimensional shape data for surfaces that are not imaged, such as the back surface of the subject, is not generated. That is, the three-dimensional shape data of the subject has the surface captured in the captured image but the back surface is not closed, and is configured to have a boundary portion (edge) between the front surface and the back surface. Therefore, when the three-dimensional shape data is changed to an angle different from that at the time of imaging for viewing purposes or the like, the edge may be in a visible state.

[0005] By the way, since the surface of the subject distributed at the boundary portion in the three-dimensional shape data is a surface extending in the imaging direction (depth direction / depth dimension) (a surface whose normal line is substantially orthogonal to the imaging direction), the accuracy of the distance derived by the phase difference detection method may decrease. As a result, when viewing the three-dimensional shape data at an angle different from that at the time of imaging, the ridgelines distributed at the edge may have a convex and concave shape in the depth direction at the time of imaging, which may interfere with a suitable viewing experience.

[0006] This invention has been made in view of the above-mentioned problems, and aims to provide an information processing device, an imaging device, a control method, and a program for generating three-dimensional shape data that provides a suitable viewing experience. [Means for solving the problem]

[0007] To achieve the aforementioned objectives, the present invention provides an information processing device having a generation means for generating three-dimensional shape data of an object based on a distance image, characterized by comprising: an identification means for identifying an edge region in which edges relating to the object are distributed in the distance image; a determination means for determining whether or not there are irregularities that do not satisfy acceptable conditions in the ridge lines corresponding to the edge region among the three-dimensional shape data generated by the generation means; and a correction means for executing a process relating to the shape correction of the ridge lines when the determination means determines that there are irregularities that do not satisfy acceptable conditions. [Effects of the Invention]

[0008] With this configuration, the present invention makes it possible to generate three-dimensional shape data that provides a suitable viewing experience. [Brief explanation of the drawing]

[0009] [Figure 1] Block diagram illustrating the hardware configuration of a digital camera 100 according to embodiments and modified examples of the present invention. [Figure 2] Figure illustrating the configuration of an image sensor according to embodiments and modified examples of the present invention. [Figure 3] Diagram to explain on-sensor phase-detection autofocus [Figure 4] A flowchart illustrating the distance image generation process performed by a digital camera 100 according to embodiments and modifications of the present invention. [Figure 5] A diagram illustrating how to derive distance information from the amount of defocus. [Figure 6] Figure illustrating the overview of the generation of 3D shape data according to embodiments and modified examples of the present invention. [Figure 7]A diagram illustrating the irregularities of the edges that appear at the ends of 3D shape data. [Figure 8] A diagram illustrating the outline of a process for increasing the angle at the vertices forming a ridge line, according to embodiments and modifications of the present invention. [Figure 9] Figure illustrating edge region images relating to embodiments and modifications of the present invention. [Figure 10] Figure illustrating the edge region image generation method according to embodiments and modified versions of the present invention. [Figure 11] Figure illustrating the generation of 3D shape data based on edge region images according to Embodiment 1 of the present invention. [Figure 12] A flowchart illustrating the shape generation process performed by the digital camera 100 according to Embodiment 1 of the present invention. [Figure 13] A flowchart illustrating the correction process according to Embodiment 1 of the present invention. [Figure 14] Figure illustrating the generation of 3D shape data based on edge region images according to Embodiment 2 of the present invention. [Figure 15] A flowchart illustrating the shape generation process performed by the digital camera 100 according to Embodiment 2 of the present invention. [Figure 16] Figure illustrating shape correction in distance images according to Modification 1 of the present invention. [Modes for carrying out the invention]

[0010] [Embodiment 1] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] The following describes an example of an embodiment applied to a digital camera that can generate three-dimensional shape data of a subject based on distance information storing information on the distance to the subject where an optical image is formed at each pixel position of an image sensor, as an example of an imaging device. However, the present invention is applicable to any device (information processing device) capable of generating a three-dimensional shape based on a distance image. Such devices can include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game machines, robots, drones, and the like.

[0012] 《Hardware Configuration of Digital Camera》 Hereinafter, the hardware configuration of the digital camera 100 according to the present embodiment will be described with reference to the block diagram of FIG. 1.

[0013] The image sensor 11 photoelectrically converts the optical image of the subject and generates an image signal (analog image signal). The image sensor 11 may be a known CCD or CMOS color image sensor having a color filter of a primary color Bayer array, for example.

[0014] An optical image (subject image) of the subject is formed on the imaging surface of the image sensor 11 by the imaging optical system 10. The imaging optical system 10 includes a plurality of lenses arranged along the optical axis 103. The plurality of lenses includes a focus lens 102 for adjusting the focal length of the imaging optical system 10. The focus lens 102 is configured to be movable along the optical axis. The operation control of the focus lens 102 is performed by the control unit 12. The control unit 12 performs drive control to change the position of the focus lens 102 based on the defocus amount derived by the processing unit 14 described later.

[0015] The imaging optical system 10 also includes an aperture 104 with an adjustable aperture value (aperture). The operation of the aperture 104 is also controlled by the control unit 12. The control unit 12 derives an aperture value based on shooting conditions determined, for example, by automatic exposure control (AE), and controls the operation of the aperture 104 based on that aperture value. The aperture 104 may also have the function of a mechanical shutter. The exit pupil 101 is an image of the imaging optical system 10 at its widest aperture as viewed from the image sensor 11 side, and the figure shows the position of the exit pupil 101.

[0016] The image sensor 11 has a pixel array in which multiple pixels (photoelectric conversion elements) are arranged in a two-dimensional array, and peripheral circuits for reading signals from each pixel. Each pixel accumulates a charge corresponding to the amount of incident light through photoelectric conversion. By reading signals with a voltage corresponding to the amount of charge accumulated during the exposure period from each pixel, the peripheral circuits obtain an analog image signal representing the subject image formed on the imaging surface by the imaging optical system 10. As will be described in detail later, in the digital camera 100 of this embodiment, each pixel of the image sensor 11 has multiple photoelectric conversion areas or photoelectric conversion elements, and is configured to generate a pair of parallax images in a single shot. As will be described later, based on this pair of parallax images, it is possible to perform phase-difference detection autofocus (phase-difference AF) and generate distance information.

[0017] The control unit 12 is a control device that controls the operation of each block of the digital camera 100. The control unit 12 includes one or more processors (hereinafter referred to as CPUs) capable of executing programs. The control unit 12 controls the operation of each block by, for example, reading a program stored in the ROM 21, loading it into the RAM 20, and having the CPU execute it. Such programs include programs that realize various functions of the digital camera 100.

[0018] ROM21 is, for example, a rewritable non-volatile memory. ROM21 stores, for example, programs that the CPU of the control unit 12 can execute, setting values, GUI data, etc. In contrast, RAM20 is a volatile memory. RAM20 is used, for example, as a program deployment area for programs executed by the CPU of the control unit 12, and as a temporary storage area for saving values ​​needed during program execution. RAM20 is also used as working memory for the processing unit 14, a buffer memory for temporarily storing image signals obtained by imaging, and video memory for the display unit 17 (described later).

[0019] The processing unit 14 applies predetermined image processing to the analog image signal read from the image sensor 11 to generate signals and image data according to the application, and acquires and / or generates various information. The processing unit 14 may be a dedicated hardware circuit, such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the processing unit 14 may be configured such that a processor, such as a DSP (Digital Signal Processor) or GPU (Graphics Processing Unit), executes software to realize a specific function. The processing unit 14 outputs the acquired or generated information and data to the control unit 12, RAM 20, etc., according to the application.

[0020] Image processing performed by the processing unit 14 may include, for example, preprocessing, color interpolation, correction, detection, data processing, special effects processing, distance image generation, edge detection, and 3D shape generation. Preprocessing may include A / D conversion, signal amplification, reference level adjustment, and defective pixel correction. Color interpolation is performed when a color filter is provided on the image sensor 11. Color interpolation is a process that interpolates the values ​​of color components not included in the individual pixel data that make up the image data. Color interpolation is also called demosaicing. Correction processing may include white balance adjustment, gradation correction, correction of image degradation caused by optical aberrations of the imaging optical system 10 (image recovery), correction of the effect of vignetting of the imaging optical system 10, and color correction. Detection processing may include detection of feature regions (e.g., face regions or human body regions) and their movement, person recognition processing, and semantic segmentation processing. Data processing may include processes such as region cropping, compositing, scaling, encoding and decoding, and header information generation (data file generation). The generation of display image data and recording image data is also included in data processing. Special effects processing may include processes such as adding blur effects, changing color tones, and relighting. Distance image generation processing includes processes for deriving the amount of parallax, converting the amount of parallax to the amount of defocus, and converting the amount of defocus to a distance image. Edge detection processing is the process of detecting the edges of the subject (3D shape) from the recording image data and distance image based on the results of the above detection processing, and generating an edge region image. The edge detection processing may also store edge detection information by detecting the edge region based on the positional relationship of the 3D point clouds that form the edge, for the 3D point cloud generated in the 3D shape generation processing described later. 3D shape generation processing is the process of generating the 3D shape of the subject based on the display image data, distance image, and edge region image. Note that these processes are merely examples of the processes performed by the processing unit 14 and do not limit all of the processes performed by the processing unit 14.

[0021] The storage unit 15 is a recording medium configured to be detachably attached to, for example, the digital camera 100. The storage unit 15 may also be, for example, a combination of a memory card and its reader / writer. The storage unit 15 can record data files containing, for example, image data obtained by imaging. The storage unit 15 can also be configured to use multiple types of recording media.

[0022] The input unit 16 is a user interface (user-operable input device) provided on the digital camera 100, such as a dial, button, switch, or touch panel. When an operation input is made to any user interface, the input unit 16 outputs a control signal corresponding to that operation input to the control unit 12. Based on the control signal, the control unit 12 executes the function or operation assigned to the input device.

[0023] The display unit 17 is, for example, a display device such as a liquid crystal display or an organic EL display. The display unit 17 functions as an electronic viewfinder (EVF) by continuously displaying the video captured in conjunction with the capture of video. The operation of making the display unit 17 function as an electronic viewfinder (EVF) is sometimes called live view display or through display. The image displayed on the display unit 17 by live view display or through display is sometimes called a live view image or through image. The display unit 17 can also be, for example, a touch display. If the display unit 17 is a touch display, software keys can be realized by combining GUI parts displayed on the display unit 17 with a touch panel.

[0024] The communication unit 18 is a communication interface with an external device. The control unit 12 can communicate with an external device through the communication unit 18 in accordance with one or more wired or wireless communication standards.

[0025] The motion sensor 19 generates a signal corresponding to the movement of the digital camera 100. The motion sensor 19 can be a combination of, for example, an accelerometer that outputs a signal corresponding to movement in each of the XYZ axes and a gyroscope that outputs a signal corresponding to movement around each axis.

[0026] 《Example of image sensor configuration》 Next, an example of the configuration of the image sensor 11 of this embodiment will be described in detail with reference to Figure 2. Figure 2(a) is a plan view of the pixel array of the image sensor 11 as seen from the imaging surface side. The pixel array is provided with primary color Bayer array color filters. Therefore, one color filter of either red (R), green (G), or blue (B) is provided for each pixel, and a 2x2 pixel group 210 is regularly arranged as a repeating unit. Note that color filters with arrays other than the primary color Bayer array can also be used.

[0027] Figure 2(b) is a vertical cross-sectional view of a single pixel. Figure 2(b) corresponds to the configuration of the I-I' cross-section in Figure 2(a). Each pixel has a light guide layer 213 and a light receiving layer 214. The light guide layer 213 has a microlens 211 and a color filter 212. The light receiving layer 214 has a first photoelectric conversion unit 215 and a second photoelectric conversion unit 216.

[0028] The microlens 211 is configured to efficiently guide the light beam incident on the pixel to the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216. The color filter 212 is either an R filter, a G filter, or a B filter.

[0029] Both the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 generate an electric charge corresponding to the amount of incident light. The image sensor 11 can selectively read signals from individual pixels from one or both of the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216. In this specification, the signal obtained from the first photoelectric conversion unit 215 may be referred to as the A signal, the signal obtained from the second photoelectric conversion unit 216 as the B signal, and the signal obtained from both the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 may be referred to as the A+B signal.

[0030] The first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 are positioned differently with respect to the exit pupil 101. As a result, the image consisting of signal A and the image consisting of signal B, read from the same pixel area, form a disparity image pair. That is, the image sensor 11 of this embodiment can separate and image light beams that have passed through different pupil areas of the imaging optical system. Therefore, by using signal A and signal B, the amount of defocus can be derived based on the principle of phase-difference autofocus. Accordingly, signal A and signal B can each be used as focus detection signals.

[0031] On the other hand, the A+B signal corresponds to the signal obtained when a pixel has one photoelectric converter. Therefore, by acquiring the A+B signal from each pixel, an analog image signal can be obtained.

[0032] The A signal can be derived by subtracting the B signal from the A+B signal, and the B signal can be derived by subtracting the A signal from the A+B signal. Therefore, by reading out the A+B signal and either the A signal or the B signal from each pixel, all three signals—A, B, and A+B—can be obtained. The type of signal read from each pixel is controlled by the control unit 12.

[0033] Figure 2 illustrates a configuration in which each pixel has two photoelectric conversion units 215 and 216 arranged horizontally. However, a configuration with a total of four photoelectric conversion units, two horizontally and two vertically, is also possible. Furthermore, a configuration in which multiple pairs of pixels, one dedicated to generating the A signal and the other dedicated to generating the B signal, are distributed within the pixel array is also possible. In other words, the structure of the image sensor 11 can adopt any configuration that corresponds to image plane phase-detection autofocus.

[0034] 《Principle of on-sensor phase-detection autofocus》 The principle for deriving the defocus amount using signals A and B will be explained with reference to Figure 3.

[0035] Figure 3(a) is a schematic diagram showing the relationship between the exit pupil 101 of the imaging optical system 10 and the light beam incident on the first photoelectric conversion unit 215 of a certain pixel. Similarly, Figure 3(b) is a schematic diagram showing the relationship between the light beam incident on the second photoelectric conversion unit 216 of the same pixel and the exit pupil 101. In this specification, the direction parallel to the optical axis of the imaging optical system is defined as the z direction or defocus direction, the direction perpendicular to the optical axis and parallel to the horizontal direction of the imaging surface is defined as the x direction, and the direction perpendicular to the optical axis and parallel to the vertical direction of the imaging surface is defined as the y direction.

[0036] The microlens 211 is positioned such that it is optically conjugate to the exit pupil 101 and the light-receiving layer 214. The light beam that passes through the exit pupil 101 of the imaging optical system 10 is focused by the microlens 211 and incident on the first photoelectric conversion unit 215 or the second photoelectric conversion unit 216. At this time, as shown in Figures 3(a) and 3(b), the light beam that has passed through different regions of the exit pupil 101 mainly incidents on the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216, respectively. Specifically, the light beam that has passed through the first pupil region 310 is incident on the first photoelectric conversion unit 215, and the light beam that has passed through the second pupil region 320 is incident on the second photoelectric conversion unit 216.

[0037] Signals A and B are acquired from each of multiple pixels arranged horizontally around the pixel of interest. In this case, the relative positional shift (phase difference or disparity) between the image signal based on the A signal sequence (image A) and the image signal based on the B signal sequence (image B) is proportional to the amount of defocus of the pixel of interest.

[0038] Figures 3(c) to 3(e) illustrate the relationship between the focus state and the phase difference (or parallax amount). In the figures, the first luminous beam 311 represents the luminous beam passing through the first pupil region 310, and the second luminous beam 321 represents the luminous beam passing through the second pupil region 320.

[0039] Figure 3(c) shows the focused state, where the first luminous beam 211 and the second luminous beam 221 converge on the imaging plane. At this time, the phase difference (or parallax) between image A and image B is 0. Figure 3(d) shows that the first luminous beam 311 and the second luminous beam 321 converge on the object side (negative z-axis side) of the imaging plane. At this time, the phase difference (or parallax) between image A and image B has a negative value (<0).

[0040] Figure 3(e) shows that the first luminous beam 311 and the second luminous beam 321 converge behind the imaging plane (towards the positive z-axis) when viewed from the object side. At this time, the phase difference (or parallax amount) between image A and image B has a positive value (>0).

[0041] Thus, the phase difference (or parallax amount) between image A and image B has a sign corresponding to the relationship between the position where the first light beam 311 and the second light beam 321 converge and the imaging plane, and a magnitude corresponding to the amount of defocus. Therefore, by deriving the correlation amount while relatively shifting image A and image B, the phase difference (or parallax amount) between image A and image B can be obtained based on the shift amount that maximizes the correlation amount.

[0042] Distance image generation process The distance image generation process in the processing unit 14 of this embodiment, which generates a distance image, will be explained using the flowchart in Figure 4. The process corresponding to the flowchart can be executed by the processing unit 14 by the control unit 12, for example, reading the corresponding processing program stored in the ROM 21, loading it into the RAM 20, and executing it. This distance image generation process will be explained as being executed prior to the shape generation process described later, for example, when an operation input related to the generation of 3D shape data of an object is made. Prior to this distance image generation process, it is assumed that A signals and B signals for each pixel of the image sensor 11 are stored in the RAM 20.

[0043] In step S401, the processing unit 14 corrects the light intensity of the A signal and the B signal. In particular, for pixels with a large image height, the difference in shape between the first pupil region 310 and the second pupil region 320 becomes large due to the aperture vignetting of the imaging optical system 10, resulting in a difference in the magnitude (signal value) of the A signal and the B signal. The processing unit 14 corrects the difference in magnitude between the A signal and the B signal by applying, for example, a correction value corresponding to the pixel position to the A signal and the B signal. The correction value can be stored in advance in the ROM 21, for example.

[0044] In S402, the processing unit 14 applies noise reduction processing to the A signal and the B signal. Generally, the higher the spatial frequency, the more noise components there are relatively. Therefore, the processing unit 14 applies a low-pass filter, whose pass-through rate decreases as the spatial frequency increases, to the A signal and the B signal. However, due to manufacturing errors in the imaging optical system 10, good results may not be obtained with the light intensity correction in S101. For this reason, in S402, the processing unit 14 may also apply a band-pass filter that blocks the DC component and has a low pass-through rate for high-frequency components.

[0045] In S403, the processing unit 14 detects the phase difference (or disparity amount) between the A signal and the B signal. The processing unit 14 generates the A signal sequence and the B signal sequence from a horizontally continuous sequence of pixels including the pixel of interest. The processing unit 14 then derives a correlation amount by relatively shifting the A signal sequence and the B signal sequence. The correlation amount can be, for example, NCC (Normalized Cross-Correlation), SSD (Sum of Squared Difference), or SAD (Sum of Absolute Difference).

[0046] The processing unit 14 derives the shift amount that maximizes the correlation between the A signal sequence and the B signal sequence in units less than one pixel, and uses this as the phase difference (or disparity amount) at the pixel of interest. The processing unit 14 detects the phase difference (or disparity amount) at each individual pixel position while changing the position of the pixel of interest. Note that the phase difference (or disparity amount) between the A signal and the B signal can also be detected by any other known method. The spatial resolution used to derive the phase difference (or disparity amount) may be lower than the spatial resolution of the captured image.

[0047] In S404, the processing unit 14 converts the detected phase difference (or parallax amount) into a defocus amount. Since the phase difference (or parallax amount) has a magnitude corresponding to the defocus amount, it can be converted into a defocus amount by applying a predetermined conversion coefficient. If the phase difference (or parallax amount) is d and the conversion coefficient is K, then the defocus amount ΔL is: This can be derived using JPEG2026054261000002.jpg1037. The processing unit 14 converts the detected phase difference (or parallax amount) into a defocus amount, thereby generating two-dimensional information (defocus image) that represents the amount of defocus according to the pixel position.

[0048] Next, we will explain how to obtain depth (distance) information based on the amount of defocus, using Figure 5. In Figure 5, OBJ represents the object plane, IMG represents the image plane, H is the front principal point, H' is the rear principal point, f is the focal length of the imaging optical system (lens), S is the distance from the object plane to the front principal point, and S' is the distance from the rear principal point to the image plane. Also, ΔS' is the amount of defocus, and ΔS is the relative distance to the object corresponding to the amount of defocus. The dashed line is the optical axis, the dotted line is the imaging beam, and the dashed line is the defocus beam.

[0049] In lens imaging, The relationship JPEG2026054261000003.jpg1434 holds true. When defocused, the relationship is transformed. The file JPEG2026054261000004.jpg1452 is valid.

[0050] Here, S and f at the time of focus can be obtained from the shooting condition information (shooting information). Therefore, S' can be derived based on equation (1). In addition, the amount of defocus ΔS' can also be obtained by, for example, phase-difference autofocus (AF). With this, ΔS can be obtained from equation (3), and the distance S to the object surface OBJ can be derived.

[0051] In S405, the processing unit 14 generates a distance image of the subject using the generated defocus image and shooting information. The distance image is two-dimensional data in which each pixel represents the distance to the subject corresponding to the pixel position of that pixel. The distance image is also called a distance map, depth map, or depth image.

[0052] Although this specification describes a method for acquiring distance images using the amount of defocus, distance images can also be acquired using other known methods. For example, the subject distance can be acquired for each pixel by identifying the focus lens position where the contrast evaluation value is maximized for each pixel. Alternatively, distance information for each pixel can be acquired based on the correlation between the amount of blur and the distance from image data obtained by taking multiple shots of the same scene with different focusing distances and the point image distribution function (PSF) of the optical system. Details of these techniques are described in, for example, Japanese Patent Application Publication No. 2010-177741 and U.S. Patent No. 4,965,840. Furthermore, in a configuration in which the digital camera 100 can acquire a parallax image pair (a pair of captured images), it is also possible to derive the subject distance for each pixel using methods such as stereo matching.

[0053] 《Generation of 3D shape data》 Next, we will explain how to generate 3D shape data based on depth images.

[0054] This generation method includes converting each pixel of a depth image to coordinate values ​​in the world coordinate system using the focal length and focus position obtained from the shooting information to generate point cloud data, and polygonizing the point cloud data so that it can be easily handled as a 3D model. Any known method can be used for polygonization. In the following description, point cloud data may also be referred to as 3D shape data.

[0055] For example, 3D shape data can be converted into a polygon mesh by defining a surface using the coordinate information of any three adjacent points in the 3D shape data. Furthermore, texture information (or simply a texture) to be applied to the polygon can be extracted from the information of the captured image corresponding to the three points used for polygonization. Additionally, filtering may be applied to the distance image before conversion to world coordinate system coordinate values, or to the 3D shape data before polygonization. This filtering may include, for example, applying a median filter to smooth small shape changes.

[0056] When polygonization is performed, the processing unit 14 converts the polygon data into 2D structured data using any known method so that the amount of data can be reduced using 2D image predictive coding techniques. Note that polygonization is not mandatory, and 3D shape data may be handled in point cloud format. As long as the data format is such that known 2D image predictive coding techniques can be applied, the method of representing the 3D shape of the object (subject) is arbitrary.

[0057] Figure 6 shows an example of the subject object, its distance image, and its 3D shape data.

[0058] In the example in Figure 6, the subject is a cylindrical object as shown in Figure 6(a). When the subject is photographed from the side and a depth image is obtained, a depth image as shown in Figure 6(b) is obtained. Here, the shading in the depth image in Figure 6(b) indicates that the lighter the color, the greater the distance (farther away). In other words, in the depth image in the figure, the central part of the cylinder is closest to the digital camera 100, and the distance increases as the parts further to the left and right of the center move away. Figure 6(c) schematically shows the state when the 3D shape data obtained by converting the depth image is plotted in the world coordinate system. Since a depth image is not generated for the parts of the object that are not photographed (outside the imaging range), only the parts corresponding to the depth image (601 corresponding to the cylinder and 602 corresponding to the background) are generated in the 3D shape data.

[0059] Shape correction of 3D shape data The following describes the outline of the shape correction of the 3D shape data generated for the subject, which is performed in the digital camera 100 of this embodiment.

[0060] As described above, when generating 3D shape data based on depth images, unnatural irregularities may occur in the depth direction (z direction) at the edges of the 3D shape. Figure 7 illustrates such irregularities at the edges. The example in Figure 7 shows the vertices distributed at the edges of the 3D shape data. To facilitate understanding of the invention, in the example in Figure 7, each vertex is shown in a 2D form projected from the y-axis direction onto the xz-plane.

[0061] In the figure, the white circles represent vertices distributed at the edges of the 3D shape data, while the black circles represent vertices distributed elsewhere. Therefore, the group of edges connecting the vertices indicated by the white circles forms the edges of the 3D shape data. As shown in the figure, when the distance values ​​indicated by the corresponding pixels in the depth image are derived with varying degrees of precision, the edges may show irregularities in the depth direction (z-direction).

[0062] In this embodiment, the state of irregularities on a ridge is evaluated using the angle between two adjacent edges on that ridge. More specifically, the state of irregularities on a ridge is evaluated at each vertex included in that ridge (hereinafter sometimes referred to as a target vertex) based on the angle between two edges connected through that vertex.

[0063] The angle formed by a target vertex can be derived based on the coordinate values ​​of the target vertex and the coordinate values ​​of two vertices adjacent to it on the ridge (both vertices forming the ridge; hereinafter referred to as adjacent points). For example, if the coordinates of target vertex A are [Xa, Ya, Za], and the coordinates of adjacent points B and C are [Xb, Yb, Zb] and [Xc, Yc, Zc], respectively, the angle θ formed by the target vertex can be derived using the dot product of the vectors connecting the vertices by the following formula. JPEG2026054261000005.jpg5149 Here, since the formula can also derive angles made by vectors primarily in the xy-plane, conditions such as the difference between the coordinate values ​​of the target vertex, adjacent points, and depth directions exceeding a threshold may be included in the evaluation from the perspective of evaluating the unevenness in the depth direction of the ridge.

[0064] In this specification, "angle" refers to the smaller of the angles formed by two edges connected via a target vertex. In the example in Figure 7, the angle formed by the two edges 702 and 703 connected via vertex 701 is angle 704, which is formed above vertex 701 (on the side where no polygon exists). On the other hand, the angle formed by the edges 703 and 706 connected via vertex 705 is angle 707, which is formed below vertex 705 (on the side where a polygon exists).

[0065] Therefore, the degree of unevenness on the edges of 3D shape data will be greater the smaller the angle (the sharper the angle). In other words, if two sides with small angles are included in the 3D shape data, unnatural unevenness may appear on the edges when the 3D shape data is viewed from an angle different from the imaging direction. For this reason, the digital camera 100 of this embodiment is configured to perform shape correction on the 3D shape data if the angle made by any vertex included in the edge of the 3D shape data falls below a threshold (angle threshold), as it is considered that the acceptable conditions are not met. As will be described in detail later, this shape correction involves processing to increase the angle made at each vertex that constitutes the edge.

[0066] <Process to increase the angle between the two points> Here, an overview of the process performed in the processing unit 14 of this embodiment to increase the angle formed by the target vertices constituting the edge will be explained with reference to Figure 8. As shown below, the processing unit 14 of this embodiment classifies the state of the target vertices into three types, and the content of the process to increase the angle formed differs for each state. First, the classification of the state of the target vertices will be explained.

[0067] If the angle formed by the target vertex is below the angle threshold, the processing unit 14 evaluates the degree of unevenness of the target vertex on the edge. In this embodiment, the degree of unevenness is evaluated based on whether the distance between the target vertex and other adjacent vertices (adjacent points) on the edge, that is, between the target vertex and the vertices forming the edge that connect the target vertex to the edge, exceeds a threshold (distance threshold). Specifically, the processing unit 14 evaluates the degree of unevenness as moderate if the distance between the target vertex and the adjacent points does not exceed the distance threshold, and evaluates the degree of unevenness as large if it exceeds the distance threshold.

[0068] Furthermore, if the degree of unevenness of the target vertex is large, the processing unit 14 further evaluates the significance of the target vertex based on the area of ​​the polygon mesh containing the target vertex. In this embodiment, significance is evaluated by whether the area of ​​any part of the polygon mesh formed with the target vertex exceeds a threshold (area threshold). Specifically, if the area of ​​the polygon mesh formed with the target vertex exceeds the area threshold, the processing unit 14 considers that there was a large error in the pixel values ​​of the depth image obtained for the target vertex, and evaluates that the significance of the target vertex is low. That is, the processing unit 14 evaluates that the significance is low because the target vertex shows protruding unevenness along the edge, and the texture applied to the polygon mesh containing the target vertex is also stretched, resulting in an appearance that is not suitable for viewing. On the other hand, if the area of ​​the polygon mesh formed with the target vertex does not exceed the area threshold, the processing unit 14 considers that the pixel values ​​of the depth image obtained for the target vertex do not have a large error, and evaluates that the significance of the target vertex is not low. In other words, the processing unit 14 evaluates that the irregularities shown by the target vertices on the ridge line may be due to the shape of the subject, or that the texture applied to the polygon mesh containing the target vertices is significant enough to ensure an appropriate appearance when viewed.

[0069] In this embodiment, the processing unit 14 classifies the state of the target vertex into the following four types based on the angle it makes with respect to the target vertex. (1) The angle formed does not fall below the angle threshold (the degree of unevenness is small: not subject to shape correction). (2) The angle is below the angle threshold, and the distance between the target vertex and adjacent points on the ridge does not exceed the distance threshold (moderate degree of unevenness). (3) The angle is below the angle threshold, the distance between the target vertex and adjacent points on the edge exceeds the distance threshold, and the area of ​​the polygon mesh formed with the target vertex does not exceed the area threshold (the degree of unevenness is large, and the significance of the target vertex is not low). (4) The angle is below the angle threshold, the distance between the target vertex and adjacent points on the edge exceeds the distance threshold, and the area of ​​the polygon mesh formed with the target vertex exceeds the area threshold (the degree of unevenness is large and the significance of the target vertex is low).

[0070] Of these, in state (1), the acceptable conditions are met, so the processing unit 14 does not perform shape correction on the target vertex. On the other hand, in states (2), (3), and (4), the processing unit 14 performs different shape correction processes on the target vertex.

[0071] In the case of state (2), as shown in Figure 8(a), the degree of unevenness at the target vertex 801 is moderate, so the processing unit 14 adjusts the depth coordinate values ​​of the target vertex 801 by applying a filter. This filter increases the angle related to the target vertex, as shown in Figure 8(b). This filter changes the depth coordinate values ​​of the target vertex 801 based on the depth coordinate values ​​of other vertices distributed in the vicinity of the target vertex 801 along the edge. More specifically, the filter changes the depth coordinate values ​​of the target vertex 801 to statistical values ​​of the depth coordinate values ​​of other vertices distributed in the vicinity of the target vertex 801 along the edge. For example, the mean, median, weighted mean, etc., can be used for these statistical values. Here, the weights used to derive the weighted mean may be set so that they become smaller the further away from the target vertex 801 is, and larger the closer to the target vertex 801 is. As a result, the variation in coordinate values ​​(distance values) in the depth direction near the target vertex 801 can be reduced, and the 3D shape data can be corrected to have a smoother shape with less unevenness along the edges.

[0072] In state (3), as shown in Figure 8(c), the degree of unevenness at the target vertex 811 is large, but since there is also significance in the coordinate values ​​of the target vertex 811 in the depth direction, the processing unit 14 adds a new vertex to increase the angle related to the target vertex 811. As shown in Figure 8(c), in state (3), there is a steep unevenness in the edge of the target vertex 811, so as shown in Figure 8(d), the unevenness of the edge is smoothed out by adding new vertices 812 and 813 adjacent to the target vertex 811.

[0073] In state (4), as shown in Figure 8(e), the degree of unevenness at target vertex 821 is large, but since there is no significant difference in the coordinate values ​​of target vertex 821 in the depth direction, the processing unit 14 deletes target vertex 821 from the vertices that form the edge. As shown in Figure 8(f), in state (4), the edge has sharp unevenness at target vertex 821, and stretching is expected in the applied texture, so as shown in Figure 8(f), the unevenness of the edge is smoothed out by deleting target vertex 821.

[0074] In this way, the processing unit 14 corrects the shape of the edges of the 3D shape data by applying processing according to the state of each vertex that is determined not to meet the tolerance conditions based on the angle it forms. This shape correction process is repeatedly performed on all vertices forming the edges, for example, by changing the target vertex until the tolerance conditions related to the angle it forms are met. As a result, the shape of the edges of the 3D shape data is made less uneven and can withstand viewing from different angles.

[0075] <Identifying edge regions> Here, the generation of 3D shape data based on depth images involves a process of forming a polygon mesh by selecting three adjacent points from the defined vertices by converting the pixels of the depth image into 3D coordinates. During the formation of this polygon mesh, a 3D search is required to identify a single vertex and the vertices that form the polygon mesh, which can lead to excessive computation.

[0076] Incidentally, the imaging range may include multiple subjects, and these subjects may be distributed at different distances in the depth direction and may overlap from the perspective of the digital camera 100. When multiple subjects overlap in this way, adjacent pixels at the boundaries between subjects in the depth image may indicate different distances. The vertices defined by converting the pixel values ​​of each of these pixels into 3D coordinates do not form a polygon mesh from the viewpoint of generating individual 3D shape data for each subject. That is, if a polygon mesh is formed for these vertices, it will exhibit an unnatural 3D shape that extends in the depth direction, and 3D shape data will be generated that makes it appear as if multiple subjects have been merged into one. In other words, there is no need to perform a search for polygon mesh formation between the vertices corresponding to each of the multiple subjects that exist at different distances in the depth direction. Therefore, when generating 3D shape data based on a depth image, it is preferable to make it easy to identify the edge region where the edges (boundaries) related to subjects in the imaging range are distributed.

[0077] Therefore, in the digital camera 100 of this embodiment, prior to generating 3D shape data based on the depth image, the processing unit 14 generates an edge region image that enables the identification of edge regions within the imaging range. In this specification, "edge region" refers to a group of pixels in a 2D image (distance image, imaging image (RGB image), etc.) corresponding to the imaging range in which the edges of the subject appear. In this embodiment, the edge region image is a 2D image with the same number of horizontal and vertical pixels as the imaging image or depth image, and is described as being constructed by storing a pixel value of "1" in pixels identified as edge regions and a pixel value of "0" in other pixels.

[0078] Edge region images can be generated using RGB images, luminance images (also called Y signal images) generated by converting RGB images, or distance images. In addition, edge region images can also be generated using variance images calculated from RGB images or Y signal images, or segmentation images generated using semantic segmentation, etc.

[0079] The following describes the method for generating edge region images, using the example of a case where the images of subjects are distributed in an RGB image captured within the imaging range as shown in Figure 9(a). In the example in Figure 9(a), a spherical subject is distributed within the imaging range at a position close to the digital camera 100 (foreground), and a cubic subject is distributed at a position further away from the digital camera 100 than the sphere (background). In this case, the ideal edge region image for the RGB image in Figure 9(a) is an image in which the pixel value "1" is stored at the pixel positions of the edges of each subject image, as shown in Figure 9(b). In order to generate this ideal edge region image, in this embodiment, the processing unit 14 employs a method for generating the edge region image using a distance image, a segmentation image, and an RGB image.

[0080] In a method where a pre-trained model that has learned the shape of an object is input with the RGB image in Figure 9(a) and subjected to semantic segmentation processing, regions showing specific shapes in the image can be extracted as the object region. For example, if the shapes of a sphere and a cube have been learned separately, a segmentation image like the one shown in Figure 10(a) will be output by the segmentation processing performed by the pre-trained model. In the figure, the regions of the sphere and the cube are detected as separate regions as the object region. With a segmentation image in this state, an edge region image like the one in Figure 9(b) can be generated by extracting the edges of each region.

[0081] On the other hand, there are cases where the types of objects that can be trained to learn the shape of a pre-trained model are limited. In such cases, even if the RGB image in Figure 9(a) is input to the pre-trained model, it may detect the sphere and cube as the same geometric object. As a result, a segmentation image is output that treats the sphere and cube as a single object, as shown in Figure 10(b). Therefore, even if edges are extracted based on this image to generate an edge region image, the boundary between the foreground object and the background object may not be identified as an edge region, as shown in Figure 10(c).

[0082] On the other hand, in methods that generate edge region images based solely on depth images, if a distance change exceeding a threshold occurs near the edges of the subject, the contour of the detected edge region may become thicker, resulting in an edge region image like that shown in Figure 10(d). This is due to the disparity calculation performed when generating the depth image, and can occur particularly in areas where actual distance differences occur in real space, when there are differences in the blur state of the subject's image, or when the region size used when deriving the correlation amount is large. In an edge region image like that shown in Figure 10(d), the boundary between the foreground and background subjects becomes ambiguous, and it may be impossible to determine whether the edge region in question is the foreground or background.

[0083] In this way, by using both a segmentation image method and a depth image method, it is possible to generate an edge region image that separates the foreground subject from the background subject, as shown in Figure 10(e). Here, since there may be differences in color, brightness, lighting conditions, etc. between the foreground and background subjects, by using information that detects these differences based on the RGB image, the edge region can be refined to generate an edge region image like the one in Figure 9(b).

[0084] In this embodiment, edge region images are described as being generated using methods based on multiple types of images, but the method for generating edge region images to identify edge regions where edges related to a subject are distributed is not limited to this. As mentioned above, the most suitable method for generating edge region images may vary depending on the shape of the subject and the imaging conditions, so for example, to reduce the amount of computation, one or more of the above methods may be combined to generate the edge region image. Furthermore, the edge detection method related to the subject is not limited to those shown in Figure 10, and it will be easy to understand that other methods can be used.

[0085] By referring to the edge region image generated in this way, it is possible to identify pixels that should be separated as subjects located at different distances when generating 3D shape data of a subject based on a depth image. In this embodiment, pixels shown as edge regions in the edge region image are identified as the edge of the subject on the near side (closer to the digital camera 100) among the subjects adjacent to that edge region at different distances.

[0086] Therefore, in a depth image, if regions with different distance values ​​are distributed through pixels identified as edge regions, those pixels shall define the edges of the regions with smaller distance values. Conversely, for regions with larger distance values, the pixels adjacent to those pixels in the direction from the regions with smaller distance values ​​toward the regions with larger distance values ​​shall define the edges.

[0087] For example, in the case where an edge region image is obtained as shown in Figure 11(a), each pixel in the edge region shown in the edge region image indicates a pixel position in the depth image where pixels corresponding to the edges of the subject on the near side, with small distance values, are distributed. In the example in Figure 11(a), the pixels shown with hatching have a pixel value of 1 and indicate the edges of the subject on the near side. By referring to this edge region image, the depth image can be separated into a region where distance values ​​related to the subject on the near side are distributed (region 1101) and a region where distance values ​​related to the subject on the far side are distributed (region 1102), as shown in Figure 11(b). At this time, in the 3D shape data generated based on region 1101, the vertices obtained by transforming the pixels corresponding to the edges of the subject on the near side, identified by the edge region image, form the edges at the ends of the 3D shape data. On the other hand, the edges of the subject on the far side appear at pixel positions adjacent to the pixels of the edge region in the direction from region 1101 to region 1102, and thus become the group of pixels shown with hatching in Figure 11(c). Therefore, in the 3D shape data generated based on region 1102, the vertices obtained by transforming the pixels corresponding to the edges of the subject on the far side, which were identified based on the edge region image, form the edges at the ends of the 3D shape data.

[0088] By identifying the edges of each subject in the depth image based on the edge region image, the depth image can be separated into regions for each subject, and the generation of 3D shape data for each subject at different depth distances can be made more efficient based on the information of each region. Furthermore, in the process of generating this 3D shape data, it is possible to determine whether a vertex corresponds to an edge of a subject based on whether the source pixel is included in the edge region, and information indicating this can be associated as an attribute of the vertex. By associating such attributes, the group of vertices that form the edges at the ends of the generated 3D shape data can be easily identified, and the processing related to shape correction can be made more efficient.

[0089] Shape generation process The shape generation process for generating 3D shape data in the processing unit 14 of this embodiment will be explained using the flowchart in Figure 12. The process corresponding to the flowchart can be executed by the processing unit 14 by the control unit 12, for example, reading the corresponding processing program stored in the ROM 21, loading it into the RAM 20, and executing it. This shape generation process will be explained as being executed, for example, when an operation input related to the generation of 3D shape data of an object is made. It should be assumed that the distance image generation process and edge region image generation described above have been completed prior to this shape generation process.

[0090] In S1201, the processing unit 14 converts the distance image into a group of vertices having three-dimensional coordinates, and generates three-dimensional shape data of the subject based on the group of vertices and the edge region image.

[0091] In S1202, the processing unit 14 executes correction processing related to the shape correction of the 3D shape data. If multiple subjects are captured in the imaging range, the processing unit 14 executes correction processing for the 3D shape data of each subject.

[0092] <Correction process> The correction process performed in this step will be explained using the flowchart in Figure 13.

[0093] In S1301, the processing unit 14 derives the angle related to each vertex that forms a ridge line in the 3D shape data. Based on the angles related to each vertex forming the ridge line derived in this step, the processing unit 14 determines whether these vertex groups satisfy the acceptable conditions by performing the following S1302 to S1309 processes (hereinafter referred to as the determination process), and performs shape correction as necessary.

[0094] In S1302, the processing unit 14 selects the vertices that were not selected in the current determination process from among the vertices forming the edge line as target vertices.

[0095] In S1303, the processing unit 14 determines whether the target vertex satisfies the acceptable conditions. That is, the processing unit 14 determines whether the angle formed by the target vertex is greater than or equal to the angle threshold. If the processing unit 14 determines that the angle formed by the target vertex is greater than or equal to the angle threshold, it determines that the target vertex satisfies the acceptable conditions and moves the process to S1309. If the processing unit 14 determines that the angle formed by the target vertex is less than the angle threshold, it determines that the target vertex does not satisfy the acceptable conditions and moves the process to S1304.

[0096] In S1304, the processing unit 14 determines whether the distance between the target vertex and its adjacent points exceeds a distance threshold. If the processing unit 14 determines that the distance between the target vertex and its adjacent points exceeds the distance threshold, it moves the process to S1306; otherwise, it moves the process to S1305.

[0097] In S1305, the processing unit 14 performs a filtering process to change the depth coordinate values ​​of the target vertex to statistical values ​​of the depth coordinate values ​​of the vertices (including adjacent points) near the target vertex that form the edge.

[0098] On the other hand, if in S1304 the processing unit 14 determines that the distance between the target vertex and an adjacent point exceeds the distance threshold, then in S1306 the processing unit 14 determines whether the area of ​​the polygon mesh formed with the target vertex (hereinafter referred to as the target polygon) exceeds the area threshold. If the processing unit 14 determines that the area of ​​the target polygon exceeds the area threshold, it moves the processing to S1308; otherwise, it moves the processing to S1307.

[0099] In S1307, the processing unit 14 adds a new vertex near the target vertex and changes the configuration so that the target vertex is connected to the new vertex instead of the adjacent points to form an edge (the new vertex is added to the vertices that form the edge). At this time, the processing unit 14 forms a new polygon mesh for the new vertex and its neighboring vertices (including the target vertex). In addition, with the addition of the new vertex, the processing unit 14 changes the attributes of the adjacent points to indicate that they do not correspond to the edges of the subject. The processing unit 14 also changes the attributes of the new vertex to indicate that it corresponds to the edges of the subject.

[0100] On the other hand, if the processing unit 14 determines in S1306 that the area of ​​the target polygon exceeds the area threshold, it disconnects the connection between the adjacent points on the edge and the target vertex in S1308 and deletes the target vertex. At this time, the polygon mesh (target polygon) that was composed of the target vertex is deleted from the 3D shape data. In addition, if there are vertices that can become the edges of the 3D shape data as a result of deleting the target vertex and target polygon, the processing unit 14 also performs a process to change the attributes of such vertices to indicate that they correspond to the edges of the subject and adds them to the vertices that form the edge.

[0101] In S1309, the processing unit 14 determines whether or not it has determined the acceptable conditions for all vertices that form the edge derived in S1301. If the processing unit 14 determines that it has determined the acceptable conditions for all vertices that form the edge, it moves the process to S1310; if it determines that it has not determined the acceptable conditions for any vertex, it returns the process to S1302.

[0102] In S1310, the processing unit 14 determines whether it has obtained a determination result that all vertices forming the edge line meet the acceptable conditions. That is, the processing unit 14 determines whether it has determined in S1303 that all vertices for which the determination process was performed meet the acceptable conditions. If the processing unit 14 determines that it has obtained a determination result that all vertices forming the edge line meet the acceptable conditions, it completes this correction process. If the processing unit 14 determines that it has obtained a determination result that any vertex forming the edge line does not meet the acceptable conditions, it returns to S1301. That is, if there are vertices for which the acceptable conditions are not met and various processes related to edge line shape correction have been performed, the processing unit 14 returns to S1301 in order to perform the determination process again on the corrected edge line.

[0103] Once the correction process is complete, the processing unit 14 moves on to the shape generation process in S1203.

[0104] In S1203, the processing unit 14 outputs the 3D shape data after the correction process is performed. The output 3D shape data can be stored in the storage unit 15 or transferred to an external device such as a server via the communication unit 18.

[0105] In addition, the 3D shape data generated in S1201 has images of the pixel positions corresponding to each 3D vertex extracted from the RGB image and applied as a texture. Therefore, if vertices are added or deleted during the correction process in S1202, the texture for application to the 3D shape data will be re-extracted based on the pixel position information of each vertex shown by projecting the corrected 3D shape data in the depth direction.

[0106] As described above, the information processing device of this embodiment can generate 3D shape data that provides a suitable viewing experience. More specifically, the information processing device performs shape correction processing on the edges of the ridges that appear at the ends of the 3D shape data generated based on the depth image, in order to reduce the unevenness in the depth direction, thereby generating 3D shape data with smoothed ridge shapes.

[0107] [Embodiment 2] In the embodiments described above, a method was described in which, after generating 3D shape data based on a distance image, shape correction is performed according to the angles made by the vertices forming the edges in order to reduce the irregularities of the edges that appear at the edges of the 3D shape data. However, the method for correcting the irregularities of the edges is not limited to a method that derives the angles made by each vertex forming the edges of the 3D shape data.

[0108] The method shown in Embodiment 1, which derives the angle formed by each vertex forming a ridge and repeatedly performs shape correction processing until the angle meets the acceptable limit, can result in excessive computation depending on the accuracy of the distance value derivation. For example, if there are many vertices with a large degree of unevenness on the ridge, iterative processing may be performed until the angles formed by these vertices meet the acceptable conditions. Furthermore, for each vertex of the 3D shape data, not only is there three types of information as 3D coordinates (x, y, and z coordinates), but also information such as which vertices it is adjacent to on the ridge and which vertices define a polygon mesh between it and those vertices. Consequently, the process of correcting the shape of 3D shape data involves a large amount of information to refer to, which can lead to increased memory access and increased computational scale.

[0109] Therefore, in this embodiment, in order to reduce the computational load in the correction process, the pixel values ​​(distance values) are adjusted at the distance image stage prior to the conversion from the distance image to 3D coordinates, thereby indirectly correcting the shape of the subsequently generated 3D shape data. That is, the digital camera 100 of this embodiment performs shape correction processing at both the distance image stage and the 3D shape data stage to reduce the unevenness of the edges of the ultimately generated 3D shape data.

[0110] Shape correction in distance images The following outline of the shape correction process performed in the digital camera 100 of this embodiment at the time of capture of the depth image (before conversion to 3D coordinates) will be described with reference to Figure 14. In the example in Figure 14, multiple (two) subjects are located at different depth positions within the imaging range of the digital camera 100, and the images of each subject are superimposed in the captured image (not shown). The two subjects are a first subject, which is the subject in the foreground and is distributed at a first distance from the digital camera 100, and a second subject, which is the subject in the background and is distributed at a second distance that is longer than the first distance. When viewed from the position of the digital camera 100, these subjects are arranged such that at least a part of the second subject is obscured by the first subject. That is, in the captured image, the image of the second subject is distributed adjacent to the image of the first subject. Furthermore, in the following examples, in order to facilitate understanding of the invention, the edge region image and the depth image (or RGB image) are both composed of 10 pixels × 10 pixels in size.

[0111] Figure 14(a) illustrates an edge region image obtained in this embodiment. In the figure, pixels with a pixel value of 1 are hatched, and these pixels indicate edge regions. In the embodiment illustrated in Figure 14(a), the edge region image represents the edge region relating to the first subject. Accordingly, the processing unit 14 classifies the distance image based on the edge region image into a region 1401 corresponding to a first distance where the first subject is distributed, as shown in Figure 14(b), and a region 1402 corresponding to a second distance where the second subject is distributed. Hereafter, the former region will be referred to as the first region 1401, and the latter region as the second region 1402. In the distance image of Figure 14(b), the hatched pixels represent the second region 1402, and the other pixels represent the first region 1401. As will be described in detail later, in the example of Figure 14(b), the first region 1401 includes pixels shown as polka dots and pixels shown in white, each having distance values ​​in different ranges as pixel values. In the example in Figure 14(b), the pixels shown with hatching, the pixels shown with polka dots, and the pixels shown in white each store distance values ​​in different ranges. The distance range of the distance values ​​is assumed to decrease in the order of the pixels shown with hatching, the pixels shown with polka dots, and the pixels shown in white, where the distance indicated by the median decreases.

[0112] Here, pixels in the depth image at the same pixel positions as pixels indicated as edge regions in the edge region image represent the boundary between the image of the first subject and the image of the second subject. Therefore, the processing unit 14, for example, derives the median distance value for each region (excluding the boundary) demarcated by the boundary, and classifies the region with the smaller median value and the boundary together as the first region 1401. The processing unit 14 also classifies the remaining region (the region with the larger median value) as the second region.

[0113] This classification allows us to identify the pixels in the depth image that should be converted to 3D coordinates when generating 3D shape data for the first subject, and the pixels in the depth image that should be converted to 3D coordinates when generating 3D shape data for the second subject.

[0114] On the other hand, as mentioned above, the method of repeatedly deriving the angles made by each vertex that forms the edge line appearing at the edge after conversion to 3D shape data can be computationally intensive. For this reason, the processing unit 14 performs adjustment processing to reduce the variation in distance values ​​for pixels (hereinafter referred to as edge pixels) that are converted into vertices that form the edge line appearing at the edge when generating 3D shape data for each region classified with respect to the distance image. The edge pixels are arranged around the outer perimeter of each region. Here, the edge pixels of the first region include pixels at the same position as the pixel position indicated as an edge region in the edge region image. The edge pixels of the second region include pixels that have a pixel position adjacent to the pixel position indicated as an edge region in the edge region image and are located within the second region.

[0115] The adjustment of the pixel value (distance value) of a single edge pixel (target pixel) is performed, for example, by reading a predetermined number of pixels arranged around the target pixel on the outer perimeter of each region, such that the adjacency relationship between edge pixels on the outer perimeter is maintained, and referring to the pixel value of that sequence. That is, the pixel value of the target pixel is adjusted based on the pixel values ​​of other edge pixels distributed in the vicinity of the target pixel. The processing unit 14 derives the pixel value of the target pixel based on statistical values ​​of distance values ​​relating to a predetermined number of pixels distributed in the vicinity of the target pixel on the outer perimeter. The statistical values ​​can include, for example, the mean, median, weighted mean, etc. Here, the weights used when deriving the weighted mean may be set so that they become smaller the further away from the target pixel in the pixel sequence and larger the closer to the target pixel. As a result, the adjustment process reduces the variation in pixel values ​​(distance values) in the vicinity of the target pixel, so that the irregularities appearing on the edges of the edges of the 3D shape data generated based on the pixel group of each region can be reduced to some extent before the correction process is performed.

[0116] For example, as shown in white in the first region 1401 in Figure 14(b), variations in distance values ​​may occur at the boundary between the first region 1401 and the second region 1402. However, by performing an adjustment process, these regions can be made to a state where the variations in distance values ​​are eliminated, as shown in Figure 14(c).

[0117] Shape generation process The shape generation process for generating 3D shape data in the processing unit 14 of this embodiment will be explained using the flowchart in Figure 15. The process corresponding to the flowchart can be executed by the processing unit 14 by the control unit 12, for example, reading the corresponding processing program stored in the ROM 21, loading it into the RAM 20, and executing it. This shape generation process will be explained as being executed, for example, when an operation input related to the generation of 3D shape data of an object is made. It should be assumed that the distance image generation process and edge region image generation described above have been completed prior to this shape generation process. In the explanation of the shape generation process of this embodiment, steps that perform the same processing as the shape generation process of Embodiment 1 described above will be given the same reference number and their explanation will be omitted, and only steps that perform processing specific to this embodiment will be explained below.

[0118] In S1501, the processing unit 14 performs adjustment processing on the distance image and then moves the processing to S1201. As described above, the adjustment processing includes region classification based on the edge region image, identification of edge pixels, and smoothing of the pixel values ​​of the edge pixels.

[0119] In this way, the information processing device of this embodiment can generate 3D shape data that provides a suitable viewing experience while reducing the amount of computation.

[0120] [Example 1] In the embodiments described above, a method was explained in which a distance image is classified into regions for each subject based on information of the edge region defined by the edge region drawing. On the other hand, in the edge region image generation process, the edges of the subject may not be detected in a suitable shape within the xy plane (the plane of the captured two-dimensional image) due to, for example, segmentation region detection errors, errors caused by the use of RGB images, and the effects of image blur. For example, even in the distance image after changing the pixel values ​​illustrated in Figure 14(c), a small region (pixel group) 1403 of the second region 1402 is distributed in part within the horizontal range where the first region 1401 is distributed. In other words, the first region 1401 has a concave shape, and the pixels of the distance image are classified with the convex portion (small region 1403) of the second region 1402 protruding from the concave portion.

[0121] In this modified example, the irregularities of the edge shape in the xy plane are assumed to be due to false edge detection or detection accuracy, and the shape of each region defined in the distance image is corrected before generating the 3D shape data. For example, if a sub-region 1403 of the second region extends into the first region 1401 as shown in Figure 14(c), the region shape can be corrected as shown in Figure 16(a) by deleting the distance value of the sub-region 1403 of the second region 1402 and filling (interpolating) it with the distance value of the first region 1401. The distance value used for filling can be, for example, the statistical value of the pixel values ​​of pixels in the first region 1401 distributed in the vicinity of the sub-region 1403. The statistical value can be the mean, median, weighted mean, etc.

[0122] In the embodiment shown in Figure 16(a), pixels classified as belonging to the second region 1402 based on edge region information were deleted from the second region 1402 and assigned to the first region 1401. On the other hand, there are cases where the irregularities of the edge shape in the xy plane are not due to false edge detection. That is, if the detected irregularities of the edge shape are significant, such as when the subject actually has similar irregularities, deleting pixels classified as belonging to the second region 1402 and reducing the region will result in the 3D shape data relating to the second subject not being a suitable shape. Therefore, for irregularities of the edge shape in the xy plane, such as in the small region 1403, different corrections may be applied to the region where concave parts exist (first region 1401) and the region where convex parts corresponding to concave parts exist (second region 1402).

[0123] For example, as shown in Figure 16(b), the information of the regions in the xy-plane where shape correction has been performed may be managed independently for each of the first region 1401 and the second region 1402. In the example in Figure 16(b), the pixels shown in black are pixels in which no information is included for each region. In the example in the figure, the first region 1401 is corrected to the same shape as in Figure 16(a), but the second region 1402 is corrected in such a way that the shape of the part of the sub-region 1403 is smoothed in the two-dimensional image plane (sub-region 1601). Therefore, in this embodiment, for the pixels at the position of the sub-region 1601 remaining after smoothing the sub-region 1403 of the second region 1402, distance value information is managed redundantly for the two types of regions related to the distance image.

[0124] In other words, if, for multiple regions defined by the classification of the depth image, a recess (small region) smaller than a predetermined size exists in the xy plane due to being classified into another region, the processing unit 14 corrects the two-dimensional shape of the region. At this time, for the region where the recess exists, the processing unit 14 corrects the shape in the xy plane by filling in the recess based on the pixel values ​​of the surrounding pixels of that region, so that the recess becomes a pixel of that region. Conversely, for the region where a convex portion corresponding to the recess exists, the processing unit 14 corrects the shape in the xy plane by smoothing the shape of the convex portion. In this way, the processing unit 14 controls the processing unit 14 to independently manage multiple types of distance values ​​for some pixels of the depth image.

[0125] In this embodiment, Figure 14 illustrates an example in which the boundary between the images of two subjects is identified by an edge region image, but it should be understood that the implementation of the present invention is not limited to this. For example, the boundary between the images of three or more subjects may be defined in the edge region image, and in this embodiment, if the processing unit 14 performs shape correction of the region in the xy plane, the same number of distance values ​​can be independently managed for some of the pixels.

[0126] [Differentiation 2] In the embodiments and modifications described above, the adjustment of distance values ​​in the depth image was explained using, for example, statistical values ​​of distance values ​​of pixels distributed in the vicinity of the target pixel. However, the implementation of the present invention is not limited to this. For example, the color of the pixel position of the target pixel may be referenced in an RGB image, and the pixel positions where similar colors are distributed in the RGB image may be identified. Then, the distance values ​​of the pixels at the same pixel positions in the depth image may be used to derive the statistical values.

[0127] Furthermore, statistical values ​​are not limited to the mean, median, weighted mean, etc. For example, if a slope (e.g., a linear change in distance values) appears in the distribution of distance values ​​in the vicinity of the target pixel, the statistical values ​​may be derived by further referring to the information of that slope.

[0128] [Difference 3] In the embodiments and modifications described above, an embodiment was described in which, after generating 3D shape data based on a distance image, a correction process is performed to reduce the depth-direction irregularities of the ridges while deriving the angles formed by each vertex that form the ridges. However, the implementation of the present invention is not limited to this. For example, as described in Embodiment 2, depth-direction irregularities can also be reduced by identifying pixels that become vertices (end pixels) that form ridges based on edge region images in the state of the distance image, and adjusting the pixel values ​​of those pixels based on the pixel values ​​of other end pixels. That is, even by simply performing the adjustment process in the state of the distance image, depth-direction irregularities of the ridges that appear at their ends can be reduced in the 3D shape data generated based on the adjusted distance image. Therefore, even in a configuration that does not perform the shape generation process S1203, it is possible to generate 3D shape data that provides a suitable viewing experience.

[0129] [Differentiation Example 4] The embodiments and modifications described above describe a method for generating 3D shape data in a digital camera 100, which is an imaging device capable of acquiring RGB images and depth images. However, it goes without saying that the present invention is not limited to this. That is, the present invention, which reduces the depth-direction irregularities of the ridges appearing at the edges of 3D shape data, may be implemented in any information processing device, not limited to an imaging device, as long as it is possible to acquire depth images and information identifying edge regions.

[0130] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0131] [Summary of Embodiments and Modifications] The disclosures herein include the following information processing devices, imaging devices, control methods, and programs. (Item 1) An information processing device having generation means for generating three-dimensional shape data of an object based on a distance image, A means for identifying edge regions in the distance image where edges relating to the subject are distributed, A determination means for determining whether or not there are irregularities that do not satisfy the tolerance conditions in the edge line corresponding to the edge region of the three-dimensional shape data generated by the generation means, If the determination means determines that there are irregularities that do not satisfy the tolerance conditions, the correction means executes a process related to correcting the shape of the ridge line, has An information processing device characterized by the following: (Item 2) The process for correcting the shape of the ridge line includes a process for increasing the angle between two sides connected via a vertex included in the ridge line of the 3D shape data. The determination means determines that if the angle is below a threshold, there are irregularities that do not satisfy the tolerance conditions. The information processing device described in item 1, characterized by the features described herein. (Item 3) The process of increasing the angle includes changing the depth coordinate value of the target vertex corresponding to the angle based on the depth coordinate values ​​of other vertices distributed in the vicinity of the target vertex along the ridge. The information processing device described in item 2, characterized by the features described herein. (Item 4) The process of changing the coordinate values ​​in the depth direction of the target vertex involves changing the coordinate values ​​in the depth direction of the target vertex to the statistical values ​​of the coordinate values ​​in the depth direction of the other vertices. The information processing device described in item 3, characterized by the features described herein. (Item 5) The process of increasing the angle includes adding a new vertex adjacent to the vertex corresponding to the angle and the edge. An information processing device according to any one of items 2 to 4, characterized in that it is an information processing device. (Item 6) The process of increasing the angle includes the process of removing the vertex corresponding to the angle from the edge. An information processing device according to any one of items 2 to 5, characterized in that it is an information processing device. (Item 7) The process relating to the correction of the shape of the ridge line includes a process for adjusting the pixel values ​​of the distance image, The generation means generates three-dimensional shape data of the subject based on the distance image after adjusting the pixel values. An information processing device according to any one of items 2 to 6, characterized in that it is an information processing device. (Item 8) The process relating to the correction of the shape of the ridge line includes a process for adjusting the pixel values ​​of the distance image, The generation means generates three-dimensional shape data of the subject based on the distance image after adjusting the pixel values. The information processing device described in item 1, characterized by the features described herein. (Item 9) The process of adjusting the pixel values ​​of the distance image includes adjusting the pixel values ​​of target pixels included in the edge region of the distance image based on the pixel values ​​of other pixels distributed in the vicinity of the target pixel and included in the edge region. An information processing device according to item 7 or 8, characterized by the features described therein. (Item 10) The process of adjusting the pixel value of the target pixel involves changing the pixel value of the target pixel to a statistical value of the pixel values ​​of the other pixels. The information processing device according to item 9, characterized in that it is a processing device. (Item 11) The other pixels are pixels at pixel locations where colors similar to the color of the target pixel are distributed in the captured image of the subject corresponding to the distance image. An information processing device according to item 9 or 10, characterized in that it is an information processing device. (Item 12) The subject includes a first subject distributed at a first distance and a second subject distributed at a second distance longer than the first distance. The edge region corresponds to the boundary between the image of the first subject and the image of the second subject in the distance image, The process of adjusting the pixel values ​​of the aforementioned distance image is: A process for classifying the distance image into a first region corresponding to the first distance and a second region corresponding to the second distance, A process of adjusting the pixel value of a first target pixel included in the edge region of the first region based on the pixel values ​​of other pixels distributed in the vicinity of the first target pixel, which are included in the edge region of the first region. A process of adjusting the pixel value of a second target pixel included in the edge region of the second region based on the pixel values ​​of other pixels distributed in the vicinity of the second target pixel, which are included in the edge region of the second region, including An information processing device according to item 7 or 8, characterized by the features described therein. (Item 13) The process of adjusting the pixel values ​​of the distance image further includes a process of correcting the two-dimensional shapes of the first and second regions after the adjustment of the pixel values. The information processing device described in item 12, characterized by the features described herein. (Item 14) The process for correcting the two-dimensional shape is performed when a recess of a two-dimensional shape smaller than a predetermined size classified in the other region exists in one of the first region and the second region. For one of the aforementioned regions, the pixels of the recess are interpolated based on the pixels of that region. With respect to the other region, the shape of the protrusion corresponding to the recess is smoothed. The information processing device further includes management means for independently managing pixels relating to the recesses in the first and second regions after the process for correcting the two-dimensional shape. The information processing device described in item 13, characterized by the features described herein. (Item 15) The system further includes an extraction means for extracting a texture to be applied to the three-dimensional shape data from the captured image of the subject, When a process for correcting the two-dimensional shape is performed, the extraction means extracts textures to be applied to the three-dimensional shape data of the first subject and the three-dimensional shape data of the second subject, based on the corrected two-dimensional shapes of the first and second regions. An information processing device according to item 13 or 14, characterized by the features described herein. (Item 16) An imaging device comprising: an imaging means; an acquisition means for acquiring a distance image corresponding to a subject; and a generation means for generating three-dimensional shape data of the subject based on the distance image, A means for identifying edge regions in the distance image where edges relating to the subject are distributed, A determination means for determining whether or not there are irregularities that do not satisfy the tolerance conditions in the edge line corresponding to the edge region of the three-dimensional shape data generated by the generation means, If the determination means determines that there are irregularities that do not satisfy the tolerance conditions, the correction means executes a process related to correcting the shape of the ridge line, has An imaging device characterized by the following features. (Item 17) The imaging means has an image sensor capable of separating and imaging light beams that have passed through different pupil regions of the imaging optical system. The acquisition means acquires the distance image generated based on a pair of captured images relating to light beams that have passed through the different pupil regions. The imaging device described in item 16, characterized by the features described herein. (Item 18) A control method for an information processing device having a generation means for generating three-dimensional shape data of an object based on a distance image, A process of identifying the edge region in the distance image in which the edges relating to the subject are distributed, A determination step of determining whether or not there are irregularities that do not satisfy the tolerance conditions in the edge line corresponding to the edge region of the three-dimensional shape data generated by the generation means, If the determination step determines that there are irregularities that do not meet the tolerance conditions, a correction step is performed to carry out a process related to correcting the shape of the ridge line. has A control method characterized by the following: (Item 19) A program to cause a computer to function as one of the means of an information processing device described in any one of items 1 through 15.

[0132] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0133] 100: Digital camera, 11: Image sensor, 12: Control unit, 14: Processing unit

Claims

1. An information processing device having a generation means for generating three-dimensional shape data of an object based on a distance image, A means for identifying edge regions in the distance image where edges relating to the subject are distributed, A determination means for determining whether or not there are irregularities that do not satisfy the tolerance conditions in the edge line corresponding to the edge region of the three-dimensional shape data generated by the generation means, If the determination means determines that there are irregularities that do not satisfy the tolerance conditions, the correction means executes a process related to correcting the shape of the ridge line, has An information processing device characterized by the following:

2. The process for correcting the shape of the ridge line includes a process for increasing the angle between two sides connected via a vertex included in the ridge line of the three-dimensional shape data. The determination means determines that if the angle is below a threshold, there are irregularities that do not satisfy the tolerance conditions. The information processing apparatus according to feature 1.

3. The process of increasing the angle includes changing the depth coordinate value of the target vertex corresponding to the angle based on the depth coordinate values ​​of other vertices distributed in the vicinity of the target vertex along the ridge. The information processing apparatus according to feature 2.

4. The process of changing the coordinate values ​​in the depth direction of the target vertex involves changing the coordinate values ​​in the depth direction of the target vertex to the statistical values ​​of the coordinate values ​​in the depth direction of the other vertices. The information processing apparatus according to feature 3.

5. The process of increasing the angle includes adding a new vertex adjacent to the vertex corresponding to the angle and the edge. The information processing apparatus according to feature 2.

6. The process of increasing the angle includes the process of removing the vertex corresponding to the angle from the edge. The information processing apparatus according to feature 2.

7. The process relating to the correction of the shape of the ridge line includes a process for adjusting the pixel values ​​of the distance image, The generation means generates three-dimensional shape data of the subject based on the distance image after adjusting the pixel values. The information processing apparatus according to feature 2.

8. The process relating to the correction of the shape of the ridge line includes a process for adjusting the pixel values ​​of the distance image, The generation means generates three-dimensional shape data of the subject based on the distance image after adjusting the pixel values. The information processing apparatus according to feature 1.

9. The process of adjusting the pixel values ​​of the distance image includes adjusting the pixel values ​​of target pixels included in the edge region of the distance image based on the pixel values ​​of other pixels distributed in the vicinity of the target pixel and included in the edge region. The information processing apparatus according to feature 7.

10. The process of adjusting the pixel value of the target pixel involves changing the pixel value of the target pixel to a statistical value of the pixel values ​​of the other pixels. The information processing apparatus according to feature 9.

11. The other pixels are pixels at pixel locations where colors similar to the color of the target pixel are distributed in the captured image of the subject corresponding to the distance image. The information processing apparatus according to feature 9.

12. The subject includes a first subject distributed at a first distance and a second subject distributed at a second distance longer than the first distance. The edge region corresponds to the boundary between the image of the first subject and the image of the second subject in the distance image, The process of adjusting the pixel values ​​of the aforementioned distance image is as follows: A process of classifying the distance image into a first region corresponding to the first distance and a second region corresponding to the second distance, A process of adjusting the pixel value of a first target pixel included in the edge region of the first region based on the pixel values ​​of other pixels distributed in the vicinity of the first target pixel, which are included in the edge region of the first region. A process of adjusting the pixel value of a second target pixel included in the edge region of the second region based on the pixel values ​​of other pixels distributed in the vicinity of the second target pixel, which are included in the edge region of the second region, including The information processing apparatus according to feature 7.

13. The process of adjusting the pixel values ​​of the distance image further includes a process of correcting the two-dimensional shapes of the first and second regions after the adjustment of the pixel values. The information processing apparatus according to feature 12.

14. The process for correcting the two-dimensional shape is performed when a recess of a two-dimensional shape smaller than a predetermined size classified in the other region exists in one of the first region and the second region. For one of the aforementioned regions, the pixels of the recess are interpolated based on the pixels of that region. With respect to the other region, the shape of the protrusion corresponding to the recess is smoothed. The information processing device further includes management means for independently managing pixels relating to the recesses in the first region and the second region after the process for correcting the two-dimensional shape. The information processing apparatus according to feature 13.

15. The system further includes an extraction means for extracting a texture to be applied to the three-dimensional shape data from the captured image of the subject, When a process for correcting the two-dimensional shape is performed, the extraction means extracts textures to be applied to the three-dimensional shape data of the first subject and the three-dimensional shape data of the second subject based on the corrected two-dimensional shapes of the first and second regions. The information processing apparatus according to feature 13.

16. An imaging device comprising: an imaging means; an acquisition means for acquiring a distance image corresponding to a subject; and a generation means for generating three-dimensional shape data of the subject based on the distance image, A means for identifying edge regions in the distance image where edges relating to the subject are distributed, A determination means for determining whether or not there are irregularities that do not satisfy the tolerance conditions in the edge line corresponding to the edge region of the three-dimensional shape data generated by the generation means, If the determination means determines that there are irregularities that do not satisfy the tolerance conditions, the correction means executes a process related to correcting the shape of the ridge line, has An imaging device characterized by the following features.

17. The imaging means has an image sensor capable of separating and imaging light beams that have passed through different pupil regions of the imaging optical system. The acquisition means acquires the distance image generated based on a pair of captured images relating to light beams that have passed through the different pupil regions. The imaging device according to feature 16.

18. A control method for an information processing device having a generation means for generating three-dimensional shape data of an object based on a distance image, A process of identifying the edge region in the distance image in which the edges relating to the subject are distributed, A determination step of determining whether or not there are irregularities that do not satisfy the tolerance conditions in the ridge line corresponding to the edge region of the three-dimensional shape data generated by the generation means, If the determination step determines that there are irregularities that do not meet the tolerance conditions, a correction step is performed to carry out a process related to correcting the shape of the ridge line. has A control method characterized by the following:

19. A program for causing a computer to function as one of the means of an information processing apparatus described in any one of claims 1 to 15.

Images