Image processing apparatus and image processing method

Abstract

To provide an image processing apparatus and an image processing method that can appropriately reduce the amount of data of three-dimensional moving images by using the correlation between frames.SOLUTION: An image processing apparatus has: acquisition means that acquires three-dimensional moving image data having three-dimensional data and texture information for every frame; and encoding means that encodes the three-dimensional moving image data by using inter-frame prediction. The encoding means encodes the three-dimensional data and the texture information separately, and selects a key frame for encoding the three-dimensional data and a key frame for encoding the texture information separately.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and an image processing method, and particularly to a technique for reducing data volume.

Background Art

[0002] There is known a photographing apparatus capable of photographing two-dimensional (2D) videos and three-dimensional (3D) videos (Patent Document 1). In Patent Document 1, the data volume is reduced by encoding a 3D video by a method compliant with the MPEG2 standard.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to efficiently reduce the data volume of a 3D video while suppressing image quality degradation by using encoding that utilizes correlation between frames, such as the MPEG standard, it is necessary to appropriately set a reference frame (key frame). In Patent Document 1, there is a disclosure of performing exposure control at the timing of an I frame, but there is no mention of how to set the I frame.

[0005] In one aspect of the present invention, there is provided an image processing apparatus and an image processing method capable of appropriately reducing the data volume of a 3D video by utilizing correlation between frames.

Means for Solving the Problems

[0006] An image processing apparatus according to one aspect of the present invention comprises an acquisition means for acquiring 3D video data having 3D data and texture information for each frame, and an encoding means for encoding the 3D video data using inter-frame prediction, wherein the encoding means separately encodes the 3D data and texture information, and separately selects keyframes for encoding the 3D data and keyframes for encoding the texture information. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an image processing apparatus and an image processing method that can appropriately reduce the amount of data in 3D video by utilizing the correlation between frames. [Brief explanation of the drawing]

[0008] [Figure 1] Block diagram showing an example of the functional configuration of a digital camera as an example of an image processing apparatus according to the embodiment. [Figure 2] Diagram showing an example of the image sensor configuration. [Figure 3] Diagram to explain on-sensor phase-detection autofocus [Figure 4] Flowchart for the defocus map generation process in the embodiment [Figure 5] A diagram illustrating how to obtain distance information from the amount of defocus. [Figure 6] A diagram illustrating the data related to a three-dimensional object generated in this embodiment. [Figure 7] Flowchart relating to the compression process of 3D video data in the first embodiment [Figure 8] A diagram illustrating the keyframe evaluation method in the first embodiment. [Figure 9] Flowchart relating to the compression process of 3D video data in the second embodiment [Figure 10] A diagram illustrating the keyframe evaluation method in the second embodiment. [Modes for carrying out the invention]

[0009] ●(First Embodiment) The present invention will be described in detail below with reference to the attached drawings, based on exemplary embodiments thereof. Note that the following embodiments do not limit the invention to the claims. Furthermore, while multiple features are described in the embodiments, not all of them are essential to the invention, and the multiple features may be combined arbitrarily. In addition, in the attached drawings, the same or similar configurations are given the same reference numeral, and redundant descriptions are omitted.

[0010] In the following embodiments, the present invention will be described in relation to its implementation using a digital camera. However, imaging functionality is not essential to the present invention, and it can be implemented using any electronic device capable of handling image data. Such electronic devices include video cameras, computer equipment (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and the like. These are examples, and the present invention can be implemented using other electronic devices as well.

[0011] <Image Information> Figure 1 is a block diagram showing an example of the functional configuration of a digital camera 100 as an image processing device according to the embodiment. The imaging optical system 10 forms an optical image of the subject on the imaging surface of the image sensor 11. The imaging optical system 10 has a plurality of lenses arranged along the optical axis 103. The plurality of lenses include a focus lens 102 for adjusting the focusing distance of the imaging optical system 10. The focus lens 102 is movable along the optical axis. The focus lens 102 is driven by the control unit 12 according to the amount of defocus generated by the image processing unit 14.

[0012] The imaging optical system 10 also has an aperture 104 whose aperture value (aperture amount) can be adjusted. The aperture value of the aperture 104 is controlled by the control unit 12 based on, for example, shooting conditions determined by automatic exposure control (AE). The aperture 104 may have the function of a mechanical shutter. The exit pupil 101 is the image of the open aperture when the imaging optical system 10 is viewed from the side of the imaging device 11, and the position of the exit pupil 101 is shown in the figure.

[0013] The imaging device 11 may be, for example, a known CCD or CMOS color image sensor having a color filter in a primary color Bayer array. The imaging device 11 has a pixel array in which a plurality of pixels are two-dimensionally arranged and a peripheral circuit for reading signals from each pixel. Each pixel accumulates charges corresponding to the amount of incident light by photoelectric conversion. By reading out from each pixel a signal having a voltage corresponding to the amount of charges accumulated during the exposure period, a group of pixel signals (analog image signals) representing the subject image formed by the imaging optical system 10 on the imaging surface is obtained.

[0014] As will be described later, the pixels of the imaging device 11 have a plurality of photoelectric conversion regions or photoelectric conversion elements and can generate a pair of parallax images in one shooting. Then, based on this pair of parallax images, automatic focus detection (phase difference AF) using the phase difference detection method can be executed or distance information can be generated. Details will be described later.

[0015] The control unit 12 has one or more processors (hereinafter referred to as CPUs) capable of executing programs. The control unit 12 reads, for example, a program stored in the ROM 21 into the RAM 20 and executes it with the CPU. The control unit 12 realizes various functions of the digital camera 100 by controlling the operations of each functional block while executing the program.

[0016] The ROM 21 is, for example, a rewritable non-volatile memory, which stores programs, set values, GUI data, etc. that can be executed by the CPU of the control unit 12. The RAM 20 is used to load the programs executed by the CPU of the control unit 12 and to store the values necessary during the execution of the programs. Also, the RAM 20 is used as a working memory for the image processing unit 14, a buffer memory for temporarily storing the images obtained by imaging, a video memory for the display unit 17, etc.

[0017] The image processing unit 14 applies predetermined image processing to the analog image signal read from the imaging device 11, generates signals and image data according to the application, and acquires and / or generates various kinds of information. The image processing unit 14 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the image processing unit 14 may have a configuration in which a processor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) realizes a specific function by executing software. The image processing unit 14 outputs the acquired or generated information and data to the control unit 12, the RAM 20, etc. according to the application.

[0018] The image processing applied by the image processing unit 14 may include, for example, preprocessing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, etc. The preprocessing may include A / D conversion, signal amplification, reference level adjustment, defective pixel correction, etc. The color interpolation processing is performed when a color filter is provided in the imaging device, and is a process of interpolating the values of color components not included in the individual pixel data constituting the image data. The color interpolation processing is also called demosaicing processing. The correction processing may include processes such as white balance adjustment, tone correction, correction of image degradation (image restoration) caused by optical aberration of the imaging optical system 10, correction of the influence of peripheral light reduction of the imaging optical system 10, color correction, etc. Detection processes may include detecting feature regions (such as face regions or human body regions) and their movements, as well as recognizing people. The evaluation value calculation process may include processes such as generating signals and evaluation values ​​used for autofocus detection (AF), and generating evaluation values ​​used for automatic exposure control (AE). In Figure 1, the function of the image processing unit 14 that generates the defocus amount, which is an evaluation value for AF, is shown as a functional block (defocus generation unit 141) for convenience. Data processing may include processes such as region extraction (trimming), merging, scaling, encoding and decoding, and header information generation (data file generation). The generation of display image data and recording image data is also included in data processing. Furthermore, the generation of distance information based on the amount of defocus is also performed as data processing. Special effects processing may include adding blur effects, changing color tones, and relighting. These are merely examples of processes that the image processing unit 14 can apply, and do not limit the processes that the image processing unit 14 can apply.

[0019] The storage unit 15 is a recording medium for recording data files containing image data obtained by imaging. The storage unit 15 may be, for example, a combination of a memory card and its reader / writer. The storage unit 15 may be capable of handling multiple recording media.

[0020] The input unit 16 is a general term for user-operable input devices provided on the digital camera 100, such as dials, buttons, switches, and touch panels. The control unit 12 monitors operations on the input unit 16. When an operation on the input unit 16 is detected, the control unit 12 executes an action according to the function and operation content assigned to the operated input device.

[0021] The display unit 17 is, for example, a display device such as a liquid crystal display or an organic EL display. By continuously capturing video and displaying the captured video on the display unit 17, the display unit 17 can be made to function as an electronic viewfinder (EVF). The operation of making the display unit 17 function as an electronic viewfinder (EVF) is sometimes called live view display or through display. The image displayed on the display unit 17 by live view display or through display is sometimes called a live view image or through image.

[0022] The display unit 17 may be a touch display. If the display unit 17 is a touch display, software keys may be implemented by combining GUI parts displayed on the display unit 17 with a touch panel. The control unit 12 handles the software keys in the same way as the input devices of the input unit 16.

[0023] The communication unit 18 is a communication interface with an external device. The control unit 12 can communicate with an external device through the communication unit 18 using one or more wired or wireless communication standards.

[0024] The motion sensor 19 generates a signal corresponding to the movement of the digital camera 100. The motion sensor 19 may be a combination of, for example, an accelerometer that outputs a signal corresponding to movement in each of the XYZ axes and a gyroscope that outputs a signal corresponding to movement around each axis.

[0025] <Example of image sensor configuration> An example of the configuration of the image sensor 11 will be explained with reference to Figure 2. Figure 2(a) is a plan view of the pixel array of the image sensor 11 as seen from the imaging surface side. The pixel array is provided with primary color Bayer array color filters. Therefore, each pixel has one of the following color filters: red (R), green (G), or blue (B), arranged regularly in a 2x2 pixel group 210 as a repeating unit. Note that color filters with arrays other than the primary color Bayer array may also be provided.

[0026] Figure 2(b) is a vertical cross-sectional view of a single pixel. This corresponds to the configuration of the I-I' cross-section in Figure 2(a). Each pixel has a light guide layer 213 and a light receiving layer 214. The light guide layer 213 has a microlens 211 and a color filter 212. The light receiving layer 214 has a first photoelectric conversion unit 215 and a second photoelectric conversion unit 216.

[0027] The microlens 211 is configured to efficiently guide the light beam incident on the pixel to the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216. The color filter 212 is either an R filter, a G filter, or a B filter.

[0028] Both the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 generate an electric charge corresponding to the amount of incident light. The image sensor 11 can selectively read signals from individual pixels from one or both of the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216. In this specification, the signal obtained from the first photoelectric conversion unit 215 may be referred to as the A signal, the signal obtained from the second photoelectric conversion unit 216 as the B signal, and the signal obtained from both the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 may be referred to as the A+B signal.

[0029] The first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 view the exit pupil 101 from different viewpoints. Therefore, the image consisting of signal A and the image consisting of signal B, read from the same pixel area, form a disparity image pair. Thus, by using signals A and B, the amount of defocus can be determined according to the principle of phase-difference autofocus. Therefore, signals A and B can be considered focus detection signals.

[0030] On the other hand, the A+B signal corresponds to the signal obtained when a pixel has one photoelectric converter; therefore, by acquiring the A+B signal from each pixel, an analog image signal can be obtained.

[0031] The A signal can also be obtained by subtracting the B signal from the A+B signal. Similarly, the B signal can be obtained by subtracting the A signal from the A+B signal. Therefore, by reading out the A+B signal and either the A signal or the B signal from each pixel, the A signal, B signal, and A+B signal can be obtained. The type of signal read out from each pixel is controlled by the control unit 12.

[0032] In Figure 2, a configuration is shown in which each pixel has two photoelectric conversion units 215 and 216 arranged horizontally. However, a configuration with four photoelectric conversion units, two arranged horizontally and two arranged vertically, is also possible. Furthermore, a configuration in which multiple pairs of pixels, one dedicated to generating the A signal and the other dedicated to generating the B signal, are distributed in the pixel array is also possible. The image sensor 11 can have any known configuration that corresponds to image plane phase-detection autofocus.

[0033] <Principle of on-sensor phase-detection autofocus> The principle by which the amount of defocus can be calculated using signals A and B will be explained with reference to Figures 3(a) to (e). Figure 3(a) is a schematic diagram showing the relationship between the exit pupil 101 of the imaging optical system 10 and the light beam incident on the first photoelectric conversion unit 215 of a certain pixel. Figure 3(b) is a schematic diagram showing the relationship between the light beam incident on the second photoelectric conversion unit 216 of the same pixel and the exit pupil 101.

[0034] In this specification, the direction parallel to the optical axis of the imaging optical system is defined as the z-direction or defocus direction, the direction perpendicular to the optical axis and parallel to the horizontal direction of the imaging plane is defined as the x-direction, and the direction perpendicular to the optical axis and parallel to the vertical direction of the imaging plane is defined as the y-direction.

[0035] The microlens 211 is positioned such that it is optically conjugate to the exit pupil 101 and the light-receiving layer 214. The light beam that passes through the exit pupil 101 of the imaging optical system 10 is focused by the microlens 211 and incident on the first photoelectric conversion unit 215 or the second photoelectric conversion unit 216. At this time, as shown in Figures 3(a) and 3(b), the light beam that has passed through different regions of the exit pupil 101 mainly incidents on the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216, respectively. Specifically, the light beam that has passed through the first pupil region 510 is incident on the first photoelectric conversion unit 215, and the light beam that has passed through the second pupil region 520 is incident on the second photoelectric conversion unit 216.

[0036] A signal and B signal are acquired from each of multiple pixels arranged horizontally around the pixel of interest. In this case, the relative positional shift (phase difference or disparity) between the image signal based on the A signal sequence (A image) and the image signal based on the B signal sequence (B image) is proportional to the amount of defocus of the pixel of interest.

[0037] In Figures 3(c) to 3(e), 511 represents the first luminous beam passing through the first pupil region 510, and 521 represents the second luminous beam passing through the second pupil region 520. Figure 3(c) shows the focused state, where the first light beam 511 and the second light beam 521 converge on the imaging plane. At this time, the phase difference or parallax between image A and image B is 0. Figure 3(d) shows that the first luminous beam 511 and the second luminous beam 521 converge on the object side (negative z-axis side) of the imaging plane. In this case, the phase difference or parallax between image A and image B has a negative value (<0). Figure 3(e) shows that the first luminous beam 511 and the second luminous beam 521 converge behind the imaging plane (towards the positive z-axis) when viewed from the object side. At this time, the phase difference or parallax between image A and image B has a positive value (>0).

[0038] Thus, the phase difference or parallax amount between image A and image B has a sign corresponding to the relationship between the position where the first light beam 511 and the second light beam 521 converge and the imaging plane, and has a magnitude corresponding to the amount of defocus. By calculating the correlation amount while relatively shifting image A and image B, the phase difference or parallax amount between image A and image B can be obtained as the shift amount that maximizes the correlation amount.

[0039] <Defocused Image Generation Process> Next, an example of the process by which the defocus generation unit 141 of the image processing unit 14 generates a defocus map will be explained using the flowchart shown in Figure 4. A defocus map is two-dimensional data that represents the amount of defocus at each pixel position in the captured image.

[0040] Here, it is assumed that the A signal and B signal for each pixel of the image sensor 11 are stored in the RAM 20.

[0041] In S1401, the defocus generation unit 141 corrects the light intensity of the A signal and the B signal. In particular, for pixels with a large image height, the difference in shape between the first pupil region 510 and the second pupil region 520 becomes large due to the aperture vignetting of the imaging optical system 10, resulting in a difference in the magnitude of the A signal and the B signal. The defocus generation unit 141 applies a correction value corresponding to the pixel position to the A signal and the B signal to correct the difference in magnitude between the A signal and the B signal. The correction value can be stored in advance, for example, in the ROM 21.

[0042] In S1402, the defocus generation unit 141 applies noise reduction processing to the A signal and the B signal. Generally, the higher the spatial frequency, the more noise components there are relatively. Therefore, the defocus generation unit 141 applies a low-pass filter, whose pass-through rate decreases as the spatial frequency increases, to the A signal and the B signal. However, due to manufacturing errors in the imaging optical system 10, good results may not be obtained with the light intensity correction in S1401. For this reason, in S1402, the defocus generation unit 141 can apply a band-pass filter that blocks the DC component and has a low pass-through rate for high-frequency components.

[0043] In S1403, the defocus generation unit 141 detects the phase difference or disparity between the A signal and the B signal. The defocus generation unit 141 generates the A signal sequence and the B signal sequence from a series of horizontally continuous pixels including the pixel of interest. The defocus generation unit 141 then calculates the correlation amount while relatively shifting the A signal sequence and the B signal sequence. The correlation amount may be, for example, NCC (Normalized Cross-Correlation), SSD (Sum of Squared Difference), or SAD (Sum of Absolute Difference).

[0044] The defocus generation unit 141 determines the amount of shift that maximizes the correlation between the A signal train and the B signal train in units less than one pixel, and uses this as the phase difference or disparity amount at the pixel of interest. The defocus generation unit 141 detects the phase difference or disparity amount at each individual pixel position while changing the position of the pixel of interest. Note that the phase difference or disparity amount between the A signal and the B signal may be detected by any other known method. The resolution used to determine the phase difference or disparity amount may be lower than the resolution of the captured image.

[0045] In S1404, the defocus generation unit 141 converts the detected phase difference or parallax amount into a defocus amount. Since the detected phase difference or parallax amount has a magnitude corresponding to the defocus amount, it can be converted into a defocus amount by applying a predetermined conversion coefficient. If the phase difference or parallax amount is d and the conversion coefficient is K, the defocus amount ΔL can be obtained by the following equation (1). ΔL = K × d (1)

[0046] The defocus generation unit 141 generates two-dimensional information (defocus map) representing the amount of defocus corresponding to the pixel position by converting the detected phase difference or disparity amount into a defocus amount.

[0047] <Acquiring distance information> Next, we will explain how to obtain depth (distance) information based on the amount of defocus, using Figure 5. In Figure 5, OBJ represents the object plane, IMG represents the image plane, H is the front principal point, H' is the rear principal point, f is the focal length of the imaging optical system (lens), S is the distance from the object plane to the front principal point, and S' is the distance from the rear principal point to the image plane. Also, ΔS' is the amount of defocus, and ΔS is the relative distance to the object side corresponding to the amount of defocus. The dashed line is the optical axis, the dotted line is the imaging beam, and the dashed line is the defocus beam.

[0048] It is known that the following equation (2) holds true for lens imaging. 1 / S + 1 / S' = 1 / f (2) Furthermore, when defocusing, equation (3), which is a transformation of equation (2), holds true. 1 / (S+ΔS) + 1 / (S'+ΔS') = 1 / f (3)

[0049] The S and f values ​​at the time of focus can be obtained from the shooting conditions information (shooting information). Therefore, S' can be calculated from equation (1). In addition, the defocus amount ΔS' can be obtained by, for example, a phase-detection autofocus (AF) system. From this, ΔS can be calculated from equation (3), and the distance S to the object surface OBJ can be determined.

[0050] The image processing unit 14 can generate distance information for the subject using the generated defocus map and shooting information. The distance information may be, for example, two-dimensional data representing the subject distance corresponding to each pixel position, and may also be called a depth map, distance image, or depth image.

[0051] Here, distance information was obtained using the amount of defocus, but distance information may also be obtained using other known methods. For example, the subject distance can be obtained for each pixel by determining the focus lens position where the contrast evaluation value is maximized for each pixel. Alternatively, distance information for each pixel can be obtained from image data obtained by taking multiple shots of the same scene with different focusing distances, and the point image distribution function (PSF) of the optical system, based on the correlation between the amount of blur and the distance. These techniques are described, for example, in Japanese Patent Publication No. 2010-177741 and U.S. Patent No. 4,965,840. Furthermore, if a pair of disparity images can be obtained, the subject distance can be obtained for each pixel using methods such as stereo matching.

[0052] <Generating 3D data> Next, we will explain an example of a method for generating 3D data using distance information. First, 3D data is generated by converting distance information (depth map) into coordinate values ​​in the world coordinate system using the focal length and focus position obtained from the imaging information. The resulting 3D data is then polygonized to make it easier to handle as a 3D model. Polygonization can be performed using any known method.

[0053] For example, 3D data can be converted into a polygon mesh by defining a surface using the coordinate information of any three adjacent points in the 3D data. Furthermore, the texture information of the polygon can be calculated from the information of the captured image corresponding to the three points used for polygonization. Additionally, filtering can be applied to the depth map before conversion to world coordinate system coordinate values, or to the 3D data before polygonization. For example, small shape changes can be smoothed by applying a median filter.

[0054] When polygonization is performed, the image processing unit 14 converts the polygon data into two-dimensional structured data using any known method so that the amount of data can be reduced using predictive coding techniques for two-dimensional images. Note that polygonization is not mandatory, and 3D data may be handled in point cloud format. The method of representing the three-dimensional shape of an object is arbitrary, as long as the data format is one to which known predictive coding techniques for two-dimensional images can be applied.

[0055] Figures 6(A) to (C) show examples of 3D objects, their depth maps, and 3D data. When a cylinder is photographed from the side as a 3D object, as shown in Figure 6(A), and distance information is obtained, a depth map like the one shown in Figure 6(B) is obtained. Here, the shading in the depth map in Figure 6(B) indicates that the lighter the color, the greater the distance (farther). In other words, the center of the cylinder is the closest in the captured image, and the distance increases as you move away from the center to the left and right. Figure 6(C) schematically shows the 3D data obtained by converting the depth map plotted in a world coordinate system. Since a depth map is not generated for parts of the 3D object that are not photographed, the 3D data is generated only for the parts corresponding to the depth map. Although not shown, texture information (RGB data) is mapped to the 3D data.

[0056] <Relationship between shooting conditions, accuracy of distance information, and quality of texture information> When acquiring distance information from captured images, shooting conditions can affect the accuracy of the distance information. For example, when acquiring distance information based on a pair of parallax images captured using an image sensor compatible with on-sensor phase-detection autofocus, increasing the aperture value shortens the baseline length of the parallax image pair, thus reducing the distance resolution.

[0057] Furthermore, regardless of the image sensor configuration, increasing the shooting sensitivity (ISO sensitivity) amplifies image noise, which reduces the accuracy of detecting the amount of defocus and thus the accuracy of distance information. Also, when the proportion of the image area occupied by the object is small (low shooting magnification), the surface area of ​​the object corresponding to one pixel becomes large, which reduces the reproducibility of the object shape.

[0058] Thus, when acquiring distance information from a captured image, the accuracy of the distance information can vary depending on the shooting conditions. For example, when using an image sensor that supports on-sensor phase-detection autofocus, the closer the aperture value is to the widest setting, the longer the baseline length of the parallax image pair becomes, and therefore the higher the accuracy of the distance information.

[0059] On the other hand, the image quality of captured images is generally higher when the aperture value is greater than the maximum aperture value. This is because vignetting and optical aberrations have the greatest impact on the image when the aperture value is maximum, and these effects are reduced when the aperture value is increased. Since better image quality results in higher quality texture information, from the perspective of texture information quality, it is better not to use a maximum aperture value. Thus, the optimal shooting conditions differ from the perspective of accuracy of distance information and 3D data based on distance information, and from the perspective of texture information quality.

[0060] This means that when attempting to reduce the amount of data generated for each frame—3D data and texture information (frame image data)—using inter-frame prediction, the optimal keyframes for 3D data and the optimal keyframes for texture information may differ. Therefore, if frames at the same timing are used as keyframes, an inefficient reduction of data volume may occur in at least one of the 3D data or texture information.

[0061] <Generating 3D video files> The digital camera 100 generates 3D video data and stores it in the memory unit 15, for example, when a shooting mode for recording 3D video is set. Specifically, the control unit 12 performs video recording at a predetermined frame rate and controls the operation of the image sensor 11 to read out the A+B signal and the A signal for each frame. It is also possible to read out the B signal instead of the A signal. The exposure conditions and focus adjustment are performed by the control unit 12, for example, frame by frame, based on evaluation values ​​generated by the image processing unit 14.

[0062] The image processing unit 14 generates frame image data for recording from the A+B signal for each frame. The frame image data for recording may be the same as that generated during general video recording. Exposure conditions used during shooting are also recorded in association with the frame image data. When recording 3D video, the frame image data of the 2D video for recording is used as texture information for the 3D data.

[0063] Furthermore, for each frame, the image processing unit 14 generates the B signal by subtracting the A signal from the A+B signal. Then, the image processing unit 14 (defocus generation unit 141) generates a defocus map from the A signal and the B signal, and further converts the defocus map into a depth map. When 3D data is to be used as polygon data, the image processing unit 14 converts the depth map into polygon data and then further converts it into 2D structured data.

[0064] The control unit 12 associates the texture information (frame image data) and 3D data (2D structured data or depth map) generated for the same frame and temporarily stores them in the RAM 20 as frame data for the 3D video. Then, the control unit 12 applies the data size reduction process (compression process) described later to the frame data of the 3D video and saves it to the storage unit 15. Alternatively, the frame data of the 3D video may be saved to the storage unit 15 without applying the compression process, and the compression process may be applied after the shooting of the 3D video is completed. In addition, the frame data of the 3D video may be saved to an external device via the communication unit 18.

[0065] <Data reduction processing for 3D videos> The data reduction (compression) process for 3D video in this embodiment will be explained using the flowchart shown in Figure 7. Here, it is assumed that the image processing unit 14 of the digital camera 100 performs the process when shooting 3D video. However, it may also be performed by an external device connected via the communication unit 18. Alternatively, it may be performed by the image processing unit 14 or an external device after the shooting of 3D video is completed. Here, the amount of data in the 3D video data is reduced using image encoding technology that uses inter-frame prediction, such as MPEG4.

[0066] In S101, the image processing unit 14 reads the 3D video data to be compressed from the storage unit 15 into the RAM 20. Here, it is assumed that at least 1 GOP (Group of Pictures) or more frames are read. If 3D video data already exists in the RAM 20, it is not necessary to read it from the storage unit 15.

[0067] In S102, the image processing unit 14 determines whether shooting information is recorded along with the 3D video data. If it determines that it is recorded, it executes S103; otherwise, it executes S104. When images are captured by an imaging device such as a digital camera 100, the shooting information is recorded, for example, as metadata.

[0068] In S103, the image processing unit 14 reads the shooting information for each frame read in S101 and stores it in the RAM 20. The shooting information read here may be, for example, the focal length, focusing distance, aperture value, ISO sensitivity, and shutter speed of the imaging optical system 10.

[0069] In S104, the image processing unit 14 performs keyframe evaluation processing. In the keyframe evaluation process, the image processing unit 14 evaluates the 3D data and texture information of each frame image data of the 3D video loaded into RAM 20, and determines whether or not it is appropriate as a keyframe (I-frame).

[0070] At this time, the image processing unit 14 evaluates the texture information and the 3D data based on separate conditions to determine separately the keyframes that are optimal for compressing the texture information and the keyframes that are optimal for compressing the 3D data.

[0071] Figure 8 schematically shows the texture information and 3D data for corresponding frames. The left column represents frame N, and the right column represents frame N+α (α≧1). Frame N was captured with aperture value a, and frame N+α was captured with aperture value b (b>a). Because frame N was captured with an aperture value closer to the widest aperture, the quality of the texture information (frame image data) is higher for frame N+α. On the other hand, the baseline length is larger for frame N, so the distance resolution of the 3D data is higher for frame N.

[0072] In the keyframe evaluation process, the image processing unit 14 can make a determination based, for example, on the aperture value at the time of shooting for all frames included in the GOP. For example, the image processing unit 14 can determine that the frame with the largest aperture value at the time of shooting is the optimal keyframe for texture information, and that the frame with the smallest aperture value at the time of shooting is the optimal keyframe for 3D data.

[0073] Alternatively, an evaluation value may be calculated for each frame based on one or more of the shooting conditions, and the frame with the highest evaluation value may be determined as the optimal keyframe. In this case, the relationship between the shooting condition items and the evaluation value can be as follows, for example.

[0074] • Aperture value As mentioned above, as the aperture value decreases, the distance resolution of the 3D data increases, while the resolution of the texture information decreases. Therefore, the smaller the aperture value, the higher the evaluation value for 3D data and the lower the evaluation value for texture information. However, when the aperture value exceeds a threshold, the contrast of the image decreases due to diffraction. Therefore, if the evaluation value for texture information is set as follows: 1st aperture value < 2nd aperture value < 3rd aperture value (threshold), the evaluation value will increase in proportion to the increase in aperture value from the 1st aperture value up to the 3rd aperture value, and then decrease beyond the 3rd aperture value. The evaluation value when the aperture value exceeds the 3rd aperture value can be a fixed value or it can be gradually decreased.

[0075] • Shutter speed A slow shutter speed increases the likelihood of camera shake and motion blur. Therefore, when the shutter speed is slower than the threshold, the evaluation values ​​for both 3D data and texture information are set lower than when the shutter speed is faster than the threshold. If the focal length of the imaging optical system 10 is variable, the threshold may be shortened as the focal length increases.

[0076] • ISO sensitivity Higher ISO sensitivity increases image noise, which reduces the reliability of 3D data. Therefore, when the ISO sensitivity is above a certain threshold, the evaluation values ​​for both 3D data and texture information are lower than when the ISO sensitivity is above the threshold.

[0077] • Magnification (combination of focal length and focusing distance) When the shooting magnification is low, the distance resolution of the 3D data decreases. The resolution of the texture information also decreases. Therefore, when the shooting magnification is below a threshold, the evaluation values ​​for both the 3D data and the texture information are set lower than when it is above the threshold. The shooting magnification can be set in advance, for example, according to the combination of the focal length and focusing distance of the imaging optical system 10. Alternatively, the shooting magnification may be associated with the proportion of the screen occupied by the area of ​​the main subject.

[0078] For example, in the case of 3D video where the main subject approaches or moves away from the digital camera 100, the higher the magnification and the shorter the focus distance, the higher the evaluation values ​​for both the 3D data and texture information.

[0079] Furthermore, evaluation of whether a frame is suitable as a keyframe may be performed based on conditions other than the shooting information. For example, texture information (frame image data) may be evaluated separately for areas with proper exposure and dark areas. Specifically, among the multiple frames to be evaluated, the frame with the dark area exposure closest to proper exposure will be considered the frame with the highest evaluation value for the dark area. Bright areas can also be evaluated separately in a similar manner.

[0080] The evaluation value may be a binary value such as OK / NG (or 1 / 0), or it may be a value with three or more values. Alternatively, it may be a value corresponding to the rank of the frame being evaluated. The image processing unit 14 stores the evaluation value in RAM 20, associating it with the frame that was evaluated.

[0081] In S105, the image processing unit 14 selects keyframes for 3D data and keyframes for texture information based on the results of the evaluation process in S104. For example, the image processing unit 14 can select the frame with the highest evaluation value as the keyframe. If there are multiple evaluation values ​​for each frame, the frame with the highest sum of evaluation values ​​can be selected as the keyframe. Keyframes may also be selected based on other conditions. Furthermore, keyframes may be selected from frames that do not have an NG evaluation value. The same applies when selecting keyframes for texture information for each region.

[0082] In S106, the image processing unit 14 encodes 3D data and texture information separately on a GOP basis using MPEG encoding with the keyframe selected in S105 as the I-frame. Since the MPEG encoding method that assigns I-frames, P-frames, and B-frames to each GOP and performs inter-frame predictive encoding for P-frames and B-frames is well known, its details will not be explained.

[0083] The image processing unit 14 encodes 3D video data by repeatedly executing the processes from S101 to S106 as needed.

[0084] In S107, the image processing unit 14 sequentially records a 3D video data file containing encoded 3D data and 3D video data including texture information into the storage unit 15. When 3D video data is read from the storage unit 15 in S101, it may be replaced with the encoded 3D video data, or the unencoded 3D video data may be retained. Alternatively, the 3D video data file containing the encoded 3D video data may be transmitted to an external device via the communication unit 18.

[0085] Here, the external device includes a decoder corresponding to the encoding method used in S106 for encoding the 3D data and texture information. The decoder decodes the 3D data and texture information of the 3D video data stored in the 3D video data file separately, referring to the keyframes set for each. The external device then generates a combination of the decoded 3D data and texture information for each frame, loads each frame into memory, and reads and displays the frames in chronological order. This makes it possible to play back and display the video while giving a sense of depth to the 3D objects included as subjects in the image. Alternatively, after the 3D data and texture information have been decoded, the combination of the 3D data and texture information may be generated for each frame and stored as a file in the external device's storage device.

[0086] According to this embodiment, when encoding 3D video data having 3D data and texture information for each frame using inter-frame prediction, keyframes for the 3D data and keyframes for the texture information are determined separately. This allows the 3D data and texture information to be encoded using optimal keyframes, thereby suppressing quality degradation due to encoding and efficiently reducing the amount of data.

[0087] ●(Second Embodiment) Next, a second embodiment of the present invention will be described. This embodiment may be the same as the first embodiment except for the 3D video data compression process. Therefore, the compression process will be described below. Figure 9 is a flowchart relating to the 3D video data compression process in this embodiment. Steps that perform the same processing as in the first embodiment are denoted by the same reference numerals as in Figure 7. In this embodiment, step S201, which performs 3D data analysis processing, is included before the keyframe evaluation processing in S104.

[0088] 3D data analysis processing is performed to better evaluate and select keyframes. When generating 3D data from disparity images, the entire disparity image is rarely included in the depth of field; it generally contains blurred areas. Areas with high in-focus areas have higher contrast than areas with low in-focus areas, resulting in higher distance resolution for the resulting 3D data.

[0089] During video recording, the focus distance can change over time, and therefore, the areas with high focus in the parallax image can also change over time. For this reason, for 3D data, 3D data with high distance resolution can be selected as keyframes for each region. The 3D data may be divided in the distance direction, or in the distance and vertical directions. The 3D data before division may be a single continuous object or multiple objects.

[0090] Figure 10 schematically shows the texture information and 3D data for the corresponding frames. The left column represents frame N, the middle column represents frame N+α (α≧1), and the right column represents frame N+β (β>α).

[0091] Frames N and N+α were captured with aperture value a, while frame N+β was captured with aperture value b (b>a). Furthermore, in frame N, the focus is on the foreground of the object, while in frame N+α, the focus is on the background. Frame N+β shows a state where the entire texture information is in focus due to the increased aperture value compared to frame N+α.

[0092] The 3D data shows areas with high distance resolution using a grid pattern. In frame N, the distance resolution is high on the near side of the object, and in frame N+α, the distance resolution is high on the far side of the object. In frame N+β, the increased aperture value reduces the distance resolution on the far side of the object, eliminating the area with high distance resolution.

[0093] In the 3D data analysis process, the image processing unit 14 divides the 3D data into a foreground and a background, and increases the evaluation value of frame N for the foreground and frame N+α for the background. In addition, it increases the evaluation value of frame N+β for the texture information.

[0094] The image processing unit 14 stores information on how the 3D data was divided, evaluation values ​​for each divided region of the 3D data, and evaluation values ​​for texture information in RAM 20. This information and evaluation values ​​are considered in the keyframe selection process of S105, along with the evaluation values ​​determined in the keyframe evaluation process of S104.

[0095] Furthermore, in the keyframe evaluation process in S104, it is not necessary to calculate evaluation values ​​for 3D data. Alternatively, evaluation values ​​may be calculated only for items not considered in the 3D data analysis process. The same applies to the evaluation values ​​for texture information.

[0096] In S105, the image processing unit 14 selects keyframes for the 3D data for each divided region. Keyframes for texture information can be selected in the same manner as in the first embodiment.

[0097] In S106, the image processing unit 14 processes the 3D data in the same manner as in the first embodiment, except that it encodes each divided region.

[0098] According to this embodiment, keyframes can be selected more precisely for 3D data, and the amount of data can be effectively reduced while further suppressing the degradation of 3D data quality.

[0099] The encoded 3D video data generated in the first and second embodiments can be decoded using known methods. The decoded 3D data is converted into a polygon mesh. Furthermore, based on the decoded texture information, textures can be mapped onto the 3D model based on the polygon mesh.

[0100] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0101] This embodiment includes the following image processing apparatus, imaging apparatus, image processing method, and program. (Item 1) Acquisition means for acquiring 3D video data having 3D data and texture information for each frame, The system includes an encoding means for encoding the aforementioned 3D video data using inter-frame prediction, The encoding means is The 3D data and the texture information are encoded separately. Keyframes for encoding the 3D data and keyframes for encoding the texture information are selected separately. An image processing apparatus characterized by the following: (Item 2) The image processing apparatus according to item 1, characterized in that the encoding means selects the key frame based on evaluation values ​​for the three-dimensional data and the texture information based on the frame's shooting information. (Item 3) The image processing apparatus according to item 2, characterized in that the encoding means obtains the evaluation value for the three-dimensional data and the texture information for each frame, and selects the frame with the highest evaluation value as the key frame. (Item 4) The image processing apparatus according to item 2 or 3, characterized in that the aforementioned shooting information includes one or more of the shutter speed, aperture value, ISO sensitivity, focusing distance, and focal length of the imaging optical system at the time of shooting. (Item 5) The aforementioned shooting information includes the aperture value at the time of shooting. The evaluation value for the three-dimensional data is higher when the shooting information is a first aperture value than when it is a second aperture value that is greater than the first aperture value. The evaluation value for the texture information is higher when the shooting information is at the second aperture value than when the shooting information is at the first aperture value. An image processing apparatus according to any one of items 2 to 4, characterized in that (Item 6) The image processing apparatus according to item 5, characterized in that the evaluation value for the texture information becomes lower when the shooting information exceeds a third aperture value greater than the second aperture value. (Item 7) The aforementioned texture information is a frame image of a 2D video. The encoding means selects keyframes for encoding the texture information for each region of the frame image. An image processing apparatus according to any one of items 1 to 6, characterized in that (Item 8) The encoding means divides the 3D data into multiple regions and selects a keyframe for encoding the 3D data for each region of the 3D data. An image processing apparatus according to any one of items 1 to 7, characterized in that (Item 9) The image processing apparatus according to item 8, characterized in that the encoding means divides the three-dimensional data into one or more of the depth direction, horizontal direction, and vertical direction. (Item 10) The aforementioned 3D data is polygon data, The encoding means converts the polygon data into two-dimensional structured data before encoding it. An image processing apparatus according to any one of items 1 to 9, characterized in that (Item 11) An imaging means capable of generating a pair of disparity images in a single shot, A generation means that generates 3D video data having 3D data and texture information for each frame based on the video captured by the aforementioned imaging means, An image processing apparatus according to any one of items 1 to 10, which processes the three-dimensional video data generated by the generation means, An imaging device characterized by having the following features. (Item 12) An image processing method performed by an image processing device, The acquisition process involves obtaining 3D video data that has 3D data and texture information for each frame, The process includes encoding the aforementioned 3D video data using inter-frame prediction, In the aforementioned encoding process, The 3D data and the texture information are encoded separately. Keyframes for encoding the 3D data and keyframes for encoding the texture information are selected separately. An image processing method characterized by the following: (Item 13) A program for causing a computer to function as one of the means of an image processing device described in any one of items 1 through 10.

[0102] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0103] 100...Digital camera, 10...Imaging optical system, 11...Image sensor, 12...Control unit, 14...Image processing unit, 15...Storage unit

Claims

1. An acquisition means for acquiring 3D video data having 3D data and texture information for each frame, The system includes an encoding means for encoding the aforementioned three-dimensional video data using inter-frame prediction, The encoding means is The three-dimensional data and the texture information are encoded separately. Keyframes for encoding the 3D data and keyframes for encoding the texture information are selected separately. An image processing apparatus characterized by the following:

2. The image processing apparatus according to claim 1, wherein the encoding means selects the key frame based on evaluation values ​​of the three-dimensional data and the texture information based on the frame's shooting information.

3. The image processing apparatus according to claim 2, characterized in that the encoding means obtains the evaluation value for the three-dimensional data and the texture information for each frame, and selects the frame with the highest evaluation value as the key frame.

4. The image processing apparatus according to claim 2, characterized in that the aforementioned shooting information includes one or more of the shutter speed, aperture value, ISO sensitivity, focusing distance, and focal length of the imaging optical system at the time of shooting.

5. The aforementioned shooting information includes the aperture value at the time of shooting. The evaluation value for the three-dimensional data is higher when the shooting information is a first aperture value than when the shooting information is a second aperture value which is greater than the first aperture value. The evaluation value for the texture information is higher when the second aperture value is used than when the first aperture value is used. The image processing apparatus according to claim 2.

6. The image processing apparatus according to claim 5, characterized in that the evaluation value for the texture information becomes lower when the shooting information exceeds a third aperture value that is greater than the second aperture value.

7. The aforementioned texture information is a frame image of a two-dimensional video. The encoding means selects keyframes for encoding the texture information for each region of the frame image. The image processing apparatus according to feature 1.

8. The encoding means divides the three-dimensional data into multiple regions and selects a keyframe for encoding the three-dimensional data for each region of the three-dimensional data. The image processing apparatus according to feature 1.

9. The image processing apparatus according to claim 8, characterized in that the encoding means divides the three-dimensional data into one or more of the depth direction, horizontal direction, and vertical direction.

10. The aforementioned three-dimensional data is polygon data. The encoding means converts the polygon data into two-dimensional structured data before encoding it. The image processing apparatus according to feature 1.

11. An imaging means capable of generating a pair of disparity images in a single shot, A generation means that generates three-dimensional video data having three-dimensional data and texture information for each frame based on the video captured by the aforementioned imaging means, An image processing apparatus according to any one of claims 1 to 10, which processes the three-dimensional video data generated by the generation means, An imaging device characterized by having the following features.

12. An image processing method performed by an image processing device, The acquisition process involves acquiring 3D video data that has 3D data and texture information for each frame, The three-dimensional video data is encoded using inter-frame prediction, and the encoding process is also included. In the aforementioned encoding process, The three-dimensional data and the texture information are encoded separately. Keyframes for encoding the 3D data and keyframes for encoding the texture information are selected separately. An image processing method characterized by the following:

13. A program for causing a computer to function as each of the means of the image processing apparatus described in any one of claims 1 to 10.

Images