Information processing apparatus, information processing method, and program

Abstract

This information processing device (IP1) comprises a depth information extraction unit (DIE1) and a processing unit (IMP). The depth information extraction unit (DIE1) can extract depth information from a plurality of pieces of infrared image information included in a plurality of pieces of image data. The plurality of pieces of image data are image data photographed from a plurality of viewpoints. Each of the plurality of pieces of image data includes visible light image information and infrared image information. The processing unit (IMP) processes, on the basis of the depth information, the visible light image generated by using the visible image information included in at least one piece of image data among the plurality of pieces of image data.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] There is known a stereo image technique for extracting depth information (depth information) of a subject using parallax information. Products using stereo image technology are generally called stereo cameras. The stereo image method includes a passive stereo method and an active stereo method. The passive stereo method is a method of extracting depth information using parallax information of a plurality of visible light images. The active stereo method is a method of extracting depth information using parallax information of a plurality of infrared images obtained by photographing an infrared light projection pattern (see, for example, Patent Documents 1 and 2).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the stereo image method, it is necessary to determine corresponding points between two images. In the active stereo method, since an infrared light projection pattern is projected onto the subject, it is easier to determine corresponding points compared to the passive stereo method. However, the infrared light projection pattern appears in the images captured by the active stereo method. Therefore, it is difficult to use the captured images as they are as images for viewing. Although it is conceivable to install a separate camera for viewing, there is a positional shift due to parallax between the depth map generated using depth information and the image for viewing. Therefore, it is difficult to perform image processing (such as foreground / background separation, refocusing, and relighting) using the depth map.

[0005] Therefore, the present disclosure proposes an information processing apparatus, an information processing method, and a program that can easily perform image processing using depth information.

Means for Solving the Problems

[0006] According to the present disclosure, there is provided an information processing apparatus including a depth information extraction unit that extracts depth information from a plurality of infrared image information included in a plurality of image data captured from a plurality of viewpoints, the plurality of image data including visible light image information and infrared image information, and a processing unit that processes a visible light image generated using the visible light image information included in at least one of the plurality of image data based on the depth information. Further, according to the present disclosure, there are provided an information processing method in which the information processing of the information processing apparatus is executed by a computer, and a program that causes a computer to realize the information processing of the information processing apparatus.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Mode for Carrying Out the Invention

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same parts are denoted by the same reference numerals, and redundant descriptions are omitted.

[0009] Note that the description will be made in the following order. [1. First Embodiment] [1-1. Configuration of Information Processing Apparatus] [1-2. Information Processing Method] [1-3. Effects] [2. Second Embodiment] [2-1. Configuration of Information Processing Apparatus] [2-2. Information Processing Method] [2-3. Effects] [3. Third Embodiment] [3-1. Configuration of Information Processing Apparatus] [3-2. Information Processing Method] [3-3. Effects] [4. Fourth Embodiment] [4-1. Configuration of Information Processing Apparatus] [4-2. Information Processing Method] [4-3. Effects] [5. Fifth Embodiment] [6. Variations of Pixel Array Section] [7. Sixth Embodiment] [7-1. Configuration of Information Processing Apparatus] [7-2. Information Processing Method] [7-3. Effects] [8. Seventh Embodiment] [8-1. Configuration of Information Processing Apparatus] [8-2. Information Processing Method] [8-3. Effects] [9. Eighth Embodiment] [9-1. Configuration of Information Processing Apparatus] [9-2. Information Processing Method] [9-3. Effects]

[0010] [1. First Embodiment] [1-1. Configuration of Information Processing Apparatus] FIG. 1 is a schematic diagram of an information processing apparatus IP1 according to the first embodiment. The information processing apparatus IP1 is, for example, a stereo camera.

[0011] The information processing apparatus IP1 includes, for example, a processing unit PU1, a plurality of cameras CA, a projector PJ, and a storage device ST1.

[0012] The processing device PU1 is a device that extracts depth information and performs image processing using a plurality of image data obtained from a plurality of cameras CA. Image processing includes, for example, foreground / background separation, refocusing, and relighting. Foreground / background separation is a process of separating the foreground and the background. Refocusing is a process of adjusting the focus only on a specified part so that a subject in the foreground stands out against the background. Relighting is a process of adjusting the brightness of a specified part so that a subject in the foreground stands out against the background. Image processing is performed based on depth information.

[0013] Figure 2 is a schematic diagram of the camera CA.

[0014] The camera CA includes a lens LE, a UV cut filter UVF, a low-pass filter LPF, and an image sensor IS. The UV cut filter UVF cuts ultraviolet rays. The low-pass filter LPF passes only light with wavelengths necessary as image information and cuts other light. The low-pass filter LPF suppresses the occurrence of moiré and false colors by intentionally blurring the image captured by the lens LE.

[0015] The image sensor IS converts the light entering from the lens LE into an electrical signal. The image sensor IS includes, for example, a lens array LA, a color filter array CFA, and a sensor plate SP. The sensor plate SP has a plurality of light source conversion elements (photodiodes) PD arranged two-dimensionally. The light source conversion element PD photoelectrically converts the amount of incident light into an amount of charge, accumulates it inside, and outputs it as a signal. The color filter array CFA has a plurality of color filters CF provided in one-to-one correspondence with the plurality of light receiving elements PD. The lens array LA has a plurality of microlenses ML that condense the light incident from the lens LE onto the plurality of light receiving elements PD.

[0016] As the image sensor IS, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor and a CCD (Charge-Coupled Device) image sensor are used. As the color filter array CFA, for example, a primary color system color filter array and a complementary color system color filter array are used. The primary color system color filter array has color filters CF of three colors: red, green, and blue. The complementary color system color filter array has color filters CF of four colors: cyan, yellow, magenta, and green. In this embodiment, a CMOS image sensor using a primary color system color filter array is used. The camera CA is used in a wide range of applications such as in-vehicle use.

[0017] FIG. 3 is a diagram showing an example of the configuration of the image sensor IS.

[0018] The image sensor IS includes a pixel array section PA, a vertical drive section VD, a column readout circuit section CRC, a column signal processing section CSP, a horizontal drive section HD, a system control section SC, and a signal processing section SP. The pixel array section PA, the vertical drive section VD, the column readout circuit section CRC, the column signal processing section CSP, the horizontal drive section HD, the system control section SC, and the signal processing section SP are realized by a processing circuit PR such as an IC (Integrated Circuit) formed on a sensor plate SP, for example.

[0019] The pixel array section PA has a plurality of pixels PX arranged two-dimensionally. The pixel PX includes a photoelectric conversion element PD and a color filter CF. In the pixel array section PA, a plurality of pixel drive lines LD extending in the horizontal direction (row direction: left-right direction in the figure) and a plurality of vertical pixel wirings LV extending in the vertical direction (column direction: up-down direction in the figure) are provided in a grid pattern. The pixel drive lines LD are provided for each pixel row extending in the horizontal direction. The vertical pixel wirings LV are provided for each pixel column extending in the vertical direction. One end of the pixel drive line LD is connected to the output end corresponding to each row of the vertical drive section VD.

[0020] The column readout circuit section CRC includes at least a circuit that supplies a constant current to each column of pixels PX in the selected row within the pixel array section PA, a current mirror circuit, a switching switch for the pixel PX to be read out, and the like. The column readout circuit section CRC forms an amplifier together with the transistors in the selected pixels within the pixel array section PA, converts the photocharge signal into a voltage signal, and outputs it to the vertical pixel wiring LV.

[0021] The vertical drive section VD includes a shift register, an address decoder, and the like. The vertical drive section VD drives each pixel PX of the pixel array section PA in row units. Although the specific configuration is omitted in the illustration, the vertical drive section VD has a configuration including a readout scanning system and a wiping scanning system or a batch wiping and batch transfer system.

[0022] The readout scanning system sequentially selects and scans the pixels PX of the pixel array section PA in row units in order to read out the pixel signals from the pixels PX. In the case of line drive (rolling shutter operation), for wiping, the wiping scan is performed for the readout row where the readout scan is performed by the readout scanning system, with a time corresponding to the shutter speed ahead of the readout scan. Also, in the case of global exposure (global shutter operation), the batch wipe is performed with a time corresponding to the shutter speed ahead of the batch transfer. By such wiping, unnecessary charges are wiped out (reset) from the photodiodes PD of the pixels PX in the readout row. Then, by the wiping out (reset) of the unnecessary charges, a so-called electronic shutter operation is performed.

[0023] Here, the electronic shutter operation refers to an operation of discarding the unnecessary photocharges that have accumulated in the photodiode PD until just before and starting a new exposure (starting the accumulation of photocharges).

[0024] The signal read out by the readout scanning system during the readout operation corresponds to the amount of light incident after the immediately preceding readout operation or electronic shutter operation. In the case of line driving, the period from the readout timing by the immediately preceding readout operation or the blanking timing by the electronic shutter operation to the readout timing by the current readout operation becomes the accumulation time (exposure time) of the photoelectric charge in the pixel PX. In the case of global exposure, the time from the batch blanking to the batch transfer becomes the accumulation time (exposure time).

[0025] The pixel signals output from each pixel PX of the pixel row selectively scanned by the vertical drive unit VD are supplied to the column signal processing unit CSP through each of the vertical pixel wirings LV. The column signal processing unit CSP performs predetermined signal processing on the pixel signals output from each pixel PX of the selected row through the vertical pixel wiring LV for each pixel column of the pixel array unit PA, and temporarily holds the pixel signals after the signal processing.

[0026] Specifically, the column signal processing unit CSP performs at least noise removal processing, for example, CDS (Correlated Double Sampling) processing, as the signal processing. By the CDS by this column signal processing unit CSP, fixed pattern noises specific to the pixel such as reset noise and threshold variations of the amplification transistor AMP are removed. In addition to the noise removal processing, the column signal processing unit CSP can also be configured to have, for example, an AD conversion function and output the pixel signal as a digital signal.

[0027] The horizontal drive unit HD includes a shift register, an address decoder, etc. The horizontal drive unit HD sequentially selects the unit circuits corresponding to the pixel columns of the column signal processing unit CSP. By the selective scanning by this horizontal drive unit HD, the pixel signals processed by the column signal processing unit CSP are sequentially output to the signal processing unit SP.

[0028] The system control unit SC includes a timing generator that generates various timing signals. Based on the various timing signals generated by the timing generator, the system control unit SC performs drive control of the vertical drive unit VD, the column signal processing unit CSP, the horizontal drive unit HD, and the like.

[0029] The image sensor IS further includes a signal processing unit SP and a data storage unit (not shown). The signal processing unit SP has at least an addition processing function and performs various signal processes such as addition processing on the pixel signals output from the column signal processing unit CSP. The data storage unit temporarily stores the data necessary for the processing when performing the signal processing in the signal processing unit SP. The processing of the signal processing unit SP and the data storage unit may be replaced by an external signal processing unit provided on a board different from the image sensor IS, for example, a DSP (Digital Signal Processor) or software.

[0030] Returning to FIG. 1, the plurality of cameras CA are installed at different positions from each other. Therefore, the positions of the viewpoints of the plurality of cameras CA when photographing a subject are different from each other. The plurality of cameras CA output the image data photographed at the plurality of viewpoints to the processing device PU1. In the example of FIG. 1, as the plurality of cameras CA, a first camera CA1 and a second camera CA2 are provided. The first camera CA1 and the second camera CA2 are installed at symmetric positions centered on the projector PJ.

[0031] The camera CA has an image sensor IS capable of detecting both visible light and infrared rays. The image sensor IS has a structure in which, for example, a plurality of pixels PX for detecting visible light image information and a plurality of pixels PX for detecting infrared ray image information are periodically arranged in a two-dimensional direction. For example, the image sensor IS has a plurality of pixel blocks PB arranged two-dimensionally. The pixel block PB has a structure in which, for example, one pixel PX1 for detecting red light, one pixel PX2 for detecting green light, one pixel PX3 for detecting blue light, and one pixel PX4 for detecting infrared rays are arranged in two rows and two columns.

[0032] Pixel PX1 includes, for example, a color filter CF that selectively transmits red light and selectively absorbs green light, blue light, and infrared rays. Pixel PX2 includes, for example, a color filter CF that selectively transmits green light and selectively absorbs red light, blue light, and infrared rays. Pixel PX3 includes, for example, a color filter CF that selectively transmits blue light and selectively absorbs red light, green light, and infrared rays. Pixel PX4 is not provided with a color filter CF that absorbs infrared rays, for example. For example, the color filter array CFA in the portion corresponding to pixel PX4 is a transparent layer and transmits red light, green light, blue light, and infrared rays.

[0033] Projector PJ projects an infrared light projection pattern onto a subject. As the infrared light projection pattern, a known pattern used in, for example, a spot light projection method, a slit light projection method, and a pattern light projection method is adopted.

[0034] Processing device PU1 has, for example, an image data acquisition unit IDO, an infrared image extraction unit IRE, a visible light image extraction unit VLE1, a depth information extraction unit DIE1, a distance detection unit DD, a processing unit IMP, and an output unit OT.

[0035] The image data acquisition unit IDO acquires, for example, a plurality of image data captured from a plurality of viewpoints from a plurality of cameras CA. Each of the plurality of image data includes visible light image information and infrared image information. The image data acquisition unit IDO outputs the plurality of image data to the infrared image extraction unit IRE and the visible light image extraction unit VLE1.

[0036] The infrared image extraction unit IRE extracts, for example, an infrared image from each of the plurality of image data using the infrared image information. The infrared image extraction unit IRE outputs the plurality of infrared images extracted from the plurality of image data to the depth information extraction unit DIE1.

[0037] The visible light image extraction unit VLE1 extracts visible light images from a plurality of image data, for each piece of image data, using visible light image information. The visible light image extraction unit VLE1 outputs a plurality of visible light images extracted from the plurality of image data to the depth information extraction unit DIE1 and the distance detection unit DD. The visible light image extraction unit VLE1 outputs at least one visible light image among the plurality of visible light images extracted from the plurality of image data to the processing unit IMP.

[0038] The extraction of the infrared image and the visible light image is performed using the light quantity values of red, green, blue, and infrared (hereinafter referred to as color values) of each pixel PX calculated by the demosaicing process. For example, the signal processing unit SP performs demosaicing on the detection value of each pixel PX. The demosaicing process is a process of complementing information on the wavelength of light (hereinafter referred to as color) missing for each pixel PX based on the detection values of surrounding pixels PX. The infrared image extraction unit IRE extracts an infrared image using, for example, the infrared color value of each pixel PX. The visible light image extraction unit VLE1 extracts a visible light image using, for example, the red, green, and blue color values of each pixel PX.

[0039] The demosaicing process can be performed by various known methods. As a simple method, there is a method of linearly interpolating using the detection values of a plurality of pixels PX responsible for the same color nearby. The color information of each pixel PX may be estimated using a machine learning technique. For example, the signal processing unit SP can estimate the color value of each color for each pixel PX from the detection value of each pixel PX using an analysis model obtained by machine learning the relationship between the known luminance distribution and the detection value of each pixel PX.

[0040] The depth information extraction unit DIE1 extracts depth information from a plurality of image data captured at a plurality of viewpoints by a plurality of cameras CA. The depth information extraction unit DIE1 outputs the depth information as a depth map to the processing unit IMP and the output unit OT. The depth map is data that defines the depths of a plurality of measurement points set in the captured image of the camera CA in association with the coordinates of each measurement point.

[0041] The depth information extraction unit DIE1 has, for example, a passive stereo mode and an active stereo mode. The passive stereo mode is a stereo mode for extracting depth information from a plurality of visible light image information included in a plurality of image data. The active stereo mode is a stereo mode for extracting depth information from a plurality of infrared image information included in a plurality of image data. The depth information extraction unit DIE1 switches between the passive stereo mode and the active stereo mode according to the situation.

[0042] Various situations can be considered as the situation in which the stereo mode is switched. The passive stereo mode and the active stereo mode each have advantages and disadvantages. The stereo mode is switched so as to complement each other's disadvantages.

[0043] For example, the depth information extraction unit DIE1 switches between the passive stereo mode and the active stereo mode according to the situation based on the distance from the subject. For example, when the distance from the subject is greater than the threshold value, the depth information extraction unit DIE1 extracts depth information in the passive stereo mode. When the distance from the subject is equal to or less than the threshold value, the depth information extraction unit DIE1 extracts depth information in the active stereo mode.

[0044] In the active stereo method, the interval of the infrared light projection pattern reflected in the image of the camera CA changes according to the distance from the subject. When the distance from the subject increases, aliasing may occur in relation to the arrangement density of the pixels PX4 for detecting the infrared image information. By switching to the passive stereo mode in such a case, depth information can be detected accurately.

[0045] The depth information extraction unit DIE1 switches the stereo mode based on, for example, the distance between the camera CA and the subject detected by the distance detection unit DD. The distance between the camera CA and the subject is calculated, for example, as the average value of the depth information (distance) of all measurement points in the captured image of the camera CA, or as the distance between the main subject and the camera CA. The distance detection unit DD extracts the depth information of some or all of the measurement points in the captured image by the passive stereo method using a plurality of visible light images extracted by the visible light image extraction unit VLE1. The distance detection unit DD detects the distance between the camera CA and the subject based on the extracted depth information.

[0046] The processing unit IMP processes the visible light image generated using the visible light image information included in at least one of the plurality of image data based on the depth information. For example, one of the plurality of cameras CA is selected as the reference camera. How to select the reference camera is arbitrary. In this embodiment, for example, the first camera CA1 is selected as the reference camera. The processing unit IMP performs image processing (such as foreground / background separation, refocusing, and relighting) based on the depth information on the visible light image (reference image) generated using the visible light image information included in the image data of the reference camera. The processing unit IMP outputs the visible light image (processed image) obtained by the image processing to the output unit OT.

[0047] The output unit OT outputs the visible light image output from the processing unit INP and the depth information output from the depth information extraction unit DIE1 to an external device.

[0048] The storage device ST1 stores, for example, the program PG1 executed by the processing device PU1. The program PG1 is a program for causing a computer to execute the information processing according to the present embodiment. The processing device PU1 performs various processes according to the program PG1 stored in the storage device ST1. The storage device ST1 may be used as a work area for temporarily storing the processing results of the processing device PU1. The storage device ST1 includes, for example, any non-transitory storage medium such as a semiconductor storage medium and a magnetic storage medium. The storage device ST1 is configured to include, for example, an optical disk, a magneto-optical disk, or a flash memory. The program PG1 is stored, for example, in a non-transitory storage medium readable by a computer.

[0049] The processing device PU1 is, for example, a computer composed of a processor and a memory. The memory of the processing device PU1 includes a RAM (Random Access Memory) and a ROM (Read Only Memory). By executing the program PG1, the processing device PU1 functions as an image data acquisition unit IDO, an infrared image extraction unit IRE, a visible light image extraction unit VLE1, a depth information extraction unit DIE1, a distance detection unit DD, a processing unit IMP, and an output unit OT.

[0050] [1-2. Information Processing Method] FIGS. 4 and 5 are diagrams showing an example of the information processing method of the present embodiment. FIG. 4 is a conceptual diagram of information processing. FIG. 5 is a flowchart showing the information processing method.

[0051] In step S1, the plurality of cameras CA capture a subject from a plurality of viewpoints. For example, the first camera CA captures image data from the first viewpoint. The second camera CA2 captures image data from the second viewpoint. The image data acquisition unit IDO acquires a plurality of image data captured from a plurality of viewpoints. The visible light image extraction unit VLE1 extracts a visible light image from the plurality of image data for each piece of image data using visible light image information.

[0052] In step S2, the distance detection unit DD extracts depth information of some or all of the measurement points in the captured image of the camera CA by the passive stereo method using a plurality of visible light images extracted from the plurality of image data. The distance detection unit DD detects the distance between the camera CA and the subject using the extracted depth information.

[0053] In step S3, the depth information extraction unit DIE1 determines whether the distance detected by the distance detection unit DD is greater than a threshold value. In step S3, if it is determined that the distance is greater than the threshold value (step S3: Yes), the process proceeds to step S4. In step S4, the depth information extraction unit DIE1 selects the passive stereo mode. The depth information extraction unit DIE1 extracts depth information by the passive stereo method using the plurality of visible light images extracted by the visible light image extraction unit VLE1. Then, the process proceeds to step S6. When the distance detection unit DD extracts the depth information of all the measurement points in the captured image of the camera CA in step S2, the depth information extraction unit DIE1 outputs the depth information extracted by the distance detection unit DD as it is to the processing unit IMP and the output unit OT.

[0054] In step S3, if it is determined that the distance is less than or equal to the threshold value (step S3: No), the process proceeds to step S5. In step S5, the depth information extraction unit DIE1 selects the active stereo mode. The depth information extraction unit DIE1 extracts depth information by the active stereo method using the plurality of infrared images extracted by the infrared image extraction unit IRE. Then, the process proceeds to step S6.

[0055] In step S6, the processing unit IMP performs preprocessing on the visible light image acquired from the visible light image extraction unit VLE1. This visible light image is a reference image generated using the visible light image information included in the image data of the first camera CA1 (reference camera). The preprocessing includes, for example, missing part interpolation processing and upsampling processing. The missing part interpolation processing is a process of obtaining missing information by interpolation. The upsampling processing is a process of converting the sampling frequency to a higher one.

[0056] In step S7, the processing unit IMP performs image processing (such as foreground / background separation, refocusing, and relighting) based on depth information on the pre-processed visible light image.

[0057] [1-3. Effect] The information processing apparatus IP1 includes a depth information extraction unit DIE1 and a processing unit IMP. The depth information extraction unit DIE1 can extract depth information from a plurality of infrared image information included in a plurality of image data. The plurality of image data are image data captured from a plurality of viewpoints. Each of the plurality of image data includes visible light image information and infrared image information. The processing unit IMP processes a visible light image generated using the visible light image information included in at least one of the plurality of image data based on the depth information. In the information processing method of the present embodiment, the information processing of the above-described information processing apparatus is executed by a computer. The program of the present embodiment causes a computer to realize the information processing of the above-described information processing apparatus.

[0058] According to this configuration, the infrared image information for detecting depth information and the visible light image information for generating a viewing image are included in the image data of the same viewpoint. Therefore, misalignment is less likely to occur between the depth map generated using the depth information and the visible light image. Thus, image processing using the depth information can be easily performed.

[0059] The depth information extraction unit DIE1 switches between a passive stereo mode and an active stereo mode according to the situation. The passive stereo mode is a stereo mode for extracting depth information from a plurality of visible light image information included in a plurality of image data. The active stereo mode is a stereo mode for extracting depth information from a plurality of infrared image information included in a plurality of image data.

[0060] The passive stereo mode and the active stereo mode each have advantages and disadvantages. By switching the stereo mode according to the situation, the disadvantages of each other can be compensated for.

[0061] The depth information extraction unit DIE1 switches between the passive stereo mode and the active stereo mode according to the situation based on the distance from the subject.

[0062] In the active stereo method, the interval of the infrared light projection pattern captured in the image of the camera CA changes according to the distance from the subject. When the distance from the subject increases, aliasing may occur in relation to the arrangement density of the pixels PX4 for detecting the infrared image information. By switching to the passive stereo mode in such a case, the depth information can be detected accurately.

[0063] The information processing device IP1 has a plurality of image sensors IS that respectively capture a plurality of image data. Each of the plurality of image sensors IS has a structure in which a plurality of pixels PX (pixels PX1, PX2, PX3) for detecting visible light image information and a plurality of pixels PX (pixels PX4) for detecting infrared image information are periodically arranged in a two-dimensional direction.

[0064] According to this configuration, the infrared image information and the visible light image information are easily separated and extracted.

[0065] Each of the plurality of image sensors IS has a plurality of pixel blocks PB arranged two-dimensionally. Each of the plurality of pixel blocks PB has a structure in which one pixel PX1 for detecting red light, one pixel PX2 for detecting green light, one pixel PX3 for detecting blue light, and one pixel PX4 for detecting infrared are arranged in a 2×2 matrix.

[0066] According to this configuration, the information of red, green, blue, and infrared is detected in a well-balanced manner.

[0067] [2. Second Embodiment] [2-1. Configuration of Information Processing Device] FIG. 6 is a schematic diagram of the information processing device IP2 according to the second embodiment. The differences from the first embodiment in this embodiment are that the extraction of depth information is performed by the active stereo method, and that the processing device PU2 has a pattern control unit PTC. Hereinafter, the description will focus on the differences from the first embodiment.

[0068] The extraction of depth information is not performed by the passive stereo method. Therefore, the visible light image extraction unit VLE2 does not output the plurality of visible light images extracted from the plurality of image data to the depth information extraction unit DIE2. The distance detection unit DD outputs information regarding the distance between the camera CA and the subject (distance from the subject) to the pattern control unit PTC. The pattern control unit PTC changes the infrared light projection pattern IRP used in the active stereo mode according to the distance from the subject.

[0069] For example, when the distance from the subject is greater than the threshold value, the pattern control unit PTC projects a rough long-distance pattern with a large interval between spots or slits as the infrared light projection pattern IRP. When the distance from the subject is equal to or less than the threshold value, the pattern control unit PTC projects a fine short-distance pattern with a narrow interval between spots or slits as the infrared light projection pattern IRP.

[0070] The storage device ST2 stores, for example, the program PG2 executed by the processing device PU2. The program PG2 is a program that causes a computer to execute the information processing according to this embodiment. The processing device PU2 performs various processes according to the program PG2 stored in the storage device ST2. By executing the program PG2, the processing device PU2 functions as an image data acquisition unit IDO, an infrared image extraction unit IRE, a visible light image extraction unit VLE2, a depth information extraction unit DIE2, a distance detection unit DD, a processing unit IMP, an output unit OT, and a pattern control unit PTC.

[0071] [2-2. Information processing method] FIGS. 7 and 8 are diagrams showing an example of the information processing method of this embodiment. FIG. 7 is a conceptual diagram of information processing. FIG. 8 is a flowchart showing the information processing method.

[0072] In step S11, a plurality of cameras CA capture a subject from a plurality of viewpoints. An image data acquisition unit IDO acquires a plurality of image data captured at the plurality of viewpoints. A visible light image extraction unit VLE1 extracts a visible light image from the plurality of image data for each piece of image data using visible light image information.

[0073] In step S12, a distance detection unit DD extracts depth information of some or all of the measurement points in the shooting area by a passive stereo method using the plurality of visible light images extracted from the plurality of image data. The distance detection unit DD detects the distance between the camera CA and the subject using the extracted depth information.

[0074] In step S13, a pattern control unit PTC determines whether the distance detected by the distance detection unit DD is greater than a threshold value. In step S13, if it is determined that the distance is greater than the threshold value (step S13: Yes), the process proceeds to step S14. In step S14, the pattern control unit PTC projects a long-distance pattern as an infrared projection pattern IRP. A depth information extraction unit DIE2 extracts depth information by an active stereo method using the plurality of infrared images in which the long-distance pattern is reflected. Then, the process proceeds to step S16.

[0075] In step S13, if it is determined that the distance is less than or equal to the threshold value (step S13: No), the process proceeds to step S15. In step S15, the pattern control unit PTC projects a short-distance pattern as an infrared projection pattern IRP. A depth information extraction unit DIE2 extracts depth information by an active stereo method using the plurality of infrared images in which the short-distance pattern is reflected. Then, the process proceeds to step S16.

[0076] In step S16, a processing unit IMP performs preprocessing on the visible light image obtained from the visible light image extraction unit VLE2. This visible light image is a reference image generated using the visible light image information included in the image data of the first camera CA1 (reference camera).

[0077] In step S17, the processing unit IMP performs image processing based on depth information on the pre-processed visible light image.

[0078] [2-3. Effect] The information processing apparatus IP2 includes a pattern control unit PTC. The pattern control unit PTC changes the infrared light projection pattern used in the active stereo mode according to the distance from the subject. According to this configuration, the occurrence of aliasing can be suppressed.

[0079] [3. Third Embodiment] [3-1. Configuration of Information Processing Apparatus] FIG. 9 is a schematic diagram of the information processing apparatus IP3 according to the third embodiment. The difference from the first embodiment in this embodiment is that the depth information extraction unit DIE3 switches between the passive stereo mode and the active stereo mode according to the situation based on the shooting scene. Hereinafter, the description will be centered on the differences from the first embodiment.

[0080] The processing device PU3 includes, for example, a scene detection unit SD. The visible light image extraction unit VLE3 outputs, for example, one of a plurality of visible light images extracted from a plurality of image data to the scene detection unit SD. The scene detection unit SD detects the shooting scene based on, for example, the visible light image output from the visible light image extraction unit VLE3. The shooting scenes to be detected include, for example, "daytime & outdoors", "indoors", and "dark". "Daytime & outdoors" indicates a shooting scene outdoors during the day. "Indoors" indicates a shooting scene indoors. "Dark" indicates a shooting scene in a dark environment.

[0081] It can be arbitrarily selected which visible light image, from which image data, is used to detect the shooting scene. In this embodiment, for example, the shooting scene is detected based on the visible light image (reference image) extracted from the image data captured by the reference camera (the first camera CA1).

[0082] For the detection of the shooting scene, a known scene recognition technology using AI (artificial intelligence) adopted in digital cameras, smartphones, etc. is used. As described in Japanese Patent Application Laid-Open No. 2011-250281, it is also possible to determine whether it is an indoor environment or an outdoor environment by estimating the number of captured GPS satellites. As described in Japanese Patent Application Laid-Open No. 2013-526215, it is also possible to determine whether it is an indoor environment or an outdoor environment from the strength of the GPS signal. When the information processing device IP3 has an illuminance sensor, the information of the illuminance sensor may be combined with the above-described method to determine the shooting scene.

[0083] The depth information extraction unit DIE3 switches the stereo mode, for example, based on the shooting scene detected by the scene detection unit SD. For example, when "daytime & outdoor" is detected as the shooting scene, the depth information extraction unit DIE3 extracts depth information in the passive stereo mode. When "indoor" or "dark" is detected as the shooting scene, the depth information extraction unit DIE3 extracts depth information in the active stereo mode.

[0084] The storage device ST3 stores, for example, the program PG3 executed by the processing device PU3. The program PG3 is a program that causes a computer to execute the information processing according to the present embodiment. The processing device PU3 performs various processes according to the program PG3 stored in the storage device ST3. By executing the program PG3, the processing device PU3 functions as an image data acquisition unit IDO, an infrared image extraction unit IRE, a visible light image extraction unit VLE3, a depth information extraction unit DIE3, a scene detection unit SD, a processing unit IMP, and an output unit OT.

[0085] [3-2. Information Processing Method] FIGS. 10 and 11 are diagrams showing an example of the information processing method of the present embodiment. FIG. 10 is a conceptual diagram of information processing. FIG. 11 is a flowchart showing the information processing method.

[0086] In step S21, the plurality of cameras CA photograph a subject from a plurality of viewpoints. The image data acquisition unit IDO acquires a plurality of image data captured from a plurality of viewpoints. The visible light image extraction unit VLE3 extracts a visible light image from the plurality of image data for each piece of image data using visible light image information.

[0087] In step S22, the scene detection unit SD detects a shooting scene based on one of the plurality of visible light images extracted by the visible light image extraction unit VLE3.

[0088] In step S23, the depth information extraction unit DIE3 determines whether "daytime & outdoors" is detected as the shooting scene. In step S23, if it is determined that "daytime & outdoors" is detected (step S23: Yes), the process proceeds to step S24. In step S24, the depth information extraction unit DIE3 selects the passive stereo mode. The depth information extraction unit DIE3 extracts depth information in the passive stereo method using the plurality of visible light images extracted by the visible light image extraction unit VLE3. Then, the process proceeds to step S26.

[0089] In step S23, if it is determined that "daytime & outdoors" is not detected (step S23: No), the process proceeds to step S25. In step S25, the depth information extraction unit DIE3 selects the active stereo mode. The depth information extraction unit DIE3 extracts depth information in the active stereo method using the plurality of infrared images extracted by the infrared image extraction unit IRE. Then, the process proceeds to step S26.

[0090] In step S26, the processing unit IMP performs preprocessing on the visible light image acquired from the visible light image extraction unit VLE3. This visible light image is a reference image generated using the visible light image information included in the image data of the first camera CA1 (reference camera).

[0091] In step S27, the processing unit IMP performs image processing based on depth information on the preprocessed visible light image.

[0092] [3-3. Effect] The depth information extraction unit DIE3 switches between, for example, the passive stereo mode and the active stereo mode according to the situation based on the shooting scene.

[0093] In the stereo image method, the detection accuracy of depth information varies depending on the shooting scene. For example, in the active stereo method, the infrared component derived from the ambient light is detected as noise. Therefore, when shooting in strong sunlight, it is difficult to accurately detect depth information. In the passive stereo method, the subject cannot be sufficiently detected in a dark environment. By switching the stereo mode according to the shooting scene, depth information can be accurately detected.

[0094] [4. Fourth Embodiment] [4-1. Configuration of Information Processing Device] FIG. 12 is a schematic diagram of the information processing device IP4 of the fourth embodiment. The difference between this embodiment and the first and third embodiments is that the depth information extraction unit DIE4 switches between the passive stereo mode and the active stereo mode according to the situation based on both the distance from the subject and the shooting scene. Hereinafter, the description will focus on the differences from the first and third embodiments.

[0095] The processing device PU4 has, for example, both a distance detection unit DD and a scene detection unit SD. The depth information extraction unit DIE4 switches the stereo mode based on both the distance from the subject detected by the distance detection unit DD and the shooting scene detected by the scene detection unit SD. For example, when "daytime & outdoors" is detected as the shooting scene, the depth information extraction unit DIE4 selects the outdoor control mode. When "indoors" or "dark" is detected as the shooting scene, the depth information extraction unit DIE4 selects the indoor control mode.

[0096] The outdoor control mode is a control in which the passive stereo mode is actively selected. The indoor control mode is a control in which the active stereo mode is actively selected. The distance conditions (threshold values) for switching the stereo mode are different between the outdoor control mode and the indoor control mode.

[0097] For example, when the outdoor control mode is selected, the following control is performed. First, when the distance from the subject is greater than the first threshold value, the depth information extraction unit DIE4 extracts depth information in the passive stereo mode. When the distance from the subject is less than or equal to the first threshold value, the depth information extraction unit DIE4 extracts depth information in the active stereo mode.

[0098] When the indoor control mode is selected, the following control is performed. First, when the distance from the subject is greater than the second threshold value, the depth information extraction unit DIE4 extracts depth information in the passive stereo mode. When the distance from the subject is less than or equal to the second threshold value, the depth information extraction unit DIE4 extracts depth information in the active stereo mode.

[0099] The first threshold value is smaller than the second threshold value. Therefore, if the distance from the subject is the same, the range of distances for which the passive stereo mode is selected is wider when the outdoor control mode is selected than when the indoor control mode is selected. Thus, when the outdoor control mode is selected, the passive stereo mode is actively selected. Conversely, if the distance from the subject is the same, the range of distances for which the active stereo mode is selected is wider when the indoor control mode is selected than when the outdoor control mode is selected. Thus, when the indoor control mode is selected, the active stereo mode is actively selected.

[0100] In the active stereo method, the infrared component derived from environmental light is detected as noise. In the outdoors during the day, due to the influence of infrared rays contained in environmental light (sunlight), the detection values derived from the infrared projection pattern are easily buried in the surrounding noise. Therefore, when shooting in strong sunlight, it is difficult to accurately detect depth information. Such a decrease in detection accuracy becomes more prominent as the distance from the subject increases. Therefore, by actively selecting the passive stereo mode in such a case, depth information can be accurately detected.

[0101] Conversely, indoors, the influence of infrared rays contained in environmental light is smaller than that outdoors during the day. Therefore, the detection values derived from the infrared projection pattern are not easily buried in the surrounding noise. Therefore, by actively selecting the active stereo mode in such a case, depth information can be accurately detected.

[0102] The storage device ST4 stores, for example, a program PG4 executed by the processing device PU4. The program PG4 is a program that causes a computer to execute information processing according to the present embodiment. The processing device PU4 performs various processes according to the program PG4 stored in the storage device ST4. By executing the program PG4, the processing device PU4 functions as an image data acquisition unit IDO, an infrared image extraction unit IRE, a visible light image extraction unit VLE4, a depth information extraction unit DIE4, a distance detection unit DD, a scene detection unit SD, a processing unit IMP, and an output unit OT.

[0103] [4-2. Information Processing Method] FIGS. 13 and 14 are diagrams showing an example of the information processing method of the present embodiment. FIG. 13 is a conceptual diagram of information processing. FIG. 14 is a flowchart showing the information processing method.

[0104] In step S31, the plurality of cameras CA photograph a subject from a plurality of viewpoints. The image data acquisition unit IDO acquires a plurality of image data photographed from a plurality of viewpoints. The visible light image extraction unit VLE4 extracts a visible light image from the plurality of image data for each piece of image data using the visible light image information.

[0105] In step S32, the scene detection unit SD detects a shooting scene based on one of the plurality of visible light images extracted by the visible light image extraction unit VLE4.

[0106] In step S33, the depth information extraction unit DIE4 determines whether "daytime & outdoors" has been detected as the shooting scene. In step S33, if it is determined that "daytime & outdoors" has been detected (step S33: Yes), the process proceeds to step S34. In step S34, the depth information extraction unit DIE3 selects an outdoor control mode. Then, the process proceeds to step S36.

[0107] In step S33, if it is determined that "daytime & outdoors" has not been detected (step S33: No), the process proceeds to step S35. In step S35, the depth information extraction unit DIE4 selects an indoor control mode. Then, the process proceeds to step S36.

[0108] In step S36, the distance detection unit DD uses the plurality of visible light images extracted from the plurality of image data to extract the depth information of some or all of the measurement points in the captured image of the camera CA by the passive stereo method. The distance detection unit DD detects the distance between the camera CA and the subject using the extracted depth information.

[0109] In step S37, the depth information extraction unit DIE4 determines whether the distance detected by the distance detection unit DD is greater than a threshold value. The threshold value used as the criterion in step S37 is different when the outdoor control mode is selected and when the indoor control mode is selected. The threshold value when the outdoor control mode is selected is the first threshold value. The threshold value when the indoor control mode is selected is the second threshold value. The first threshold value is smaller than the second threshold value.

[0110] In step S37, if it is determined that the distance is greater than the threshold value (step S37: Yes), the process proceeds to step S38. In step S38, the depth information extraction unit DIE4 selects the passive stereo mode. The depth information extraction unit DIE4 extracts depth information in the passive stereo method using a plurality of visible light images extracted by the visible light image extraction unit VLE4. Then, the process proceeds to step S40. If the distance detection unit DD extracts the depth information of all the measurement points in the captured image of the camera CA in step S36, the depth information extraction unit DIE4 outputs the depth information extracted by the distance detection unit DD as it is to the processing unit IMP and the output unit OT.

[0111] In step S37, if it is determined that the distance is less than or equal to the threshold value (step S37: No), the process proceeds to step S39. In step S39, the depth information extraction unit DIE4 selects the active stereo mode. The depth information extraction unit DIE4 extracts depth information in the active stereo method using a plurality of infrared images extracted by the infrared image extraction unit IRE. Then, the process proceeds to step S40.

[0112] In step S40, the processing unit IMP performs preprocessing on the visible light image acquired from the visible light image extraction unit VLE4. This visible light image is a reference image generated using the visible light image information included in the image data of the first camera CA1 (reference camera).

[0113] In step S41, the processing unit IMP performs image processing based on depth information on the preprocessed visible light image.

[0114] [4-3. Effect] The depth information extraction unit DIE4 switches between the passive stereo mode and the active stereo mode according to the situation based on both the distance from the subject and the shooting scene. Therefore, depth information is accurately detected in various situations.

[0115] [5. Fifth Embodiment] FIG. 15 is a schematic diagram of the information processing apparatus IP5 according to the fifth embodiment. The difference from the fourth embodiment in this embodiment is that the distance information from the subject detected by the distance detection unit DD is used for controlling the infrared light projection pattern by the pattern control unit PTC shown in the second embodiment. Hereinafter, the description will focus on the differences from the second and fourth embodiments.

[0116] The processing device PU2 has a pattern control unit PTC. The function of the pattern control unit PTC is the same as that described in the second embodiment. The pattern control unit PTC changes the infrared light projection pattern IRP used in the active stereo mode according to the distance from the subject.

[0117] The distance detection unit DD detects the distance from the subject based on a plurality of visible light images extracted by the visible light image extraction unit VLE5. When the distance from the subject is greater than the threshold value, the pattern control unit PTC projects the long-distance pattern as the infrared light projection pattern IRP. The depth information extraction unit DIE5 extracts the depth information by the active stereo method using a plurality of infrared images in which the long-distance pattern is reflected. When the distance from the subject is equal to or less than the threshold value, the pattern control unit PTC projects the short-distance pattern as the infrared light projection pattern IRP. The depth information extraction unit DIE5 extracts the depth information by the active stereo method using a plurality of infrared images in which the short-distance pattern is reflected.

[0118] The distance condition (threshold value) for switching the infrared light projection pattern IRP is different between the outdoor control mode and the indoor control mode.

[0119] For example, when the outdoor control mode is selected, the following control is performed. First, when the distance from the subject is greater than the first threshold value, the pattern control unit PTC projects the long-distance pattern as the infrared light projection pattern IRP. When the distance from the subject is equal to or less than the first threshold value, the pattern control unit PTC projects the short-distance pattern as the infrared light projection pattern IRP.

[0120] For example, when the indoor control mode is selected, the following control is performed. First, when the distance from the subject is greater than the second threshold, the pattern control unit PTC projects a long-distance pattern as the infrared light projection pattern IRP. When the distance from the subject is equal to or less than the second threshold, the pattern control unit PTC projects a short-distance pattern as the infrared light projection pattern IRP.

[0121] The storage device ST5 stores, for example, a program PG5 executed by the processing device PU5. The program PG5 is a program that causes a computer to execute information processing according to the present embodiment. The processing device PU5 performs various processes according to the program PG5 stored in the storage device ST5. By executing the program PG5, the processing device PU5 functions as an image data acquisition unit IDO, an infrared image extraction unit IRE, a visible light image extraction unit VLE5, a depth information extraction unit DIE5, a distance detection unit DD, a scene detection unit SD, a processing unit IMP, an output unit OT, and a pattern control unit PTC.

[0122] In the present embodiment, in addition to the effects of the fourth embodiment, an effect is obtained in that the occurrence of aliasing is suppressed when the distance from the subject increases.

[0123] [6. Variations of the Pixel Array Unit] FIGS. 16 to 21 are diagrams showing variations of the pixel array unit PA.

[0124] FIG. 16 is a diagram showing a pixel array unit PA1 according to the first variation. The pixel array unit PA1 is the same as that shown in the first to fifth embodiments. The image sensor IS has a plurality of pixel blocks PB1 arranged two-dimensionally. Each of the plurality of pixel blocks PB1 has a structure in which one pixel PX1 for detecting red light, one pixel PX2 for detecting green light, one pixel PX3 for detecting blue light, and one pixel PX4 for detecting infrared rays are arranged in a 2×2 matrix.

[0125] According to this configuration, information on red, green, blue, and infrared light is detected in a well-balanced manner.

[0126] FIG. 17 is a diagram showing a pixel array unit PA2 according to a second variation. The image sensor IS has a structure in which a plurality of pixel blocks (first pixel blocks) PB2 and a plurality of pixel blocks (second pixel blocks) PB3 are periodically arranged in a two-dimensional direction. Each of the plurality of pixel blocks PB2 has a structure in which one pixel PX1 for detecting red light, one pixel PX2 for detecting green light, and two pixels PX4 for detecting infrared light are arranged in a 2×2 matrix. Each of the plurality of pixel blocks PB3 has a structure in which, for example, one pixel PX2 for detecting green light, one pixel PX3 for detecting blue light, and two pixels PX4 for detecting infrared light are arranged in a 2×2 matrix.

[0127] According to this configuration, pixels PX4 for detecting infrared light are arranged at a high density. Therefore, the resolution of infrared light is increased. Since the sensitivity of infrared light is also increased, the distance (threshold value) to a subject from which depth information can be extracted in the active stereo mode becomes larger. In addition, pixels PX1 for detecting red light, pixels PX2 for detecting green light, and pixels PX3 for detecting blue light are uniformly arranged at the same period. Therefore, information on red, blue, and green is detected in a well-balanced manner.

[0128] FIG. 18 is a diagram showing a pixel array unit PA3 according to a third variation. The image sensor IS has a plurality of pixel units PU1 arranged two-dimensionally. Each of the plurality of pixel units PU1 has a plurality of pixel blocks PB to which different colors are assigned. Each of the plurality of pixel blocks PB includes a plurality of pixels PX arranged adjacent to each other. The plurality of pixels PX constituting the pixel block PB detect light of the color assigned to this pixel block PB.

[0129] For example, the pixel unit PU1 has a structure in which the pixel blocks PB1, PB2, PB3, and PB4 are arranged in a 2x2 matrix. The pixel block PB1 is a pixel block to which red is assigned. In the pixel block PB1, four pixels PX1 for detecting red light are arranged in a 2x2 matrix. The pixel block PB2 is a pixel block to which green is assigned. In the pixel block PB2, four pixels PX2 for detecting green light are arranged in a 2x2 matrix. The pixel block PB3 is a pixel block to which blue is assigned. In the pixel block PB3, four pixels PX3 for detecting blue light are arranged in a 2x2 matrix. The pixel block PB4 is a pixel block to which infrared is assigned. In the pixel block PB4, four pixels PX4 for detecting infrared are arranged in a 2x2 matrix.

[0130] According to this configuration, the image sensor IS has a structure in which a plurality of pixel blocks PB1 to which red is assigned, a plurality of pixel blocks PB2 to which green is assigned, a plurality of pixel blocks PB3 to which blue is assigned, and a plurality of pixel blocks PB4 to which infrared is assigned are periodically arranged in a two-dimensional direction. Therefore, binning can be performed for each pixel block, and information on red, green, blue, and infrared can be detected with high sensitivity. Since the sensitivity to infrared also increases, the distance (threshold value) to the subject from which depth information can be extracted in the active stereo mode becomes larger. In addition, the pixels PX1 for detecting red light, the pixels PX2 for detecting green light, and the pixels PX3 for detecting blue light are uniformly arranged with the same period. Therefore, information on red, blue, and green is detected in a well-balanced manner.

[0131] FIG. 19 is a diagram showing a pixel array section PA4 according to a fourth variation. The image sensor IS has a plurality of pixel units PU2 arranged two-dimensionally. The pixel unit PU2 has a structure in which one pixel block PB2, one pixel block PB4, and two pixel blocks PB5 are arranged in a 2-row and 2-column manner. The pixel block PB5 has a structure in which one pixel PX1, one pixel PX2, and two pixels PX3 are arranged in a 2-row and 2-column manner. The pixel block PB2 and the pixel block PB4 are arranged so as not to be adjacent to each other in both the row direction and the column direction.

[0132] In this configuration, the number of pixels PX2 that detect green light is the largest. Green is the color with the highest visual sensitivity of the human eye. By increasing the number of pixels PX2, the apparent resolution is increased.

[0133] FIG. 20 is a diagram showing a pixel array section PA5 according to a fifth variation. The image sensor IS has a plurality of pixel units PU3 arranged two-dimensionally. The pixel unit PU3 has a structure in which one pixel block PB2, one pixel block PB4, and two pixel blocks PB5 are arranged in a 2-row and 2-column manner. The pixel block PB2 and the pixel block PB4 are arranged adjacent to each other in the column direction.

[0134] Also in this configuration, since the number of pixels PX2 is the largest, the apparent resolution is increased.

[0135] FIG. 21 is a diagram showing a pixel array section PA6 according to a sixth variation. The image sensor IS has a plurality of pixel units PU4 arranged two-dimensionally. The pixel unit PU4 has a structure in which one pixel block PB5, one pixel block PB6, one pixel block PB7, and one pixel block PB8 are arranged in a 2-row and 2-column manner.

[0136] The pixel block PB6 has a structure in which one pixel PX2 and three pixels PX4 are arranged in two rows and two columns. The pixel block PB7 has a structure in which two pixels PX2, one pixel PX3, and one pixel PX4 are arranged in two rows and two columns. The pixel block PB8 has a structure in which one pixel PX1, two pixels PX2, and one pixel PX4 are arranged in two rows and two columns.

[0137] In this configuration, the number of pixels PX2 for detecting green light and pixels PX4 for detecting infrared rays is the largest. Therefore, the sensitivity to infrared rays is high, and the apparent resolution for visible light images is also high. There is a region where five pixels PX4 are arranged in a cross shape. Therefore, by binning these five pixels PX, the sensitivity to infrared rays is further increased.

[0138] [7. Sixth Embodiment] [7-1. Configuration of Information Processing Apparatus] FIG. 22 is a schematic diagram of the information processing apparatus IP6 according to the sixth embodiment. The differences from the first embodiment in this embodiment are that the sensitivities to infrared rays of the plurality of cameras CA are different, the exposure times of the plurality of cameras CA are different according to the sensitivities to infrared rays, and the processing device PU6 has a synthesizing unit IMC that synthesizes a plurality of visible light images with different exposure times. Hereinafter, the description will be centered on the differences from the first embodiment.

[0139] In the first to fifth embodiments, all of the plurality of image sensors IS included in the plurality of cameras CA have the same structure. In this embodiment, the sensitivities to infrared rays of the plurality of image sensors IS are different from each other. For example, infrared cut filters are provided for pixels PX (PX5, PX6) for detecting infrared image information of one or more cameras CA. This infrared cut filter absorbs a part of the infrared rays incident on the pixels PX for detecting infrared image information.

[0140] The processing device PU6 has, for example, an exposure control unit ETC. The exposure control unit ETC varies the exposure times of the plurality of image sensors IS according to the infrared sensitivities of the respective plurality of image sensors IS. The exposure control unit ETC increases the exposure time for an image sensor IS with lower infrared sensitivity. Thereby, the exposure control unit ETC equalizes the brightness levels of the infrared images detected by the plurality of image sensors.

[0141] FIG. 23 is a diagram showing the relationship between the infrared transmittance and the exposure time of the first camera CA3 and the second camera CA4. FIG. 24 is a diagram showing the visible light exposure amounts of the first camera CA3 and the second camera CA4.

[0142] For example, the infrared transmittance detected by the light receiving element PD of the pixel PX6 of the second camera CA4 is less than the infrared transmittance detected by the light receiving element PD of the pixel PX5 of the first camera CA3. If the ratio of the infrared transmittances of the two cameras CA (infrared transmittance of the second camera CA4 / infrared transmittance of the first camera CA3) is, for example, Q, the exposure control unit ETC makes the exposure time of the second camera CA4 1 / Q times longer than the exposure time of the first camera CA3. Therefore, the detected value of the infrared rays of the pixel PX6 is equal to the detected value of the infrared rays of the pixel PX5. The visible light exposure amounts of the pixels PX1, PX2, and PX3 are larger for the second camera CA4 than for the first camera CA3.

[0143] Returning to FIG. 22, the image data acquisition unit IDO acquires a plurality of image data captured under different exposure conditions from the plurality of cameras CA. Each of the plurality of image data includes visible light image information and infrared image information. The image data acquisition unit IDO outputs the plurality of image data to the infrared image extraction unit IRE and the visible light image extraction unit VLE6.

[0144] The infrared image extraction unit IRE extracts infrared images from a plurality of image data, for each piece of image data, using infrared image information. The plurality of image data are captured under exposure conditions such that the detected value of the infrared ray of pixel PX5 is equal to the detected value of the infrared ray of pixel PX6. Therefore, the brightness levels of the plurality of infrared images extracted from the plurality of image data are equal to each other. The infrared image extraction unit IRE outputs the plurality of infrared images extracted from the plurality of image data to the depth information extraction unit DIE6. The depth information extraction unit DIE6 extracts depth information from the plurality of infrared images extracted by the infrared image extraction unit IRE in an active stereo method.

[0145] The visible light image extraction unit VLE6 extracts visible light images from a plurality of image data with different visible light exposure times, for each piece of image data, using visible light image information. The brightness levels of the plurality of extracted visible light images are different from each other, unlike in the first embodiment. The visible light image extracted from the image data of the second camera CA4 is an image with a high brightness level (long exposure image). The visible light image extracted from the image data of the first camera CA3 is an image with a low brightness level (short exposure image). Hereinafter, the second camera CA4 that acquires the long exposure image may be referred to as the long exposure camera, and the first camera CA3 that acquires the short exposure image may be referred to as the short exposure camera. The visible light image extraction unit VLE6 outputs the plurality of visible light images (long exposure image, short exposure image) extracted from the plurality of image data to the synthesis unit IMC.

[0146] The synthesis unit IMC synthesizes the plurality of visible light images (long exposure image, short exposure image) extracted from the plurality of image data. The synthesis unit IMC first detects the parallax of the plurality of cameras CA based on the plurality of visible light images. The synthesis unit IMC corrects the positional shift due to the parallax of the plurality of visible light images (warp processing). Next, the synthesis unit IMC synthesizes the plurality of visible light images whose positional shift due to parallax has been corrected (synthesis processing).

[0147] A long-exposure image is an image taken with a long exposure time. Therefore, the color reproducibility in the low-tone region is high. A short-exposure image is an image taken with a short exposure time. Therefore, the color reproducibility in the high-tone region is high. The synthesis unit IMC generates a visible light image (synthesized image) with a wide dynamic range based on the tone information of the low-tone region extracted from the long-exposure image and the tone information of the high-tone region extracted from the short-exposure image.

[0148] FIGS. 25 and 26 are diagrams for explaining an example of the warping process. FIG. 25 is a diagram showing a perspective projection model. FIG. 26 is a diagram for explaining the warping process.

[0149] As one of the warping processes, there is a method using a perspective projection model and depth information. With this method, it is possible to correct the misalignment without performing matching such as block matching.

[0150] The perspective projection model is a model for converting world coordinates (X W , Y W , Z W ) into image coordinates (u, v). In FIG. 25, P and (u, v) indicate the coordinates of the points projected onto the image plane. K indicates the internal parameter matrix. The internal parameter matrix K describes the optical system (lens) used for imaging. (C x , C y ) indicates the principal point (the position of the optical axis, usually at the center of the image). fk x , fk y indicates the focal length represented in pixel units. [R|T] indicates the external parameter matrix. The external parameter matrix [R|T] describes the location and orientation of the camera CA. R is a parameter representing the rotation of the camera CA. T is a parameter representing the translation of the camera CA.

[0151] The parameters of the internal parameter matrix K (intrinsic parameters) and the parameters of the external parameter matrix [R|T] (extrinsic parameters) can be estimated by using calibration charts photographed from multiple viewpoints (for example, refer to the Zhang method described in http: / / staff.fh-hagenberg.at / burger / publications / reports / 2016Calibration / Burger-CameraCalibration-20160516.pdf).

[0152] Using the intrinsic parameters and extrinsic parameters obtained by camera calibration, a plurality of visible light images can be oriented directly in front of the subject (frontalization). As a result, the epipolar line becomes horizontal, and the influence of parallax (the direction in which warping is required) is only in the horizontal direction (X-axis direction). In practice, it is necessary to remove the influence of lens distortion, but if it is assumed to be an image of a pinhole camera, the influence of lens distortion can be ignored.

[0153] When frontalization is performed using the intrinsic parameters and extrinsic parameters of the perspective projection model, if the depth information Z is known, using the triangulation method shown in FIG. 26, the parallax amount (X L -X R ) of a plurality of cameras CA can be obtained. By moving the visible light image extracted from the image data of the non-reference camera (the second camera CA4) by the parallax amount, a visible light image can be generated as if it were photographed from the same viewpoint as the reference image.

[0154] FIG. 27 is a conceptual diagram of the synthesis process.

[0155] The synthesis unit IMC performs synthesis using common methods (see, for example, "Radiometric Self Calibration", Tomoo Mitsunaga, etc.). The symbols Z1, Z2, Z3, ··· Zn represent pixel values. The symbol n is the number of cameras CA (the number of visible light images). For each pixel value, processing is performed to linearly convert the non-linear image signal by the camera response function CRF. Note that if the input is a linear signal, the processing by the camera response function CRF is unnecessary.

[0156] In the brightness level normalization, processing is performed to align the brightness to an arbitrary standard (long exposure image or short exposure image). By performing the normalization, the brightness levels of the long exposure image and the short exposure image are made uniform. As a result, the dynamic range is expanded downward in the long exposure image and upward in the short exposure image. The addition unit ITP performs the process of adding the normalized brightness levels E1, E2, ···, En according to the formula shown in FIG. 27. Thereby, the long exposure image and the short exposure image are synthesized, and a visible light image (synthesized image) with an expanded dynamic range is generated. The processing unit IMP processes the visible light image (synthesized image) generated by the synthesis unit IMC based on the depth information.

[0157] The storage device ST6 stores, for example, the program PG6 executed by the processing device PU6. The program PG6 is a program that causes a computer to execute the information processing according to the present embodiment. The processing device PU6 performs various processes according to the program PG6 stored in the storage device ST6. By executing the program PG6, the processing device PU6 functions as an image data acquisition unit IDO, an infrared image extraction unit IRE, a visible light image extraction unit VLE6, a depth information extraction unit DIE6, a synthesis unit IMC, a processing unit IMP, an output unit OT, and an exposure control unit ETC.

[0158] [7-2. Information Processing Method] FIGS. 28 and 29 are diagrams showing an example of the information processing method of the present embodiment. FIG. 28 is a conceptual diagram of information processing. FIG. 29 is a flowchart showing the information processing method.

[0159] In step S51, the exposure control unit ETC starts the exposure of the long exposure camera (second camera CA4). In step S52, the exposure control unit ETC starts the exposure of the short exposure camera (first camera CA3). Then, in step S53, the exposure control unit ETC stops the exposure of the long exposure camera and the short exposure camera.

[0160] The exposure control unit ETC varies the exposure times of the plurality of image sensors IS according to the infrared sensitivity of each of the plurality of image sensors IS. The exposure control unit ETC increases the exposure time of the long exposure camera with low infrared sensitivity so that the brightness level of the infrared image detected by the long exposure camera matches the brightness level of the infrared image detected by the short exposure camera.

[0161] The image data acquisition unit IDO acquires a plurality of image data captured by the plurality of cameras CA. The infrared image extraction unit IRE extracts an infrared image from each of the plurality of image data using infrared image information. The visible light image extraction unit VLE6 extracts a visible light image from each of the plurality of image data using visible light image information.

[0162] In step S54, the depth information extraction unit DIE6 extracts depth information in an active stereo method using the plurality of infrared images extracted by the infrared image extraction unit IRE.

[0163] In step S55, the synthesis unit IMC performs a warp process on the non-reference image to correct the positional shift due to the parallax of the plurality of visible light images. Then, in step S56, the synthesis unit IMC performs a synthesis process on the plurality of visible light images whose positional shift due to parallax has been corrected. After that, the processing unit IMP processes the visible light image (synthesized image) obtained by the synthesis process based on the depth information.

[0164] [7-3. Effect] The information processing apparatus IP6 includes a visible light image extraction unit VLE6 and a synthesis unit IMC. The visible light image extraction unit VLE6 extracts a visible light image from a plurality of image data with different exposure times of visible light for each piece of image data using visible light image information. The synthesis unit IMC synthesizes a plurality of visible light images extracted from the plurality of image data.

[0165] According to this configuration, a visible light image (synthesized image) with a wide dynamic range is generated.

[0166] The infrared sensitivities of the plurality of image sensors IS are different from each other. The information processing apparatus IP6 includes an exposure control unit ETC. The exposure control unit ETC varies the exposure times of the plurality of image sensors IS according to the infrared sensitivity of each of the plurality of image sensors IS.

[0167] According to this configuration, with the execution of the active stereo mode, a plurality of visible light image information with different exposure times is acquired. Therefore, a visible light image with a wide dynamic range is easily generated.

[0168] [8. Seventh Embodiment] [8-1. Configuration of Information Processing Apparatus] FIG. 30 is a schematic diagram of the information processing apparatus IP7 according to the seventh embodiment. The difference from the first embodiment in this embodiment is that a plurality of pixels PX that detect both visible light and infrared light are two-dimensionally arranged. A special pixel PX (the pixel PX4 in the first embodiment) for detecting infrared light is not provided, and infrared light is detected by all the pixels PX. Hereinafter, the description will be centered on the differences from the first embodiment.

[0169] The image sensor IS has, for example, a plurality of pixel blocks PB two-dimensionally arranged. Each of the plurality of pixel blocks PB has a structure in which one pixel PX7, two pixels PX8, and one pixel PX9 are arranged in two rows and two columns. The pixel PX7 detects, for example, red light and infrared light. The pixel PX8 detects, for example, green light and infrared light. The pixel PX9 detects, for example, blue light and infrared light.

[0170] The image data acquisition unit IDO acquires a plurality of image data captured from a plurality of cameras CA at a plurality of viewpoints. Each of the plurality of image data includes information regarding the total amount of received visible light and infrared light for each pixel PX as visible light image information and infrared light image information. The image data acquisition unit IDO outputs the plurality of image data to the luminance image extraction unit BIE and the visible light image extraction unit VLE7.

[0171] Depth information is extracted, for example, from a plurality of luminance images showing the distribution of the total amount of received light. The processing device PU7 has a luminance image extraction unit BIE instead of the infrared light image extraction unit IRE used in the first embodiment.

[0172] The luminance image extraction unit BIE extracts, for example, from a plurality of image data, a luminance image including both infrared light image information and visible light image information for each image data. The luminance image includes only luminance information indicating the detection value of each pixel PX and does not include color information. The luminance image extraction unit BIE outputs a plurality of luminance images extracted from the plurality of image data to the depth information extraction unit DIE7.

[0173] The depth information extraction unit DIE7 extracts depth information from the plurality of luminance images output from the luminance image extraction unit BIE. The luminance image includes infrared light image information indicating an infrared light projection pattern. The luminance image includes the detection value of visible light as a noise component. However, if the infrared light intensity of the projector PJ is high, the detection value of infrared light becomes larger than the detection value of visible light, and the shape of the infrared light projection pattern is clearly reflected in the luminance image. Therefore, the depth information extraction unit DIE7 can extract depth information from the plurality of infrared light image information included in the plurality of image data. The depth information extraction unit DIE7 outputs the depth information as a depth map to the visible light image extraction unit VLE7, the processing unit IMP, and the output unit OT.

[0174] The visible light image extraction unit VLE7 separates, for example, infrared image information and visible light image information. The visible light image extraction unit VLE7 extracts a visible light image from the visible light image information obtained by separation. The visible light image extraction unit VLE7 extracts, for example, a visible light image for each piece of image data from a plurality of pieces of image data. The visible light image extraction unit VLE7 outputs at least one visible light image among the plurality of visible light images extracted from the plurality of pieces of image data to the processing unit IMP.

[0175] For example, the visible light image extraction unit VLE7 estimates distribution information of an infrared projection pattern reflected in an image using depth information and correction information CI. The visible light image extraction unit VLE7 separates the infrared image information and the visible light image information based on the distribution information.

[0176] The correction information CI includes, for example, calibration information between the camera CA and the projector PJ (including information regarding the focal length and baseline length of the camera CA). The correction information CI includes, for example, information regarding the attenuation and scattering modes of infrared rays according to the distance. The correction information CI includes, for example, information on the infrared projection pattern (including information regarding the shape and position of the infrared projection pattern). The correction information CI includes, for example, information regarding the deterioration process such as blurring of the infrared projection pattern by the lens of the projector PJ. The correction information CI includes, for example, information on a color conversion matrix for correcting color shift caused by ambient light.

[0177] For example, the visible light image extraction unit VLE7 estimates the position where the infrared light projection pattern should be projected using depth information and calibration information. The position is specified using the triangulation method shown in FIG. 26. The projector PJ can also be treated as having the same perspective projection model as the camera CA. Therefore, the infrared light projection pattern of the projector PJ can be warped to the viewpoint of the reference camera in the same way as the warping process described in the sixth embodiment. However, in the case of the projector PJ, unlike the camera CA, it is not possible to directly estimate the parameters by photographing the pattern board. Therefore, for example, calibration is performed indirectly using the camera CA by the method disclosed at https: / / www.jstage.jst.go.jp / article / itej / 62 / 12 / 62_12_1964 / _pdf / -char / ja.

[0178] The visible light image extraction unit VLE7 estimates the shape of the infrared light projection pattern reflected in the image, taking into account, for example, the power of the projector PJ, the attenuation and scattering modes of infrared rays due to distance, and the degradation process such as blurring by the lens. The visible light image extraction unit VLE7 estimates the position and shape of the infrared light projection pattern obtained by calculation as the distribution information of the infrared light projection pattern.

[0179] FIGS. 31 and 32 are diagrams for explaining a method of imparting blurring (degradation) to a dot pattern.

[0180] The user measures in advance the point spread function (PSF) of the projector PJ. The PSF changes in shape for each image height of the lens of the projector PJ. Therefore, the user measures for each image height. If only the first quadrant is measured, symmetric values can be used for the remaining three quadrants. By performing convolution integration on the measured PSF and the infrared light projection pattern projected by the projector PJ, a blurred image can be reproduced. As shown in FIG. 31, the blurring is small at the center of the lens, but becomes large at the peripheral portion of the lens, and the shape of the blurring also becomes peculiar.

[0181] The visible light image extraction unit VLE7 performs correction processing on, for example, the visible light image information separated from the infrared image information. The correction processing is processing for correcting color deviation caused by infrared rays included in the ambient light.

[0182] FIG. 33 is an explanatory diagram of the correction processing.

[0183] The pixel PX detects the total amount of received light of visible light and infrared rays. Therefore, when demosaicing processing is performed on the detection value of each pixel PX, the color values of red, green, and blue of each pixel PX are increased by the detection value of infrared rays. Even if the infrared component derived from the infrared projection pattern is separated by the above-described processing, the infrared component derived from the ambient light is not separated. Therefore, the visible light image extraction unit VLE7 corrects the color deviation derived from the ambient light using a color conversion matrix.

[0184] FIG. 34 is a diagram showing an example of a method for calculating the color conversion matrix.

[0185] The color conversion matrix is calculated, for example, by the following method. First, the user photographs a Macbeth chart whose correct color is known with the camera CA. The user uses a computer to obtain a color conversion matrix having parameters ω0 to ω8 such that the mean square error between the detected value and the correct value is minimized.

[0186] Returning to FIG. 30, the storage device ST7 stores, for example, a program PG7 executed by the processing device PU7 and correction information CI. The program PG7 is a program for causing a computer to execute information processing according to the present embodiment. The processing device PU7 performs various processes according to the program PG7 stored in the storage device ST7. By executing the program PG7, the processing device PU7 functions as an image data acquisition unit IDO, a luminance image extraction unit BIE, a visible light image extraction unit VLE7, a depth information extraction unit DIE7, a processing unit IMP, and an output unit OT.

[0187] [8-2. Information Processing Method] FIG. 35 and FIG. 36 are diagrams showing an example of the information processing method of the present embodiment. FIG. 35 is a conceptual diagram of information processing. FIG. 36 is a flowchart showing the information processing method.

[0188] In step S61, a plurality of cameras CA photograph a subject from a plurality of viewpoints. The image data acquisition unit IDO acquires a plurality of image data captured from a plurality of viewpoints.

[0189] In step S62, the luminance image extraction unit BIE extracts a luminance image including both infrared image information and visible light image information for each piece of image data from the plurality of image data. The depth information extraction unit DIE7 extracts depth information by an active stereo method from the plurality of luminance images extracted from the plurality of image data.

[0190] In step S63, the visible light image extraction unit VLE7 acquires correction information CI from the storage device ST7.

[0191] In step S64, the visible light image extraction unit VLE7 estimates a position where an infrared light projection pattern should be projected based on the depth information and the correction information CI.

[0192] In step S65, the visible light image extraction unit VLE7 superimposes an infrared light projection pattern on the position estimated in step S64 in the image of the reference camera using the information on the infrared light projection pattern included in the correction information CI.

[0193] In step S66, the visible light image extraction unit VLE7 estimates distribution information of the infrared light projection pattern by applying a degradation model to the infrared light projection pattern using the information on the degradation process included in the correction information CI.

[0194] In step S67, the visible light image extraction unit VLE7 separates the visible light image information and the infrared image information included in the image data based on the distribution information estimated in step S66. The visible light image extraction unit VLE7 performs a correction process for correcting the color shift caused by the infrared rays included in the ambient light on the visible light image information separated from the infrared image information. The visible light image extraction unit VLE7 generates a visible light image using the visible light image information separated from the infrared image information.

[0195] In step S68, the processing unit IMP performs preprocessing on the visible light image acquired from the visible light image extraction unit VLE1. This visible light image is a reference image generated using the visible light image information included in the image data of the first camera CA5 (reference camera).

[0196] In step S69, the processing unit IMP performs image processing based on the depth information on the preprocessed visible light image.

[0197] [8-3. Effect] Each of the plurality of image data includes information on the total received light amount of visible light and infrared rays for each pixel as visible light image information and infrared image information.

[0198] According to this configuration, infrared rays are detected in all pixels. Therefore, the sensitivity to infrared rays is increased. Since the density of the pixels that detect infrared rays is high, aliasing is also less likely to occur.

[0199] The visible light image extraction unit VLE7 separates the infrared image information and the visible light image information. The visible light image extraction unit VLE7 extracts a visible light image from the separated visible light image information.

[0200] According to this configuration, a visible light image that does not include a noise component caused by the infrared image information can be obtained.

[0201] The visible light image extraction unit VLE7 estimates the distribution information of the infrared light projection pattern captured in the image. The visible light image extraction unit VLE7 separates the infrared light image information and the visible light image information based on the distribution information.

[0202] According to this configuration, the infrared light image information and the visible light image information are accurately separated.

[0203] The visible light image extraction unit VLE7 performs a correction process to correct the color shift caused by the infrared light included in the ambient light on the visible light image information separated from the infrared light image information.

[0204] According to this configuration, a visible light image with high color reproducibility can be obtained.

[0205] [9. Eighth Embodiment] [9-1. Configuration of Information Processing Apparatus] FIG. 37 is a schematic diagram of the information processing apparatus IP8 according to the eighth embodiment. The differences from the seventh embodiment in this embodiment are that the infrared sensitivities of the plurality of cameras CA are different, the exposure times of the plurality of cameras CA are different according to the infrared sensitivities, and the processing apparatus PU8 has a synthesis unit IMC that synthesizes a plurality of visible light images with different exposure times. The point of synthesizing a plurality of visible light images with different exposure times to generate a composite image with a wide dynamic range is the same as in the sixth embodiment. Hereinafter, the description will be centered on the differences from the sixth and seventh embodiments.

[0206] In the seventh embodiment, all of the plurality of image sensors IS included in the plurality of cameras CA have the same structure. In this embodiment, the infrared sensitivities of the plurality of image sensors IS are different from each other. For example, an infrared cut filter is provided for each pixel PX of one or more cameras CA. This infrared cut filter absorbs a part of the infrared light incident on the pixel PX.

[0207] The processing device PU8 has, for example, an exposure control unit ETC as disclosed in the sixth embodiment. The exposure control unit ETC varies the exposure times of the plurality of image sensors IS according to the infrared sensitivity of each of the plurality of image sensors IS. The exposure control unit ETC increases the exposure time for an image sensor IS with lower infrared sensitivity. Thereby, the exposure control unit ETC equalizes the brightness levels of the infrared images detected by the plurality of image sensors.

[0208] For example, the amount of infrared rays transmitted and detected by the light receiving element PD of the pixel PX of the second camera CA8 is less than the amount of infrared rays transmitted and detected by the light receiving element PD of the pixel PX of the first camera CA7. If the ratio of the infrared transmission amounts of the two cameras CA (infrared transmission amount of the second camera CA8 / infrared transmission amount of the first camera CA7) is, for example, Q, the exposure control unit ETC makes the exposure time of the second camera CA8 1 / Q times longer than the exposure time of the first camera CA7. Therefore, the infrared detection value of the pixel PX is equal between the first camera CA1 and the second camera CA2. The amount of visible light exposure of the pixel PX is larger for the second camera CA8 than for the first camera CA7.

[0209] The image data acquisition unit IDO acquires a plurality of image data captured under different exposure conditions from a plurality of cameras CA. Each of the plurality of image data includes information on the total received light amounts of visible light and infrared rays for each pixel PX as visible light image information and infrared ray image information. The image data acquisition unit IDO outputs the plurality of image data to the luminance image extraction unit BIE and the visible light image extraction unit VLE8.

[0210] The luminance image extraction unit BIE extracts, for example, a luminance image including both infrared ray image information and visible light image information for each image data from the plurality of image data. The depth information extraction unit DIE8 extracts depth information from the plurality of luminance images output from the luminance image extraction unit BIE. The depth information extraction unit DIE8 outputs the depth information as a depth map to the visible light image extraction unit VLE8, the processing unit IMP, and the output unit OT.

[0211] The visible light image extraction unit VLE8 separates infrared image information and visible light image information by, for example, the method described in the seventh embodiment. The visible light image extraction unit VLE8 extracts a visible light image from the separated visible light image information. The visible light image extraction unit VLE7 extracts a visible light image for each piece of image data from, for example, a plurality of pieces of image data with different exposure times of visible light. The brightness levels of the plurality of extracted visible light images are different from each other. The visible light image extracted from the image data of the second camera CA8 is an image with a high brightness level (long exposure image). The visible light image extracted from the image data of the first camera CA7 is an image with a low brightness level (short exposure image). The visible light image extraction unit VLE6 outputs a plurality of visible light images (long exposure images, short exposure images) extracted from the plurality of pieces of image data to the synthesis unit IMC.

[0212] The synthesis unit IMC synthesizes a plurality of visible light images (long exposure images, short exposure images) extracted from the plurality of pieces of image data. The synthesis method is the same as that described in the sixth embodiment.

[0213] The storage device ST8 stores, for example, a program PG8 executed by the processing device PU8 and correction information CI. The program PG8 is a program that causes a computer to execute information processing according to the present embodiment. The processing device PU8 performs various processes according to the program PG8 stored in the storage device ST8. By executing the program PG8, the processing device PU8 functions as an image data acquisition unit IDO, a luminance image extraction unit BIE, a visible light image extraction unit VLE8, a depth information extraction unit DIE8, a synthesis unit IMC, a processing unit IMP, an output unit OT, and an exposure control unit ETC.

[0214] [9-2. Information Processing Method] FIGS. 38 and 39 are diagrams showing an example of the information processing method of the present embodiment. FIG. 38 is a conceptual diagram of information processing. FIG. 39 is a flowchart showing the information processing method.

[0215] In step S71, the exposure control unit ETC starts the exposure of the long exposure camera (second camera CA8). In step S72, the exposure control unit ETC starts the exposure of the short exposure camera (first camera CA7). Then, in step S73, the exposure control unit ETC stops the exposure of the long exposure camera and the short exposure camera.

[0216] The exposure control unit ETC varies the exposure times of the plurality of image sensors IS according to the infrared sensitivity of each of the plurality of image sensors IS. The exposure control unit ETC increases the exposure time of the long exposure camera with low infrared sensitivity to make the brightness level of the infrared image detected by the long exposure camera coincide with the brightness level of the infrared image detected by the short exposure camera.

[0217] In step S74, the image data acquisition unit IDO acquires a plurality of image data captured by the plurality of cameras CA. The luminance image extraction unit BIE extracts a luminance image including both infrared image information and visible light image information for each piece of image data from the plurality of image data. The depth information extraction unit DIE8 extracts depth information by an active stereo method from the plurality of luminance images extracted from the plurality of image data.

[0218] In step S75, the visible light image extraction unit VLE8 extracts a visible light image for each piece of image data from the plurality of image data with different visible light exposure times. First, the visible light image extraction unit VLE8 estimates the distribution information of the infrared projection pattern reflected in the image. The visible light image extraction unit VLE8 separates the infrared image information and the visible light image information included in the image data based on the distribution information. The visible light image extraction unit VLE8 performs a correction process for correcting the color shift caused by the infrared rays included in the ambient light on the visible light image information separated from the infrared image information. The visible light image extraction unit VLE8 extracts a visible light image from the visible light image information obtained by separation.

[0219] In step S76, the synthesis unit IMC performs warp processing on the non-reference image to correct the positional shift due to the parallax of the plurality of visible light images. Then, in step S77, the synthesis unit IMC performs synthesis processing on the plurality of visible light images whose positional shift due to parallax has been corrected. Thereafter, the processing unit IMP processes the visible light image (synthesized image) obtained by the synthesis processing based on the depth information.

[0220] [9-3. Effect] The visible light image extraction unit VLE8 extracts visible light images for each piece of image data from a plurality of pieces of image data having different exposure times of visible light. The synthesis unit IMC synthesizes the plurality of visible light images extracted from the plurality of pieces of image data.

[0221] According to this configuration, a visible light image with a wide dynamic range is generated.

[0222] The infrared sensitivities of the plurality of image sensors IS are different from each other. The exposure control unit ETC varies the exposure times of the plurality of image sensors IS according to the infrared sensitivity of each of the plurality of image sensors IS.

[0223] According to this configuration, with the execution of the active stereo mode, a plurality of visible light image information with different exposure times is acquired. Therefore, a visible light image with a wide dynamic range is easily generated.

[0224] Note that the effects described in this specification are merely examples and are not limiting, and there may be other effects.

[0225] Note that the present technology can also adopt the following configuration.

[0226] (1) A depth information extraction unit capable of extracting depth information from a plurality of infrared image information included in a plurality of pieces of image data captured from a plurality of viewpoints, including visible light image information and infrared image information. A processing unit that processes a visible light image generated using the visible light image information included in at least one of the plurality of image data based on the depth information; An information processing apparatus having the same. (2) The depth information extraction unit switches between a passive stereo mode for extracting the depth information from a plurality of visible light image information included in the plurality of image data and an active stereo mode for extracting the depth information from a plurality of infrared image information included in the plurality of image data according to the situation. The information processing apparatus according to (1) above. (3) The depth information extraction unit switches between the passive stereo mode and the active stereo mode according to the situation based on the distance from the subject. The information processing apparatus according to (2) above. (4) The depth information extraction unit switches between the passive stereo mode and the active stereo mode according to the situation based on the shooting scene. The information processing apparatus according to (2) or (3) above. (5) It has a pattern control unit that changes the infrared projection pattern used in the active stereo mode according to the distance from the subject. The information processing apparatus according to any one of (2) to (4) above. (6) It has a plurality of image sensors that respectively capture the plurality of image data, Each of the plurality of image sensors has a structure in which a plurality of pixels for detecting the visible light image information and a plurality of pixels for detecting the infrared image information are periodically arranged in a two-dimensional direction. The information processing apparatus according to any one of (1) to (5) above. (7) Each of the plurality of image sensors has a plurality of pixel blocks arranged two-dimensionally, Each of the plurality of pixel blocks has a structure in which one pixel for detecting red light, one pixel for detecting green light, one pixel for detecting blue light, and one pixel for detecting infrared rays are arranged in a 2x2 matrix. The information processing apparatus according to (6) above. (8) Each of the plurality of image sensors has a structure in which a plurality of first pixel blocks and a plurality of second pixel blocks are periodically arranged in a two-dimensional direction. Each of the plurality of first pixel blocks has a structure in which one pixel for detecting red light, one pixel for detecting green light, and two pixels for detecting infrared rays are arranged in a 2x2 matrix. Each of the plurality of second pixel blocks has a structure in which one pixel for detecting green light, one pixel for detecting blue light, and two pixels for detecting infrared rays are arranged in a 2x2 matrix. The information processing apparatus according to (6) above. (9) Each of the plurality of image sensors has a plurality of pixel blocks for detecting infrared rays. Each of the plurality of pixel blocks for detecting infrared rays has a structure in which a plurality of pixels for detecting infrared rays are arranged adjacent to each other. The information processing apparatus according to (6) above. (10) Each of the plurality of image sensors has a plurality of pixel blocks arranged two-dimensionally. Each of the plurality of pixel blocks includes a plurality of pixels arranged adjacent to each other. Each of the plurality of image sensors has a structure in which a plurality of pixel blocks assigned red, a plurality of pixel blocks assigned green, a plurality of pixel blocks assigned blue, and a plurality of pixel blocks assigned infrared rays are periodically arranged in a two-dimensional direction. The information processing apparatus according to (6) above. (11) A visible light image extraction unit that extracts a visible light image using the visible light image information for each of the plurality of image data from the plurality of image data having different exposure times of visible light. A synthesizing unit that synthesizes a plurality of visible light images extracted from the plurality of image data; The information processing apparatus according to (1) above, which has the above. (12) Having a plurality of image sensors that respectively capture the plurality of image data, Each of the plurality of image sensors has a structure in which a plurality of pixels for detecting visible light image information and a plurality of pixels for detecting infrared image information are periodically arranged in a two-dimensional direction, The infrared sensitivities of the plurality of image sensors are different from each other, Having an exposure control unit that varies the exposure times of the plurality of image sensors according to the infrared sensitivity of each of the plurality of image sensors The information processing apparatus according to (11) above. (13) Each of the plurality of image data includes information on the total amount of received visible light and infrared light for each pixel as the visible light image information and the infrared light image information. The information processing apparatus according to (1) above. (14) Having a visible light image extraction unit that separates the infrared light image information and the visible light image information and extracts a visible light image from the separated visible light image information. The information processing apparatus according to (13) above. (15) The visible light image extraction unit estimates the distribution information of the infrared light projection pattern reflected in the image, and separates the infrared light image information and the visible light image information based on the distribution information. The information processing apparatus according to (14) above. (16) The visible light image extraction unit extracts the visible light image for each image data from the plurality of image data having different exposure times of visible light, Having a synthesizing unit that synthesizes a plurality of visible light images extracted from the plurality of image data. The information processing apparatus according to (15) above. (17) Having a plurality of image sensors that respectively capture the plurality of image data, Each of the plurality of image sensors has a structure in which a plurality of pixels for detecting both visible light and infrared light are two-dimensionally arranged. The infrared sensitivities of the plurality of image sensors are different from each other. It has an exposure control unit that varies the exposure times of the plurality of image sensors according to the infrared sensitivity of each of the plurality of image sensors. The information processing apparatus according to (16) above. (18) The visible light image extraction unit performs correction processing for correcting color deviation caused by infrared light included in ambient light on the visible light image information separated from the infrared light image information. The information processing apparatus according to (17) above. (19) Obtain a plurality of image data captured from a plurality of viewpoints, including visible light image information and infrared light image information. Extract depth information from the plurality of infrared light image information included in the plurality of image data. Process a visible light image generated using the visible light image information included in at least one of the plurality of image data based on the depth information. An information processing method executed by a computer, which includes the above. (20) Obtain a plurality of image data captured from a plurality of viewpoints, including visible light image information and infrared light image information. Extract depth information from the plurality of infrared light image information included in the plurality of image data. Process a visible light image generated using the visible light image information included in at least one of the plurality of image data based on the depth information. A program that causes a computer to realize the above.

Explanation of Signs

[0227] DIE1, DIE2, DIE3, DIE4, DIE5, DIE6, DIE7, DIE8 Depth information extraction unit ETC Exposure control unit IMC Composition unit IMP Processing Unit IP1, IP2, IP3, IP4, IP5, IP6, IP7, IP8 Information Processing Devices IS Image Sensor PB Pixel Block PTC Pattern Control Unit PX Pixel VLE1, VLE2, VLE3, VLE4, VLE5, VLE6, VLE7, VLE8 Visible Light Image Extraction Units

Claims

1. A depth information extraction unit capable of extracting depth information from a plurality of infrared image information included in a plurality of image data captured from a plurality of viewpoints, including visible light image information and infrared image information; A processing unit that processes a visible light image generated using the visible light image information included in at least one of the plurality of image data based on the depth information; having; The depth information extraction unit switches between a passive stereo mode for extracting the depth information from a plurality of visible light image information included in the plurality of image data and an active stereo mode for extracting the depth information from a plurality of infrared image information included in the plurality of image data according to the situation. An information processing apparatus.

2. The depth information extraction unit switches between the passive stereo mode and the active stereo mode according to a situation based on the distance from the subject. The information processing apparatus according to claim 1.

3. The depth information extraction unit switches between the passive stereo mode and the active stereo mode according to a situation based on the shooting scene. The information processing apparatus according to claim 1.

4. having a pattern control unit that changes the infrared projection pattern used in the active stereo mode according to the distance from the subject. The information processing apparatus according to claim 1.

5. having a plurality of image sensors that respectively capture the plurality of image data, Each of the plurality of image sensors has a structure in which a plurality of pixels for detecting the visible light image information and a plurality of pixels for detecting the infrared image information are periodically arranged in a two-dimensional direction. The information processing apparatus according to claim 1.

6. Each of the plurality of image sensors has a plurality of pixel blocks arranged two-dimensionally, Each of the plurality of pixel blocks has a structure in which one pixel for detecting red light, one pixel for detecting green light, one pixel for detecting blue light, and one pixel for detecting infrared rays are arranged in a 2-row 2-column configuration. The information processing apparatus according to claim 5.

7. Each of the plurality of image sensors has a structure in which a plurality of first pixel blocks and a plurality of second pixel blocks are periodically arranged in a two-dimensional direction, Each of the plurality of first pixel blocks has a structure in which one pixel for detecting red light, one pixel for detecting green light, and two pixels for detecting infrared rays are arranged in a 2-row 2-column configuration. Each of the plurality of second pixel blocks has a structure in which one pixel for detecting green light, one pixel for detecting blue light, and two pixels for detecting infrared rays are arranged in a 2x2 matrix. The information processing apparatus according to claim 5.

8. Each of the plurality of image sensors has a plurality of pixel blocks for detecting infrared rays. Each of the plurality of pixel blocks for detecting infrared rays has a structure in which a plurality of pixels for detecting infrared rays are arranged adjacent to each other. The information processing apparatus according to claim 5.

9. Each of the plurality of image sensors has a plurality of pixel blocks arranged two-dimensionally. Each of the plurality of pixel blocks includes a plurality of pixels arranged adjacent to each other. Each of the plurality of image sensors has a structure in which a plurality of pixel blocks assigned red, a plurality of pixel blocks assigned green, a plurality of pixel blocks assigned blue, and a plurality of pixel blocks assigned infrared rays are periodically arranged in a two-dimensional direction. The information processing apparatus according to claim 5.

10. A depth information extraction unit capable of extracting depth information from a plurality of infrared image information included in a plurality of image data captured from a plurality of viewpoints, the plurality of image data including visible light image information and infrared image information; A processing unit that processes a visible light image generated using the visible light image information included in at least one of the plurality of image data based on the depth information; A visible light image extraction unit that extracts a visible light image using the visible light image information for each of the plurality of image data from the plurality of image data having different exposure times for visible light; A synthesis unit that synthesizes a plurality of visible light images extracted from the plurality of image data; An information processing apparatus having the above components.

11. The information processing apparatus has a plurality of image sensors that respectively capture the plurality of image data. Each of the plurality of image sensors has a structure in which a plurality of pixels for detecting visible light image information and a plurality of pixels for detecting infrared ray image information are periodically arranged in a two-dimensional direction. The infrared sensitivities of the plurality of image sensors are different from each other. The information processing apparatus has an exposure control unit that varies the exposure times of the plurality of image sensors according to the respective infrared sensitivities of the plurality of image sensors. The information processing apparatus according to claim 10.

12. A depth information extraction unit that can extract depth information from a plurality of infrared image information included in a plurality of image data captured from a plurality of viewpoints, including visible light image information and infrared image information; A processing unit that processes a visible light image generated using the visible light image information included in at least one of the plurality of image data based on the depth information; It has, Each of the plurality of image data includes information regarding the total amount of received light of visible light and infrared light for each pixel as the visible light image information and the infrared image information; It has a visible light image extraction unit that separates the infrared image information and the visible light image information, and extracts a visible light image from the separated visible light image information; The visible light image extraction unit estimates distribution information of an infrared light projection pattern reflected in an image, and separates the infrared image information and the visible light image information based on the distribution information. An information processing apparatus.

13. The visible light image extraction unit extracts the visible light image for each image data from the plurality of image data having different exposure times of visible light; It has a composition unit that composes a plurality of visible light images extracted from the plurality of image data. The information processing apparatus according to Claim 12.

14. It has a plurality of image sensors that respectively capture the plurality of image data; Each of the plurality of image sensors has a structure in which a plurality of pixels that detect both visible light and infrared light are two-dimensionally arranged; The infrared sensitivities of the plurality of image sensors are different from each other; It has an exposure control unit that varies the exposure times of the plurality of image sensors according to the infrared sensitivity of each of the plurality of image sensors. The information processing apparatus according to Claim 13.

15. The visible light image extraction unit performs a correction process for correcting color deviation caused by infrared light included in ambient light on the visible light image information separated from the infrared image information. The information processing apparatus according to Claim 14.

16. Acquire a plurality of image data captured from a plurality of viewpoints, including visible light image information and infrared image information; Extract depth information from a plurality of infrared image information included in the plurality of image data; Process a visible light image generated using the visible light image information included in at least one of the plurality of image data based on the depth information; A passive stereo mode for extracting the depth information from a plurality of visible light image information included in the plurality of image data, and an active stereo mode for extracting the depth information from a plurality of infrared image information included in the plurality of image data are switched according to the situation. An information processing method executed by a computer, which has the above.

17. Acquire a plurality of image data captured from a plurality of viewpoints, including visible light image information and infrared image information. Extract depth information from a plurality of infrared image information included in the plurality of image data. Process a visible light image generated using the visible light image information included in at least one of the plurality of image data based on the depth information. A passive stereo mode for extracting the depth information from a plurality of visible light image information included in the plurality of image data, and an active stereo mode for extracting the depth information from a plurality of infrared image information included in the plurality of image data are switched according to the situation. A program for causing a computer to realize the above.

Images