Information processing method, information processor, and program

Abstract

To provide an information processing method that can obtain image depth.SOLUTION: In the information processing method, an IR image is obtained by capturing an image with reflected light from an object emitted from a light source 10, a BW image is obtained by capturing an image of the object with visible light, to detect a flare region based on the IR image and the BW image.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to a depth acquisition device that acquires the depth of an image. [Background technology]

[0002] Conventionally, a distance measuring device that measures the distance to a subject has been proposed (see, for example, Patent Document 1). This distance measuring device includes a light source and an imaging unit. The light source irradiates light onto the subject. The imaging unit captures the light reflected by the subject. The distance measuring device then measures the distance to the subject by converting each pixel value of the image obtained by the imaging into the distance to the subject. In other words, the distance measuring device obtains the depth of the image obtained by the imaging unit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-64498 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the distance measuring device of Patent Document 1 has a problem in that it is not possible to accurately obtain the depth of an image.

[0005] Therefore, the present disclosure provides a depth acquisition device that can accurately acquire the depth of an image. [Means for solving the problem]

[0006] In an information processing method according to one embodiment of the present disclosure, an IR image is obtained by capturing light irradiated from a light source and reflected by a subject, an image of the subject is captured using visible light to obtain a BW image, and a flare region is detected based on the IR image and the BW image.

[0007] An imaging device according to one embodiment of the present disclosure includes a light source that irradiates a subject with illumination light; a solid-state imaging element that performs a first image of the subject and a second image using light reflected by the subject from the illumination light; a flare detection unit that detects a flare area using subject information regarding the subject output by the solid-state imaging element; and an output unit that generates and outputs output information according to the detected flare area, wherein the first image and the second image are images taken from substantially the same viewpoint.

[0008] A depth acquisition device according to one aspect of the present disclosure includes a memory and a processor, wherein the processor acquires timing information indicating a timing at which a light source irradiates a subject with infrared light, acquires an infrared image obtained by imaging based on infrared light of a scene including the subject in accordance with the timing indicated by the timing information and stored in the memory, acquires a visible light image which is imaging based on visible light of substantially the same scene as the infrared image, the visible light image being obtained by imaging from substantially the same viewpoint and at substantially the same imaging time as the infrared image and stored in the memory, detects a flare region from the infrared image, and estimates a depth of the flare region based on the infrared image, the visible light image, and the flare region.

[0009] These comprehensive or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized by any combination of the system, the method, the integrated circuit, the computer program, and the recording medium. The recording medium may be a non-transitory recording medium. Effect of the Invention

[0010] The depth acquisition device of the present disclosure can accurately acquire the depth of an image. Further advantages and effects of one aspect of the present disclosure will be apparent from the specification and drawings. Such advantages and / or effects are provided by some embodiments and features described in the specification and drawings, respectively, but not all of them are necessarily provided to obtain one or more identical features. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing a hardware configuration of a depth acquisition device according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing a pixel array of the solid-state imaging device according to the embodiment. [Diagram 3] FIG. 3 is a timing chart showing the relationship between the light emission timing of the light emitting element of the light source and the exposure timing of the first pixel of the solid-state imaging element in the embodiment. [Figure 4] FIG. 4 is a block diagram illustrating an example of a functional configuration of the depth acquisition device according to the embodiment. [Diagram 5] FIG. 5 is a block diagram illustrating another example of the functional configuration of the depth acquisition device according to the embodiment. [Figure 6] FIG. 6 is a flowchart showing the overall processing operation of the depth acquisition device according to the embodiment. [Figure 7] FIG. 7 is a flowchart showing an overall processing operation by the processor of the depth acquisition device according to the embodiment. [Figure 8] FIG. 8 is a block diagram illustrating a specific functional configuration of a processor of the depth acquisition device according to the embodiment. [Figure 9A] FIG. 9A is a diagram showing an example of an IR image. [Figure 9B] FIG. 9B is a diagram showing an example of a BW image. [Figure 10] FIG. 10 is a diagram showing an example of a binarized image obtained by binarizing an IR image. [Figure 11]FIG. 11 is a diagram showing an example of a high luminance region in an IR image. [Figure 12] FIG. 12 is a diagram showing an area of ​​a BW image corresponding to a high luminance area of ​​an IR image. [Figure 13] FIG. 13 is a diagram showing an example of a flare region detected from an IR image. [Figure 14] FIG. 14 is a diagram showing a simulation result of the depth acquisition device according to the embodiment. [Figure 15] FIG. 15 is a flowchart showing the overall processing operation of the depth acquisition device shown in FIG. [Figure 16] FIG. 16 is a flowchart showing detailed processing of steps S31 to S34 in FIG. [Figure 17] FIG. 17 is a flow chart showing an example of a process instead of steps S31 to S34 in FIG. [Figure 18] FIG. 18 is a flowchart showing another example of processing instead of steps S31 to S34 in FIG. [Figure 19] FIG. 19 is a flowchart showing another example of processing instead of steps S31 to S34 in FIG. [Figure 20] FIG. 20 is a block diagram illustrating an example of a functional configuration of a depth acquisition device according to the first modification of the embodiment. As illustrated in FIG. [Figure 21] FIG. 21 is a block diagram illustrating another example of the functional configuration of the depth acquisition device according to the first modification of the embodiment. In FIG. [Figure 22] FIG. 22 is a flowchart showing a processing operation of the flare detection unit in the second modification of the embodiment. [Figure 23] FIG. 23 is a flowchart showing an overall processing operation of the depth acquisition device according to the third modification of the embodiment. [Figure 24] FIG. 24 is a flowchart showing the detailed processing of steps S31a to S34a in FIG. [Diagram 25] FIG. 25 is a flow chart showing an example of a process instead of steps S31a to S34a in FIG. [Figure 26] FIG. 26 is a flowchart showing another example of processing instead of steps S31a to S34a in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (Findings on which this disclosure is based) The present inventors have found that the distance measuring device of Patent Document 1 described in the "Background Art" section has the following problems.

[0013] As described above, the distance measuring device of Patent Document 1 irradiates a subject with light from a light source, captures an image of the subject irradiated with the light, and measures the depth of the image. Time of Flight (TOF) is used for measuring the depth. In such a distance measuring device, imaging is performed under different imaging conditions to improve the accuracy of distance measurement. That is, the distance measuring device captures an image according to a predetermined imaging condition, and sets an imaging condition different from the predetermined imaging condition depending on the imaging result. Then, the distance measuring device captures an image again according to the set imaging condition.

[0014] However, the image obtained by imaging may have flare, ghost, or saturation of brightness. The depth cannot be measured correctly only from the image in which the flare occurs. In addition, even if the imaging conditions are changed, it may be difficult to easily suppress the occurrence of flare. Furthermore, for example, if a distance measuring device mounted on a vehicle repeatedly captures images under different imaging conditions while the vehicle is running, the viewpoint positions of the repeated images are different, so that the scenes of the multiple images obtained are different. In other words, imaging cannot be repeated for the same scene, and the depth of the image in which the scene is displayed, especially the depth of the area in which the flare occurs, cannot be appropriately estimated.

[0015] In order to solve such a problem, a depth acquisition device according to one aspect of the present disclosure includes a memory and a processor, and the processor acquires timing information indicating a timing at which a light source irradiates an object with infrared light, acquires an infrared image obtained by imaging a scene including the object based on infrared light according to the timing indicated by the timing information and stored in the memory, acquires a visible light image of a scene substantially identical to the infrared image, which is obtained by imaging from substantially the same viewpoint and imaging time as the infrared image and stored in the memory, detects a flare region from the infrared image, and estimates the depth of the flare region based on the infrared image, the visible light image, and the flare region. Note that the flare region is a region where flare, ghost, saturation of brightness, or smear occurs.

[0016] As a result, a flare region is detected from the infrared image, and in the flare region, the depth is estimated based on not only the infrared image but also the visible light image, so that the depth of the flare region can be appropriately acquired. That is, the scene to be imaged is substantially the same in the infrared image and the visible light image, and the viewpoint and the image capturing time are also substantially the same. Here, an example of an image of a substantially identical scene captured at substantially the same viewpoint and image capturing time is an image captured at different pixels of the same image sensor. Such an image is similar to each of the red, green, and blue channel images of a color image captured with a Bayer array color filter, and the angle of view, viewpoint, and image capturing time of each image are almost equal. That is, in the images of a substantially identical scene captured at substantially the same viewpoint and image capturing time, the position of the subject on the image does not differ by more than two pixels in each captured image. For example, when a point light source having visible light and infrared components exists in the scene, and only one pixel is captured with high luminance in the visible light image, the point light source is also captured in the infrared image closer than two pixels to the pixel corresponding to the pixel position captured in the visible light image. Furthermore, substantially the same imaging time means that the difference between the imaging times is equal to one frame or less. Therefore, the infrared image and the visible light image have a high correlation. Furthermore, flare is a phenomenon that depends on wavelength, and even if flare occurs in the infrared image, there is a high possibility that flare does not occur in the visible light image. Therefore, information missing in the flare region can be compensated for from a region in the visible light image that corresponds to the flare region (i.e., a corresponding region). As a result, the depth of the flare region can be appropriately acquired.

[0017] For example, in estimating the depth of the flare region, the processor may estimate first depth information indicating the depth at each position in the infrared image, correct the depth at each position in the flare region indicated by the first depth information based on the visible light image, thereby estimating second depth information indicating the corrected depth at each position in the flare region, and further generate third depth information indicating the depth at each position outside the flare region of the infrared image indicated by the first depth information and the depth at each position in the flare region of the infrared image indicated by the second depth information. Note that, in estimating the first depth information, TOF or the like may be applied to the infrared image.

[0018] As a result, the third depth information indicates a depth obtained from the infrared image as a depth outside the flare region of the infrared image, and indicates a depth obtained from the infrared image and corrected based on the visible light image as a depth of the flare region of the infrared image. Therefore, even if the infrared image has a flare region, the entire depth of the infrared image can be appropriately estimated.

[0019] In addition, in detecting the flare region, the processor may detect a region in the infrared image having a brightness equal to or greater than a first threshold as the flare region.

[0020] Since the brightness within a flare region tends to be higher than the brightness outside the flare region, the flare region can be appropriately detected by detecting a region in the infrared image having a brightness equal to or greater than a first threshold as the flare region.

[0021] In addition, in detecting the flare region, the processor detects a region of the infrared image that has a brightness equal to or greater than a first threshold and satisfies a predetermined condition as the flare region, and the predetermined condition may be a condition in which a correlation value between an image feature amount in a region of the infrared image and an image feature amount in a region of the visible light image corresponding to the region is less than a second threshold. For example, the image feature amount in each region of the infrared image and the visible light image may be an edge included in an image in the region. Or, the image feature amount in each region of the infrared image and the visible light image may be a brightness in the region.

[0022] Since there is a tendency for the correlation between image features in a flare region of an infrared image and image features in a region of a visible light image corresponding to the flare region to be low, it is possible to more appropriately detect a flare region by detecting a region in the infrared image that has high brightness and low correlation between the image features as a flare region.

[0023] In addition, in detecting the flare region, the processor may (i) obtain a first converted image by performing a CENSUS conversion on an image within at least one high-brightness region in the infrared image having a brightness equal to or greater than a first threshold, and (ii) obtain a second converted image by performing a CENSUS conversion on an image within a region of the visible light image corresponding to the high-brightness region, and detect as the flare region a high-brightness region among the at least one high-brightness region in which the Hamming distance between the first converted image and the second converted image exceeds a third threshold.

[0024] This allows the flare region to be detected appropriately.

[0025] In addition, when estimating the depth of the flare region, the processor may estimate depth information indicating the depth at each position within the infrared image, and correct the depth at each position within the flare region indicated by the depth information by inputting the infrared image, the visible light image, the flare region, and the depth information into a learning model.

[0026] Thus, if the learning model is trained in advance so that the correct depth at each position in the flare region is output for the input of the infrared image, the visible light image, the flare region, and the depth information, the depth information estimated from the infrared image can be appropriately corrected. In other words, the depth at each position in the flare region indicated by the depth information can be appropriately corrected.

[0027] In addition, a depth acquisition device according to another aspect of the present disclosure may include a memory and a processor, wherein the processor acquires timing information indicating a timing at which a light source irradiates a subject with infrared light, acquires an infrared image obtained by imaging a scene including the subject based on infrared light in accordance with the timing indicated by the timing information and stored in the memory, acquires a visible light image that is imaging substantially the same scene as the infrared image based on visible light, obtained by imaging from substantially the same viewpoint and at substantially the same imaging time as the infrared image and stored in the memory, estimates depth information indicating a depth at each position within the infrared image, and corrects the depth at each position within a flare region of the infrared image indicated by the depth information by inputting the infrared image, the visible light image, and the depth information into a learning model.

[0028] Thus, if the learning model is trained in advance so that the correct depth at each position within the flare region of the infrared image is output in response to the input of the infrared image, the visible light image, and the depth information, the depth information estimated from the infrared image can be appropriately corrected. In other words, the depth at each position within the flare region indicated by the depth information can be appropriately corrected without detecting the flare region.

[0029] A depth acquisition device according to another aspect of the present disclosure includes a memory and a processor, the processor acquires an infrared image stored in the memory, where the infrared image is obtained by imaging based on infrared light, acquires a visible light image stored in the memory, where the visible light image is obtained by imaging based on visible light at substantially the same viewpoint and imaging time as the infrared image, detects an area in which a flare is projected from the infrared image as a flare area, and estimates a depth of the flare area based on the flare area. When each of the visible light image and the infrared image is divided into a flare area and other areas, the depth of the flare area is estimated based on the visible light image, and the depth of the other areas is estimated based on the infrared image.

[0030] This makes it possible to appropriately acquire the depth of the flare region, similar to the depth acquisition device according to the above aspect of the present disclosure.

[0031] These comprehensive or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized by any combination of the system, the method, the integrated circuit, the computer program, or the recording medium. The recording medium may be a non-transitory recording medium.

[0032] Hereinafter, the embodiment will be specifically described with reference to the drawings.

[0033] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component arrangement and connection forms, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in an independent claim showing a top concept are described as optional components.

[0034] In addition, each drawing is a schematic diagram and is not necessarily a precise illustration. In addition, in each drawing, the same components are denoted by the same reference numerals.

[0035] (Embodiment) [Hardware configuration] 1 is a block diagram showing a hardware configuration of a depth acquisition device 1 according to an embodiment. The depth acquisition device 1 in this embodiment has a hardware configuration capable of acquiring an image based on infrared light (or near-infrared light) and an image based on visible light by capturing images of substantially the same scene from substantially the same viewpoint and at substantially the same capturing time. Note that "substantially the same" means that the images are the same to the extent that the effects of the present disclosure can be achieved.

[0036] As shown in FIG. 1, the depth acquisition device 1 includes a light source 10, a solid-state imaging element 20, a processing circuit 30, a diffuser 50, a lens 60, and a band-pass filter .

[0037] The light source 10 emits illumination light. More specifically, the light source 10 emits illumination light to irradiate a subject at a timing indicated by a light emission signal generated by the processing circuit 30.

[0038] The light source 10 includes, for example, a capacitor, a driving circuit, and a light emitting element, and emits light by driving the light emitting element with electrical energy stored in the capacitor. The light emitting element is realized by, for example, a laser diode, a light emitting diode, etc. The light source 10 may include one type of light emitting element, or may include multiple types of light emitting elements according to the purpose.

[0039] In the following, the light emitting element is, for example, a laser diode that emits near-infrared light, or a light emitting diode that emits near-infrared light. However, the irradiation light irradiated by the light source 10 does not need to be limited to near-infrared light. The irradiation light irradiated by the light source 10 may be, for example, infrared light (also called infrared light) in a frequency band other than near-infrared light. Hereinafter, in this embodiment, the irradiation light irradiated by the light source 10 is described as infrared light, but the infrared light may be near-infrared light or infrared light in a frequency band other than near-infrared light.

[0040] The solid-state imaging element 20 captures an image of a subject and outputs an imaging signal indicating the amount of exposure. More specifically, the solid-state imaging element 20 performs exposure at a timing indicated by an exposure signal generated by the processing circuit 30, and outputs an imaging signal indicating the amount of exposure.

[0041] The solid-state imaging element 20 has a pixel array in which first pixels that capture an image using light reflected by a subject from irradiation light and second pixels that capture an image of the subject are arranged in an array. The solid-state imaging element 20 may have a logic function such as a cover glass, an AD converter, etc., as necessary.

[0042] In the following description, the reflected light is assumed to be infrared light, like the irradiated light; however, the reflected light does not have to be limited to infrared light, so long as it is light that is the irradiated light reflected by the subject.

[0043] FIG. 2 is a schematic diagram showing a pixel array 2 of the solid-state imaging device 20. As shown in FIG.

[0044] As shown in FIG. 2, the pixel array 2 is configured such that first pixels 21 (IR pixels) that capture images using light reflected by a subject from irradiated light and second pixels 22 (BW pixels) that capture images of the subject are arranged in an array in alternating columns.

[0045] 2, in the pixel array 2, the second pixels 22 and the first pixels 21 are arranged adjacent to each other in the row direction and arranged in a stripe pattern in the row direction, but the present invention is not limited to this and the pixels may be arranged every several rows (as an example, every two rows). That is, the first row in which the second pixels 22 are arranged adjacent to each other in the row direction and the second row in which the first pixels 21 are arranged adjacent to each other in the row direction may be arranged alternately every M rows (M is a natural number). Furthermore, the first row in which the second pixels 22 are arranged adjacent to each other in the row direction and the second row in which the first pixels 21 are arranged adjacent to each other in the row direction may be arranged at different intervals (the first row is alternately N rows and the second row is alternately L rows (N and L are different natural numbers)).

[0046] The first pixel 21 is realized by, for example, an infrared pixel having sensitivity to infrared light, which is reflected light, and the second pixel 22 is realized by, for example, a visible pixel having sensitivity to visible light.

[0047] The infrared light pixel includes, for example, an optical filter (also called an IR filter) that transmits only infrared light, a microlens, a light receiving element as a photoelectric conversion unit, and an accumulation unit that accumulates electric charges generated by the light receiving element. Therefore, an image showing the luminance of infrared light is expressed by imaging signals output from a plurality of infrared light pixels (i.e., first pixels 21) included in the pixel array 2. Hereinafter, this infrared light image is also called an IR image or an infrared image.

[0048] Also, the visible light pixel includes, for example, an optical filter (also called a BW filter) that transmits only visible light, a microlens, a light receiving element as a photoelectric conversion unit, and an accumulation unit that accumulates the electric charge converted by the light receiving element. Therefore, the visible light pixel, i.e., the second pixel 22, outputs an image signal indicating the luminance and color difference. That is, a color image indicating the luminance and color difference of visible light is expressed by the image signal output from the second pixels 22 included in the pixel array 2. Note that the optical filter of the visible light pixel may transmit both visible light and infrared light, or may transmit only light of a specific wavelength band such as red (R), green (G), or blue (B) of visible light.

[0049] Also, the visible light pixels may detect only the luminance of visible light. In this case, the visible light pixels, i.e., the second pixels 22, output imaging signals indicating the luminance. Therefore, a black and white image indicating the luminance of visible light, in other words, a monochrome image, is expressed by the imaging signals output from the second pixels 22 included in the pixel array 2. Hereinafter, this monochrome image is also referred to as a BW image. The above-mentioned color image and BW image are also collectively referred to as a visible light image.

[0050] Returning to FIG. 1 again, the description of the depth acquisition device 1 will continue.

[0051] The processing circuit 30 uses the imaging signal output by the solid-state imaging device 20 to calculate subject information relating to the subject.

[0052] The processing circuit 30 is configured, for example, by an arithmetic processing device such as a microcomputer. The microcomputer includes a processor (microprocessor), memory, etc., and generates a light emission signal and an exposure signal by executing a driving program stored in the memory by the processor. The processing circuit 30 may use an FPGA or an ISP, etc., and may be configured by one piece of hardware or multiple pieces of hardware.

[0053] The processing circuit 30 calculates the distance to the subject, for example, by a TOF distance measuring method using an imaging signal from the first pixel 21 of the solid-state imaging device 20.

[0054] Hereinafter, calculation of the distance to the subject by the TOF distance measurement method performed by the processing circuit 30 will be described with reference to the drawings.

[0055] FIG. 3 is a timing diagram showing the relationship between the light emission timing of the light-emitting element of the light source 10 and the exposure timing of the first pixel 21 of the solid-state imaging element 20 when the processing circuit 30 calculates the distance to the subject using the TOF distance measurement method.

[0056] 3, Tp is the light emission period during which the light emitting element of the light source 10 emits the irradiation light, and Td is the delay time from when the light emitting element of the light source 10 emits the irradiation light to when the reflected light of the irradiation light reflected by the subject returns to the solid-state imaging element 20. The first exposure period has the same timing as the light emission period during which the light source 10 emits the irradiation light, and the second exposure period has the timing from the end of the first exposure period until the light emission period Tp has elapsed.

[0057] In Figure 3, q1 indicates the total amount of exposure to reflected light at the first pixel 21 of the solid-state imaging element 20 during the first exposure period, and q2 indicates the total amount of exposure to reflected light at the first pixel 21 of the solid-state imaging element 20 during the second exposure period.

[0058] By emitting light by the light-emitting element of the light source 10 and exposing the first pixel 21 of the solid-state imaging element 20 at the timing shown in Figure 3, the distance d to the subject can be expressed by the following (Equation 1), where c is the speed of light.

[0059] d=c×Tp / 2×q2 / (q1+q2) (Formula 1)

[0060] Therefore, by utilizing (Equation 1), the processing circuit 30 can calculate the distance to the subject using the imaging signal from the first pixel 21 of the solid-state imaging device 20.

[0061] Furthermore, the first pixels 21 of the solid-state imaging device 20 may be exposed for only a third exposure period Tp after the first and second exposure periods are completed. The first pixels 21 can detect noise other than reflected light based on the amount of exposure obtained in the third exposure period Tp. In other words, the processing circuit 30 can more accurately calculate the distance d to the subject by removing noise from the amount of exposure q1 in the first exposure period and the amount of exposure q2 in the second exposure period in the above formula 1.

[0062] Returning to FIG. 1 again, the description of the depth acquisition device 1 will continue.

[0063] The processing circuit 30 may use, for example, an imaging signal from the second pixel 22 of the solid-state imaging device 20 to detect the subject and calculate the distance to the subject.

[0064] That is, the processing circuit 30 may detect the subject and calculate the distance to the subject based on the visible light image captured by the multiple second pixels 22 of the solid-state imaging device 20. Here, the subject may be detected, for example, by determining the shape by pattern recognition using edge detection of the singular points of the subject, or by processing such as deep learning using a learning model learned in advance. The distance to the subject may be calculated using world coordinate transformation. Of course, the subject may be detected by multimodal learning processing using not only the visible light image but also the luminance and distance information of the infrared light captured by the first pixels 21.

[0065] The processing circuit 30 generates a light emission signal indicating the timing of light emission and an exposure signal indicating the timing of exposure. Then, the processing circuit 30 outputs the generated light emission signal to the light source 10 and outputs the generated exposure signal to the solid-state imaging element 20.

[0066] The processing circuit 30 may, for example, generate and output a light emission signal to cause the light source 10 to emit light at a predetermined cycle, and generate and output an exposure signal to cause the solid-state imaging element 20 to be exposed at a predetermined cycle, thereby causing the depth acquisition device 1 to realize continuous imaging at a predetermined frame rate. The processing circuit 30 may also include, for example, a processor (microprocessor), memory, etc., and generate the light emission signal and the exposure signal by the processor executing a driving program stored in the memory.

[0067] The diffusion plate 50 adjusts the intensity distribution and angle of the irradiated light. In adjusting the intensity distribution, the diffusion plate 50 makes the intensity distribution of the irradiated light from the light source 10 uniform. Note that, although the depth acquisition device 1 includes the diffusion plate 50 in the example shown in FIG. 1, the diffusion plate 50 may not be included.

[0068] The lens 60 is an optical lens that focuses light entering from outside the depth acquisition device 1 onto the surface of the pixel array 2 of the solid-state imaging element 20.

[0069] The bandpass filter 70 is an optical filter that transmits infrared light, which is reflected light, and visible light. In the example shown in FIG. 1, the depth acquisition device 1 includes the bandpass filter 70, but the bandpass filter 70 may not be included.

[0070] The depth acquisition device 1 having the above configuration is mounted on a transport device and used. For example, the depth acquisition device 1 is mounted on a vehicle that travels on a road surface and used. Note that the transport device on which the depth acquisition device 1 is mounted is not necessarily limited to a vehicle. The depth acquisition device 1 may be mounted on a transport device other than a vehicle, such as a motorcycle, a boat, an airplane, etc.

[0071] [Outline of the depth acquisition device] The depth acquisition device 1 in this embodiment acquires an IR image and a BW image by capturing images of substantially the same scene from substantially the same viewpoint and at substantially the same time, using the hardware configuration shown in Fig. 1. The depth acquisition device 1 then corrects the depth at each position in the IR image obtained from the IR image, using the BW image. Specifically, when a flare region, which will be described later, exists in the IR image, the depth acquisition device 1 corrects the depth at each position in the flare region obtained from the IR image, using an image in the region of the BW image corresponding to the flare region.

[0072] FIG. 4 is a block diagram showing an example of a functional configuration of the depth acquisition device 1. As shown in FIG.

[0073] The depth acquisition device 1 includes a light source 101, an IR camera 102, a BW camera 103, a depth estimation unit 111, and a flare detection unit 112.

[0074] The light source 101 may be composed of the light source 10 and the diffusion plate 50 shown in FIG.

[0075] 1, a lens 60, and a bandpass filter 70. Such an IR camera 102 captures an IR image by capturing an image of a scene including a subject based on infrared light in response to the timing at which a light source 101 irradiates the subject with infrared light.

[0076] The BW camera 103 may be composed of a plurality of second pixels 22 of the solid-state imaging element 20 shown in Fig. 1, a lens 60, and a bandpass filter 70. Such a BW camera 103 captures an image based on visible light of a scene that is substantially the same as the infrared image, and obtains a visible light image (specifically, a BW image) by capturing an image from substantially the same viewpoint and at substantially the same capturing time as the infrared image.

[0077] The depth estimation unit 111 and the flare detection unit 112 may be realized as a function of the processing circuitry 30 shown in FIG.

[0078] The flare detection unit 112 detects a flare region from an IR image obtained by imaging using the IR camera 102 and a BW image obtained by imaging using the BW camera 103 .

[0079] The flare region in this embodiment is a region where flare, ghost, saturation of brightness, or smear occurs. Flare is a light fogging phenomenon that occurs when harmful light is reflected on the lens surface or lens barrel when the lens of the IR camera 102 is pointed toward a strong light source. Flare also makes the image whitish and reduces sharpness. Ghost is a type of flare, in which light that has been repeatedly reflected in a complex manner on the lens surface is clearly captured as an image. Smear is a phenomenon in which a linear white portion occurs when a subject that has a brightness difference of a certain level or more from the surroundings is photographed by a camera.

[0080] In this disclosure, a phenomenon including at least one of flare, ghost, luminance saturation, and smear is referred to as flare or the like.

[0081] The depth estimation unit 111 estimates the depth at each position in the IR image including the flare region detected by the flare detection unit 112. Specifically, the depth estimation unit 111 acquires an IR image captured by the IR camera 102 in accordance with the timing of irradiation of the subject with infrared light by the light source 101, and estimates the depth at each position in the IR image based on the IR image. Furthermore, the depth estimation unit 111 corrects the depth at each position estimated in the flare region detected by the flare detection unit 112 based on the BW image. That is, the depth estimation unit 111 estimates the depth of the flare region based on the IR image, the BW image, and the flare region.

[0082] FIG. 5 is a block diagram showing another example of the functional configuration of the depth acquisition device 1. As shown in FIG.

[0083] The depth acquisition device 1 may include a memory 200 and a processor 110.

[0084] 5, processor 110 may include not only depth estimation unit 111 and flare detection unit 112, but also light emission timing acquisition unit 113, IR image acquisition unit 114, and BW image acquisition unit 115. These components are realized as functions of processor 110.

[0085] The light emission timing acquisition unit 113 acquires timing information indicating the timing at which the light source 101 irradiates the subject with infrared light. That is, the light emission timing acquisition unit 113 outputs the light emission signal shown in Fig. 1 to the light source 101, and acquires information indicating the output timing as the above-mentioned timing information.

[0086] The IR image acquisition unit 114 acquires an IR image that is obtained by capturing an image of a scene including a subject based on infrared light in accordance with the timing indicated by the timing information and that is stored in the memory 200 .

[0087] The BW image acquisition unit 115 acquires a BW image that is captured based on visible light of substantially the same scene as the above-mentioned IR image, obtained by capturing an image from substantially the same viewpoint and at substantially the same capturing time as the IR image, and is stored in the memory 200.

[0088] As described above, the flare detection unit 112 detects a flare region from the IR image, and the depth estimation unit 111 estimates the depth based on the IR image, the BW image, and the flare region.

[0089] The depth acquisition device 1 in this embodiment may be configured with the processor 110 and the memory 200 without including the light source 101, the IR camera 102, and the BW camera 103.

[0090] FIG. 6 is a flowchart showing the overall processing operation of the depth acquisition device 1.

[0091] (Step S11) First, the light source 101 emits light to irradiate the subject with infrared light.

[0092] (Step S12) Next, the IR camera 102 acquires an IR image. That is, the IR camera 102 captures an image of a scene including a subject illuminated with infrared light by the light source 101. In this way, the IR camera 102 acquires an IR image based on the infrared light reflected from the subject. Specifically, the IR camera 102 acquires an IR image obtained by the respective timings and exposure amounts of the first exposure period, the second exposure period, and the third exposure period shown in FIG.

[0093] (Step S13) Next, the BW camera 103 acquires a BW image. That is, the BW camera 103 acquires a BW image corresponding to the IR image acquired in step S12, that is, a BW image of the same scene and the same viewpoint as the IR image.

[0094] (Step S14) Then, the flare detection unit 112 detects a flare region from the IR image acquired in step S12.

[0095] (Step S15) Next, the depth estimation unit 111 estimates the depth of the flare region based on the IR image acquired in step S12, the BW image acquired in step S13, and the flare region detected in step S14.

[0096] FIG. 7 is a flowchart showing the overall processing operation by the processor 110 of the depth acquisition device 1. As shown in FIG.

[0097] (Step S21) First, light emission timing acquisition section 113 of processor 110 acquires timing information indicating the timing at which light source 101 irradiates the subject with infrared light.

[0098] (Step S22) Next, the IR image acquisition unit 114 acquires an IR image from the IR camera 102 that has captured an image in accordance with the timing indicated by the timing information acquired in step S21. For example, the IR image acquisition unit 114 outputs an exposure signal to the IR camera 102 at the timing when the light emission signal shown in FIG. 1 is output from the light emission timing acquisition unit 113. This causes the IR image acquisition unit 114 to start capturing an image in the IR camera 102, and acquires an IR image obtained by the capturing from the IR camera 102. At this time, the IR image acquisition unit 114 may acquire the IR image from the IR camera 102 via the memory 200, or may acquire the IR image directly from the IR camera 102.

[0099] (Step S23) Next, the BW image acquisition unit 115 acquires a BW image corresponding to the IR image acquired in step S22 from the BW camera 103. At this time, the BW image acquisition unit 115 may acquire the BW image from the BW camera 103 via the memory 200, or may acquire the BW image directly from the BW camera 103.

[0100] (Step S24) Then, the flare detection unit 112 detects a flare region from the IR image.

[0101] (Step S25) Next, the depth estimation unit 111 estimates the depth of the flare region based on the IR image acquired in step S22, the BW image acquired in step S23, and the flare region detected in step S24. This allows calculation of depth information indicating at least the depth of the flare region. Note that at this time, the depth estimation unit 111 may estimate the depth of the entire IR image, not just the flare region, and calculate depth information indicating the estimation result.

[0102] Specifically, the depth estimation unit 111 in this embodiment estimates the depth at each position in the IR image acquired in step S22. Then, the depth estimation unit 111 corrects the depth at each position in the flare region using the BW image. Note that each position may be the position of each of a plurality of pixels, or may be the position of a block consisting of a plurality of pixels.

[0103] In the depth acquisition device 1 according to the present embodiment, a flare region is detected from an IR image, and the depth of the flare region is estimated based on not only the IR image but also the BW image, so that the depth of the flare region can be appropriately acquired. That is, the scenes to be imaged are substantially the same in the IR image and the BW image, and the viewpoint and image acquisition time are also substantially the same. Therefore, the IR image and the BW image have a high correlation. Also, flare is a phenomenon that depends on wavelength, and even if flare occurs in the IR image, there is a high possibility that flare does not occur in the BW image. Therefore, information missing in the flare region can be compensated for from a region in the BW image that corresponds to the flare region (i.e., a corresponding region). As a result, the depth of the flare region can be appropriately acquired.

[0104] [Specific functional configuration of the depth acquisition device] FIG. 8 is a block diagram showing a specific functional configuration of the processor 110 of the depth acquisition device 1. As shown in FIG.

[0105] The processor 110 includes a first depth estimation unit 111a, a second depth estimation unit 111b, a flare detection unit 112, a high brightness area detection unit 116, a first edge detection unit 117IR, a second edge detection unit 117BW, and an output unit 118. Note that the first depth estimation unit 111a and the second depth estimation unit 111b correspond to the depth estimation unit 111 shown in Fig. 5. The processor 110 may also include the light emission timing acquisition unit 113, the IR image acquisition unit 114, and the BW image acquisition unit 115 described above.

[0106] The high-luminance region detection unit 116 detects a region having a luminance equal to or greater than a first threshold in the IR image as a high-luminance region. The first edge detection unit 117IR detects edges in the IR image. The second edge detection unit 117BW detects edges in the BW image.

[0107] The flare detection unit 112 compares, for each of at least one high-luminance region in the IR image, an edge detected for that high-luminance region with an edge detected for a region in the BW image corresponding to that high-luminance region. Through this comparison, the flare detection unit 112 determines whether or not the high-luminance region is a flare region. That is, through this determination, a flare region is detected. In other words, the flare detection unit 112 divides the captured IR image into regions, that is, a flare region and a non-flare region that is not a flare region, thereby dividing the IR image.

[0108] Here, flare and the like are phenomena that depend on the wavelength of light. Therefore, flare and the like that occurs in an IR image often does not occur in a BW image. It is generally known that an IR image and a BW image have a strong correlation. However, in the case of a flare or the like that occurs in an IR image, the edge of the IR image is crushed, so that the correlation value between the edge of the area where the flare or the like occurs and the edge of the area of ​​the BW image corresponding to that area becomes low. Also, when a flare or the like occurs, the brightness of the area where the flare or the like occurs increases. Therefore, the flare detection unit 112 in this embodiment uses this relationship to determine the flare area from the captured IR image.

[0109] That is, the flare detection unit 112 in this embodiment detects, as a flare region, a region in the IR image that has a luminance equal to or greater than a first threshold and satisfies a predetermined condition. The predetermined condition is that a correlation value between an image feature amount in a region of the IR image and an image feature amount in a region of the BW image corresponding to the region is less than a second threshold. Here, the image feature amount in each region of the IR image and the BW image is an edge included in the image in the region. Note that the region of the BW image corresponding to the region of the IR image is a region that is spatially in the same position as the region of the IR image and has the same shape and size as the region of the IR image.

[0110] As described above, the correlation between the image feature amount in the flare area of ​​the IR image and the image feature amount in the area of ​​the BW image corresponding to the flare area tends to be low. Therefore, in the present embodiment, the flare area can be more appropriately detected by detecting an area in the IR image that has high brightness and low correlation of the image feature amount as the flare area.

[0111] The first depth estimation section 111a and the second depth estimation section 111b have the function of the depth estimation section 111 described above.

[0112] The first depth estimation unit 111a estimates the depth at each position in the IR image based on the IR image acquired in accordance with the timing of irradiation of infrared light by the light source 101. The first depth estimation unit 111a outputs information indicating the estimated depth at each position in the IR image as first depth information. In other words, the first depth estimation unit 111a estimates the first depth information indicating the depth at each position in the IR image.

[0113] The second depth estimation unit 111b corrects the first depth information based on the BW image and the flare region in the IR image. As a result, the depth of the flare region among the depths at each position in the IR image indicated by the first depth information is corrected. The second depth estimation unit 111b outputs information indicating the corrected depth at each position in the flare region as the second depth information. That is, the second depth estimation unit 111b estimates the second depth information indicating the corrected depth at each position in the flare region by correcting the depth at each position in the flare region indicated by the first depth information based on the BW image.

[0114] The output unit 118 replaces the depth at each position in the flare region indicated by the first depth information with the corrected depth at each position in the flare region indicated by the second depth information. This generates third depth information including the depth at each position outside the flare region of the IR image indicated by the first depth information and the corrected depth at each position in the flare region of the IR image indicated by the second depth information. The output unit 118 outputs the third depth information.

[0115] As a result, the third depth information indicates a depth obtained from the IR image as a depth outside the flare region of the IR image, and indicates a depth obtained from the IR image and corrected based on the BW image as a depth of the flare region of the IR image. Therefore, in this embodiment, even if the IR image has a flare region, the entire depth of the IR image can be appropriately estimated.

[0116] Figure 9A shows an example of an IR image, and Figure 9B shows an example of a BW image.

[0117] As shown in Fig. 9B, the BW image shows a scene in which a signboard is placed on a road. The signboard contains, for example, a material that easily reflects infrared light. When the IR camera 102 captures the same scene as that shown in Fig. 9B from the same viewpoint as that of the BW camera 103, the IR image shown in Fig. 9A is acquired.

[0118] As shown in FIG. 9A, the IR image acquired as described above has a high luminance flare in an area including a range corresponding to the signboard in the BW image. This is because infrared light from the light source 101 is reflected specularly by the signboard on the road, and strong infrared light is incident on the IR camera 102 as reflected light. Materials that easily reflect infrared light are often used for clothes worn by construction workers or multiple poles erected along roads. Therefore, when a scene including a subject using such materials is captured, there is a high possibility that flare will occur in the IR image. However, there is a low possibility that flare will occur in the BW image. As a result, the correlation between the image feature amount of the flare area of ​​the IR image and the image feature amount of the area of ​​the BW image corresponding to the flare area is low. On the other hand, the correlation between the image feature amount of the area other than the flare area of ​​the IR image (i.e., the non-flare area) and the image feature amount of the area of ​​the BW image corresponding to the non-flare area is high.

[0119] FIG. 10 shows an example of a binarized image obtained by binarizing an IR image.

[0120] The high-luminance area detection unit 116 detects an area having a luminance equal to or greater than a first threshold as a high-luminance area in the IR image shown in Fig. 9A. That is, the high-luminance area detection unit 116 binarizes the luminance at each position (i.e., each pixel) in the IR image. As a result, for example, as shown in Fig. 10, a binarized image consisting of a white area and a black area (hatched area in Fig. 10) is generated.

[0121] FIG. 11 shows an example of a high brightness area in an IR image.

[0122] The high-luminance region detection unit 116 detects white regions in the binarized image as high-luminance regions. For example, as shown in Fig. 11, when six white regions are included in the binarized image, the high-luminance region detection unit 116 detects the six white regions as high-luminance regions A to F. That is, the IR image or the binarized image is divided into six high-luminance regions A to F and a non-high-luminance region that is not a high-luminance region.

[0123] FIG. 12 shows an area of ​​the BW image that corresponds to a high brightness area of ​​the IR image.

[0124] The flare detection unit 112 identifies, for each of at least one high-luminance region in the binarized image (i.e., the IR image), an image feature of a region in the BW image corresponding to the high-luminance region. The image feature is, for example, an edge. The region in the BW image corresponding to the high-luminance region is located at the same spatial position as the high-luminance region in the binarized image or the IR image, and has the same shape and size as the high-luminance region. Hereinafter, such a region in the BW region corresponding to a region in the IR image is also referred to as a corresponding region.

[0125] For example, as shown in FIG. 11, when high luminance areas A to F are detected, the flare detection unit 112 identifies image feature amounts of areas corresponding to each of these high luminance areas A to F in the BW image.

[0126] FIG. 13 shows an example of a flare region detected from an IR image.

[0127] Flare detection unit 112 determines whether each of high luminance regions A to F is a flare region. That is, flare detection unit 112 determines whether each high luminance region is a flare region by comparing an image feature amount of high luminance region A in the IR image with an image feature amount of a corresponding region in the BW image corresponding to high luminance region A. As a result, for example, as shown in Fig. 13, flare detection unit 112 determines that high luminance regions A, C, D, and E of high luminance regions A to F are flare regions.

[0128] FIG. 14 shows a simulation result of the depth acquisition device 1.

[0129] The depth acquisition device 1 acquires a BW image shown in (a) of Fig. 14 by imaging with the BW camera 103, and further acquires an IR image shown in (b) of Fig. 14 by imaging with the IR camera 102. The BW image and the IR image are images obtained by imaging the same scene from the same viewpoint and at the same imaging time. In the example shown in (b) of Fig. 14, a large flare area occurs at the right end of the IR image.

[0130] The first depth estimation unit 111a generates the first depth information shown in FIG. 14(c) by estimating the depth from the IR image. This first depth information is expressed as a first depth image that indicates the depth at each position in the IR image by brightness. In this first depth image, the depth of the flare region is inappropriately expressed.

[0131] The second depth estimation unit 111b corrects the inappropriate depth in the flare region. Then, the output unit 118 generates third depth information indicating the corrected depth of the flare region and the depth of the non-flare region, as shown in (e) of FIG. 14. This third depth information is also expressed as a third depth image indicating the depth by brightness, similar to the first depth information. Note that the second depth estimation unit 111b may also correct the depth of the non-flare region in the first depth image based on the image feature amount of the corresponding region of the BW image.

[0132] In this way, in the depth acquisition device 1 according to the present embodiment, the third depth image can be made closer to the correct depth image shown in FIG. 14(d) in the entire image including the flare region.

[0133] [Specific processing flow of the depth acquisition device] FIG. 15 is a flowchart showing the overall processing operation of the depth acquisition device 1 shown in FIG.

[0134] (Step S31) First, the high-luminance region detection unit 116 detects high-luminance regions from the IR image.

[0135] (Step S32) The first edge detector 117IR detects edges in the IR image.

[0136] (Step S33) The second edge detector 117BW detects edges in the BW image.

[0137] (Step S34) The flare detection unit 112 detects a flare region in the IR image by comparing an edge in at least one high-luminance region of the IR image with an edge in a corresponding region of the BW image. That is, the flare detection unit 112 detects a high-luminance region as a flare region when a correlation value between an edge in the high-luminance region and an edge in a corresponding region of the BW image is less than a second threshold. This divides the IR image into at least one flare region and a non-flare region.

[0138] (Step S35) The first depth estimation unit 111a generates first depth information from the IR image using, for example, TOF.

[0139] (Step S36) The second depth estimation unit 111b generates second depth information indicating the depth of the flare region based on the first depth information of the IR image and the BW image.

[0140] (Step S37) The output unit 118 generates third depth information by replacing the depth of the flare region indicated by the first depth information with the depth indicated by the second depth information.

[0141] FIG. 16 is a flowchart showing detailed processing of steps S31 to S34 in FIG.

[0142] (Step S41) First, the high-luminance region detection unit 116 judges whether the luminance at each position in the IR image is equal to or greater than the first threshold. Here, the first threshold may be about 1500 if the IR image is an image with 12-bit gradation, for example. Of course, this first threshold may be a value that changes depending on the environmental conditions or the settings of the IR camera 102. For example, when a dark scene such as at night is captured, the luminance of the entire IR image is low, so the first threshold may be a smaller value than when a bright scene is captured in the daytime. Also, when the exposure time of the IR camera 102 is long, the luminance of the entire IR image is high, so the first threshold may be a larger value than when the exposure time is short.

[0143] (Step S42) If it is determined that the luminance at any position is not equal to or greater than the first threshold (No in step S41), the high luminance area detection unit 116 determines that no flare occurs in the IR image (step S42). In other words, the entire IR image is determined to be a non-flare area.

[0144] (Step S43) On the other hand, if it is determined that the luminance at any position is equal to or greater than the first threshold (Yes in step S41), the high luminance area detection unit 116 divides the IR image into areas. That is, the high luminance area detection unit 116 divides the IR image into at least one high luminance area and an area other than the high luminance area. For this area division, a luminance-based method such as SuperPixel may be used.

[0145] (Step S44) Next, the first edge detector 117IR and the second edge detector 117BW perform edge detection on the IR image and the BW image, respectively. The edge detection may utilize the Canny method or a Sobel filter.

[0146] (Step S45) The flare detection unit 112 compares an edge in at least one high-luminance region of the IR image with an edge in a region of the BW image corresponding to the high-luminance region. That is, the flare detection unit 112 determines whether or not a correlation value between an edge in a high-luminance region of the IR image and an edge in a corresponding region of the BW image is equal to or greater than a second threshold. The correlation value is calculated by arranging each value output by edge detection for each of the IR image and the BW image in a vector form for each region, and normalizing the inner product value. That is, the flare detection unit 112 normalizes the inner product value between a vector consisting of a plurality of values ​​obtained by edge detection in a high-luminance region of the IR image and a vector consisting of a plurality of values ​​obtained by edge detection in a region of the BW image corresponding to the high-luminance region. In this way, the correlation value for the high-luminance region is calculated.

[0147] (Step S46) Here, if it is determined that the correlation value is not equal to or greater than the second threshold, that is, is less than the second threshold (No in step S45), the flare detection unit 112 determines the high-luminance region as a flare region. That is, because there is no edge correlation between the region in the IR image where the flare or the like occurs and the corresponding region in the BW image due to the influence of the flare or the like, the flare detection unit 112 determines the high-luminance region in the IR image as a flare region.

[0148] On the other hand, if flare detection unit 112 determines in step S45 that the correlation value is equal to or greater than the second threshold, i.e., is not less than the second threshold (Yes in step S45), it determines that no flare or the like has occurred in the high-luminance region. In other words, flare detection unit 112 determines that the high-luminance region is a non-flare region.

[0149] In such a method, an IR image and a BW image having substantially the same viewpoint are required. In the depth acquisition device 1 in this embodiment, the filter used for each pixel is set to either an IR filter or a BW filter. That is, as shown in FIG. 2, a first pixel 21 having an IR filter and a second pixel 22 having a BW filter are alternately arranged in the column direction. This makes it possible to acquire an IR image and a BW image having substantially the same viewpoint, so that the flare area can be appropriately determined.

[0150] <Using brightness correlation> In the above description, edges are used to distinguish between flare and non-flare regions, but edges do not have to be used for this distinction. For example, the correlation value of the luminance of the IR image and the BW image may be used. As described above, when no flare or the like occurs, the IR image and the BW image have a strong correlation, but in a region where a flare or the like occurs, the correlation becomes weak. Therefore, by using the correlation of the luminance of the IR image and the BW image, the flare region can be appropriately distinguished.

[0151] Fig. 17 is a flowchart showing an example of a process instead of steps S31 to S34 in Fig. 15. That is, Fig. 17 is a flowchart showing a process of detecting a flare area using the correlation between the luminance of an IR image and a BW image. In Fig. 17, the same steps as in Fig. 16 are given the same reference numerals, and detailed explanations are omitted. Unlike the flowchart shown in Fig. 16, the flowchart shown in Fig. 17 does not include step S44, and includes step S45a instead of step S45.

[0152] (Step S45a) In step S45a, for each high-brightness region obtained by the region division in step S43, the flare detection unit 112 calculates the correlation value between the luminance of each pixel in the high-brightness region and the luminance of each pixel in the region of the BW image corresponding to the high-brightness region. The correlation value is obtained by arranging the luminances of the respective pixels of the IR image and the BW image in a vector form for each region and normalizing the inner product value by the number of pixels. That is, the flare detection unit 112 normalizes the inner product value between the vector composed of the luminances of the respective pixels in the high-brightness region of the IR image and the vector composed of the luminances of the respective pixels in the corresponding region of the BW image. Thereby, the correlation value for the high-brightness region is calculated.

[0153] Here, when the correlation value is greater than or equal to the second threshold value, that is, not less than the second threshold value (Yes in step S45a), the flare detection unit 112 determines that no flare or the like has occurred in the high-brightness region (step S42). On the other hand, when the correlation value is less than the second threshold value (No in step S45a), due to the influence of flare or the like, the correlation between the luminance of each pixel in the high-brightness region of the IR image and the luminance of each pixel in the corresponding region of the BW image has decreased. Therefore, in such a case, the flare detection unit 112 determines the high-brightness region as a flare region (step S46).

[0154] That is, the image feature amounts in the respective regions of the IR image and the BW image used for detecting the flare region are the edges included in the image in the region in the example shown in FIG. 16, but are the luminance in the region in the example shown in FIG. 17. Here, as described above, the correlation between the luminance in the flare region of the IR image and the luminance in the region of the BW image corresponding to the flare region tends to be low. Therefore, in the IR image, by detecting a region with high luminance and low correlation of the luminance as a flare region, the flare region can be detected more appropriately.

[0155] <Using CENSUS transform> Of course, the evaluation value for distinguishing between flare and non-flare regions does not have to be a correlation value. For example, Hamming distance and CENSUS transform may be used. The CENSUS transform is disclosed in, for example, R. Zabih and J. Woodfill, "Non-parametric Local Transforms for Computing Visual Correspondence", Proc. of ECCV, pp.151-158, 1994. The CENSUS transform sets a window in an image and converts the magnitude relationship between the center pixel of the window and the surrounding pixels into a binary vector.

[0156] Fig. 18 is a flowchart showing another example of processing instead of steps S31 to S34 in Fig. 15. That is, Fig. 18 is a flowchart showing detection processing of a flare region using CENSUS transform of each of an IR image and a BW image. Note that in Fig. 18, the same steps as in Fig. 16 are given the same reference numerals, and detailed explanations are omitted. The flowchart shown in Fig. 18 differs from the flowchart shown in Fig. 16 in that it includes steps S44b and S45b instead of steps S44 and S45.

[0157] (Step S44b) In step S44b, for each high-luminance region obtained by the region division in step S43, flare detection unit 112 performs a CENSUS transformation on the image of that high-luminance region in the IR image and the image of the corresponding region in the BW image, thereby generating a CENSUS-transformed image for the image of that high-luminance region in the IR image and a CENSUS-transformed image for the image of the corresponding region in the BW image.

[0158] (Step S45b) Next, in step S45b, the flare detection unit 112 calculates the Hamming distance between the CENSUS transformed image of the IR image obtained in step S44b and the CENSUS transformed image of the BW image. Then, when the value obtained by normalizing this Hamming distance by the number of pixels in the high-brightness region is equal to or less than the third threshold (Yes in step S45b), the flare detection unit 112 determines that no flare or the like has occurred in the high-brightness region (step S42). On the other hand, when the value of the normalized Hamming distance is greater than the third threshold (No in step S45b), the flare detection unit 112 determines that the texture has disappeared in the image of the high-brightness region due to the influence of a flare or the like. As a result, the flare detection unit 112 discriminates the high-brightness region as a flare region (step S46).

[0159] That is, for each of at least one high-brightness region having a luminance equal to or higher than the first threshold in the IR image, the flare detection unit 112 obtains a first transformed image by performing CENSUS transformation on the image within the high-brightness region. Then, the flare detection unit 112 obtains a second transformed image by performing CENSUS transformation on the image within the region of the BW image corresponding to the high-brightness region. Note that the first transformed image and the second transformed image are the CENSUS transformed images described above. Next, the flare detection unit 112 detects, as a flare region, a high-brightness region in which the Hamming distance between the first transformed image and the second transformed image exceeds the third threshold among at least one high-brightness region. Even when using CENSUS transformation in this way, the flare region can be appropriately detected.

[0160] <Using the luminance of the IR image> In the example shown in FIG. 17, the correlation value between the luminance of the IR image and the luminance of the IR image and the BW image is used to discriminate between the flare region and the non-flare region, but only the luminance of the IR image may be used.

[0161] Fig. 19 is a flowchart showing another example of processing instead of steps S31 to S34 in Fig. 15. That is, Fig. 19 is a flowchart showing detection processing of a flare area using only the luminance of an IR image. Note that in Fig. 19, the same steps as in Fig. 16 are given the same reference numerals, and detailed explanations will be omitted. Steps S43 to S45 shown in Fig. 16 are omitted in the flowchart shown in Fig. 19.

[0162] That is, in step S41, flare detection unit 112 determines whether the luminance of a pixel in the IR image is equal to or greater than a first threshold. If it is determined that the luminance of the pixel is less than the first threshold (No in step S41), flare detection unit 112 determines that no flare has occurred in the region including that pixel (step S42). On the other hand, if it is determined that the luminance of the pixel is equal to or greater than the first threshold (Yes in step S41), flare detection unit 112 determines that a flare has occurred in the region including that pixel. That is, flare detection unit 112 determines that the region is a flare region (step S46).

[0163] 19, the flare detection unit 112 detects an area having a luminance equal to or greater than a first threshold value in the IR image as a flare area. Since the luminance in the flare area tends to be higher than the luminance outside the flare area, the flare area can be appropriately detected by detecting an area having a luminance equal to or greater than a first threshold value in the IR image as a flare area.

[0164] Also, the discrimination between the flare region and the non-flare region may be realized by using a learning process. For example, a process such as deep learning may be used for the learning process. In this case, for learning, an IR image, a BW image, and a correct answer image in which the IR image is divided into a flare region and a non-flare region are prepared in advance. Next, the IR image and the BW image are given as inputs to a learning model. Then, the learning model is trained so that the output from the learning model for the input matches the correct answer image. For example, the learning model is a neural network. The output from the learning model is an image showing a value "0" or a value "1" for each pixel, where the value "0" indicates that the pixel belongs to a non-flare region, and the value "1" indicates that the pixel belongs to a flare region.

[0165] The flare detection unit 112 distinguishes between flare regions and non-flare regions by using the learning model that has been trained in advance in this way. That is, the flare detection unit 112 inputs an IR image and a BW image to the learning model as inputs. Then, the flare detection unit 112 distinguishes a region including pixels corresponding to a numerical value "0" output from the learning model as a non-flare region. Furthermore, the flare detection unit 112 distinguishes a region including pixels corresponding to a numerical value "1" output from the learning model as a flare region.

[0166] Through the above processing, the flare detection unit 112 divides the captured IR image into a flare region where a flare or the like occurs and a non-flare region.

[0167] <Depth correction processing> The second depth estimation unit 111b generates second depth information by using the BW image, the first depth information, and the flare region (that is, the discrimination result of the above-mentioned region).

[0168] Flare and other phenomena depend on the wavelength of light. Therefore, flare and other phenomena that occur in an IR image often do not occur in a BW image. Therefore, by correcting the first depth information only in the flare region using the BW image, rather than the first depth information obtained from the IR image, it is possible to obtain second depth information that is not affected by flare and other phenomena that occur in the IR image.

[0169] A guided filter, which is a type of image correction filter, may be used to obtain the second depth information. The guided filter is disclosed in a non-patent document (Kaiming He, Jian Sun and Xiaoou Tang, “Guided Image Filtering”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 35, Iss. 6, pp.1397-1409,2013.). The guided filter is a filter that corrects a target image by utilizing the correlation between the target image and a reference image. In the guided filter, it is assumed that the reference image I and the target image p are expressed by parameters a and b as shown in the following (Equation 2).

[0170]

number

[0171] Here, q is the output image obtained by correcting the target image p, i is the number of each pixel, and ωk is the peripheral area of ​​pixel k. Furthermore, parameters a and b are expressed by the following (Equation 3).

[0172]

number

[0173] Here, ε is a regularization parameter, and μ and σ are the mean and variance within a block of the reference image, and are calculated by the following (Equation 4).

[0174]

number

[0175] However, in order to suppress the noise contained in the obtained parameters a and b, averaged parameters are used and the output is calculated as shown in the following (Equation 5).

[0176]

number

[0177] In this embodiment, the second depth estimation unit 111b corrects the first depth information (or the first depth image) which is the target image by providing a BW image as a reference image. This generates or acquires the second depth information. To generate such second depth information, an IR image and a BW image having substantially the same viewpoint position are required. In the depth acquisition device 1 of this embodiment, the filter used for each pixel is set to either an IR filter or a BW filter. That is, as shown in FIG. 2, the first pixel 21 having an IR filter and the second pixel 22 having a BW filter are alternately arranged in the column direction. This makes it possible to acquire an IR image and a BW image having substantially the same viewpoint, and therefore to acquire appropriate second depth information.

[0178] Of course, the second depth estimation unit 111b may use a process other than the guided filter. For example, the second depth estimation unit 111b may use a process such as a bilateral filter (Non-Patent Document: C. Tomasi, R. Manduchi, “Bilateral filtering for gray and color images”, IEEE International Conference on Computer Vision (ICCV), pp. 839-846, 1998) or Mutual-Structure for Joint Filtering (Non-Patent Document: Xiaoyong Shen, Chao Zhou, Li Xu and Jiaya Jia, “Mutual-Structure for Joint Filtering”, IEEE International Conference on Computer Vision (ICCV), 2015.).

[0179] As described above, in this embodiment, the first depth information is used for an area where it is determined that no flare or the like is occurring (i.e., a non-flare area), and the second depth information is used for an area where a flare or the like is occurring (i.e., a flare area). This makes it possible to obtain more accurate depth information even if a flare or the like is occurring in the IR image.

[0180] (Variation 1) In the above embodiment, a filter such as a guided filter is used to generate the second depth information, but the second depth information may be generated using a learning model.

[0181] For example, as in the non-patent document (Shuran Song, Fisher Yu, Andy Zeng, Angel X. Chang, Manolis Savva and Thomas Funkhouser, “Semantic Scene Completion from a Single Depth Image”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 190-198, 2017.), a learning process, deep learning, may be used. That is, a learning model may be trained so that when a BW image and first depth information are input, second depth information is output. The above non-patent document proposes a network that, when depth information including a missing area and a color image are input, interpolates the missing area of ​​the depth information. The second depth estimation unit 111b in this modification provides an IR image, a BW image, and the first depth information to a network (i.e., a learning model) similar to that non-patent document, and further provides the flare area detected by the flare detection unit 112 as a mask image of the missing area. This makes it possible to acquire second depth information with higher accuracy from the network.

[0182] FIG. 20 is a block diagram showing an example of a functional configuration of the depth acquisition device 1 in this modified example.

[0183] The depth acquisition device 1 in this modification includes the components shown in FIG. 8, and further includes a learning model 104 including, for example, a neural network.

[0184] The second depth estimation unit 111b inputs three types of data, namely, an IR image, a BW image, and the first depth information, to the learning model 104, and generates the second depth information by using the flare area as a mask area to be corrected.

[0185] In the learning of the learning model 104, in addition to the IR image, the BW image, and the first depth information, a correct depth image is prepared in advance. Next, the IR image, the BW image, the first depth information, and the mask image specifying the flare region are given to the learning model 104 as inputs. Then, the learning model 104 is trained so that the output from the learning model 104 in response to the input matches the correct depth image. During training, the mask image is given randomly. The second depth estimation unit 111b uses the learning model 104 that has been trained in advance in this way. That is, the second depth estimation unit 111b can acquire the second depth information output from the learning model 104 by inputting the IR image, the BW image, the first depth information, and the mask image specifying the flare region to the learning model 104.

[0186] In this way, in this modification, the second depth estimation unit 111b estimates depth information indicating the depth at each position in the IR image, and corrects the depth at each position in the flare area indicated by the depth information by inputting the IR image, BW image, flare area, and the depth information into the learning model. Therefore, if the learning model is trained in advance so that the correct depth at each position in the flare area is output for the input of the IR image, BW image, flare area, and depth information, the depth information estimated from the IR image can be appropriately corrected. In other words, the depth at each position in the flare area indicated by the depth information can be appropriately corrected.

[0187] As described above, second depth estimation unit 111b may use deep learning. In this case, it is not necessary to directly output the flare region, and second depth information may be generated directly by deep learning.

[0188] FIG. 21 is a block diagram showing another example of the functional configuration of the depth acquisition device 1 in this modified example.

[0189] The depth acquisition device 1 in this modification does not include the flare detection unit 112, the high brightness area detection unit 116, the first edge detection unit 117IR, and the second edge detection unit 117BW among the components shown in FIG. 20, but includes the other components.

[0190] In the learning of the learning model 104, as in the example shown in FIG. 20, in addition to an IR image, a BW image, and the first depth information, a correct depth image is prepared in advance. Next, the IR image, the BW image, and the first depth information are given to the learning model 104 as inputs. Then, the learning model 104 is trained so that the output from the learning model 104 in response to the input matches the correct depth image. As the learning model 104, a VGG-16 network with an added skip connection may be used as in non-patent literature (Caner Hazirbas, Laura Leal-Taixe and Daniel Cremers C. Hazirbas, “Deep Depth From Focus”, In ArXiv preprint arXiv, 1704.01085, 2017.). The number of channels of the learning model 104 is changed so that the IR image, the BW image, and the first depth information are given to the learning model 104 as inputs. By utilizing the learning model 104 that has been trained in advance in this manner, the second depth estimation unit 111b can easily obtain the second depth information from the learning model 104 by inputting the IR image, the BW image, and the first depth information into the learning model 104.

[0191] That is, the depth acquisition device 1 shown in FIG. 21 includes a memory and a processor 110. Although the memory is not shown in FIG. 21, it may be included in the depth acquisition device 1 as shown in FIG. 5. The processor 110 acquires timing information indicating the timing at which the light source 101 irradiates the subject with infrared light. Next, the processor 110 acquires an IR image obtained by imaging a scene including the subject based on infrared light according to the timing indicated by the timing information and stored in the memory. Next, the processor 110 acquires a BW image obtained by imaging a scene substantially identical to the IR image based on visible light, from substantially the same viewpoint and imaging time as the IR image, and stored in the memory. Then, the first depth estimation unit 111a of the processor 110 estimates depth information indicating the depth at each position in the IR image. The second depth estimation unit 111b corrects the depth at each position in the flare region of the IR image indicated by the depth information by inputting the IR image, the BW image, and the depth information into the learning model 104.

[0192] Therefore, if the learning model 104 is trained in advance so that the correct depth at each position in the flare region of the IR image is output in response to the input of an IR image, a BW image, and depth information, the depth information estimated from the IR image can be appropriately corrected. In other words, the depth at each position in the flare region indicated by the depth information can be appropriately corrected without detecting the flare region.

[0193] (Variation 2) <Use of time direction> Here, the flare detection unit 112 may use information in the time direction to distinguish between a flare region and a non-flare region. Flare is a phenomenon that occurs inside the IR camera 102 due to the relationship between the subject and the light source 101, not due to the subject itself. Therefore, when the IR camera 102 moves, the shape of the flare region changes. Therefore, the flare detection unit 112 detects a candidate for a flare region from the IR image by the above-mentioned method, and determines whether or not the shape of the candidate has changed from the flare region detected in the previous IR image. Then, when the flare detection unit 112 determines that the shape has not changed, it may determine that the candidate is not a flare region but a non-flare region.

[0194] FIG. 22 is a flowchart showing the processing operation of the flare detection section 112 in this modified example.

[0195] (Step S51) First, the flare detection unit 112 determines whether or not a candidate for a flare region is present in the target frame. That is, the flare detection unit 112 determines whether or not a flare region detected based on the flowcharts shown in, for example, Figs. 16 to 19 is not a final flare region, but is present in the target frame as a candidate for a flare region. The target frame is an IR image to be determined, that is, an IR image in which the presence of a flare region is determined.

[0196] (Step S54) Here, if there is no candidate for a flare region in the target frame (No in step S51), flare detection section 112 determines that no flare is present in the target frame (that is, the IR image).

[0197] (Step S52) On the other hand, if a candidate for a flare region exists in the target frame (Yes in step S51), the flare detection unit 112 determines whether or not a flare region exists in the frame before the target frame. Note that the frame before the target frame is an IR image acquired by imaging using the IR camera 102 before the target frame.

[0198] (Step S55) Here, if no flare region is present in the frame preceding the target frame (No in step S52), flare detection section 112 determines that the detected candidate for the flare region is a region caused by a flare, that is, a flare region.

[0199] (Step S53) On the other hand, if a flare region exists in the frame preceding the target frame (Yes in step S52), the flare detection unit 112 compares the shape of the candidate flare region in the target frame with the shape of the flare region in the preceding frame. Here, if the shape of the candidate flare region in the target frame is similar to the shape of the flare region in the preceding frame (Yes in step S53), the flare detection unit 112 executes the process of step S54. That is, the flare detection unit 112 updates the detected candidate flare region to a non-flare region and determines that no flare exists in the target frame (i.e., the IR image). On the other hand, if the shape of the candidate flare region in the target frame is not similar to the shape of the flare region in the preceding frame (No in step S53), the flare detection unit 112 executes the process of step S55. That is, the flare detection unit 112 determines that the detected candidate flare region is a region caused by a flare, that is, a flare region.

[0200] In this manner, flare detection section 112 in this modification uses information in the time direction. That is, flare detection section 112 uses the shapes of the flare region or its candidates in the respective frames acquired at different times. This makes it possible to distinguish between a flare region and a non-flare region with higher accuracy.

[0201] The flare detection unit 112 may determine the flare area for each of a plurality of adjacent frames, without determining the flare area for each frame. The plurality of adjacent frames are, for example, a plurality of IR images acquired in a time-series manner by imaging using the IR camera 102. That is, the flare detection unit 112 may detect candidates for the flare area in each of the plurality of adjacent frames, for example, based on the flowcharts shown in Figs. 16 to 19 described above, and determine whether or not the candidates are flare areas. More specifically, the flare detection unit 112 compares the shapes of the candidates for the flare area in each frame, and if the shapes of the candidates for the flare area are substantially the same, the flare detection unit 112 determines that the candidates are not flare areas, that is, non-flare areas. On the other hand, if the shapes of the candidates for the flare area are not similar, the flare detection unit 112 determines that the candidates are flare areas.

[0202] Alternatively, the flare detection unit 112 may determine whether the two shapes are similar by determining whether the similarity between the two shapes is equal to or greater than a threshold value. The similarity may be calculated as a correlation value between the two shapes, for example.

[0203] (Variation 3) <Using multiple original IR images> The depth acquisition device 1 in the above embodiment and its modified examples 1 and 2 detects a flare area using an IR image and a BW image, but the BW image may not be used. The depth acquisition device 1 in this modified example detects a flare area using multiple IR original images. The multiple IR original images are, for example, an infrared image obtained in the first exposure period and an infrared image obtained in the second exposure period shown in FIG. 3.

[0204] That is, in this modification, when the IR camera 102 acquires a plurality of infrared images at different timings to estimate the first depth information using TOF or the like, the flare region and the non-flare region are distinguished using these plurality of infrared images. In the following description, each of these plurality of infrared images is called an IR original image. It should be noted that the above-mentioned IR image is composed of these plurality of IR original images.

[0205] There are two types of TOF depth estimation: direct TOF, which directly measures the arrival time of emitted light, and indirect TOF, which estimates depth from multiple IR original images obtained by different timings of light emission and reception. In this modification, the depth acquisition device 1 distinguishes between flare and non-flare regions from multiple IR original images acquired during indirect TOF.

[0206] The timing at which each of the multiple IR original images is obtained, that is, the timing at which each of the multiple IR original images receives light, is different. Therefore, even if a flare occurs in a first IR original image among the multiple IR original images, there is a high possibility that a flare does not occur in a second IR original image, which has a different timing of receiving light from the first IR original image. Therefore, the depth acquisition device 1 in this modification distinguishes between a flare area and a non-flare area by comparing multiple IR original images acquired by changing the timing of receiving light. This makes it possible to make the third depth image closer to the correct depth image shown in (d) of FIG. 14.

[0207] [Specific processing flow of the depth acquisition device] FIG. 23 is a flowchart showing the overall processing operation of the depth acquisition device 1 in this modified example.

[0208] (Step S31a) First, the high-luminance region detection unit 116 detects high-luminance regions from the first original IR image.

[0209] (Step S32a) The first edge detection unit 117IR detects edges in the first original IR image.

[0210] (Step S33a) The second edge detector 117BW detects edges in the second original IR image instead of the BW image.

[0211] (Step S34a) The flare detection unit 112 detects a flare region in the IR image by comparing an edge in at least one high-luminance region of the first IR original image with an edge in a corresponding region of the second IR original image. That is, when a correlation value between an edge in a high-luminance region and an edge in a corresponding region of the second IR original image is less than a fourth threshold, the flare detection unit 112 detects the high-luminance region as a flare region. As a result, the IR image is divided into at least one flare region and a non-flare region.

[0212] (Step S35) The first depth estimation unit 111a generates first depth information from the IR image using, for example, TOF.

[0213] (Step S36) The second depth estimation unit 111b generates second depth information indicating the depth of the flare region based on the first depth information of the IR image and the BW image.

[0214] (Step S37) The output unit 118 generates third depth information by replacing the depth of the flare region indicated by the first depth information with the depth indicated by the second depth information.

[0215] FIG. 24 is a flowchart showing the detailed processing of steps S31a to S34a in FIG.

[0216] (Step S41a) First, the high-luminance region detection unit 116 judges whether the luminance at each position in the first IR original image is equal to or greater than the fifth threshold. Here, the fifth threshold may be about 1500 if the first IR original image is an image with 12-bit gradation, for example. Of course, this fifth threshold may be a value that changes according to the environmental conditions or the settings of the IR camera 102. For example, when a dark scene such as at night is captured, the luminance of the entire first IR original image is low, so the fifth threshold may be a smaller value than when a bright scene is captured in the daytime. Also, when the exposure time of the IR camera 102 is long, the luminance of the entire first IR original image is high, so the fifth threshold may be a larger value than when the exposure time is short.

[0217] (Step S42a) Here, if it is determined that the luminance at any position is not equal to or greater than the fifth threshold (No in step S41a), the high luminance area detection unit 116 determines that no flare occurs in the IR image constructed using the first original IR image (step S42). In other words, the entire IR image is determined to be a non-flare area.

[0218] (Step S43a) On the other hand, when it is determined that the luminance at any position is equal to or greater than the fifth threshold (Yes in step S41a), the high luminance region detection unit 116 divides the first IR original image into regions. That is, the high luminance region detection unit 116 divides the first IR original image into at least one high luminance region and a region other than the high luminance region. For this region division, a method based on luminance, such as SuperPixel, may be used.

[0219] (Step S44a) Next, the first edge detector 117IR and the second edge detector 117BW perform edge detection on the first original IR image and the second original IR image, respectively. The Canny method or a Sobel filter may be used for edge detection.

[0220] (Step S45a) The flare detection unit 112 compares, for each of at least one high-luminance region of the first IR original image, an edge in the high-luminance region with an edge in a region of the second IR original image corresponding to the high-luminance region. That is, the flare detection unit 112 determines whether or not a correlation value between an edge in a high-luminance region of the first IR original image and an edge in a corresponding region of the second IR original image is equal to or greater than a fourth threshold value. The correlation value is calculated by arranging each value output by edge detection for each of the first IR original image and the second IR original image in a vector shape for each region, and normalizing the inner product value. That is, the flare detection unit 112 normalizes the inner product value between a vector consisting of a plurality of values ​​obtained by edge detection in a high-luminance region of the first IR original image and a vector consisting of a plurality of values ​​obtained by edge detection in a region of the second IR original image corresponding to the high-luminance region. This allows the correlation value for the high-luminance region to be calculated.

[0221] (Step S46a) Here, if it is determined that the correlation value is not equal to or greater than the fourth threshold, that is, is less than the fourth threshold (No in step S45a), the flare detection unit 112 determines the high-luminance region as a flare region. That is, because there is no edge correlation between the region in the first IR original image where the flare or the like occurs and the corresponding region in the second IR original image due to the influence of the flare or the like, the flare detection unit 112 determines the high-luminance region in the first IR original image as a flare region. That is, the flare detection unit 112 determines the high-luminance region in the IR image as a flare region. Note that the high-luminance region in the IR image is the same region as the high-luminance region in the first IR original image.

[0222] On the other hand, if flare detection unit 112 determines in step S45a that the correlation value is equal to or greater than the fourth threshold, that is, is not less than the fourth threshold (Yes in step S45a), it determines that no flare or the like has occurred in the high-luminance region. In other words, flare detection unit 112 determines that the high-luminance region is a non-flare region.

[0223] <Using brightness correlation> In the above description, edges are used to distinguish between flare and non-flare regions, but edges do not have to be used for the distinction. For example, the correlation value of the luminance of the first IR original image and the second IR original image may be used. As described above, when no flare or the like occurs, the first IR original image and the second IR original image have a strong correlation, but in an area where a flare or the like occurs, the correlation becomes weak. Therefore, by using the correlation of the luminance of the first IR original image and the second IR original image, the flare region can be appropriately distinguished.

[0224] Fig. 25 is a flowchart showing an example of a process instead of steps S31a to S34a in Fig. 23. That is, Fig. 25 is a flowchart showing a process of detecting a flare area using the correlation between the luminance of the first original IR image and the second original IR image. In Fig. 25, the same steps as those in Fig. 24 are given the same reference numerals, and detailed explanations are omitted. Unlike the flowchart shown in Fig. 24, the flowchart shown in Fig. 25 does not include step S44a, and includes step S45b instead of step S45a.

[0225] (Step S45b) In step S45b, the flare detection unit 112 calculates a correlation value between the luminance of each pixel in each high-luminance region obtained by the region division in step S43a and the luminance of each pixel in the region of the second IR original image corresponding to the high-luminance region. The correlation value is obtained by arranging the luminance of each pixel in each of the first IR original image and the second IR original image in a vector shape for each region and normalizing the inner product value by the number of pixels. In other words, the flare detection unit 112 normalizes the inner product value between the vector consisting of the luminance of each pixel in the high-luminance region of the first IR original image and the vector consisting of the luminance of each pixel in the corresponding region of the second IR original image. This calculates the correlation value for the high-luminance region.

[0226] Here, when the correlation value is greater than or equal to the fourth threshold value, that is, not less than the fourth threshold value (Yes in step S45b), the flare detection unit 112 determines that no flare or the like has occurred in the high-luminance region (step S42a). On the other hand, when the correlation value is less than the fourth threshold value (No in step S45b), due to the influence of flares or the like, the correlation between the luminance of each pixel in the high-luminance region of the first IR original image and the luminance of each pixel in the corresponding region of the second IR original image becomes low. Therefore, in such a case, the flare detection unit 112 determines the high-luminance region as a flare region (step S46a).

[0227] That is, the image feature amounts in the respective regions of the first IR original image and the second IR original image used for detecting the flare region are edges included in the image in that region in the example shown in FIG. 24, but are luminance in that region in the example shown in FIG. 25. Here, as described above, the correlation between the luminance in the flare region of the first IR original image and the luminance in the region of the second IR original image corresponding to the flare region tends to be low. Therefore, in the first IR original image, by detecting a region with high luminance and low correlation of the luminance as a flare region, the flare region can be detected more appropriately.

[0228] <Using CENSUS transform> Of course, the evaluation value for discriminating between the flare region and the non-flare region does not necessarily have to be the correlation value. For example, the aforementioned Hamming distance and CENSUS transform may be used.

[0229] FIG. 26 is a flowchart showing another example of the process instead of steps S31a to S34a in FIG. 23. That is, FIG. 26 is a flowchart showing the flare region detection process using the CENSUS transform of each of the first IR original image and the second IR original image. In FIG. 26, the same steps as those in FIG. 24 are given the same reference numerals, and detailed descriptions thereof are omitted. The flowchart shown in FIG. 26 includes steps S44c and S45c instead of steps S44a and S45a, unlike the flowchart shown in FIG. 24.

[0230] (Step S44c) In step S44c, for each high-luminance region obtained by the region division in step S43a, the flare detection unit 112 performs a CENSUS transformation on the image of the high-luminance region in the first IR original image and the image of the corresponding region in the second IR original image, thereby generating a CENSUS-transformed image for the image of the high-luminance region in the first IR original image and a CENSUS-transformed image for the image of the corresponding region in the second IR original image.

[0231] (Step S45c) Next, in step S45c, the flare detection unit 112 calculates the Hamming distance between the CENSUS-converted image of the first IR original image obtained in step S44c and the CENSUS-converted image of the second IR original image. If the value of this Hamming distance normalized by the number of pixels in the high-luminance region is equal to or less than a sixth threshold (Yes in step S45c), the flare detection unit 112 determines that no flare or the like has occurred in the high-luminance region (step S42a). On the other hand, if the value of the normalized Hamming distance is greater than the sixth threshold (No in step S45c), the flare detection unit 112 determines that the texture has disappeared in the image of the high-luminance region due to the influence of a flare or the like. As a result, the flare detection unit 112 determines that the high-luminance region is a flare region (step S46a).

[0232] That is, the flare detection unit 112 obtains a first converted image by CENSUS converting an image in at least one high-luminance region having a luminance equal to or greater than a fifth threshold in the first IR original image. Then, the flare detection unit 112 obtains a second converted image by CENSUS converting an image in a region of the second IR original image corresponding to the high-luminance region. The first converted image and the second converted image are the above-mentioned CENSUS converted images. Next, the flare detection unit 112 detects, as a flare region, a high-luminance region in which the Hamming distance between the first converted image and the second converted image exceeds a sixth threshold among at least one high-luminance region. In this way, even if the CENSUS conversion is used, the flare region can be appropriately detected.

[0233] As described above, in the depth acquisition device 1 of this embodiment and its modified examples, even if there is a flare region in the IR image, the appropriate depth at each position within the flare region can be acquired by using an image of the corresponding region in the BW image.

[0234] In each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software for realizing the depth acquisition device of the above embodiment and modified example causes a computer to execute each step included in any of the flowcharts of Figures 6, 7, 15 to 19, and 22 to 26.

[0235] Although the depth acquisition device according to one or more aspects has been described based on the embodiment and its modified examples, the present disclosure is not limited to the embodiment and its modified examples. As long as it does not deviate from the spirit of the present disclosure, various modifications conceived by a person skilled in the art to the present embodiment and modified examples, and forms constructed by combining the components in the embodiment and modified examples may also be included within the scope of the present disclosure.

[0236] In addition, in this disclosure, all or part of the units, devices, or all or part of the functional blocks of the block diagrams shown in Figures 1, 4, 5, 8, 20, and 21 may be implemented by one or more electronic circuits including a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). The LSI or IC may be integrated into one chip, or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated into one chip. Although LSI and IC are referred to here, the names may change depending on the degree of integration, and may be called system LSI, VLSI (very large scale integration), or ULSI (ultra large scale integration). Field programmable gate arrays (FPGAs) that are programmed after the manufacture of LSIs, or reconfigurable logic devices that can reconfigure the junction relationship within LSIs or set up circuit partitions within LSIs, can also be used for the same purpose.

[0237] Furthermore, all or part of the functions or operations of a unit, device, or part of a device can be executed by software processing. In this case, the software is recorded in one or more non-transitory recording media such as a ROM, an optical disk, a hard disk drive, etc., and when the software is executed by a processor, the software causes the processor and peripheral devices to execute specific functions in the software. The system or device may include one or more non-transitory recording media in which the software is recorded, a processor, and necessary hardware devices such as an interface. [Industrial Applicability]

[0238] The present disclosure is applicable to a depth acquisition device that acquires depth from an image obtained by shooting, and can be used as, for example, an in-vehicle device. [Explanation of symbols]

[0239] 1. Depth acquisition device 10, 101 light source 20 Solid-state imaging element 21 1st pixel (IR) 22 2nd pixel (BW) 30 Processing circuit 50 Diffuser 60 Lenses 70 Bandpass Filter 102 IR Camera 103 BW Camera 104 Learning Model 110 Processor 111 Depth Estimation Unit 111a First depth estimation unit 111b Second depth estimation unit 112 Flare detection unit 113 Light emission timing acquisition unit 114 IR image acquisition unit 115 BW image acquisition unit 116 High brightness area detection unit 117IR First edge detection section 117BW 2nd edge detection section 118 Output section 200 Memory

Claims

1. An infrared image is obtained by capturing light that is irradiated from a light source and reflected by a subject. acquiring a visible light image by capturing an image of the subject using visible light; detecting a flare region based on the infrared image and the visible light image; Information processing methods.

2. In the detection of the flare region, the flare region is detected based on a region in the infrared image corresponding to a region in the visible light image. The information processing method according to claim 1 .

3. In the detection of the flare region, the flare region is detected based on a luminance of a region in the infrared image corresponding to a region in the visible light image.

3. The information processing method according to claim 1.

4. In the detection of the flare region, it is detected whether a luminance of a region in the infrared image corresponding to a region in the visible light image is equal to or greater than a threshold value. The information processing method according to claim 3.

5. In the detection of the flare region, when a luminance of a region in the infrared image corresponding to a region in the visible light image is equal to or greater than a threshold, the region in the infrared image is detected as the flare region. The information processing method according to claim 4.

6. In the detection of the flare region, the flare region is detected based on the infrared image and the visible light image generated by capturing an image of the same scene as the infrared image. The information processing method according to claim 1 .

7. In the detection of the flare region, the flare region is detected based on the infrared image and the visible light image generated by capturing the infrared image at the same viewpoint as that of the infrared image. The information processing method according to claim 1 .

8. In the detection of the flare region, the flare region is detected based on the infrared image and the visible light image generated by capturing the infrared image at the same time as the infrared image. The information processing method according to claim 1 .

9. The infrared image is acquired using ToF (Time of Flight). The information processing method according to claim 1 .

10. In the detection of the flare region, detecting, based on the infrared image and the visible light image, a region in the infrared image that has a luminance equal to or greater than a first threshold and satisfies a predetermined condition, as the flare region; The information processing method according to claim 1 .

11. the predetermined condition being a condition that a correlation value between an image feature amount in a region of the infrared image and an image feature amount in a region of the visible light image corresponding to the region is less than a second threshold value; The information processing method according to claim 10.

12. The image feature amount in each region of the infrared image and the visible light image is an edge included in the image in the region. The information processing method according to claim 11.

13. The image feature amount in each region of the infrared image and the visible light image is the luminance in the region. The information processing method according to claim 11.

14. The acquisition of the infrared image and the acquisition of the visible light image are performed by a solid-state image sensor, the solid-state imaging element includes a plurality of first pixels used for capturing the infrared image, and a plurality of second pixels different from the plurality of first pixels and used for capturing the visible light image; The information processing method according to claim 1 .

15. When the infrared images are captured continuously at a predetermined frame rate, the difference between the capture time of the visible light image and the capture time of the infrared image is equal to or less than the time for one frame at the frame rate. The information processing method according to claim 1 .

16. A processor; A memory. The processor uses the memory to: Acquiring an infrared image generated by capturing light irradiated from a light source and reflected by a subject; acquiring a visible light image generated by capturing an image of the subject using visible light; detecting a flare region based on the infrared image and the visible light image; Information processing device.

17. Acquiring an infrared image generated by capturing light irradiated from a light source and reflected by a subject; acquiring a visible light image generated by capturing an image of the subject using visible light; detecting a flare region based on the infrared image and the visible light image; A program that causes a computer to do something.

Images