Information processing method, information processing apparatus, and computer program product

Abstract

The invention provides an information processing method, an information processing apparatus, and a computer program product. The information processing method acquires an infrared image by imaging light irradiated from a light source and reflected by a subject, acquires a visible light image by imaging the subject with visible light, and detects a glare region on the basis of the infrared image and the visible light image.

Description

[0001] This application is a divisional application of an invention application with an international filing date of September 9, 2019 and a national application number of 201980050094.4. The invention name of the invention application is Depth Acquisition Device, Depth Acquisition Method, and Recording Medium. Technical Field

[0002] The present disclosure relates to a depth acquisition device that acquires the depth of an image, etc. Background Art

[0003] In the past, a distance measurement device that measures the distance to a subject has been proposed (for example, refer to Patent Document 1). The distance measurement device includes a light source and an imaging unit. The light source irradiates light on the subject. The imaging unit images the reflected light reflected by the subject. Then, the distance measurement device measures the distance to the subject by converting the pixel value of each pixel of the image obtained by the imaging into the distance to the subject. That is, the distance measurement device acquires the depth of the image obtained by the imaging unit.

[0004] Prior Art Documents

[0005] Patent Documents

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 2011-64498 Summary of the Invention

[0007] Problems to be Solved by the Invention

[0008] However, in the distance measurement device of Patent Document 1 described above, there is a problem that the depth of the image cannot be correctly acquired.

[0009] Therefore, the present disclosure provides a depth acquisition device that can correctly acquire the depth of an image.

[0010] Means for Solving the Problems

[0011] The depth acquisition device according to one aspect of the present disclosure includes a memory and a processor. The processor acquires timing information indicating the timing at which a light source irradiates infrared light on a subject, acquires an infrared image, the infrared image being obtained by imaging based on infrared light of a scene including the subject corresponding to the timing indicated by the timing information and being held in the memory, acquires a visible light image, the visible light image being obtained by imaging based on visible light of a scene substantially the same as the infrared image at a viewpoint and a imaging time substantially the same as the infrared image and being held in the memory, detects a glare area from the infrared image, and estimates the depth of the glare area based on the infrared image, the visible light image, and the glare area.

[0012] In addition, these general or specific solutions can be implemented using a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or any combination of a system, a method, an integrated circuit, a computer program, and a recording medium. Additionally, the recording medium can be a non-transitory recording medium.

[0013] Effects of the Invention

[0014] The depth acquisition device of the present disclosure can accurately acquire the depth of an image. Further advantages and effects in one aspect of the present disclosure are clarified from the description and the drawings. The related advantages and / or effects are provided separately by several embodiments and the features described in the description and the drawings, but it is not necessary to provide all in order to obtain one or more of the same features. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a block diagram showing the hardware configuration of the depth acquisition device in the embodiment.

[0016] Figure 2 It is a schematic diagram showing the pixel array of the solid-state imaging device in the embodiment.

[0017] Figure 3 It is a timing chart showing the relationship between the light emission timing of the light-emitting element of the light source and the exposure timing of the first pixel of the solid-state imaging device in the embodiment.

[0018] Figure 4 It is a block diagram showing an example of the functional configuration of the depth acquisition device in the embodiment.

[0019] Figure 5 It is a block diagram showing another example of the functional configuration of the depth acquisition device in the embodiment.

[0020] Figure 6 It is a flowchart showing the overall processing operation of the depth acquisition device in the embodiment.

[0021] Figure 7 It is a flowchart showing the overall processing operation of the processor of the depth acquisition device in the embodiment.

[0022] Figure 8 It is a block diagram showing the specific functional configuration of the processor of the depth acquisition device in the embodiment.

[0023] Figure 9A It is a diagram showing an example of an IR image.

[0024] Figure 9B It is a diagram showing an example of a BW image.

[0025] Figure 10This is a diagram showing an example of a binary image obtained by binarizing an IR image.

[0026] Figure 11 This is a diagram showing an example of a high-brightness region within an IR image.

[0027] Figure 12 This is a diagram showing the region of a BW image corresponding to the high-brightness region of an IR image.

[0028] Figure 13 This is a diagram showing an example of a flare region detected from an IR image.

[0029] Figure 14 This is a diagram showing the simulation result of the depth acquisition device in the embodiment.

[0030] Figure 15 This is a diagram showing Figure 8 A flowchart showing the overall processing operation of the depth acquisition device shown.

[0031] Figure 16 This is a diagram showing Figure 15 A flowchart showing the detailed processing of steps S31 to S34 of

[0032] Figure 17 This is a diagram showing an example of a flowchart replacing the processing of steps S31 to S34 of Figure 15 A flowchart showing an example of a flowchart replacing the processing of steps S31 to S34 of

[0033] Figure 18 This is a diagram showing an example of a flowchart replacing the processing of steps S31 to S34 of Figure 15 A flowchart showing another example of the processing of steps S31 to S34 of

[0034] Figure 19 This is a diagram showing an example of a flowchart replacing the processing of steps S31 to S34 of Figure 15 A flowchart showing another example of the processing of steps S31 to S34 of

[0035] Figure 20 This is a block diagram showing an example of the functional structure of the depth acquisition device in Modification 1 of the embodiment.

[0036] Figure 21 This is a block diagram showing another example of the functional structure of the depth acquisition device in Modification 1 of the embodiment.

[0037] Figure 22 This is a flowchart showing the processing operation of the flare detection unit in Modification 2 of the embodiment.

[0038] Figure 23 This is a flowchart showing the overall processing operation of the depth acquisition device in Modification 3 of the embodiment.

[0039] Figure 24 This is a diagram showingFigure 23 Flowchart of the detailed processing of steps S31a to S34a.

[0040] Figure 25 Indicates substitution Figure 23 Flowchart showing an example of the processing of steps S31a to S34a.

[0041] Figure 26 Indicates substitution Figure 23 Flowchart showing another example of the processing of steps S31a to S34a. Detailed implementation mode

[0042] (Insight underlying the present disclosure)

[0043] The inventors of the present invention found the following problems with the distance measuring device of Patent Document 1 described in the "Background Art" section.

[0044] The distance measuring device of Patent Document 1 irradiates light from a light source onto a subject, captures an image of the irradiated subject, thereby obtains an image, and measures the depth of the image. ToF (Time Of Flight) is used in the measurement of this depth. In such a distance measuring device, imaging is performed under mutually different imaging conditions in order to improve the distance measurement accuracy. That is, the distance measuring device performs imaging according to given imaging conditions, and sets imaging conditions different from the given imaging conditions corresponding to the imaging result. Then, the distance measuring device performs imaging again according to the set imaging conditions.

[0045] However, in the image obtained by imaging, there are cases where flare, ghosting, or saturation of brightness occurs. The depth cannot be correctly measured only from the image in which flare or the like occurs. In addition, there are cases where it is difficult to simply suppress the occurrence of flare or the like even when the imaging conditions are changed. Furthermore, for example, if a distance measuring device mounted on a vehicle repeatedly images under mutually different imaging conditions during the running of the vehicle, since the viewpoint positions of the repeatedly performed imaging are different, the scenes of the obtained multiple images will be different. That is, it is not possible to repeatedly image the same scene, and it is not possible to appropriately estimate the depth of the image showing the scene, particularly the depth of the area where flare or the like occurs.

[0046] To solve such problems, a depth acquisition device according to an aspect of the present disclosure includes a memory and a processor. The processor acquires timing information indicating the timing of the light source irradiating the subject with infrared light, acquires an infrared image. The infrared image is obtained by infrared light-based imaging of a scene corresponding to the timing indicated by the timing information and including the subject, and is stored in the memory. The processor acquires a visible light image. The visible light image is obtained by visible light-based imaging of a scene substantially the same as the infrared image, at substantially the same viewpoint and imaging time as the infrared image, and is stored in the memory. The processor detects a flare region from the infrared image, and estimates the depth of the flare region based on the infrared image, the visible light image, and the flare region. In addition, the flare region is a region where flare, ghosting, brightness saturation, or bright lines (スミヤ, smear) occur.

[0047] Accordingly, a flare region is detected from the infrared image. Since the depth of the flare region is detected based not only on the infrared image but also on the visible light image in this flare region, the depth of the flare region can be appropriately acquired. That is, in the infrared image and the visible light image, the scene that is the object of imaging is substantially the same, and the viewpoint and imaging time are also substantially the same. Here, as an example of images of substantially the same scene imaged at substantially the same viewpoint and imaging time, images imaged by different pixels of the same imaging element are used. Such images are similar to the red, green, and blue channel images of a color image imaged with a Bayer-arranged color filter, and the viewing angle, viewpoint, and imaging time of each image are substantially equal. That is, in images of substantially the same scene imaged at substantially the same viewpoint and imaging time, the position of the subject image in each imaged image does not differ by more than 2 pixels. For example, when there is a point light source having visible and infrared components in the scene and only 1 pixel is imaged with high brightness in the visible light image, in the infrared image, the point light source is imaged closer than 2 pixels to the pixel corresponding to the pixel position imaged in the visible light image. In addition, the so-called substantially the same imaging time means that the difference in imaging time is equal to or less than 1 frame. Therefore, the infrared image and the visible light image have a high correlation. In addition, flare and the like are wavelength-dependent phenomena, and even if flare or the like occurs in the infrared image, it is highly likely that flare or the like does not occur in the visible light image. Therefore, information missing in the flare region can be compensated from the region within the visible light image corresponding to the flare region (i.e., the corresponding region). As a result, the depth of the flare region can be appropriately acquired.

[0048] For example, the processor may perform the following: in the estimation of the depth in the glare area, estimate first depth information representing the depth at each position in the infrared image, and estimate second depth information representing the corrected depth at each position in the glare area by correcting the depth at each position in the glare area represented by the first depth information based on the visible light image. Furthermore, generate third depth information representing the depth at each position outside the glare area of the infrared image represented by the first depth information and the depth at each position inside the glare area of the infrared image represented by the second depth information. Additionally, in the estimation of the first depth information, TOF or the like may also be applied to the infrared image.

[0049] Thus, the third depth information represents the depth obtained from this infrared image. As the depth outside the glare area of the infrared image, it represents the depth obtained from this infrared image and corrected based on the visible light image, as the depth of the glare area of the infrared image. Therefore, even when there is a glare area in the infrared image, the overall depth of this infrared image can be appropriately estimated.

[0050] Alternatively, the processor may detect, in the detection of the glare area, an area in the infrared image having a luminance equal to or higher than a first threshold as the glare area.

[0051] Since the luminance in the glare area tends to be higher than that outside the glare area, by detecting an area in the infrared image having a luminance equal to or higher than the first threshold as the glare area, the glare area can be appropriately detected.

[0052] Alternatively, the processor may detect, in the detection of the glare area, an area in the infrared image having a luminance equal to or higher than a first threshold and satisfying a predetermined condition as the glare area, where the predetermined condition is a condition that the correlation value between the image feature amount in the area of the infrared image and the image feature amount in the area of the visible light image corresponding to the area is less than a second threshold. For example, the image feature amount in each area of the infrared image and the visible light image may be the edges contained in the image in the area. Or alternatively, the image feature amount in each area of the infrared image and the visible light image may be the luminance in the area.

[0053] The correlation between the image feature amount in the glare area of the infrared image and the image feature amount in the area of the visible light image corresponding to the glare area tends to be low. Therefore, in the infrared image, by detecting an area with high luminance and low correlation of the image feature amount as the glare area, the glare area can be detected more appropriately.

[0054] Alternatively, in the detection of the flare region, the processor may separately perform the following operations on at least one high-brightness region in the infrared image having a brightness equal to or higher than a first threshold: (i) obtaining a first transformed image by performing a CENSUS transform on the image within the high-brightness region; and (ii) obtaining a second transformed image by performing a CENSUS transform on the image within the region of the visible light image corresponding to the high-brightness region, and detecting, as the flare region, a high-brightness region among the at least one high-brightness regions where the Hamming distance between the first transformed image and the second transformed image exceeds a third threshold.

[0055] Thereby, the flare region can be appropriately detected.

[0056] Alternatively, in the estimation of the depth of the flare region, the processor may estimate depth information representing the depth at each position in the infrared image, and correct the depth at each position in the flare region represented by the depth information by inputting the infrared image, the visible light image, the flare region, and the depth information into a learning model.

[0057] Thereby, by pre-training the learning model to output the correct depth at each position in the flare region for the input of the infrared image, the visible light image, the flare region, and the depth information, the depth information estimated from the infrared image can be appropriately corrected. That is, the depth at each position in the flare region represented by the depth information can be appropriately corrected.

[0058] Alternatively, the depth acquisition device according to other aspects of the present disclosure includes a memory and a processor. The processor obtains timing information indicating the timing of the light source irradiating the subject with infrared light, obtains an infrared image, which is obtained by infrared light-based imaging of a scene including the subject corresponding to the timing indicated by the timing information and is stored in the memory, obtains a visible light image, which is obtained by visible light-based imaging of a scene substantially the same as the infrared image at a substantially the same viewpoint and imaging time as the infrared image and is stored in the memory, estimates depth information representing the depth at each position in the infrared image, and corrects the depth at each position in the flare region of the infrared image represented by the depth information by inputting the infrared image, the visible light image, and the depth information into a learning model.

[0059] Thereby, if the learning model is pre-trained to output the correct depth at each position in the flare region of the infrared image for the input of the infrared image, the visible light image, and the depth information, the depth information estimated from the infrared image can be appropriately corrected. That is, the depth at each position in the flare region represented by the depth information can be appropriately corrected without detecting the flare region.

[0060] In addition, the depth acquisition device involved in other schemes of the present disclosure includes a memory and a processor, wherein the processor acquires an infrared image stored in the memory, wherein the infrared image is obtained by imaging based on infrared light, acquires a visible light image stored in the memory, wherein the visible light image is obtained by imaging based on visible light at substantially the same viewpoint and imaging time as the infrared image, detects an area reflecting glare from the infrared image as a glare area, and estimates the depth of the glare area based on the visible light image. In addition, when the visible light image and the infrared image are respectively divided into a glare area and other areas, the depth of the glare area is estimated based on the visible light image, and the depth of the other areas is estimated based on the infrared image.

[0061] Thereby, similarly to the depth acquisition device according to the above-mentioned one aspect of the present disclosure, the depth of the glare area can be appropriately acquired.

[0062] In addition, these general or specific solutions can be implemented by a system, method, integrated circuit, computer program or computer-readable CD-ROM or other recording medium, or by any combination of systems, methods, integrated circuits, computer programs or recording media. In addition, the recording medium can be a non-transitory recording medium.

[0063] The embodiments are described in detail below with reference to the drawings.

[0064] In addition, the embodiments described below all represent general or specific examples. The numerical values, shapes, materials, components, configuration positions of components, connection forms, steps, and the order of steps shown in the following embodiments are examples and do not limit the subject matter of the present disclosure. In addition, the components in the following embodiments that are not recorded in the independent claims representing the highest concept are described as arbitrary components.

[0065] In addition, each figure is a schematic diagram and is not necessarily a strict illustration. In addition, in each figure, the same reference numeral is attached to the same component.

[0066] (Implementation Method)

[0067] [Hardware structure]

[0068] Figure 1 This is a block diagram showing the hardware structure of the depth acquisition device 1 involved in the embodiment. The depth acquisition device 1 in this embodiment has a hardware structure that can acquire an image based on infrared light (or near-infrared light) and an image based on visible light by shooting substantially the same scene and substantially the same viewpoint and shooting time. In addition, the so-called substantially the same means the same degree as that of being able to achieve the effect in the present disclosure.

[0069] As Figure 1 shown, the depth acquisition device 1 includes a light source 10, a solid-state imaging element 20, a processing circuit 30, a diffusion plate 50, a lens 60, and a band-pass filter 70.

[0070] The light source 10 irradiates irradiation light. More specifically, the light source 10 emits the irradiation light for irradiating the subject at the timing indicated by the light emission signal generated by the processing circuit 30.

[0071] The light source 10 is constituted by, for example, a capacitor, a drive circuit, and a light-emitting element, and emits light by driving the light-emitting element with the electric energy stored in the capacitor. As an example, the light-emitting element is implemented by a laser diode, a light-emitting diode, etc. In addition, the light source 10 may be configured to include one type of light-emitting element, or may be configured to include multiple types of light-emitting elements corresponding to the purpose.

[0072] Hereinafter, as an example, the light-emitting element is a laser diode that emits near-infrared light, or a light-emitting diode that emits near-infrared light, etc. However, the irradiation light irradiated by the light source 10 does not need to be limited to near-infrared light. The irradiation light irradiated by the light source 10 may be, for example, infrared light (also referred to as infrared ray light) in a band other than near-infrared light. In the following embodiments, the irradiation light irradiated by the light source 10 is described as infrared light, but this infrared light may be near-infrared light or infrared light in a band other than near-infrared light.

[0073] The solid-state imaging element 20 images the subject and outputs an imaging signal indicating the exposure amount. More specifically, the solid-state imaging element 20 performs exposure at the timing indicated by the exposure signal generated in the processing circuit 30, and outputs an imaging signal indicating the exposure amount.

[0074] The solid-state imaging element 20 has a pixel array in which a first pixel that images the reflected light reflected by the subject using the irradiation light and a second pixel that images the subject are arranged in an array. The solid-state imaging element 20 may have logical functions such as a protective glass and an AD converter as needed, for example.

[0075] Hereinafter, similar to the irradiation light, the reflected light is described as infrared light, but the reflected light does not need to be limited to infrared light as long as it is the light reflected by the subject from the irradiation light.

[0076] Figure 2 is a schematic diagram showing the pixel array 2 included in the solid-state imaging element 20.

[0077] As Figure 2As shown, the pixel array 2 is configured in an array such that the first pixels 21 (IR pixels) that perform imaging using the reflected light reflected from the subject by the irradiation light and the second pixels 22 (BW pixels) that image the subject are alternately arranged in columns.

[0078] In addition, in Figure 2 , in the pixel array 2, the second pixels 22 and the first pixels 21 are arranged adjacent to each other in the row direction and arranged in a strip shape in the row direction, but it is not limited to this, and they may be arranged every multiple rows (for example, every 2 rows). That is, the first row where the second pixels 22 are arranged adjacent to each other in the row direction and the second row where the first pixels 21 are arranged adjacent to each other in the row direction may be alternately arranged every M rows (M is a natural number). Further, the first row where the second pixels 22 are arranged adjacent to each other in the row direction and the second row where the first pixels 21 are arranged adjacent to each other in the row direction may also be arranged with different rows separated (alternately repeating N rows for the first row and L rows for the second row (N and L are different natural numbers)).

[0079] The first pixels 21 are realized, for example, by infrared light pixels that are sensitive to infrared light as the reflected light. The second pixels 22 are realized, for example, by visible light pixels that are sensitive to visible light.

[0080] The infrared light pixels are composed of, for example, an optical filter that only transmits infrared light (also called an IR filter), a microlens, a light-receiving element as a photoelectric conversion unit, and a storage unit that stores the charge generated by the light-receiving element. Therefore, an image representing the brightness of infrared light is represented by the imaging signals output from the plurality of infrared light pixels (i.e., the first pixels 21) contained in the pixel array 2. Hereinafter, this image of infrared light is also called an IR image or an infrared image.

[0081] In addition, the visible light pixels are composed of, for example, an optical filter that only transmits visible light (also called a BW filter), a microlens, a light-receiving element as a photoelectric conversion unit, and a storage unit that stores the charge converted by the light-receiving element. Therefore, the visible light pixels, that is, the second pixels 22, output imaging signals representing brightness and color difference. That is, a color image representing the brightness and color difference of visible light is represented by the imaging signals output from the plurality of second pixels 22 contained in the pixel array 2. In addition, the optical filter of the visible light pixels may transmit both visible light and infrared light, or may only transmit light in a specific wavelength band such as red (R), green (G), or blue (B) among visible light.

[0082] In addition, the visible light pixels can detect only the brightness of visible light. In this case, the visible light pixels, i.e., the second pixels 22, output a captured image signal representing the brightness. Therefore, a black-and-white image, in other words, a monochromatic image representing the brightness of visible light is represented by the captured image signals output from the plurality of second pixels 22 included in the pixel array 2. Hereinafter, this monochromatic image will also be referred to as a BW image. In addition, the above-described color image and BW image will also be collectively referred to as visible light images.

[0083] Returning again to Figure 1 , continue with the description of the depth acquisition device 1.

[0084] The processing circuit 30 uses the captured image signal output from the solid-state imaging device 20 to calculate the subject information related to the subject.

[0085] The processing circuit 30 includes, for example, an arithmetic processing device such as a microcomputer. The microcomputer includes a processor (microprocessor), a memory, etc., and generates a light emission signal and an exposure signal by the processor executing the driver program stored in the memory. In addition, the processing circuit 30 can use an FPGA or an ISP, etc., and can include one piece of hardware or multiple pieces of hardware.

[0086] The processing circuit 30 calculates the distance to the subject, for example, by the TOF ranging method using the captured image signal from the first pixel 21 of the solid-state imaging device 20.

[0087] Hereinafter, the calculation of the distance to the subject by the TOF ranging method performed by the processing circuit 30 will be described with reference to the accompanying drawings.

[0088] Figure 3 is a timing chart showing the relationship between the light emission timing of the light emitting element of the light source 10 and the exposure timing of the first pixel 21 of the solid-state imaging device 20 when the processing circuit 30 calculates the distance to the subject using the TOF ranging method.

[0089] In Figure 3 , Tp is the light emission period during which the light emitting element of the light source 10 emits the irradiation light, and Td is the delay time from when the irradiation light is emitted from the light emitting element of the light source 10 until the reflected light reflected by the subject returns to the solid-state imaging device 20. And the first exposure period becomes the same timing as the light emission period during which the light source 10 emits the irradiation light, and the second exposure period becomes the timing from the end time point of the first exposure period until the light emission period Tp elapses.

[0090] In Figure 3 , q1 represents the total amount of the exposure amount in the first pixel 21 of the solid-state imaging device 20 caused by the reflected light during the first exposure period, and q2 represents the total amount of the exposure amount in the first pixel 21 of the solid-state imaging device 20 caused by the reflected light during the second exposure period.

[0091] By performing the light emission of the irradiation light by the light emitting element of the light source 10 and the exposure by the first pixel 21 of the solid-state imaging device 20 at the timing shown in Figure 3 , setting the speed of light to c, the distance d to the subject can be expressed by the following (Equation 1).

[0092] d = c × Tp / 2 × q2 / (ql + q2) … (Equation 1)

[0093] Therefore, by using (Equation 1), the processing circuit 30 can calculate the distance to the subject using the imaging signal from the first pixel 21 of the solid-state imaging device 20.

[0094] In addition, the plurality of first pixels 21 of the solid-state imaging device 20 can be exposed only during the third exposure period Tp after the end of the first exposure period and the second exposure period. The plurality of first pixels 21 can detect noise other than the reflected light based on the exposure amount obtained during the third exposure period Tp. That is, the processing circuit 30 can calculate the distance d to the subject more accurately by removing noise from the exposure amount q1 during the first exposure period and the exposure amount q2 during the second exposure period in the above (Equation 1).

[0095] Returning again to Figure 1 , continue with the description of the depth acquisition device 1.

[0096] The processing circuit 30 can, for example, use the imaging signal from the second pixel 22 of the solid-state imaging device 20 to detect the subject and calculate the distance to the subject.

[0097] That is, the processing circuit 30 can detect the subject and calculate the distance to the subject based on the visible light image captured by the plurality of second pixels 22 of the solid-state imaging device 20. Here, the detection of the subject can be achieved, for example, by detecting the edges of the singularities of the subject and performing shape discrimination by pattern recognition, or can be achieved by using a pre-learned learning model through processing such as deep learning. In addition, the calculation of the distance to the subject can be performed using world coordinate transformation. Of course, it is also possible to detect the subject not only using the visible light image but also using the brightness and distance information of the infrared light captured by the first pixel 21 through multi-modal learning processing.

[0098] The processing circuit 30 generates a light emission signal indicating the timing of light emission and an exposure signal indicating the timing of exposure. Then, the processing circuit 30 outputs the generated light emission signal to the light source 10 and outputs the generated exposure signal to the solid-state imaging device 20.

[0099] The processing circuit 30 can, for example, generate and output a light emission signal to cause the light source 10 to emit light at a given period, and generate and output an exposure signal to cause the solid-state imaging element 20 to be exposed at a given period, so that the depth acquisition device 1 can perform continuous imaging based on a given frame rate. In addition, the processing circuit 30 includes, for example, a processor (microprocessor), a memory, etc., and generates a light emission signal and an exposure signal by the processor executing a driver program stored in the memory.

[0100] The diffusion plate 50 adjusts the intensity distribution and angle of the irradiation light. In addition, in the adjustment of the intensity distribution, the diffusion plate 50 makes the intensity distribution of the irradiation light from the light source 10 uniform. In addition, in Figure 1 In the example shown, the depth acquisition device 1 includes the diffusion plate 50, but it may not include the diffusion plate 50.

[0101] The lens 60 is an optical lens that condenses light entering from the outside of the depth acquisition device 1 onto the surface of the pixel array 2 of the solid-state imaging element 20.

[0102] The band-pass filter 70 is an optical filter that transmits infrared light and visible light as reflected light. In addition, in Figure 1 In the example shown, the depth acquisition device 1 includes the band-pass filter 70, but it may not include the band-pass filter 70.

[0103] The depth acquisition device 1 with the above structure is mounted on a transportation device for use. For example, the depth acquisition device 1 is mounted on a vehicle traveling on a road for use. In addition, the transportation device on which the depth acquisition device 1 is mounted does not necessarily need to be limited to a vehicle. The depth acquisition device 1 can also be mounted on a transportation device other than a vehicle, such as a motorcycle, a ship, an airplane, etc., for use.

[0104] [Overview of the Depth Acquisition Device]

[0105] The depth acquisition device 1 in this embodiment uses Figure 1 the hardware structure shown to acquire an IR image and a BW image by imaging the same scene, the same viewpoint, and the same time. Then, the depth acquisition device 1 uses the BW image to correct the depth at each position in the IR image obtained from the IR image. Specifically, when there is a flare region described later in the IR image, the depth acquisition device 1 uses the image in the region of the BW image corresponding to the flare region to correct the depth at each position in the flare region obtained from the IR image.

[0106] Figure 4 is a block diagram showing an example of the functional structure of the depth acquisition device 1.

[0107] The depth acquisition device 1 includes a light source 101, an IR camera 102, a BW camera 103, a depth estimation unit 111, and a flare detection unit 112.

[0108] The light source 101 may include Figure 1 the light source 10 and the diffusion plate 50 shown.

[0109] The IR camera 102 may include Figure 1 a plurality of first pixels 21 of the solid-state imaging element 20, a lens 60, and a band-pass filter 70 shown. Such an IR camera 102 acquires an IR image by performing IR-based imaging of a scene including the subject at a timing corresponding to the infrared light irradiation of the subject by the light source 101.

[0110] The BW camera 103 may include Figure 1 a plurality of second pixels 22 of the solid-state imaging element 20, a lens 60, and a band-pass filter 70 shown. Such a BW camera 103 acquires a visible light image (specifically, a BW image) by performing visible light-based imaging of a scene substantially the same as the IR image and imaging at substantially the same viewpoint and imaging time as the IR image.

[0111] The depth estimation unit 111 and the flare detection unit 112 may be implemented as Figure 1 the functions of the processing circuit 30 shown, specifically, the functions of the processor 110.

[0112] The flare detection unit 112 detects a flare region from the IR image based on the IR image obtained by imaging with the IR camera 102 and the BW image obtained by imaging with the BW camera 103.

[0113] The flare region in the present embodiment is a region where flare, ghosting, saturation of brightness, or bright lines occur. Flare is over-sensitivity of light generated by harmful light reflected on the lens surface or lens barrel when the lens of the IR camera 102 is directed towards a strong light source. In addition, in flare, the image becomes white and loses sharpness. Ghosting is a type of flare and is a phenomenon where light repeatedly reflected complexly on the lens surface is clearly captured as an image. A bright line is a phenomenon where a linear white portion is generated when the camera captures a subject having a brightness difference greater than a given value compared to the surroundings.

[0114] In addition, in the present disclosure, a phenomenon including at least one of flare, ghosting, saturation of brightness, and bright lines is referred to as flare and the like.

[0115] The depth estimation unit 111 estimates the depth at each position in the R image including the flare region detected by the flare detection unit 112. Specifically, the depth estimation unit 111 obtains an IR image captured by the IR camera 102 corresponding to the timing when the light source 101 irradiates the subject with infrared light, and estimates the depth at each position in the IR image based on this IR image. Further, the depth estimation unit 111 corrects the depth at each position estimated in the flare region detected by the flare detection unit 112 based on the BW image. That is, the depth estimation unit 111 estimates the depth of the flare region based on the IR image, the BW image, and the flare region.

[0116] Figure 5 It is a block diagram showing another example of the functional structure of the depth acquisition device 1.

[0117] The depth acquisition device 1 may include a memory 200 and a processor 110.

[0118] In addition, the processor 110 may not only include a depth estimation unit 111 and a flare detection unit 112, but also Figure 5 include a light emission timing acquisition unit 113, an IR image acquisition unit 114, and a BW image acquisition unit 115 as shown. In addition, these components are implemented as functions of the processor 110.

[0119] The light emission timing acquisition unit 113 acquires timing information indicating the timing when the light source 101 irradiates the subject with infrared light. That is, the light emission timing acquisition unit 113 outputs the Figure 1 shown light emission signal to the light source 101 to acquire information indicating the timing of this output as the above-mentioned timing information.

[0120] The IR image acquisition unit 114 acquires an IR image obtained by capturing a scene including the subject based on infrared light corresponding to the timing indicated by the timing information, and stores it in the memory 200.

[0121] The BW image acquisition unit 115 acquires a BW image obtained by capturing a scene based on visible light that is substantially the same as the above-mentioned IR image, at substantially the same viewpoint and at the same imaging time as the IR image, and stores it in the memory 200.

[0122] The flare detection unit 112 detects the flare region from the IR image as described above, and the depth estimation unit 111 estimates the depth based on this IR image, the BW image, and the flare region.

[0123] Alternatively, the depth acquisition device 1 in the present embodiment may not include the light source 101, the IR camera 102, and the BW camera 103, but may include a processor 110 and a memory 200.

[0124] Figure 6 It is a flowchart showing the overall processing operation of the depth acquisition device 1.

[0125] (Step S11)

[0126] First, the light source 101 emits light to irradiate the subject with infrared light.

[0127] (Step S12)

[0128] Next, the IR camera 102 acquires an IR image. That is, the IR camera 102 captures an image of a scene including the subject irradiated with infrared light by the light source 101. Thus, the IR camera 102 acquires an IR image based on the infrared light reflected from the subject. Specifically, the IR camera 102 acquires an IR image obtained through Figure 3 the timings and exposure amounts of the first exposure period, the second exposure period, and the third exposure period shown.

[0129] (Step S13)

[0130] Next, the BW camera 103 acquires a BW image. That is, the BW camera 103 acquires a BW image corresponding to the IR image acquired in step S12, that is, a BW image of the same scene and the same viewpoint as the IR image.

[0131] (Step S14)

[0132] Then, the flare detection unit 112 detects a flare area from the IR image acquired in step S12.

[0133] (Step S15)

[0134] Next, the depth estimation unit 111 estimates the depth of the flare area based on the IR image acquired in step S12, the BW image acquired in step S13, and the flare area detected in step S14.

[0135] Figure 7 It is a flowchart showing the overall processing operation of the processor 110 of the depth acquisition device 1.

[0136] (Step S21)

[0137] First, the light emission timing acquisition unit 113 of the processor 110 acquires timing information indicating the timing at which the light source 101 irradiates the subject with infrared light.

[0138] (Step S22)

[0139] Next, the IR image acquisition unit 114 acquires an IR image from the IR camera 102 that has performed imaging corresponding to the timing indicated by the timing information acquired in step S21. For example, the IR image acquisition unit 114 outputs an exposure signal to the IR camera 102 at the timing of the light emission signal shown in Figure 1 Thus, the IR image acquisition unit 114 causes the IR camera 102 to start imaging and acquires the IR image obtained by this imaging from the IR camera 102. At this time, the IR image acquisition unit 114 can acquire the IR image from the IR camera 102 via the memory 200 or directly from the IR camera 102.

[0140] (Step S23)

[0141] Next, the BW image acquisition unit 115 acquires a BW image corresponding to the IR image acquired in step S22 from the BW camera 103. At this time, the BW image acquisition unit 115 can acquire the BW image from the BW camera 103 via the memory 200 or directly from the BW camera 103.

[0142] (Step S24)

[0143] Then, the flare detection unit 112 detects a flare region from the IR image.

[0144] (Step S25)

[0145] Next, the depth estimation unit 111 estimates the depth of the flare region based on the IR image acquired in step S22, the BW image acquired in step S23, and the flare region detected in step S24. Thus, at least depth information indicating the depth of the flare region is calculated. In addition, at this time, the depth estimation unit 111 can estimate not only the flare region but also the overall depth of the IR image and calculate depth information indicating the estimation result.

[0146] Specifically, the depth estimation unit 111 in the present embodiment estimates the depth at each position in the IR image acquired in step S22. Then, the depth estimation unit 111 corrects the depth at each position in the flare region using the BW image. In addition, each position can be the position of each of a plurality of pixels or the position of a block including a plurality of pixels.

[0147] In the depth acquisition device 1 in such an embodiment, a flare region is detected from an IR image, and in this flare region, the depth is estimated based not only on the IR image but also on the BW image. Therefore, the depth of this flare region can be appropriately acquired. That is, in the IR image and the BW image, the scene to be photographed is substantially the same, and the viewpoints and the photographing times are also substantially the same. Therefore, the IR image and the BW image have a high correlation. In addition, flares and the like are wavelength-dependent phenomena. Even if flares or the like occur in the IR image, there is a high possibility that they do not occur in the BW image. Therefore, the information missing in the flare region can be compensated from the region within the BW image corresponding to the flare region (i.e., the corresponding region). As a result, the depth of the flare region can be appropriately acquired.

[0148] [Specific functional structure of the depth acquisition device]

[0149] Figure 8 It is a block diagram showing the specific functional structure of the processor 110 of the depth acquisition device 1.

[0150] The processor 110 includes a first depth estimation unit 111a, a second depth estimation unit 111b, a flare detection unit 112, a high-brightness region detection unit 116, a first edge detection unit 117IR, a second edge detection unit 117BW, and an output unit 118. In addition, the first depth estimation unit 111a and the second depth estimation unit 111b correspond to Figure 5 the depth estimation unit 111 shown. In addition, the processor 110 may include the above-mentioned light emission timing acquisition unit 113, IR image acquisition unit 114, and BW image acquisition unit 115.

[0151] The high-brightness region detection unit 116 detects, in the IR image, a region having a brightness equal to or higher than a first threshold as a high-brightness region. The first edge detection unit 117IR detects the edge located in the IR image. The second edge detection unit 117BW detects the edge located in the BW image.

[0152] The flare detection unit 112 compares, for each of at least one high-brightness region in the IR image, the edge detected for the high-brightness region with the edge detected for the region within the BW image corresponding to the high-brightness region. Through this comparison, the flare detection unit 112 determines whether the high-brightness region is a flare region. That is, the flare region is detected by this determination. In other words, the flare detection unit 112 performs region segmentation of the IR image by discriminating the photographed IR image into a flare region and a non-flare region that is not a flare region.

[0153] Here, phenomena such as flare are wavelength-dependent. Therefore, flare and the like generated in the IR image are mostly not generated in the BW image. Generally, it is known that the IR image and the BW image have a strong correlation. However, among the flare and the like generated in the IR image, due to the edge collapse of the IR image, the correlation value between the edge of the region where the flare and the like are generated and the edge of the region of the BW image corresponding to the region becomes low. In addition, when flare and the like are generated, the brightness of the region where they are generated becomes large. Therefore, the flare detection unit 112 in the present embodiment discriminates the flare region from the captured IR image using this relationship.

[0154] That is, the flare detection unit 112 in the present embodiment detects, as the flare region, a region in the IR image having a brightness equal to or higher than the first threshold and satisfying a predetermined condition. The predetermined condition is a condition that the correlation value between the image feature amount in the region of the IR image and the image feature amount in the region of the BW image corresponding to the region is less than the second threshold. Here, the image feature amount in each region of the IR image and the BW image is the edge contained in the image in the region. In addition, the region of the BW image corresponding to the region of the IR image is a region located at the same spatial position as the region of the IR image, having the same shape and size as the region of the IR image.

[0155] As described above, the correlation between the image feature amount in the flare region of the IR image and the image feature amount in the region of the BW image corresponding to the flare region tends to be low. Therefore, in the present embodiment, by detecting, in the IR image, a region having a high brightness and a low correlation of the image feature amount as the flare region, the flare region can be detected more appropriately.

[0156] The first depth estimation unit 111a and the second depth estimation unit 111b have the functions of the depth estimation unit 111 described above.

[0157] The first depth estimation unit 111a estimates the depth at each position in the IR image based on the IR image obtained corresponding to the timing of the irradiation of the infrared light from the light source 101. The first depth estimation unit 111a outputs the information indicating the depth at each position in the estimated IR image as the first depth information. That is, the first depth estimation unit 111a estimates the first depth information indicating the depth at each position in the IR image.

[0158] The second depth estimation unit 111b corrects the first depth information based on the flare regions in the BW image and the IR image. Thereby, the depth of the flare region among the depths at each position in the IR image represented by the first depth information is corrected. The second depth estimation unit 111b outputs, as the second depth information, information indicating the corrected depths at each position within the flare region. That is, the second depth estimation unit 111b estimates the second depth information indicating the corrected depths at each position within the flare region by correcting the depths at each position within the flare region represented by the first depth information based on the BW image.

[0159] The output unit 118 replaces the depth at each position within the flare region represented by the first depth information with the corrected depth at each position within the flare region represented by the second depth information. Thereby, third depth information is generated that includes the depth at each position outside the flare region of the IR image represented by the first depth information and the corrected depth at each position within the flare region of the IR image represented by the second depth information. The output unit 118 outputs the third depth information.

[0160] Thereby, the third depth information represents the depth obtained from the IR image. As the depth outside the flare region of the IR image, it represents the depth obtained from the IR image and corrected based on the BW image. As the depth of the flare region of the IR image. Therefore, in the present embodiment, even when there is a flare region in the IR image, the overall depth of the IR image can be appropriately estimated.

[0161] Figure 9A Shows an example of an IR image. Figure 9B Shows an example of a BW image.

[0162] As Figure 9B shown, a scene where a sign is placed on a road is reflected in the BW image. The sign includes, for example, a material that easily reflects infrared light. For this reason, if the IR camera 102 captures an image of the same scene as the scene shown Figure 9B from the same viewpoint as the viewpoint of the BW camera 103, the IR image shown Figure 9A is obtained.

[0163] The IR image obtained as described above is as Figure 9AAs shown, glare with high brightness is generated in the area including the range equivalent to the sign of the BW image. This is because the infrared light from the light source 101 is specularly reflected by the road sign, and the infrared light with strong intensity enters the IR camera 102 as reflected light. In addition, materials that are prone to reflecting infrared light are often used in the clothes worn by construction workers or multiple poles erected along the road. Therefore, when shooting a scene including a subject using such materials, there is a high possibility of generating glare or the like in the IR image. However, the possibility of generating glare or the like in the BW image is low. As a result, the correlation between the image feature amount of the glare area in the IR image and the image feature amount of the area of the BW image corresponding to the glare area becomes low. On the other hand, the correlation between the image feature amount of the area outside the glare area (i.e., the non-glare area) in the IR image and the image feature amount of the area of the BW image corresponding to the non-glare area becomes high.

[0164] Figure 10 Shows an example of a binary image obtained by binarizing the IR image.

[0165] The high-brightness area detection unit 116 detects, in the Figure 9A shown IR image, an area with a brightness equal to or higher than the first threshold as a high-brightness area. That is, the high-brightness area detection unit 116 binarizes the brightness at each position (i.e., each pixel) in the IR image. As a result, for example, as Figure 10 shown, a binary image including a white area and a black area ( Figure 10 the shaded area in ) is generated.

[0166] Figure 11 Shows an example of a high-brightness area in the IR image.

[0167] The high-brightness area detection unit 116 detects the white area in the binary image as a high-brightness area. For example, in the case where there are 6 white areas in the binary image as Figure 11 shown, the high-brightness area detection unit 116 detects the 6 white areas as high-brightness areas A to F. That is, the IR image or the binary image area is divided into 6 high-brightness areas A to F and a non-high-brightness area that is not a high-brightness area.

[0168] Figure 12 Shows the area of the BW image corresponding to the high-brightness area of the IR image.

[0169] The flare detection unit 112 determines the image feature amounts of the regions in the BW image corresponding to at least one high-brightness region in the binary image (i.e., the IR image). Additionally, the image feature amount is, for example, an edge. Further, the region in the BW image corresponding to the high-brightness region is a region that is located at a spatially identical position to the high-brightness region in the binary image or the IR image, and has the same shape and size as the high-brightness region. Hereinafter, the region in the BW region corresponding to the region of the IR image will also be referred to as a corresponding region.

[0170] For example, in the case where high-brightness regions A to F are detected as shown in Figure 11 , the flare detection unit 112 determines the image feature amounts of the regions in the BW image corresponding to these high-brightness regions A to F, respectively.

[0171] Figure 13 This shows an example of the flare region detected from the IR image.

[0172] The flare detection unit 112 determines for each of the high-brightness regions A to F whether the high-brightness region is a flare region. That is, the flare detection unit 112 determines whether the high-brightness region is a flare region by comparing the image feature amount of the high-brightness region A in the IR image with the image feature amount of the corresponding region in the BW image corresponding to the high-brightness region A. As a result, for example, as shown in Figure 13 , the flare detection unit 112 determines that the high-brightness regions A, C, D, and E among the high-brightness regions A to F are flare regions.

[0173] Figure 14 This shows the simulation result of the depth acquisition device 1.

[0174] The depth acquisition device 1 acquires the BW image shown in (a) of Figure 14 through the imaging of the BW camera 103, and further acquires the IR image shown in (b) of Figure 14 through the imaging of the IR camera 102. The BW image and the IR image are images obtained by imaging the same scene at the same viewing point and imaging time. In the example shown in (b) of Figure 14 , a large flare region is generated at the right end of the IR image.

[0175] The first depth estimation unit 111a generates the first depth information shown in (c) of Figure 14 by estimating the depth from the IR image. The first depth information is represented as a first depth image indicating the depth at each position in the IR image by the brightness. In this first depth image, the depth of the flare region is inappropriately represented.

[0176] The second depth estimation unit 111b corrects inappropriate depth in the flare region. Then, as shown in (e) of Figure 14 , the output unit 118 generates third depth information indicating the corrected depth of the flare region and the depth of the non-flare region. Similar to the first depth information, this third depth information is represented as a third depth image in which depth is represented by luminance. In addition, the second depth estimation unit 111b can also correct the depth of the non-flare region in the first depth image based on the image feature amount of the corresponding region of the BW image.

[0177] In this way, in the depth acquisition device 1 of the present embodiment, in the entire image including the flare region, the third depth image can be made closer to Figure 14 the correct depth image shown in (d) of

[0178] [Specific processing flow of depth acquisition device]

[0179] Figure 15 is a flowchart showing the overall processing operation of the depth acquisition device 1 shown in Figure 8 .

[0180] (Step S31)

[0181] First, the high-brightness region detection unit 116 detects a high-brightness region from the IR image.

[0182] (Step S32)

[0183] The first edge detection unit 117IR detects the edge located in the IR image.

[0184] (Step S33)

[0185] The second edge detection unit 117BW detects the edge located in the BW image.

[0186] (Step S34)

[0187] The flare detection unit 112 detects a flare region in the IR image by comparing the edge in at least one high-brightness region of the IR image with the edge in the corresponding region of the BW image. That is, the flare detection unit 112 detects the high-brightness region as a flare region when the correlation value between the edge in the high-brightness region and the edge in the corresponding region of the BW image is less than the second threshold. Thus, the IR image is regionally divided into at least one flare region and a non-flare region.

[0188] (Step S35)

[0189] The first depth estimation unit 111a generates first depth information from the IR image using, for example, TOF.

[0190] (Step S36)

[0191] The second depth estimation unit 111b generates second depth information representing the depth of the flare region based on the first depth information of the IR image and the BW image.

[0192] (Step S37)

[0193] The output unit 118 generates third depth information by replacing the depth of the flare region represented by the first depth information with the depth represented by the second depth information.

[0194] Figure 16 represents Figure 15 a flowchart of the detailed processing of steps S31 to S34.

[0195] (Step S41)

[0196] First, the high-brightness region detection unit 116 determines whether the brightness at each position in the IR image is equal to or greater than the first threshold. Here, for example, if the IR image is a 12-bit grayscale image, the first threshold is about 1500. Of course, this first threshold can be a value that varies according to environmental conditions or the settings of the IR camera 102. For example, in the case of photographing a dark scene such as at night, since the overall brightness of the IR image is low, the first threshold can be a value smaller than that in the case of photographing a bright scene during the day. In addition, when the exposure time of the IR camera 102 is long, since the overall brightness of the IR image is high, the first threshold can be a value larger than that in the case of a short exposure time.

[0197] (Step S42)

[0198] Here, if it is determined that the brightness at none of the positions is equal to or greater than the first threshold (No in step S41), the high-brightness region detection unit 116 determines that no flare has occurred in the IR image (step S42). That is, the entire IR image is determined to be a non-flare region.

[0199] (Step S43)

[0200] On the other hand, if it is determined that the brightness at any position is equal to or greater than the first threshold (Yes in step S41), the high-brightness region detection unit 116 performs region segmentation on the IR image. That is, the high-brightness region detection unit 116 divides the IR image into at least one high-brightness region and a region other than the high-brightness region. In this region segmentation, for example, a luminance-based method such as SuperPixel can be used.

[0201] (Step S44)

[0202] Next, the first edge detection unit 117IR and the second edge detection unit 117BW perform edge detection on the IR image and the BW image, respectively. In edge detection, the Canny method, the Sobel filter, or the like can be used.

[0203] (Step S45)

[0204] The flare detection unit 112 compares the edges in at least one high-brightness region of the IR image with the edges in the region of the BW image corresponding to the high-brightness region. That is, the flare detection unit 112 determines whether the correlation value between the edges in the high-brightness region of the IR image and the edges in the corresponding region of the BW image is equal to or greater than the second threshold value. The respective values output by edge detection of the IR image and the BW image are arranged in a vector shape for each region, and the inner product value is normalized to obtain the correlation value. That is, the flare detection unit 112 normalizes the inner product value of the vector including a plurality of values obtained by edge detection in the high-brightness region of the IR image and the vector including a plurality of values obtained by edge detection in the region of the BW image corresponding to the high-brightness region. Thus, the correlation value for the high-brightness region is calculated.

[0205] (Step S46)

[0206] Here, if it is determined that the correlation value is not equal to or greater than the second threshold value, that is, less than the second threshold value (No in Step S45), the flare detection unit 112 determines that the high-brightness region is a flare region. That is, due to the influence of flare or the like, the correlation of the edges between the region where flare or the like occurs in the IR image and the corresponding region of the BW image disappears. Therefore, the flare detection unit 112 discriminates the high-brightness region in the IR image as a flare region.

[0207] On the other hand, if the flare detection unit 112 determines in Step S45 that the correlation value is equal to or greater than the second threshold value, that is, not less than the second threshold value (Yes in Step S45), it is determined that no flare or the like has occurred in the high-brightness region. That is, the flare detection unit 112 determines that the high-brightness region is a non-flare region.

[0208] In such a method, IR images and BW images with substantially equal viewpoint positions are required. In the depth acquisition device 1 in the present embodiment, for each pixel, the filter used in the pixel is set to either an IR filter or a BW filter. That is, as Figure 2 shown, the first pixel 21 having an IR filter and the second pixel 22 having a BW filter are alternately arranged in the column direction. Thus, since IR images and BW images with substantially the same viewpoint can be obtained, the flare region can be appropriately discriminated.

[0209] <Using the correlation of luminance>

[0210] In the above description, edges are used in the discrimination between the glare area and the non-glare area, but edges may not be used in this discrimination. For example, the correlation values of the brightnesses of the IR image and the BW image themselves can be used. As described above, when there is no glare or the like, the IR image and the BW image have a strong correlation, but in the area where glare or the like occurs, the correlation becomes weak. Therefore, by using the correlation of the brightnesses of the IR image and the BW image, the glare area can be appropriately discriminated.

[0211] Figure 17 represents substitution Figure 15 A flowchart showing an example of the processing of steps S31 to S34. That is, Figure 17 is a flowchart showing the detection process of the glare area using the correlation of the brightnesses of the IR image and the BW image. In addition, in Figure 17 in, in the same step as Figure 16 the same reference numerals are given, and detailed descriptions are omitted. Figure 17 The flowchart shown in Figure 16 is different from the flowchart shown in

[0212] (Step S45a)

[0213] In step S45a, the glare detection unit 112 calculates the correlation value between the brightness of each pixel in the high-brightness area obtained by the area division in step S43 and the brightness of each pixel in the area of the BW image corresponding to the high-brightness area. The brightnesses of each pixel of the IR image and the BW image are arranged in a vector shape for each area, and the inner product value is normalized by the number of pixels to obtain the correlation value. That is, the glare detection unit 112 normalizes the inner product value of the vector including the brightness of each pixel in the high-brightness area of the IR image and the vector including the brightness of each pixel in the corresponding area of the BW image. Thus, the correlation value for the high-brightness area is calculated.

[0214] Here, when the correlation value is equal to or greater than the second threshold, that is, not less than the second threshold (step S45a "Yes"), the glare detection unit 112 determines that no glare or the like has occurred in the high-brightness area (step S42). On the other hand, when the correlation value is less than the second threshold (step S45a "No"), the correlation between the brightness of each pixel in the high-brightness area of the IR image and the brightness of each pixel in the corresponding area of the BW image becomes low due to the influence of glare or the like. Therefore, in such a case, the glare detection unit 112 discriminates the high-brightness area as a glare area (step S46).

[0215] That is, the image feature amounts in the areas of the IR image and the BW image used for the detection of the glare area are inFigure 16 In the example shown, they are the edges contained in the image within this region, but in Figure 17 the example shown, they are the brightness within this region. Here, as described above, the correlation between the brightness within the flare region of the IR image and the brightness within the region of the BW image corresponding to this flare region tends to be low. Therefore, in the IR image, by detecting as the flare region the region with high brightness and low correlation of this brightness, the flare region can be detected more appropriately.

[0216] <Using CENSUS transform>

[0217] Of course, the evaluation value for discriminating between the flare region and the non-flare region does not need to be the correlation value. For example, the Hamming distance and the CENSUS transform can be used. Regarding the CENSUS transform, it is disclosed, for example, in the non-patent literature (R. Zabih and J. Woodfill, “Non-parametric Local Transforms for Computing Visual Correspondence”, Proc. of ECCV, pp. 151-158, 1994). The CENSUS transform sets a window in the image and transforms the size relationship between the central pixel of this window and the surrounding pixels into a binary vector.

[0218] Figure 18 represents replacing Figure 15 the processes of steps S31 to S34. That is, Figure 18 is a flowchart showing another example of the processes of steps S31 to S34. In addition, in Figure 18 the same reference numerals are given to the same steps as Figure 16 and detailed description thereof is omitted. Figure 18 The flowchart shown in Figure 16 is different from the flowchart shown in

[0219] (Step S44b)

[0220] In step S44b, the flare detection unit 112 performs the CENSUS transform on the image of the high-brightness region in the IR image and the image of the corresponding region in the BW image for each high-brightness region obtained by the region division in step S43. Thereby, a CENSUS transform image of the image of the high-brightness region in the IR image and a CENSUS transform image of the image of the corresponding region in the BW image are generated.

[0221] (Step S45b)

[0222] Next, the flare detection unit 112 calculates the Hamming distance between the CENSUS transformed image of the IR image obtained in step S44b and the CENSUS transformed image of the BW image in step S45b. Then, when the value obtained by normalizing the Hamming distance by the number of pixels in the high-brightness region is equal to or less than the third threshold (step S45b "Yes"), it is determined that no flare or the like has occurred in the high-brightness region (step S42). On the other hand, when the value of the normalized Hamming distance is greater than the third threshold (step S45b "No"), the flare detection unit 112 determines that the texture has disappeared from the image in the high-brightness region due to the influence of flare or the like. As a result, the flare detection unit 112 determines the high-brightness region as a flare region (step S46).

[0223] That is, for each of at least one high-brightness region in the IR image having a brightness equal to or higher than the first threshold, the flare detection unit 112 obtains a first transformed image by performing a CENSUS transform on the image within the high-brightness region. Then, the flare detection unit 112 obtains a second transformed image by performing a CENSUS transform on the image within the region of the BW image corresponding to the high-brightness region. In addition, the first transformed image and the second transformed image are the above-mentioned CENSUS transformed images. Next, the flare detection unit 112 detects, as a flare region, a high-brightness region in which the Hamming distance between the first transformed image and the second transformed image exceeds the third threshold among at least one high-brightness region. By using the CENSUS transform in this way, the flare region can also be appropriately detected.

[0224] <Using the brightness of the IR image>

[0225] In Figure 17 the illustrated example, the brightness of the IR image and the correlation value between the brightness of the IR image and the BW image are used in the determination of the flare region and the non-flare region, but the brightness of the IR image alone may also be used.

[0226] Figure 19 represents Figure 15 is a flowchart showing another example of the processing of steps S31 to S34. That is, Figure 19 is a flowchart showing the detection process of the flare region using only the brightness of the IR image. In addition, in Figure 19 the same steps as those in Figure 16 are denoted by the same reference numerals, and detailed descriptions thereof are omitted. In the flowchart shown in Figure 19 the steps S43 to S45 shown in Figure 16 are omitted.

[0227] That is, in step S41, the flare detection unit 112 determines whether the luminance of a pixel in the IR image is equal to or higher than the first threshold. Here, if it is determined that the luminance of the pixel is less than the first threshold (step S41, "No"), the flare detection unit 112 determines that no flare has occurred in the region including the pixel (step S42). On the other hand, if it is determined that the luminance of the pixel is equal to or higher than the first threshold (step S41, "Yes"), the flare detection unit 112 determines that a flare has occurred in the region including the pixel. That is, the flare detection unit 112 determines the region as a flare region (step S46).

[0228] That is, in Figure 19 the example shown, the flare detection unit 112 detects, as a flare region, a region in the IR image having a luminance equal to or higher than the first threshold. Since the luminance within the flare region tends to be higher than the luminance outside the flare region, by detecting, as a flare region, a region in the IR image having a luminance equal to or higher than the first threshold, the flare region can be appropriately detected.

[0229] In addition, in the discrimination between the flare region and the non-flare region, learning processing can be used. In the learning processing, for example, processing such as deep learning can be used. In this case, for learning, an IR image, a BW image, and a ground truth image that divides the IR image into a flare region and a non-flare region are prepared in advance. Next, the IR image and the BW image are provided as inputs to the learning model. Then, the learning model is made to learn so that the output from the learning model for the input matches the ground truth image. For example, the learning model is a neural network. The output from the learning model is an image showing the numerical value "0" or the numerical value "1" for each pixel, where the numerical value "0" indicates that the pixel belongs to the non-flare region and the numerical value "1" indicates that the pixel belongs to the flare region.

[0230] The flare detection unit 112 discriminates between the flare region and the non-flare region by using the learning model that has been learned in advance in this way. That is, the flare detection unit 112 inputs the IR image and the BW image as inputs to the learning model. Then, the flare detection unit 112 determines the region including the pixels corresponding to the numerical value "0" output from the learning model as the non-flare region. Further, the flare detection unit 112 determines the region including the pixels corresponding to the numerical value "1" output from the learning model as the flare region.

[0231] Through the above processing, the flare detection unit 112 divides the captured IR image into a flare region where flares occur, etc., and a non-flare region.

[0232] <Depth correction processing>

[0233] The second depth estimation unit 111b generates second depth information using the BW image, the first depth information, and the flare region (i.e., the discrimination result of the above-mentioned region).

[0234] Flare and the like are phenomena that depend on the wavelength of light. For this reason, flare and the like generated in the IR image are mostly not generated in the BW image. For this reason, only the first depth information obtained from the IR image in the flare region is corrected by using the BW image to obtain second depth information that is not affected by flare and the like generated in the IR image.

[0235] In obtaining the second depth information, a guided filter, which is a type of image correction filter, can be used. The guided filter is disclosed in a non-patent document (Kaiming He, Jian Sun and Xiaoou Tang, “Guided Image Filtering”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.35, Iss.6, Pp.1397-1409, 2013.). The guided filter is a filter that corrects an object image by using the correlation between the object image and a reference image. In the guided filter, the reference image I and the object image p are assumed to be represented by parameters a and b as shown in the following (Equation 2).

[0236] [Mathematical formula 1]

[0237]

[0238] Here, q is the output image obtained by correcting the object image p, i is the number of each pixel, and ωk represents the surrounding region of pixel k. In addition, the parameters a and b are represented by the following (Equation 3).

[0239] [Mathematical formula 2]

[0240]

[0241] Among them, ε is a normalization parameter. In addition, μ and σ are the mean and variance within the block of the reference image, and are calculated by the following (Equation 4).

[0242] [Mathematical formula 3]

[0243]

[0244] Among them, in order to suppress the noise contained in the obtained parameters a and b, the output is obtained as shown in the following (Equation 5) by using the averaged parameters.

[0245] [Mathematical formula 4]

[0246]

[0247] In this embodiment, the second depth estimation unit 111b corrects the object image, that is, the first depth information (or the first depth image), by giving a BW image as a reference image. Thereby, the second depth information is generated or obtained. In the generation of such second depth information, IR images and BW images with substantially equal viewing point positions are required. In the depth acquisition device 1 of this embodiment, for each pixel, the filter used in that pixel is set to either an IR filter or a BW filter. That is, as Figure 2 shown, the first pixel 21 having an IR filter and the second pixel 22 having a BW filter are alternately arranged in the column direction. Thereby, since IR images and BW images with substantially the same viewing point can be obtained, appropriate second depth information can be obtained.

[0248] Of course, the second depth estimation unit 111b may also use processing other than the guided filter. For example, the second depth estimation unit 111b may utilize a bilateral filter (Non-Patent Document: C. Tomasi, R. Manduchi, “Bilateral filtering for gray and color images”, IEEE International Conference on Computer Vision (ICCV), pp. 839-846, 1998), or processing such as Mutual-Structure for Joint Filtering (Non-Patent Document: Xiaoyong Shen, Chao Zhou, Li Xu and Jiaya Jia, “Mutual-Structure for Joint Filtering”, IEEE International Conference on Computer Vision (ICCV), 2015.).

[0249] As described above, in this embodiment, the first depth information is used in the region where it is determined that no flare or the like is generated (i.e., the non-flare region), and the second depth information is used in the region where flare or the like is generated (i.e., the flare region). Thereby, even if flare or the like is generated in the IR image, higher-precision depth information can be obtained.

[0250] (Modification Example 1)

[0251] In the above embodiment, a filter such as a guided filter is used in the generation of the second depth information, but a learning model may also be used to generate the second depth information.

[0252] For example, deep learning, which is a learning process, can be utilized as in the non-patent literature (Shuran Song, Fisher Yu, Andy Zeng, Angel X. Chang, Manolis Savva and Thomas Funkhouser, “Semantic Scene Completion from a Single Depth Image”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 190-198, 2017.). That is, the learning model can be made to learn so that when a BW image and first depth information are input, second depth information is output. In the above non-patent literature, a network is proposed that interpolates a missing region of the depth information when depth information and a color image containing the missing region are input. The second depth estimation unit 111b in this modified example provides the IR image, the BW image, and the first depth information to the same network (i.e., the learning model) as in this non-patent literature, and further provides a mask image of the flare region detected by the flare detection unit 112 as the missing region. Thereby, higher-precision second depth information can be obtained from this network.

[0253] Figure 20 FIG. is a block diagram showing an example of the functional structure of the depth acquisition device 1 in this modified example.

[0254] The depth acquisition device 1 in this modified example includes Figure 8 each of the components shown, and further includes a learning model 104 such as a neural network.

[0255] The second depth estimation unit 111b inputs three types of data, namely the IR image, the BW image, and the first depth information, to the learning model 104, and uses the flare region as the corrected mask region to generate the second depth information.

[0256] In the learning of the learning model 104, in addition to the IR image, the BW image, and the first depth information, a ground truth depth image is also prepared in advance. Next, the IR image, the BW image, the first depth information, and a mask image specifying the flare region are provided as inputs to the learning model 104. Then, the learning model 104 is made to learn so that the output from the learning model 104 for this input matches the ground truth depth image. Additionally, during learning, the mask image is given randomly. The second depth estimation unit 111b uses the learning model 104 that has been learned in this way. That is, the second depth estimation unit 111b can obtain the second depth information output from the learning model 104 by inputting the IR image, the BW image, the first depth information, and the mask image specifying the flare region into the learning model 104.

[0257] In this way, in this modification example, the second depth estimation unit 111b estimates the depth information indicating the depth at each position within the IR image, and corrects the depth at each position within the flare region represented by this depth information by inputting the IR image, the BW image, the flare region, and this depth information into the learning model. Therefore, if the learning model is pre-learned so that the output for the input of the IR image, the BW image, the flare region, and the depth information is the ground truth depth at each position within the flare region, the depth information estimated from the IR image can be appropriately corrected. That is, the depth at each position within the flare region represented by the depth information can be appropriately corrected.

[0258] As described above, the second depth estimation unit 111b can use Deep Learning. In this case, it is not necessary to directly output the flare region, and the second depth information can be directly generated by Deep Learning.

[0259] Figure 21 It is a block diagram showing another example of the functional structure of the depth acquisition device 1 in this modification example.

[0260] The depth acquisition device 1 in this modification example does not have Figure 20 among the respective components shown, such as the flare detection unit 112, the high-brightness region detection unit 116, the first edge detection unit 117IR, and the second edge detection unit 117BW, but has components other than these.

[0261] In the learning of the learning model 104, similar to Figure 20Similarly, for the example shown, in addition to the IR image, the BW image, and the first depth information, a ground truth depth image is prepared in advance. Next, the IR image, the BW image, and the first depth information are provided as inputs to the learning model 104. Then, the learning model 104 is made to learn so that the output from the learning model 104 for this input is consistent with the ground truth depth image. As the learning model 104, a VGG-16 network with skip connections added can be used as in the non-patent literature (Caner Hazirbas, Laura Leal-Taixe and Daniel Cremers C. Hazirbas, “Deep Depth From Focus”, In ArXiv preprint arXiv, 1704.01085, 2017.). The number of channels of the learning model 104 is changed so that the IR image, the BW image, and the first depth information are given as inputs to the learning model 104. By using the learning model 104 that has been pre-learned in this way, the second depth estimation unit 111b can simply obtain the second depth information from the learning model 104 by inputting the IR image, the BW image, and the first depth information into the learning model 104.

[0262] That is, Figure 21 the depth acquisition device 1 shown includes a memory and a processor 110. In addition, Figure 21 although not shown in the figure, the memory can be loaded into the depth acquisition device 1 as Figure 5 shown. The processor 110 acquires timing information indicating the timing at which the light source 101 irradiates the subject with infrared light. Next, the processor 110 acquires an IR image, which is obtained by capturing an infrared light-based image of a scene including the subject corresponding to the timing indicated by the timing information and is held in the memory. Next, the processor 110 acquires a BW image, which is obtained by capturing a visible light-based image of substantially the same scene as the IR image and at substantially the same viewpoint and capture time as the IR image and is held in the memory. Then, the first depth estimation unit 111a of the processor 110 estimates depth information indicating the depth at each position in the IR image. The second depth estimation unit 111b corrects the depth at each position in the glare area of the IR image represented by the depth information by inputting the IR image, the BW image, and the depth information into the learning model 104.

[0263] Therefore, if the learning model 104 is pre-learned so that, for the input of the IR image, the BW image, and the depth information, the correct depth at each position in the glare area of the IR image is output, the depth information estimated from the IR image can be appropriately corrected. That is, the depth at each position in the glare area represented by the depth information can be appropriately corrected without detecting the glare area.

[0264] (Modification Example 2) Utilization of the Temporal Direction

[0265] Here, the flare detection unit 112 may also utilize information in the temporal direction for discriminating between a flare region and a non-flare region. Flare is a phenomenon that occurs inside the IR camera 102 not due to the subject itself but due to the relationship between the subject and the light source 101. For this reason, when the IR camera 102 moves, the shape of the flare region changes. For this reason, the flare detection unit 112 detects candidates for the flare region from the IR image by the above-described method, and determines whether the shape of the candidate has changed based on the flare region detected in the previous IR image. Then, if the flare detection unit 112 determines that the shape has not changed, it can be determined that the candidate is not a flare region but a non-flare region.

[0266] Figure 22 It is a flowchart showing the processing operation of the flare detection unit 112 in this modification example.

[0267] (Step S51)

[0268] First, the flare detection unit 112 determines whether there are candidates for the flare region in the target frame. That is, the flare detection unit 112 discriminates, for example, whether the flare region detected based on the flowchart shown above is not the final flare region and thus exists as a candidate for the flare region in the target frame. In addition, the target frame is the IR image set as the discrimination target, that is, the IR image for discriminating the existence of the flare region. Figures 16 - 19 shown flowchart and detected flare region is not the final flare region and thus exists as a candidate for the flare region in the target frame. In addition, the target frame is the IR image set as the discrimination target, that is, the IR image for discriminating the existence of the flare region.

[0269] (Step S54)

[0270] Here, when there are no candidates for the flare region in the target frame (No in Step S51), the flare detection unit 112 determines that there is no flare in the target frame (i.e., the IR image).

[0271] (Step S52)

[0272] On the other hand, when there are candidates for the flare region in the target frame (Yes in Step S51), the flare detection unit 112 determines whether there is also a flare region in the frame before the target frame. In addition, the frame before the target frame is the IR image obtained by photographing with the IR camera 102 before the target frame.

[0273] (Step S55)

[0274] Here, when there is no flare region in the frame before the target frame (No in Step S52), the flare detection unit 112 determines that the detected candidate for the flare region is the region where flare is generated, that is, the flare region.

[0275] (Step S53)

[0276] On the other hand, when there is a flare area in the frame before the object frame (Yes in Step S52), the flare detection unit 112 compares the shape of the candidate for the flare area of the object frame with the shape of the flare area of the previous frame. Here, when the shape of the candidate for the flare area of the object frame is similar to the shape of the flare area of the previous frame (Yes in Step S53), the flare detection unit 112 performs the process of Step S54. That is, the flare detection unit 112 updates the detected candidate for the flare area to a non-flare area and determines that there is no flare in the object frame (i.e., the IR image). On the other hand, when the shape of the candidate for the flare area of the object frame is not similar to the shape of the flare area of the previous frame (No in Step S53), the flare detection unit 112 performs the process of Step S55. That is, the flare detection unit 112 determines that the detected candidate for the flare area is the area where flare is generated, i.e., the flare area.

[0277] In this way, the flare detection unit 112 in this modification uses the information in the time direction. That is, the flare detection unit 112 uses the shape of the flare area or its candidate in each frame obtained at different times. Thereby, the flare area and the non-flare area can be discriminated with higher accuracy.

[0278] In addition, the flare detection unit 112 may discriminate the flare area not for each frame but for every plurality of adjacent frames. In addition, the plurality of adjacent frames are, for example, a plurality of IR images continuously obtained in time by the imaging of the IR camera 102. That is, the flare detection unit 112 may also detect candidates for the flare area in each of the plurality of adjacent frames based on, for example, the Figures 16 - 19 flowchart shown above, and determine whether these candidates are flare areas. More specifically, the flare detection unit 112 compares the shapes of the candidates for the flare area in each frame, and when the shapes of these candidates for the flare area are substantially equal, determines that these candidates are not flare areas, that is, determines that they are non-flare areas. On the other hand, when the shapes of these candidates for the flare area are not similar, the flare detection unit 112 determines that these candidates are flare areas.

[0279] Alternatively, the flare detection unit 112 may determine whether these shapes are similar by determining whether the similarity of the two shapes is above a threshold value. This similarity can be calculated, for example, as the correlation value of the two shapes.

[0280] (Modification 3) [Use of Multiple IR Source Images]

[0281] In the above-described embodiments, the depth acquisition device 1 in Modifications 1 and 2 thereof uses an IR image and a BW image to detect a flare area, but the BW image may not be used. The depth acquisition device 1 in this modification uses a plurality of IR source images to detect a flare area. The plurality of IR source images are, for example, an infrared image obtained during the first exposure period shown in Figure 3 and an infrared image obtained during the second exposure period.

[0282] That is, in this modification, when the IR camera 102 acquires a plurality of infrared images at mutually different timings in order to estimate the first depth information using TOF or the like, these plurality of infrared images are used to discriminate between a flare area and a non-flare area. In the following description, each of the plurality of infrared images is referred to as an IR source image. In addition, it can be said that the above-described IR image is composed of these plurality of IR source images.

[0283] In depth estimation based on TOF, there are two types: direct TOF that directly measures the arrival time of emitted light, and indirect TOF that estimates depth from a plurality of IR source images obtained due to different timings of light emission and light reception. In this modification, the depth acquisition device 1 discriminates between a flare area and a non-flare area from the plurality of IR source images obtained in the case of indirect TOF.

[0284] The timings for obtaining each of the plurality of IR source images, that is, the light reception timings of the plurality of IR source images are different. Therefore, even if flare occurs in the first IR source image among the plurality of IR source images, it is highly likely that flare does not occur in the second IR source image having a light reception timing different from that of the first IR source image. For this reason, the depth acquisition device 1 in this modification compares the plurality of IR source images obtained by changing the light reception timing, and thereby discriminates between a flare area and a non-flare area. As a result, the third depth image can be made closer to Figure 14 the correct depth image shown in (d) of

[0285] [Specific processing flow of depth acquisition device]

[0286] Figure 23 is a flowchart showing the overall processing operation of the depth acquisition device 1 in this modification.

[0287] (Step S31a)

[0288] First, the high-brightness area detection unit 116 detects a high-brightness area from the first IR source image.

[0289] (Step S32a)

[0290] The first edge detection unit 117 IR detects the edge located in the first IR source image.

[0291] (Step S33a)

[0292] The second edge detection unit 117BW detects the edges located in the second IR source image instead of the BW image.

[0293] (Step S34a)

[0294] The flare detection unit 112 detects the flare region in the IR image by comparing the edges in at least one high-brightness region of the first IR source image with the edges in the corresponding region of the second IR source image respectively. That is, when the correlation value between the edge in the high-brightness region and the edge in the corresponding region of the second IR source image is less than the fourth threshold, the flare detection unit 112 detects the high-brightness region as the flare region. Thus, the IR image region is divided into at least one flare region and a non-flare region.

[0295] (Step S35)

[0296] The first depth estimation unit 111a generates the first depth information from the IR image using, for example, TOF.

[0297] (Step S36)

[0298] The second depth estimation unit 111b generates the second depth information representing the depth of the flare region based on the first depth information of the IR image and the BW image.

[0299] (Step S37)

[0300] The output unit 118 generates the third depth information by replacing the depth of the flare region represented by the first depth information with the depth represented by the second depth information.

[0301] Figure 24 represents Figure 23 The flowchart of the detailed processing of steps S31a to S34a.

[0302] (Step S41a)

[0303] First, the high-brightness region detection unit 116 determines whether the brightness at each position in the first IR source image is equal to or higher than the fifth threshold. Here, for example, if the first IR source image is a 12-bit grayscale image, the fifth threshold can be about 1500. Of course, the fifth threshold can be a value that varies according to the environmental conditions or the settings of the IR camera 102. For example, when shooting a dark scene such as at night, since the overall brightness of the first IR source image is low, the fifth threshold can be a value smaller than that for shooting a bright scene during the day. In addition, when the exposure time of the IR camera 102 is long, since the overall brightness of the first IR source image is high, the fifth threshold can be a value larger than that for a short exposure time.

[0304] (Step S42a)

[0305] Here, if it is determined that the luminance at any position is not above the fifth threshold value (step S41a "No"), the high-luminance region detection unit 116 determines that flare has not occurred in the IR image formed using the first IR source image (step S42). That is, the entire IR image is determined as a non-flare region.

[0306] (Step S43a)

[0307] On the other hand, if it is determined that the luminance at an arbitrary position is above the fifth threshold value (step S41a "Yes"), the high-luminance region detection unit 116 performs region division on the first IR source image. That is, the high-luminance region detection unit 116 divides the first IR source image into at least one high-luminance region and a region other than the high-luminance region. In this region division, for example, a luminance-based method such as SuperPixel can be used.

[0308] (Step S44a)

[0309] Next, the first edge detection unit 117IR and the second edge detection unit 117BW perform edge detection on the first IR source image and the second IR source image, respectively. In edge detection, the Canny method or Sobel filter, etc. can be used.

[0310] (Step S45a)

[0311] The flare detection unit 112 compares the edges within at least one high-luminance region of the first IR source image with the edges within the region of the second IR source image corresponding to the high-luminance region. That is, the flare detection unit 112 determines whether the correlation value between the edges within the high-luminance region of the first IR source image and the edges within the corresponding region of the second IR source image is above the fourth threshold value. The respective values output through edge detection of the first IR source image and the second IR source image are arranged in a vector shape for each region, and the inner product value is normalized to obtain the correlation value. That is, the flare detection unit 112 normalizes the inner product value of the vector including a plurality of values obtained through edge detection within the high-luminance region of the first IR source image and the vector including a plurality of values obtained through edge detection within the region of the second IR source image corresponding to the high-luminance region. Thereby, the correlation value for the high-luminance region is calculated.

[0312] (Step S46a)

[0313] Here, if it is determined that the correlation value is not above the fourth threshold, that is, less than the fourth threshold (step S45a "No"), the glare detection unit 112 determines that this high-brightness area is a glare area. That is, due to the influence of glare or the like, the correlation of the edges disappears between the area where glare or the like is generated in the first IR source image and the corresponding area of the second IR source image. Therefore, the glare detection unit 112 discriminates this high-brightness area in the first IR source image as a glare area. That is, the glare detection unit 112 discriminates this high-brightness area in the IR image as a glare area. In addition, the high-brightness area in the IR image is the same area as the high-brightness area in the first IR source image.

[0314] On the other hand, if the glare detection unit 112 determines in step S45a that the correlation value is above the fourth threshold, that is, not less than the fourth threshold (step S45a "Yes"), it is determined that no glare or the like is generated in this high-brightness area. That is, the glare detection unit 112 determines that this high-brightness area is a non-glare area.

[0315] <Using the correlation of brightness>

[0316] In the above description, edges are used in the discrimination between glare areas and non-glare areas, but edges may not be used in the discrimination. For example, the correlation value of the brightness itself of the first IR source image and the second IR source image can be used. As described above, in the case where no glare or the like is generated, the first IR source image and the second IR source image have a strong correlation, but in the area where glare or the like is generated, the correlation becomes weak. Therefore, by using the correlation of the brightness of the first IR source image and the second IR source image, the glare area can be appropriately discriminated.

[0317] Figure 25 It represents replacing Figure 23 The flowchart of an example of the processing of steps S31a to S34a. That is, Figure 25 It is a flowchart showing the detection process of the glare area using the correlation of the brightness of the first IR source image and the second IR source image respectively. In addition, in Figure 25 For the same steps as those in Figure 24 The same reference numerals are used, and the detailed description is omitted. Figure 25 The flowchart shown in Figure 24 is different from the flowchart shown in

[0318] (Step S45b)

[0319] In step S45b, the flare detection unit 112 calculates the correlation value between the brightness of each pixel in each high-brightness region obtained by region segmentation in step S43a and the brightness of each pixel in the region of the second IR source image corresponding to the high-brightness region. The brightness of each pixel of the first IR source image and the second IR source image is arranged in a vector form for each region, and the inner product value is normalized by the number of pixels, thereby obtaining the correlation value. That is, the flare detection unit 112 normalizes the inner product value of the vector including the brightness of each pixel in the high-brightness region of the first IR source image and the vector including the brightness of each pixel in the corresponding region of the second IR source image. Thus, the correlation value for the high-brightness region is calculated.

[0320] Here, when the correlation value is equal to or greater than the fourth threshold, that is, not less than the fourth threshold (step S45b "Yes"), the flare detection unit 112 determines that no flare or the like has occurred in the high-brightness region (step S42a). On the other hand, when the correlation value is less than the fourth threshold (step S45b "No"), due to the influence of flare or the like, the correlation between the brightness of each pixel in the high-brightness region of the first IR source image and the brightness of each pixel in the corresponding region of the second IR source image becomes low. Therefore, in such a case, the flare detection unit 112 determines the high-brightness region as a flare region (step S46a).

[0321] That is, the image feature amounts in the respective regions of the first IR source image and the second IR source image used for detecting the flare region are Figure 24 the edges included in the image in the region in the example shown, but Figure 25 the brightness in the region in the example shown. Here, as described above, the correlation between the brightness in the flare region of the first IR source image and the brightness in the region of the second IR source image corresponding to the flare region tends to be low. Therefore, in the first IR source image, by detecting the region with high brightness and low correlation of the brightness as the flare region, the flare region can be detected more appropriately.

[0322] <Using CENSUS transform>

[0323] Of course, the evaluation value used for discriminating the flare region and the non-flare region does not need to be the correlation value. For example, the aforementioned Hamming distance and CENSUS transform can also be used.

[0324] Figure 26 represents other examples of the processes in steps S31a to S34a that replace Figure 23 . That is, Figure 26 is a flowchart showing the detection process of the flare region using the CENSUS transform for the first IR source image and the second IR source image respectively. In addition, in Figure 26 , forFigure 24 The same steps are given the same reference numerals, and detailed descriptions thereof are omitted. Figure 26 The flowchart shown is different from Figure 24 the flowchart shown, and includes steps S44c and S45c in place of steps S44a and S45a.

[0325] (Step S44c)

[0326] In step S44c, the flare detection unit 112 performs a CENSUS transform on the image of each high-brightness region obtained by the region division in step S43a in the first IR source image and the image of the corresponding region in the second IR source image, respectively. Thus, a CENSUS transform image of the image of the high-brightness region in the first IR source image and a CENSUS transform image of the image of the corresponding region in the second IR source image are generated.

[0327] (Step S45c)

[0328] Next, in step S45c, the flare detection unit 112 calculates the Hamming distance between the CENSUS transform image of the first IR source image obtained in step S44c and the CENSUS transform image of the second IR source image. Then, when the value obtained by normalizing the Hamming distance by the number of pixels in the high-brightness region is equal to or less than the sixth threshold (step S45c "Yes"), the flare detection unit 112 determines that no flare or the like has occurred in the high-brightness region (step S42a). On the other hand, when the value of the normalized Hamming distance is greater than the sixth threshold (step S45c "No"), the flare detection unit 112 determines that the texture has disappeared in the image of the high-brightness region due to the influence of flare or the like. As a result, the flare detection unit 112 determines the high-brightness region as a flare region (step S46a).

[0329] That is, the flare detection unit 112 obtains a first transform image by performing a CENSUS transform on the image within at least one high-brightness region having a brightness equal to or higher than the fifth threshold in the first IR source image. Then, the flare detection unit 112 obtains a second transform image by performing a CENSUS transform on the image within the region of the second IR source image corresponding to the high-brightness region. In addition, the first transform image and the second transform image are the above-mentioned CENSUS transform images. Next, the flare detection unit 112 detects, as flare regions, the high-brightness regions in which the Hamming distance between the first transform image and the second transform image exceeds the sixth threshold among at least one high-brightness region. By using the CENSUS transform in this way, flare regions can also be appropriately detected.

[0330] As described above, in the depth acquisition device 1 in the present embodiment and its modified examples, even when there is a glare area in the IR image, appropriate depths at respective positions within the glare area can be acquired by using the image of the corresponding area of the BW image.

[0331] In addition, in each of the above-described embodiments, each component may be configured by dedicated hardware or may be implemented by executing a software program suitable for each component. Each component may be implemented by a program execution unit such as a CPU or a processor reading out and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software for implementing the depth acquisition device and the like in the above-described embodiment and modified examples causes a computer to execute Figure 6 、 Figure 7 、 Figures 15 - 19 and Figures 22 - 26 each step included in any one of the flowcharts.

[0332] The depth acquisition device related to one or more solutions has been described based on the embodiment and its modified examples above, but the present disclosure is not limited to the embodiment and its modified examples. As long as it does not deviate from the gist of the present disclosure, solutions obtained by applying various modifications conceived by those skilled in the art to the present embodiment and its modified examples, and configurations constructed by combining the components in the embodiment and its modified examples may also be included within the scope of the present disclosure.

[0333] In addition, in the present disclosure, all or a part of a component, device, or Figure 1 、 Figure 4 、 Figure 5 、 Figure 8 、 Figure 20 and Figure 21All or part of the functional blocks of the block diagram shown can be executed by a semiconductor device, one or more electronic circuits including a semiconductor integrated circuit (IC) or LSI (large scale integration). The LSI or IC can be integrated on a single chip or formed by combining multiple chips. For example, functional blocks other than storage elements can be integrated on a single chip. Here, it is called LSI or IC, but depending on the degree of integration, the name may change and it may be called system LSI, VLSI (very large scale integration), or ULSI (ultra large scale integration). For the same purpose, a field programmable gate array (FPGA, Field Programmable Gate Array) programmed after the manufacture of the LSI, or a reconfigurable logic device capable of reconstructing the bonding relationship inside the LSI or setting the circuit division inside the LSI can also be used.

[0334] Furthermore, all or part of the functions or operations of a component, device, or part of a device can be executed by software processing. In this case, the software is recorded on one or more non-transitory recording media such as ROM, optical disc, hard disk drive, etc. When the software is executed by a processing device (processor), the software causes the processing device (processor) and peripheral devices to execute specific functions within the software. The system or device may include one or more non-transitory recording media for recording the software, a processing device (processor), and necessary hardware devices such as interfaces.

[0335] Industrial Applicability

[0336] The present disclosure can be applied to a depth acquisition device that obtains depth from an image obtained by shooting, and can be used, for example, as in-vehicle equipment.

[0337] Explanation of Reference Numerals

[0338] 1 Depth acquisition device

[0339] 10, 101 Light source

[0340] 20 Solid-state imaging element

[0341] 21 First pixel (IR)

[0342] 22 Second pixel (BW)

[0343] 30 Processing circuit

[0344] 50 Diffusion plate

[0345] 60 lens

[0346] 70 band - pass filter

[0347] 102 IR camera

[0348] 103 BW camera

[0349] 104 learning model

[0350] 110 processor

[0351] 111 depth estimation unit

[0352] 111a First depth estimation unit

[0353] 111b Second depth estimation unit

[0354] 112 flare detection unit

[0355] 113 emission timing acquisition unit

[0356] 114 IR image acquisition unit

[0357] 115 BW image acquisition unit

[0358] 116 high - brightness area detection unit

[0359] 117 First IR edge detection unit

[0360] 117 Second BW edge detection unit

[0361] 118 output unit

[0362] 200 memory

Claims

1. An information processing method, obtaining an infrared image by performing imaging that receives light irradiated from a light source and reflected by a subject, obtaining a visible light image by imaging the subject with visible light, detecting a glare area based on the infrared image and the visible light image.

2. The information processing method according to claim 1, wherein in the detection of the glare area, the glare area is detected based on an area in the infrared image corresponding to an area in the visible light image.

3. The information processing method according to claim 1 or 2, wherein in the detection of the glare area, the glare area is detected based on the brightness of an area in the infrared image corresponding to an area in the visible light image.

4. The information processing method according to claim 3, wherein in the detection of the glare area, it is detected that the brightness of an area in the infrared image corresponding to an area in the visible light image is equal to or higher than a threshold value.

5. The information processing method according to claim 4, wherein in the detection of the glare area, when the brightness of an area in the infrared image corresponding to an area in the visible light image is equal to or higher than the threshold value, the area in the infrared image is detected as the glare area.

6. The information processing method according to claim 1, wherein in the detection of the glare area, the glare area is detected based on the infrared image and the visible light image generated by imaging the same scene as the infrared image.

7. The information processing method according to claim 1, wherein in the detection of the glare area, the glare area is detected based on the infrared image and the visible light image generated by imaging from the same viewpoint as the infrared image.

8. The information processing method according to claim 1, wherein in the detection of the glare area, the glare area is detected based on the infrared image and the visible light image generated by imaging at the same time as the infrared image.

9. The information processing method according to claim 1, wherein In the acquisition of the infrared image, T O F, i.e., time of flight, is used.

10. The information processing method according to claim 1, wherein in the detection of the glare area, based on the infrared image and the visible light image, an area in the infrared image having a brightness equal to or higher than a first threshold value and satisfying a predetermined condition is detected as the glare area.

11. The information processing method according to claim 10, wherein the predetermined condition is a condition that a correlation value between an image feature amount in an area of the infrared image and an image feature amount in an area of the visible light image corresponding to the area is less than a second threshold value.

12. The information processing method according to claim 11, wherein the image feature amount in each area of the infrared image and the visible light image is an edge included in the image in the area.

13. The information processing method according to claim 11, wherein the image feature amount in each area of the infrared image and the visible light image is the brightness in the area.

14. The information processing method according to claim 1, wherein, the acquisition of the infrared image and the acquisition of the visible light image are performed by a solid-state imaging device, the solid-state imaging device includes a plurality of first pixels for imaging the infrared image and a plurality of second pixels different from the plurality of first pixels and for imaging the visible light image.

15. The information processing method according to claim 1, wherein, when the imaging of the infrared image is continuously performed at a given frame rate, the difference between the imaging time of the visible light image and the imaging time of the infrared image is within the time of one frame amount of the frame rate.

16. An information processing apparatus comprising a memory and a processor, the processor uses the memory, to acquire an infrared image generated by performing imaging of light reflected by a subject irradiated from a light source, to acquire a visible light image generated by imaging the subject with visible light, and to detect a flare region based on the infrared image and the visible light image.

17. A computer program product including a program for causing a computer to perform the following steps: acquire an infrared image generated by performing imaging of light reflected by a subject irradiated from a light source, acquire a visible light image generated by imaging the subject with visible light, and detect a flare region based on the infrared image and the visible light image.

Images