Depth acquisition device, depth acquisition method, and recording medium

Abstract

A depth acquisition device capable of correctly acquiring the depth of an image is provided. The depth acquisition device (1) comprises a memory (200) and a processor (110), wherein the processor (110) acquires timing information indicating the timing at which a light source (101) irradiates infrared light to a subject, acquires an IR image obtained by capturing a scene including the subject based on infrared light at the timing indicated by the timing information and stored in the memory (200), acquires a BW image obtained by capturing a scene substantially the same as the IR image based on visible light and at substantially the same viewpoint and capturing time as the IR image and stored in the memory (200), detects a glare area from the IR image, and estimates the depth of the glare area based on the IR image, the BW image, and the glare area.

Description

Technical Field

[0001] The present disclosure relates to a depth acquisition device and the like for acquiring the depth of an image. Background Art

[0002] In the past, a distance measuring device for measuring the distance to a subject has been proposed (for example, refer to Patent Document 1). The distance measuring device includes a light source and an imaging unit. The light source irradiates light to the subject. The imaging unit captures the reflected light reflected by the subject. Then, the distance measuring device measures the distance to the subject by converting each pixel value of the image obtained by the imaging into the distance to the subject. That is, the distance measuring device obtains the depth of the image obtained by the imaging unit.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2011-64498 Summary of the invention

[0006] Problems to be solved by the invention

[0007] However, the distance measuring device of Patent Document 1 has a problem that the depth of an image cannot be accurately acquired.

[0008] To this end, the present disclosure provides a depth acquisition device that can accurately acquire the depth of an image.

[0009] Means for solving problems

[0010] A depth acquisition device involved in one scheme of the present disclosure includes a memory and a processor, wherein the processor obtains timing information indicating the timing of a light source irradiating infrared light to a subject, obtains an infrared image obtained by infrared-light-based imaging of a scene corresponding to the timing indicated by the timing information and including the subject, and the image is stored in the memory; obtains a visible light image obtained by visible-light-based imaging of a scene substantially the same as the infrared image and at substantially the same viewpoint and imaging time as the infrared image, and the image is stored in the memory; detects a glare area from the infrared image, and estimates the depth of the glare area based on the infrared image, the visible light image, and the glare area.

[0011] In addition, these general or specific solutions can be implemented by a system, method, integrated circuit, computer program or a computer-readable CD-ROM or other recording medium, or by any combination of the system, method, integrated circuit, computer program and recording medium. In addition, the recording medium can be a non-transitory recording medium.

[0012] Effects of the Invention

[0013] The depth acquisition device disclosed in the present invention can correctly acquire the depth of an image. Further advantages and effects in one scheme of the present invention are made clear by the specification and the accompanying drawings. The relevant advantages and / or effects are provided respectively by several embodiments and the features recorded in the specification and the accompanying drawings, but it is not necessary to provide all of them in order to obtain one or more of the same features. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a block diagram showing the hardware configuration of the depth acquisition device in the embodiment.

[0015] Figure 2 It is a schematic diagram showing a pixel array included in the solid-state imaging element according to the embodiment.

[0016] Figure 3 This is a timing chart showing the relationship between the light emission timing of the light emitting element of the light source and the exposure timing of the first pixel of the solid-state imaging element in the embodiment.

[0017] Figure 4 This is a block diagram showing an example of the functional structure of the depth acquisition device in the embodiment.

[0018] Figure 5 This is a block diagram showing another example of the functional structure of the depth acquisition device in the embodiment.

[0019] Figure 6 This is a flowchart showing the overall processing operation of the depth acquisition device in the embodiment.

[0020] Figure 7 This is a flowchart showing the overall processing operation of the processor of the depth acquisition device in the embodiment.

[0021] Figure 8 This is a block diagram showing a specific functional structure of a processor of the depth acquisition device in the embodiment.

[0022] Fig. 9A This is a diagram showing an example of an IR image.

[0023] Fig. 9B This is a diagram showing an example of a BW image.

[0024] Fig.10 This is a diagram showing an example of a binarized image obtained by binarizing an IR image.

[0025] Fig.11 This is a diagram showing an example of a high-brightness area in an IR image.

[0026] Fig.12It is a diagram showing a region of the BW image corresponding to a high-brightness region of the IR image.

[0027] Fig.13 This is a diagram showing an example of a glare area detected from an IR image.

[0028] Fig.14 It is a diagram showing the simulation results of the depth acquisition device in the embodiment.

[0029] Fig.15 Yes means Figure 8 The flowchart of the overall processing operation of the depth acquisition device is shown.

[0030] Fig.16 Yes means Fig.15 Flowchart of detailed processing of steps S31 to S34.

[0031] Fig.17 It means to replace Fig.15 Flowchart of an example of the processing of steps S31 to S34.

[0032] Fig.18 It means to replace Fig.15 This is a flowchart of another example of the processing of steps S31 to S34.

[0033] Fig.19 It means to replace Fig.15 This is a flowchart of another example of the processing of steps S31 to S34.

[0034] Fig. 20 This is a block diagram showing an example of the functional configuration of the depth acquisition device in the first variation of the embodiment.

[0035] Fig.21 This is a block diagram showing another example of the functional structure of the depth acquisition device in the first modification of the embodiment.

[0036] Fig. 22 This is a flowchart showing the processing operation of the glare detection unit in Modification 2 of the embodiment.

[0037] Fig.23 This is a flowchart showing the overall processing operation of the depth acquisition device in Modification 3 of the embodiment.

[0038] Fig.24 Yes means Fig.23 Flowchart of detailed processing of steps S31a to S34a.

[0039] Fig.25 It means to replace Fig.23 Flowchart of an example of the processing of steps S31a to S34a.

[0040] Fig.26 It means to replace Fig.23 This is a flowchart of another example of the processing of steps S31a to S34a. DETAILED DESCRIPTION

[0041] (Insights that form the basis of this disclosure)

[0042] The inventors of the present invention have found that the following problems occur with respect to the distance measuring device of Patent Document 1 described in the “Background Art” column.

[0043] The distance measuring device of patent document 1 irradiates light from a light source to a subject as described above, captures an image of the irradiated subject, thereby obtaining an image, and measuring the depth of the image. ToF (Time Of Flight) is used in the measurement of the depth. In such a distance measuring device, different imaging conditions are used to improve the distance measurement accuracy. That is, the distance measuring device captures images according to given imaging conditions, and sets imaging conditions different from the given imaging conditions according to the imaging results. Then, the distance measuring device captures images again according to the set imaging conditions.

[0044] However, in the images obtained by camera, glare, ghosting, or brightness saturation may occur. The depth cannot be accurately measured only from the image that produces glare. In addition, even if the camera conditions are changed, it is difficult to simply suppress the generation of glare. Furthermore, if, for example, a distance measuring device mounted on a vehicle repeatedly performs camera shooting under different camera conditions while the vehicle is traveling, the scenes of the multiple images obtained will be different because the viewpoint positions of the repeated cameras are different. That is, the same scene cannot be repeatedly photographed, and the depth of the image reflecting the scene, especially the depth of the area where glare occurs, cannot be properly estimated.

[0045] In order to solve such a problem, a depth acquisition device involved in one scheme of the present disclosure includes a memory and a processor, wherein the processor acquires timing information indicating the timing of irradiating infrared light to a subject by a light source, acquires an infrared image obtained by infrared light-based imaging of a scene corresponding to the timing indicated by the timing information and including the subject, and is stored in the memory, acquires a visible light image obtained by visible light-based imaging of a scene substantially the same as the infrared image and at substantially the same viewpoint and imaging time as the infrared image, and is stored in the memory, detects a glare area from the infrared image, and estimates the depth of the glare area based on the infrared image, the visible light image, and the glare area. In addition, the glare area is an area where glare, ghosting, brightness saturation, or bright lines (smear) are generated.

[0046] Thus, the glare area is detected from the infrared image, and since the depth in the glare area is detected not only based on the infrared image but also based on the visible light image, the depth of the glare area can be appropriately obtained. That is, in the infrared image and the visible light image, the scene that becomes the object of the camera is substantially the same, and the viewpoint and the shooting time are also substantially the same. Here, as an example of an image of substantially the same scene captured at substantially the same viewpoint and shooting time, it is an image captured by different pixels of the same imaging element. Such an image is similar to the red, green and blue channel images of a color image captured by a Bayer arrangement color filter, and the viewing angle, viewpoint and shooting time of each image are approximately equal. That is, the position of the image of the substantially same scene captured at substantially the same viewpoint and shooting time on the image of the subject in each captured image will not differ by more than 2 pixels. For example, when there is a point light source having visible light and infrared components in the scene, and only one pixel is photographed at high brightness in the visible light image, the point light source will also be photographed in the infrared image at a position closer to the pixel corresponding to the position of the pixel photographed in the visible light image. In addition, the so-called substantially the same photographing time means that the difference in photographing time is equal to less than one frame. Therefore, the infrared image and the visible light image are highly correlated. In addition, glare and the like are phenomena that depend on wavelength, and even if glare and the like are generated in the infrared image, there is a high possibility that glare and the like will not be generated in the visible light image. Therefore, the information missing in the glare area can be supplemented from the area in the visible light image corresponding to the glare area (i.e., the corresponding area). As a result, the depth of the glare area can be appropriately obtained.

[0047] For example, the processor may perform: in estimating the depth of the glare area, estimating the first depth information indicating the depth at each position in the infrared image, estimating the second depth information indicating the corrected depth at each position in the glare area by correcting the depth at each position in the glare area indicated by the first depth information based on the visible light image, and further generating the third depth information indicating the depth at each position outside the glare area of ​​the infrared image indicated by the first depth information and the depth at each position in the glare area of ​​the infrared image indicated by the second depth information. In addition, in estimating the first depth information, TOF or the like may be applied to the infrared image.

[0048] Thus, the third depth information indicates the depth obtained from the infrared image as the depth outside the glare area of ​​the infrared image, and indicates the depth obtained from the infrared image and corrected based on the visible light image as the depth of the glare area of ​​the infrared image. Therefore, even if there is a glare area in the infrared image, the overall depth of the infrared image can be properly estimated.

[0049] Alternatively, the processor may detect, during the detection of the glare area, an area in the infrared image having a brightness greater than a first threshold value as the glare area.

[0050] Since the brightness in the glare area tends to be higher than the brightness outside the glare area, the glare area can be appropriately detected by detecting the area having a brightness greater than the first threshold value in the infrared image as the glare area.

[0051] Alternatively, the processor may detect, in the detection of the glare area, an area in the infrared image having a brightness above a first threshold value and satisfying a predetermined condition as the glare area, wherein the predetermined condition is a condition in which a correlation value between an image feature quantity in the area of ​​the infrared image and an image feature quantity in the area of ​​the visible light image corresponding to the area is less than a second threshold value. For example, the image feature quantity in each area of ​​the infrared image and the visible light image may be an edge contained in the image in the area. Alternatively, the image feature quantity in each area of ​​the infrared image and the visible light image may be the brightness in the area.

[0052] The correlation between the image feature quantity in the glare area of ​​the infrared image and the image feature quantity in the area of ​​the visible light image corresponding to the glare area tends to be low. Therefore, in the infrared image, by detecting the area with high brightness and low correlation of the image feature quantity as the glare area, the glare area can be detected more appropriately.

[0053] Alternatively, the processor may, in the detection of the glare area, perform (i) CENSUS transformation on the image in the high-brightness area to obtain a first transformed image, and (ii) perform CENSUS transformation on the image in the area of ​​the visible light image corresponding to the high-brightness area to obtain a second transformed image, for at least one high-brightness area in the infrared image having a brightness above a first threshold value, and detect a high-brightness area in at least one of the high-brightness areas in which the Hamming distance between the first transformed image and the second transformed image exceeds a third threshold value as the glare area.

[0054] Thereby, the glare area can be appropriately detected.

[0055] Alternatively, the processor may estimate depth information representing the depth at each position within the infrared image in estimating the depth of the glare area, and correct the depth at each position within the glare area represented by the depth information by inputting the infrared image, the visible light image, the glare area, and the depth information into a learning model.

[0056] Thus, by making the learning model learn in advance so as to output the correct depth of each position in the glare area for the input of the infrared image, the visible light image, the glare area, and the depth information, the depth information estimated from the infrared image can be appropriately corrected. In other words, the depth of each position in the glare area represented by the depth information can be appropriately corrected.

[0057] Alternatively, other schemes of the present disclosure involve a depth acquisition device comprising a memory and a processor, wherein the processor acquires timing information indicating the timing of a light source irradiating infrared light to a subject, acquires an infrared image obtained by infrared-light-based imaging of a scene corresponding to the timing indicated by the timing information and including the subject, and the image is stored in the memory, acquires a visible light image obtained by visible-light-based imaging of a scene substantially the same as the infrared image and at substantially the same viewpoint and imaging time as the infrared image, and the image is stored in the memory, estimates depth information indicating the depth at each position in the infrared image, and corrects the depth at each position in the glare area of ​​the infrared image indicated by the depth information by inputting the infrared image, the visible light image, and the depth information into a learning model.

[0058] Therefore, if the learning model is pre-learned so that the correct depth at each position in the glare area of ​​the infrared image is outputted for the input of the infrared image, the visible light image, and the depth information, the depth information estimated from the infrared image can be appropriately corrected. In other words, the depth at each position in the glare area represented by the depth information can be appropriately corrected without detecting the glare area.

[0059] In addition, the depth acquisition device involved in other schemes of the present disclosure includes a memory and a processor, wherein the processor acquires an infrared image stored in the memory, wherein the infrared image is obtained by imaging based on infrared light, acquires a visible light image stored in the memory, wherein the visible light image is obtained by imaging based on visible light at substantially the same viewpoint and imaging time as the infrared image, detects an area reflecting glare from the infrared image as a glare area, and estimates the depth of the glare area based on the visible light image. In addition, when the visible light image and the infrared image are respectively divided into a glare area and other areas, the depth of the glare area is estimated based on the visible light image, and the depth of the other areas is estimated based on the infrared image.

[0060] Thereby, similarly to the depth acquisition device according to the above-mentioned one aspect of the present disclosure, the depth of the glare area can be appropriately acquired.

[0061] In addition, these general or specific solutions can be implemented by a system, method, integrated circuit, computer program or computer-readable CD-ROM or other recording medium, or by any combination of systems, methods, integrated circuits, computer programs or recording media. In addition, the recording medium can be a non-transitory recording medium.

[0062] The embodiments are described in detail below with reference to the drawings.

[0063] In addition, the embodiments described below all represent general or specific examples. The numerical values, shapes, materials, components, configuration positions of components, connection forms, steps, and the order of steps shown in the following embodiments are examples and do not limit the subject matter of the present disclosure. In addition, the components in the following embodiments that are not recorded in the independent claims representing the highest concept are described as arbitrary components.

[0064] In addition, each figure is a schematic diagram and is not necessarily a strict illustration. In addition, in each figure, the same reference numeral is attached to the same component.

[0065] (Implementation Method)

[0066] [Hardware structure]

[0067] Figure 1 This is a block diagram showing the hardware structure of the depth acquisition device 1 involved in the embodiment. The depth acquisition device 1 in this embodiment has a hardware structure that can acquire an image based on infrared light (or near-infrared light) and an image based on visible light by shooting substantially the same scene and substantially the same viewpoint and shooting time. In addition, the so-called substantially the same means the same degree as that of being able to achieve the effect in the present disclosure.

[0068] like Figure 1 As shown, the depth acquisition device 1 includes a light source 10 , a solid-state imaging element 20 , a processing circuit 30 , a diffusion plate 50 , a lens 60 , and a bandpass filter 70 .

[0069] The light source 10 emits illumination light. More specifically, the light source 10 emits illumination light for illuminating a subject at a timing indicated by a light emission signal generated by the processing circuit 30 .

[0070] The light source 10 is composed of, for example, a capacitor, a driving circuit, and a light-emitting element, and emits light by driving the light-emitting element with electric energy stored in the capacitor. The light-emitting element is realized by, for example, a laser diode, a light-emitting diode, etc. In addition, the light source 10 may be a structure including one type of light-emitting element, or may be a structure including multiple types of light-emitting elements according to the purpose.

[0071] In the following, the light emitting element is, for example, a laser diode that emits near-infrared light, or a light emitting diode that emits near-infrared light. However, the irradiation light irradiated by the light source 10 is not necessarily limited to near-infrared light. The irradiation light irradiated by the light source 10 may be, for example, infrared light (also referred to as infrared light) in a frequency band other than near-infrared light. In the following, in this embodiment, the irradiation light irradiated by the light source 10 is described as infrared light, but the infrared light may be near-infrared light or infrared light in a frequency band other than near-infrared light.

[0072] The solid-state imaging element 20 captures an image of a subject and outputs an imaging signal indicating an exposure amount. More specifically, the solid-state imaging element 20 performs exposure at a timing indicated by an exposure signal generated by the processing circuit 30 and outputs an imaging signal indicating an exposure amount.

[0073] The solid-state imaging device 20 has a pixel array in which first pixels for imaging using reflected light of an irradiated light reflected from an object and second pixels for imaging the object are arranged in an array. The solid-state imaging device 20 may have logic functions such as a cover glass and an AD converter as necessary.

[0074] In the following description, the reflected light is assumed to be infrared light, similarly to the irradiated light. However, the reflected light is not limited to infrared light as long as it is light obtained by reflecting the irradiated light from the subject.

[0075] Figure 2 Schematic diagram showing the pixel array 2 included in the solid-state imaging element 20 .

[0076] like Figure 2 As shown, the pixel array 2 is configured in an array so that first pixels 21 (IR pixels) for capturing images using reflected light obtained by irradiating light reflected from a subject and second pixels 22 (BW pixels) for capturing images of the subject are alternately arranged in columns.

[0077] In addition, Figure 2 In the pixel array 2, the second pixel 22 and the first pixel 21 are arranged adjacent to each other in the row direction and arranged in a strip shape in the row direction, but this is not limited to this, and they may be arranged every multiple rows (for example, every 2 rows). That is, the second pixel 22 is arranged in the first row adjacent to each other in the row direction and the first pixel 21 is arranged in the second row adjacent to each other in the row direction, and they may be arranged alternately every M rows (M is a natural number). Furthermore, the second pixel 22 is arranged in the first row adjacent to each other in the row direction and the first pixel 21 is arranged in the second row adjacent to each other in the row direction, and they may be arranged separated by different rows (alternatingly repeating the first row as N rows and the second row as L rows (N and L are different natural numbers)).

[0078] The first pixel 21 is realized by, for example, an infrared light pixel having sensitivity to infrared light as reflected light, and the second pixel 22 is realized by, for example, a visible light pixel having sensitivity to visible light.

[0079] The infrared light pixel is composed of, for example, an optical filter (also referred to as an IR filter) that transmits only infrared light, a microlens, a light-receiving element as a photoelectric conversion unit, and a storage unit that stores the charge generated by the light-receiving element. Therefore, an image representing the brightness of infrared light is expressed by the imaging signal output from the plurality of infrared light pixels (i.e., the first pixel 21) included in the pixel array 2. Hereinafter, the infrared light image is also referred to as an IR image or an infrared image.

[0080] In addition, the visible light pixel is composed of, for example, an optical filter (also called a BW filter) that transmits only visible light, a microlens, a light receiving element as a photoelectric conversion unit, and a storage unit that stores the charge converted by the light receiving element. Therefore, the visible light pixel, that is, the second pixel 22, outputs an imaging signal representing brightness and color difference. That is, a color image representing the brightness and color difference of visible light is expressed by the imaging signal output from the plurality of second pixels 22 included in the pixel array 2. In addition, the optical filter of the visible light pixel can transmit both visible light and infrared light, or can transmit only light in a specific wavelength band such as red (R), green (G) or blue (B) among the visible light.

[0081] In addition, the visible light pixel can detect only the brightness of the visible light. In this case, the visible light pixel, i.e., the second pixel 22, outputs an imaging signal indicating the brightness. Therefore, the imaging signal outputted from the plurality of second pixels 22 contained in the pixel array 2 represents a black and white image indicating the brightness of the visible light, in other words, a monochrome image. Hereinafter, the monochrome image is also referred to as a BW image. In addition, the above-mentioned color image and BW image are also collectively referred to as a visible light image.

[0082] Back again Figure 1 , continue with the description of the acquisition device 1 in depth.

[0083] The processing circuit 30 calculates subject information related to the subject using the imaging signal outputted from the solid-state imaging element 20 .

[0084] The processing circuit 30 includes, for example, a microcomputer or other computing device. The microcomputer includes a processor (microprocessor), a memory, etc., and generates a light emission signal and an exposure signal by executing a driver stored in the memory by the processor. In addition, the processing circuit 30 may use an FPGA or an ISP, etc., and may include one hardware or multiple hardware.

[0085] The processing circuit 30 calculates the distance to the subject by, for example, a TOF distance measurement method using the imaging signal from the first pixel 21 of the solid-state imaging element 20 .

[0086] The calculation of the distance to the subject by the TOF distance measurement method performed by the processing circuit 30 will be described below with reference to the drawings.

[0087] Figure 3 This is a timing chart showing the relationship between the light emission timing of the light emitting element of the light source 10 and the exposure timing of the first pixel 21 of the solid-state imaging element 20 when the processing circuit 30 calculates the distance to the subject using the TOF ranging method.

[0088] exist Figure 3 In the figure, Tp is the light emission period during which the light emitting element of the light source 10 emits the irradiation light, and Td is the delay time from when the light emitting element of the light source 10 emits the irradiation light to when the reflected light of the irradiation light reflected by the subject returns to the solid-state imaging element 20. Furthermore, the first exposure period becomes the same timing as the light emission period during which the light source 10 emits the irradiation light, and the second exposure period becomes the timing from the end time point of the first exposure period to the passage of the light emission period Tp.

[0089] exist Figure 3 In the figure, q1 represents the total amount of exposure in the first pixel 21 of the solid-state imaging element 20 caused by the reflected light during the first exposure period, and q2 represents the total amount of exposure in the first pixel 21 of the solid-state imaging element 20 caused by the reflected light during the second exposure period.

[0090] By Figure 3 The light emitting element of the light source 10 emits the irradiation light and the first pixel 21 of the solid-state imaging element 20 exposes the light at the timing shown. Assuming the speed of light is c, the distance d to the subject can be expressed by the following (Equation 1).

[0091] d=c×Tp / 2×q2 / (q1+q2)…(Formula 1)

[0092] Therefore, the processing circuit 30 can calculate the distance to the subject using the imaging signal from the first pixel 21 of the solid-state imaging element 20 by using (Formula 1).

[0093] In addition, the plurality of first pixels 21 of the solid-state imaging element 20 can be exposed only in the third exposure period Tp after the first exposure period and the second exposure period have ended. The plurality of first pixels 21 can detect noise other than the reflected light based on the exposure amount obtained in the third exposure period Tp. That is, the processing circuit 30 can more accurately calculate the distance d to the subject by removing the noise from the exposure amount q1 in the first exposure period and the exposure amount q2 in the second exposure period in the above (Formula 1).

[0094] Back again Figure 1 , continue with the description of the acquisition device 1 in depth.

[0095] The processing circuit 30 can detect the subject and calculate the distance to the subject using the imaging signal from the second pixel 22 of the solid-state imaging element 20 , for example.

[0096] That is, the processing circuit 30 can detect the subject and calculate the distance to the subject based on the visible light image captured by the multiple second pixels 22 of the solid-state imaging element 20. Here, the detection of the subject can be achieved, for example, by using edge detection of the singularity of the subject to distinguish the shape by pattern recognition, or it can be achieved by using a learning model that has been learned in advance through deep learning and other processes. In addition, the calculation of the distance to the subject can be performed using world coordinate transformation. Of course, it is also possible to detect the subject by multimodal learning processing using not only the visible light image but also the brightness and distance information of the infrared light captured by the first pixel 21.

[0097] The processing circuit 30 generates a light emission signal indicating a timing for light emission and an exposure signal indicating a timing for exposure. The processing circuit 30 then outputs the generated light emission signal to the light source 10 and outputs the generated exposure signal to the solid-state imaging element 20 .

[0098] The processing circuit 30 can, for example, generate and output a light emission signal to cause the light source 10 to emit light at a given cycle, and generate and output an exposure signal to cause the solid-state imaging element 20 to be exposed at a given cycle, thereby enabling the depth acquisition device 1 to achieve continuous imaging at a given frame rate. In addition, the processing circuit 30 includes, for example, a processor (microprocessor), a memory, etc., and generates a light emission signal and an exposure signal by executing a driver program stored in the memory by the processor.

[0099] The diffuser 50 adjusts the intensity distribution and angle of the irradiated light. In addition, in the adjustment of the intensity distribution, the diffuser 50 makes the intensity distribution of the irradiated light from the light source 10 uniform. Figure 1 In the illustrated example, the depth acquisition device 1 includes the diffusion plate 50 , but may not include the diffusion plate 50 .

[0100] The lens 60 is an optical lens that focuses light entering from the outside of the depth acquisition device 1 on the surface of the pixel array 2 of the solid-state imaging element 20 .

[0101] The bandpass filter 70 is an optical filter that transmits infrared light as reflected light and visible light. Figure 1 In the illustrated example, the depth acquisition device 1 includes the bandpass filter 70 , but may not include the bandpass filter 70 .

[0102] The depth acquisition device 1 of the above structure is mounted on a transport device and used. For example, the depth acquisition device 1 is mounted on a vehicle traveling on a road and used. In addition, the transport device on which the depth acquisition device 1 is mounted is not necessarily limited to a vehicle. The depth acquisition device 1 can also be mounted on a transport device other than a vehicle such as a motorcycle, a ship, or an airplane and used.

[0103] [Overview of the depth acquisition device]

[0104] The depth acquisition device 1 in this embodiment is used Figure 1 The hardware structure shown in the figure acquires an IR image and a BW image by photographing substantially the same scene, substantially the same viewpoint and at the same time. Then, the depth acquisition device 1 uses the BW image to correct the depth at each position in the IR image obtained from the IR image. Specifically, when there is a glare area described later in the IR image, the depth acquisition device 1 uses an image in the area of ​​the BW image corresponding to the glare area to correct the depth at each position in the glare area obtained from the IR image.

[0105] Figure 4 This is a block diagram showing an example of the functional structure of the depth acquisition device 1 .

[0106] The depth acquisition device 1 includes a light source 101 , an IR camera 102 , a BW camera 103 , a depth estimation unit 111 , and a glare detection unit 112 .

[0107] The light source 101 may include Figure 1 The light source 10 and the diffuser plate 50 are shown.

[0108] IR camera 102 may include Figure 1 The solid-state imaging element 20 shown has a plurality of first pixels 21, a lens 60, and a bandpass filter 70. Such an IR camera 102 acquires an IR image by capturing a scene including the subject with infrared light in accordance with the timing at which the light source 101 irradiates the subject with infrared light.

[0109] The BW camera 103 may include Figure 1 The solid-state imaging element 20 shown has a plurality of second pixels 22, a lens 60, and a bandpass filter 70. Such a BW camera 103 acquires a visible light image (specifically, a BW image) by performing visible light imaging of a scene substantially the same as the infrared image and imaging at substantially the same viewpoint and imaging time as the infrared image.

[0110] The depth estimation unit 111 and the glare detection unit 112 can be used as Figure 1The functions of the processing circuit 30 shown, specifically the functions of the processor 110, are implemented.

[0111] The glare detection unit 112 detects a glare area from the IR image based on the IR image obtained by the IR camera 102 and the BW image obtained by the BW camera 103 .

[0112] The glare area in this embodiment is an area where glare, ghosting, brightness saturation or bright lines occur. Glare is the over-sensitivity of light caused by harmful light reflected on the lens surface or lens barrel when the lens of the IR camera 102 is pointed towards a strong light source. In addition, in glare, the image becomes whitish and loses sharpness. Ghosting is a type of glare, and is a phenomenon in which light reflected repeatedly and complexly on the lens surface is clearly captured as an image. Bright lines are a phenomenon in which a straight white portion is generated when a camera is used to shoot a subject that has a brightness difference of a given value or more compared to the surroundings.

[0113] In addition, in the present disclosure, a phenomenon including at least one of glare, ghosting, saturation of brightness, and bright lines is referred to as glare or the like.

[0114] The depth estimation unit 111 estimates the depth at each position in the R image including the glare area detected by the glare detection unit 112. Specifically, the depth estimation unit 111 obtains an IR image obtained by the IR camera 102 in accordance with the timing of the light source 101 irradiating the subject with infrared light, and estimates the depth at each position in the IR image based on the IR image. Furthermore, the depth estimation unit 111 corrects the depth at each position estimated in the glare area detected by the glare detection unit 112 based on the BW image. That is, the depth estimation unit 111 estimates the depth of the glare area based on the IR image, the BW image, and the glare area.

[0115] Figure 5 It is a block diagram showing another example of the functional structure of the depth acquisition device 1 .

[0116] The depth acquiring device 1 may include a memory 200 and a processor 110 .

[0117] In addition, the processor 110 may include not only the depth estimation unit 111 and the glare detection unit 112, but also Figure 5 As shown in the figure, the light emission timing acquisition unit 113, the IR image acquisition unit 114, and the BW image acquisition unit 115 are provided. In addition, these components are realized as functions of the processor 110.

[0118] The light emission timing acquisition unit 113 acquires timing information indicating the timing at which the light source 101 radiates infrared light to the subject. Figure 1The light emission signal shown is output to the light source 101, and information indicating the timing of the output is obtained as the above-mentioned timing information.

[0119] The IR image acquisition unit 114 acquires an IR image obtained by capturing an image of a scene including a subject using infrared light at a timing indicated by the timing information, and stores the IR image in the memory 200 .

[0120] The BW image acquisition unit 115 acquires a BW image obtained by imaging substantially the same scene as the IR image using visible light and at substantially the same viewpoint and imaging time as the IR image, and stores the BW image in the memory 200 .

[0121] As described above, the glare detection unit 112 detects the glare area from the IR image, and the depth estimation unit 111 estimates the depth based on the IR image, the BW image, and the glare area.

[0122] Furthermore, the depth acquisition device 1 in the present embodiment may include the processor 110 and the memory 200 instead of the light source 101 , the IR camera 102 , and the BW camera 103 .

[0123] Figure 6 It is a flowchart showing the overall processing operation of the depth acquisition device 1 .

[0124] (Step S11)

[0125] First, the light source 101 emits light to irradiate the subject with infrared light.

[0126] (Step S12)

[0127] Next, the IR camera 102 acquires an IR image. That is, the IR camera 102 captures a scene including a subject irradiated with infrared light by the light source 101. Thus, the IR camera 102 acquires an IR image based on the infrared light reflected from the subject. Specifically, the IR camera 102 acquires the infrared image through Figure 3 IR images obtained by changing the timing and exposure amount of each of the first exposure period, the second exposure period, and the third exposure period shown.

[0128] (Step S13)

[0129] Next, the BW camera 103 acquires a BW image. That is, the BW camera 103 acquires a BW image corresponding to the IR image acquired in step S12, that is, a BW image of the same scene and the same viewpoint as the IR image.

[0130] (Step S14)

[0131] Then, the glare detection unit 112 detects a glare area from the IR image acquired in step S12 .

[0132] (Step S15)

[0133] Next, the depth estimation unit 111 estimates the depth of the glare region based on the IR image acquired in step S12 , the BW image acquired in step S13 , and the glare region detected in step S14 .

[0134] Figure 7 It is a flowchart showing the overall processing operation of the processor 110 of the depth acquisition device 1 .

[0135] (Step S21)

[0136] First, the light emission timing acquisition unit 113 of the processor 110 acquires timing information indicating the timing at which the light source 101 radiates infrared light onto the subject.

[0137] (Step S22)

[0138] Next, the IR image acquisition unit 114 acquires an IR image from the IR camera 102 that has captured an image at the timing indicated by the timing information acquired in step S21. Figure 1 The exposure signal is output to the IR camera 102 at the timing of the light emission signal shown. As a result, the IR image acquisition unit 114 starts the IR camera 102 to start imaging, and acquires the IR image obtained by the imaging from the IR camera 102. At this time, the IR image acquisition unit 114 may acquire the IR image from the IR camera 102 via the memory 200, or may acquire it directly from the IR camera 102.

[0139] (Step S23)

[0140] Next, the BW image acquisition unit 115 acquires a BW image corresponding to the IR image acquired in step S22 from the BW camera 103. At this time, the BW image acquisition unit 115 may acquire the BW image from the BW camera 103 via the memory 200 or directly from the BW camera 103.

[0141] (Step S24)

[0142] Then, the glare detection unit 112 detects a glare area from the IR image.

[0143] (Step S25)

[0144] Next, the depth estimation unit 111 estimates the depth of the glare area based on the IR image obtained in step S22, the BW image obtained in step S23, and the glare area detected in step S24. Thus, at least the depth information indicating the depth of the glare area is calculated. In addition, at this time, the depth estimation unit 111 may estimate not only the glare area but also the depth of the entire IR image and calculate the depth information indicating the estimation result.

[0145] Specifically, the depth estimation unit 111 in this embodiment estimates the depth at each position in the IR image from the IR image obtained in step S22. Then, the depth estimation unit 111 uses the BW image to correct the depth at each position in the glare area. In addition, each position may be the position of each of a plurality of pixels or the position of a block including a plurality of pixels.

[0146] In the depth acquisition device 1 of the present embodiment, a glare area is detected from an IR image, and in the glare area, the depth is estimated not only based on the IR image but also based on the BW image, so that the depth of the glare area can be properly acquired. That is, in the IR image and the BW image, the scene that becomes the object of photography is substantially the same, and the viewpoint and the photography time are also substantially the same. Therefore, the IR image and the BW image are highly correlated. In addition, glare and the like are phenomena that depend on wavelength, and even if glare and the like are generated in the IR image, there is a high possibility that glare and the like will not be generated in the BW image. Therefore, the missing information in the glare area can be supplemented from the area in the BW image corresponding to the glare area (i.e., the corresponding area). As a result, the depth of the glare area can be properly acquired.

[0147] [Specific functional structure of the depth acquisition device]

[0148] Figure 8 It is a block diagram showing a specific functional structure of the processor 110 of the depth acquisition device 1 .

[0149] The processor 110 includes a first depth estimation unit 111a, a second depth estimation unit 111b, a glare detection unit 112, a high brightness region detection unit 116, a first edge detection unit 117IR, a second edge detection unit 117BW, and an output unit 118. The first depth estimation unit 111a and the second depth estimation unit 111b correspond to Figure 5 The depth estimation unit 111 is shown. In addition, the processor 110 may include the light emission timing acquisition unit 113, the IR image acquisition unit 114, and the BW image acquisition unit 115 described above.

[0150] The high-brightness region detection unit 116 detects a region having a brightness equal to or higher than a first threshold value in the IR image as a high-brightness region. The first edge detection unit 117IR detects an edge in the IR image. The second edge detection unit 117BW detects an edge in the BW image.

[0151] The glare detection unit 112 compares the edge detected for at least one high brightness area in the IR image with the edge detected for the area in the BW image corresponding to the high brightness area. Through this comparison, the glare detection unit 112 determines whether the high brightness area is a glare area. That is, the glare area is detected through this determination. In other words, the glare detection unit 112 performs area segmentation of the IR image by distinguishing the captured IR image into a glare area and a non-glare area that is not a glare area.

[0152] Here, glare and the like are phenomena that depend on the wavelength of light. For this reason, glare and the like generated in IR images are mostly not generated in BW images. Generally, it is known that IR images and BW images have a strong correlation. However, in glare and the like generated in IR images, since the edges of the IR images collapse, the correlation value between the edge of the area where the glare and the like is generated and the edge of the area of ​​the BW image corresponding to the area becomes low. In addition, when glare and the like are generated, the brightness of the area where they are generated becomes larger. For this reason, the glare detection unit 112 in the present embodiment uses this relationship to discriminate the glare area from the captured IR image.

[0153] That is, the glare detection unit 112 in the present embodiment detects a region in the IR image that has a brightness greater than the first threshold and satisfies a predetermined condition as a glare region. The predetermined condition is that the correlation value between the image feature quantity in the region of the IR image and the image feature quantity in the region of the BW image corresponding to the region is less than the second threshold. Here, the image feature quantity in each region of the IR image and the BW image is the edge contained in the image in the region. In addition, the region of the BW image corresponding to the region of the IR image is a region that is located at the same spatial position as the region of the IR image and has the same shape and size as the region of the IR image.

[0154] As described above, the correlation between the image feature quantity in the glare area of ​​the IR image and the image feature quantity in the area of ​​the BW image corresponding to the glare area tends to be low. Therefore, in this embodiment, by detecting the area with high brightness and low correlation of the image feature quantity as the glare area in the IR image, the glare area can be detected more appropriately.

[0155] The first depth estimating unit 111 a and the second depth estimating unit 111 b have a function as the depth estimating unit 111 described above.

[0156] The first depth estimation unit 111a estimates the depth of each position in the IR image based on the IR image obtained in accordance with the timing of the irradiation of the infrared light from the light source 101. The first depth estimation unit 111a outputs information indicating the estimated depth of each position in the IR image as the first depth information. That is, the first depth estimation unit 111a estimates the first depth information indicating the depth of each position in the IR image.

[0157] The second depth estimation unit 111b corrects the first depth information based on the glare area in the BW image and the IR image. Thus, the depth of the glare area is corrected among the depths at each position in the IR image represented by the first depth information. The second depth estimation unit 111b outputs information representing the corrected depth at each position in the glare area as the second depth information. That is, the second depth estimation unit 111b estimates the second depth information representing the corrected depth at each position in the glare area by correcting the depth at each position in the glare area represented by the first depth information based on the BW image.

[0158] The output unit 118 replaces the depth at each position in the glare area indicated by the first depth information with the corrected depth at each position in the glare area indicated by the second depth information. Thus, the third depth information including the depth at each position outside the glare area of ​​the IR image indicated by the first depth information and the corrected depth at each position in the glare area of ​​the IR image indicated by the second depth information is generated. The output unit 118 outputs the third depth information.

[0159] Thus, the third depth information indicates the depth obtained from the IR image as the depth outside the glare area of ​​the IR image, and indicates the depth obtained from the IR image and corrected based on the BW image as the depth of the glare area of ​​the IR image. Therefore, in the present embodiment, even when there is a glare area in the IR image, the overall depth of the IR image can be appropriately estimated.

[0160] Fig. 9A An example of an IR image is shown. Fig. 9B An example of a BW image is shown.

[0161] like Fig. 9B As shown in FIG. 1 , a scene in which a sign is arranged on a road is shown in the BW image. The sign is made of a material that easily reflects infrared light. Therefore, if the IR camera 102 is directed to the sign from the same viewpoint as the BW camera 103, Fig. 9B The same scene as shown in the picture is taken. Fig. 9A IR image shown.

[0162] The IR image obtained as described above is Fig. 9AAs shown, a high-brightness glare is generated in the area including the range of the signboard in the BW image. This is because, due to the regular reflection of the infrared light from the light source 101 on the road signboard, the strong-intensity infrared light enters the IR camera 102 as reflected light. In addition, materials that easily reflect infrared light are often used in clothes worn by workers on construction sites or multiple poles erected along the road. Therefore, when photographing a scene including a subject using such materials, there is a high possibility that glare will be generated in the IR image. However, the possibility of glare in the BW image is low. As a result, the correlation between the image feature quantity of the glare area of ​​the IR image and the image feature quantity of the area of ​​the BW image corresponding to the glare area becomes low. On the other hand, the correlation between the image feature quantity of the area outside the glare area of ​​the IR image (i.e., the non-glare area) and the image feature quantity of the area of ​​the BW image corresponding to the non-glare area becomes high.

[0163] Fig.10 An example of a binarized image obtained by binarizing an IR image is shown.

[0164] The high brightness area detection unit 116 Fig. 9A In the IR image shown in FIG. 1 , the area with brightness above the first threshold is detected as a high brightness area. That is, the high brightness area detection unit 116 binarizes the brightness at each position (i.e., each pixel) in the IR image. As a result, for example, Fig.10 As shown in the figure, a white area and a black area are generated ( Fig.10 The binary image of the shaded area in the figure.

[0165] Fig.11 An example of a high-brightness area in an IR image is shown.

[0166] The high brightness region detection unit 116 detects the white region in the binary image as a high brightness region. Fig.11 As shown, when the binary image includes six white areas, the high brightness area detection unit 116 detects the six white areas as high brightness areas A to F. That is, the IR image or the binary image area is divided into the six high brightness areas A to F and the non-high brightness area which is not a high brightness area.

[0167] Fig.12 Indicates the area of ​​the BW image corresponding to the high brightness area of ​​the IR image.

[0168] The glare detection unit 112 determines the image feature quantity of the area in the BW image corresponding to the high brightness area for at least one high brightness area in the binary image (i.e., IR image). In addition, the image feature quantity is, for example, an edge. In addition, the area in the BW image corresponding to the high brightness area is an area located at the same spatial position as the high brightness area in the binary image or IR image and has the same shape and size as the high brightness area. Hereinafter, such an area in the BW area corresponding to the area of ​​the IR image is also referred to as a corresponding area.

[0169] For example, in Fig.11 When the high-luminance regions A to F are detected as shown, the glare detection unit 112 specifies the image feature quantities of the regions corresponding to the high-luminance regions A to F, respectively, in the BW image.

[0170] Fig.13 An example of a glare area detected from an IR image is shown.

[0171] The glare detection unit 112 determines whether the high brightness area A to F is a glare area. That is, the glare detection unit 112 determines whether the high brightness area is a glare area by comparing the image feature of the high brightness area A in the IR image with the image feature of the corresponding area in the BW image corresponding to the high brightness area A. The result is, for example, Fig.13 As shown, the glare detection unit 112 determines that the high-luminance regions A, C, D, and E among the high-luminance regions A to F are glare regions.

[0172] Fig.14 The simulation results of the depth acquisition device 1 are shown.

[0173] The depth acquisition device 1 acquires the depth through the image captured by the BW camera 103. Fig.14 The BW image shown in (a) is obtained by the IR camera 102. Fig.14 The IR image shown in (b) of FIG. The BW image and the IR image are images obtained by photographing the same scene at the same viewpoint and at the same photographing time. Fig.14 In the example shown in (b), a large glare area is generated at the right end of the IR image.

[0174] The first depth estimation unit 111a estimates the depth from the IR image and generates Fig.14 The first depth information is shown in (c). The first depth information is expressed as a first depth image in which the depth at each position in the IR image is represented by brightness. In the first depth image, the depth of the glare area is not properly expressed.

[0175] The second depth estimation unit 111b corrects the inappropriate depth in the glare area. Fig.14 As shown in (e) of FIG. 1 , the third depth information indicating the corrected depth of the glare area and the depth of the non-glare area is generated. The third depth information is also expressed as a third depth image in which the depth is expressed by brightness, similarly to the first depth information. In addition, the second depth estimation unit 111b may also correct the depth of the non-glare area in the first depth image based on the image feature amount of the corresponding area of ​​the BW image.

[0176] In this way, in the depth acquisition device 1 of the present embodiment, the third depth image can be made close to Fig.14 The depth image of the correct solution shown in (d).

[0177] [Specific processing flow of the depth acquisition device]

[0178] Fig.15 Yes means Figure 8 The flowchart of the overall processing operation of the depth acquisition device 1 is shown.

[0179] (Step S31)

[0180] First, the high-luminance region detection unit 116 detects a high-luminance region from the IR image.

[0181] (Step S32)

[0182] The first edge detection unit 117IR detects an edge located in the IR image.

[0183] (Step S33)

[0184] The second edge detection unit 117BW detects an edge located in the BW image.

[0185] (Step S34)

[0186] The glare detection unit 112 detects the glare area in the IR image by comparing the edge in the high brightness area with the edge in the corresponding area of ​​the BW image for at least one high brightness area of ​​the IR image. That is, when the correlation value between the edge in the high brightness area and the edge in the corresponding area of ​​the BW image is less than the second threshold value, the glare detection unit 112 detects the high brightness area as the glare area. Thus, the IR image is divided into at least one glare area and a non-glare area.

[0187] (Step S35)

[0188] The first depth estimating unit 111 a generates first depth information from an IR image using, for example, TOF.

[0189] (Step S36)

[0190] The second depth estimating unit 111 b generates second depth information indicating the depth of the glare region based on the first depth information of the IR image and the BW image.

[0191] (Step S37)

[0192] The output unit 118 generates third depth information by replacing the depth of the glare area indicated by the first depth information with the depth indicated by the second depth information.

[0193] Fig.16 Yes means Fig.15 Flowchart of detailed processing of steps S31 to S34.

[0194] (Step S41)

[0195] First, the high-brightness area detection unit 116 determines whether the brightness at each position in the IR image is above the first threshold value. Here, for example, if the IR image is a 12-bit grayscale image, the first threshold value is about 1500. Of course, the first threshold value may be a value that changes according to environmental conditions or the settings of the IR camera 102. For example, when shooting a dark scene such as at night, since the overall brightness of the IR image becomes lower, the first threshold value may be a smaller value than when shooting a bright scene during the day. In addition, when the exposure time of the IR camera 102 is long, since the overall brightness of the IR image becomes higher, the first threshold value may be a larger value than when the exposure time is short.

[0196] (Step S42)

[0197] If it is determined that the brightness at any position is not equal to or greater than the first threshold (step S41: No), the high brightness region detection unit 116 determines that no glare occurs in the IR image (step S42). That is, the entire IR image is determined to be a non-glare region.

[0198] (Step S43)

[0199] On the other hand, if it is determined that the brightness at any position is greater than the first threshold value (step S41 "Yes"), the high brightness region detection unit 116 performs region segmentation on the IR image. That is, the high brightness region detection unit 116 segments the IR image into at least one high brightness region and a region other than the high brightness region. For example, a brightness-based technique such as SuperPixel can be used in the region segmentation.

[0200] (Step S44)

[0201] Next, the first edge detection unit 117IR and the second edge detection unit 117BW perform edge detection on the IR image and the BW image, respectively. The Canny method or the Sobel filter can be used for edge detection.

[0202] (Step S45)

[0203] The glare detection unit 112 compares the edge in at least one high brightness area of ​​the IR image with the edge in the area of ​​the BW image corresponding to the high brightness area. That is, the glare detection unit 112 determines whether the correlation value between the edge in the high brightness area of ​​the IR image and the edge in the corresponding area of ​​the BW image is greater than the second threshold value. The values ​​output by edge detection of the IR image and the BW image are arranged in a vector form for each area, and the inner product values ​​are normalized to obtain the correlation value. That is, the glare detection unit 112 normalizes the inner product value of the vector containing multiple values ​​obtained by edge detection in the high brightness area of ​​the IR image and the vector containing multiple values ​​obtained by edge detection in the area of ​​the BW image corresponding to the high brightness area. The correlation value for the high brightness area is calculated in this way.

[0204] (Step S46)

[0205] Here, if it is determined that the correlation value is not greater than the second threshold value, that is, less than the second threshold value (step S45 "No"), the glare detection unit 112 determines the high brightness area as a glare area. That is, due to the influence of glare, the edge correlation between the area where glare, etc. in the IR image is generated and the corresponding area of ​​the BW image disappears, so the glare detection unit 112 determines the high brightness area in the IR image as a glare area.

[0206] On the other hand, if the glare detection unit 112 determines in step S45 that the correlation value is greater than or equal to the second threshold value, that is, not less than the second threshold value (step S45 "Yes"), it is determined that no glare occurs in the high-brightness area. In other words, the glare detection unit 112 determines the high-brightness area as a non-glare area.

[0207] In such a method, an IR image and a BW image with substantially the same viewpoint position are required. In the depth acquisition device 1 of the present embodiment, for each pixel, the filter used in the pixel is set to be either an IR filter or a BW filter. Figure 2 As shown, the first pixels 21 having IR filters and the second pixels 22 having BW filters are arranged alternately in the column direction. This allows the IR image and the BW image to be acquired from substantially the same viewpoint, and thus the glare area can be appropriately determined.

[0208] <Related information on using brightness>

[0209] In the above description, the edge is used in the discrimination of the glare area and the non-glare area, but the edge may not be used in the discrimination. For example, the correlation value of the brightness of the IR image and the BW image may be used. As mentioned above, when no glare or the like is generated, the IR image and the BW image have a strong correlation, but in the area where glare or the like is generated, the correlation becomes weak. Therefore, by using the correlation of the brightness of the IR image and the BW image, the glare area can be properly discriminated.

[0210] Fig.17 It means to replace Fig.15 This is a flowchart of an example of the processing of steps S31 to S34. That is, Fig.17 FIG. 1 is a flowchart showing a glare area detection process using the correlation between the brightness of the IR image and the BW image. Fig.17 In Fig.16 The same steps are denoted by the same reference numerals and detailed descriptions are omitted. Fig.17 The flowchart shown is similar to Fig.16 The flowchart shown is different in that step S44 is not included and step S45a is included instead of step S45.

[0211] (Step S45a)

[0212] In step S45a, the glare detection unit 112 calculates the correlation value between the brightness of each pixel in each high brightness area obtained by the area segmentation in step S43 and the brightness of each pixel in the area of ​​the BW image corresponding to the high brightness area. The brightness of each pixel of the IR image and the BW image is arranged in a vector shape for each area, and the inner product value is normalized by the number of pixels to obtain the correlation value. That is, the glare detection unit 112 normalizes the inner product value of the vector containing the brightness of each pixel in the high brightness area of ​​the IR image and the vector containing the brightness of each pixel in the corresponding area of ​​the BW image. In this way, the correlation value for the high brightness area is calculated.

[0213] Here, when the correlation value is greater than or equal to the second threshold value, that is, not less than the second threshold value (step S45a "Yes"), the glare detection unit 112 determines that no glare or the like is generated in the high-brightness area (step S42). On the other hand, when the correlation value is less than the second threshold value (step S45a "No"), the correlation between the brightness of each pixel in the high-brightness area of ​​the IR image and the brightness of each pixel in the corresponding area of ​​the BW image becomes low due to the influence of glare or the like. Therefore, in such a case, the glare detection unit 112 determines the high-brightness area as a glare area (step S46).

[0214] That is, the image feature quantities in the respective areas of the IR image and the BW image used for the detection of the glare area are Fig.16 The example shown is an edge contained in the image within this region, but Fig.17 In the example shown, it is the brightness in the area. Here, as described above, the correlation between the brightness in the glare area of ​​the IR image and the brightness in the area of ​​the BW image corresponding to the glare area tends to be low. Therefore, in the IR image, by detecting an area with high brightness and low brightness correlation as a glare area, the glare area can be detected more appropriately.

[0215] <Conversion using CENSUS>

[0216] Of course, the evaluation value used to distinguish between the glare area and the non-glare area does not need to be a correlation value. For example, the Hamming distance and the CENSUS transform can be used. Regarding the CENSUS transform, for example, it is disclosed in non-patent literature (R. Zabih and J. Woodfill, "Non-parametric Local Transforms for Computing Visual Correspondence", Proc. of ECCV, pp. 151-158, 1994). The CENSUS transform sets a window in the image and transforms the size relationship between the central pixel and the surrounding pixels of the window into a binary vector.

[0217] Fig.18 It means to replace Fig.15 This is a flowchart of another example of the processing of steps S31 to S34. That is, Fig.18 FIG. 1 is a flowchart showing the detection process of the glare area using CENSUS transformation for each of the IR image and the BW image. Fig.18 In, with Fig.16 The same steps are given the same reference numerals and detailed descriptions are omitted. Fig.18 The flowchart shown is similar to Fig.16 The flowchart shown is different in that steps S44b and S45b are included instead of steps S44 and S45.

[0218] (Step S44b)

[0219] In step S44b, the glare detection unit 112 performs CENSUS transformation on the image of the high brightness area in the IR image and the image of the corresponding area in the BW image for each high brightness area obtained by the area segmentation in step S43, thereby generating a CENSUS transformed image of the image of the high brightness area in the IR image and a CENSUS transformed image of the image of the corresponding area in the BW image.

[0220] (Step S45b)

[0221] Next, the glare detection unit 112 calculates the Hamming distance between the CENSUS transformed image of the IR image obtained in step S44b and the CENSUS transformed image of the BW image in step S45b. Then, when the value of the Hamming distance normalized by the number of pixels in the high-brightness area is less than the third threshold value (step S45b "Yes"), the glare detection unit 112 determines that glare, etc., does not occur in the high-brightness area (step S42). On the other hand, when the value of the normalized Hamming distance is greater than the third threshold value (step S45b "No"), the glare detection unit 112 determines that the texture disappears in the image of the high-brightness area due to the influence of glare, etc. As a result, the glare detection unit 112 determines that the high-brightness area is a glare area (step S46).

[0222] That is, the glare detection unit 112 obtains a first transformed image by performing CENSUS transformation on the image in the high-brightness region of at least one high-brightness region having a brightness greater than the first threshold value in the IR image. Then, the glare detection unit 112 obtains a second transformed image by performing CENSUS transformation on the image in the region of the BW image corresponding to the high-brightness region. In addition, the first transformed image and the second transformed image are the above-mentioned CENSUS transformed images. Next, the glare detection unit 112 detects a high-brightness region in at least one high-brightness region where the Hamming distance between the first transformed image and the second transformed image exceeds the third threshold value as a glare region. By using CENSUS transformation in this way, the glare region can also be appropriately detected.

[0223] <Brightness of IR image>

[0224] exist Fig.17 In the example shown, the brightness of the IR image and the correlation value between the brightness of the IR image and the brightness of the BW image are used to distinguish between the glare area and the non-glare area, but only the brightness of the IR image may be used.

[0225] Fig.19 It means to replace Fig.15 This is a flowchart of another example of the processing of steps S31 to S34. That is, Fig.19 FIG. 4 is a flowchart showing the detection process of the glare area using only the brightness of the IR image. Fig.19 In, with Fig.16 The same steps are marked with the same reference numerals and detailed descriptions are omitted. Fig.19 In the flowchart shown, Fig.16 Steps S43 to S45 are shown.

[0226] That is, the glare detection unit 112 determines whether the brightness of the pixel in the IR image is greater than the first threshold value in step S41. Here, if it is determined that the brightness of the pixel is less than the first threshold value (step S41 "No"), the glare detection unit 112 determines that glare is not generated in the area including the pixel (step S42). On the other hand, if it is determined that the brightness of the pixel is greater than the first threshold value (step S41 "Yes"), the glare detection unit 112 determines that glare is generated in the area including the pixel. That is, the glare detection unit 112 determines the area as a glare area (step S46).

[0227] That is, in Fig.19 In the example shown, the glare detection unit 112 detects a region in the IR image having a brightness greater than a threshold value 1 as a glare region. Since the brightness in the glare region tends to be higher than the brightness outside the glare region, the glare region can be appropriately detected by detecting a region in the IR image having a brightness greater than a first threshold value as a glare region.

[0228] In addition, in the discrimination between the glare area and the non-glare area, it can be achieved by using a learning process. In the learning process, for example, a process such as deep learning can be used. In this case, for learning, an IR image and a BW image, and a correct image that divides the IR image into a glare area and a non-glare area are prepared in advance. Next, the IR image and the BW image are provided as input to a learning model. Then, the learning model is learned so that the output from the learning model for the input is consistent with the correct image. For example, the learning model is a neural network. The output from the learning model is an image showing a numerical value "0" or a numerical value "1" for each pixel, where the numerical value "0" indicates that the pixel belongs to the non-glare area, and the numerical value "1" indicates that the pixel belongs to the glare area.

[0229] The glare detection unit 112 distinguishes between the glare area and the non-glare area by using the learning model that has been learned in advance. That is, the glare detection unit 112 inputs the IR image and the BW image into the learning model as input. Then, the glare detection unit 112 distinguishes the area including the pixel corresponding to the value "0" output from the learning model as the non-glare area. Furthermore, the glare detection unit 112 distinguishes the area including the pixel corresponding to the value "1" output from the learning model as the glare area.

[0230] Through the above processing, the glare detection unit 102 divides the captured IR image into a glare area where glare etc. is generated and a non-glare area.

[0231] <Depth Correction Processing>

[0232] The second depth estimation unit 111 b generates the second depth information using the BW image, the first depth information, and the glare area (that is, the above-mentioned area determination result).

[0233] Glare and the like are phenomena that depend on the wavelength of light. Therefore, glare and the like generated in IR images are often not generated in BW images. Therefore, by using the BW image to correct the first depth information that is not obtained from the IR image only in the glare area, the second depth information that is not affected by glare and the like generated in the IR image can be obtained.

[0234] In acquiring the second depth information, a guided filter, which is a type of image correction filter, can be used. The guided filter is disclosed in a non-patent document (Kaiming He, Jian Sun and Xiaoou Tang, "Guided Image Filtering", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 35, Iss. 6, PP. 1397-1409, 2013.). The guided filter is a filter that corrects the object image by using the correlation between the object image and the reference image. In the guided filter, the reference image I and the object image p are assumed to be represented by parameters a and b as shown in the following (Formula 2).

[0235] [Mathematical formula 1]

[0236]

[0237] Here, q is an output image after correction of the target image p, i is the number of each pixel, and ωk represents the surrounding area of ​​the pixel k. In addition, parameters a and b are expressed by the following (Formula 3).

[0238] [Mathematical formula 2]

[0239]

[0240]

[0241] Here, ε is a normalization parameter. In addition, μ and σ are the mean and variance within the block of the reference image, and are calculated using the following (Equation 4).

[0242] [Mathematical formula 3]

[0243]

[0244]

[0245] However, in order to suppress the noise included in the obtained parameters a and b, the output is obtained as shown in the following (Equation 5) using the averaged parameters.

[0246] [Formula 4]

[0247]

[0248]

[0249]

[0250] In this embodiment, the second depth estimation unit 111b corrects the target image, i.e., the first depth information (or the first depth image), by giving the BW image as a reference image. The second depth information is thereby generated or acquired. In the generation of such second depth information, an IR image and a BW image having substantially the same viewpoint position are required. In the depth acquisition device 1 of this embodiment, for each pixel, the filter used in the pixel is set to be either an IR filter or a BW filter. That is, Figure 2 As shown, the first pixels 21 having IR filters and the second pixels 22 having BW filters are arranged alternately in the column direction. Thus, since an IR image and a BW image of substantially the same viewpoint can be obtained, appropriate second depth information can be obtained.

[0251] Of course, the second depth estimation unit 111b may also use processing other than guided filtering. For example, the second depth estimation unit 111b may use processing such as bilateral filtering (non-patent literature: C. Tomasi, R. Manduchi, "Bilateral filtering for gray and color images", IEEE International Conference on Computer Vision (ICCV), pp. 839-846, 1998) or Mutual-Structure for Joint Filtering (non-patent literature: Xiaoyong Shen, Chao Zhou, Li Xu and Jiaya Jia, "Mutual-Structure for Joint Filtering", IEEE International Conference on Computer Vision (ICCV), 2015).

[0252] As described above, in this embodiment, the first depth information is used in an area determined to not generate glare, etc. (i.e., non-glare area), and the second depth information is used in an area determined to generate glare, etc. (i.e., glare area). Thus, even if glare, etc., is generated in the IR image, more accurate depth information can be obtained.

[0253] (Variant 1)

[0254] In the above-described embodiment, a filter such as a guided filter is used to generate the second depth information. However, the second depth information may be generated using a learning model.

[0255] For example, deep learning as a learning process can be used as in the non-patent document (Shuran Song, Fisher Yu, Andy Zeng, Angel X. Chang, Manolis Savva and Thomas Funkhouser, "Semantic Scene Completion from a Single Depth Image", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 190-198, 2017.). That is, the learning model can be made to learn so that if a BW image and the first depth information are input, the second depth information is output. In the above non-patent document, a network is proposed that interpolates the defective area of ​​the depth information if the depth information and the color image containing the defective area are input. The second depth estimation unit 111b in this variant provides the IR image, the BW image and the first depth information to the same network (i.e., the learning model) as the non-patent document, and further provides the glare area detected by the glare detection unit 112 as a mask image of the defective area. As a result, the second depth information with higher accuracy can be obtained from the network.

[0256] Fig. 20 This is a block diagram showing an example of the functional configuration of the depth acquisition device 1 in this modification.

[0257] The depth acquisition device 1 in this modified example includes Figure 8 The components shown in the figure further include a learning model 104 including, for example, a neural network.

[0258] The second depth estimation unit 111 b inputs three types of data, namely, the IR image, the BW image, and the first depth information, to the learning model 104 , and generates the second depth information using the glare area as a mask area to be corrected.

[0259] In the learning of the learning model 104, in addition to the IR image, the BW image and the first depth information, the correct depth image is prepared in advance. Next, the IR image, the BW image, the first depth information, and the mask image of the designated glare area are provided as inputs to the learning model 104. Then, the learning model 104 is made to learn so that the output from the learning model 104 for the input is consistent with the correct depth image. In addition, the mask image is randomly given during learning. The second depth estimation unit 111b uses the learning model 104 that has been learned in advance in this way. That is, the second depth estimation unit 111b can obtain the second depth information output from the learning model 104 by inputting the IR image, the BW image, the first depth information, and the mask image of the designated glare area into the learning model 104.

[0260] Thus, in this modification, the second depth estimation unit 111b estimates the depth information indicating the depth at each position in the IR image, and corrects the depth at each position in the glare area indicated by the depth information by inputting the IR image, the BW image, the glare area, and the depth information into the learning model. Therefore, if the learning model is pre-learned so as to output the correct depth at each position in the glare area in response to the input of the IR image, the BW image, the glare area, and the depth information, the depth information estimated from the IR image can be appropriately corrected. That is, the depth at each position in the glare area indicated by the depth information can be appropriately corrected.

[0261] As described above, the second depth estimation unit 111b may use deep learning. In this case, there is no need to directly output the glare area, and the second depth information may be directly generated by deep learning.

[0262] Fig.21 This is a block diagram showing another example of the functional configuration of the depth acquisition device 1 in this modification.

[0263] The depth acquisition device 1 in this modification does not include Fig. 20 The components shown in the figure include the glare detection unit 112 , the high-luminance region detection unit 116 , the first edge detection unit 117IR, and the second edge detection unit 117BW, but the device also includes components other than these.

[0264] In the learning of the learning model 104, Fig. 20Similarly to the example shown, in addition to the IR image, the BW image and the first depth information, a correct depth image is prepared in advance. Next, the IR image, the BW image and the first depth information are provided as inputs to the learning model 104. Then, the learning model 104 is made to learn so that the output from the learning model 104 for the input is consistent with the correct depth image. As the learning model 104, a VGG-16 network with skip combining added can be used as described in the non-patent literature (Caner Hazirbas, Laura Leal-Taixe and Daniel Cremers C. Hazirbas, "Deep Depth From Focus", In ArXiv preprint arXiv, 1704.01085, 2017.). The number of channels of the learning model 104 is changed so that the IR image, the BW image and the first depth information are given as inputs to the learning model 104. By using the learning model 104 that has been learned in advance in this manner, the second depth estimating unit 111 b can easily acquire the second depth information from the learning model 104 by inputting the IR image, the BW image, and the first depth information into the learning model 104 .

[0265] Right now, Fig.21 The depth acquisition device 1 shown includes a memory and a processor 110. Fig.21 Although not shown in the figure, the memory can be Figure 5 As shown in the figure, it is loaded on the depth acquisition device 1. The processor 110 obtains timing information indicating the timing of the light source 101 irradiating the subject with infrared light. Next, the processor 110 obtains an IR image, wherein the IR image is obtained by infrared light-based imaging of a scene including the subject corresponding to the timing indicated by the timing information, and is stored in the memory. Next, the processor 110 obtains a BW image, wherein the BW image is obtained by visible light-based imaging of a scene substantially the same as the IR image and imaging at substantially the same viewpoint and imaging time as the IR image, and is stored in the memory. Then, the first depth estimation unit 111a of the processor 110 estimates the depth information indicating the depth at each position in the IR image. The second depth estimation unit 111b corrects the depth at each position in the glare area of ​​the IR image indicated by the depth information by inputting the IR image, the BW image and the depth information into the learning model 104.

[0266] Therefore, if the learning model 104 is pre-learned so as to output the correct depth at each position in the glare area of ​​the IR image in response to the input of the IR image, the BW image, and the depth information, the depth information estimated from the IR image can be appropriately corrected. In other words, the depth at each position in the glare area indicated by the depth information can be appropriately corrected without detecting the glare area.

[0267] (Variant 2) <Use in the Time Direction>

[0268] Here, the glare detection unit 112 may also use information in the time direction to distinguish between the glare area and the non-glare area. The glare is a phenomenon that is generated inside the IR camera 102 not by the subject itself but by the relationship between the subject and the light source 101. For this reason, when the IR camera 102 moves, the shape of the glare area changes. For this reason, the glare detection unit 112 detects a candidate for the glare area from the IR image using the above-mentioned method, and determines whether the shape of the candidate has changed based on the glare area detected in the previous IR image. Then, if the glare detection unit 112 determines that the shape has not changed, it can be determined that the candidate is not a glare area but a non-glare area.

[0269] Fig. 22 : is a flowchart showing the processing operation of the glare detection unit 112 in this modification.

[0270] (Step S51)

[0271] First, the glare detection unit 112 determines whether there is a candidate for a glare region in the target frame. Figure 16 to Figure 19 The glare region detected by the flowchart shown in the figure is not the final glare region and exists in the target frame as a candidate of the glare region. In addition, the target frame is an IR image set as a determination target, that is, an IR image for determining the presence of the glare region.

[0272] (Step S54)

[0273] Here, when there is no candidate for a glare region in the target frame (No in step S51 ), the glare detection unit 112 determines that there is no glare in the target frame (ie, the IR image).

[0274] (Step S52)

[0275] On the other hand, when there is a candidate for a glare region in the target frame (step S51: Yes), the glare detection unit 112 determines whether there is a glare region in a frame before the target frame. The frame before the target frame is an IR image acquired by the IR camera 102 before the target frame.

[0276] (Step S55)

[0277] Here, when there is no glare region in the frame before the target frame (No in step S52 ), the glare detection unit 112 determines that the detected glare region candidate is a glare region where glare occurs.

[0278] (Step S53)

[0279] On the other hand, when there is a glare area in the frame before the target frame (step S52 "Yes"), the glare detection unit 112 compares the shape of the candidate for the glare area of ​​the target frame with the shape of the glare area of ​​the previous frame. Here, when the shape of the candidate for the glare area of ​​the target frame is similar to the shape of the glare area of ​​the previous frame (step S53 "Yes"), the glare detection unit 112 performs the processing of step S54. That is, the glare detection unit 112 updates the candidate for the detected glare area to a non-glare area, and determines that there is no glare in the target frame (i.e., the IR image). On the other hand, when the shape of the candidate for the glare area of ​​the target frame is not similar to the shape of the glare area of ​​the previous frame (step S53 "No"), the glare detection unit 112 performs the processing of step S55. That is, the glare detection unit 112 determines that the candidate for the detected glare area is an area generated by glare, i.e., a glare area.

[0280] In this way, the glare detection unit 112 in this modification uses information in the time direction. That is, the glare detection unit 112 uses the shapes of the glare area or its candidate in each frame acquired at different times. This makes it possible to distinguish the glare area and the non-glare area with higher accuracy.

[0281] In addition, the glare detection unit 112 may not identify the glare area for each frame, but may identify the glare area for each of a plurality of nearby frames. In addition, the plurality of nearby frames may be, for example, a plurality of IR images acquired in succession in time by the IR camera 102. That is, the glare detection unit 112 may also identify the glare area for each of the plurality of nearby frames based on, for example, the above-mentioned Figure 16 to Figure 19 , to detect the candidates of the glare area and determine whether the candidates are the glare area. More specifically, the glare detection unit 112 compares the shapes of the candidates of the glare area in each frame, and when the shapes of the candidates of the glare area are substantially equal, determines that the candidates are not the glare area, that is, determines that they are the non-glare area. On the other hand, when the shapes of the candidates of the glare area are not similar, the glare detection unit 112 determines that the candidates are the glare area.

[0282] Alternatively, the glare detection unit 112 may determine whether the shapes are similar by determining whether the similarity between the two shapes is equal to or greater than a threshold value. The similarity may be calculated as a correlation value between the two shapes, for example.

[0283] (Variant 3) <Utilization of Multiple IR Source Images>

[0284] The depth acquisition device 1 in the above embodiment and its variants 1 and 2 uses IR images and BW images to detect the glare area, but it is also possible not to use the BW image. The depth acquisition device 1 in this variant uses multiple IR source images to detect the glare area. For example, multiple IR source images are Figure 3 The infrared image obtained during the first exposure period and the infrared image obtained during the second exposure period are shown.

[0285] That is, in this modification, when the IR camera 102 acquires a plurality of infrared images at different timings in order to estimate the first depth information using TOF or the like, these plurality of infrared images are used to distinguish between the glare area and the non-glare area. In the following description, each of the plurality of infrared images is referred to as an IR source image. In addition, the above-mentioned IR image can be said to be composed of these plurality of IR source images.

[0286] There are two types of depth estimation based on TOF: direct TOF that directly measures the arrival time of emitted light, and indirect TOF that estimates depth from multiple IR source images obtained due to different timings of light emission and light reception. In this variant, the depth acquisition device 1 distinguishes glare areas and non-glare areas from multiple IR source images obtained during indirect TOF.

[0287] The timings of obtaining the multiple IR source images, that is, the timings of receiving light of the multiple IR source images, are different. Therefore, even if the first IR source image among the multiple IR source images generates glare, it is highly likely that the second IR source image that receives light at a different timing from the first IR source image will not generate glare. For this reason, the depth acquisition device 1 in this modified example compares the multiple IR source images obtained by changing the timing of receiving light, thereby distinguishing the glare area and the non-glare area. In this way, the third depth image can be made close to Fig.14 The depth image of the correct solution shown in (d).

[0288] [Specific processing flow of the depth acquisition device]

[0289] Fig.23 This is a flowchart showing the overall processing operation of the depth acquisition device 1 in this modification.

[0290] (Step S31a)

[0291] First, the high-brightness region detection unit 116 detects a high-brightness region from the first IR source image.

[0292] (Step S32a)

[0293] The first edge detection unit 117IR detects an edge located in the first IR source image.

[0294] (Step S33a)

[0295] The second edge detection unit 117BW detects an edge located in the second IR source image instead of the BW image.

[0296] (Step S34a)

[0297] The glare detection unit 112 detects the glare area in the IR image by comparing the edge in the high brightness area of ​​at least one high brightness area of ​​the first IR source image with the edge in the corresponding area of ​​the second IR source image. That is, when the correlation value between the edge in the high brightness area and the edge in the corresponding area of ​​the second IR source image is less than the fourth threshold value, the glare detection unit 112 detects the high brightness area as the glare area. In this way, the IR image area is divided into at least one glare area and a non-glare area.

[0298] (Step S35)

[0299] The first depth estimating unit 111 a generates first depth information from an IR image using, for example, TOF.

[0300] (Step S36)

[0301] The second depth estimating unit 111 b generates second depth information indicating the depth of the glare region based on the first depth information of the IR image and the BW image.

[0302] (Step S37)

[0303] The output unit 118 generates third depth information by replacing the depth of the glare area indicated by the first depth information with the depth indicated by the second depth information.

[0304] Fig.24 Yes means Fig.23 Flowchart of detailed processing of steps S31a to S34a.

[0305] (Step S41a)

[0306] First, the high-brightness area detection unit 116 determines whether the brightness at each position in the first IR source image is above the fifth threshold value. Here, for example, if the first IR source image is a 12-bit grayscale image, the fifth threshold value may be about 1500. Of course, the fifth threshold value may be a value that varies according to environmental conditions or the settings of the IR camera 102. For example, in the case of photographing a dark scene such as at night, since the overall brightness of the first IR source image becomes lower, the fifth threshold value may be a smaller value than the case of photographing a bright scene during the day. In addition, in the case of a long exposure time of the IR camera 102, since the overall brightness of the first IR source image becomes higher, the fifth threshold value may be a larger value than the case of a short exposure time.

[0307] (Step S42a)

[0308] Here, if it is determined that the brightness at any position is not greater than the fifth threshold value (step S41a "No"), the high brightness area detection unit 116 determines that no glare occurs in the IR image formed using the first IR source image (step S42). That is, the entire IR image is determined to be a non-glare area.

[0309] (Step S43a)

[0310] On the other hand, if it is determined that the brightness at any position is greater than the fifth threshold value (step S41a "Yes"), the high brightness region detection unit 116 will perform region segmentation on the 1IR source image. That is, the high brightness region detection unit 116 divides the first IR source image into at least one high brightness region and regions other than the high brightness region. For example, a brightness-based technique such as SuperPixel can be used in this region segmentation.

[0311] (Step S44a)

[0312] Next, the first edge detection unit 117IR and the second edge detection unit 117BW perform edge detection on the first IR source image and the second IR source image, respectively. The Canny method or the Sobel filter can be used for edge detection.

[0313] (Step S45a)

[0314] The glare detection unit 112 compares the edge in at least one high-brightness area of ​​the first IR source image with the edge in the area of ​​the second IR source image corresponding to the high-brightness area. That is, the glare detection unit 112 determines whether the correlation value between the edge in the high-brightness area of ​​the first IR source image and the edge in the corresponding area of ​​the second IR source image is greater than the fourth threshold value. The values ​​output by edge detection of the first IR source image and the second IR source image are arranged in a vector form for each area, and the inner product values ​​are normalized to obtain the correlation value. That is, the glare detection unit 112 normalizes the inner product value of the vector containing multiple values ​​obtained by edge detection in the high-brightness area of ​​the first IR source image and the vector containing multiple values ​​obtained by edge detection in the area of ​​the second IR source image corresponding to the high-brightness area. The correlation value for the high-brightness area is thus calculated.

[0315] (Step S46a)

[0316] Here, if it is determined that the correlation value is not above the fourth threshold value, that is, is less than the fourth threshold value (step S45a "No"), the glare detection unit 112 determines the high-brightness area as a glare area. That is, due to the influence of glare, etc., the edge correlation between the area where glare, etc. is generated in the first IR source image and the corresponding area of ​​the second IR source image disappears, so the glare detection unit 112 determines the high-brightness area in the first IR source image as a glare area. That is, the glare detection unit 112 determines the high-brightness area in the IR image as a glare area. In addition, the high-brightness area in the IR image is the same area as the high-brightness area in the first IR source image.

[0317] On the other hand, if the glare detection unit 112 determines in step S45a that the correlation value is greater than or equal to the fourth threshold value, that is, not less than the fourth threshold value (step S45a "Yes"), it determines that no glare occurs in the high-brightness area. In other words, the glare detection unit 112 determines the high-brightness area as a non-glare area.

[0318] <Using the correlation of brightness>

[0319] In the above description, the edge is used in the discrimination of the glare area and the non-glare area, but the edge may not be used in the discrimination. For example, the correlation value of the brightness of each of the first IR source image and the second IR source image may be used. As mentioned above, when no glare or the like is generated, the first IR source image and the second IR source image have a strong correlation, but in the area where glare or the like is generated, the correlation becomes weak. For this reason, by using the correlation of the brightness of each of the first IR source image and the second IR source image, the glare area can be properly discriminated.

[0320] Fig.25 It means to replace Fig.23 This is a flowchart of an example of the processing of steps S31a to S34a. That is, Fig.25 is a flowchart showing a glare area detection process using the correlation between the brightness of the first IR source image and the second IR source image. Fig.25 In, with Fig.24 The same steps are marked with the same reference numerals and detailed descriptions are omitted. Fig.25 The flowchart shown is similar to Fig.24 The flowchart shown is different in that step S44a is not included and step S45b is included instead of step S45a.

[0321] (Step S45b)

[0322] In step S45b, the glare detection unit 112 calculates the correlation value between the brightness of each pixel in each high-brightness area obtained by the area segmentation in step S43a and the brightness of each pixel in the area of ​​the second IR source image corresponding to the high-brightness area. The brightness of each pixel of the first IR source image and the second IR source image are arranged in a vector form for each area, and their inner product values ​​are normalized by the number of pixels, thereby obtaining the correlation value. That is, the glare detection unit 112 normalizes the inner product value of the vector containing the brightness of each pixel in the high-brightness area of ​​the first IR source image and the vector containing the brightness of each pixel in the corresponding area of ​​the second IR source image. In this way, the correlation value for the high-brightness area is calculated.

[0323] Here, when the correlation value is greater than the fourth threshold value, that is, not less than the fourth threshold value (step S45b "Yes"), the glare detection unit 112 determines that no glare or the like is generated in the high-brightness area (step S42a). On the other hand, when the correlation value is less than the fourth threshold value (step S45b "No"), due to the influence of glare or the like, the correlation between the brightness of each pixel in the high-brightness area of ​​the first IR source image and the brightness of each pixel in the corresponding area of ​​the second IR source image becomes low. Therefore, in such a case, the glare detection unit 112 determines the high-brightness area as a glare area (step S46a).

[0324] That is, the image feature quantities in the respective areas of the first IR source image and the second IR source image used for the detection of the glare area are Fig.24 The example shown is an edge contained in the image within this region, but Fig.25 In the example shown, it is the brightness in the area. Here, as described above, the correlation between the brightness in the glare area of ​​the first IR source image and the brightness in the area of ​​the second IR source image corresponding to the glare area tends to be low. Therefore, in the first IR source image, by detecting an area with high brightness and low brightness correlation as a glare area, the glare area can be detected more appropriately.

[0325] <Conversion using CENSUS>

[0326] Of course, the evaluation value used for distinguishing the glare area and the non-glare area does not need to be a correlation value. For example, the aforementioned Hamming distance and CENSUS conversion may be used.

[0327] Fig.26 It means to replace Fig.23 This is a flowchart of another example of the processing of steps S31a to S34a. That is, Fig.26 is a flowchart showing the detection process of the glare area using CENSUS transformation for each of the first IR source image and the second IR source image. Fig.26 In, with Fig.24 The same steps are given the same reference numerals and detailed descriptions are omitted. Fig.26 The flowchart shown is similar to Fig.24 The flowchart shown is different in that steps S44c and S45c are included instead of steps S44a and S45a.

[0328] (Step S44c)

[0329] In step S44c, the glare detection unit 112 performs CENSUS transformation on the image of the high brightness area in the first IR source image and the image of the corresponding area in the second IR source image for each high brightness area obtained by the area segmentation in step S43a. Thus, a CENSUS transformed image of the image of the high brightness area in the first IR source image and a CENSUS transformed image of the image of the corresponding area in the second IR source image are generated.

[0330] (Step S45c)

[0331] Next, the glare detection unit 112 calculates the Hamming distance between the CENSUS transformed image of the first IR source image obtained in step S44c and the CENSUS transformed image of the second IR source image in step S45c. Then, when the value of the Hamming distance normalized by the number of pixels in the high-brightness area is less than the sixth threshold value (step S45c "Yes"), the glare detection unit 112 determines that no glare or the like is generated in the high-brightness area (step S42a). On the other hand, when the value of the normalized Hamming distance is greater than the sixth threshold value (step S45c "No"), the glare detection unit 112 determines that the texture disappears in the image of the high-brightness area due to the influence of glare or the like. As a result, the glare detection unit 112 determines that the high-brightness area is a glare area (step S46a).

[0332] That is, the glare detection unit 112 obtains a first transformed image by performing CENSUS transformation on the image in the high-brightness area of ​​at least one high-brightness area having a brightness greater than the fifth threshold value in the first IR source image. Then, the glare detection unit 112 obtains a second transformed image by performing CENSUS transformation on the image in the area of ​​the second IR source image corresponding to the high-brightness area. In addition, the first transformed image and the second transformed image are the above-mentioned CENSUS transformed images. Next, the glare detection unit 112 detects a high-brightness area in at least one high-brightness area where the Hamming distance between the first transformed image and the second transformed image exceeds the sixth threshold value as a glare area. By using CENSUS transformation in this way, the glare area can also be appropriately detected.

[0333] As described above, in the depth acquisition device 1 in the present embodiment and its modified example, even when there is a glare area in the IR image, the appropriate depth at each position in the glare area can be acquired by using the image of the corresponding area of ​​the BW image.

[0334] In addition, in each of the above embodiments, each component can be formed by dedicated hardware, or can be implemented by executing a software program suitable for each component. Each component can be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software that implements the depth acquisition device of the above embodiments and modified examples causes a computer to execute Figure 6 , Figure 7 , Figure 15 to Figure 19 as well as Figure 22 to Figure 26 Each step contained in any one of the flowcharts.

[0335] The depth acquisition device involved in one or more schemes is described above based on the embodiment and its variants, but the present disclosure is not limited to the embodiment and its variants. As long as it does not deviate from the main purpose of the present disclosure, the scheme obtained by implementing various modifications thought of by those skilled in the art to the embodiment and the variant, and the form constructed by combining the constituent elements in the embodiment and the variant can also be included in the scope of the present disclosure.

[0336] In addition, in the present disclosure, all or part of a component, device, or Figure 1 , Figure 4 , Figure 5 , Figure 8 , Fig. 20 as well as Fig.21All or part of the functional blocks of the block diagram shown can be executed by a semiconductor device, one or more electronic circuits including a semiconductor integrated circuit (IC) or LSI (large scale integration). LSI or IC can be integrated on a single chip, or can be composed of a combination of multiple chips. For example, functional blocks other than storage elements can be integrated on a single chip. Here, they are called LSI or IC, but the name changes depending on the degree of integration, and may be called system LSI, VLSI (very large scale integration) or ULSI (ultra large scale integration). For the same purpose, a field programmable gate array (FPGA) programmed after the manufacture of LSI, or a reconfigurable logic device that can reconfigure the connection relationship within the LSI or set the circuit division within the LSI can also be used.

[0337] Furthermore, all or part of the functions or operations of a component, device or part of a device can be executed by software processing. In this case, the software is recorded on one or more non-temporary recording media such as ROM, optical disk, hard disk drive, etc., and when the software is executed by a processor, the software enables the processor and peripheral devices to perform specific functions in the software. The system or device may have one or more non-temporary recording media recording the software, a processor, and necessary hardware devices such as interfaces.

[0338] Industrial Applicability

[0339] The present disclosure can be applied to a depth acquisition device that acquires depth from an image obtained by photographing, and can be used as, for example, an in-vehicle device.

[0340] Description of Reference Numerals

[0341] 1 Depth acquisition device

[0342] 10.101 Light Source

[0343] 20 Solid-state imaging element

[0344] 21 1st pixel (IR)

[0345] 22 2nd pixel (BW)

[0346] 30 Processing circuit

[0347] 50 Diffuser Plate

[0348] 60 shots

[0349] 70 Bandpass filter

[0350] 102 IR Camera

[0351] 103 BW Camera

[0352] 104 Learning Model

[0353] 110 Processor

[0354] 111 Depth Estimation Department

[0355] 111a 1st Depth Estimation Unit

[0356] 111b Second Depth Estimation Unit

[0357] 112 Glare Detection Department

[0358] 113 Lighting timing acquisition unit

[0359] 114 IR image acquisition unit

[0360] 115 BW image acquisition unit

[0361] 116 High brightness area detection unit

[0362] 117IR 1st edge detection unit

[0363] 117BW 2nd edge detection unit

[0364] 118 Output

[0365] 200 Memory.

Claims

1. A depth acquisition device, comprising a memory and a processor, the processor, Acquiring timing information indicating the timing at which the light source irradiates the subject with infrared light, acquiring an infrared image, the infrared image being obtained by capturing an image of a scene including the subject corresponding to the timing indicated by the timing information using infrared light, and being stored in the memory, Obtaining a visible light image, wherein the visible light image is obtained by photographing a scene substantially the same as the infrared image based on visible light and photographing at substantially the same viewpoint and photographing time as the infrared image, and is stored in the memory, detecting a glare area from the infrared image, estimating the depth of the glare area based on the infrared image, the visible light image and the glare area, the processor, In estimating the depth of the glare area, estimating depth information representing the depth at each position within the infrared image, The depth at each position in the glare area indicated by the depth information is corrected by inputting the infrared image, the visible light image, the glare area, and the depth information into a learning model.

2. The depth acquisition device according to claim 1, wherein: the processor, In estimating the depth of the glare area, estimating first depth information indicating the depth at each position in the infrared image, By correcting the depth of each position in the glare area indicated by the first depth information based on the visible light image, second depth information indicating the corrected depth of each position in the glare area is estimated, Furthermore, the third depth information is generated, which represents the depth of each position outside the glare area of ​​the infrared image represented by the first depth information and the depth of each position within the glare area of ​​the infrared image represented by the second depth information.

3. The depth acquisition device according to claim 1 or 2, wherein: The processor, in detecting the glare area, An area in the infrared image having a brightness above a first threshold is detected as the glare area.

4. The depth acquisition device according to claim 1 or 2, wherein: The processor, in detecting the glare area, The area in the infrared image having a brightness above a first threshold and satisfying a predetermined condition is detected as the glare area. The predetermined condition is a condition that a correlation value between an image feature quantity within a region of the infrared image and an image feature quantity within a region of the visible light image corresponding to the region is smaller than a second threshold value.

5. The depth acquisition device according to claim 4, wherein: The image feature amount within each area of ​​the infrared image and the visible light image is an edge contained in the image within the area.

6. The depth acquisition device according to claim 4, wherein: The image feature quantity within each area of ​​the infrared image and the visible light image is the brightness within the area.

7. The depth acquisition device according to claim 1 or 2, wherein: The processor, in detecting the glare area, For at least one high brightness area in the infrared image having brightness above a first threshold, respectively: (i) performing CENSUS transformation on the image in the high brightness region to obtain a first transformed image, (ii) obtaining a second transformed image by performing CENSUS transformation on an image in a region of the visible light image corresponding to the high brightness region, A high-luminance region in which a Hamming distance between the first converted image and the second converted image exceeds a third threshold value among at least one of the high-luminance regions is detected as the glare region.

8. A depth acquisition device, comprising a memory and a processor, the processor, Acquiring timing information indicating the timing at which the light source irradiates the subject with infrared light, acquiring an infrared image, the infrared image being obtained by capturing an image of a scene including the subject corresponding to the timing indicated by the timing information using infrared light, and being stored in the memory, Obtaining a visible light image, wherein the visible light image is obtained by photographing a scene substantially the same as the infrared image based on visible light and photographing at substantially the same viewpoint and photographing time as the infrared image, and is stored in the memory, estimating depth information representing the depth at each position within the infrared image, The depth at each position in the glare area of ​​the infrared image represented by the depth information is corrected by inputting the infrared image, the visible light image, and the depth information into a learning model.

9. A depth acquisition device, comprising: light source; an infrared camera that acquires an infrared image by performing infrared light-based imaging of a scene including the subject in accordance with a timing at which the light source irradiates the subject with infrared light; A visible light camera that obtains a visible light image by performing visible light-based imaging of a scene that is substantially the same as the infrared image and at substantially the same viewpoint and imaging time as the infrared image; A glare detection unit, configured to detect a glare area from the infrared image; and a depth estimating unit, configured to estimate the depth of the glare area based on the infrared image, the visible light image, and the glare area, In estimating the depth of the glare area, estimating depth information representing the depth at each position within the infrared image, The depth at each position in the glare area indicated by the depth information is corrected by inputting the infrared image, the visible light image, the glare area, and the depth information into a learning model.

10. A depth acquisition device, comprising: light source; an infrared camera that acquires an infrared image by performing infrared light-based imaging of a scene including the subject in accordance with a timing at which the light source irradiates the subject with infrared light; A visible light camera that obtains a visible light image by performing visible light-based imaging of a scene that is substantially the same as the infrared image and at substantially the same viewpoint and imaging time as the infrared image; a first depth estimating unit that estimates depth information indicating the depth at each position in the infrared image; and The second depth estimating unit corrects the depth at each position in the glare area of ​​the infrared image indicated by the depth information by inputting the infrared image, the visible light image, and the depth information into a learning model.

11. A depth acquisition device, comprising a memory and a processor, the processor, The infrared image stored in the memory is obtained by taking an infrared light-based camera. Obtaining a visible light image stored in the memory, wherein the visible light image is obtained by imaging based on visible light at substantially the same viewpoint and imaging time as the infrared image, Detecting the area reflecting the glare as the glare area from the infrared image, estimating the depth of the glare area based on the visible light image, the processor, In estimating the depth of the glare area, estimating depth information representing the depth at each position within the infrared image, The depth at each position in the glare area indicated by the depth information is corrected by inputting the infrared image, the visible light image, the glare area, and the depth information into a learning model.

12. A depth acquisition method, Acquiring timing information indicating the timing at which the light source irradiates the subject with infrared light, acquiring an infrared image, the infrared image being obtained by capturing an image of a scene including the subject corresponding to the timing indicated by the timing information using infrared light, and being stored in a memory, Obtaining a visible light image, wherein the visible light image is obtained by photographing a scene substantially the same as the infrared image based on visible light and photographing at substantially the same viewpoint and photographing time as the infrared image, and is stored in a memory, detecting a glare area from the infrared image, estimating the depth of the glare area based on the infrared image, the visible light image and the glare area, In estimating the depth of the glare area, estimating depth information representing the depth at each position within the infrared image, The depth at each position in the glare area indicated by the depth information is corrected by inputting the infrared image, the visible light image, the glare area, and the depth information into a learning model.

13. A depth acquisition method, Acquiring timing information indicating the timing at which the light source irradiates the subject with infrared light, acquiring an infrared image, the infrared image being obtained by capturing an image of a scene including the subject corresponding to the timing indicated by the timing information using infrared light, and being stored in a memory, Obtaining a visible light image, wherein the visible light image is obtained by photographing a scene substantially the same as the infrared image based on visible light and photographing at substantially the same viewpoint and photographing time as the infrared image, and is stored in a memory, estimating depth information representing the depth at each position within the infrared image, The depth at each position in the glare area of ​​the infrared image represented by the depth information is corrected by inputting the infrared image, the visible light image, and the depth information into a learning model.

14. A depth acquisition method, using a depth acquisition device having a memory and a processor, the processor, The infrared image stored in the memory is obtained by taking an infrared light-based camera. Obtaining a visible light image stored in the memory, wherein the visible light image is obtained by imaging based on visible light at substantially the same viewpoint and imaging time as the infrared image, Detecting the area reflecting the glare as the glare area from the infrared image, estimating the depth of the glare area based on the visible light image, the processor, In estimating the depth of the glare area, estimating depth information representing the depth at each position within the infrared image, The depth at each position in the glare area indicated by the depth information is corrected by inputting the infrared image, the visible light image, the glare area, and the depth information into a learning model.

15. A recording medium storing a program, wherein the program causes a computer to execute the following steps: Acquiring timing information indicating the timing at which the light source irradiates the subject with infrared light, acquiring an infrared image, the infrared image being obtained by capturing an image of a scene including the subject corresponding to the timing indicated by the timing information using infrared light, and being stored in a memory, Obtaining a visible light image, wherein the visible light image is obtained by photographing a scene substantially the same as the infrared image based on visible light and photographing at substantially the same viewpoint and photographing time as the infrared image, and is stored in a memory, detecting a glare area from the infrared image, estimating the depth of the glare area based on the infrared image, the visible light image and the glare area, In estimating the depth of the glare area, estimating depth information representing the depth at each position within the infrared image, The depth at each position in the glare area indicated by the depth information is corrected by inputting the infrared image, the visible light image, the glare area, and the depth information into a learning model.

16. A recording medium storing a program, wherein the program causes a computer to execute the following steps: Acquiring timing information indicating the timing at which the light source irradiates the subject with infrared light, acquiring an infrared image, the infrared image being obtained by capturing an image of a scene including the subject corresponding to the timing indicated by the timing information using infrared light, and being stored in a memory, Obtaining a visible light image, wherein the visible light image is obtained by photographing a scene substantially the same as the infrared image based on visible light and photographing at substantially the same viewpoint and photographing time as the infrared image, and is stored in a memory, estimating depth information representing the depth at each position within the infrared image, The depth at each position in the glare area of ​​the infrared image represented by the depth information is corrected by inputting the infrared image, the visible light image, and the depth information into a learning model.

Images