Information processing method, information processing device, and program

Abstract

To provide an information processing method capable of accurately acquiring the depth of an image.SOLUTION: This information processing method acquires an IR image by performing imaging for receiving light emitted from a light source 10 and reflected by a subject, acquires a BW image by imaging the subject by visible light, and detects a dust area on the basis of the IR image and the BW image.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to a depth acquisition device that acquires the depth of an image. [Background technology]

[0002] Conventionally, a distance measuring device that measures the distance to a subject has been proposed (see, for example, Patent Document 1). This distance measuring device includes a light source and an imaging unit. The light source irradiates light onto the subject. The imaging unit captures the light reflected by the subject. The distance measuring device then measures the distance to the subject by converting each pixel value of the image obtained by the imaging into the distance to the subject. In other words, the distance measuring device obtains the depth of the image obtained by the imaging unit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-64498 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the distance measuring device of Patent Document 1 has a problem in that it is not possible to accurately obtain the depth of an image.

[0005] Therefore, the present disclosure provides a depth acquisition device that can accurately acquire the depth of an image. [Means for solving the problem]

[0006] In an information processing method according to one embodiment of the present disclosure, an IR image is obtained by capturing light irradiated from a light source and reflected by a subject, an image of the subject is captured using visible light to obtain a BW image, and a dust area is detected based on the IR image and the BW image.

[0007] An imaging device according to one aspect of the present disclosure includes a light source that irradiates a subject with irradiation light, a solid-state imaging element that performs a first image of the subject and a second image using light reflected by the subject from the irradiation light, a dust detection unit that detects a dust area in which dust is displayed using subject information regarding the subject output by the solid-state imaging element, and an output unit that generates and outputs output information according to the detected dust area.

[0008] A depth acquisition device according to one aspect of the present disclosure includes a memory and a processor, wherein the processor acquires timing information indicating a timing at which a light source irradiates a subject with infrared light, acquires an infrared image obtained by imaging a scene including the subject based on infrared light in accordance with the timing indicated by the timing information and stored in the memory, acquires a visible light image of substantially the same scene as the infrared image, obtained by imaging based on visible light from substantially the same viewpoint and at substantially the same imaging time as the infrared image and stored in the memory, detects an area in which dust is reflected from the infrared image as a dust area, and estimates a depth of the dust area based on the infrared image, the visible light image, and the dust area.

[0009] These comprehensive or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized by any combination of the system, the method, the integrated circuit, the computer program, and the recording medium. The recording medium may be a non-transitory recording medium. Effect of the Invention

[0010] The depth acquisition device of the present disclosure can accurately acquire the depth of an image. Further advantages and effects of one aspect of the present disclosure will be apparent from the specification and drawings. Such advantages and / or effects are provided by some embodiments and features described in the specification and drawings, respectively, but not all of them are necessarily provided to obtain one or more identical features. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing a hardware configuration of a depth acquisition device according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing a pixel array of the solid-state imaging device according to the embodiment. [Diagram 3] FIG. 3 is a timing chart showing the relationship between the light emission timing of the light emitting element of the light source and the exposure timing of the first pixel of the solid-state imaging element in the embodiment. [Figure 4] FIG. 4 is a block diagram illustrating an example of a functional configuration of the depth acquisition device according to the embodiment. [Diagram 5] FIG. 5 is a block diagram illustrating another example of the functional configuration of the depth acquisition device according to the embodiment. [Figure 6] FIG. 6 is a flowchart showing the overall processing operation of the depth acquisition device according to the embodiment. [Figure 7] FIG. 7 is a flowchart showing an overall processing operation by the processor of the depth acquisition device in the embodiment. [Figure 8] FIG. 8 is a block diagram showing a specific functional configuration of a processor of the depth acquisition device according to the embodiment. [Figure 9A] FIG. 9A is a diagram showing an example of an IR image. [Figure 9B] FIG. 9B is a diagram showing an example of a BW image. [Figure 10] FIG. 10 is a diagram showing an example of a binarized image obtained by binarizing an IR image. [Figure 11] FIG. 11 is a diagram showing an example of a dust candidate region in an IR image. [Figure 12] FIG. 12 is a diagram showing an example of FOE detected in a BW image. [Figure 13] FIG. 13 is a diagram showing an example of a principal axis detected for a dust region. [Figure 14]FIG. 14 is a diagram showing an example of a dust region and a non-dust region. [Figure 15A] FIG. 15A is a diagram showing another example of an IR image. [Figure 15B] FIG. 15B is a diagram showing another example of the BW image. [Figure 16] FIG. 16 is a diagram showing an example of a binarized image obtained by binarizing an IR image. [Figure 17] FIG. 17 is a diagram showing an example of a dust candidate region in an IR image. [Figure 18] FIG. 18 is a diagram showing an example of FOE detected in a BW image. [Figure 19] FIG. 19 is a diagram showing an example of the arrangement of each dust candidate region. [Figure 20] FIG. 20 is a diagram showing a simulation result of the depth acquisition device according to the embodiment. [Figure 21] FIG. 21 is a flowchart showing the overall processing operation of the depth acquisition device shown in FIG. [Figure 22] FIG. 22 is a flowchart showing an example of detailed processing of steps S31 to S34 in FIG. [Diagram 23] FIG. 23 is a flowchart showing another example of the detailed processing of steps S31 to S34 in FIG. [Figure 24] FIG. 24 is a flow chart showing an example of a process instead of steps S31 to S34 in FIG. [Diagram 25] FIG. 25 is a flowchart showing another example of the detailed processing of steps S31 to S34 in FIG. [Figure 26] FIG. 26 is a block diagram illustrating an example of a functional configuration of a depth acquisition device according to a modification of the embodiment. In FIG. [Figure 27] FIG. 27 is a block diagram illustrating another example of a functional configuration of a depth acquisition device according to the modification of the embodiment. In FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (Findings on which this disclosure is based) The present inventors have found that the distance measuring device of Patent Document 1 described in the "Background Art" section has the following problems.

[0013] As described above, the distance measuring device of Patent Document 1 irradiates a subject with light from a light source, captures an image of the subject irradiated with the light, and measures the depth of the image. Time of Flight (TOF) is used for measuring the depth. In such a distance measuring device, imaging is performed under different imaging conditions to improve the accuracy of distance measurement. That is, the distance measuring device captures an image according to a predetermined imaging condition, and sets an imaging condition different from the predetermined imaging condition depending on the imaging result. Then, the distance measuring device captures an image again according to the set imaging condition.

[0014] However, dust particles near the distance measuring device may appear as noise in the image obtained by imaging. As a result, the dust particles cannot be removed from the image in which the dust particles are reflected, and the depth cannot be measured correctly. In addition, even if the imaging conditions are changed, it may be difficult to easily suppress the dust particles from being reflected. Furthermore, for example, if a distance measuring device mounted on a vehicle repeatedly captures images under different imaging conditions while the vehicle is traveling, the viewpoint positions of the repeated images are different, and the scenes of the multiple images obtained are different. In other words, it is not possible to repeatedly capture the same scene, and it is not possible to properly estimate the depth of the image in which the scene is captured, particularly the depth of the area in which the dust particles are captured.

[0015] In order to solve such problems, a depth acquisition device according to one aspect of the present disclosure includes a memory and a processor, wherein the processor acquires timing information indicating a timing at which a light source irradiates a subject with infrared light, acquires an infrared image obtained by imaging a scene including the subject based on infrared light in accordance with the timing indicated by the timing information and stored in the memory, acquires a visible light image of substantially the same scene as the infrared image, obtained by imaging based on visible light from substantially the same viewpoint and at substantially the same imaging time as the infrared image and stored in the memory, detects an area in which dust is reflected from the infrared image as a dust area, and estimates a depth of the dust area based on the infrared image, the visible light image, and the dust area.

[0016] As a result, a dust region is detected from the infrared image, and the depth of the dust region is estimated based on not only the infrared image but also the visible light image, so that the depth of the dust region can be appropriately obtained. That is, the scene to be captured is substantially the same in the infrared image and the visible light image, and the viewpoint and the capture time are also substantially the same. Here, an example of an image of a substantially identical scene captured at substantially the same viewpoint and capture time is an image captured at different pixels of the same image sensor. Such an image is similar to the red, green, and blue channel images of a color image captured with a Bayer array color filter, and the angle of view, viewpoint, and capture time of each image are approximately equal. That is, in the images of a substantially identical scene captured at substantially the same viewpoint and capture time, the position of the subject on the image does not differ by more than two pixels in each captured image. For example, when a point light source having visible light and infrared components exists in the scene and only one pixel is captured with high luminance in the visible light image, the point light source is also captured in the infrared image closer than two pixels to the pixel position captured in the visible light image. Furthermore, substantially the same imaging time means that the difference between the imaging times is equal to one frame or less. Therefore, the infrared image and the visible light image have a high correlation. However, if there is dust near the camera that captures the infrared image and the visible light image, the irradiated infrared light may be strongly reflected by the dust, and the dust may be reflected in the infrared image due to the strong reflected light. Therefore, even if the dust is reflected in the infrared image, it is highly likely that it will not be reflected in the visible light image. Therefore, the missing information in the dust region can be compensated for from the region in the visible light image that corresponds to the dust region (i.e., the corresponding region). As a result, the depth of the dust region can be appropriately acquired without the influence of the noise of the dust.

[0017] For example, the processor may estimate first depth information indicating the depth at each position in the infrared image, correct the depth at each position in the dust region indicated by the first depth information based on the visible light image, and estimate second depth information indicating the corrected depth at each position in the dust region, and generate third depth information indicating the depth at each position outside the dust region in the infrared image indicated by the first depth information and the depth at each position in the dust region in the infrared image indicated by the second depth information. Note that, in estimating the first depth information, TOF or the like may be applied to the infrared image.

[0018] Thus, the third depth information indicates a depth obtained from the infrared image as a depth outside the dusty region of the infrared image, and indicates a depth obtained from the infrared image and corrected based on the visible light image as a depth of the dusty region of the infrared image. Therefore, even if the infrared image has a dusty region, the overall depth of the infrared image can be appropriately estimated.

[0019] In addition, in detecting the dust region, the processor may detect a high-brightness region in the infrared image having a brightness equal to or greater than a first threshold as the dust region if the high-brightness region satisfies a first condition.

[0020] The brightness tends to be high in dusty regions. Therefore, by detecting high-brightness regions having brightness equal to or greater than the first threshold in the infrared image, it is possible to easily narrow down the regions in which dust is likely to be present. Furthermore, since high-brightness regions that satisfy the first condition are detected as dusty regions, dusty regions can be detected with high accuracy by appropriately setting the first condition.

[0021] The first condition may be a condition that the center of gravity of the high-brightness area is located on a straight line or an arc intersecting the center of gravity of each of at least two other high-brightness areas different from the high-brightness area in the infrared image and the FOE (Focus of Expansion) of the infrared image or the visible light image.

[0022] For example, when a camera captures one dust particle, if multiple exposures are performed for capturing one frame of an infrared image, the dust particle will be displayed in the infrared image as a high-luminance area each time the exposure is performed. Therefore, when the above-mentioned camera is mounted on a moving body such as a vehicle and moves, the dust particle is likely to be displayed in the infrared image as if it is blowing out of the FOE. For example, if the moving speed of the above-mentioned moving body is high, these multiple high-luminance areas caused by the dust particle tend to be arranged on a straight line intersecting with the FOE. Alternatively, if the distortion of the lens of the camera is large, the multiple high-luminance areas caused by the dust particle tend to be arranged on an arc intersecting with the FOE. Therefore, by setting the first condition that the centers of gravity of at least three high-luminance areas are arranged on the above-mentioned straight line or arc, the dust particle area can be detected with high accuracy.

[0023] In addition, since the scenes captured in the infrared image and the visible light image are substantially the same and the viewpoints are also substantially the same, the FOE of the infrared image and the FOE of the visible light image are substantially the same.

[0024] Alternatively, the first condition may be a condition that a principal axis of the high-luminance region or an extension of the principal axis intersects with a focus of expansion (FOE) of the infrared image or the visible light image.

[0025] For example, contrary to the above example, if the moving speed of the moving body equipped with the camera is slow, the above-mentioned multiple high-brightness areas will overlap. As a result, these multiple high-brightness areas will be displayed in the infrared image as a single high-brightness area with an elongated shape, in other words, a tail shape. The main axis or an extension of the main axis of such a high-brightness area caused by dust tends to intersect with the FOE. Therefore, by setting the first condition that the main axis of the high-brightness area or an extension of the main axis intersects with the FOE, the dust area can be detected with high accuracy.

[0026] In addition, in detecting the dust region, the processor may detect the high luminance region as the dust region if the high luminance region further satisfies a second condition.

[0027] For example, the property that a dust region is observed in an infrared image but not in a visible light image may be used as the second condition, which allows the dust region to be detected with higher accuracy.

[0028] The second condition may be a condition that the luminance at a position in the visible light image corresponding to the center of gravity of the high luminance area in the infrared image is less than a second threshold value.

[0029] Since dust regions are not observed in BW images, the luminance of the position in the visible light image corresponding to the center of gravity of the dust region tends to be low. Therefore, by setting the luminance of the position in the visible light image corresponding to the center of gravity of the high luminance region in the infrared image to be less than the second threshold as the second condition, dust regions can be detected with higher accuracy.

[0030] The second condition may be a condition that a correlation coefficient between a luminance in a high luminance region of the infrared image and a luminance in a region of the visible light image corresponding to the high luminance region is less than a third threshold value.

[0031] Due to the nature of dust regions being observed in infrared images but not in visible light images, there tends to be a low correlation between the luminance in a dust region in an infrared image and the luminance in a region in a visible light image corresponding to that dust region. Therefore, by setting the correlation coefficient between the luminance in a high-luminance region in an infrared image and the luminance in a region in a visible light image corresponding to that high-luminance region to be less than a third threshold as the second condition, dust regions can be detected with higher accuracy.

[0032] In addition, when estimating the depth of the dust region, the processor may estimate depth information indicating the depth at each position in the infrared image, and correct the depth at each position in the dust region indicated by the depth information by inputting the infrared image, the visible light image, the dust region, and the depth information into a learning model.

[0033] Thus, if the learning model is trained in advance to output the correct depth at each position in the dust region in response to the input of the infrared image, the visible light image, the dust region, and the depth information, the depth information estimated from the infrared image can be appropriately corrected. In other words, the depth at each position in the dust region indicated by the depth information can be appropriately corrected.

[0034] In addition, a depth acquisition device according to another aspect of the present disclosure includes a memory and a processor, wherein the processor acquires timing information indicating a timing at which a light source irradiates a subject with infrared light, acquires an infrared image obtained by imaging a scene including the subject based on infrared light in accordance with the timing indicated by the timing information and stored in the memory, acquires a visible light image of substantially the same scene as the infrared image, obtained by imaging based on visible light from substantially the same viewpoint and at substantially the same imaging time as the infrared image and stored in the memory, estimates depth information indicating a depth at each position in the infrared image, and corrects the depth at each position in a dust area where dust is reflected in the infrared image, as indicated by the depth information, by inputting the infrared image, the visible light image, and the depth information into a learning model.

[0035] Thus, if the learning model is trained in advance to output the correct depth at each position in the dust region of the infrared image in response to the input of the infrared image, visible light image, and depth information, the depth information estimated from the infrared image can be appropriately corrected. In other words, the depth at each position in the dust region indicated by the depth information can be appropriately corrected without detecting the dust region.

[0036] A depth acquisition device according to another aspect of the present disclosure includes a memory and a processor, the processor acquires an infrared image stored in the memory, where the infrared image is obtained by imaging based on infrared light, acquires a visible light image stored in the memory, where the visible light image is obtained by imaging based on visible light at substantially the same viewpoint and imaging time as the infrared image, detects an area in which dust is reflected as a dust area from the infrared image, and estimates a depth of the dust area based on the visible light image. When the visible light image and the infrared image are each divided into a dust area and other areas, the depth of the dust area is estimated based on the visible light image, and the depth of the other areas is estimated based on the infrared image.

[0037] This makes it possible to properly obtain the depth of the dust region while eliminating the influence of noise, which is dust, in the same way as the depth obtaining device according to the above aspect of the present disclosure.

[0038] These comprehensive or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized by any combination of the system, the method, the integrated circuit, the computer program, or the recording medium. The recording medium may be a non-transitory recording medium.

[0039] Hereinafter, the embodiment will be specifically described with reference to the drawings.

[0040] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component arrangement and connection forms, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in an independent claim showing a top concept are described as optional components.

[0041] In addition, each drawing is a schematic diagram and is not necessarily a precise illustration. In addition, in each drawing, the same components are denoted by the same reference numerals.

[0042] (Embodiment) [Hardware configuration] 1 is a block diagram showing a hardware configuration of a depth acquisition device 1 according to an embodiment. The depth acquisition device 1 in this embodiment has a hardware configuration capable of acquiring an image based on infrared light (or near-infrared light) and an image based on visible light by capturing images of substantially the same scene from substantially the same viewpoint and at substantially the same capturing time. Note that "substantially the same" means that the images are the same to the extent that the effects of the present disclosure can be achieved.

[0043] As shown in FIG. 1, the depth acquisition device 1 includes a light source 10, a solid-state imaging element 20, a processing circuit 30, a diffuser 50, a lens 60, and a band-pass filter .

[0044] The light source 10 emits illumination light. More specifically, the light source 10 emits illumination light to irradiate a subject at a timing indicated by a light emission signal generated by the processing circuit 30.

[0045] The light source 10 includes, for example, a capacitor, a driving circuit, and a light emitting element, and emits light by driving the light emitting element with electrical energy stored in the capacitor. The light emitting element is realized by, for example, a laser diode, a light emitting diode, etc. The light source 10 may include one type of light emitting element, or may include multiple types of light emitting elements according to the purpose.

[0046] In the following, the light emitting element is, for example, a laser diode that emits near-infrared light, or a light emitting diode that emits near-infrared light. However, the irradiation light irradiated by the light source 10 does not need to be limited to near-infrared light. The irradiation light irradiated by the light source 10 may be, for example, infrared light (also called infrared light) in a frequency band other than near-infrared light. Hereinafter, in this embodiment, the irradiation light irradiated by the light source 10 is described as infrared light, but the infrared light may be near-infrared light or infrared light in a frequency band other than near-infrared light.

[0047] The solid-state imaging element 20 captures an image of a subject and outputs an imaging signal indicating the amount of exposure. More specifically, the solid-state imaging element 20 performs exposure at a timing indicated by an exposure signal generated by the processing circuit 30, and outputs an imaging signal indicating the amount of exposure.

[0048] The solid-state imaging element 20 has a pixel array in which first pixels that capture an image using light reflected by a subject from irradiation light and second pixels that capture an image of the subject are arranged in an array. The solid-state imaging element 20 may have a logic function such as a cover glass, an AD converter, etc., as necessary.

[0049] In the following description, the reflected light is assumed to be infrared light, like the irradiated light; however, the reflected light does not have to be limited to infrared light, so long as it is light that is the irradiated light reflected by the subject.

[0050] FIG. 2 is a schematic diagram showing a pixel array 2 of the solid-state imaging device 20. As shown in FIG.

[0051] As shown in FIG. 2, the pixel array 2 is configured such that first pixels 21 (IR pixels) that capture images using light reflected by a subject from irradiated light and second pixels 22 (BW pixels) that capture images of the subject are arranged in an array in alternating columns.

[0052] 2, in the pixel array 2, the second pixels 22 and the first pixels 21 are arranged adjacent to each other in the row direction and arranged in a stripe pattern in the row direction, but the present invention is not limited to this and the pixels may be arranged every several rows (as an example, every two rows). That is, the first row in which the second pixels 22 are arranged adjacent to each other in the row direction and the second row in which the first pixels 21 are arranged adjacent to each other in the row direction may be arranged alternately every M rows (M is a natural number). Furthermore, the first row in which the second pixels 22 are arranged adjacent to each other in the row direction and the second row in which the first pixels 21 are arranged adjacent to each other in the row direction may be arranged at different intervals (the first row is alternately N rows and the second row is alternately L rows (N and L are different natural numbers)).

[0053] The first pixel 21 is realized, for example, by an infrared pixel having sensitivity to infrared light, which is reflected light, and the second pixel 22 is realized, for example, by a visible pixel having sensitivity to visible light.

[0054] The infrared light pixel includes, for example, an optical filter (also called an IR filter) that transmits only infrared light, a microlens, a light receiving element as a photoelectric conversion unit, and an accumulation unit that accumulates electric charges generated by the light receiving element. Therefore, an image showing the luminance of infrared light is expressed by imaging signals output from a plurality of infrared light pixels (i.e., first pixels 21) included in the pixel array 2. Hereinafter, this infrared light image is also called an IR image or an infrared image.

[0055] Also, the visible light pixel includes, for example, an optical filter (also called a BW filter) that transmits only visible light, a microlens, a light receiving element as a photoelectric conversion unit, and an accumulation unit that accumulates the electric charge converted by the light receiving element. Therefore, the visible light pixel, i.e., the second pixel 22, outputs an image signal indicating the luminance and color difference. That is, a color image indicating the luminance and color difference of visible light is expressed by the image signal output from the second pixels 22 included in the pixel array 2. Note that the optical filter of the visible light pixel may transmit both visible light and infrared light, or may transmit only light of a specific wavelength band such as red (R), green (G), or blue (B) of visible light.

[0056] Also, the visible light pixels may detect only the luminance of visible light. In this case, the visible light pixels, i.e., the second pixels 22, output imaging signals indicating the luminance. Therefore, a black and white image indicating the luminance of visible light, in other words, a monochrome image, is expressed by the imaging signals output from the second pixels 22 included in the pixel array 2. Hereinafter, this monochrome image is also referred to as a BW image. The above-mentioned color image and BW image are also collectively referred to as a visible light image.

[0057] Returning to FIG. 1 again, the description of the depth acquisition device 1 will continue.

[0058] The processing circuit 30 uses the imaging signal output by the solid-state imaging device 20 to calculate subject information relating to the subject.

[0059] The processing circuit 30 is configured, for example, by an arithmetic processing device such as a microcomputer. The microcomputer includes a processor (microprocessor), memory, etc., and generates a light emission signal and an exposure signal by executing a driving program stored in the memory by the processor. The processing circuit 30 may use an FPGA or an ISP, etc., and may be configured by one piece of hardware or multiple pieces of hardware.

[0060] The processing circuit 30 calculates the distance to the subject, for example, by a TOF distance measuring method using an imaging signal from the first pixel 21 of the solid-state imaging device 20.

[0061] Hereinafter, calculation of the distance to the subject by the TOF distance measurement method performed by the processing circuit 30 will be described with reference to the drawings.

[0062] FIG. 3 is a timing diagram showing the relationship between the light emission timing of the light-emitting element of the light source 10 and the exposure timing of the first pixel 21 of the solid-state imaging element 20 when the processing circuit 30 calculates the distance to the subject using the TOF distance measurement method.

[0063] 3, Tp is the light emission period during which the light emitting element of the light source 10 emits the irradiation light, and Td is the delay time from when the light emitting element of the light source 10 emits the irradiation light to when the reflected light of the irradiation light reflected by the subject returns to the solid-state imaging element 20. The first exposure period has the same timing as the light emission period during which the light source 10 emits the irradiation light, and the second exposure period has the timing from the end of the first exposure period until the light emission period Tp has elapsed.

[0064] In Figure 3, q1 indicates the total amount of exposure to reflected light at the first pixel 21 of the solid-state imaging element 20 during the first exposure period, and q2 indicates the total amount of exposure to reflected light at the first pixel 21 of the solid-state imaging element 20 during the second exposure period.

[0065] By emitting light by the light-emitting element of the light source 10 and exposing the first pixel 21 of the solid-state imaging element 20 at the timing shown in Figure 3, the distance d to the subject can be expressed by the following (Equation 1), where c is the speed of light.

[0066] d=c×Tp / 2×q2 / (q1+q2) (Formula 1)

[0067] Therefore, by utilizing (Equation 1), the processing circuit 30 can calculate the distance to the subject using the imaging signal from the first pixel 21 of the solid-state imaging device 20.

[0068] Furthermore, the first pixels 21 of the solid-state imaging device 20 may be exposed for only a third exposure period Tp after the first and second exposure periods are completed. The first pixels 21 can detect noise other than reflected light based on the amount of exposure obtained in the third exposure period Tp. In other words, the processing circuit 30 can more accurately calculate the distance d to the subject by removing noise from the amount of exposure q1 in the first exposure period and the amount of exposure q2 in the second exposure period in the above formula 1.

[0069] Returning to FIG. 1 again, the description of the depth acquisition device 1 will continue.

[0070] The processing circuit 30 may use, for example, an imaging signal from the second pixel 22 of the solid-state imaging device 20 to detect the subject and calculate the distance to the subject.

[0071] That is, the processing circuit 30 may detect the subject and calculate the distance to the subject based on the visible light image captured by the multiple second pixels 22 of the solid-state imaging device 20. Here, the subject may be detected, for example, by determining the shape by pattern recognition using edge detection of the singular points of the subject, or by processing such as deep learning using a learning model learned in advance. The distance to the subject may be calculated using world coordinate transformation. Of course, the subject may be detected by multimodal learning processing using not only the visible light image but also the luminance and distance information of the infrared light captured by the first pixels 21.

[0072] The processing circuit 30 generates a light emission signal indicating the timing of light emission and an exposure signal indicating the timing of exposure. The processing circuit 30 then outputs the generated light emission signal to the light source 10 and outputs the generated exposure signal to the solid-state imaging element 20.

[0073] The processing circuit 30 may, for example, generate and output a light emission signal to cause the light source 10 to emit light at a predetermined cycle, and generate and output an exposure signal to cause the solid-state imaging element 20 to be exposed at a predetermined cycle, thereby causing the depth acquisition device 1 to realize continuous imaging at a predetermined frame rate. The processing circuit 30 may also include, for example, a processor (microprocessor), memory, etc., and generate the light emission signal and the exposure signal by the processor executing a driving program stored in the memory.

[0074] The diffusion plate 50 adjusts the intensity distribution and angle of the irradiated light. In adjusting the intensity distribution, the diffusion plate 50 makes the intensity distribution of the irradiated light from the light source 10 uniform. Note that, although the depth acquisition device 1 includes the diffusion plate 50 in the example shown in FIG. 1, the diffusion plate 50 may not be included.

[0075] The lens 60 is an optical lens that focuses light entering from outside the depth acquisition device 1 onto the surface of the pixel array 2 of the solid-state imaging element 20.

[0076] The bandpass filter 70 is an optical filter that transmits infrared light, which is reflected light, and visible light. In the example shown in FIG. 1, the depth acquisition device 1 includes the bandpass filter 70, but the bandpass filter 70 may not be included.

[0077] The depth acquisition device 1 having the above configuration is mounted on a transport device and used. For example, the depth acquisition device 1 is mounted on a vehicle that travels on a road surface and used. Note that the transport device on which the depth acquisition device 1 is mounted is not necessarily limited to a vehicle. The depth acquisition device 1 may be mounted on a transport device other than a vehicle, such as a motorcycle, a boat, an airplane, etc.

[0078] [Outline of the depth acquisition device] The depth acquisition device 1 in this embodiment acquires an IR image and a BW image by imaging substantially the same scene from substantially the same viewpoint and at the same time using the hardware configuration shown in Fig. 1. The depth acquisition device 1 then corrects the depth at each position in the IR image obtained from the IR image using the BW image. Specifically, if the IR image contains an area in which dust is reflected (hereinafter referred to as a dust area), the depth acquisition device 1 corrects the depth at each position in the dust area obtained from the IR image using an image in the area of ​​the BW image corresponding to the dust area. This makes it possible to properly acquire the depth of the dust area without the influence of noise, which is dust.

[0079] FIG. 4 is a block diagram showing an example of a functional configuration of the depth acquisition device 1. As shown in FIG.

[0080] The depth acquisition device 1 includes a light source 101, an IR camera 102, a BW camera 103, a depth estimation unit 111, and a dust detection unit 112.

[0081] The light source 101 may be composed of the light source 10 and the diffusion plate 50 shown in FIG.

[0082] 1, a lens 60, and a bandpass filter 70. Such an IR camera 102 captures an IR image by capturing an image of a scene including a subject based on infrared light in response to the timing at which a light source 101 irradiates the subject with infrared light.

[0083] The BW camera 103 may be composed of a plurality of second pixels 22 of the solid-state imaging element 20 shown in Fig. 1, a lens 60, and a bandpass filter 70. Such a BW camera 103 captures an image based on visible light of a scene that is substantially the same as the infrared image, and obtains a visible light image (specifically, a BW image) by capturing an image from substantially the same viewpoint and at substantially the same capturing time as the infrared image.

[0084] The depth estimation unit 111 and the dust detection unit 112 may be realized as a function of the processing circuit 30 shown in FIG.

[0085] The dust detection unit 112 detects a dust region from the IR image based on the IR image obtained by imaging with the IR camera 102 and the BW image obtained by imaging with the BW camera 103. In other words, the dust detection unit 112 divides the IR image obtained by imaging into a dust region where dust is reflected and a non-dust region where no dust is reflected.

[0086] When fine particles such as dust are present near the IR camera 102, the dust appears as large noise in the IR image. The dust region in this embodiment is a high-luminance region in which dust is displayed. For example, when dust is present near the depth acquisition device 1, infrared light is irradiated onto the dust from the light source 101, and reflected by the dust is received by the solid-state imaging element 20 while maintaining a high luminance. Therefore, in the IR image, the luminance of each pixel in the region in which dust is displayed, i.e., the dust region, is high.

[0087] The depth estimation unit 111 estimates the depth at each position in the IR image including the dust region detected by the dust detection unit 112. Specifically, the depth estimation unit 111 acquires an IR image captured by the IR camera 102 in accordance with the timing of irradiation of the subject with infrared light by the light source 101, and estimates the depth at each position in the IR image based on the IR image. Furthermore, the depth estimation unit 111 corrects the depth at each position estimated in the dust region detected by the dust detection unit 112 based on the BW image. That is, the depth estimation unit 111 estimates the depth of the dust region based on the IR image, the BW image, and the dust region.

[0088] FIG. 5 is a block diagram showing another example of the functional configuration of the depth acquisition device 1. As shown in FIG.

[0089] The depth acquisition device 1 may include a memory 200 and a processor 110.

[0090] 5, the processor 110 may include not only the depth estimation unit 111 and the dust detection unit 112, but also a light emission timing acquisition unit 113, an IR image acquisition unit 114, and a BW image acquisition unit 115. These components are realized as functions of the processor 110.

[0091] The light emission timing acquisition unit 113 acquires timing information indicating the timing at which the light source 101 irradiates the subject with infrared light. That is, the light emission timing acquisition unit 113 outputs the light emission signal shown in Fig. 1 to the light source 101, and acquires information indicating the output timing as the above-mentioned timing information.

[0092] The IR image acquisition unit 114 acquires an IR image that is obtained by capturing an image of a scene including a subject based on infrared light in accordance with the timing indicated by the timing information and that is stored in the memory 200 .

[0093] The BW image acquisition unit 115 acquires a BW image that is captured based on visible light of substantially the same scene as the above-mentioned IR image, obtained by capturing an image from substantially the same viewpoint and at substantially the same capturing time as the IR image, and is stored in the memory 200.

[0094] As described above, the dust detection unit 112 detects a dust region from an IR image, and the depth estimation unit 111 estimates the depth based on the IR image, the BW image, and the dust region.

[0095] The depth acquisition device 1 in this embodiment may be configured with the processor 110 and the memory 200 without including the light source 101, the IR camera 102, and the BW camera 103.

[0096] FIG. 6 is a flowchart showing the overall processing operation of the depth acquisition device 1.

[0097] (Step S11) First, the light source 101 emits light to irradiate the subject with infrared light.

[0098] (Step S12) Next, the IR camera 102 acquires an IR image. That is, the IR camera 102 captures an image of a scene including a subject illuminated with infrared light by the light source 101. In this way, the IR camera 102 acquires an IR image based on the infrared light reflected from the subject. Specifically, the IR camera 102 acquires an IR image obtained by the respective timings and exposure amounts of the first exposure period, the second exposure period, and the third exposure period shown in FIG.

[0099] (Step S13) Next, the BW camera 103 acquires a BW image. That is, the BW camera 103 acquires a BW image corresponding to the IR image acquired in step S12, that is, a BW image of the same scene, the same viewpoint, and the same imaging time as the IR image.

[0100] (Step S14) Then, the dust detection unit 112 detects a dust region from the IR image acquired in step S12.

[0101] (Step S15) Next, the depth estimation unit 111 estimates the depth of the dust region based on the IR image acquired in step S12, the BW image acquired in step S13, and the dust region detected in step S14.

[0102] FIG. 7 is a flowchart showing the overall processing operation by the processor 110 of the depth acquisition device 1. As shown in FIG.

[0103] (Step S21) First, light emission timing acquisition section 113 of processor 110 acquires timing information indicating the timing at which light source 101 irradiates the subject with infrared light.

[0104] (Step S22) Next, the IR image acquisition unit 114 acquires an IR image from the IR camera 102 that has captured an image in accordance with the timing indicated by the timing information acquired in step S21. For example, the IR image acquisition unit 114 outputs an exposure signal to the IR camera 102 at the timing when the light emission signal shown in FIG. 1 is output from the light emission timing acquisition unit 113. This causes the IR image acquisition unit 114 to start capturing an image in the IR camera 102, and acquires an IR image obtained by the capturing from the IR camera 102. At this time, the IR image acquisition unit 114 may acquire the IR image from the IR camera 102 via the memory 200, or may acquire the IR image directly from the IR camera 102.

[0105] (Step S23) Next, the BW image acquisition unit 115 acquires a BW image corresponding to the IR image acquired in step S22 from the BW camera 103. At this time, the BW image acquisition unit 115 may acquire the BW image from the BW camera 103 via the memory 200, or may acquire the BW image directly from the BW camera 103.

[0106] (Step S24) Then, the dust detection unit 112 detects a dust region from the IR image.

[0107] (Step S25) Next, the depth estimation unit 111 estimates the depth of the dust region based on the IR image acquired in step S22, the BW image acquired in step S23, and the dust region detected in step S24. This allows calculation of depth information indicating at least the depth of the dust region. Note that at this time, the depth estimation unit 111 may estimate the depth of the entire IR image, not just the dust region, and calculate depth information indicating the estimation result.

[0108] Specifically, the depth estimation unit 111 in this embodiment estimates the depth at each position in the IR image acquired in step S22 from the IR image. Then, the depth estimation unit 111 corrects the depth at each position in the dust region using the BW image. Note that each position may be the position of each of a plurality of pixels, or may be the position of a block consisting of a plurality of pixels.

[0109] In the depth acquisition device 1 according to the present embodiment, a dust region is detected from an IR image, and the depth of the dust region is estimated based on not only the IR image but also the BW image, so that the depth of the dust region can be appropriately acquired. That is, the scenes to be captured are substantially the same in the IR image and the BW image, and the viewpoint and the capture time are also substantially the same. Therefore, the IR image and the BW image have a high correlation. Also, even if dust is captured in the IR image, there is a high possibility that the dust will not be captured in the BW image. Therefore, the information missing in the dust region can be compensated for from the region in the BW image corresponding to the dust region (i.e., the corresponding region). As a result, the depth of the dust region can be appropriately acquired.

[0110] [Specific functional configuration of the depth acquisition device] FIG. 8 is a block diagram showing a specific functional configuration of the processor 110 of the depth acquisition device 1. As shown in FIG.

[0111] The processor 110 includes a first depth estimation unit 111a, a second depth estimation unit 111b, a dust detection unit 112, a high brightness area detection unit 116, an FOE detection unit 117, and an output unit 118. The first depth estimation unit 111a and the second depth estimation unit 111b correspond to the depth estimation unit 111 shown in Fig. 5. The processor 110 may also include the light emission timing acquisition unit 113, the IR image acquisition unit 114, and the BW image acquisition unit 115 described above.

[0112] The high-luminance region detection unit 116 detects a region in the IR image having a luminance equal to or greater than a first threshold as a high-luminance region.

[0113] The FOE detection unit 117 detects the FOE (Focus of Expansion) in the BW image. The FOE is also called a vanishing point. When the IR camera 102 moves in parallel and the subject is stationary, it is known that the optical flow, which is the apparent movement on the screen, intersects at one point. That one point is the FOE.

[0114] The dust detection unit 112 determines whether or not each of at least one high-luminance region in the IR image is a dust region. Here, the inventors have discovered that each dust particle shown in the IR image is formed long along a straight line or a circular arc intersecting the FOE, as if it is being blown out of the FOE, or is arranged along the straight line or the circular arc. When the IR camera 120 is installed on a moving body such as an automobile, the movement of the dust particle is sufficiently small compared to the movement of the IR camera 120, and it can be assumed that the dust particle is stationary. Therefore, the dust particle is shown in the IR image as if it is being blown out of the FOE.

[0115] In addition, in the depth acquisition device 1 of this embodiment, from the viewpoint of noise removal, each of the IR image and the BW image is an image obtained by imaging in which exposure and shading are repeated multiple times. Therefore, the same dust is projected at multiple locations in one frame by exposure at different timings within a frame period. Since the dust moves as if blowing out of the FOE on the image, the dust areas at each of the multiple locations and the FOE are arranged on a straight line. Furthermore, if the IR camera 120 that images the same dust moves slowly, multiple dust areas generated by multiple exposures in one frame period will overlap in the IR image. As a result, one dust area with a tail-like shape is formed. In this case, the FOE is present in the direction of the tail as seen from the dust area. The dust detection unit 112 detects the dust area by utilizing these properties.

[0116] That is, the dust detection unit 112 in this embodiment detects a high-luminance region having a luminance equal to or greater than a first threshold in an IR image as a dust region when the high-luminance region satisfies a first condition. Specifically, the first condition is that the center of gravity of the high-luminance region is located on a line or an arc intersecting the centers of gravity of at least two other high-luminance regions different from the high-luminance region in the IR image and the FOE (Focus of Expansion) of the IR image or the BW image. Alternatively, the first condition is that the main axis of the high-luminance region or an extension of the main axis intersects with the FOE of the IR image or the BW image. Note that this main axis is the axis of the tail when the high-luminance region has a tail shape. This allows the dust region to be detected with high accuracy.

[0117] Furthermore, the dust detection unit 112 may detect a dust region by using the property that a dust region is observed in an IR image but not in a BW image. That is, the dust detection unit 112 may detect a high-luminance region as a dust region when the high-luminance region further satisfies a second condition. For example, the second condition is a condition that the luminance of a position in the BW image corresponding to the center of gravity of the high-luminance region in the IR image is less than a second threshold. Or, the second condition is a condition that the correlation coefficient between the luminance in the high-luminance region of the IR image and the luminance in the region of the BW image corresponding to the high-luminance region is less than a third threshold. Note that the region of the BW image corresponding to the high-luminance region of the IR image is in the same spatial position as the high-luminance region of the IR image and has the same shape and size as the high-luminance region of the IR image. This allows the dust region to be detected with even higher accuracy.

[0118] The first depth estimation section 111a and the second depth estimation section 111b have the function of the depth estimation section 111 described above.

[0119] The first depth estimation unit 111a estimates the depth at each position in the IR image based on the IR image acquired in accordance with the timing of irradiation of infrared light by the light source 101. The first depth estimation unit 111a outputs information indicating the estimated depth at each position in the IR image as first depth information. In other words, the first depth estimation unit 111a estimates the first depth information indicating the depth at each position in the IR image.

[0120] The second depth estimation unit 111b corrects the first depth information based on the BW image and the dust region in the IR image. As a result, the depth of the dust region among the depths at each position in the IR image indicated by the first depth information is corrected. The second depth estimation unit 111b outputs information indicating the corrected depth at each position in the dust region as the second depth information. In other words, the second depth estimation unit 111b estimates the second depth information indicating the corrected depth at each position in the dust region by correcting the depth at each position in the dust region indicated by the first depth information based on the BW image.

[0121] The output unit 118 replaces the depth at each position in the dust region indicated by the first depth information with the corrected depth at each position in the dust region indicated by the second depth information. This generates third depth information including the depth at each position outside the dust region of the IR image indicated by the first depth information and the corrected depth at each position in the dust region of the IR image indicated by the second depth information. The output unit 118 outputs the third depth information.

[0122] As a result, the third depth information indicates a depth outside the dust region of the IR image obtained from the IR image, and indicates a depth obtained from the IR image and corrected based on the BW image as a depth of the dust region of the IR image. Therefore, in this embodiment, even if the IR image has a dust region, the entire depth of the IR image can be appropriately estimated.

[0123] Figure 9A shows an example of an IR image, and Figure 9B shows an example of a BW image.

[0124] As shown in Fig. 9B, the BW image shows a scene captured by, for example, a BW camera 103 attached to a vehicle, depicting the surroundings of the road on which the vehicle is traveling moving away from the BW camera 103. On this road, for example, dust is being stirred up by the vehicle traveling. Then, when the IR camera 102 captures the same scene as that shown in Fig. 9B from the same viewpoint and at the same time as the BW camera 103, the IR image shown in Fig. 9A is acquired.

[0125] As shown in FIG. 9A, the IR image acquired as described above has high brightness areas. These areas include areas where dust is reflected, that is, dust areas. For example, the dust areas are confirmed to exist in the center and right parts of the IR image. On the other hand, dust is not reflected in the BW image. This is because, in capturing an IR image, infrared light irradiated from the light source 101 is diffusely reflected by dust in the vicinity of the IR camera 102 and the BW camera 103, but the effect of the diffuse reflection is small in capturing a BW image. Therefore, the dust detection unit 112 detects the dust area by utilizing the property that the dust area is observed in the IR image but not in the BW image.

[0126] FIG. 10 shows an example of a binarized image obtained by binarizing an IR image.

[0127] The high-luminance area detection unit 116 detects an area having a luminance equal to or greater than a first threshold as a high-luminance area in the IR image shown in Fig. 9A. That is, the high-luminance area detection unit 116 binarizes the luminance at each position (i.e., each pixel) in the IR image. As a result, for example, as shown in Fig. 10, a binarized image consisting of a white area and a black area (hatched area in Fig. 10) is generated.

[0128] FIG. 11 shows an example of a candidate dust region in an IR image.

[0129] The dust detection unit 112 first detects the center of gravity of a high-luminance region, which is a white region, in the binarized image. Such detection of the center of gravity is performed for each of all high-luminance regions in the binarized image. Next, for each of all high-luminance regions in the binarized image, the dust detection unit 112 determines whether or not the luminance of a position in the BW image corresponding to the center of gravity of the high-luminance region (hereinafter referred to as the center-of-gravity corresponding position) is less than a second threshold value. That is, the dust detection unit 112 determines whether or not the high-luminance region satisfies the above-mentioned second condition. As a result, the dust detection unit 112 determines the high-luminance region corresponding to the center-of-gravity corresponding position, whose luminance is determined to be less than the second threshold value, as a candidate for a dust region. For example, as shown in FIG. 11, the dust detection unit 112 detects each of five high-luminance regions A to E as a candidate for a dust region. That is, the IR image or the binarized image is divided into five dust candidate regions A to E and a non-dust region that is not a dust region.

[0130] FIG. 12 shows an example of FOE detected in a BW image.

[0131] The FOE detection unit 117 detects FOE by detecting optical flows from multiple frames of BW images including the BW image shown in FIG. 9B, and finding intersections of the optical flows by robust estimation such as RANSAC (Random Sample Consensus).

[0132] FIG. 13 shows an example of the principal axes detected for the dust candidate regions A to E.

[0133] The dust detection unit 112 detects a principal axis for each of the detected dust candidate regions A to E as shown in Fig. 11. Specifically, the dust detection unit 112 detects the first principal component axis as the principal axis of the dust candidate region by performing principal component analysis on each pixel of the dust candidate region in the IR image or binarized image.

[0134] FIG. 14 shows an example of dust and non-dust regions.

[0135] The dust detection unit 112 judges whether or not the main axis or the extension of the main axis of each of the dust candidate regions A to E in the IR image or the binarized image intersects with the FOE detected by the FOE detection unit 117. That is, the dust detection unit 112 judges whether or not the dust candidate region satisfies the above-mentioned first condition. Then, the dust detection unit 112 detects a dust candidate region having a main axis or an extension that intersects with the FOE as a dust region, and detects a dust candidate region having a main axis or an extension that does not intersect with the FOE as a non-dust region. For example, as shown in FIG. 14, the extension of the main axis of each of the dust candidate regions B, C, and E intersects with the FOE, and the main axis and the extension of each of the dust candidate regions A and D do not intersect with the FOE. Therefore, the dust detection unit 112 detects the dust candidate regions B, C, and E as dust regions, and detects the dust candidate regions A and D as non-dust regions.

[0136] Fig. 15A shows another example of an IR image, and Fig. 15B shows another example of a BW image.

[0137] The BW image shown in Fig. 15B is an image obtained by capturing an image at a different timing from the BW image shown in Fig. 9B. In the BW image shown in Fig. 15B, as in the example shown in Fig. 9B, a scene of the surroundings of the road on which the vehicle is traveling approaching the BW camera 103 is captured by the BW camera 103 attached to the vehicle. In addition, for example, dust is stirred up on the road by the traveling vehicle. Therefore, when the IR camera 102 captures the same scene as the scene shown in Fig. 15B from the same viewpoint and at the same time as the BW camera 103, the IR image shown in Fig. 15A is acquired.

[0138] In the IR image acquired in this way, there are areas of high brightness, as shown in Fig. 15A. These areas include areas where dust is reflected, that is, dust areas. For example, it is confirmed that the dust area exists in the left part of the IR image. On the other hand, as in the example shown in Fig. 9B, no dust is reflected in the BW image.

[0139] FIG. 16 shows an example of a binarized image obtained by binarizing an IR image.

[0140] The high-luminance area detection unit 116 detects an area having a luminance equal to or greater than a first threshold as a high-luminance area in the IR image shown in Fig. 15A. That is, the high-luminance area detection unit 116 binarizes the luminance at each position (i.e., each pixel) in the IR image. As a result, for example, as shown in Fig. 16, a binarized image consisting of a white area and a black area (hatched area in Fig. 16) is generated.

[0141] FIG. 17 shows an example of a candidate dust region in an IR image.

[0142] The dust detection unit 112 first detects the center of gravity of a high-luminance region, which is a white region in the binarized image. Such detection of the center of gravity is performed for each of all high-luminance regions in the binarized image. Next, for each of all high-luminance regions in the binarized image, the dust detection unit 112 determines whether the luminance of a position in the BW image corresponding to the center of gravity of the high-luminance region (i.e., the center-of-gravity corresponding position) is less than the second threshold value. That is, the dust detection unit 112 determines whether the high-luminance region satisfies the above-mentioned second condition. As a result, the dust detection unit 112 determines the high-luminance region corresponding to the center-of-gravity corresponding position, whose luminance is determined to be less than the second threshold value, as a candidate for the dust region. For example, as shown in FIG. 17, the dust detection unit 112 detects a region group A consisting of a plurality of high-luminance regions, a high-luminance region B, a region group C consisting of a plurality of high-luminance regions, and a high-luminance region D as candidates for the dust region. That is, the IR image or the binarized image is divided into dust candidate regions and non-dust regions that are not dust regions. The dust candidate regions consist of the regions included in region group A, region B, the regions included in region group C, and region D.

[0143] FIG. 18 shows an example of FOE detected in a BW image.

[0144] The FOE detection unit 117 detects FOE by detecting optical flows from multiple frames of BW images including the BW image shown in FIG. 15B, and finding intersections of the optical flows by robust estimation such as RANSAC (Random Sample Consensus).

[0145] FIG. 19 shows an example of the arrangement of each dust candidate region.

[0146] For example, when the vehicle on which the IR camera 102 is mounted is moving at a high speed, the same dust is projected at multiple locations in one frame of an IR image by exposure at different times within a frame period, as described above. Since the dust moves as if it is blown out from the FOE, the dust areas at each of the multiple locations and the FOE are arranged on a straight line. Furthermore, when a wide-angle lens or a fisheye lens is used for the lens of the IR camera 102, the lens distortion is large. When the effect of the lens distortion is large, the optical flow, which is the apparent movement on the screen, does not intersect at a single point, and therefore each dust area and the FOE do not exist on a straight line. In this way, when the lens distortion is large, each dust area and the FOE are arranged on an arc due to the effect of the lens distortion.

[0147] In the example shown in FIG. 19, the dust detection unit 112 judges whether each dust candidate region in the region group A, the dust candidate region B, each dust candidate region in the region group C, and the dust candidate region D detected as shown in FIG. 17 are arranged on an arc together with the FOE. That is, when making a judgment for one dust candidate region, the dust detection unit 112 judges whether the center of gravity of the dust candidate region to be judged is arranged on an arc intersecting with the center of gravity of at least two other dust candidate regions different from the dust candidate region to be judged and the FOE of the BW image. That is, the dust detection unit 112 judges whether the high brightness region that is the dust candidate region satisfies the above-mentioned first condition. If the center of gravity of the dust candidate region to be judged is arranged on the arc, the dust detection unit 112 detects the dust candidate region as a dust region. Conversely, if the center of gravity of the dust candidate region to be judged is not arranged on the arc, the dust detection unit 112 detects the dust candidate region as a non-dust region.

[0148] 19, since each of the multiple dust candidate regions included in area group A and the FOE are arranged on an arc, the dust detection unit 112 detects each of the multiple dust candidate regions included in area group A as a dust region. Similarly, since each of the multiple dust candidate regions included in area group B and the FOE are arranged on an arc, the dust detection unit 112 also detects each of the multiple dust candidate regions included in area group B as a dust region. On the other hand, each of dust candidate regions B and D is not arranged on an arc that intersects with at least two other dust candidate regions and the FOE, so the dust detection unit 112 detects each of dust candidate regions B and D as a non-dust region.

[0149] In addition, when the lens distortion is large, the dust detection unit 112 may perform a distortion correction process on the captured BW image and IR image. For example, the dust detection unit 112 may perform the distortion correction process by using a camera calibration method such as the non-patent document (R. Tsai, “A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses”, IEEE Journal on Robotics and Automation, Vol. 3, Iss. 4, pp. 323 - 344, 1987.). In this case, the dust detection unit 112 detects the dust candidate region to be determined as a dust region if the dust candidate region to be determined is located on a straight line intersecting at least two other dust candidate regions and the FOE in the IR image or its binarized image after the distortion correction process.

[0150] FIG. 20 shows a simulation result of the depth acquisition device 1.

[0151] The depth acquisition device 1 acquires a BW image shown in (a) of Fig. 20 by capturing images using the BW camera 103, and further acquires an IR image shown in (b) of Fig. 20 by capturing images using the IR camera 102. The BW image and the IR image are images obtained by capturing images of the same scene from the same viewpoint and at the same time. In the example shown in (b) of Fig. 20, some dust areas exist in the IR image.

[0152] The first depth estimation unit 111a estimates the depth from the IR image to generate the first depth information shown in (c) of Fig. 20. This first depth information is expressed as a first depth image that indicates the depth at each position in the IR image by brightness. In this first depth image, the depth of the dust region is inappropriately expressed.

[0153] The second depth estimation unit 111b corrects the inappropriate depth in the dust region. Then, the output unit 118 generates third depth information indicating the corrected depth of the dust region and the depth of the non-dust region, as shown in (e) of FIG. 20. This third depth information is also expressed as a third depth image indicating the depth by brightness, similar to the first depth information. Note that the second depth estimation unit 111b may also correct the depth of the non-dust region in the first depth image based on the corresponding region in the BW image.

[0154] In this way, the depth acquisition device 1 in this embodiment can make the third depth image closer to the correct depth image shown in FIG. 20(d) in the entire image including the dust region.

[0155] [Specific processing flow of the depth acquisition device] FIG. 21 is a flowchart showing the overall processing operation of the depth acquisition device 1 shown in FIG.

[0156] (Step S31) First, the high-luminance region detection unit 116 detects high-luminance regions from the IR image.

[0157] (Step S32) The dust detection unit 112 determines whether a high brightness region is a dust candidate region, thereby dividing the IR image into dust candidate regions and non-dust regions.

[0158] (Step S33) The FOE detection unit 117 detects the FOE by using the BW image.

[0159] (Step S34) The dust detection unit 112 detects dust regions based on the dust candidate regions and the FOE, thereby dividing the IR image into dust regions and non-dust regions.

[0160] (Step S35) The first depth estimation unit 111a generates first depth information from the IR image using, for example, TOF.

[0161] (Step S36) The second depth estimation unit 111b generates second depth information indicating the depth of the dust region based on the first depth information of the IR image and the BW image.

[0162] (Step S37) The output unit 118 generates third depth information by replacing the depth of the dust region indicated by the first depth information with the depth indicated by the second depth information.

[0163] FIG. 22 is a flowchart showing an example of detailed processing of steps S31 to S34 in FIG.

[0164] (Step S41) First, the high-luminance region detection unit 116 judges whether the luminance at each position in the IR image is equal to or greater than the first threshold. Here, the first threshold may be about 256 if the IR image is an image with 12-bit gradation, for example. Of course, this first threshold may be a value that changes depending on the environmental conditions or the settings of the IR camera 102. For example, when a dark scene such as at night is captured, the luminance of the entire IR image is low, so the first threshold may be a smaller value than when a bright scene is captured in the daytime. Also, when the exposure time of the IR camera 102 is long, the luminance of the entire IR image is high, so the first threshold may be a larger value than when the exposure time is short.

[0165] (Step S42) If it is determined that the luminance at any position is not equal to or greater than the first threshold value (No in step S41), the high luminance area detection unit 116 determines that no dust is shown in the IR image. In other words, the entire IR image is determined to be a non-dust area.

[0166] (Step S43) On the other hand, when it is determined that the luminance at any position is equal to or greater than the first threshold (Yes in step S41), the high-luminance region detection unit 116 divides the IR image into regions. That is, the high-luminance region detection unit 116 divides the IR image into at least one high-luminance region and a region other than the high-luminance region. For this region division, a luminance-based method such as SuperPixel may be used. Note that the high-luminance region detection unit 116 may perform a filtering process using the size of the region for the region division. For example, if the number of pixels in a high-luminance region is equal to or less than a predetermined number, the high-luminance region detection unit 116 may delete the high-luminance region. That is, even if the high-luminance region detection unit 116 detects a high-luminance region, if the number of pixels in the region is small, the high-luminance region detection unit 116 may reclassify the high-luminance region into a region other than the high-luminance region.

[0167] (Step S44) Next, the dust detection unit 112 detects the center of gravity of at least one high-luminance area detected by the area division in step S43. Specifically, the dust detection unit 112 detects the center of gravity of the high-luminance area by calculating the average values ​​of the X-axis coordinate positions and the Y-axis coordinate positions of multiple pixels included in the high-luminance area.

[0168] (Step S45a) The dust detection unit 112 judges whether the luminance of the position in the BW image corresponding to the center of gravity of the high-luminance area (i.e., the center-of-gravity corresponding position) is less than the second threshold value. That is, the dust detection unit 112 judges whether the high-luminance area satisfies the second condition. If it is judged that the luminance is not less than the second threshold value (No in step S45a), the dust detection unit 112 judges the high-luminance area as a non-dust area (step S42). That is, in this case, it is estimated that a subject with a high light reflectance is captured in each of the high-luminance area of ​​the IR image and the area corresponding to the high-luminance area in the BW image. Therefore, in this case, the high-luminance area is judged as a non-dust area. Note that the second threshold value may be about 20,000 if the BW image is a 12-bit gradation image, for example. Of course, this second threshold value may be a value that changes according to the environmental conditions or the settings of the BW camera 103. For example, when a dark scene such as at night is captured, the luminance of the entire BW image is low, so the second threshold may be a smaller value than when a bright scene is captured in the daytime. Also, when the exposure time of the BW camera 103 is long, the luminance of the entire BW image is high, so the second threshold may be a larger value than when the exposure time is short.

[0169] (Step S46) On the other hand, if the dust detection unit 112 determines that the luminance at the center-of-gravity corresponding position is less than the second threshold value (Yes in step S45a), the FOE detection unit 117 detects the FOE based on the BW image.

[0170] (Step S47a) The dust detection unit 112 judges whether the centers of gravity of the three or more dust candidate regions detected in step S44 and the FOE are arranged on a straight line. That is, the dust detection unit 112 judges whether the dust candidate regions satisfy the above-mentioned first condition. Specifically, the dust detection unit 112 performs fitting of the centers of gravity of the three or more dust candidate regions to a line intersecting the FOE, and judges whether the error (i.e., distance) between the line and each center of gravity is equal to or less than a tolerance. This judges whether the centers of gravity of the three or more dust candidate regions and the FOE are arranged on a straight line. If the error is equal to or less than a tolerance, it is judged that the centers of gravity of the dust candidate regions and the FOE are arranged on a straight line, and if the error is not equal to or less than a tolerance, it is judged that the centers of gravity of the dust candidate regions and the FOE are not arranged on a straight line.

[0171] (Step S50) When the dust detection unit 112 determines that the center of gravity of each dust candidate region and the FOE are arranged on a straight line (Yes in step S47a), it determines that the dust candidate region is a dust region.

[0172] (Step S48) On the other hand, if the dust detection unit 112 determines that the centers of gravity of three or more dust candidate regions and the FOE are not located on a straight line (No in step S47a), it detects the main axis of each dust candidate region.

[0173] (Step S49) Next, the dust detection unit 112 determines whether or not the main axis or an extension of each dust candidate region detected in step S48 intersects with the FOE. In other words, the dust detection unit 112 determines whether or not the dust candidate region satisfies a first condition different from the first condition of step S47a. Here, if the dust detection unit 112 determines that the main axis or an extension of the main axis intersects with the FOE (Yes in step S49), it determines the dust candidate region having that main axis as a dust region (step S50). On the other hand, if the dust detection unit 112 determines that the main axis or an extension of the main axis does not intersect with the FOE (No in step S49), it determines the dust candidate region having that main axis as a non-dust region (step S42).

[0174] In such a method, an IR image and a BW image having substantially the same viewpoint are required. In the depth acquisition device 1 in this embodiment, the filter used for each pixel is set to either an IR filter or a BW filter. That is, as shown in FIG. 2, a first pixel 21 having an IR filter and a second pixel 22 having a BW filter are alternately arranged in the column direction. This makes it possible to acquire an IR image and a BW image having substantially the same viewpoint and at the same time, so that the dust region can be appropriately identified.

[0175] Fig. 23 is a flowchart showing another example of the detailed processes of steps S31 to S in Fig. 21. The flowchart shown in Fig. 23 includes step S47b instead of step S47a among the steps in the flowchart in Fig. 22.

[0176] (Step S47b) For example, if the lens distortion of the IR camera 102 is large, as described above, the dust regions and FOEs are not arranged on a straight line but are arranged on an arc according to the lens distortion.

[0177] Therefore, the dust detection unit 112 may determine whether the centers of gravity of the three or more dust candidate regions detected in step S44 and the FOE are arranged on a circular arc. Specifically, the dust detection unit 112 obtains an approximation curve between the centers of gravity of the three or more dust candidate regions and the FOE, and determines whether the error (i.e., distance) between the approximation curve and each center of gravity is equal to or less than a tolerance. This determines whether the centers of gravity of the three or more dust candidate regions and the FOE are arranged on a circular arc. That is, if the error is equal to or less than a tolerance, it is determined that the centers of gravity of the dust candidate regions and the FOE are arranged on a circular arc, and if the error is not equal to or less than a tolerance, it is determined that the centers of gravity of the dust candidate regions and the FOE are not arranged on a circular arc. Note that the above-mentioned approximation curve is expressed in an order equal to or less than the number of the three or more dust candidate regions to be determined.

[0178] In step S46, FOE detection unit 117 may detect optical flows from multiple IR images instead of the BW image, obtain an intersection of the optical flows by robust estimation such as RANSAC, and detect the intersection as the FOE. FOE detection unit 117 may also detect the movement of IR camera 102 or BW camera 103, and detect the FOE by calculation using the movement and internal parameters of IR camera 102 or BW camera 103.

[0179] Fig. 24 is a flowchart showing an example of a process instead of steps S31 to S34 in Fig. 21. Step S32 in Fig. 21 is omitted in the flowchart shown in Fig. 24. In other words, step S45a in the flowchart in Fig. 22 is omitted in the flowchart shown in Fig. 24.

[0180] That is, in the examples shown in Figs. 21 to 23, the dust detection unit 112 determines whether or not a high brightness region is a dust candidate region by utilizing the property that dust is observed in an IR image but not in a BW image. That is, as shown in step S45a in each of Figs. 22 and 23, the dust detection unit 112 determines whether or not a high brightness region is a dust candidate region based on the brightness of the position corresponding to the center of gravity in the BW image. However, as in the flowchart shown in Fig. 24, the dust detection unit 112 does not have to perform the determination in step S45a. In this case, any high brightness region obtained by the region division in step S43 is treated as a dust candidate region.

[0181] Fig. 25 is a flowchart showing another example of the detailed processes of steps S31 to S in Fig. 21. The flowchart shown in Fig. 25 includes step S45b instead of step S45a among the steps in the flowchart in Fig. 22.

[0182] In order to determine the dust candidate region, the property that dust is observed in the IR image but not in the BW image may be more actively utilized. For example, the correlation coefficient between the IR image and the BW image may be utilized as the property. As described above, if dust exists, the dust will be reflected in the IR image, but not in the BW image. In other words, since the BW image shows a more distant subject, the IR image and the BW image, which are viewed from the same viewpoint, have significantly different images. Therefore, by utilizing the correlation coefficient between the IR image and the BW image to determine whether the subjects shown are the same or not, it is possible to determine whether each high-brightness region is a dust candidate region or not.

[0183] (Step S45b) In step S45b, for each of at least one high-brightness region obtained by the region division in step S43, the dust detection unit 112 calculates a correlation coefficient of brightness between that high-brightness region of the IR image and a region in the BW image corresponding to that high-brightness region (i.e., a corresponding region). The correlation coefficient is found by arranging the brightness of each pixel in the IR image and the BW image in a vector shape for each region, calculating the inner product value, and normalizing it by the number of pixels. In other words, the dust detection unit 112 normalizes the inner product value of the vector consisting of the brightness of each pixel in the high-brightness region of the IR image and the vector consisting of the brightness of each pixel in the corresponding region of the BW image. This calculates the correlation coefficient for that high-brightness region.

[0184] Then, the dust detection unit 112 judges whether the calculated correlation coefficient is less than the third threshold value. That is, the dust detection unit 112 judges whether the high brightness region satisfies the above-mentioned second condition. If the correlation coefficient is not less than the third threshold value (No in step S45b), the same subject is captured in the high brightness region of the IR image and the corresponding region of the BW image, so the correlation coefficient may be high. Therefore, in this case, the dust detection unit 112 determines that the high brightness region is not a dust region (step S42). On the other hand, if the correlation coefficient is less than the third threshold value (Yes in step S45b), different subjects are captured in the high brightness region of the IR image and the corresponding region of the BW image, so the correlation coefficient may be low. Therefore, in this case, the dust detection unit 112 determines that the high brightness region is a dust candidate region (step S50).

[0185] Also, the discrimination between dusty and non-dusty regions may be realized by using a learning process. For example, deep learning or other processes may be used for the learning process. In this case, for learning, an IR image, a BW image, and a ground truth image in which the IR image is divided into dusty and non-dusty regions are prepared in advance. Next, the IR image and the BW image are given as input to a learning model. Then, the learning model is trained so that the output from the learning model for that input matches the ground truth image. For example, the learning model is a neural network. The output from the learning model is an image showing the value "0" or the value "1" for each pixel, where the value "0" indicates that the pixel belongs to a non-dust region and the value "1" indicates that the pixel belongs to a dusty region.

[0186] The dust detection unit 112 distinguishes between dust and non-dust regions by using the learning model that has been trained in advance in this way. That is, the dust detection unit 112 inputs an IR image and a BW image to the learning model. The dust detection unit 112 then distinguishes as non-dust regions areas that include pixels that correspond to the numerical value "0" output from the learning model. Furthermore, the dust detection unit 112 distinguishes as dust regions areas that include pixels that correspond to the numerical value "1" output from the learning model.

[0187] Through the above process, the dust detection unit 112 divides the captured IR image into a dust region where dust is reflected and a non-dust region where no dust is reflected.

[0188] <Depth correction processing> The second depth estimation unit 111b generates second depth information by using the BW image, the first depth information, and the dust region (that is, the discrimination result of the above-mentioned region).

[0189] As described above, noise caused by dust is a phenomenon caused by diffuse reflection of infrared light emitted from the light source 101. Therefore, dust that appears as noise in the IR image is often not shown in the BW image. Therefore, by correcting the first depth information only in the dust region using the BW image, rather than the first depth information obtained from the IR image, it is possible to obtain second depth information that is not affected by dust shown in the IR image.

[0190] A guided filter, which is a type of image correction filter, may be used to obtain the second depth information. The guided filter is disclosed in a non-patent document (Kaiming He, Jian Sun and Xiaoou Tang, “Guided Image Filtering”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 35, Iss. 6, pp.1397-1409,2013.). The guided filter is a filter that corrects a target image by utilizing the correlation between the target image and a reference image. In the guided filter, it is assumed that the reference image I and the target image p are expressed by parameters a and b as shown in the following (Equation 2).

[0191]

number

[0192] Here, q is the output image obtained by correcting the target image p, i is the number of each pixel, and ωk is the peripheral area of ​​pixel k. Furthermore, parameters a and b are expressed by the following (Equation 3).

[0193]

number

[0194] Here, ε is a regularization parameter, and μ and σ are the mean and variance within a block of the reference image, and are calculated by the following (Equation 4).

[0195]

number

[0196] However, in order to suppress the noise contained in the obtained parameters a and b, averaged parameters are used and the output is calculated as shown in the following (Equation 5).

[0197]

number

[0198] In this embodiment, the second depth estimation unit 111b corrects the first depth information (or the first depth image) which is the target image by providing a BW image as a reference image. This generates or acquires the second depth information. To generate such second depth information, an IR image and a BW image having substantially the same viewpoint position are required. In the depth acquisition device 1 of this embodiment, the filter used for each pixel is set to either an IR filter or a BW filter. That is, as shown in FIG. 2, the first pixel 21 having an IR filter and the second pixel 22 having a BW filter are alternately arranged in the column direction. This makes it possible to acquire an IR image and a BW image having substantially the same viewpoint, and therefore to acquire appropriate second depth information.

[0199] Of course, the second depth estimation unit 111b may use a process other than the guided filter. For example, the second depth estimation unit 111b may use a process such as a bilateral filter (Non-Patent Document: C. Tomasi, R. Manduchi, “Bilateral filtering for gray and color images”, IEEE International Conference on Computer Vision (ICCV), pp. 839-846, 1998) or Mutual-Structure for Joint Filtering (Non-Patent Document: Xiaoyong Shen, Chao Zhou, Li Xu and Jiaya Jia, “Mutual-Structure for Joint Filtering”, IEEE International Conference on Computer Vision (ICCV), 2015.).

[0200] As described above, in this embodiment, the first depth information is used for areas determined not to include dust (i.e., non-dust areas), and the second depth information is used for areas that include dust (i.e., dust areas). This makes it possible to obtain more accurate depth information even if dust is included in the IR image.

[0201] (Modification) In the above embodiment, a filter such as a guided filter is used to generate the second depth information, but the second depth information may be generated using a learning model.

[0202] For example, as in the non-patent document (Shuran Song, Fisher Yu, Andy Zeng, Angel X. Chang, Manolis Savva and Thomas Funkhouser, “Semantic Scene Completion from a Single Depth Image”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 190-198, 2017.), a learning process, deep learning, may be used. That is, a learning model may be trained so that when a BW image and first depth information are input, second depth information is output. The above non-patent document proposes a network that, when depth information including a missing area and a color image are input, interpolates the missing area of ​​the depth information. The second depth estimation unit 111b in this embodiment provides an IR image, a BW image, and the first depth information to a network (i.e., a learning model) similar to that non-patent document, and further provides the dust area detected by the dust detection unit 112 as a mask image of the missing area. This makes it possible to acquire second depth information with higher accuracy from the network.

[0203] FIG. 26 is a block diagram showing an example of a functional configuration of the depth acquisition device 1 in this modified example.

[0204] The depth acquisition device 1 in this modification includes the components shown in FIG. 8, and further includes a learning model 104 including, for example, a neural network.

[0205] The second depth estimation unit 111b inputs three types of data, namely, an IR image, a BW image, and the first depth information, to the learning model 104, and generates second depth information by using the dust area as a mask area to be corrected.

[0206] In the learning of the learning model 104, in addition to the IR image, the BW image, and the first depth information, a correct depth image is prepared in advance. Next, the IR image, the BW image, the first depth information, and a mask image specifying the dust region are provided as inputs to the learning model 104. Then, the learning model 104 is trained so that the output from the learning model 104 in response to the input matches the correct depth image. During training, the mask image is provided randomly. The second depth estimation unit 111b uses the learning model 104 that has been trained in advance in this way. That is, the second depth estimation unit 111b can obtain the second depth information output from the learning model 104 by inputting the IR image, the BW image, the first depth information, and the mask image specifying the dust region to the learning model 104.

[0207] In this way, in this modification, the second depth estimation unit 111b estimates depth information indicating the depth at each position in the IR image, and corrects the depth at each position in the dust region indicated by the depth information by inputting the IR image, BW image, dust region, and the depth information into the learning model. Therefore, if the learning model is trained in advance to output the correct depth at each position in the dust region in response to the input of the IR image, BW image, dust region, and depth information, the depth information estimated from the IR image can be appropriately corrected. In other words, the depth at each position in the dust region indicated by the depth information can be appropriately corrected.

[0208] As described above, the second depth estimation unit 111b may use deep learning. In this case, it is not necessary to output the dust region, and the second depth information may be generated directly by deep learning.

[0209] FIG. 27 is a block diagram showing another example of the functional configuration of the depth acquisition device 1 in this modified example.

[0210] The depth acquisition device 1 in this modification does not include the dust detection unit 112, the high brightness area detection unit 116, and the FOE detection unit 117 among the components shown in FIG. 26, but includes other components.

[0211] In the learning of the learning model 104, as in the example shown in FIG. 26, in addition to an IR image, a BW image, and the first depth information, a correct depth image is prepared in advance. Next, the IR image, the BW image, and the first depth information are given to the learning model 104 as inputs. Then, the learning model 104 is trained so that the output from the learning model 104 in response to the input matches the correct depth image. As the learning model 104, a VGG-16 network with an added skip connection may be used as in non-patent literature (Caner Hazirbas, Laura Leal-Taixe and Daniel Cremers C. Hazirbas, “Deep Depth From Focus”, In ArXiv preprint arXiv, 1704.01085, 2017.). The number of channels of the learning model 104 is changed so that the IR image, the BW image, and the first depth information are given to the learning model 104 as inputs. By utilizing the learning model 104 that has been trained in advance in this manner, the second depth estimation unit 111b can easily obtain the second depth information from the learning model 104 by inputting the IR image, the BW image, and the first depth information into the learning model 104.

[0212] That is, the depth acquisition device 1 shown in FIG. 27 includes a memory and a processor 110. Although the memory is not shown in FIG. 27, it may be included in the depth acquisition device 1 as shown in FIG. 5. The processor 110 acquires timing information indicating the timing at which the light source 101 irradiates the subject with infrared light. Next, the processor 110 acquires an IR image obtained by imaging a scene including the subject based on infrared light according to the timing indicated by the timing information and stored in the memory. Next, the processor 110 acquires a BW image obtained by imaging a scene substantially identical to the IR image based on visible light, from substantially the same viewpoint and imaging time as the IR image, and stored in the memory. Then, the first depth estimation unit 111a of the processor 110 estimates depth information indicating the depth at each position in the IR image. The second depth estimation unit 111b corrects the depth at each position in the dust region of the IR image indicated by the depth information by inputting the IR image, the BW image, and the depth information into the learning model 104.

[0213] Therefore, if the learning model 104 is trained in advance so that the correct depth at each position in the dust region of the IR image is output in response to the input of an IR image, a BW image, and depth information, the depth information estimated from the IR image can be appropriately corrected. In other words, the depth at each position in the dust region indicated by the depth information can be appropriately corrected without detecting the dust region.

[0214] As described above, in the depth acquisition device 1 of this embodiment and its modified examples, even if there is a dust area in the IR image, the appropriate depth at each position within the dust area can be acquired by using an image of the corresponding area in the BW image.

[0215] In each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software for realizing the depth acquisition device of the above embodiment and modified example causes a computer to execute each step included in any of the flowcharts of Figures 6, 7, and 21 to 25.

[0216] Although the depth acquisition device according to one or more aspects has been described based on the embodiment and its modified examples, the present disclosure is not limited to the embodiment and its modified examples. As long as it does not deviate from the spirit of the present disclosure, various modifications conceived by a person skilled in the art to the present embodiment and modified examples, and forms constructed by combining the components in the embodiment and modified examples may also be included within the scope of the present disclosure.

[0217] For example, in the above embodiment and its modified example, the dust detection unit 112 detects a dust region, but it may detect a region other than the dust region as long as the region is an object that is reflected as noise similar to dust. For example, the dust detection unit 112 may detect a rain region in which raindrops are reflected, or a snow region in which snowdrops are reflected. In an environment where light rain is falling, if the size of the raindrops is sufficiently small relative to the resolution of the BW image acquired by the BW camera 103 or the BW image acquisition unit 115, the raindrops are not reflected in the BW image. However, in the IR image acquired by the IR camera 102 or the IR image acquisition unit 114, the infrared light from the light source 101 is reflected by the raindrops and observed as high brightness. For this reason, the depth of the rain region becomes inappropriate in the first depth information or the first depth image generated by the first depth estimation unit 111a. Similarly, in a snowfall environment, if the size of snow particles is sufficiently small relative to the resolution of the BW image acquired by the BW camera 103 or the BW image acquisition unit 115, the snow particles are not captured in the BW image. However, in the IR image acquired by the IR camera 102 or the IR image acquisition unit 114, the infrared light from the light source 101 is reflected by the snow particles and observed as high brightness. For this reason, the first depth information or the first depth image generated by the first depth estimation unit 111a has an inappropriate depth of the snow region. Therefore, the dust detection unit 112 detects the rain region or the snow region using a method similar to that for detecting the dust region. As a result, the second depth estimation unit 111b generates the second depth information using the BW image, the first depth information, and the region (i.e., the rain region or the snow region) detected by the dust detection unit 112. This makes it possible to acquire the second depth information that is not affected by rain or snow. Note that the dust in this disclosure includes solid particles such as dust, but is not limited to solid particles and may include liquid particles. For example, dust in the present disclosure may include at least one of dust particles, raindrops, and snow particles.

[0218] In addition, in this disclosure, all or part of the units, devices, or all or part of the functional blocks of the block diagrams shown in Figures 1, 4, 5, 8, 26, and 27 may be implemented by one or more electronic circuits including a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). The LSI or IC may be integrated into one chip, or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated into one chip. Although LSI and IC are referred to here, the names may change depending on the degree of integration, and may be called system LSI, VLSI (very large scale integration), or ULSI (ultra large scale integration). Field programmable gate arrays (FPGAs) that are programmed after the manufacture of LSIs, or reconfigurable logic devices that can reconfigure the junction relationship within LSIs or set up circuit partitions within LSIs, can also be used for the same purpose.

[0219] Furthermore, all or part of the functions or operations of a unit, device, or part of a device can be executed by software processing. In this case, the software is recorded in one or more non-transitory recording media such as a ROM, an optical disk, a hard disk drive, etc., and when the software is executed by a processor, the software causes the processor and peripheral devices to execute specific functions in the software. The system or device may include one or more non-transitory recording media in which the software is recorded, a processor, and necessary hardware devices such as an interface. [Industrial Applicability]

[0220] The present disclosure is applicable to a depth acquisition device that acquires depth from an image obtained by shooting, and can be used as, for example, an in-vehicle device. [Explanation of symbols]

[0221] 1. Depth acquisition device 10, 101 light source 20 Solid-state imaging element 21 1st pixel (IR) 22 2nd pixel (BW) 30 Processing circuit 50 Diffuser 60 Lenses 70 Bandpass Filter 102 IR Camera 103 BW Camera 104 Learning Model 110 Processor 111 Depth Estimation Unit 111a First depth estimation unit 111b Second depth estimation unit 112 Dust detection unit 113 Light emission timing acquisition unit 114 IR image acquisition unit 115 BW image acquisition unit 116 High brightness area detection unit 117 FOE Detector 118 Output section 200 Memory

Claims

1. An IR image is obtained by capturing light that is emitted from a light source and reflected by a subject; Obtaining a BW image by imaging the subject with visible light; detecting a dust area based on the IR image and the BW image; Information processing methods.

2. The detection of the dust region includes detecting the dust region based on a region in the IR image corresponding to a region in the BW image. The information processing method according to claim 1 .

3. In the detection of the dust region, the dust region is detected based on a luminance of a region in the IR image corresponding to a region in the BW image.

3. The information processing method according to claim 1.

4. In the detection of the dust region, it is detected whether a brightness of a region in the IR image corresponding to a region in the BW image is equal to or greater than a threshold value. The information processing method according to claim 3.

5. In the detection of the dust region, when a luminance of a region in the IR image corresponding to a region in the BW image is equal to or greater than a threshold, the region in the IR image is detected as the dust region. The information processing method according to claim 4.

6. In the detection of the dusty region, the dusty region is detected based on the IR image and the BW image generated by capturing the same scene as the IR image. The information processing method according to claim 1 .

7. In the detection of the dusty region, the dusty region is detected based on the IR image and the BW image generated by capturing the IR image from the same viewpoint as the IR image. The information processing method according to claim 1 .

8. In the detection of the dusty region, the dusty region is detected based on the IR image and the BW image generated by capturing the IR image at the same time. The information processing method according to claim 1 .

9. The IR image is acquired using Time of Flight (ToF). The information processing method according to claim 1 .

10. In detecting the dust area, based on the IR image and the BW image, when a high-luminance region in the IR image having a luminance equal to or greater than a first threshold satisfies a first condition, the high-luminance region is detected as the dust region; The information processing method according to claim 1 .

11. The first condition is: The condition is that the center of gravity of the high-luminance area is located on a straight line or an arc intersecting the center of gravity of at least two other high-luminance areas different from the high-luminance area in the IR image and the FOE (Focus of Expansion) of the IR image or the BW image. The information processing method according to claim 10.

12. The first condition is: A condition in which the principal axis of the high-brightness region or an extension of the principal axis intersects with the FOE (Focus of Expansion) of the IR image or the BW image. The information processing method according to claim 10.

13. In detecting the dust area, If the high-luminance region further satisfies a second condition, the high-luminance region is detected as the dust region. The information processing method according to any one of claims 10 to 12.

14. The second condition is: a condition that the luminance at a position in the BW image corresponding to the center of gravity of the high luminance area in the IR image is less than a second threshold value; The information processing method according to claim 13.

15. The second condition is: a correlation coefficient between a luminance in a high luminance region of the IR image and a luminance in a region of the BW image corresponding to the high luminance region is less than a third threshold. The information processing method according to claim 13.

16. A processor; A memory. The processor uses the memory to: Acquiring an IR image generated by capturing light irradiated from a light source and reflected by a subject; acquiring a BW image generated by imaging the subject with visible light; detecting a dust area based on the IR image and the BW image; Information processing device.

17. Acquiring an IR image generated by capturing light irradiated from a light source and reflected by a subject; acquiring a BW image generated by imaging the subject with visible light; detecting a dust area based on the IR image and the BW image; A program that causes a computer to do something.

Images