Imaging method, imaging device, and recording medium

Abstract

The invention provides an imaging method, an imaging apparatus, and a recording medium. The imaging method includes acquiring an infrared image by performing imaging that receives light irradiated from a light source and reflected by a subject, acquiring a visible light image by imaging the subject with visible light, and detecting a dust region based on the infrared image and the visible light image.

Description

[0001] This application is a divisional application of the invention application with an international filing date of August 28, 2019 and a national application number of 201980049509.6. The invention name of the invention application is depth acquisition device, depth acquisition method and program. Technical Field

[0002] The present disclosure relates to a depth acquisition device and the like for acquiring the depth of an image. Background Art

[0003] In the past, a distance measuring device for measuring the distance to a subject has been proposed (for example, refer to Patent Document 1). The distance measuring device includes a light source and an imaging unit. The light source irradiates light to the subject. The imaging unit captures the reflected light reflected by the subject. Then, the distance measuring device measures the distance to the subject by converting each pixel value of the image obtained by the imaging into the distance to the subject. That is, the distance measuring device obtains the depth of the image obtained by the imaging unit.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2011-64498 Summary of the invention

[0007] Problems to be solved by the invention

[0008] However, the distance measuring device of Patent Document 1 has a problem that the depth of an image cannot be accurately acquired.

[0009] To this end, the present disclosure provides a depth acquisition device that can accurately acquire the depth of an image.

[0010] Means for solving problems

[0011] A depth acquisition device involved in one scheme of the present disclosure includes a memory and a processor, wherein the processor obtains timing information indicating the timing of a light source irradiating infrared light to a subject, obtains an infrared image, the infrared image is obtained by infrared-light-based imaging of a scene corresponding to the timing indicated by the timing information and including the subject, and is maintained in the memory, obtains a visible light image, the visible light image is obtained by visible-light-based imaging of substantially the same scene as the infrared image and at substantially the same viewpoint and imaging time as the infrared image, and is maintained in the memory, detects an area where dust is reflected from the infrared image as a dust area, and estimates the depth of the dust area based on the infrared image, the visible light image, and the dust area.

[0012] In addition, these general or specific solutions can be implemented by a system, method, integrated circuit, computer program or a computer-readable CD-ROM or other recording medium, or by any combination of the system, method, integrated circuit, computer program and recording medium. In addition, the recording medium can be a non-temporary recording medium.

[0013] Effects of the Invention

[0014] The depth acquisition device disclosed in the present invention can correctly acquire the depth of an image. Further advantages and effects in one scheme of the present invention are clearly defined according to the specification and the accompanying drawings. The relevant advantages and / or effects are provided respectively by several embodiments and the features recorded in the specification and the accompanying drawings, but it is not necessary to provide all the features in order to obtain one or more of the same features. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a block diagram showing the hardware configuration of the depth acquisition device in the embodiment.

[0016] Figure 2 It is a schematic diagram showing a pixel array included in the solid-state imaging element according to the embodiment.

[0017] Figure 3 This is a timing chart showing the relationship between the light emission timing of the light emitting element of the light source and the exposure timing of the first pixel of the solid-state imaging element in the embodiment.

[0018] Figure 4 This is a block diagram showing an example of the functional structure of the depth acquisition device in the embodiment.

[0019] Figure 5 This is a block diagram showing another example of the functional structure of the depth acquisition device in the embodiment.

[0020] Figure 6 This is a flowchart showing the overall processing operation of the depth acquisition device in the embodiment.

[0021] Figure 7 This is a flowchart showing the overall processing operation of the processor of the depth acquisition device in the embodiment.

[0022] Figure 8 This is a block diagram showing a specific functional structure of a processor of the depth acquisition device in the embodiment.

[0023] Fig. 9A This is a diagram showing an example of an IR image.

[0024] Fig. 9B This is a diagram showing an example of a BW image.

[0025] Fig.10 This is a diagram showing an example of a binarized image obtained by binarizing an IR image.

[0026] Fig.11 This is a diagram showing an example of a dust candidate region in an IR image.

[0027] Fig.12 This is a diagram showing an example of FOE detected on a BW image.

[0028] Fig.13 This is a diagram showing an example of a main axis for detecting a dust region.

[0029] Fig.14 This is a diagram showing an example of a dust area and a non-dust area.

[0030] Fig.15A FIG. 2 is a diagram showing another example of an IR image.

[0031] Fig. 15B FIG. 4 is a diagram showing another example of a BW image.

[0032] Fig.16 This is a diagram showing an example of a binarized image obtained by binarizing an IR image.

[0033] Fig.17 This is a diagram showing an example of a dust candidate region in an IR image.

[0034] Fig.18 This is a diagram showing an example of FOE detected on a BW image.

[0035] Fig.19 It is a diagram showing an example of arrangement of each dust candidate region.

[0036] Fig. 20 It is a diagram showing the simulation results of the depth acquisition device in the embodiment.

[0037] Fig.21 Yes means Figure 8 The flowchart of the overall processing operation of the depth acquisition device is shown.

[0038] Fig. 22 Yes means Fig.21 A flowchart of an example of detailed processing of steps S31 to S34.

[0039] Fig.23 Yes means Fig.21 A flowchart of another example of the detailed processing of steps S31 to S34 is shown.

[0040] Fig.24 It means to replace Fig.21Flowchart of an example of the processing of steps S31 to S34.

[0041] Fig.25 Yes means Fig.21 A flowchart of another example of the detailed processing of steps S31 to S34 is shown.

[0042] Fig.26 This is a block diagram showing an example of the functional configuration of a depth acquisition device in a modified example of the embodiment.

[0043] Fig. 27 This is a block diagram showing another example of the functional structure of the depth acquisition device in a modification of the embodiment. DETAILED DESCRIPTION

[0044] (Insights that form the basis of this disclosure)

[0045] The inventors of the present invention have found that the following problems occur with respect to the distance measuring device of Patent Document 1 described in the “Background Art” column.

[0046] The distance measuring device of patent document 1 irradiates light from a light source to a subject as described above, captures an image of the irradiated subject, thereby obtaining an image, and measuring the depth of the image. ToF (Time Of Flight) is used in the measurement of the depth. In such a distance measuring device, different imaging conditions are used to improve the distance measurement accuracy. That is, the distance measuring device captures images according to given imaging conditions, and sets imaging conditions different from the given imaging conditions according to the imaging results. Then, the distance measuring device captures images again according to the set imaging conditions.

[0047] However, in the image obtained by the camera, dust located near the distance measuring device may be captured as noise. As a result, the dust, i.e., noise, cannot be removed from the image reflecting the dust, and the depth cannot be measured correctly. In addition, even if the camera conditions are changed, it is difficult to simply suppress the dust from being captured. Furthermore, if, for example, a distance measuring device mounted on a vehicle repeatedly captures images under different camera conditions while the vehicle is traveling, the scenes of the multiple images obtained will be different because the viewpoint positions of the repeated captures are different. That is, the same scene cannot be captured repeatedly, and the depth of the image reflecting the scene, especially the depth of the area reflecting the dust, cannot be properly estimated.

[0048] In order to solve such a problem, a depth acquisition device involved in a scheme of the present invention includes a memory and a processor, wherein the processor obtains timing information indicating the timing of a light source irradiating infrared light to a subject, obtains an infrared image, wherein the infrared image is obtained by infrared light-based imaging of a scene corresponding to the timing indicated by the timing information and including the subject, and is maintained in the memory, obtains a visible light image, wherein the visible light image is obtained by visible light-based imaging of substantially the same scene as the infrared image and at substantially the same viewpoint and imaging time as the infrared image, and is maintained in the memory, detects an area reflecting dust from the infrared image as a dust area, and estimates the depth of the dust area based on the infrared image, the visible light image, and the dust area.

[0049] Thus, since the dust region is detected from the infrared image, in the dust region, the depth is estimated not only based on the infrared image but also based on the visible light image, so the depth of the dust region can be properly obtained. That is, in the infrared image and the visible light image, the scene that is the object of the camera is substantially the same, and the viewpoint and the shooting time are also substantially the same. Here, as an example of an image of substantially the same scene captured at substantially the same viewpoint and shooting time, it is an image captured by different pixels of the same camera element. Such an image is similar to the red, green and blue channel images of a color image captured by a Bayer arrangement color filter, and the viewing angle, viewpoint and shooting time of each image are roughly equal. That is, the position of the image of the substantially same scene captured at substantially the same viewpoint and shooting time on the image of the subject in each captured image will not differ by more than 2 pixels. For example, when there is a point light source having visible light and infrared components in the scene, and only one pixel is photographed at high brightness in the visible light image, in the infrared image, the point light source will also be photographed closer than 2 pixels to the pixel corresponding to the pixel position photographed in the visible light image. In addition, the so-called substantially the same photographing time means that the difference in photographing time is equal to less than 1 frame. Therefore, the infrared image and the visible light image have a high correlation. However, when there is dust near the camera that takes the infrared image and the visible light image, the irradiated infrared light is strongly reflected by the dust, and the dust is sometimes photographed in the infrared image due to the strong reflected light. Therefore, even if the dust is photographed in the infrared image, there is a high possibility that it will not be photographed in the visible light image. For this reason, the information missing in the dust area can be supplemented from the area in the visible light image corresponding to the dust area (i.e., the corresponding area). As a result, the influence of dust, i.e., noise, can be removed and the depth of the dust area can be properly obtained.

[0050] For example, the processor may estimate the first depth information indicating the depth at each position in the infrared image in estimating the depth of the dust area, and estimate the second depth information indicating the corrected depth at each position in the dust area by correcting the depth at each position in the dust area indicated by the first depth information based on the visible light image, thereby generating the third depth information indicating the depth at each position outside the dust area of ​​the infrared image indicated by the first depth information and the depth at each position in the dust area of ​​the infrared image indicated by the second depth information. In addition, in estimating the first depth information, TOF or the like may be applied to the infrared image.

[0051] Thus, the third depth information indicates the depth obtained from the infrared image as the depth outside the dust region of the infrared image, and indicates the depth obtained from the infrared image and corrected based on the visible light image as the depth of the dust region of the infrared image. Therefore, even if there is a dust region in the infrared image, the overall depth of the infrared image can be properly estimated.

[0052] In addition, the processor may detect the high-brightness region as the dust region when a high-brightness region having a brightness greater than a first threshold value in the infrared image satisfies a first condition during the detection of the dust region.

[0053] The dust region has a tendency to be high in brightness. Therefore, by detecting a high brightness region having a brightness above the first threshold in the infrared image, the region with a high possibility of reflecting dust can be easily narrowed down. Furthermore, since a high brightness region satisfying the first condition is detected as a dust region, the dust region can be detected with high accuracy by appropriately setting the first condition.

[0054] In addition, the first condition may also be a condition that the center of gravity of the high-brightness area is arranged on a straight line or an arc where the center of gravity of at least two other high-brightness areas different from the high-brightness area in the infrared image intersects with the FOE (Focus of Expansion) of the infrared image or the visible light image.

[0055] For example, when a camera is photographing a speck of dust, if a single frame of infrared image is exposed multiple times, the dust will appear in the infrared image as a high-brightness area each time the exposure is performed. For this reason, when the camera is mounted on a moving object such as a vehicle and moves, there is a high possibility that the dust will be blown out from the FOE and appear in the infrared image. For example, if the moving object moves at a high speed, the multiple high-brightness areas caused by the dust will tend to be arranged on a straight line intersecting the FOE. Alternatively, if the distortion of the camera lens is large, the multiple high-brightness areas caused by the dust will tend to be arranged on an arc intersecting the FOE. Therefore, by setting the first condition that the center of gravity of at least three high-brightness areas is arranged on the straight line or arc, the dust area can be detected with high accuracy.

[0056] In addition, in the infrared image and the visible light image, since the scene that becomes the object of imaging is substantially the same and the viewpoint is also substantially the same, the FOE of the infrared image and the FOE of the visible light image are substantially the same.

[0057] Alternatively, the first condition may be a condition that the main axis of the high brightness area or an extension of the main axis intersects with the FOE (Focus of Expansion) of the infrared image or the visible light image.

[0058] For example, if the moving speed of the mobile body equipped with the camera is low, in contrast to the above example, the above-mentioned multiple high brightness areas overlap. As a result, these multiple high brightness areas are reflected in the infrared image as one high brightness area of ​​an elongated shape, in other words, a trailing shape. The main axis of the high brightness area caused by such dust or the extension of the main axis tends to intersect with the FOE. Therefore, by setting the main axis of the high brightness area or the extension of the main axis to intersect with the FOE as the first condition, the dust area can be detected with high accuracy.

[0059] Furthermore, the processor may detect the high-brightness region as the dust region when the high-brightness region further satisfies a second condition during the detection of the dust region.

[0060] For example, even if a dust region is observed in an infrared image, the property that the dust region is not observed in a visible light image can be used as the second condition. This makes it possible to detect the dust region with higher accuracy.

[0061] In addition, the second condition may be a condition that the brightness of a position in the visible light image corresponding to the center of gravity of the high brightness area in the infrared image is less than a second threshold value.

[0062] Since the dust area cannot be observed in the BW image, the brightness of the position in the visible light image corresponding to the center of gravity of the dust area tends to be low. Therefore, by setting the brightness of the position in the visible light image corresponding to the center of gravity of the high-brightness area in the infrared image to be less than the second threshold as the second condition, the dust area can be detected with higher accuracy.

[0063] In addition, the second condition may be a condition that a correlation coefficient between the brightness in a high-brightness region of the infrared image and the brightness in a region of the visible light image corresponding to the high-brightness region is less than a third threshold value.

[0064] Since there is a property that even if a dust region is observed in an infrared image, the dust region cannot be observed in a visible light image, the correlation between the brightness in the dust region of the infrared image and the brightness in the region of the visible light image corresponding to the dust region tends to be low. Therefore, by setting the correlation coefficient between the brightness in the high brightness region of the infrared image and the brightness in the region of the visible light image corresponding to the high brightness region to be less than the third threshold as the second condition, the dust region can be detected with higher accuracy.

[0065] In addition, the processor may also estimate the depth information representing the depth at each position within the infrared image when estimating the depth of the dust area, and correct the depth at each position within the dust area represented by the depth information by inputting the infrared image, the visible light image, the dust area, and the depth information into a learning model.

[0066] Therefore, if the learning model is pre-learned so that the correct depth of each position in the dust area is outputted for the input of the infrared image, the visible light image, the dust area, and the depth information, the depth information estimated from the infrared image can be appropriately corrected. In other words, the depth of each position in the dust area represented by the depth information can be appropriately corrected.

[0067] In addition, other schemes of the present disclosure involve a depth acquisition device comprising a memory and a processor, wherein the processor acquires timing information indicating the timing of a light source irradiating infrared light to a subject, acquires an infrared image, wherein the infrared image is obtained by infrared light-based imaging of a scene including the subject corresponding to the timing indicated by the timing information, and is maintained in the memory, acquires a visible light image, wherein the visible light image is obtained by visible light-based imaging of substantially the same scene as the infrared image and at substantially the same viewpoint and imaging time as the infrared image, and is maintained in the memory, estimates depth information indicating the depth at each position in the infrared image, and corrects the depth at each position in a dust area where dust is reflected in the infrared image indicated by the depth information by inputting the infrared image, the visible light image, and the depth information into a learning model.

[0068] Therefore, if the learning model is pre-learned so that the correct depth of each position in the dust area of ​​the infrared image is outputted for the input of the infrared image, the visible light image, and the depth information, the depth information estimated from the infrared image can be properly corrected. In other words, the depth of each position in the dust area represented by the depth information can be properly corrected without detecting the dust area.

[0069] In addition, the depth acquisition device involved in other schemes of the present disclosure includes a memory and a processor, and the processor executes: obtaining an infrared image stored in the memory, wherein the infrared image is obtained by imaging based on infrared light, obtaining a visible light image stored in the memory, wherein the visible light image is obtained by imaging based on visible light at substantially the same viewpoint and imaging time as the infrared image, detecting an area reflecting dust from the infrared image as a dust area, and estimating the depth of the dust area based on the visible light image. In addition, when the visible light image and the infrared image are respectively divided into a dust area and other areas, the depth of the dust area is estimated based on the visible light image, and the depth of the other areas is estimated based on the infrared image.

[0070] Thus, similarly to the depth acquisition device according to the above-mentioned one aspect of the present disclosure, the influence of dust, ie, noise, can be eliminated and the depth of the dust area can be appropriately acquired.

[0071] In addition, these general or specific solutions can be implemented by a system, method, integrated circuit, computer program or computer-readable CD-ROM or other recording medium, or by any combination of systems, methods, integrated circuits, computer programs or recording media. In addition, the recording medium can be a non-transitory recording medium.

[0072] The embodiments are described in detail below with reference to the drawings.

[0073] In addition, the embodiments described below all represent general or specific examples. The numerical values, shapes, materials, components, configuration positions of components, connection forms, steps, and the order of steps shown in the following embodiments are examples and do not limit the subject matter of the present disclosure. In addition, the components in the following embodiments that are not recorded in the independent claims representing the highest concept are described as arbitrary components.

[0074] In addition, each figure is a schematic diagram and is not necessarily a strict illustration. In addition, in each figure, the same reference numeral is attached to the same component.

[0075] (Implementation Method)

[0076] [Hardware structure]

[0077] Figure 1 This is a block diagram showing the hardware structure of the depth acquisition device 1 involved in the embodiment. The depth acquisition device 1 in this embodiment has a hardware structure that can acquire an image based on infrared light (or near-infrared light) and an image based on visible light by shooting substantially the same scene and substantially the same viewpoint and shooting time. In addition, the so-called substantially the same means the same degree as that of being able to achieve the effect in the present disclosure.

[0078] like Figure 1 As shown, the depth acquisition device 1 includes a light source 10 , a solid-state imaging element 20 , a processing circuit 30 , a diffusion plate 50 , a lens 60 , and a bandpass filter 70 .

[0079] The light source 10 emits illumination light. More specifically, the light source 10 emits illumination light for illuminating a subject at a timing indicated by a light emission signal generated by the processing circuit 30 .

[0080] The light source 10 is composed of, for example, a capacitor, a driving circuit, and a light-emitting element, and emits light by driving the light-emitting element with electric energy stored in the capacitor. The light-emitting element is realized by, for example, a laser diode, a light-emitting diode, etc. In addition, the light source 10 may be a structure including one type of light-emitting element, or may be a structure including multiple types of light-emitting elements according to the purpose.

[0081] In the following, the light emitting element is, for example, a laser diode that emits near-infrared light, or a light emitting diode that emits near-infrared light. However, the irradiation light irradiated by the light source 10 is not necessarily limited to near-infrared light. The irradiation light irradiated by the light source 10 may be, for example, infrared light (also referred to as infrared light) in a frequency band other than near-infrared light. In the following, in this embodiment, the irradiation light irradiated by the light source 10 is described as infrared light, but the infrared light may be near-infrared light or infrared light in a frequency band other than near-infrared light.

[0082] The solid-state imaging element 20 captures an image of a subject and outputs an imaging signal indicating an exposure amount. More specifically, the solid-state imaging element 20 performs exposure at a timing indicated by an exposure signal generated by the processing circuit 30 and outputs an imaging signal indicating an exposure amount.

[0083] The solid-state imaging device 20 has a pixel array in which first pixels for imaging using reflected light of an irradiated light reflected from an object and second pixels for imaging the object are arranged in an array. The solid-state imaging device 20 may have logic functions such as a cover glass and an AD converter as necessary.

[0084] In the following description, the reflected light is assumed to be infrared light, similarly to the irradiated light. However, the reflected light is not limited to infrared light as long as it is light obtained by reflecting the irradiated light from the subject.

[0085] Figure 2 Schematic diagram showing the pixel array 2 included in the solid-state imaging element 20 .

[0086] like Figure 2 As shown, the pixel array 2 is configured in an array so that first pixels 21 (IR pixels) for capturing images using reflected light obtained by irradiating light reflected from a subject and second pixels 22 (BW pixels) for capturing images of the subject are alternately arranged in columns.

[0087] In addition, Figure 2 In the pixel array 2, the second pixel 22 and the first pixel 21 are arranged adjacent to each other in the row direction and arranged in a strip shape in the row direction, but this is not limited to this, and they may be arranged every multiple rows (for example, every 2 rows). That is, the second pixel 22 is arranged in the first row adjacent to each other in the row direction and the first pixel 21 is arranged in the second row adjacent to each other in the row direction, and they may be arranged alternately every M rows (M is a natural number). Furthermore, the second pixel 22 is arranged in the first row adjacent to each other in the row direction and the first pixel 21 is arranged in the second row adjacent to each other in the row direction, and they may be arranged separated by different rows (alternatingly repeating the first row as N rows and the second row as L rows (N and L are different natural numbers)).

[0088] The first pixel 21 is realized by, for example, an infrared light pixel having sensitivity to infrared light as reflected light, and the second pixel 22 is realized by, for example, a visible light pixel having sensitivity to visible light.

[0089] The infrared light pixel is composed of, for example, an optical filter (also referred to as an IR filter) that transmits only infrared light, a microlens, a light-receiving element as a photoelectric conversion unit, and a storage unit that stores the charge generated by the light-receiving element. Therefore, an image representing the brightness of infrared light is expressed by the imaging signal output from the plurality of infrared light pixels (i.e., the first pixel 21) included in the pixel array 2. Hereinafter, the infrared light image is also referred to as an IR image or an infrared image.

[0090] In addition, the visible light pixel is composed of, for example, an optical filter (also called a BW filter) that transmits only visible light, a microlens, a light receiving element as a photoelectric conversion unit, and a storage unit that stores the charge converted by the light receiving element. Therefore, the visible light pixel, that is, the second pixel 22, outputs an imaging signal representing brightness and color difference. That is, a color image representing the brightness and color difference of visible light is expressed by the imaging signal output from the plurality of second pixels 22 included in the pixel array 2. In addition, the optical filter of the visible light pixel can transmit both visible light and infrared light, or can transmit only light in a specific wavelength band such as red (R), green (G) or blue (B) among the visible light.

[0091] In addition, the visible light pixel can detect only the brightness of the visible light. In this case, the visible light pixel, i.e., the second pixel 22, outputs an imaging signal indicating the brightness. Therefore, the imaging signal outputted from the plurality of second pixels 22 contained in the pixel array 2 represents a black and white image indicating the brightness of the visible light, in other words, a monochrome image. Hereinafter, the monochrome image is also referred to as a BW image. In addition, the above-mentioned color image and BW image are also collectively referred to as a visible light image.

[0092] Back again Figure 1 , continue with the description of the acquisition device 1 in depth.

[0093] The processing circuit 30 calculates subject information related to the subject using the imaging signal outputted from the solid-state imaging element 20 .

[0094] The processing circuit 30 includes, for example, a microcomputer or other computing device. The microcomputer includes a processor (microprocessor), a memory, etc., and generates a light emission signal and an exposure signal by executing a driver stored in the memory by the processor. In addition, the processing circuit 30 may use an FPGA or an ISP, etc., and may include one hardware or multiple hardware.

[0095] The processing circuit 30 calculates the distance to the subject by, for example, a TOF distance measurement method using the imaging signal from the first pixel 21 of the solid-state imaging element 20 .

[0096] The calculation of the distance to the subject by the TOF distance measurement method performed by the processing circuit 30 will be described below with reference to the drawings.

[0097] Figure 3 This is a timing chart showing the relationship between the light emission timing of the light emitting element of the light source 10 and the exposure timing of the first pixel 21 of the solid-state imaging element 20 when the processing circuit 30 calculates the distance to the subject using the TOF ranging method.

[0098] exist Figure 3In the figure, Tp is the light emission period during which the light emitting element of the light source 10 emits the irradiation light, and Td is the delay time from when the light emitting element of the light source 10 emits the irradiation light to when the reflected light of the irradiation light reflected by the subject returns to the solid-state imaging element 20. Furthermore, the first exposure period becomes the same timing as the light emission period during which the light source 10 emits the irradiation light, and the second exposure period becomes the timing from the end time point of the first exposure period to the passage of the light emission period Tp.

[0099] exist Figure 3 In the figure, q1 represents the total amount of exposure in the first pixel 21 of the solid-state imaging element 20 caused by the reflected light during the first exposure period, and q2 represents the total amount of exposure in the first pixel 21 of the solid-state imaging element 20 caused by the reflected light during the second exposure period.

[0100] By Figure 3 The light emitting element of the light source 10 emits the irradiation light and the first pixel 21 of the solid-state imaging element 20 exposes the light at the timing shown. Assuming the speed of light is c, the distance d to the subject can be expressed by the following (Equation 1).

[0101] d=c×Tp / 2×q2 / (q1+q2)…(Formula 1)

[0102] Therefore, the processing circuit 30 can calculate the distance to the subject using the imaging signal from the first pixel 21 of the solid-state imaging element 20 by using (Formula 1).

[0103] In addition, the plurality of first pixels 21 of the solid-state imaging element 20 can be exposed only in the third exposure period Tp after the first exposure period and the second exposure period have ended. The plurality of first pixels 21 can detect noise other than the reflected light based on the exposure amount obtained in the third exposure period Tp. That is, the processing circuit 30 can more accurately calculate the distance d to the subject by removing the noise from the exposure amount q1 in the first exposure period and the exposure amount q2 in the second exposure period in the above (Formula 1).

[0104] Back again Figure 1 , continue with the description of the acquisition device 1 in depth.

[0105] The processing circuit 30 can detect the subject and calculate the distance to the subject using the imaging signal from the second pixel 22 of the solid-state imaging element 20 , for example.

[0106] That is, the processing circuit 30 can detect the subject and calculate the distance to the subject based on the visible light image captured by the multiple second pixels 22 of the solid-state imaging element 20. Here, the detection of the subject can be achieved, for example, by using edge detection of the singularity of the subject to distinguish the shape by pattern recognition, or it can be achieved by using a learning model that has been learned in advance through deep learning and other processes. In addition, the calculation of the distance to the subject can be performed using world coordinate transformation. Of course, it is also possible to detect the subject by multimodal learning processing using not only the visible light image but also the brightness and distance information of the infrared light captured by the first pixel 21.

[0107] The processing circuit 30 generates a light emission signal indicating a timing for light emission and an exposure signal indicating a timing for exposure. The processing circuit 30 then outputs the generated light emission signal to the light source 10 and outputs the generated exposure signal to the solid-state imaging element 20 .

[0108] The processing circuit 30 can, for example, generate and output a light emission signal to cause the light source 10 to emit light at a given cycle, and generate and output an exposure signal to cause the solid-state imaging element 20 to be exposed at a given cycle, thereby enabling the depth acquisition device 1 to achieve continuous imaging at a given frame rate. In addition, the processing circuit 30 includes, for example, a processor (microprocessor), a memory, etc., and generates a light emission signal and an exposure signal by executing a driver program stored in the memory by the processor.

[0109] The diffuser 50 adjusts the intensity distribution and angle of the irradiated light. In addition, in the adjustment of the intensity distribution, the diffuser 50 makes the intensity distribution of the irradiated light from the light source 10 uniform. Figure 1 In the illustrated example, the depth acquisition device 1 includes the diffusion plate 50 , but may not include the diffusion plate 50 .

[0110] The lens 60 is an optical lens that focuses light entering from the outside of the depth acquisition device 1 on the surface of the pixel array 2 of the solid-state imaging element 20 .

[0111] The bandpass filter 70 is an optical filter that transmits infrared light as reflected light and visible light. Figure 1 In the illustrated example, the depth acquisition device 1 includes the bandpass filter 70 , but may not include the bandpass filter 70 .

[0112] The depth acquisition device 1 of the above structure is mounted on a transport device and used. For example, the depth acquisition device 1 is mounted on a vehicle traveling on a road and used. In addition, the transport device on which the depth acquisition device 1 is mounted is not necessarily limited to a vehicle. The depth acquisition device 1 can also be mounted on a transport device other than a vehicle such as a motorcycle, a ship, or an airplane and used.

[0113] [Overview of the depth acquisition device]

[0114] The depth acquisition device 1 in this embodiment is used Figure 1 The hardware structure shown acquires an IR image and a BW image by photographing substantially the same scene, substantially the same viewpoint, and at the same time. Then, the depth acquisition device 1 uses the BW image to correct the depth at each position in the IR image obtained from the IR image. Specifically, when there is an area in which dust is reflected in the IR image (hereinafter referred to as the dust area), the depth acquisition device 1 uses the image in the area of ​​the BW image corresponding to the dust area to correct the depth at each position in the dust area obtained from the IR image. In this way, the influence of dust, that is, noise, can be removed and the depth of the dust area can be properly acquired.

[0115] Figure 4 This is a block diagram showing an example of the functional structure of the depth acquisition device 1 .

[0116] The depth acquisition device 1 includes a light source 101 , an IR camera 102 , a BW camera 103 , a depth estimation unit 111 , and a dust detection unit 112 .

[0117] The light source 101 may include Figure 1 The light source 10 and the diffuser plate 50 are shown.

[0118] IR camera 102 may include Figure 1 The solid-state imaging element 20 shown has a plurality of first pixels 21, a lens 60, and a bandpass filter 70. Such an IR camera 102 acquires an IR image by capturing a scene including a subject with infrared light in accordance with the timing at which the light source 101 irradiates the subject with infrared light.

[0119] The BW camera 103 may include Figure 1 The solid-state imaging element 20 shown has a plurality of second pixels 22, a lens 60, and a bandpass filter 70. Such a BW camera 103 acquires a visible light image (specifically, a BW image) by performing visible light imaging of a scene substantially the same as the infrared image and imaging at substantially the same viewpoint and imaging time as the infrared image.

[0120] The depth estimation unit 111 and the dust detection unit 112 can be used as Figure 1 The functions of the processing circuit 30 shown, specifically the functions of the processor 110, are implemented.

[0121] The dust detection unit 112 detects a dust area from the IR image based on the IR image obtained by the IR camera 102 and the BW image obtained by the BW camera 103. In other words, the dust detection unit 112 divides the IR image obtained by the imaging into a dust area where dust is reflected and a non-dust area where dust is not reflected.

[0122] When there are fine particles such as dust near the IR camera 102, the dust is reflected as large noise in the IR image. The dust area in this embodiment is a high-brightness area where dust is reflected. For example, when there is dust near the depth acquisition device 1, the dust is irradiated from the light source 101 and the infrared light reflected by the dust is received by the solid-state imaging element 20 while maintaining a high brightness. Therefore, in the IR image, the brightness of each pixel in the area where dust is reflected, that is, the dust area, is high.

[0123] The depth estimation unit 111 estimates the depth of each position in the IR image including the dust region detected by the dust detection unit 112. Specifically, the depth estimation unit 111 obtains an IR image captured by the IR camera 102 corresponding to the timing of the light source 101 irradiating the subject with infrared light, and estimates the depth of each position in the IR image based on the IR image. Furthermore, the depth estimation unit 111 corrects the depth of each position estimated in the dust region detected by the dust detection unit 112 based on the BW image. That is, the depth estimation unit 111 estimates the depth of the dust region based on the IR image, the BW image, and the dust region.

[0124] Figure 5 It is a block diagram showing another example of the functional structure of the depth acquisition device 1 .

[0125] The depth acquiring device 1 may include a memory 200 and a processor 110 .

[0126] In addition, the processor 110 may include not only the depth estimation unit 111 and the dust detection unit 112, but also Figure 5 As shown in the figure, the light emission timing acquisition unit 113, the IR image acquisition unit 114, and the BW image acquisition unit 115 are provided. In addition, these components are realized as functions of the processor 110.

[0127] The light emission timing acquisition unit 113 acquires timing information indicating the timing at which the light source 101 radiates infrared light to the subject. Figure 1 The light emission signal shown is output to the light source 101, and information indicating the timing of the output is obtained as the above-mentioned timing information.

[0128] The IR image acquisition unit 114 acquires an IR image obtained by capturing an image of a scene including a subject using infrared light at a timing indicated by the timing information, and stores the IR image in the memory 200 .

[0129] The BW image acquisition unit 115 acquires a BW image obtained by imaging substantially the same scene as the IR image using visible light and at substantially the same viewpoint and imaging time as the IR image, and stores the BW image in the memory 200 .

[0130] The dust detection unit 112 detects the dust region from the IR image as described above, and the depth estimation unit 111 estimates the depth based on the IR image, the BW image, and the dust region.

[0131] In addition, the depth acquisition device 1 in this embodiment may not include the light source 101 , the IR camera 102 , and the BW camera 103 , and may be composed of the processor 110 and the memory 200 .

[0132] Figure 6 It is a flowchart showing the overall processing operation of the depth acquisition device 1 .

[0133] (Step S11)

[0134] First, the light source 101 emits light to irradiate the subject with infrared light.

[0135] (Step S12)

[0136] Next, the IR camera 102 acquires an IR image. That is, the IR camera 102 captures a scene including a subject irradiated with infrared light by the light source 101. Thus, the IR camera 102 acquires an IR image based on the infrared light reflected from the subject. Specifically, the IR camera 102 acquires the infrared image through Figure 3 IR images obtained by changing the timing and exposure amount of the first exposure period, the second exposure period, and the third exposure period shown.

[0137] (Step S13)

[0138] Next, the BW camera 103 acquires a BW image. That is, the BW camera 103 acquires a BW image corresponding to the IR image acquired in step S12, that is, a BW image of the same scene, the same viewpoint, and the same imaging time as the IR image.

[0139] (Step S14)

[0140] Then, the dust detection unit 112 detects a dust region from the IR image acquired in step S12 .

[0141] (Step S15)

[0142] Next, the depth estimation unit 111 estimates the depth of the dust region based on the IR image acquired in step S12 , the BW image acquired in step S13 , and the dust region detected in step S14 .

[0143] Figure 7 This is a flowchart showing the overall processing operation of the processor 110 of the depth acquisition device 1 .

[0144] (Step S21)

[0145] First, the light emission timing acquisition unit 113 of the processor 110 acquires timing information indicating the timing at which the light source 101 radiates infrared light onto the subject.

[0146] (Step S22)

[0147] Next, the IR image acquisition unit 114 acquires an IR image from the IR camera 102 that has captured an image at the timing indicated by the timing information acquired in step S21. Figure 1 The exposure signal is output to the IR camera 102 at the timing of the light emission signal shown. As a result, the IR image acquisition unit 114 causes the IR camera 102 to start imaging and acquires an IR image obtained by the imaging from the IR camera 102. At this time, the IR image acquisition unit 114 may acquire the IR image from the IR camera 102 via the memory 200 or directly from the IR camera 102.

[0148] (Step S23)

[0149] Next, the BW image acquisition unit 115 acquires a BW image corresponding to the IR image acquired in step S22 from the BW camera 103. At this time, the BW image acquisition unit 115 may acquire the BW image from the BW camera 103 via the memory 200 or directly from the BW camera 103.

[0150] (Step S24)

[0151] Then, the dust detection unit 112 detects a dust region from the IR image.

[0152] (Step S25)

[0153] Next, the depth estimation unit 111 estimates the depth of the dust region based on the IR image obtained in step S22, the BW image obtained in step S23, and the dust region detected in step S24. Thus, at least the depth information indicating the depth of the dust region is calculated. In addition, at this time, the depth estimation unit 111 estimates not only the dust region but also the depth of the entire IR image, and calculates the depth information indicating the estimation result.

[0154] Specifically, the depth estimation unit 111 in this embodiment estimates the depth of each position in the IR image from the IR image obtained in step S22. Then, the depth estimation unit 111 uses the BW image to correct the depth of each position in the dust area. In addition, each position may be the position of each of a plurality of pixels or the position of a block composed of a plurality of pixels.

[0155] In the depth acquisition device 1 of the present embodiment, a dust area is detected from an IR image, and in the dust area, the depth is estimated not only based on the IR image but also based on the BW image, so that the depth of the dust area can be properly acquired. That is, in the IR image and the BW image, the scene that becomes the object of photography is substantially the same, and the viewpoint and the photography time are also substantially the same. Therefore, the IR image and the BW image have a high correlation. In addition, even if dust is captured in the IR image, there is a high possibility that the dust will not be captured in the BW image. Therefore, the missing information in the dust area can be supplemented from the area in the BW image corresponding to the dust area (i.e., the corresponding area). As a result, the depth of the dust area can be properly acquired.

[0156] [Specific functional structure of the depth acquisition device]

[0157] Figure 8 It is a block diagram showing a specific functional structure of the processor 110 of the depth acquisition device 1 .

[0158] The processor 110 includes a first depth estimation unit 111a, a second depth estimation unit 111b, a dust detection unit 112, a high brightness region detection unit 116, a FOE detection unit 117, and an output unit 118. The first depth estimation unit 111a and the second depth estimation unit 111b correspond to Figure 5 The depth estimation unit 111 is shown. In addition, the processor 110 may include the light emission timing acquisition unit 113, the IR image acquisition unit 114, and the BW image acquisition unit 115 described above.

[0159] The high-luminance region detection unit 116 detects a region having luminance equal to or higher than a first threshold value in the IR image as a high-luminance region.

[0160] The FOE detection unit 117 detects FOE (Focus of Expansion) in the BW image. FOE is also called a vanishing point. It is known that when the IR camera 102 moves in parallel and the subject stops, the visible motion on the screen, that is, the optical flow, intersects at one point. This point is the FOE.

[0161] The dust detection unit 112 determines whether at least one high brightness area in the IR image is a dust area. Here, the inventors found that each dust reflected in the IR image is formed long along a straight line or arc intersecting the FOE as if blown out from the FOE, or is arranged along the straight line or arc. In the case where the IR camera 120 is set on a moving body such as a car, the movement of the dust is small enough compared to the movement of the IR camera 120, and it can be assumed to be stationary. For this reason, in the IR image, the dust is reflected as being blown out from the FOE.

[0162] In addition, in the depth acquisition device 1 in the present embodiment, from the viewpoint of noise removal, the images of each of the IR image and the BW image are obtained by repeatedly exposing and blocking the light. For this reason, the same dust is reflected in multiple locations in one frame by exposures at different times within the frame period. On the image, since the dust moves as if blown out from the FOE, the respective dust regions and the FOE located at the multiple locations are arranged on a straight line. In addition, when the IR camera 120 that photographs the same dust moves slowly, the multiple dust regions generated by multiple exposures in one frame period overlap in the IR image. As a result, a dust region with a shape like a tail is formed. In this case, there is an FOE in the direction of the tail viewed from the dust region. The dust detection unit 112 detects the dust region using these properties.

[0163] That is, the dust detection unit 112 in this embodiment detects a high-brightness area with a brightness above the first threshold in the IR image as a dust area when the first condition is satisfied. Specifically, the first condition is a condition that the center of gravity of the high-brightness area is arranged on a straight line or arc that intersects with the center of gravity of at least two other high-brightness areas different from the high-brightness area in the IR image and the FOE (Focus of Expansion) of the IR image or the BW image. Alternatively, the first condition is a condition that the main axis of the high-brightness area or an extension of the main axis intersects with the FOE of the IR image or the BW image. In addition, the main axis is the axis of the tail when the high-brightness area is in the shape of a tail. In this way, the dust area can be detected with high precision.

[0164] In addition, the dust detection unit 112 can further detect the dust area by using the property that even if the dust area is observed in the IR image, the dust area cannot be observed in the BW image. That is, the dust detection unit 112 can detect the high-brightness area as a dust area when the high-brightness area further satisfies the second condition. For example, the second condition is the condition that the brightness of the position in the BW image corresponding to the center of gravity of the high-brightness area in the IR image is less than the second threshold. Alternatively, the second condition is the condition that the correlation coefficient between the brightness in the high-brightness area of ​​the IR image and the brightness in the area of ​​the BW image corresponding to the high-brightness area is less than the third threshold. In addition, the area of ​​the BW image corresponding to the high-brightness area of ​​the IR image is a region that is at the same spatial position as the high-brightness area of ​​the IR image and has the same shape and size as the high-brightness area of ​​the IR image. Thereby, the dust area can be further detected with high accuracy.

[0165] The first depth estimating unit 111 a and the second depth estimating unit 111 b have a function as the depth estimating unit 111 described above.

[0166] The first depth estimation unit 111a estimates the depth at each position in the IR image based on the IR image acquired in accordance with the timing of the irradiation of the infrared light from the light source 101. The first depth estimation unit 111a outputs information indicating the estimated depth at each position in the IR image as first depth information. That is, the first depth estimation unit 111a estimates the first depth information indicating the depth at each position in the IR image.

[0167] The second depth estimation unit 111b corrects the first depth information based on the dust region in the BW image and the IR image. Thus, the depth of the dust region is corrected among the depths at each position in the IR image represented by the first depth information. The second depth estimation unit 111b outputs information indicating the corrected depth at each position in the dust region as the second depth information. That is, the second depth estimation unit 111b estimates the second depth information indicating the corrected depth at each position in the dust region by correcting the depth at each position in the dust region represented by the first depth information based on the BW image.

[0168] The output unit 118 replaces the depth at each position in the dust region indicated by the first depth information with the corrected depth at each position in the dust region indicated by the second depth information. Thus, the third depth information is generated, which includes the depth at each position outside the dust region of the IR image indicated by the first depth information and the corrected depth at each position in the dust region of the IR image indicated by the second depth information. The output unit 118 outputs the third depth information.

[0169] Thus, the third depth information indicates the depth obtained from the IR image as the depth outside the dust region of the IR image, and indicates the depth obtained from the IR image and corrected based on the BW image as the depth of the dust region of the IR image. Therefore, in the present embodiment, even when there is a dust region in the IR image, the overall depth of the IR image can be appropriately estimated.

[0170] Fig. 9A An example of an IR image is shown. Fig. 9B An example of a BW image is shown.

[0171] like Fig. 9B As shown in FIG. 1 , in the BW image, for example, a BW camera 103 mounted on a vehicle captures a scene around a road on which the vehicle is traveling, which is far from the BW camera 103. On the road, for example, dust is flying due to the traveling of the vehicle. Therefore, if the IR camera 102 is directed at the same viewpoint and at the same time as the BW camera 103, Fig. 9B The same scene as shown in the picture is recorded. Fig. 9A IR image shown.

[0172] In the IR image obtained as described above, Fig. 9A As shown in the figure, there are areas with high brightness. These areas include areas where dust is reflected, that is, dust areas. For example, it is confirmed that the dust area exists in the central part and the right part of the IR image. On the other hand, dust is not reflected in the BW image. This is because, in the shooting of the IR image, the infrared light irradiated from the light source 101 is diffusely reflected by the dust located near the IR camera 102 and the BW camera 103, but in the shooting of the BW image, the influence of the diffuse reflection is small. For this reason, the dust detection unit 112 detects the dust area by utilizing the property that the dust area is observed in the IR image but not in the BW image.

[0173] Fig.10 An example of a binarized image obtained by binarizing an IR image is shown.

[0174] The high brightness area detection unit 116 Fig. 9A In the IR image shown in FIG. 1 , the area with brightness above the first threshold is detected as a high brightness area. That is, the high brightness area detection unit 116 binarizes the brightness at each position (i.e., each pixel) in the IR image. As a result, for example, Fig.10 As shown in the figure, a white area and a black area are generated ( Fig.10 The binary image of the shaded area in the figure.

[0175] Fig.11 An example of a candidate for a dust region in an IR image is shown.

[0176] The dust detection unit 112 first detects the white area in the binary image, that is, the center of gravity of the high-brightness area. Such center of gravity detection is performed separately for all high-brightness areas in the binary image. Next, the dust detection unit 112 determines whether the brightness of the position in the BW image corresponding to the center of gravity of the high-brightness area (hereinafter referred to as the center of gravity corresponding position) is less than the second threshold value for all high-brightness areas in the binary image. That is, the dust detection unit 112 determines whether the high-brightness area satisfies the above-mentioned second condition. As a result, the dust detection unit 112 determines the high-brightness area corresponding to the center of gravity corresponding position judged to have a brightness less than the second threshold value as a candidate for the dust area. For example, Fig.11 As shown, the dust detection unit 112 detects the five high-luminance regions A to E as candidates for dust regions. That is, the IR image or the binary image is region-divided into the five dust candidate regions A to E and a non-dust region that is not a dust region.

[0177] Fig.12 An example of FOE detected on a BW image is shown.

[0178] The FOE detection unit 117 includes, for example, Fig. 9B The optical flow is detected from multiple frames of the BW image shown, and the FOE is detected by obtaining the intersection of the optical flows using a robust estimation such as RANSAC (Random Sample Consensus).

[0179] Fig.13 An example of a main axis for detecting dust candidate areas A to E is shown.

[0180] The dust detection unit 112 detects Fig.11 The main axis is detected for each of the dust candidate regions A to E detected as shown. Specifically, the dust detection unit 112 performs principal component analysis on each pixel of the dust candidate region in the IR image or the binary image to detect the first principal component axis as the main axis of the dust candidate region.

[0181] Fig.14 An example showing the dust area and the non-dust area.

[0182] The dust detection unit 112 determines whether the main axis of the dust candidate area A to E in the IR image or the binary image or the extension line of the main axis intersects with the FOE detected by the FOE detection unit 117. That is, the dust detection unit 112 determines whether the dust candidate area meets the first condition mentioned above. Then, the dust detection unit 112 detects the dust candidate area with the main axis or extension line intersecting with the FOE as a dust area, and detects the dust candidate area with the main axis or extension line not intersecting with the FOE as a non-dust area. For example, Fig.14 As shown, the extension lines of the main axes of the dust candidate areas B, C, and E intersect with the FOE, while the main axes and extension lines of the dust candidate areas A and D do not intersect with the FOE. Therefore, the dust detection unit 112 detects the dust candidate areas B, C, and E as dust areas, and detects the dust candidate areas A and D as non-dust areas.

[0183] Fig.15A Shows other examples of IR images. Fig. 15B Represents other examples of BW images.

[0184] Fig. 15B The BW image shown is obtained by Fig. 9B The BW images shown are images obtained by taking pictures at different timings. Fig. 15B In the BW image shown, Fig. 9B In the example shown, the BW camera 103 installed on the vehicle captures the scene near the BW camera 103 around the road on which the vehicle is traveling. In addition, on the road, for example, dust is flying due to the driving of the vehicle. Therefore, if the IR camera 102 captures the same scene at the same viewpoint and time as the BW camera 103, Fig. 15B The same scene as shown in the scene is photographed, and the Fig.15A IR image shown.

[0185] In the IR image thus obtained, Fig.15A As shown in FIG. 1 , there are areas with high brightness. These areas include areas where dust is reflected, that is, dust areas. For example, it is confirmed that the dust area exists in the left part of the IR image. On the other hand, Fig. 9B In the example shown, the dust is also not reflected in the BW image.

[0186] Fig.16 An example of a binarized image obtained by binarizing an IR image is shown.

[0187] The high brightness area detection unit 116 Fig.15A In the IR image shown in FIG. 1 , the area with brightness above the first threshold is detected as a high brightness area. That is, the high brightness area detection unit 116 binarizes the brightness at each position (i.e., each pixel) in the IR image. As a result, for example, Fig.16 As shown in the figure, a white area and a black area are generated ( Fig.16 The binary image of the shaded area in the figure.

[0188] Fig.17 An example of a candidate for a dust region in an IR image is shown.

[0189] The dust detection unit 112 first detects the white area in the binary image, that is, the center of gravity of the high-brightness area. Such center of gravity detection is performed separately for all high-brightness areas in the binary image. Next, the dust detection unit 112 determines whether the brightness of the position in the BW image corresponding to the center of gravity of the high-brightness area (that is, the position corresponding to the center of gravity) is less than the second threshold value for all high-brightness areas in the binary image. That is, the dust detection unit 112 determines whether the high-brightness area satisfies the second condition mentioned above. As a result, the dust detection unit 112 determines the high-brightness area corresponding to the position corresponding to the center of gravity determined to have a brightness less than the second threshold value as a candidate for the dust area. For example, Fig.17 As shown, the dust detection unit 112 detects the region group A including a plurality of high brightness regions, the high brightness region B, the region group C including a plurality of high brightness regions, and the high brightness region D as candidates for dust regions. That is, the IR image or the binary image is region-divided into dust candidate regions and non-dust regions that are not dust regions. The dust candidate regions include the regions included in the region group A, the region B, the regions included in the region group C, and the region D.

[0190] Fig.18 An example of FOE detected on a BW image is shown.

[0191] The FOE detection unit 117 includes, for example, Fig. 15B The optical flow is detected from multiple frames of the BW image shown, and the FOE is detected by obtaining the intersection of the optical flows using a robust estimation method such as RANSAC (Random Sample Consensus).

[0192] Fig.19 An example of arrangement of each dust candidate area is shown.

[0193] For example, when the speed of the vehicle equipped with the IR camera 102 is high, as described above, the same dust is reflected in multiple locations in the IR image of one frame by exposures at different times within the frame period. Since the dust moves as if blown out from the FOE, the respective dust areas and FOEs located at the multiple locations are arranged on a straight line. In addition, when a wide-angle lens or a fisheye lens is used in the lens of the IR camera 102, the distortion of the lens is large. When the influence of the lens distortion is large, the visible movement on the screen, that is, the light flow, does not intersect at one point, and for this reason, each dust area and FOE also exist on a straight line. In this way, when the lens distortion is large, each dust area and FOE are arranged on an arc due to the influence of the lens distortion.

[0194] exist Fig.19 In the example shown, the dust detection unit 112 determines Fig.17As shown in the figure, the dust candidate areas of the area group A, the dust candidate areas B, the dust candidate areas of the area group C, and the dust candidate areas D are respectively arranged on the arc together with the FOE. That is, when the dust detection unit 112 judges one dust candidate area, it determines whether the center of gravity of the dust candidate area of ​​the judgment object is arranged on the arc intersecting with the FOE of the BW image at the center of gravity of each of at least two other dust candidate areas different from the dust candidate area of ​​the judgment object. That is, the dust detection unit 112 determines whether the dust candidate area, i.e., the high-brightness area, satisfies the first condition mentioned above. If the center of gravity of the dust candidate area of ​​the judgment object is arranged on the arc, the dust detection unit 112 detects the dust candidate area as a dust area. On the contrary, if the center of gravity of the dust candidate area of ​​the judgment object is not arranged on the arc, the dust detection unit 112 detects the dust candidate area as a non-dust area.

[0195] Therefore Fig.19 In the example shown, since each of the plurality of dust candidate regions included in the region group A and the FOE are arranged on an arc, the dust detection unit 112 detects the plurality of dust candidate regions included in the region group A as dust regions. Similarly, since each of the plurality of dust candidate regions included in the region group B and the FOE are arranged on an arc, the dust detection unit 112 also detects the plurality of dust candidate regions included in the region group B as dust regions. On the other hand, since the dust candidate regions B and D are not arranged on an arc intersecting with at least two other dust candidate regions and the FOE, the dust detection unit 112 detects the dust candidate regions B and D as non-dust regions.

[0196] In addition, when the lens distortion is large, the dust detection unit 112 may perform distortion correction processing on the captured BW image and IR image. For example, the dust detection unit 112 may perform distortion correction processing using a camera calibration method disclosed in a non-patent document (R. Tsai, "A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses", IEEE Journal on Robotics and Automation, Vol. 3, Iss. 4, pp. 323-344, 1987.). In this case, if the dust candidate region to be determined is arranged on a straight line intersecting with at least two other dust candidate regions and FOE in the IR image or the binarized image to which the distortion correction processing is performed, the dust detection unit 112 detects the dust candidate region to be determined as a dust region.

[0197] Fig. 20 The simulation results of the depth acquisition device 1 are shown.

[0198] The depth acquisition device 1 acquires the depth through the image captured by the BW camera 103. Fig. 20 The BW image shown in (a) is obtained by the IR camera 102. Fig. 20 The BW image and the IR image are images obtained by photographing the same scene at the same viewpoint and at the same time. Fig. 20 In the example shown in (b), there are several dust regions in the IR image.

[0199] The first depth estimation unit 111a estimates the depth from the IR image and generates Fig. 20 The first depth information shown in (c) of FIG. 1 is expressed as a first depth image that expresses the depth at each position in the IR image by brightness. In the first depth image, the depth of the dust region is not properly expressed.

[0200] The second depth estimation unit 111b corrects the inappropriate depth in the dust area. Fig. 20 As shown in (e) of FIG. 1 , the output unit 118 generates the third depth information indicating the corrected depth of the dust area and the depth of the non-dust area. The third depth information is also expressed as a third depth image in which the depth is expressed by brightness, similar to the first depth information. In addition, the second depth estimation unit 111b can also correct the depth of the non-dust area in the first depth image based on the corresponding area of ​​the BW image.

[0201] In this way, in the depth acquisition device 1 of the present embodiment, the third depth image can be made close to Fig. 20 The depth image of the correct solution shown in (d).

[0202] [Specific processing flow of the depth acquisition device]

[0203] Fig.21 Yes means Figure 8 The flowchart of the overall processing operation of the depth acquisition device 1 is shown.

[0204] (Step S31)

[0205] First, the high-luminance region detection unit 116 detects a high-luminance region from the IR image.

[0206] (Step S32)

[0207] The dust detection unit 112 determines whether the high-luminance region is a dust candidate region, thereby dividing the IR image into a dust candidate region and a non-dust region.

[0208] (Step S33)

[0209] The FOE detection unit 117 detects FOE using the BW image.

[0210] (Step S34)

[0211] The dust detection unit 112 detects the dust region based on the dust candidate region and the FOE. Thus, the IR image is divided into the dust region and the non-dust region.

[0212] (Step S35)

[0213] The first depth estimating unit 111 a generates first depth information from the IR image using, for example, TOF.

[0214] (Step S36)

[0215] The second depth estimating unit 111 b generates second depth information indicating the depth of the dust region based on the first depth information of the IR image and the BW image.

[0216] (Step S37)

[0217] The output unit 118 generates the third depth information by replacing the depth of the dust region indicated by the first depth information with the depth indicated by the second depth information.

[0218] Fig. 22 Yes means Fig.21 A flowchart of an example of detailed processing of steps S31 to S34.

[0219] (Step S41)

[0220] First, the high-brightness area detection unit 116 determines whether the brightness at each position in the IR image is above the first threshold value. Here, for example, if the IR image is a 12-bit grayscale image, the first threshold value may be 256. Of course, the first threshold value may also be a value that varies according to environmental conditions or the settings of the IR camera 102. For example, when shooting a dark scene such as at night, since the overall brightness of the IR image becomes lower, the first threshold value may be a smaller value than when shooting a bright scene during the day. In addition, when the exposure time of the IR camera 102 is long, since the overall brightness of the IR image becomes higher, the first threshold value may be a larger value than when the exposure time is short.

[0221] (Step S42)

[0222] If it is determined that the brightness at any position is not equal to or greater than the first threshold (step S41: No), the high brightness region detection unit 116 determines that dust is not reflected in the IR image. That is, the entire IR image is determined to be a non-dust region.

[0223] (Step S43)

[0224] On the other hand, if it is determined that the brightness at any position is above the first threshold value (step S41 "Yes"), the high-brightness area detection unit 116 performs area segmentation on the IR image. That is, the high-brightness area detection unit 116 divides the IR image into at least one high-brightness area and an area outside the high-brightness area. In this area segmentation, brightness-based techniques such as SuperPixel can be used. In addition, the high-brightness area detection unit 116 can perform filtering processing using the size of the area in the area segmentation. For example, if the number of pixels in the high-brightness area is less than a predetermined number, the high-brightness area detection unit 116 can also delete the high-brightness area. That is, even if the high-brightness area detection unit 116 detects a high-brightness area, if the number of pixels in the area is small, the high-brightness area is reclassified into an area outside the high-brightness area.

[0225] (Step S44)

[0226] Next, the dust detection unit 112 detects the centroid of at least one high brightness region detected by the region segmentation in step S43. Specifically, the dust detection unit 112 detects the centroid of the high brightness region by calculating the average values ​​of the X-axis coordinate positions and the Y-axis coordinate positions of the plurality of pixels included in the high brightness region.

[0227] (Step S45a)

[0228] The dust detection unit 112 determines whether the brightness of the position in the BW image corresponding to the center of gravity of the high-brightness area (i.e., the position corresponding to the center of gravity) is less than the second threshold value. That is, the dust detection unit 112 determines whether the high-brightness area satisfies the second condition. If it is determined that it is not less than the second threshold value (step S45a "No"), the dust detection unit 112 determines the high-brightness area as a non-dust area (step S42). That is, in this case, it is estimated that the subject with a high reflectivity of light is reflected in each of the high-brightness area of ​​the IR image and the area corresponding to the high-brightness area in the BW image. Therefore, in this case, the high-brightness area is determined to be a non-dust area. In addition, for example, if the BW image is a 12-bit grayscale image, the second threshold value may also be about 20,000. Of course, the second threshold value may be a value that changes according to environmental conditions or the settings of the BW camera 103. For example, when photographing a dark scene such as at night, the overall brightness of the BW image becomes low, so the second threshold value may be a smaller value than when photographing a bright scene during the day. In addition, when the exposure time of the BW camera 103 is long, the overall brightness of the BW image becomes high, so the second threshold value may be a larger value than when the exposure time is short.

[0229] (Step S46)

[0230] On the other hand, when the dust detection unit 112 determines that the brightness of the position corresponding to the center of gravity is lower than the second threshold value (Yes in step S45a), the FOE detection unit 117 detects the FOE based on the BW image.

[0231] (Step S47a)

[0232] The dust detection unit 112 determines whether the center of gravity and FOE of more than three dust candidate areas detected in step S44 are arranged on a straight line. That is, the dust detection unit 112 determines whether the dust candidate area meets the first condition mentioned above. Specifically, the dust detection unit 112 fits the center of gravity of each of the more than three dust candidate areas and the straight line intersecting with the FOE, and determines whether the error (i.e., the distance) between the straight line and each center of gravity is below the allowable value. In this way, it is determined whether the center of gravity and FOE of each of the more than three dust candidate areas are arranged on a straight line. If the error is below the allowable value, it is determined that the center of gravity and FOE of these dust candidate areas are arranged on a straight line. If the error is not below the allowable value, it is determined that the center of gravity and FOE of these dust candidate areas are not arranged on a straight line.

[0233] (Step S50)

[0234] When the dust detection unit 112 determines that the centroid and FOE of each dust candidate region are arranged on a straight line (Yes in step S47a), the dust detection unit 112 determines that the dust candidate region is a dust region.

[0235] (Step S48)

[0236] On the other hand, when dust detection unit 112 determines that the centroids and FOEs of three or more dust candidate regions are not arranged on a straight line (No in step S47a), it detects the main axis of each dust candidate region.

[0237] (Step S49)

[0238] Next, the dust detection unit 112 determines whether the main axis or its extension line of each dust candidate area detected in step S48 intersects with the FOE. That is, the dust detection unit 112 determines whether the dust candidate area satisfies other first conditions that are different from the first condition of step S47a. Here, if the dust detection unit 112 determines that the main axis or its extension line intersects with the FOE (step S49 "Yes"), the dust candidate area with the main axis is judged as a dust area (step S50). On the other hand, if the dust detection unit 112 determines that the main axis or its extension line does not intersect with the FOE (step S49 "No"), the dust candidate area with the main axis is judged as a non-dust area (step S42).

[0239] In such a method, an IR image and a BW image with substantially the same viewpoint position are required. In the depth acquisition device 1 of the present embodiment, the filter used in each pixel is set to be either an IR filter or a BW filter. Figure 2 As shown, the first pixels 21 having IR filters and the second pixels 22 having BW filters are arranged alternately in the column direction. This allows the IR image and the BW image to be acquired at substantially the same viewpoint and time, and thus the dust region can be appropriately determined.

[0240] Fig.23 Yes means Fig.21 A flowchart of another example of the detailed processing of steps S31 to S34 is shown. Fig.23 The flowchart shown replaces Fig. 22 The flowchart includes step S47a and step S47b among the steps.

[0241] (Step S47b)

[0242] For example, when the distortion of the lens of the IR camera 102 is large, as described above, each dust region and FOE is not arranged on a straight line but on an arc corresponding to the lens distortion.

[0243] Therefore, the dust detection unit 112 can determine whether the center of gravity and FOE of more than three dust candidate areas detected in step S44 are arranged on a circular arc. Specifically, the dust detection unit 112 obtains an approximate curve of the center of gravity and FOE of each of the more than three dust candidate areas, and determines whether the error (i.e., the distance) between the approximate curve and each center of gravity is below the allowable value. In this way, it is determined whether the center of gravity and FOE of each of the more than three dust candidate areas are arranged on a circular arc. That is, if the error is below the allowable value, it is determined that the center of gravity and FOE of these dust candidate areas are arranged on a circular arc, and if the error is not below the allowable value, it is determined that the center of gravity and FOE of these dust candidate areas are not arranged on a circular arc. In addition, the above-mentioned approximate curve is expressed with a number of times less than the number of more than three dust candidate areas set as the determination object.

[0244] In addition, in step S46, the FOE detection unit 117 detects optical flows from a plurality of IR images instead of the BW image, obtains the intersection of the optical flows by robust estimation such as RANSAC, and detects the intersection as the FOE. Alternatively, the FOE detection unit 117 may detect the motion of the IR camera 102 or the BW camera 103, and detect the FOE by calculation using the motion and the internal parameters of the IR camera 102 or the BW camera 103.

[0245] Fig.24 It means to replace Fig.21 A flowchart of an example of the processing of steps S31 to S34. Fig.24 The flowchart shown omits Fig.21 In other words, in step S32 Fig.24 In the flowchart shown, Fig. 22 4. The flowchart of step S45a is shown in FIG.

[0246] That is, in Figure 21 to Figure 23 In the example shown in FIG. 1 , the dust detection unit 112 determines whether the high-brightness area is a dust candidate area by using the property that dust can be observed in the IR image but cannot be observed in the BW image. Fig. 22 as well as Fig.23 As shown in each step S45a of FIG. 1 , the dust detection unit 112 determines whether the high-brightness area is a dust candidate area based on the brightness of the position corresponding to the center of gravity in the BW image. Fig.24 As shown in the flowchart, the dust detection unit 112 performs the determination in step S45a. In this case, any high-brightness region obtained by the region division in step S43 is handled as a dust candidate region.

[0247] Fig.25 Yes means Fig.21 A flowchart of another example of the detailed processing of steps S31 to S34 is shown. Fig.25The flowchart shown replaces Fig. 22 The flowchart includes step S45a and step S45b among the steps.

[0248] In order to determine the dust candidate area, the property of dust that can be observed in the IR image but not in the BW image can be more actively utilized. For example, the correlation coefficient between the IR image and the BW image can be used as this property. As mentioned above, if there is dust, the dust is reflected in the IR image but not in the BW image. That is, in the BW image, since a farther object is reflected, the images in the IR image and the BW image with the same viewpoint positions are greatly different. For this reason, by using the correlation coefficient between the IR image and the BW image to determine whether the reflected objects are equal or unequal, it is possible to determine whether each high-brightness area is a dust candidate area.

[0249] (Step S45b)

[0250] In step S45b, the dust detection unit 112 calculates the correlation coefficient of brightness between the high brightness area of ​​the IR image and the area in the BW image corresponding to the high brightness area (i.e., the corresponding area) for at least one high brightness area obtained by the area segmentation in step S43. The brightness of each pixel of the IR image and the BW image is arranged in a vector form in each area, and the inner product value is calculated and normalized by the number of pixels to obtain the correlation coefficient. That is, the dust detection unit 112 normalizes the inner product value of the vector of the brightness of each pixel in the high brightness area of ​​the IR image and the vector of the brightness of each pixel in the corresponding area of ​​the BW image. In this way, the correlation coefficient for the high brightness area is calculated.

[0251] Then, the dust detection unit 112 determines whether the calculated correlation coefficient is less than the third threshold value. That is, the dust detection unit 112 determines whether the high-brightness area satisfies the second condition mentioned above. Here, when it is not less than the third threshold value (step S45b "No"), since the same subject is reflected in the high-brightness area of ​​the IR image and the corresponding area of ​​the BW image, there is a possibility that the correlation coefficient is high. Therefore, in this case, the dust detection unit 112 determines that the high-brightness area is not a dust area (step S42). On the other hand, when the correlation coefficient is less than the third threshold value (step S45b "Yes"), since different subjects are reflected in the high-brightness area of ​​the IR image and the corresponding area of ​​the BW image, there is a possibility that the correlation coefficient is low. Therefore, in this case, the dust detection unit 112 determines that the high-brightness area is a dust candidate area (step S50).

[0252] In addition, learning processing can be used to distinguish between dust areas and non-dust areas. In the learning process, for example, deep learning or other processing can be used. In this case, for learning, an IR image and a BW image, and a correct image in which the IR image is divided into dust areas and non-dust areas are prepared in advance. Next, the IR image and the BW image as input are given to the learning model. Then, the learning model is learned so that the output from the learning model for the input is consistent with the correct image. For example, the learning model is a neural network. The output from the learning model is an image that represents a numerical value "0" or a numerical value "1" for each pixel, where the numerical value "0" indicates that the pixel belongs to the non-dust area, and the numerical value "1" indicates that the pixel belongs to the dust area.

[0253] The dust detection unit 112 distinguishes between dust areas and non-dust areas by using the learning model learned in advance. That is, the dust detection unit 112 inputs the IR image and the BW image into the learning model as input. Then, the dust detection unit 112 distinguishes the area containing pixels corresponding to the value "0" output from the learning model as a non-dust area. Furthermore, the dust detection unit 112 distinguishes the area containing pixels corresponding to the value "1" output from the learning model as a dust area.

[0254] Through the above-described processing, the dust detection unit 112 divides the captured IR image into a dust region where dust is reflected and a non-dust region where dust is not reflected.

[0255] <Depth correction processing>

[0256] The second depth estimating unit 111 b generates the second depth information using the BW image, the first depth information, and the dust region (that is, the above-mentioned region determination result).

[0257] As described above, the noise caused by dust is a phenomenon caused by diffuse reflection of infrared light irradiated from the light source 101. Therefore, dust reflected as noise in the IR image is often not reflected in the BW image. Therefore, only the dust area is not the first depth information obtained from the IR image, and by correcting the first depth information using the BW image, the second depth information that is not affected by the dust reflected in the IR image can be obtained.

[0258] In acquiring the second depth information, a guided filter as a type of image correction filter can be used. The guided filter is disclosed in a non-patent document (Kaiming He, Jian Sun and Xiaoou Tang, "Guided Image Filtering", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 35, Iss. 6, PP. 1397-1409, 2013.). The guided filter is a filter that corrects the object image by using the correlation between the object image and the reference image. In the guided filter, the reference image I and the object image p are assumed to be represented by parameters a and b as shown in the following (Formula 2).

[0259] [Mathematical formula 1]

[0260]

[0261] Here, q represents an output image obtained by correcting the target image p, i represents the number of each pixel, and ωk represents the surrounding area of ​​the pixel k. Parameters a and b are expressed by the following (Formula 3).

[0262] [Mathematical formula 2]

[0263]

[0264] Here, ε is a regularization parameter. In addition, μ and σ are the mean and variance within the block of the reference image, and are calculated using the following (Equation 4).

[0265] [Mathematical formula 3]

[0266]

[0267] However, in order to suppress the noise included in the obtained parameters a and b, the output is obtained as shown in the following (Equation 5) using the averaged parameters.

[0268] [Formula 4]

[0269]

[0270] In this embodiment, the second depth estimation unit 111b corrects the object image, i.e., the first depth information (or the first depth image), by providing a BW image as a reference image. The second depth information is thereby generated or acquired. In the generation of such second depth information, an IR image and a BW image having substantially the same viewpoint position are required. In the depth acquisition device 1 of this embodiment, for each pixel, the filter used in the pixel is set to be either an IR filter or a BW filter. That is, Figure 2 As shown, the first pixels 21 having IR filters and the second pixels 22 having BW filters are arranged alternately in the column direction. Thus, since an IR image and a BW image of substantially the same viewpoint can be obtained, appropriate second depth information can be obtained.

[0271] Of course, the second depth estimation unit 111b may use processing other than guided filtering. For example, the second depth estimation unit 111b may use processing such as bilateral filtering (non-patent literature: C. Tomasi, R. Manduchi, "Bilateral filtering for gray and color images", IEEE International Conference on Computer Vision (ICCV), pp. 839-846, 1998) or Mutual-Structure for Joint Filtering (non-patent literature: Xiaoyong Shen, Chao Zhou, Li Xu and Jiaya Jia, "Mutual-Structure for Joint Filtering", IEEE International Conference on Computer Vision (ICCV), 2015).

[0272] As described above, in this embodiment, the first depth information is used in the area determined as not reflecting dust (i.e., non-dust area), and the second depth information is used in the area reflecting dust (i.e., dust area). Thus, even if dust is reflected in the IR image, more accurate depth information can be obtained.

[0273] (Variation Example)

[0274] In the above-described embodiment, a filter such as a guided filter is used to generate the second depth information. However, the second depth information may be generated using a learning model.

[0275] For example, deep learning as a learning process can be used as in the non-patent literature (Shuran Song, Fisher Yu, Andy Zeng, Angel X. Chang, Manolis Savva and Thomas Funkhouser, "Semantic Scene Completion from a Single Depth Image", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 190-198, 2017.). That is, the learning model can be made to learn so that if a BW image and the first depth information are input, the second depth information is output. In the above non-patent literature, a network is proposed that interpolates the defective area of ​​the depth information if the depth information and color image containing the defective area are input. The second depth estimation unit 111b in this embodiment gives the IR image, the BW image and the first depth information to the same network (i.e., the learning model) as the non-patent literature, and further gives the dust area detected by the dust detection unit 112 as a mask image of the defective area. As a result, a higher-precision second depth information can be obtained from the network.

[0276] Fig.26 This is a block diagram showing an example of the functional configuration of the depth acquisition device 1 in this modification.

[0277] The depth acquisition device 1 in this modified example includes Figure 8 Each of the components shown further includes a learning model 104 including a structure such as a neural network.

[0278] The second depth estimation unit 111 b inputs three types of data, namely, the IR image, the BW image, and the first depth information, to the learning model 104 , and generates the second depth information using the dust region as a mask region for correction.

[0279] In the learning of the learning model 104, in addition to the IR image, the BW image and the first depth information, the correct depth image is prepared in advance. Next, the IR image, the BW image, the first depth information, and the mask image of the designated dust area are provided as inputs to the learning model 104. Then, the learning model 104 is made to learn so that the output from the learning model 104 for the input is consistent with the correct depth image. In addition, the mask image is randomly given during the learning. The second depth estimation unit 111b uses the learning model 104 that has been learned in advance in this way. That is, the second depth estimation unit 111b can obtain the second depth information output from the learning model 104 by inputting the IR image, the BW image, the first depth information, and the mask image of the designated dust area into the learning model 104.

[0280] Thus, in this modification, the second depth estimation unit 111b estimates the depth information indicating the depth at each position in the IR image, and corrects the depth at each position in the dust region indicated by the depth information by inputting the IR image, the BW image, the dust region, and the depth information into the learning model. Therefore, if the learning model is pre-learned so that the correct depth at each position in the dust region is outputted in response to the input of the IR image, the BW image, the dust region, and the depth information, the depth information estimated from the IR image can be appropriately corrected. That is, the depth at each position in the dust region indicated by the depth information can be appropriately corrected.

[0281] As described above, the second depth estimation unit 111b may use deep learning. In this case, it is not necessary to output the dust region, but the second depth information may be directly generated by deep learning.

[0282] Fig. 27 This is a block diagram showing another example of the functional configuration of the depth acquisition device 1 in this modification.

[0283] The depth acquisition device 1 in this modification does not include Fig.26 The dust detector 112 , the high-luminance region detector 116 , and the FOE detector 117 among the components shown in the figure include components other than these.

[0284] In the learning of the learning model 104, Fig.26Similarly to the example shown, in addition to the IR image, the BW image and the first depth information, a correct depth image is prepared in advance. Next, the IR image, the BW image and the first depth information are provided as inputs to the learning model 104. Then, the learning model 104 is made to learn so that the output from the learning model 104 for the input is consistent with the correct depth image. As the learning model 104, a VGG-16 network with skip combining added can be used as described in the non-patent literature (Caner Hazirbas, Laura Leal-Taixe and Daniel CremersC.Hazirbas, "Deep Depth From Focus", InArXiv preprint arXiv, 1704.01085, 2017.). The number of channels of the learning model 104 is changed so that the IR image, the BW image and the first depth information are given as inputs to the learning model 104. By using the learning model 104 that has been learned in advance in this manner, the second depth estimating unit 111 b can easily acquire the second depth information from the learning model 104 by inputting the IR image, the BW image, and the first depth information into the learning model 104 .

[0285] Right now, Fig. 27 The depth acquisition device 1 shown in the figure includes a memory and a processor 110. Fig. 27 Not shown in the figure, but can be Figure 5 As shown in the figure, the depth acquisition device 1 is equipped. The processor 110 obtains timing information indicating the timing at which the light source 101 irradiates the subject with infrared light. Next, the processor 110 obtains an IR image, wherein the IR image is obtained by infrared light-based imaging of a scene including the subject corresponding to the timing indicated by the timing information, and is stored in a memory. Next, the processor 110 obtains a BW image, wherein the BW image is obtained by visible light-based imaging of a scene substantially the same as the IR image and imaging at substantially the same viewpoint and imaging time as the IR image, and is stored in a memory. Then, the first depth estimation unit 111a of the processor 110 estimates depth information indicating the depth at each position in the IR image. The second depth estimation unit 111b corrects the depth at each position in the dust region of the IR image indicated by the depth information by inputting the IR image, the BW image, and the depth information into the learning model 104.

[0286] Therefore, if the learning model 104 is pre-learned so as to output the correct depth at each position in the dust region of the IR image in response to the input of the IR image, the BW image, and the depth information, the depth information estimated from the IR image can be appropriately corrected. In other words, the depth at each position in the dust region indicated by the depth information can be appropriately corrected without detecting the dust region.

[0287] As described above, in the depth acquisition device 1 in the present embodiment and its modified example, even when there is a dust region in the IR image, the appropriate depth at each position in the dust region can be acquired by using the image of the corresponding region of the BW image.

[0288] In addition, in each of the above embodiments, each component can be formed by dedicated hardware, or can be implemented by executing a software program suitable for each component. Each component can be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software that implements the depth acquisition device of the above embodiments and modified examples causes a computer to execute Figure 6 , Figure 7 ,as well as Figure 21 to Figure 25 Each step contained in any one of the flowcharts.

[0289] The depth acquisition device involved in one or more schemes is described above based on the embodiment and its variants, but the present disclosure is not limited to the embodiment and its variants. As long as it does not deviate from the main purpose of the present disclosure, the scheme obtained by implementing various modifications thought of by those skilled in the art to the embodiment and the variant, and the form constructed by combining the constituent elements in the embodiment and the variant can also be included in the scope of the present disclosure.

[0290] For example, in the above-mentioned embodiment and its modified example, the dust detection unit 112 detects the dust area, but as long as it is an area where an object is reflected as noise like dust, an area other than the dust area may be detected. For example, the dust detection unit 112 may detect a rain area where raindrops are reflected, or a snow area where snow particles are reflected. In an environment where it is drizzling, if the size of the raindrops is sufficiently small relative to the resolution of the BW image obtained by the BW camera 103 or the BW image acquisition unit 115, the raindrops will not be captured in the BW image. However, in the IR image obtained by the IR camera 102 or the IR image acquisition unit 114, the infrared light from the light source 101 is reflected by the raindrops and observed as high brightness. For this reason, in the first depth information or the first depth image generated by the first depth estimation unit 111a, the depth of the rain area becomes inappropriate. Similarly, in a snowy environment, when the size of snow particles is sufficiently small relative to the resolution of the BW image acquired by the BW camera 103 or the BW image acquisition unit 115, the snow particles will not be captured in the BW image. However, in the IR image acquired by the IR camera 102 or the IR image acquisition unit 114, the infrared light from the light source 101 is reflected by the snow particles and observed as high brightness. For this reason, the depth of the snow area becomes inappropriate in the first depth information or the first depth image generated by the first depth estimation unit 111a. For this reason, the dust detection unit 112 detects the rain area or the snow area using the same method as the dust area detection. As a result, the second depth estimation unit 111b generates the second depth information using the BW image, the first depth information, and the area detected by the dust detection unit 112 (i.e., the rain area or the snow area). In this way, the second depth information that is not affected by rain or snow can be acquired. In addition, the dust in the present disclosure includes solid particles such as dust, but is not limited to solid particles, and may also include liquid particles. For example, the dust in the present disclosure may include at least one of dust, raindrops, and snow particles.

[0291] In addition, in the present disclosure, all or part of a component, device, or Figure 1 , Figure 4 , Figure 5 , Figure 8 , Fig.26 as well as Fig. 27All or part of the functional blocks of the block diagram shown can be executed by a semiconductor device, one or more electronic circuits including a semiconductor integrated circuit (IC) or LSI (large scale integration). LSI or IC can be integrated on a single chip, or can be composed of a combination of multiple chips. For example, functional blocks other than storage elements can be integrated on a single chip. Here, they are called LSI or IC, but the name changes depending on the degree of integration, and may be called system LSI, VLSI (very large scale integration) or ULSI (ultra large scale integration). For the same purpose, a field programmable gate array (FPGA) programmed after the manufacture of LSI, or a reconfigurable logic device that can reconfigure the connection relationship within the LSI or set the circuit division within the LSI can also be used.

[0292] Furthermore, all or part of the functions or operations of a component, device or part of a device can be executed by software processing. In this case, the software is recorded on one or more non-temporary recording media such as ROM, optical disk, hard disk drive, etc. When the software is executed by a processor, the software enables the processor and peripheral devices to perform specific functions in the software. The system or device may have one or more non-temporary recording media recording the software, a processor, and necessary hardware devices such as interfaces.

[0293] Industrial Applicability

[0294] The present disclosure can be applied to a depth acquisition device that acquires depth from an image obtained by photographing, and can be used as, for example, an in-vehicle device.

[0295] Description of Reference Numerals

[0296] 1Depth acquisition device

[0297] 10.101 Light Source

[0298] 20Solid-state imaging element

[0299] 211st pixel (IR)

[0300] 22 2nd pixel (BW)

[0301] 30 processing circuit

[0302] 50 diffuser plate

[0303] 60 Lens

[0304] 70 Bandpass Filter

[0305] 102 IR Camera

[0306] 103 BW Camera

[0307] 104 Learning Model

[0308] 110 processors

[0309] 111 Depth Estimation Department

[0310] 1 1 1a1st Depth Estimation Unit

[0311] 1 1 1b Second depth estimation unit

[0312] 112 Dust Detection Department

[0313] 113 Lighting timing acquisition unit

[0314] 114IR image acquisition unit

[0315] 115BW image acquisition unit

[0316] 116 High brightness area detection unit

[0317] 117FOE Inspection Department

[0318] 118 Output

[0319] 200 memory.

Claims

1. A camera method, comprising the following steps: By taking pictures of the light emitted from the light source and reflected by the subject, an infrared image is obtained. By photographing the subject with visible light to obtain a visible light image, A dust area is detected based on the infrared image and the visible light image.

2. The imaging method according to claim 1, wherein: A dust region is detected based on a region in the infrared image corresponding to a region in the visible light image.

3. The imaging method according to claim 1 or 2, wherein: A dust area is detected based on the brightness of an area in the infrared image corresponding to an area in the visible light image.

4. The imaging method according to claim 1 or 2, wherein: In the detection of the dust area, When a high-brightness region having a brightness equal to or higher than a first threshold value in the infrared image satisfies a first condition, the high-brightness region is detected as the dust region.

5. The imaging method according to claim 4, wherein: The first condition is that the center of gravity of the high-brightness area is arranged on a straight line or arc where the center of gravity of at least two other high-brightness areas different from the high-brightness area in the infrared image intersects with the FOE, i.e., the extended focus, of the infrared image or the FOE, i.e., the extended focus, of the visible light image.

6. The imaging method according to claim 4, wherein: The first condition is a condition that the main axis of the high brightness area or an extension of the main axis intersects with the FOE (extended focus) of the infrared image or the FOE (extended focus) of the visible light image.

7. The imaging method according to claim 4, wherein: In the detection of the dust area, When the high-brightness region further satisfies a second condition, the high-brightness region is detected as the dust region.

8. The imaging method according to claim 7, wherein: The second condition is a condition that the brightness of a position in the visible light image corresponding to the center of gravity of the high brightness area in the infrared image is less than a second threshold value.

9. The imaging method according to claim 7, wherein: The second condition is a condition that a correlation coefficient between the brightness in a high brightness region of the infrared image and the brightness in a region of the visible light image corresponding to the high brightness region is smaller than a third threshold value.

10. The imaging method according to claim 1, wherein: In estimating the depth of the dust region, estimating depth information representing the depth at each position within the infrared image, The depth at each position in the dust region indicated by the depth information is corrected by inputting the infrared image, the visible light image, the dust region, and the depth information into a learning model.

11. The imaging method according to claim 4, wherein: When the brightness of a region in the infrared image corresponding to the region in the visible light image is equal to or smaller than a threshold value, the region in the infrared image is detected as a dust region.

12. The imaging method according to claim 1, wherein: A dust area is detected based on the infrared image and a visible light image generated by capturing the same scene as the infrared image.

13. The imaging method according to claim 1, wherein: A dust region is detected based on the infrared image and a visible light image generated by imaging from the same viewpoint as the infrared image.

14. The imaging method according to claim 1, wherein: A dust region is detected based on the infrared image and a visible light image generated by imaging at the same time as the infrared image.

15. The imaging method according to claim 1, wherein: In acquiring the infrared image, ToF, or time of flight, is used.

16. A camera device comprising: light source; an infrared camera that receives light irradiated from the light source and reflected by the subject, thereby acquiring an infrared image; A visible light camera, which obtains a visible light image by photographing the subject using visible light; and The dust region detecting unit detects a dust region based on the infrared image and the visible light image.

17. A recording medium storing a program for causing a computer to execute the following steps: By taking pictures of the light emitted from the light source and reflected by the subject, an infrared image is obtained. By photographing the subject with visible light to obtain a visible light image, A dust area is detected based on the infrared image and the visible light image.

Images