Imaging method, imaging device, and recording medium

Abstract

The invention provides an imaging method, an imaging apparatus, and a recording medium. The imaging method includes acquiring an infrared light image by performing imaging that receives light irradiated from a light source and reflected by a subject, acquiring a visible light image by imaging the subject with visible light, and detecting an edge region based on the infrared light image and the visible light image.

Description

[0001] This application is a divisional application of the invention application with an international filing date of September 11, 2019 and a national application number of 201980050152.3. The invention name of the invention application is depth acquisition device, depth acquisition method and recording medium. Technical Field

[0002] The present disclosure relates to a depth acquisition device and the like that acquires the distance to a subject as the depth. Background Art

[0003] In the past, a distance measuring device for measuring the distance to a subject has been proposed (for example, refer to Patent Document 1). The distance measuring device includes a light source and an imaging unit. The light source irradiates light to the subject. The imaging unit captures the reflected light reflected by the subject. Then, the distance measuring device measures the distance to the subject by converting each pixel value of the image obtained by the imaging into the distance to the subject. That is, the distance measuring device obtains the depth of the image obtained by the imaging unit.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2011-64498 Summary of the invention

[0007] Problems to be solved by the invention

[0008] However, the distance measuring device of Patent Document 1 has a problem that the depth cannot be accurately obtained.

[0009] To this end, the present disclosure provides a depth acquisition device that can accurately acquire the distance to a subject, that is, the depth.

[0010] Means for solving problems

[0011] A depth acquisition device involved in one scheme of the present disclosure includes a memory and a processor, wherein the processor acquires the intensity of infrared light irradiated from a light source and reflected by a subject, the intensity of the infrared light being measured by imaging a plurality of pixels included in an imaging element respectively receiving the infrared light and stored in the memory, calculates the distance to the subject as the depth for each of the plurality of pixels included in the imaging element based on the intensity of the infrared light received in the pixel, thereby generating a depth image, acquires a visible light image, the visible light image being generated by imaging a scene substantially the same as the infrared light image formed based on the intensity of the infrared light received by the plurality of pixels included in the imaging element respectively and at substantially the same viewpoint and time as the infrared light image, and stored in the memory, detects from the visible light image an area including an edge along a direction perpendicular to the direction of movement of the visible light image, i.e., an edge area, and corrects the depth of an area within the depth image corresponding to the edge area, i.e., an area to be corrected.

[0012] In addition, these general or specific solutions can be implemented by a system, method, integrated circuit, computer program or a computer-readable CD-ROM or other recording medium, or by any combination of the system, method, integrated circuit, computer program and recording medium. In addition, the recording medium can be a non-temporary recording medium.

[0013] Effects of the Invention

[0014] The depth acquisition device disclosed in the present invention can correctly acquire the distance to the subject, i.e., the depth. Further advantages and effects in one scheme of the present invention will be clear from the specification and the drawings. The relevant advantages and / or effects are provided respectively through several embodiments and the features recorded in the specification and the drawings, but it is not necessary to provide all of them in order to obtain one or more of the same features. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a block diagram showing the hardware configuration of the depth acquisition device in the embodiment.

[0016] Figure 2 It is a schematic diagram showing a pixel array included in the solid-state imaging element according to the embodiment.

[0017] Figure 3 This is a timing chart showing the relationship between the light emission timing of the light emitting element of the light source and the exposure timing of the first pixel of the solid-state imaging element in the embodiment.

[0018] Figure 4 This is a block diagram showing an example of the functional structure of the depth acquisition device in the embodiment.

[0019] Figure 5 It is a diagram showing an example of a BW image and an IR image.

[0020] Figure 6 This is a flowchart showing the overall processing operation of the depth acquisition device in the embodiment.

[0021] Figure 7 This is a flowchart showing an example of the processing operation of the boundary area detection unit in the embodiment.

[0022] Figure 8 It is a diagram for explaining edge detection in the embodiment.

[0023] Fig. 9 This is a flowchart showing another example of the processing operation of the boundary area detection unit in the embodiment.

[0024] Fig.10 This is a block diagram showing an example of the functional configuration of a depth acquisition device in a modified example of the embodiment.

[0025] Fig.11A This is a diagram showing an example of simulation results of the depth acquisition device in the embodiment.

[0026] Fig. 11B It is a diagram showing another example of the simulation result of the depth acquisition device in the embodiment. DETAILED DESCRIPTION

[0027] (Insights that form the basis of this disclosure)

[0028] The inventors of the present invention have found that the following problems occur with respect to the distance measuring device of Patent Document 1 described in the “Background Art” column.

[0029] The distance measuring device of patent document 1 irradiates light from a light source to a subject as described above, photographs the irradiated subject, thereby obtaining an image, and measures the depth of the image. ToF (Time Of Flight) is used in the measurement of the depth. In such a distance measuring device, different photographing conditions are used to improve the distance measurement accuracy. That is, the distance measuring device photographs according to given photographing conditions, and sets photographing conditions different from the given photographing conditions according to the photographing results. Then, the distance measuring device photographs again according to the set photographing conditions.

[0030] However, when capturing an image of a boundary between two objects having different light reflectances, the distance measuring device of Patent Document 1 may have difficulty in accurately measuring the depth around the boundary even if the imaging conditions are changed.

[0031] In order to solve such a problem, a depth acquisition device involved in one scheme of the present disclosure includes a memory and a processor, wherein the processor acquires the intensity of infrared light irradiated from a light source and reflected by a subject, the intensity of the infrared light being measured by imaging a plurality of pixels included in an imaging element respectively receiving the infrared light, and the processor is stored in the memory, and the distance to the subject is calculated as the depth for each of the plurality of pixels included in the imaging element based on the intensity of the infrared light received in the pixel, thereby generating a depth image, and acquiring a visible light image, the visible light image being generated by imaging a scene substantially the same as the infrared light image formed based on the intensity of the infrared light received by the plurality of pixels included in the imaging element respectively and at substantially the same viewpoint and time as the infrared light image, and the processor is stored in the memory, and an area including an edge along a direction perpendicular to the direction of change of the visible light image, i.e., an edge area, is detected from the visible light image, and the depth of an area within the depth image corresponding to the edge area, i.e., a correction target area, is corrected. In addition, the area in the depth image corresponding to the edge area refers to the area in the depth image that is located at the same position as the edge area in the visible light image and has the same shape and size as the edge area.

[0032] For example, when capturing images of the boundary between two objects with different reflectances of infrared light, if the camera used in the capture is changed, the reflectance at the measurement target position around the boundary may change significantly in the ToF measurement. In such a case, noise is obtained at the measurement target position, and a depth image indicating an incorrect depth is obtained.

[0033] However, in the depth acquisition device involved in one of the above schemes, an area including an edge along a direction perpendicular to the direction of change of the visible light image is detected from the visible light image as an edge area. The edge included in the edge area is equivalent to the boundary of the above-mentioned two objects, for example. That is, the edge area is an area around the above-mentioned boundary where there is a high possibility that the reflectivity of infrared light changes significantly in the ToF-based measurement. Then, the area in the depth image corresponding to such an edge area is set as the correction target area, and the depth around the boundary can be correctly acquired by reducing noise by correcting the depth of the correction target area.

[0034] Here, as an example of an image of substantially the same scene captured at substantially the same viewpoint and time, there is an image captured by different pixels of the same imaging element. Such an image is similar to the red, green and blue channel images of a color image captured by a Bayer arrangement color filter, and the viewing angle, viewpoint and shooting time of each image are roughly equal. That is, the position of the image of the substantially same scene captured at substantially the same viewpoint and time on the image of the subject in each captured image will not differ by more than 2 pixels. For example, when there is a point light source with visible light and infrared components in the scene, and only 1 pixel is captured at high brightness in the visible light image, in the infrared light image, the point light source will also be captured closer to 2 pixels than the pixel corresponding to the pixel position captured in the visible light image. In addition, the so-called capturing at substantially the same time means that the difference in the shooting time is equal to less than 1 frame.

[0035] In addition, the edge region is equivalent to the bright-dark boundary region. That is, the processor detects a region including a boundary, i.e., the bright-dark boundary region, from the visible light image, wherein the boundary is a boundary between two regions having different brightnesses and is along a direction perpendicular to the direction of change of the visible light image. In this case, the processor corrects the depth of the region in the depth image corresponding to the bright-dark boundary region, i.e., the depth of the correction target region.

[0036] Furthermore, in the detection of the edge region, the edge having an intensity greater than a first threshold value may be detected, and the edge region is a region from the edge having an intensity greater than the first threshold value to a distance corresponding to a change in the visible light image.

[0037] This makes it possible to appropriately detect the edge region. As a result, it is possible to appropriately detect the region where noise is likely to occur in the depth image.

[0038] Furthermore, in the detection of the edge region, a region in which a difference in pixel value between the visible light image and a past visible light image is greater than or equal to a second threshold may be detected as the edge region.

[0039] This makes it possible to easily detect the edge area.

[0040] Alternatively, in the correction of the correction target area, the depth of the correction target area may be corrected using the depth of the surrounding area, which is an area surrounding the correction target area in the depth image. For example, the surrounding area is an area connected to the lower side of the correction target area, and in the correction of the depth of the correction target area, the depth of the correction target area may be replaced by the depth of the surrounding area.

[0041] There is a high possibility that the depth of the surrounding area is close to the correct depth of the correction target area. Therefore, by using the depth of the surrounding area, the depth of the correction target area can be accurately corrected.

[0042] Furthermore, in the correction of the depth of the correction target region, the depth of the correction target region may be corrected by filtering the depth image using the visible light image as a reference image.

[0043] For example, an extended filter of a guided filter may be used for filtering, thereby accurately correcting the depth of the correction target area.

[0044] Furthermore, in the correction of the depth of the correction target area, the depth of the correction target area in the depth image may be corrected by inputting the infrared light image, the visible light image, the depth image, and the edge area into a learning model.

[0045] Therefore, if the learning model is pre-learned to output a corrected depth image for the input of a depth image, an infrared image, a visible light image, and an edge region, an appropriate depth image can be easily obtained without detecting the edge region.

[0046] Alternatively, the depth acquisition device involved in other schemes of the present disclosure may also include a memory and a processor, wherein the processor acquires the intensity of infrared light irradiated from a light source and reflected by a subject, the intensity of the infrared light being measured by imaging a plurality of pixels included in an imaging element respectively receiving the infrared light, and the processor is stored in the memory, and the distance to the subject is calculated as the depth for each of the plurality of pixels included in the imaging element based on the intensity of the infrared light received in the pixel, thereby generating a depth image, acquiring a visible light image, the visible light image being generated by imaging a scene based on visible light and at a viewpoint and time substantially the same as the infrared light image formed based on the intensity of the infrared light received by the plurality of pixels included in the imaging element, and the processor is stored in the memory, and the depth of the depth image is corrected by inputting the depth image, the infrared light image, and the visible light image into a learning model.

[0047] Therefore, if the learning model is pre-learned so as to output a corrected depth image in response to the input of a depth image, an infrared light image, and a visible light image, an appropriate depth image can be easily acquired without detecting an edge region.

[0048] In addition, these general or specific solutions can be implemented by a system, method, integrated circuit, computer program or computer-readable CD-ROM or other recording medium, or by any combination of systems, methods, integrated circuits, computer programs or recording media. In addition, the recording medium can be a non-transitory recording medium.

[0049] The embodiments are described in detail below with reference to the drawings.

[0050] In addition, the embodiments described below all represent general or specific examples. The numerical values, shapes, materials, components, configuration positions of components, connection forms, steps, and the order of steps shown in the following embodiments are examples and do not limit the subject matter of the present disclosure. In addition, the components in the following embodiments that are not recorded in the independent claims representing the highest concept are described as arbitrary components.

[0051] In addition, each figure is a schematic diagram, and does not necessarily show the figure strictly. In addition, in each figure, the same reference numeral is attached|subjected to the same component.

[0052] (Implementation Method)

[0053] [Hardware structure]

[0054] Figure 1 This is a block diagram showing the hardware structure of the depth acquisition device 1 involved in the embodiment. The depth acquisition device 1 in this embodiment has a hardware structure that can acquire an image based on infrared light (or near-infrared light) and an image based on visible light by photographing substantially the same scene at substantially the same viewpoint and time. In addition, the so-called substantially the same means the same degree as that of being able to produce the effect in the present disclosure.

[0055] like Figure 1 As shown, the depth acquisition device 1 includes a light source 10 , a solid-state imaging element 20 , a processing circuit 30 , a diffusion plate 50 , a lens 60 , and a bandpass filter 70 .

[0056] The light source 10 emits illumination light. More specifically, the light source 10 emits illumination light for illuminating a subject at a timing indicated by a light emission signal generated in the processing circuit 30 .

[0057] The light source 10 is composed of, for example, a capacitor, a driving circuit, and a light-emitting element, and emits light by driving the light-emitting element with electric energy stored in the capacitor. The light-emitting element is realized by, for example, a laser diode, a light-emitting diode, etc. In addition, the light source 10 may be a structure including one type of light-emitting element, or may be a structure including multiple types of light-emitting elements according to the purpose.

[0058] In the following, the light emitting element is, for example, a laser diode that emits near-infrared light, or a light emitting diode that emits near-infrared light. However, the irradiation light irradiated by the light source 10 is not necessarily limited to near-infrared light. The irradiation light irradiated by the light source 10 may be, for example, infrared light (also referred to as infrared light) in a frequency band other than near-infrared light. In the following, in this embodiment, the irradiation light irradiated by the light source 10 is described as infrared light, but the infrared light may be near-infrared light or infrared light in a frequency band other than near-infrared light.

[0059] The solid-state imaging element 20 captures an image of a subject and outputs an imaging signal indicating an exposure amount. More specifically, the solid-state imaging element 20 performs exposure at a timing indicated by an exposure signal generated by the processing circuit 30 and outputs an imaging signal indicating an exposure amount.

[0060] The solid-state imaging device 20 has a pixel array in which first pixels for imaging using reflected light of an irradiated light reflected from an object and second pixels for imaging the object are arranged in an array. The solid-state imaging device 20 may have logic functions such as a cover glass and an AD converter as necessary.

[0061] In the following description, the reflected light is assumed to be infrared light, similarly to the irradiation light. However, the reflected light is not limited to infrared light as long as it is light obtained by reflecting the irradiation light from the subject.

[0062] Figure 2 Schematic diagram showing the pixel array 2 included in the solid-state imaging element 20 .

[0063] like Figure 2 As shown, the pixel array 2 is configured in an array so that first pixels 21 (IR pixels) for capturing images using reflected light obtained by irradiating light reflected from a subject and second pixels 22 (BW pixels) for capturing images of the subject are alternately arranged in columns.

[0064] In addition, Figure 2 In the pixel array 2, the second pixel 22 and the first pixel 21 are arranged adjacent to each other in the row direction and arranged in a strip shape in the row direction, but this is not limited to this, and they may be arranged every multiple rows (for example, every 2 rows). That is, the second pixel 22 is arranged in the first row adjacent to each other in the row direction and the first pixel 21 is arranged in the second row adjacent to each other in the row direction, and they may be arranged alternately every M rows (M is a natural number). Furthermore, the second pixel 22 is arranged in the first row adjacent to each other in the row direction and the first pixel 21 is arranged in the second row adjacent to each other in the row direction, and they may be arranged separated by different rows (alternatingly repeating the first row as N rows and the second row as L rows (N and L are different natural numbers)).

[0065] The first pixel 21 is realized by, for example, an infrared light pixel having sensitivity to infrared light as reflected light, and the second pixel 22 is realized by, for example, a visible light pixel having sensitivity to visible light.

[0066] The infrared light pixel is composed of, for example, an optical filter (also referred to as an IR filter) that transmits only infrared light, a microlens, a light-receiving element as a photoelectric conversion unit, and a storage unit that stores the charge generated by the light-receiving element. Therefore, an image representing the brightness of infrared light is expressed by the imaging signal output from the plurality of infrared light pixels (i.e., the first pixel 21) included in the pixel array 2. Hereinafter, the infrared light image is also referred to as an IR image or an infrared light image.

[0067] In addition, the visible light pixel is composed of, for example, an optical filter (also called a BW filter) that transmits only visible light, a microlens, a light receiving element as a photoelectric conversion unit, and a storage unit that stores the charge converted by the light receiving element. Therefore, the visible light pixel, that is, the second pixel 22, outputs an imaging signal representing brightness and color difference. That is, a color image representing the brightness and color difference of visible light is expressed by the imaging signal output from the plurality of second pixels 22 included in the pixel array 2. In addition, the optical filter of the visible light pixel can transmit both visible light and infrared light, or can transmit only light in a specific wavelength band such as red (R), green (G) or blue (B) among the visible light.

[0068] In addition, the visible light pixel can detect only the brightness of the visible light. In this case, the visible light pixel, i.e., the second pixel 22, outputs an imaging signal indicating the brightness. Therefore, the imaging signal outputted by the plurality of second pixels 22 included in the pixel array 2 represents a black and white image indicating the brightness of the visible light, in other words, a monochrome image. Hereinafter, the monochrome image is also referred to as a BW image. In addition, the above-mentioned color image and BW image are also collectively referred to as a visible light image.

[0069] Back again Figure 1 , continue with the description of the acquisition device 1 in depth.

[0070] The processing circuit 30 calculates subject information related to the subject using the imaging signal outputted from the solid-state imaging element 20 .

[0071] The processing circuit 30 includes, for example, a microcomputer or other computing device. The microcomputer includes a processor (microprocessor), a memory, etc., and generates a light emission signal and an exposure signal by executing a driver stored in the memory by the processor. In addition, the processing circuit 30 may use an FPGA or an ISP, etc., and may include one hardware or multiple hardware.

[0072] The processing circuit 30 calculates the distance to the subject by, for example, a ToF distance measurement method using the imaging signal from the first pixel 21 of the solid-state imaging element 20 .

[0073] The calculation of the distance to the subject by the ToF distance measurement method performed by the processing circuit 30 will be described below with reference to the drawings.

[0074] Figure 3 This is a timing chart showing the relationship between the light emission timing of the light emitting element of the light source 10 and the exposure timing of the first pixel 21 of the solid-state imaging element 20 when the processing circuit 30 calculates the distance to the subject using the ToF distance measurement method.

[0075] exist Figure 3 In the figure, Tp is the light emission period during which the light emitting element of the light source 10 emits the irradiation light, and Td is the delay time from when the light emitting element of the light source 10 emits the irradiation light to when the reflected light of the irradiation light reflected by the subject returns to the solid-state imaging element 20. Furthermore, the first exposure period becomes the same timing as the light emission period during which the light source 10 emits the irradiation light, and the second exposure period becomes the timing from the end time point of the first exposure period to the passage of the light emission period Tp.

[0076] exist Figure 3 In the figure, q1 represents the total amount of exposure in the first pixel 21 of the solid-state imaging element 20 caused by the reflected light during the first exposure period, and q2 represents the total amount of exposure in the first pixel 21 of the solid-state imaging element 20 caused by the reflected light during the second exposure period.

[0077] By Figure 3 The light emitting element of the light source 10 emits the irradiation light and the first pixel 21 of the solid-state imaging element 20 exposes the light at the timing shown. Assuming the speed of light is c, the distance d to the subject can be expressed by the following (Equation 1).

[0078] d=c×Tp / 2×q2 / (q1+q2)…(Formula 1)

[0079] Therefore, the processing circuit 30 can calculate the distance to the subject using the imaging signal from the first pixel 21 of the solid-state imaging element 20 by using (Formula 1).

[0080] In addition, the plurality of first pixels 21 of the solid-state imaging element 20 can be exposed only in the third exposure period Tp after the first exposure period and the second exposure period have ended. The plurality of first pixels 21 can detect noise other than the reflected light based on the exposure amount obtained in the third exposure period Tp. That is, the processing circuit 30 can more accurately calculate the distance d to the subject by removing the noise from the exposure amount q1 in the first exposure period and the exposure amount q2 in the second exposure period in the above (Formula 1).

[0081] Back again Figure 1 , continue with the description of the acquisition device 1 in depth.

[0082] The processing circuit 30 can detect the subject and calculate the distance to the subject using the imaging signal from the second pixel 22 of the solid-state imaging element 20 , for example.

[0083] That is, the processing circuit 30 can detect the subject and calculate the distance to the subject based on the visible light image captured by the multiple second pixels 22 of the solid-state imaging element 20. Here, the detection of the subject can be achieved, for example, by using edge detection of the singularity of the subject to distinguish the shape by pattern recognition, or it can be achieved by using a learning model that has been learned in advance through deep learning and other processes. In addition, the calculation of the distance to the subject can be performed using world coordinate transformation. Of course, it is also possible to detect the subject by multimodal learning processing using not only the visible light image but also the brightness and distance information of the infrared light captured by the first pixel 21.

[0084] The processing circuit 30 generates a light emission signal indicating a timing for light emission and an exposure signal indicating a timing for exposure. The processing circuit 30 then outputs the generated light emission signal to the light source 10 and outputs the generated exposure signal to the solid-state imaging element 20 .

[0085] The processing circuit 30 can, for example, generate and output a light emission signal to cause the light source 10 to emit light at a given cycle, and generate and output an exposure signal to cause the solid-state imaging element 20 to be exposed at a given cycle, thereby enabling the depth acquisition device 1 to achieve continuous imaging at a given frame rate. In addition, the processing circuit 30 includes, for example, a processor (microprocessor), a memory, etc., and generates a light emission signal and an exposure signal by executing a driver program stored in the memory by the processor.

[0086] The diffuser 50 adjusts the intensity distribution and angle of the irradiated light. In addition, in the adjustment of the intensity distribution, the diffuser 50 makes the intensity distribution of the irradiated light from the light source 10 uniform. Figure 1 In the illustrated example, the depth acquisition device 1 includes the diffusion plate 50 , but may not include the diffusion plate 50 .

[0087] The lens 60 is an optical lens that focuses light entering from the outside of the depth acquisition device 1 on the surface of the pixel array 2 of the solid-state imaging element 20 .

[0088] The bandpass filter 70 is an optical filter that transmits infrared light as reflected light and visible light. Figure 1 In the illustrated example, the depth acquisition device 1 includes the bandpass filter 70 , but may not include the bandpass filter 70 .

[0089] The depth acquisition device 1 of the above structure is mounted on a transport device and used. For example, the depth acquisition device 1 is mounted on a vehicle traveling on a road and used. In addition, the transport device on which the depth acquisition device 1 is mounted is not necessarily limited to a vehicle. The depth acquisition device 1 can also be mounted on a transport device other than a vehicle such as a motorcycle, a ship, or an airplane and used.

[0090] [Functional structure of depth acquisition device]

[0091] The depth acquisition device 1 in this embodiment is used Figure 1 The hardware structure shown acquires an IR image and a BW image by photographing substantially the same scene and substantially the same viewpoint and time. Here, the IR image is formed based on the intensity of infrared light respectively received by the plurality of first pixels 21 included in the solid-state imaging element 20. Therefore, the depth acquisition device 1 acquires the intensity of infrared light in each of the plurality of first pixels 21 by photographing the IR image. The depth acquisition device 1 acquires a depth image that represents the distance to the subject projected in the IR image as the depth based on the intensity of the infrared light of these first pixels 21. Then, the depth acquisition device 1 detects the edge area in the BW image and corrects the depth of the area in the depth image corresponding to the edge area.

[0092] In addition, in the present disclosure, the second region in the second image corresponding to the first region in the first image is a region in the second image that is located at the same position as the first region in the first image and has the same shape and size as the first region. The first image and the second image are arbitrary images, and the first region and the second region are also arbitrary regions.

[0093] In addition, the edge in the present disclosure is the boundary between two areas with different brightness. The two areas are a bright area and a dark area. In addition, the average brightness in the bright area is higher than the average brightness in the dark area. Therefore, the depth acquisition device 1 in the present disclosure detects, for example, a region including the boundary between a bright area and a dark area in a visible light image such as a BW image, i.e., a bright-dark boundary region, as the above-mentioned edge region.

[0094] Figure 4 This is a block diagram showing an example of the functional structure of the depth acquisition device 1 .

[0095] The depth acquisition device 1 includes a light source 101, an IR camera 102, a BW camera 103, a processor 110a, and a memory 200. In addition, the depth acquisition device 1 in this embodiment includes the light source 101, the IR camera 102, and the BW camera 103, but may include only the processor 110a and the memory 200 without these components.

[0096] The light source 101 may include Figure 1 The light source 10 and the diffuser plate 50 shown in the figure emit light to irradiate the subject with infrared light.

[0097] The IR camera 102 may be referred to as an infrared camera and may include Figure 1 The IR camera 102 is a solid-state imaging element 20 shown in FIG. 1 , and the plurality of first pixels 21, the lens 60, and the bandpass filter 70 are shown. Such an IR camera 102 acquires an IR image by performing infrared light-based imaging of a scene including the subject in accordance with the timing of the light source 101 irradiating the subject with infrared light. In addition, the IR camera 102 measures the intensity of infrared light by capturing the IR image. That is, the IR camera 102 has a solid-state imaging element 20, and measures the intensity of infrared light by capturing the plurality of first pixels 21 included in the solid-state imaging element 20, respectively, receiving infrared light irradiated from the light source 101 and reflected by the subject.

[0098] The BW camera 103 may be referred to as a visible light camera and may include Figure 1 The solid-state imaging device 20 shown in FIG. 1 includes a plurality of second pixels 22, a lens 60, and a bandpass filter 70. Such a BW camera 103 acquires a visible light image (specifically, a BW image) by performing imaging based on visible light of a scene substantially identical to the IR image and performing imaging at substantially the same viewpoint and time as the IR image. That is, the BW camera 103 generates a BW image by performing imaging based on visible light of a scene substantially identical to the IR image formed based on the intensity of infrared light received by each of the plurality of first pixels 21 included in the solid-state imaging device 20 and performing imaging at substantially the same viewpoint and time as the IR image.

[0099] The memory 200 is a recording medium for storing an IR image obtained by the imaging performed by the IR camera 102 and a BW image obtained by the imaging performed by the BW camera 103. In addition, as described above, the IR image is formed based on the intensity of infrared light respectively received by the plurality of first pixels 21 included in the solid-state imaging element 20. Therefore, the IR image shows the intensity of infrared light for each pixel. That is, the memory 200 can be said to store the intensity of infrared light. In addition, such a memory 200 can specifically be a ROM (Read Only Memory), a RAM (Random access memory) or an SSD (solid state drive), etc., and can be non-volatile or volatile. In addition, the memory 200 can be a hard disk.

[0100] The processor 110a obtains the IR image and the BW image from the memory 200, calculates the depth image from the IR image, and detects the bright-dark boundary region in the BW image based on the IR image and the BW image. Then, the processor 110a corrects the depth of the region in the depth image corresponding to the bright-dark boundary region.

[0101] The bright-dark boundary region of the BW image is a region including the boundary between the bright region and the dark region and along the boundary in a direction perpendicular to the direction of fluctuation of the BW image. In the region in the depth image corresponding to the bright-dark boundary region, inappropriate depth may be shown.

[0102] The reason is as follows. The brightness in the bright-dark boundary area of ​​the BW image changes greatly in a short period of time due to the change of the BW image. That is, in a short period of time, the object reflected in the bright-dark boundary area is replaced by another object whose light reflectance is greatly different from that of the object. In addition, the change of the BW image is caused by the change of the BW camera 103, for example. When the IR camera 102 and the BW camera 103 are changed together, for example, when the depth acquisition device 1 is mounted on a vehicle, the change of the BW image is equal to the change of the IR image. Therefore, the same phenomenon as the BW image is generated in the IR image. That is, the object reflected in the area within the IR image corresponding to the bright-dark boundary area is replaced by another object whose infrared light reflectance is greatly different from that of the object in a short period of time. As a result, the depth measured by ToF using the intensity of infrared light corresponding to the bright-dark boundary area becomes incorrect. That is, the depth obtained by such a measurement appears as noise in the depth image.

[0103] In this way, the area where noise appears in the depth image corresponds to the area containing the above-mentioned boundary in the BW image. Therefore, by detecting the area containing such a boundary, that is, the bright-dark boundary area, the area to be corrected can be appropriately found in the depth image. The depth acquisition device 1 in this embodiment corrects the depth of the area to be corrected in the depth image, that is, the correction target area which is the area corresponding to the bright-dark boundary area.

[0104] Such a processor 110 a includes a light emission control unit 113 , an IR acquisition unit 114 , a BW acquisition unit 115 , a depth calculation unit 111 a , a boundary region detection unit 112 , and a depth correction unit 111 b .

[0105] The light control unit 113 controls the light source 101. That is, the light control unit 113 outputs the light emission signal to the light source 101 to make the light source 101 emit light. Thus, infrared light is irradiated from the light source 101 to the subject, and the light reflected by the subject, that is, the reflected light, enters the IR camera 102.

[0106] The IR acquisition unit 114 acquires an IR image from the IR camera 102 via the memory 20. Here, each pixel of the IR image represents the intensity of the infrared light received at the position of the pixel as a pixel value (specifically, brightness). Therefore, the IR acquisition unit 114 acquires the intensity of the infrared light by acquiring the IR image. That is, the IR acquisition unit 114 acquires the intensity of the infrared light irradiated from the light source 101 and reflected by the subject, and the intensity of the infrared light is measured by the plurality of first pixels 21 included in the solid-state imaging element 20 receiving the infrared light respectively, and is stored in the memory 200.

[0107] The BW acquisition unit 115 acquires a BW image from the BW camera 103 via the memory 200. That is, the BW acquisition unit 115 acquires a BW image generated by imaging substantially the same scene as the IR image formed based on the intensity of infrared light respectively received by the plurality of first pixels 21 included in the solid-state imaging element 20 using visible light and imaging at substantially the same viewpoint and time as the IR image, and stores the BW image in the memory 200.

[0108] The depth calculation unit 111 a generates a depth image by calculating the distance to the subject as the depth for each of the plurality of first pixels 21 included in the solid-state imaging element 20 based on the intensity of infrared light received by the first pixel 2 .

[0109] The boundary region detection unit 112 detects a bright-dark boundary region from the BW image. The bright-dark boundary region includes a boundary between a bright region and a dark region and is a boundary along a direction perpendicular to the direction of fluctuation of the BW image.

[0110] The depth correction unit 111 b corrects the depth of the correction target area, which is an area in the depth image corresponding to the bright-dark boundary area.

[0111] For example, when capturing an image of the boundary between two objects having different reflectances to infrared light, if the IR camera 102 used in the image capturing is changed, the reflectance at the measurement target position around the boundary may change significantly in the ToF measurement. In such a case, noise is obtained at the measurement target position, and a depth image indicating an incorrect depth is obtained.

[0112] However, in the depth acquisition device 1 in the present embodiment, as described above, from the BW image, an area including the boundary between the bright area and the dark area and along the boundary in a direction perpendicular to the direction of change of the visible light image is detected as a bright-dark boundary area. The bright-dark boundary area is an area around the above-mentioned boundary where there is a high possibility that the reflectivity of infrared light changes significantly in the measurement based on ToF. And, by setting the area in the depth image corresponding to such a bright-dark boundary area as a correction target area and correcting the depth of the correction target area, noise can be reduced and the depth around the boundary can be correctly acquired.

[0113] Therefore, even if the boundary between two objects having different reflectivities for infrared light is along a direction perpendicular to the direction of image movement, the depth acquisition device 1 in this embodiment can accurately acquire the distance to the periphery of the boundary between these objects, that is, the depth.

[0114] [An example of each image]

[0115] Figure 5 An example of a BW image and an IR image is shown.

[0116] For example, when a vehicle is traveling on a road, the IR camera 102 and the BW camera 103 of the depth acquisition device 1 mounted on the vehicle respectively take images at substantially the same time. At this time, the IR camera 102 and the BW camera 103 respectively perform exposure at three different timings. Thus, three IR images are generated by the imaging of the IR camera 102, and three BW images are generated by the imaging of the BW camera 103.

[0117] In the IR image generated by the IR camera 102 and acquired by the IR acquisition unit 114, for example, Figure 5 The road surface and the surrounding trees are reflected as shown in (a) to (c).

[0118] In addition, in the BW image generated by the BW camera 103 and acquired by the BW acquisition unit 115, Figure 5 As shown in (d) to (f) of Figure 5 The IR images shown in (a) to (c) are substantially the same scene. Figure 5 The BW images shown in (d) to (f) are obtained by Figure 5 The IR images shown in (a) to (c) are images taken from substantially the same viewpoint and at the same time. Figure 5 The IR images shown in (a) to (c) and Figure 5In the BW images shown in (d) to (f) of FIG. 1 , the same object is shown in the corresponding regions. The corresponding regions are regions having the same position, size, and shape in the respective images.

[0119] Here, in the BW image, the brightness of the image of the leaves of the trees is low, and the brightness of the image of the sky in the background is high. Therefore, the area along the boundary in the direction perpendicular to the direction of image movement among the boundaries of the images of the leaves and the sky is equivalent to the bright-dark boundary area. In such an area within the IR image corresponding to the bright-dark boundary area, since an inappropriate intensity of infrared light can be obtained, there is a tendency for the image to become blurred. In addition, the reliability of the depth calculated by such inappropriate intensity of infrared light is also low. The depth correction unit 111b in this embodiment improves the reliability of the depth by correcting the depth.

[0120] [Processing Flow]

[0121] Figure 6 This is a flowchart showing the overall processing operation of the depth acquisition device 1 in this embodiment.

[0122] (Step S110)

[0123] First, the BW camera 103 generates a BW image by performing imaging using visible light, and then the BW acquisition unit 115 acquires the BW image via the memory 200 .

[0124] (Step S120)

[0125] Next, the IR camera 102 measures the intensity of the infrared light by performing imaging based on the infrared light. Then, the IR acquisition unit 114 acquires the intensity of the infrared light via the memory 200. Here, the intensity of the infrared light acquired is the intensity of the infrared light received by each of the plurality of first pixels 21 of the solid-state imaging element 20. Therefore, the IR acquisition unit 114 acquires the IR image formed based on the intensity of the infrared light by acquiring the intensity of the infrared light.

[0126] Specifically, the intensity of infrared light obtained from the memory 200 includes at least three intensities measured by exposing the solid-state imaging element 20 at least three different timings when the infrared light irradiated from the light source 101 and reflected by the subject is received by the solid-state imaging element 20. For example, the intensity of infrared light includes Figure 3 As shown, three intensities are measured during the first exposure period, the second exposure period, and the third exposure period. For example, each pixel value included in one frame of IR image is represented as a cumulative value of the intensity of infrared light measured by the at least three exposures.

[0127] (Step S130)

[0128] Next, the boundary region detection unit 112 detects a bright-dark boundary region from the BW image acquired in step S110 and outputs a binary mask image indicating 1 for each pixel in the detected bright-dark boundary region and 0 for each pixel in other regions.

[0129] (Step S140)

[0130] Next, the depth calculation unit 111a calculates the depth based on the intensity of the infrared light obtained in step S120. That is, the depth calculation unit 111a generates a depth image by calculating the distance to the subject as the depth for each of the plurality of first pixels 21 included in the solid-state imaging element 20 based on the intensity of the infrared light received at the first pixel 2. Specifically, the depth calculation unit 111a generates a depth image by calculating the depth for each of the plurality of first pixels 21 included in the solid-state imaging element 20 based on the at least three intensities measured at the first pixel 21. For example, the depth calculation unit 111a calculates the depth using the above-mentioned (Formula 1) according to the ToF ranging method.

[0131] (Step S150)

[0132] Next, the depth correction unit 111 b corrects the depth of the correction target area, which is an area in the depth image corresponding to the bright-dark boundary area.

[0133] Specifically, the depth correction unit 111b obtains a BW image, a depth image, and a mask image, and corrects the depth (i.e., the pixel value in the depth image) in the area indicated as 1 by the mask image in the depth image. The depth correction is performed, for example, by using an extended filter of a guided filter using the BW image as a guide image. The extended filter is disclosed in a non-patent document (Jiangbo Lu, Keyang Shi, Dongbo Min, Liang Lin, and Minh N. Do, "Cross-Based Local Multipoint Filtering", 2012 IEEE Conference on Computer Vision and Pattern Recognition). Alternatively, depth correction can also be achieved by a method disclosed in a non-patent document (Dongbo Min, Sunghwan Choi, Jiangbo Lu, Bumsub Ham, Kwanghoon Sohn, and Minh N. Do, "Fast Global Image Smoothing Based on Weighted Least Squares", IEEE Transactions on Image Processing, Vol. 23, No. 12, December 2014).

[0134] As described above, the depth correction unit 111b in this embodiment corrects the depth of the correction target region by filtering the depth image using the BW image as a guide image or a reference image.

[0135] In addition, the correction of depth is not limited to the above content. For example, the depth in the area connected to the lower part of the bright and dark boundary area can also be emphasized for interpolation. That is, the depth correction unit 111b can use the depth of the area surrounding the correction target area in the depth image, that is, the depth of the surrounding area, to correct the depth of the correction target area. In addition, the surrounding area can be an area connected to the lower side of the correction target area. In this case, the depth correction unit 111b can also replace the depth of the correction target area with the depth of the surrounding area. There is a high possibility that the depth of the surrounding area is similar to the correct depth of the correction target area. Therefore, by using the depth of such a surrounding area, the depth of the correction target area can be correctly corrected.

[0136] In addition, in addition to the BW image, the depth image and the mask image, the IR image may also be used as an input. For example, the depth correction unit 111b may correct the depth of the correction target area in the depth image by inputting the IR image, the BW image, the depth image and the light-dark boundary area into the learning model. In addition, the above-mentioned mask image may be used instead of the light-dark boundary area. For example, the learning model is a neural network constructed by deep learning. In the learning of the learning model, the IR image, the BW image, the depth image and the light-dark boundary area are provided as input to the learning model. Then, the learning model is learned so that the output from the learning model for the input is consistent with the correct depth image after correction. By using such a completed learning model, the depth correction unit 111b can easily obtain the corrected depth image.

[0137] Figure 7 : is a flowchart showing an example of the processing operation of the boundary area detection unit 112. Figure 7 Detailed representation Figure 6 The process of step S130.

[0138] (Step S131)

[0139] First, the boundary region detection unit 112 detects a change from the BW image. The change is detected by, for example, performing a change search by block matching between the BW image and a past BW image.

[0140] (Step S132)

[0141] Next, the boundary region detection unit 112 detects an edge along a direction perpendicular to the direction of the fluctuation detected in step S131 as the above-mentioned boundary from the BW image. At this time, the boundary region detection unit 112 may detect an edge having a strength equal to or greater than a first threshold.

[0142] (Step S133)

[0143] The boundary region detection unit 112 detects the region including the edge detected in step S132 as the bright-dark boundary region. At this time, the boundary region detection unit 112 extracts, for example, a region located at a certain distance from the edge as the bright-dark boundary region. Specifically, the bright-dark boundary region is a region from an edge having an intensity greater than or equal to the first threshold to a distance corresponding to the change in the BW image.

[0144] Here, the above distance is equal to the length of the change vector representing the change of the BW image, or is the length of the change vector multiplied by a constant (e.g., 1.1 times the length of the change vector, etc.). Alternatively, the distance may be the length of the change vector plus a constant (e.g., the length of the change vector + 3 pixels or + 5 pixels, etc.). In addition, the direction of the change vector may be considered in the distance. For example, the distance in the direction of the change vector from the edge is greater than the length of the change vector, and the distance in the opposite direction to the change vector may be, for example, 0.1 times the length of the change vector, or may be a constant such as 3 pixels or 5 pixels.

[0145] In this way, the boundary region detection unit 112 in the present embodiment detects the edge along the direction perpendicular to the direction of the change in the BW image as the boundary from the BW image, and detects the area including the edge as the bright-dark boundary area. Specifically, the boundary region detection unit 112 detects the edge having an intensity greater than the first threshold. Then, the area including the edge is the area from the edge having an intensity greater than the first threshold to the distance corresponding to the change in the BW image. Thus, the bright-dark boundary area can be appropriately detected. As a result, the area prone to noise can be appropriately detected in the depth image.

[0146] Figure 8 It is used to illustrate Figure 7 FIG. 5 is a diagram of edge detection in step S132.

[0147] For example, objects reflected in the BW image are Figure 8 The object moves with the passage of time as shown in the time t0, t1 and t2. Such a change of the object is caused, for example, by the movement of a vehicle equipped with the depth acquisition device 1. Such a change in the pixel (x, y) in the BW image is expressed as a change vector (u(x, y), v(x, y)).

[0148] The boundary region detection unit 112 calculates the edge strength as the amount of change in the pixel value by differentiating the distribution of the pixel value in the spatial direction. The differentiation in any direction n is represented by (Formula 2).

[0149] [Mathematical formula 1]

[0150]

[0151] Here, the direction n is considered to be the direction of the image change vector (u(x, y), v(x, y)), that is, a unit vector in the same direction and parallel to the change vector. That is, the x component cosθ and the y component sinθ of n are represented by (Formula 3). In addition, the image change vector is obtained by a known method.

[0152] [Mathematical formula 2]

[0153]

[0154] In addition, the brightness gradients in the horizontal direction and the vertical direction in (Formula 2) are calculated by (Formula 4) and (Formula 5), ​​respectively.

[0155] [Mathematical formula 3]

[0156]

[0157] [Formula 4]

[0158]

[0159] In addition, the calculation of the brightness gradient in the horizontal direction and the vertical direction is not limited to the calculation based on the difference between the two sides shown in (Formula 4) and (Formula 5), ​​and can also be based on the calculation of the front difference shown in (Formula 6) and (Formula 7), or use the method based on the rear difference shown in (Formula 8) and (Formula 9). The same result can be obtained by any of these calculations.

[0160] [Formula 5]

[0161]

[0162] [Mathematical formula 6]

[0163]

[0164] [Formula 7]

[0165]

[0166] [Mathematical formula 8]

[0167]

[0168] In addition, in (Formula 4) to (Formula 9), calculations are performed only between pixels in the same row or the same column, but this is not limited to this. As shown in (Formula 10) to (Formula 15), a stable brightness gradient calculation can be performed to reduce the influence of noise by calculating the pixel values ​​of adjacent rows or columns.

[0169] [Mathematical formula 9]

[0170]

[0171] [Formula 10]

[0172]

[0173] [Mathematical formula 11]

[0174]

[0175] [Mathematical formula 12]

[0176]

[0177] [Mathematical formula 13]

[0178]

[0179] [Formula 14]

[0180]

[0181] Furthermore, when calculating the brightness gradient using pixel values ​​of multiple rows or columns, the weight of the focus row or column may be increased compared to the adjacent rows or columns as in (Equation 16) to (Equation 21).

[0182] [Mathematical formula 15]

[0183]

[0184] [Formula 16]

[0185]

[0186] [Mathematical formula 17]

[0187]

[0188] [Mathematical formula 18]

[0189]

[0190] [Mathematical formula 19]

[0191]

[0192] [Mathematical formula 20]

[0193]

[0194] The boundary area detection unit 112 uses the horizontal brightness gradient and the vertical brightness gradient shown in (Equation 4) to (Equation 21) and the horizontal component cosθ and the vertical component sinθ of the direction vector n of the change shown in (Equation 3) to differentiate the direction along the change direction n by (Equation 2). Calculated as the strength of the edge.

[0195] In addition, the boundary region detection unit 112 can calculate the directional differential not only by the method of (Formula 2) but also by other methods. For example, the boundary region detection unit 112 calculates the pixel value of at least one pixel located on the straight line determined by the focus point and the differential direction by interpolation. Then, the boundary region detection unit 112 calculates the directional differential based on the difference between the pixel value of at least one pixel calculated and any two pixel values ​​among the pixel values ​​of the focus point. This method can also obtain the same result as the method of (Formula 2) above.

[0196] The boundary region detection unit 112 detects pixels having an absolute value of a directional differential equal to or greater than a first threshold value as edges.

[0197] Fig. 9 2 is a flowchart showing another example of the processing operation of the boundary area detection unit 112. Fig. 9 Detailed representation Figure 6 The process of step S130.

[0198] (Step S135)

[0199] The boundary region detection unit 112 calculates the difference between the BW image and the immediately preceding BW image.

[0200] (Step S136)

[0201] Then, the boundary region detection unit 112 detects a region in the BW image where the difference is equal to or greater than a second threshold value as a bright-dark boundary region.

[0202] As described above, the boundary region detection unit 112 in the present embodiment detects a region where the difference in pixel value between the BW image and the past BW image is equal to or greater than the second threshold as a bright-dark boundary region.

[0203] (Variation Example)

[0204] In the above-described embodiment, the bright-dark boundary region is detected to correct the depth. However, the depth may be corrected using a learning model without detecting the bright-dark boundary region.

[0205] Fig.10 is a block diagram showing an example of the functional structure of the depth acquisition device 1 in this modification. Fig.10 Among the components shown, Figure 4 The same components as those shown in the figure are marked with Figure 4 The same reference numerals are used for the constituent elements shown, and detailed description thereof will be omitted.

[0206] In this modification, the depth acquisition device 1 replaces Figure 4The processor 110a shown includes a processor 110b and further includes a learning model 104.

[0207] The learning model 104 is, for example, a neural network and is formed by deep learning. For example, a BW image, an IR image, and a depth image are used as input data to the learning model 104. The learning model 104 has been learned so that a corrected depth image is outputted for a combination of these input data.

[0208] Processor 110b does not have Figure 4 The boundary region detection unit 112 shown replaces Figure 4 The depth correction portion 111b shown in the figure is provided with a depth correction portion 111c.

[0209] The depth correction unit 111c inputs the above-mentioned input data to the above-mentioned learning model 104. As a result, the depth correction unit 111c obtains a corrected depth image from the learning model 104 as output data for the input data.

[0210] Right now, Fig.10 The depth acquisition device 1 shown includes a memory 200 and a processor 110b. The processor 110b acquires the intensity of infrared light irradiated from the light source 101 and reflected by the subject, and the intensity of the infrared light is measured by imaging the plurality of first pixels 21 included in the solid-state imaging element 20 respectively receiving the infrared light, and is stored in the memory 200. In addition, the processor 110b generates a depth image by calculating the distance to the subject as the depth for each of the plurality of first pixels 21 included in the solid-state imaging element 20 based on the intensity of the infrared light received at the first pixel 21. Furthermore, the processor 110b acquires a BW image, which is generated by imaging substantially the same scene as the IR image formed based on the intensity of the infrared light received by each of the plurality of first pixels 21 included in the solid-state imaging element 20, based on visible light, and at substantially the same viewpoint and time as the IR image, and is stored in the memory 200. Then, the processor 110 b corrects the depth of the depth image by inputting the depth image, the IR image, and the BW image into the learning model 104 .

[0211] Therefore, if the learning model 104 is previously trained to output a corrected depth image for the input of the depth image, IR image, and BW image, the depth of the depth image can be appropriately corrected without detecting the bright-dark boundary region.

[0212] Fig.11A An example of simulation results of the depth acquisition device 1 according to this modification is shown.

[0213] The depth acquisition device 1 acquires the depth through the image captured by the BW camera 103. Figure 1 The BW image shown in (a) of FIG. 1A is obtained by the IR camera 102. Fig.11A The BW image and the IR image are images obtained by photographing the same scene at the same viewpoint and at the same time.

[0214] The depth calculation unit 111a generates a depth image based on the intensity of infrared light forming the IR image. Fig.11A The depth image shown in (c) is shown in Fig.11A In the bright and dark boundary area of ​​the BW image shown in (a), the outline of the object, such as the white line of the crosswalk drawn on the road surface, is clearly reflected. Fig.11A In the depth image shown in (c), the depth of the area corresponding to the bright and dark boundary area is not clearly expressed.

[0215] The depth correction portion 111b is formed by Fig.11A The BW image, IR image, and depth image shown in (a) to (c) are input to the learning model 104 to obtain the Fig.11A The corrected depth image shown in (d).

[0216] As a result, in the depth acquisition device 1 according to the present modification, the corrected depth image can be made close to Fig.11A The depth image of the correct solution shown in (e).

[0217] Figure 1 1B shows another example of the simulation result of the depth acquisition device 1 according to the present modification.

[0218] The depth acquisition device 1 acquires the depth through the image captured by the BW camera 103. Figure 1 The BW image shown in (a) of FIG. 1B is obtained by the IR camera 102. Fig. 11B The IR image shown in (b) is shown in Figure 1 In the example shown in 1B, Fig.11A In the illustrated example, the BW image and the IR image are images obtained by photographing the same scene from the same viewpoint and at the same time.

[0219] The depth calculation unit 111a generates a depth image based on the intensity of the infrared light forming the IR image. Figure 1 The depth image shown in (c) of 1B. Figure 1 In the bright-dark boundary region of the BW image shown in (a) of FIG1B , the outline of an object, such as the outline of the white line of a crosswalk drawn on the road surface, is clearly reflected. Figure 1In the depth image shown in (c) of FIG1B , the depth of the region corresponding to the bright-dark boundary region is not clearly expressed.

[0220] The depth correction part 1 1 1b is formed by Figure 1 The BW image, IR image, and depth image shown in (a) to (c) of FIG1B are input to the learning model 104, and the learning model 104 obtains Fig. 11B The corrected depth image shown in (d).

[0221] As a result, in the depth acquisition device 1 according to the present modification, the corrected depth image can be made close to Fig. 11B The depth image of the correct solution shown in (e).

[0222] As described above, in the depth acquisition device 1 in the present embodiment and its modified example, even when the boundary between two objects having different reflectances to infrared light is imaged, the depth around the boundary can be accurately acquired.

[0223] In addition, in each of the above embodiments, each component can be formed by dedicated hardware, or can be implemented by executing a software program suitable for each component. Each component can be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software that implements the depth acquisition device of the above embodiments and modified examples causes a computer to execute Figure 6 , Figure 7 as well as Fig. 9 The steps contained in any of the flowcharts.

[0224] The depth acquisition device involved in one or more schemes is described above based on the embodiment and its variants, but the present disclosure is not limited to the embodiment and its variants. As long as it does not deviate from the main purpose of the present disclosure, the scheme obtained by implementing various modifications thought of by those skilled in the art to the embodiment and its variants, and the form constructed by combining the constituent elements in the embodiment and its variants can also be included in the scope of the present disclosure.

[0225] For example, the depth calculation unit 111a calculates the depth based on three infrared light intensities measured at three different exposure timings. However, the number of exposure timings and infrared light intensities is not limited to three, and may be four or more.

[0226] In addition, in the modified example of the above embodiment, the depth acquisition device 1 includes the learning model 104, but it may not include the learning model 104. In this case, the depth acquisition device 1 inputs the above-mentioned input data to the learning model 104 via a communication network, for example, and acquires the output data from the learning model 104, that is, the corrected depth image, via the communication network.

[0227] In addition, in the present disclosure, all or part of a component, device, or Figure 1 , Figure 4 ,as well as Fig.10 All or part of the functional blocks of the block diagram shown can be executed by a semiconductor device, one or more electronic circuits including a semiconductor integrated circuit (IC) or LSI (large-scale integration). LSI or IC can be integrated on a single chip, or can be composed of a combination of multiple chips. For example, functional blocks other than storage elements can be integrated on a single chip. Here, they are called LSI or IC, but the name changes depending on the degree of integration, and may be called system LSI, VLSI (very large scale integration) or ULSI (ultra large scale integration). For the same purpose, a field programmable gate array (FPGA) programmed after the manufacture of LSI, or a reconfigurable logic device (reconiigurable logic device) that can reconstruct the connection relationship inside the LSI or set the circuit division inside the LSI can also be used.

[0228] Furthermore, all or part of the functions or operations of a component, device or part of a device can be executed by software processing. In this case, the software is recorded on one or more non-temporary recording media such as ROM, optical disk, hard disk drive, etc., and when the software is executed by a processor, the software enables the processor and peripheral devices to perform specific functions in the software. The system or device can have one or more non-temporary recording media recording the software, a processor, and necessary hardware devices such as interfaces.

[0229] Industrial Applicability

[0230] The present disclosure can be applied to a depth acquisition device that acquires the distance to a subject, that is, the depth, and can be used as, for example, an in-vehicle device.

[0231] Description of Reference Numerals

[0232] 1Depth acquisition device

[0233] 10.101 Light Source

[0234] 20Solid-state imaging element

[0235] 211st pixel (IR)

[0236] 22 2nd pixel (BW)

[0237] 30 processing circuit

[0238] 50 diffuser plate

[0239] 60 lenses

[0240] 70 Bandpass Filter

[0241] 102 IR Camera

[0242] 103 BW Camera

[0243] 104 Learning Model

[0244] 110a, 110b processor

[0245] 111a Depth calculation unit

[0246] 111b, 111c depth correction part

[0247] 112 Boundary Area Detection Unit

[0248] 113 Lighting control unit

[0249] 114IR Acquisition Department

[0250] 115BW Acquisition Department

[0251] 200 memory.

Claims

1. A camera method, comprising the following steps: By taking pictures of the light emitted from the light source and reflected by the subject, an infrared light image is obtained. By photographing the subject with visible light to obtain a visible light image, An edge area is detected based on the infrared light image and the visible light image.

2. The imaging method according to claim 1, wherein: The edge region is detected according to the visible light image.

3. The imaging method according to claim 1, wherein: In the detection of the edge area, Detecting edges with strengths above a first threshold, The edge region is a region extending from the edge having an intensity equal to or greater than the first threshold value to a distance corresponding to a change in the visible light image.

4. The imaging method according to claim 1, wherein: In the detection of the edge region, A region in which the difference in pixel value between the visible light image and a past visible light image is greater than or equal to a second threshold is detected as the edge region.

5. The imaging method according to claim 1, wherein: The edge region is detected based on the infrared light image and a visible light image generated by capturing the same scene as the infrared light image.

6. The imaging method according to claim 1, wherein: The edge region is detected based on the infrared light image and a visible light image generated by capturing images from the same viewpoint as the infrared light image.

7. The imaging method according to claim 1, wherein: The edge region is detected based on the infrared light image and a visible light image generated by imaging at the same time as the infrared light image.

8. The imaging method according to claim 1, wherein: In acquiring the infrared light image, the infrared light camera uses ToF, or time of flight.

9. A camera device comprising: light source; an infrared camera that receives light irradiated from the light source and reflected by the subject, thereby acquiring an infrared image; A visible light camera that obtains a visible light image by photographing the subject using visible light; and The edge region detecting unit detects an edge region based on the infrared light image and the visible light image.

10. A recording medium storing a program for causing a computer to execute the following steps: By taking pictures of the light emitted from the light source and reflected by the subject, an infrared light image is obtained. By photographing the subject with visible light to obtain a visible light image, An edge area is detected based on the infrared light image and the visible light image.

Images