Information processing method, information processing apparatus, and program product

Abstract

An information processing method, an information processing apparatus, and a program product are provided. An information processing method acquires a first image by imaging light irradiated from a light source and reflected by a subject, acquires a second image by imaging the subject with visible light, and detects a low-reflection region on the basis of the first image and the second image.

Description

[0001] This application is a divisional application of the invention application with an international application date of July 18, 2019 and a national application number of 201980049530.6. The invention name of the invention application is depth acquisition device, depth acquisition method and recording medium. Technical Field

[0002] The present disclosure relates to a depth acquisition device and the like that acquires the distance to a subject as the depth. Background Art

[0003] In the past, a distance measuring device for measuring the distance to a subject has been proposed (for example, refer to Patent Document 1). The distance measuring device includes a light source and an imaging unit. The light source irradiates light to the subject. The imaging unit captures the reflected light reflected by the subject. Then, the distance measuring device measures the distance to the subject by converting each pixel value of the image obtained by the imaging into the distance to the subject. That is, the distance measuring device obtains the depth of the image obtained by the imaging unit.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2011-64498 Summary of the invention

[0007] Problems to be solved by the invention

[0008] However, the distance measuring device of Patent Document 1 has a problem that the depth cannot be accurately obtained.

[0009] To this end, the present disclosure provides a depth acquisition device that can accurately acquire the distance to a subject, that is, the depth.

[0010] Means for solving problems

[0011] A depth acquisition device involved in one scheme of the present disclosure includes a memory and a processor, wherein the processor acquires the intensity of infrared light irradiated from a light source and reflected by a subject, the intensity of the infrared light being measured by imaging the infrared light received by a plurality of pixels included in an imaging element, and stored in the memory, and the distance to the subject is calculated as the depth for each of the plurality of pixels included in the imaging element based on the intensity of the infrared light received in the pixel, thereby generating a depth image, and acquiring a visible light image, the visible light image being generated by imaging a scene substantially the same as the infrared light image formed based on the intensity of the infrared light received by the plurality of pixels included in the imaging element, and at substantially the same viewpoint and time as the infrared light image, and stored in the memory, and detecting an area in the infrared light image where an object with low reflectivity to infrared light in the subject is reflected, i.e., a low-reflection area, based on the infrared light image and the visible light image, and correcting the low-reflection area of ​​the depth image based on the visible light image, and outputting the corrected depth image.

[0012] In addition, these general or specific solutions can be implemented by a system, method, integrated circuit, computer program or a computer-readable CD-ROM or other recording medium, or by any combination of the system, method, integrated circuit, computer program and recording medium. In addition, the recording medium can be a non-temporary recording medium.

[0013] Effects of the Invention

[0014] The depth acquisition device disclosed in the present invention can correctly acquire the distance to the subject, i.e., the depth. Further advantages and effects in one scheme of the present invention will be clear from the specification and the drawings. The relevant advantages and / or effects are provided respectively by several embodiments and the features recorded in the specification and the drawings, but it is not necessary to provide all the features in order to obtain one or more of the same features. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a block diagram showing the hardware configuration of the depth acquisition device in the embodiment.

[0016] Figure 2 It is a schematic diagram showing a pixel array included in the solid-state imaging element according to the embodiment.

[0017] Figure 3 This is a timing chart showing the relationship between the light emission timing of the light emitting element of the light source and the exposure timing of the first pixel of the solid-state imaging element in the embodiment.

[0018] Figure 4 This is a block diagram showing an example of the functional structure of the depth acquisition device in the embodiment.

[0019] Figure 5 The diagram shows an example of a BW image, an IR image, and a depth image.

[0020] Figure 6 The diagrams show other examples of the BW image, the IR image, and the depth image.

[0021] Figure 7 It is a diagram showing the simulation results of the depth acquisition device in the embodiment.

[0022] Figure 8 This is a flowchart showing the overall processing operation of the depth acquisition device in the embodiment.

[0023] Fig. 9 This is a flowchart showing an example of the processing operation of the low-reflection area detection unit in the embodiment.

[0024] Fig.10 This is a flowchart showing another example of the processing operation of the low-reflection area detection unit in the embodiment.

[0025] Fig.11 This is a diagram showing an example of correction of a depth image performed by the depth correction unit in the embodiment.

[0026] Fig.12 This is a flowchart showing the correction processing operation performed by the depth correction unit in the embodiment.

[0027] Fig.13 It is a diagram showing another example of the correction of the depth image performed by the depth correction unit in the embodiment.

[0028] Fig.14 This is a block diagram showing an example of the functional configuration of a depth acquisition device in a modified example of the embodiment. DETAILED DESCRIPTION

[0029] (Insights that form the basis of this disclosure)

[0030] The inventors of the present invention have found that the following problems occur with respect to the distance measuring device of Patent Document 1 described in the “Background Art” column.

[0031] The distance measuring device of Patent Document 1 irradiates light from a light source to a subject as described above, captures an image of the irradiated subject, thereby obtaining an image, and measuring the depth of the image. ToF (Time Of Flight) is used in the measurement of the depth. In such a distance measuring device, different imaging conditions are used to improve the distance measurement accuracy. That is, the distance measuring device captures images according to given imaging conditions, and sets imaging conditions different from the given imaging conditions according to the imaging results. Then, the distance measuring device captures images again according to the set imaging conditions.

[0032] However, in the distance measuring device of Patent Document 1, it is difficult to accurately measure the distance to a subject having a low reflectivity to irradiated light, that is, the depth, even if the imaging conditions are changed.

[0033] In order to solve such a problem, a depth acquisition device involved in one scheme of the present disclosure includes a memory and a processor, wherein the processor acquires the intensity of infrared light irradiated from a light source and reflected by a subject, the intensity of the infrared light being measured by imaging the infrared light received by multiple pixels included in an imaging element, and stored in the memory, and the distance to the subject is calculated as the depth for each of the multiple pixels included in the imaging element based on the intensity of the infrared light received in the pixel, thereby generating a depth image, and acquiring a visible light image, wherein the visible light image is generated by imaging a scene substantially the same as the infrared light image formed based on the intensity of the infrared light received by the multiple pixels included in the imaging element, and at substantially the same viewpoint and time as the infrared light image, and stored in the memory, and an area where an object with low reflectivity to infrared light in the subject is reflected in the infrared light image, i.e., a low-reflection area, is detected based on the infrared light image and the visible light image, and the low-reflection area of ​​the depth image is corrected based on the visible light image, and the corrected depth image is output. In addition, the so-called low reflection area of ​​the depth image is an area in the depth image that is located at the same position as the low reflection area in the infrared light image and has the same shape and size as the low reflection area. Hereinafter, such a low reflection area of ​​the depth image is also referred to as a correction target area.

[0034] For example, if the subject includes an object with a low reflectivity to infrared light (hereinafter referred to as a low-reflection object), the distance to the low-reflection object, i.e., the depth, calculated based on the intensity of infrared light reflected by the low-reflection object will be calculated to be longer than the actual depth. That is, the area corresponding to the low-reflection object in the depth image does not show the correct depth.

[0035] However, in the depth acquisition device involved in the above-mentioned scheme, the area where the low-reflection object is reflected, that is, the low-reflection area, is detected in the infrared light image, and the area in the depth image corresponding to the low-reflection area is set as the correction target area. That is, since the subject includes a low-reflection object, the area in the depth image showing an inappropriate depth is determined as the correction target area. Then, by correcting the depth of the correction target area, the distance to the low-reflection object, that is, the depth, can be correctly acquired.

[0036] Here, as an example of an image of substantially the same scene captured at substantially the same viewpoint and time, there is an image captured by different pixels of the same imaging element. Such an image is similar to the red, green and blue channel images of a color image captured by a Bayer arrangement color filter, and the viewing angle, viewpoint and shooting time of each image are roughly equal. That is, the position of the image of the substantially same scene captured at substantially the same viewpoint and time on the image of the subject in each captured image will not differ by more than 2 pixels. For example, when there is a point light source with visible light and infrared components in the scene, and only one pixel is captured at high brightness in the visible light image, in the infrared light image, the point light source will also be captured closer than 2 pixels to the pixel corresponding to the pixel position captured in the visible light image. In addition, the so-called capturing at substantially the same time means that the difference in the shooting time is equal to less than 1 frame.

[0037] Alternatively, in the detection of the low-reflection area, the area in which the object is reflected in the visible light image is detected as the object area, and it is determined whether the brightness of the area in the infrared light image corresponding to the object area, i.e., the determination object area, is below a first threshold value, and the determination object area having a brightness determined to be below the first threshold value is detected as the low-reflection area. For example, the object area is detected by processing such as pattern matching or image recognition. In addition, the area in the infrared light image corresponding to the object area is an area in the infrared light image that is located at the same position as the object area in the visible light image and has the same shape and size as the object area.

[0038] Thus, the area where the object is reflected is detected from the visible light image as the object area. The object reflected in the object area detected in this way is not a distant background but is located nearby, and there is a high possibility that the infrared light from the light source irradiates the object. Furthermore, if the brightness in the area within the infrared light image corresponding to the object area is low, there is a high possibility that the object is a low-reflection object. Therefore, in the depth acquisition device involved in one of the above schemes, since the area with a brightness below the first threshold value within the infrared light image corresponding to the object area is detected as a low-reflection area, the low-reflection area can be appropriately detected.

[0039] In addition, in the detection of the low-reflection area, it is further possible to determine whether the noise intensity contained in the intensity of the infrared light in the determination object area among the intensity of the infrared light obtained from the memory is below a second threshold value, and the determination object area with a noise intensity determined to be below the second threshold value is detected as the low-reflection area.

[0040] For example, in the subject, there is a case where it is irradiated not only by the infrared light from the light source but also by the infrared light contained in external light such as sunlight. For this reason, the intensity of the infrared light contained in the external light is treated as noise intensity, and an infrared light image can be formed based on the intensity of the infrared light other than the noise intensity. However, if the noise intensity is large, such as in bright daytime, the brightness of the infrared light image becomes low in order to remove the noise intensity. Therefore, if the noise intensity is large, there is a possibility that even if the brightness of the area in the infrared light image corresponding to the object area of ​​the visible light image is low as described above, the area is not a low-reflection area. Therefore, in the depth acquisition device involved in one of the above schemes, since the area with a small noise intensity determined to be below the second threshold is detected as a low-reflection area, the low-reflection area can be detected more appropriately.

[0041] Alternatively, the intensity of the infrared light obtained from the memory may include at least three intensities measured by exposing the imaging element at least three different timed exposures when the infrared light irradiated from the light source and reflected from the subject is received by the imaging element, and in generating the depth image, the depth image is generated by calculating the depth of each of the plurality of pixels contained in the imaging element based on the at least three intensities measured for the pixels.

[0042] This makes it possible to generate a depth image with high precision using ToF.

[0043] Alternatively, the noise intensity may be the smallest intensity among the at least three intensities.

[0044] Thereby, the noise intensity can be appropriately obtained.

[0045] In addition, in the correction of the low-reflection area of ​​the depth image, the category of the object reflected in the area is determined based on the image of the area in the visible light image corresponding to the low-reflection area, and the depth of the area is estimated based on the determined category of the object and the size of the object reflected in the area, so as to correct the depth of the low-reflection area of ​​the depth image. For example, the category of the object is determined by processing such as pattern matching or image recognition. In addition, the area in the visible light image corresponding to the low-reflection area is an area in the visible light image that is located at the same position as the low-reflection area in the infrared light image and has the same shape and size as the low-reflection area.

[0046] Therefore, as long as the correspondence between the type and size of the object and the depth is known, the appropriate depth can be estimated based on the type and size of the object determined from the image of the area in the visible light image corresponding to the low-reflection area. As a result, the depth of the low-reflection area of ​​the depth image, i.e., the correction target area, can be corrected to the correct depth.

[0047] Alternatively, the depth of the correction target region may be corrected to the depth of a region located below the correction target region without using a visible light image.

[0048] For example, when the low-reflective object is a vehicle, the road surface is in front of the vehicle, and the depth of the road surface is similar to the depth of the vehicle. Therefore, in the depth acquisition device involved in the above-mentioned scheme, since the depth of the correction target area is corrected to the depth of the area below the correction target area, the depth of the correction target area can be corrected to the correct depth.

[0049] Alternatively, other schemes of the present disclosure involve a depth acquisition device comprising a memory and a processor, wherein the processor acquires the intensity of infrared light irradiated from a light source and reflected by a subject, the intensity of the infrared light being measured by imaging a plurality of pixels included in an imaging element respectively receiving the infrared light, and the processor is stored in the memory, and the distance to the subject is calculated as the depth for each of the plurality of pixels included in the imaging element based on the intensity of the infrared light received in the pixel, thereby generating a depth image, acquiring a visible light image, the visible light image being generated by imaging a scene based on visible light and at a viewpoint and time substantially the same as the infrared light image formed based on the intensity of the infrared light received by the plurality of pixels included in the imaging element, and the processor is stored in the memory, and the depth of the depth image is corrected by inputting the depth image, the infrared light image, and the visible light image into a learning model.

[0050] Therefore, if the learning model is pre-learned so as to output a corrected depth image for the input of a depth image, an infrared light image, and a visible light image, a suitable depth image can be easily obtained without detecting a low-reflection area.

[0051] In addition, these general or specific solutions can be implemented by a system, method, integrated circuit, computer program or computer-readable CD-ROM or other recording medium, or by any combination of systems, methods, integrated circuits, computer programs or recording media. In addition, the recording medium can be a non-transitory recording medium.

[0052] The embodiments are described in detail below with reference to the drawings.

[0053] In addition, the embodiments described below all represent general or specific examples. The numerical values, shapes, materials, components, configuration positions of components, connection forms, steps, and the order of steps shown in the following embodiments are examples and do not limit the subject matter of the present disclosure. In addition, the components in the following embodiments that are not recorded in the independent claims representing the highest concept are described as arbitrary components.

[0054] In addition, each figure is a schematic diagram, and does not necessarily show the figure strictly. In addition, in each figure, the same reference numeral is attached|subjected to the same component.

[0055] (Implementation Method)

[0056] [Hardware structure]

[0057] Figure 1 This is a block diagram showing the hardware structure of the depth acquisition device 1 involved in the embodiment. The depth acquisition device 1 in this embodiment has a hardware structure that can acquire an image based on infrared light (or near-infrared light) and an image based on visible light by photographing substantially the same scene at substantially the same viewpoint and time. In addition, the so-called substantially the same means the same degree as that of being able to produce the effect in the present disclosure.

[0058] like Figure 1 As shown, the depth acquisition device 1 includes a light source 10 , a solid-state imaging element 20 , a processing circuit 30 , a diffusion plate 50 , a lens 60 , and a bandpass filter 70 .

[0059] The light source 10 emits illumination light. More specifically, the light source 10 emits illumination light for illuminating a subject at a timing indicated by a light emission signal generated in the processing circuit 30 .

[0060] The light source 10 is composed of, for example, a capacitor, a driving circuit, and a light-emitting element, and emits light by driving the light-emitting element with electric energy stored in the capacitor. The light-emitting element is realized by, for example, a laser diode, a light-emitting diode, etc. In addition, the light source 10 may be a structure including one type of light-emitting element, or may be a structure including multiple types of light-emitting elements according to the purpose.

[0061] In the following, the light emitting element is, for example, a laser diode that emits near-infrared light, or a light emitting diode that emits near-infrared light. However, the irradiation light irradiated by the light source 10 is not necessarily limited to near-infrared light. The irradiation light irradiated by the light source 10 may be, for example, infrared light (also referred to as infrared light) in a frequency band other than near-infrared light. In the following, in this embodiment, the irradiation light irradiated by the light source 10 is described as infrared light, but the infrared light may be near-infrared light or infrared light in a frequency band other than near-infrared light.

[0062] The solid-state imaging element 20 captures an image of a subject and outputs an imaging signal indicating an exposure amount. More specifically, the solid-state imaging element 20 performs exposure at a timing indicated by an exposure signal generated by the processing circuit 30 and outputs an imaging signal indicating an exposure amount.

[0063] The solid-state imaging device 20 has a pixel array in which first pixels for imaging using reflected light of an irradiated light reflected from an object and second pixels for imaging the object are arranged in an array. The solid-state imaging device 20 may have logic functions such as a cover glass and an AD converter as necessary.

[0064] In the following description, the reflected light is assumed to be infrared light, similarly to the irradiated light. However, the reflected light is not limited to infrared light as long as it is light obtained by reflecting the irradiated light from the subject.

[0065] Figure 2 Schematic diagram showing the pixel array 2 included in the solid-state imaging element 20 .

[0066] like Figure 2 As shown, the pixel array 2 is arranged in an array such that first pixels 21 (IR pixels) for capturing images using reflected light obtained by irradiating light reflected from a subject and second pixels 22 (BW pixels) for capturing images of the subject are alternately arranged in columns.

[0067] In addition, Figure 2 In the pixel array 2, the second pixel 22 and the first pixel 21 are arranged adjacent to each other in the row direction and arranged in a strip shape in the row direction, but this is not limited to this, and they may be arranged every multiple rows (for example, every 2 rows). That is, the second pixel 22 is arranged in the first row adjacent to each other in the row direction and the first pixel 21 is arranged in the second row adjacent to each other in the row direction, and they may be arranged alternately every M rows (M is a natural number). Furthermore, the second pixel 22 is arranged in the first row adjacent to each other in the row direction and the first pixel 21 is arranged in the second row adjacent to each other in the row direction, and they may be arranged separated by different rows (alternatingly repeating the first row as N rows and the second row as L rows (N and L are different natural numbers)).

[0068] The first pixel 21 is realized by, for example, an infrared light pixel having sensitivity to infrared light as reflected light, and the second pixel 22 is realized by, for example, a visible light pixel having sensitivity to visible light.

[0069] The infrared light pixel is composed of, for example, an optical filter (also referred to as an IR filter) that transmits only infrared light, a microlens, a light-receiving element as a photoelectric conversion unit, and a storage unit that stores the charge generated by the light-receiving element. Therefore, an image representing the brightness of infrared light is expressed by the imaging signal output from the plurality of infrared light pixels (i.e., the first pixel 21) included in the pixel array 2. Hereinafter, the infrared light image is also referred to as an IR image or an infrared light image.

[0070] In addition, the visible light pixel is composed of, for example, an optical filter (also called a BW filter) that transmits only visible light, a microlens, a light receiving element as a photoelectric conversion unit, and a storage unit that stores the charge converted by the light receiving element. Therefore, the visible light pixel, that is, the second pixel 22, outputs an imaging signal representing brightness and color difference. That is, a color image representing the brightness and color difference of visible light is expressed by the imaging signal output from the plurality of second pixels 22 included in the pixel array 2. In addition, the optical filter of the visible light pixel can transmit both visible light and infrared light, or can transmit only light in a specific wavelength band such as red (R), green (G) or blue (B) among the visible light.

[0071] In addition, the visible light pixel can detect only the brightness of the visible light. In this case, the visible light pixel, i.e., the second pixel 22, outputs an imaging signal indicating the brightness. Therefore, the imaging signal outputted from the plurality of second pixels 22 contained in the pixel array 2 represents a black and white image indicating the brightness of the visible light, in other words, a monochrome image. Hereinafter, the monochrome image is also referred to as a BW image. In addition, the above-mentioned color image and BW image are also collectively referred to as a visible light image.

[0072] Back again Figure 1 , continue with the description of the acquisition device 1 in depth.

[0073] The processing circuit 30 calculates subject information related to the subject using the imaging signal outputted from the solid-state imaging element 20 .

[0074] The processing circuit 30 includes, for example, a microcomputer or other computing device. The microcomputer includes a processor (microprocessor), a memory, etc., and the processor executes a driver program stored in the memory to generate a light emission signal and an exposure signal. In addition, the processing circuit 30 may use an FPGA or an ISP, etc., and may include one hardware or multiple hardware.

[0075] The processing circuit 30 calculates the distance to the subject by, for example, a ToF distance measurement method using the imaging signal from the first pixel 21 of the solid-state imaging element 20 .

[0076] The calculation of the distance to the subject by the ToF distance measurement method performed by the processing circuit 30 will be described below with reference to the drawings.

[0077] Figure 3 This is a timing chart showing the relationship between the light emission timing of the light emitting element of the light source 10 and the exposure timing of the first pixel 21 of the solid-state imaging element 20 when the processing circuit 30 calculates the distance to the subject using the ToF distance measurement method.

[0078] exist Figure 3 In the figure, Tp is the light emission period during which the light emitting element of the light source 10 emits the irradiation light, and Td is the delay time from when the light emitting element of the light source 10 emits the irradiation light to when the reflected light of the irradiation light reflected by the subject returns to the solid-state imaging element 20. Furthermore, the first exposure period becomes the same timing as the light emission period during which the light source 10 emits the irradiation light, and the second exposure period becomes the timing from the end time point of the first exposure period to the passage of the light emission period Tp.

[0079] exist Figure 3 In the figure, q1 represents the total amount of exposure in the first pixel 21 of the solid-state imaging element 20 caused by the reflected light during the first exposure period, and q2 represents the total amount of exposure in the first pixel 21 of the solid-state imaging element 20 caused by the reflected light during the second exposure period.

[0080] By Figure 3 The light emitting element of the light source 10 emits the irradiation light and the first pixel 21 of the solid-state imaging element 20 exposes the light at the timing shown, and assuming that the speed of light is c, the distance d to the subject can be expressed by the following (Equation 1).

[0081]

[0082] Therefore, the processing circuit 30 can calculate the distance to the subject using the imaging signal from the first pixel 21 of the solid-state imaging element 20 by using (Formula 1).

[0083] In addition, the plurality of first pixels 21 of the solid-state imaging element 20 may be exposed only in the third exposure period Tp after the first exposure period and the second exposure period have ended. The plurality of first pixels 21 can detect noise other than the reflected light based on the exposure amount obtained in the third exposure period Tp. That is, the processing circuit 30 can more accurately calculate the distance d to the subject by removing the noise from the exposure amount q1 in the first exposure period and the exposure amount q2 in the second exposure period in the above (Formula 1).

[0084] Back again Figure 1 , continue with the description of the acquisition device 1 in depth.

[0085] The processing circuit 30 can detect the subject and calculate the distance to the subject using the imaging signal from the second pixel 22 of the solid-state imaging element 20 , for example.

[0086] That is, the processing circuit 30 can detect the subject and calculate the distance to the subject based on the visible light image captured by the plurality of second pixels 22 of the solid-state imaging element 20. Here, the detection of the subject can be achieved, for example, by using edge detection of the singularity of the subject to distinguish the shape by pattern recognition, or it can be achieved by using a learning model that has been learned in advance through deep learning and other processes. In addition, the calculation of the distance to the subject can be performed using world coordinate transformation. Of course, it is also possible to detect the subject by multimodal learning processing using not only the visible light image but also the brightness and distance information of the infrared light captured by the first pixel 21.

[0087] The processing circuit 30 generates a light emission signal indicating a timing for light emission and an exposure signal indicating a timing for exposure. The processing circuit 30 then outputs the generated light emission signal to the light source 10 and outputs the generated exposure signal to the solid-state imaging element 20 .

[0088] The processing circuit 30 can, for example, generate and output a light emission signal to cause the light source 10 to emit light at a given cycle, and generate and output an exposure signal to cause the solid-state imaging element 20 to be exposed at a given cycle, thereby enabling the depth acquisition device 1 to achieve continuous imaging at a given frame rate. In addition, the processing circuit 30 includes, for example, a processor (microprocessor), a memory, etc., and generates a light emission signal and an exposure signal by executing a driver program stored in the memory by the processor.

[0089] The diffuser 50 adjusts the intensity distribution and angle of the irradiated light. In addition, in the adjustment of the intensity distribution, the diffuser 50 makes the intensity distribution of the irradiated light from the light source 10 uniform. Figure 1 In the illustrated example, the depth acquisition device 1 includes the diffusion plate 50 , but may not include the diffusion plate 50 .

[0090] The lens 60 is an optical lens that focuses light entering from the outside of the depth acquisition device 1 on the surface of the pixel array 2 of the solid-state imaging element 20 .

[0091] The bandpass filter 70 is an optical filter that transmits infrared light as reflected light and visible light. Figure 1 In the illustrated example, the depth acquisition device 1 includes the bandpass filter 70 , but may not include the bandpass filter 70 .

[0092] The depth acquisition device 1 of the above structure is mounted on a transport device and used. For example, the depth acquisition device 1 is mounted on a vehicle traveling on a road and used. In addition, the transport device on which the depth acquisition device 1 is mounted is not necessarily limited to a vehicle. The depth acquisition device 1 can also be mounted on a transport device other than a vehicle such as a motorcycle, a ship, or an airplane and used.

[0093] [Functional structure of depth acquisition device]

[0094] The depth acquisition device 1 in this embodiment is used Figure 1 The hardware structure shown acquires an IR image and a BW image by photographing substantially the same scene and substantially the same viewpoint and time. Here, the IR image is formed based on the intensity of infrared light received by each of the plurality of first pixels 21 contained in the solid-state imaging element 20. Therefore, the depth acquisition device 1 acquires the intensity of infrared light in each of the plurality of first pixels 21 by photographing the IR image. The depth acquisition device 1 acquires a depth image that represents the distance to the subject projected in the IR image as the depth based on the intensity of the infrared light of these first pixels 21. Then, the depth acquisition device 1 detects a low-reflection area in the IR image based on the IR image and the BW image, and corrects the depth of the area in the depth image corresponding to the low-reflection area. In addition, in the present disclosure, the second area in the second image corresponding to the first area in the first image is an area in the second image that is located at the same position as the first area in the first image and has the same shape and size as the first area. The first image and the second image are arbitrary images, and the first area and the second area are also arbitrary areas.

[0095] Figure 4 This is a block diagram showing an example of the functional structure of the depth acquisition device 1 .

[0096] The depth acquisition device 1 includes a light source 101, an IR camera 102, a BW camera 103, a processor 110a, and a memory 200. In addition, the depth acquisition device 1 in this embodiment includes the light source 101, the IR camera 102, and the BW camera 103, but may include only the processor 110a and the memory 200 without these components.

[0097] The light source 101 may include Figure 1 The light source 10 and the diffuser plate 50 shown in the figure emit light to irradiate the subject with infrared light.

[0098] The IR camera 102 may be referred to as an infrared camera and may include Figure 1The IR camera 102 includes a plurality of first pixels 21, a lens 60, and a bandpass filter 70 of the solid-state imaging element 20 shown in the figure. Such an IR camera 102 acquires an IR image by performing infrared light imaging of a scene including the subject in accordance with the timing of the light source 101 irradiating the subject with infrared light. In addition, the IR camera 102 measures the intensity of infrared light by imaging the IR image. That is, the IR camera 102 includes a solid-state imaging element 20, and measures the intensity of infrared light by performing imaging of the plurality of first pixels 21 included in the solid-state imaging element 20 receiving infrared light irradiated from the light source 101 and reflected by the subject.

[0099] The BW camera 103 may be referred to as a visible light camera and may include Figure 1 The solid-state imaging device 20 shown in FIG. 1 includes a plurality of second pixels 22, a lens 60, and a bandpass filter 70. Such a BW camera 103 acquires a visible light image (specifically, a BW image) by performing imaging based on visible light of a scene substantially identical to the IR image and performing imaging at substantially the same viewpoint and time as the IR image. That is, the BW camera 103 generates a BW image by performing imaging based on visible light of a scene substantially identical to the IR image formed based on the intensity of infrared light received by each of the plurality of first pixels 21 included in the solid-state imaging device 20 and performing imaging at substantially the same viewpoint and time as the IR image.

[0100] The memory 200 is a recording medium for storing an IR image obtained by the imaging performed by the IR camera 102 and a BW image obtained by the imaging performed by the BW camera 103. In addition, as described above, the IR image is formed based on the intensity of infrared light respectively received by the plurality of first pixels 21 contained in the solid-state imaging element 20. Therefore, the IR image shows the intensity of infrared light for each pixel. That is, the memory 200 can be said to store the intensity of infrared light. In addition, such a memory 200 can specifically be a ROM (Read Only Memory), a RAM (Random access memory) or an SSD (solid state drive), etc., and can be non-volatile or volatile. In addition, the memory 200 can be a hard disk.

[0101] The processor 110a obtains the IR image and the BW image from the memory 200, calculates the depth image from the IR image, and detects the low reflection area in the IR image based on the IR image and the BW image. Then, the processor 110a corrects the depth of the area in the depth image corresponding to the low reflection area. In addition, the low reflection area is an area reflected in the IR image for an object with low infrared light reflectivity. The low reflection area is an area detected using the first threshold value described later, or the first threshold value and the second threshold value, and the reflectivity of the object is defined by the above threshold value.

[0102] Such a processor 110 a includes a light emission control unit 113 , an IR acquisition unit 114 , a BW acquisition unit 115 , a depth calculation unit 111 a , a low-reflection region detection unit 112 , and a depth correction unit 111 b .

[0103] The light control unit 113 controls the light source 101. That is, the light control unit 113 outputs the light emission signal to the light source 101 to make the light source 101 emit light. Thus, infrared light is irradiated from the light source 101 to the subject, and the light reflected by the subject, that is, the reflected light, enters the IR camera 102.

[0104] The IR acquisition unit 114 acquires an IR image from the IR camera 102 via the memory 20. Here, each pixel of the IR image represents the intensity of the infrared light received at the position of the pixel as a pixel value (specifically, brightness). Therefore, the IR acquisition unit 114 acquires the intensity of the infrared light by acquiring the IR image. That is, the IR acquisition unit 114 acquires the intensity of the infrared light irradiated from the light source 101 and reflected by the subject, and the intensity of the infrared light is measured by the plurality of first pixels 21 included in the solid-state imaging element 20 receiving the infrared light respectively, and is stored in the memory 200.

[0105] The BW acquisition unit 115 acquires a BW image from the BW camera 103 via the memory 200. That is, the BW acquisition unit 115 acquires a BW image generated by imaging substantially the same scene as the IR image formed based on the intensity of infrared light received by each of the plurality of first pixels 21 included in the solid-state imaging element 20 using visible light and imaging at substantially the same viewpoint and time as the IR image, and stores the BW image in the memory 200.

[0106] The depth calculation unit 111 a generates a depth image by calculating the distance to the subject as the depth for each of the plurality of first pixels 21 included in the solid-state imaging element 20 based on the intensity of infrared light received by the first pixel 2 .

[0107] The low-reflection region detection unit 112 detects a low-reflection region, which is a region of an object having a low reflectivity to infrared light among the objects reflected in the IR image, based on the IR image and the BW image.

[0108] The depth correction unit 111 b corrects the depth of the correction target area, which is an area in the depth image corresponding to the low-reflection area.

[0109] Even if the subject includes an object having a low reflectivity to infrared light (ie, a low-reflection object), the depth acquisition device 1 in this embodiment can accurately acquire the distance to the subject, ie, the depth.

[0110] For example, if the subject includes a low-reflection object, the distance to the low-reflection object, i.e., the depth, calculated based on the intensity of infrared light reflected by the low-reflection object will be longer than the actual depth. That is, the area corresponding to the low-reflection object in the depth image does not show the correct depth.

[0111] However, in the depth acquisition device 1 of the present embodiment, the area where the low-reflection object is reflected, i.e., the low-reflection area, is detected in the IR image, and the area in the depth image corresponding to the low-reflection area is set as the correction target area. That is, since the subject includes the low-reflection object, the area showing an inappropriate depth in the depth image is determined as the correction target area. And, by correcting the depth of the correction target area, the distance to the low-reflection object, i.e., the depth, can be correctly acquired.

[0112] [An example of each image]

[0113] Figure 5 An example of a BW image, an IR image, and a depth image is shown.

[0114] In the BW image generated by the BW camera 103 and acquired by the BW acquisition unit 115, for example, Figure 5 As shown in (a) of FIG. 1 , a plurality of vehicles parked in a parking lot are shown.

[0115] In addition, in the IR image generated by the IR camera 102 and acquired by the IR acquisition unit 114, Figure 5 As shown in (b) Figure 5 The BW image shown in (a) is substantially the same scene. Figure 5 The IR image shown in (b) is obtained by Figure 5 The BW image shown in (a) is an image obtained by taking the image from substantially the same viewpoint and at the same time. Figure 5 The BW image shown in (a) and Figure 5 In the IR images shown in (b), the same object is shown in the corresponding regions. The corresponding regions are regions having the same position, size, and shape in the respective images.

[0116] The depth calculation unit 111a calculates the depth of each of the plurality of first pixels 21 included in the solid-state imaging element 20 based on the intensity of infrared light received at the first pixel 21, thereby generating, for example, Figure 5 The depth image shown in (c) of FIG. The brightness of each pixel in the depth image is higher as the depth is longer, and conversely, it is lower as the depth is shorter. Figure 5 The depth image shown in (c) is Figure 5 The IR image shown in (b) corresponds to . That is, Figure 5 As described above, the IR image shown in (b) is formed based on the intensity of infrared light received by each of the plurality of first pixels 21 included in the solid-state imaging element 20 . Figure 5 The depth image shown in (c) is also generated based on the intensity of the infrared light received by each of the plurality of first pixels 21. Figure 5 The IR image shown in (b) and Figure 5 In the depth image shown in (c) of FIG. 1 , the same object is also reflected in the corresponding regions.

[0117] Here, if Figure 5 As shown in (b), there is a low reflection area in the IR image (i.e. Figure 5 The low reflection area is an area where an object with low reflectivity to infrared light is reflected. Figure 5 In the BW image shown in (a) of FIG. 1 , a vehicle located nearby is reflected in the area corresponding to the low reflection area. Figure 5 In the low-reflection area of ​​the IR image shown in (b), the brightness is low. As a result, Figure 5 As shown in (c) of FIG. 1 , in the area within the depth image corresponding to the low reflection area (i.e. Figure 5 In the region within the dotted circle in (c) of FIG. 1 ), even if the distance to the object, i.e., the depth, is actually short, the depth is expressed as long (i.e., the brightness is high).

[0118] Therefore, the low reflection area detection unit 112 of the depth acquisition device 1 in this embodiment is based on Figure 5 The BW image shown in (a) and Figure 5The IR image shown in (b) is used to detect the low-reflection area. Then, the depth correction unit 111b determines the area within the depth image corresponding to the low-reflection area as the correction target area, and corrects the depth of the correction target area. Specifically, the depth correction unit 111b corrects the low-reflection area of ​​the depth image based on the BW image, and outputs the corrected depth image. In addition, the low-reflection area of ​​the depth image is the area within the depth image corresponding to the low-reflection area of ​​the above-mentioned IR image, which is also called the correction target area. For example, the depth correction unit 111b displays the depth image on a display by outputting the corrected depth image to the display. Alternatively, the depth correction unit 111b may also output the corrected depth image to a control component such as an ECU (Electronic Control Unit) mounted on the vehicle, so that the control component performs driving control of the vehicle based on the corrected depth image.

[0119] Figure 6 Other examples of BW images, IR images, and depth images are shown.

[0120] In the BW image generated by the BW camera 103 and acquired by the BW acquisition unit 115, for example, Figure 6 As shown in (a) of FIG. 8 , a plurality of vehicles parked in the parking lot are shown.

[0121] In addition, in the IR image generated by the IR camera 102 and acquired by the IR acquisition unit 114, Figure 6 As shown in (b), the Figure 6 The BW image shown in (a) is substantially the same scene. Figure 6 The IR image shown in (b) is obtained by Figure 6 The BW image shown in (a) is an image obtained by taking the image from substantially the same viewpoint and at the same time. Figure 6 The BW image shown in (a) and Figure 6 In the IR image shown in (b), the same subject is reflected in the corresponding regions.

[0122] The depth calculation unit 111a calculates the depth of each of the plurality of first pixels 21 included in the solid-state imaging element 20 based on the intensity of infrared light received at the first pixel 21, thereby generating, for example, Figure 6 The depth image shown in (c) is Figure 6 The depth image shown in (c) is Figure 6 The corresponding IR image shown in (b).

[0123] Here, if Figure 6 As shown in (b), there is a low reflection area in the IR image (i.e. Figure 6 The low reflection area is an area where an object with low reflectivity to infrared light is reflected. Figure 6 In the BW image shown in (a) of FIG. 1 , a vehicle located nearby is reflected in the area corresponding to the low reflection area. Figure 6 In the low-reflection area of ​​the IR image shown in (b), the brightness is low. As a result, Figure 6 As shown in (c) of FIG. 1 , in the area within the depth image corresponding to the low reflection area (i.e. Figure 5 In the region within the dotted circle in (c) of FIG. 1 ), even if the distance to the object, i.e., the depth, is actually short, the depth appears to be long (i.e., the brightness is high).

[0124] Therefore, the low reflection area detection unit 112 of the depth acquisition device 1 in this embodiment is based on Figure 6 The BW image shown in (a) and Figure 6 The low reflection area is detected by using the IR image shown in (b) of FIG. Then, the depth correction unit 111 b determines the area in the depth image corresponding to the low reflection area as the correction target area, and corrects the depth of the correction target area.

[0125] Figure 7 The simulation results of the depth acquisition device 1 are shown.

[0126] The depth acquisition device 1 acquires the depth through the imaging of the BW camera 103. Figure 7 The BW image shown in (a) is obtained by the IR camera 102. Figure 7 The IR image shown in (b) of FIG. The BW image and the IR image are images obtained by photographing the same scene at the same viewpoint and at the same time. Figure 7 In the example shown in (b), there is a large low-reflection area on the left side of the IR image (i.e. Figure 7 The area within the dotted circle in (b).

[0127] The depth calculation unit 111a generates a depth image based on the intensity of infrared light forming the IR image. Figure 7 The depth image shown in (c) is Figure 7 In the BW image shown in (a), the area corresponding to the low reflection area is not Figure 7 In the depth image shown in (c) of FIG. 1 , the depth of the low-reflection region is still inappropriately expressed as long. That is, even if an object appears close in the BW image, the object still appears to be far away in the depth image.

[0128] The depth correction part 111b is connected to the low reflection area. Figure 7The area in the depth image shown in (c) is treated as a correction target area, and the depth of the correction target area is corrected to generate Figure 7 For example, the depth correction unit 111b uses the depth image corresponding to the low reflection area. Figure 7 The depth correction unit 111 b may correct the depth of the correction target area by using the image of the area within the BW image shown in (a). In addition, the depth correction unit 111 b may perform adjustments such as adding an offset to the overall brightness of the depth image.

[0129] As a result, in the depth acquisition device 1 of this embodiment, the corrected depth image can be made close to Figure 7 (e) shows the depth image of the correct solution.

[0130] [Processing Flow]

[0131] Figure 8 This is a flowchart showing the overall processing operation of the depth acquisition device 1 in this embodiment.

[0132] (Step S110)

[0133] First, the BW camera 103 generates a BW image by performing imaging using visible light, and then the BW acquisition unit 115 acquires the BW image via the memory 200 .

[0134] (Step S120)

[0135] Next, the IR camera 102 measures the intensity of the infrared light by taking an image based on the infrared light. Then, the IR acquisition unit 114 acquires the intensity of the infrared light via the memory 200. Here, the intensity of the infrared light acquired is the intensity of the infrared light received by each of the plurality of first pixels 21 of the solid-state imaging element 20. Therefore, the IR acquisition unit 114 acquires the IR image formed based on the intensity of the infrared light by acquiring the intensity of the infrared light.

[0136] Specifically, the intensity of infrared light obtained from the memory 200 includes at least three intensities measured by exposing the solid-state imaging element 20 at least three different timings when the infrared light irradiated from the light source 101 and reflected by the subject is received by the solid-state imaging element 20. For example, the intensity of infrared light includes Figure 3 As shown, three intensities are measured during the first exposure period, the second exposure period, and the third exposure period. For example, each pixel value included in one frame of IR image is represented as a cumulative value of the infrared light intensity measured by the at least three timed exposures.

[0137] (Step S130)

[0138] Next, the low-reflection region detection unit 112 detects a low-reflection region in the IR image based on the BW image acquired in step S110 and the IR image acquired in step S120 .

[0139] (Step S140)

[0140] Next, the depth calculation unit 111a calculates the depth based on the intensity of the infrared light obtained in step S120. That is, the depth calculation unit 111a generates a depth image by calculating the distance to the subject as the depth for each of the multiple first pixels 21 contained in the solid-state imaging element 20 based on the intensity of the infrared light received at the first pixel 21. Specifically, the depth calculation unit 111a generates a depth image by calculating the depth for each of the multiple first pixels 21 contained in the solid-state imaging element 20 based on the above-mentioned at least three intensities measured at the first pixel 21. For example, the depth calculation unit 111a calculates the depth using the above-mentioned (Formula 1) according to the ToF ranging method.

[0141] (Step S150)

[0142] Next, the depth correction unit 111 b corrects the depth of the area in the depth image corresponding to the low-reflection area, that is, the depth of the correction target area.

[0143] Fig. 9 : is a flowchart showing an example of the processing operation of the low reflection area detection unit 112. Fig. 9 Shown in detail Figure 8 The process of step S130.

[0144] (Step S131)

[0145] First, the low-reflection area detection unit 112 detects the area where the object is reflected in the BW image as the object area. If multiple objects are reflected in the BW image, the low-reflection area detection unit 112 detects the area where the object is reflected as the object area for each of the multiple objects. For example, the low-reflection area detection unit 112 detects the object area by processing such as pattern matching or image recognition. In addition, the low-reflection area detection unit 112 can detect an area larger than a predetermined size as the object area.

[0146] (Step S132)

[0147] Next, the low-reflection region detection unit 112 determines whether the brightness of the determination target region, which is a region in the IR image corresponding to the object region, is less than or equal to a first threshold. The brightness of the determination target region may be, for example, the average brightness of each pixel included in the determination target region.

[0148] (Step S133)

[0149] When the low-reflection area detection unit 112 determines that the brightness of the determination target area is equal to or lower than the first threshold value (Yes in step S132 ), it determines that the determination target area is a low-reflection area.

[0150] (Step S134)

[0151] On the other hand, when the low-reflection area detection unit 112 determines that the brightness of the determination target area is greater than the first threshold value (No in step S132 ), it determines that the determination target area is a non-low-reflection area.

[0152] In this way, the low-reflection area detection unit 112 in this embodiment detects the area where the object is reflected in the BW image as the object area. Then, the low-reflection area detection unit 112 determines whether the brightness of the area in the IR image corresponding to the object area, that is, the brightness of the determination target area, is below the first threshold, and detects the determination target area with brightness determined to be below the first threshold as the low-reflection area.

[0153] Thus, in the present embodiment, the area where the object is projected is detected as the object area from the BW image. The object projected in the object area detected in this way is not a distant background but is located nearby, and there is a high possibility that the infrared light from the light source 101 irradiates the object. Furthermore, if the brightness in the area within the IR image corresponding to the object area is low, there is a high possibility that the object is a low-reflection object. Therefore, in the present embodiment, since the area within the infrared light image corresponding to the object area and having a brightness below the first threshold is detected as a low-reflection area, the low-reflection area can be appropriately detected.

[0154] Fig.10 2 is a flowchart showing another example of the processing operation of the low-reflection area detection unit 112. Fig.10 Shown in detail Figure 8 The process of step S130.

[0155] Fig.10 The flowchart shown contains Fig. 9 The steps of the flowchart shown further include step S135.

[0156] (Step S135)

[0157] If the low-reflection area detection unit 112 determines that the brightness of the determination target area is less than the first threshold value (step S132: Yes), it further determines whether the intensity of the background noise of infrared light in the determination target area is less than the second threshold value. The intensity of the background noise is hereinafter also referred to as noise intensity.

[0158] Specifically, the intensity of infrared light acquired by the IR acquisition unit 114 includes at least three intensities measured by exposing the solid-state imaging element 20 at at least three different timings as described above. Figure 3 As shown in FIG. 1 , the three intensities measured during the first exposure period, the second exposure period, and the third exposure period are shown in FIG. 1 . For example, the noise intensity is the smallest intensity among the at least three intensities. The low reflection area detection unit 112 is, for example, Figure 3 In the example shown, the intensity of infrared light measured by exposure in the third exposure period is treated as noise intensity. That is, the low-reflection area detection unit 112 determines whether the intensity of infrared light measured by exposure in the third exposure period is less than the second threshold value.

[0159] If the low-reflection area detection unit 112 determines that the noise intensity is less than the second threshold value (step S135 "Yes"), the determination target area of ​​the noise intensity is determined as a low-reflection area (step S133). On the other hand, if the low-reflection area detection unit 112 determines that the noise intensity is greater than the second threshold value (step S135 "No"), the determination target area of ​​the noise intensity is determined as a non-low-reflection area (step S134).

[0160] In this way, the low-reflection area detection unit 112 in the present embodiment further determines whether the noise intensity contained in the intensity of the infrared light in the determination target area among the intensities of the infrared light acquired by the IR acquisition unit 114 is below the second threshold. Then, the low-reflection area detection unit 112 detects the determination target area with the noise intensity determined to be below the second threshold as a low-reflection area.

[0161] For example, there is a case where the subject is irradiated not only with infrared light from the light source 101 but also with infrared light contained in external light such as sunlight. For this reason, the intensity of the infrared light contained in the external light is treated as noise intensity, and an infrared light image is formed based on the intensity of the infrared light with the noise intensity removed. However, if the noise intensity is large, such as during bright daytime, the brightness of the infrared light image becomes low in order to remove the noise intensity. Therefore, if the noise intensity is large, as described above, there is a possibility that even if the brightness of the area within the infrared light image corresponding to the object area of ​​the BW image is low, the area is not a low-reflection area. Therefore, in the present embodiment, since an area with a small noise intensity determined to be below the second threshold is detected as a low-reflection area, the low-reflection area can be detected more appropriately.

[0162] Fig.11 An example of correction of the depth image performed by the depth correction unit 111 b is shown.

[0163] The depth correction unit 111b corrects Fig.11The depth of the correction target area in the depth image shown in (c) is generated by Fig.11 (e) shows the corrected depth image.

[0164] Specifically, Fig.11 The depth image shown in (c) is formed based on Fig.11 The depth correction unit 111b determines the category of the object reflected in the area based on the image of the area within the BW image corresponding to the low reflection area. For example, the depth correction unit 111b determines the category of the object reflected in the area through processing such as pattern matching or image recognition. In addition, a learning model such as a neural network can be used to determine the category of the object. Fig.11 In the example shown, the depth correction unit 111b determines the front of a normal car as the type of the object. Furthermore, the depth correction unit 111b determines the size of the object reflected in the area. The size of the object may be the horizontal width or vertical width (i.e., height) of the image of the object, or the area of ​​the image of the object, or the length of the diagonal of the rectangular frame surrounding the image of the object.

[0165] Then, the depth correction portion 111b is as follows Fig.11 As shown in (d), for each category of the object, a database representing the correspondence between the size and distance of the category is referenced. For example, the database represents the correspondence between the size and distance of the object when the category of the object is the front of an ordinary car. Specifically, the database establishes a corresponding distance "d1" for the size "s1" and a corresponding distance "d2" for the size "s2". The depth correction unit 111b refers to such a database and estimates the distance corresponding to the category and size of the determined object as the distance to the object, that is, the depth. For example, if the determined size is "s2", the depth correction unit 111b estimates the distance "d2" corresponding to the size "s2" as the depth of the correction target area. Then, the depth correction unit 111b estimates the distance "d2" corresponding to the size "s2" as the depth of the correction target area. Fig.11 The depth of the correction target area in the depth image shown in (c) of FIG. 1 is replaced with the estimated depth to correct the depth of the correction target area. Fig.11 The depth image shown in (c) is transformed into Fig.11 (e) shows the corrected depth image.

[0166] in addition, Fig.11The database shown in (d) may be stored in the depth acquisition device 1 or in a server or the like outside the depth acquisition device 1. When the database is stored in the server, the depth correction unit 111b accesses the server via a communication network, for example, and refers to the database.

[0167] in addition, Fig.11 The database shown in (d) shows the correspondence between the type and size of the object and the distance, but the type of the object may also include the image, that is, the direction of the object as seen from the BW camera 103 (for example, the front, side or back, etc.). In addition, when the lens of the BW camera 103 is a lens of an optical system with a wide viewing angle such as a fisheye lens, in the database, not only the type and size of the object but also the position of the area where the object is reflected in the BW image may be associated with the above distance. In this case, the depth correction unit 111b further determines the position of the area where the object is reflected from the BW image, and in the database, the distance associated with the determined type, size and position of the object is estimated as the depth of the correction target area. In addition, the type of the object may be not only a vehicle but also a license plate contained in the vehicle.

[0168] Fig.12 11 is a flowchart showing the correction processing operation of the depth correction unit 111b. Fig.12 Detailed representation Figure 8 The process of step S150.

[0169] (Step S151)

[0170] The depth correction unit 111 b specifies the type of an object reflected in the low-reflection area based on the image of the area in the BW image corresponding to the area.

[0171] (Step S152)

[0172] Next, the depth correction unit 111 b determines the size of the object reflected in the area.

[0173] (Step S153)

[0174] Next, the depth correction unit 111 b acquires a distance associated with the type determined in step S151 and the size determined in step S152 .

[0175] (Step S154)

[0176] Then, the depth correction unit 111b replaces the depth of the correction target region in the depth image with the distance acquired in step S154. Thus, the depth of the correction target region is corrected.

[0177] In this way, the depth correction unit 111b in this embodiment determines the type of the object reflected in the area based on the image of the area in the BW image corresponding to the low-reflection area. Then, the depth correction unit 111b estimates the depth of the area based on the determined type of the object and the size of the object reflected in the area, thereby correcting the depth of the correction target area.

[0178] Therefore, if a database such as the above is used, the appropriate depth can be estimated based on the type and size of the object determined from the image of the area in the BW image corresponding to the low-reflection area. As a result, the depth of the correction target area can be corrected to the correct depth. In addition, a database is used in the above example, but any means can be used as long as it can show the corresponding relationship between the type and size of the object and the depth. For example, a function can be used instead of the database, and a learning model such as a neural network can also be used.

[0179] Fig.13 Another example of the correction of the depth image by the depth correction unit 111 b is shown.

[0180] When the depth correction unit 111b corrects the depth of the correction target area in the depth image, it may use the depth of the area below the correction target area. Fig.13 In this case, the depth correction unit 111b uses the depth of the front area located below the correction target area. That is, the depth correction unit 111b replaces the depth of the correction target area with the depth of the front area. In other words, the depth correction unit 111b corrects the depth of the correction target area to the depth of the area located below the correction target area.

[0181] In such Fig.13 In the case where the low-reflective object is a vehicle as in the example shown, the road surface is in front of the vehicle, and the depth of the road surface is similar to the depth of the vehicle. Therefore, in this embodiment, since the depth of the correction target area is corrected to the depth of the area below the correction target area, the depth of the correction target area can be corrected to the correct depth.

[0182] (Variation)

[0183] In the above-described embodiment, a low-reflection area is detected to correct the depth. However, the depth may be corrected using a learning model without detecting the low-reflection area.

[0184] Fig.14 is a block diagram showing an example of the functional structure of the depth acquisition device 1 in this modification. Fig.14 Among the components shown, Figure 4The same components as those shown in the figure are marked with Figure 4 The same reference numerals are used for the constituent elements shown, and detailed description thereof will be omitted.

[0185] In this modification, the depth acquisition device 1 replaces Figure 4 The processor 110 a shown in the figure includes a processor 110 b and further includes a learning model 104 .

[0186] The learning model 104 is, for example, a neural network and is formed by deep learning. For example, a BW image, an IR image, and a depth image are used as input data to the learning model 104. The learning model 104 has been learned so that a corrected depth image is outputted for a combination of these input data.

[0187] Processor 110b does not have Figure 4 The low reflection area detection unit 112 shown replaces Figure 4 The depth correction portion 111b shown in the figure is provided with a depth correction portion 111c.

[0188] The depth correction unit 111c inputs the above-mentioned input data to the above-mentioned learning model 104. As a result, the depth correction unit 111c obtains the corrected depth image from the learning model 104 as output data for the input data. Furthermore, the depth correction unit 111c outputs the corrected depth image. For example, the depth correction unit 111c causes the depth image to be displayed on the display by outputting the corrected depth image to the display. Alternatively, the depth correction unit 111c can cause the control unit such as an ECU mounted on the vehicle to perform driving control of the vehicle based on the corrected depth image by outputting the corrected depth image to the control unit.

[0189] Right now, Fig.14The depth acquisition device 1 shown includes a memory 200 and a processor 110b. The processor 110b acquires the intensity of infrared light irradiated from the light source 101 and reflected by the subject, and the intensity of the infrared light is measured by imaging the plurality of first pixels 21 included in the solid-state imaging element 20 respectively receiving the infrared light, and is stored in the memory 200. In addition, the processor 110b generates a depth image by calculating the distance to the subject as the depth for each of the plurality of first pixels 21 included in the solid-state imaging element 20 based on the intensity of the infrared light received at the first pixel 21. Furthermore, the processor 110b acquires a BW image, which is generated by imaging substantially the same scene as the IR image formed based on the intensity of the infrared light received by each of the plurality of first pixels 21 included in the solid-state imaging element 20, based on visible light, and at substantially the same viewpoint and time as the IR image, and is stored in the memory 200. Then, the processor 110 b corrects the depth of the depth image by inputting the depth image, the IR image, and the BW image into the learning model 104 , and outputs the corrected depth image.

[0190] Therefore, if the learning model 104 is pre-learned so as to output a corrected depth image for the input of the depth image, IR image, and BW image, the depth of the depth image can be appropriately corrected without detecting the low-reflection area.

[0191] As described above, in the depth acquisition device 1 in the present embodiment and its modified example, even if the subject includes a low-reflection object, the distance to the subject, that is, the depth, can be accurately acquired.

[0192] In addition, in each of the above embodiments, each component can be formed by dedicated hardware, or can be implemented by executing a software program suitable for each component. Each component can be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software that implements the depth acquisition device of the above embodiments and modified examples causes a computer to execute Figure 8 to Figure 10 as well as Fig.12 The steps contained in any of the flowcharts.

[0193] The depth acquisition device involved in one or more schemes is described above based on the embodiment and its variants, but the present disclosure is not limited to the embodiment and its variants. As long as it does not deviate from the main purpose of the present disclosure, the scheme obtained by implementing various modifications thought of by those skilled in the art to the embodiment and its variants, and the form constructed by combining the constituent elements in the embodiment and its variants can also be included in the scope of the present disclosure.

[0194] For example, the depth calculation unit 111a calculates the depth based on three infrared light intensities measured at three different exposure timings. However, the number of exposure timings and infrared light intensities is not limited to three, and may be four or more.

[0195] In addition, in the modified example of the above embodiment, the depth acquisition device 1 includes the learning model 104, but it may not include the learning model 104. In this case, the depth acquisition device 1 inputs the above-mentioned input data to the learning model 104 via a communication network, for example, and acquires the output data from the learning model 104, that is, the corrected depth image, via the communication network.

[0196] In addition, in the present disclosure, all or part of a component, device, or Figure 1 , Figure 4 ,as well as Fig.14 All or part of the functional blocks of the block diagram shown can be executed by a semiconductor device, one or more electronic circuits including a semiconductor integrated circuit (IC) or LSI (large-scale integration). LSI or IC can be integrated on a single chip or can be composed of a combination of multiple chips. For example, functional blocks other than storage elements can be integrated on a single chip. Here, it is called LSI or IC, but the name changes depending on the degree of integration, and it may be called system LSI, VLSI (very large scale integration) or ULSI (ultra large scale integration). For the same purpose, a field programmable gate array (FPGA) programmed after the manufacture of LSI, or a reconfigurable logic device that can reconfigure the connection relationship within the LSI or set the circuit division within the LSI can also be used.

[0197] Furthermore, all or part of the functions or operations of a component, device or part of a device can be executed by software processing. In this case, the software is recorded on one or more non-temporary recording media such as ROM, optical disk, hard disk drive, etc. When the software is executed by a processor, the software enables the processor and peripheral devices to perform specific functions in the software. The system or device may have one or more non-temporary recording media recording the software, a processor, and necessary hardware devices such as interfaces.

[0198] Industrial Applicability

[0199] The present disclosure can be applied to a depth acquisition device that acquires the distance to a subject, that is, the depth, and can be used as, for example, an in-vehicle device.

[0200] Description of Reference Numerals

[0201] 1 Depth acquisition device

[0202] 10.101 Light Source

[0203] 20 Solid-state imaging element

[0204] 21 1st pixel (IR)

[0205] 22 2nd pixel (BW)

[0206] 30 Processing circuit

[0207] 50 Diffuser Plate

[0208] 60 shots

[0209] 70 Bandpass filter

[0210] 102 IR Camera

[0211] 103 BW Camera

[0212] 104 Learning Model

[0213] 110a, 110b processors

[0214] 111a Depth calculation unit

[0215] 111b, 111c Depth correction section

[0216] 112 Low reflection area detection unit

[0217] 113 Lighting control unit

[0218] 114 IR Acquisition Department

[0219] 115 BW Acquisition Department

[0220] 200 Memory.

Claims

1. An information processing method, The first image is acquired by capturing light emitted from a light source and reflected by a subject. The second image is acquired by photographing the subject using visible light. A low-reflection area is detected based on the first image and the second image.

2. The information processing method according to claim 1, wherein: The first image is an infrared image captured by an infrared camera. The second image is a visible light image captured by a visible light camera.

3. The information processing method according to claim 2, wherein: The low-reflection area is detected based on an area within the infrared light image corresponding to an area within the visible light image.

4. The information processing method according to claim 2 or 3, wherein: The low-reflection area is detected based on the brightness of an area in the infrared light image corresponding to an area in the visible light image.

5. The information processing method according to claim 4, wherein: It is detected that the brightness of an area in the infrared light image corresponding to the area in the visible light image is below a threshold value.

6. The information processing method according to claim 5, wherein: When the brightness of a region in the infrared light image corresponding to a region in the visible light image is below a threshold value, the region in the infrared light image is detected as the low-reflection region.

7. The information processing method according to claim 2, wherein: The low-reflection area is detected based on the infrared light image and the visible light image generated by capturing the same scene as the infrared light image.

8. The information processing method according to claim 2, wherein: The low-reflection area is detected based on the infrared light image and the visible light image generated by imaging from the same viewpoint as the infrared light image.

9. The information processing method according to claim 2, wherein: The low-reflection area is detected based on the infrared light image and the visible light image generated by imaging at the same time as the infrared light image.

10. The information processing method according to claim 2, wherein: The infrared camera uses ToF, or time of flight.

11. The information processing method according to claim 2, wherein: The low-reflection area is detected based on the brightness of the infrared light image and the object recognition result of the visible light image.

12. The information processing method according to claim 11, wherein: When the brightness of the area in the infrared light image is below a threshold value and it is determined based on the object recognition result that an object is located nearby, the area is determined to be the low-reflection area.

13. The information processing method according to claim 2, wherein: Based on the brightness of the area in the infrared light image and the object recognition result of the visible light image, it is determined whether the area is the low-reflection area or the area reflecting the object located far away.

14. The information processing method according to claim 2, wherein: Based on the brightness of the area in the infrared light image and the object recognition result of the visible light image, it is determined whether the area is the low-reflection area or the area reflecting the distant background.

15. An information processing device comprising: Memory; and processor, the processor, acquiring a first image obtained by receiving light emitted from a light source and reflected by a subject; acquiring a second image obtained by photographing the subject using visible light, A low-reflection area is detected based on the first image and the second image.

16. A program product, comprising a program, the program causing a computer to execute: acquiring a first image obtained by receiving light emitted from a light source and reflected by a subject; acquiring a second image obtained by photographing the subject using visible light, A low-reflection area is detected based on the first image and the second image.

Images