Imaging system and subject depth estimation method

Abstract

To improve practicality in subject depth estimation by coded imaging.SOLUTION: An imaging system includes an imaging device having an optical system and an imaging element, and encoding and imaging an object using the optical system and the imaging element, a temperature specification unit specifying an environmental temperature of the imaging device, a function specification unit specifying a point spread function to be used for decoding the captured image on the basis of the specified environmental temperature, and a depth estimation unit decoding the image captured by the imaging device using the specified point spread function, and estimating the depth from the optical system at each position of the object.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an imaging system and a subject depth estimation method. [Background technology]

[0002] In the field of coded imaging, a technique called DFD (Depth From Defocus) is known. DFD is a technique that estimates the distance from the optical system of the imaging device to a subject, i.e., the depth or perspective of the subject, based on the degree of edge blurring in an image obtained by imaging.

[0003] The DFD technique is described in, for example, Non-Patent Document 1. In the DFD technique, coded imaging is performed in which a mask called a coded aperture is placed in the light entrance region of an optical system to capture an image of a subject. Next, a decoding process based on a point spread function specific to the mask is performed on the captured image obtained by the coded imaging, and the depth of the subject is estimated. The point spread function is generally called a PSF (Point Spread Function), and is also called a blur function, blur spread function, point image distribution function, etc. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] "Coded Aperture Pairs for Depth from Defocus and Defocus Deblurring" C. Zhou, S. Lin and SK Nayar, International Journal of Computer Vision, Vol. 93, No. 1, pp. 53, May. 2011. Summary of the Invention [Problem to be solved by the invention]

[0005] DFD technology is still in its infancy, and there is still much room for improvement in terms of practicality. Due to the above circumstances, there is a demand for more practical DFD technology. [Means for solving the problem]

[0006] Among the inventions disclosed in this application, representative ones will be outlined as follows.

[0007] A representative embodiment of the present invention is an imaging system including an imaging device having an optical system and an imaging element, and encoding and imaging an object using the optical system and the imaging element; a temperature identification unit that identifies an environmental temperature of the imaging device; a function identification unit that identifies a point spread function to be used in decoding the captured image based on the identified environmental temperature; and a depth estimation unit that decodes the image captured by the imaging device using the identified point spread function and estimates a depth from the optical system at each position of the object. It is.

[0008] Another representative embodiment of the present invention is a subject depth estimation method that encodes and captures an image of a subject using an imaging device having an optical system and an image sensor, identifies an environmental temperature of the imaging device, identifies a point spread function to be used for decoding the captured image based on the identified environmental temperature, and decodes the image captured by the imaging device using the identified point spread function, thereby estimating a depth from the optical system at each position of the subject. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of installation of an imaging system according to a first embodiment. [Diagram 2] FIG. 1 illustrates an example of a configuration of an imaging system. [Diagram 3] FIG. 2 is a diagram illustrating an example of the configuration of an arithmetic control processing device. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional block of the arithmetic control processing device. [Diagram 5]4 is a flowchart showing an example of an operation flow in the imaging system according to the first embodiment. [Figure 6] FIG. 11 is a process flow diagram of another method for correcting deviations in subject depth estimates caused by changes in environmental temperature. [Figure 7] FIG. 4 is a functional block diagram of an arithmetic control processing device according to a modified example of the first embodiment. [Figure 8] FIG. 13 is a diagram showing a table showing the correspondence relationship between the environmental temperature and the point spread function applied in the case of that environmental temperature. [Figure 9] FIG. 11 is a diagram illustrating an example of a relationship between an estimated depth value of a subject and an environmental temperature. [Figure 10] 5 is a diagram illustrating an example of the relationship between the distance between the optical system and the image sensor and the environmental temperature. FIG. [Figure 11] 11A and 11B are diagrams for explaining how an imaging system changes due to a change in environmental temperature, and thus a subject depth estimation value changes. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] <Details of the assignment> Before describing each embodiment of the present invention, the basic content of DFD technology and problems discovered by the present inventors will be described.

[0011] The state of blur of a subject in a captured image generally depends on a point spread function determined by the optical system of an imaging device, the shape of a light incidence region of the optical system, etc. When a mask that partially blocks light is installed in the light incidence region of the optical system, the point spread function is determined for each mask. Imaging a subject with an imaging device equipped with a mask is called coded imaging. When coding imaging of a subject, an image that is blurred based on a point spread function specific to the mask used is acquired as a captured image.

[0012] When this blurred captured image is subjected to a decoding process that performs deconvolution based on a point spread function specific to the mask used, a decoded image with reduced blur and depth information of objects corresponding to each position of the subject contained in the decoded image are obtained.

[0013] On the other hand, the inventors have discovered that when, as described above, a decoding process is performed on an image obtained by encoding and capturing an image of a subject using an imaging device, and the depth at each position of the subject is estimated, the following problem occurs.

[0014] Fig. 9 is a diagram showing an example of the relationship between the subject depth estimate and the environmental temperature. As shown in Fig. 9, the inventors have found through research and development of DFD technology that the subject depth estimate dj varies depending on the environmental temperature T of the imaging device. For example, as shown in Fig. 9, when the actual distance from the optical system of the subject is Dr1 and the environmental temperature T is T0, the subject depth estimate dj is obtained as dj0. On the other hand, if the environmental temperature T changes to T1, the subject depth estimate dj changes to dj1, and if the environmental temperature T changes to T2, the subject depth estimate dj changes to dj2.

[0015] The phenomenon in which the subject depth estimate changes depending on the environmental temperature as described above is believed to be due to the fact that the positional relationship between the optical system and the image sensor of the imaging device changes depending on the environmental temperature T, and this change is reflected in the captured image. This change in the positional relationship between the optical system and the image sensor is believed to be mainly due to a change in the distance between the optical system and the image sensor.

[0016] Fig. 10 is a diagram showing an example of the relationship between the distance dw between the optical system and the imaging element and the environmental temperature T. It is considered that the distance between the optical system and the imaging element increases as the environmental temperature T increases, for example, as shown in Fig. 10. It is considered that such a change in the distance between the optical system and the imaging element due to the environmental temperature may occur, for example, due to deformation caused by the temperature of a member supporting at least one of the optical system and the imaging element, as well as based on the temperature characteristics of a device that controls the positional relationship between the optical system and the imaging element.

[0017] FIG. 11 is a diagram for explaining how the imaging system changes due to a change in the environmental temperature, and the subject depth estimation value changes accordingly.

[0018] 11, for example, when the environmental temperature is 25° C. and the in-focus position is 300 mm from the optical system, light L1 from a point light source 91 at a position ze1 300 mm away from the optical system 22 forms an image on the light receiving surface at position z1 of the image sensor 23 when the environmental temperature is 25° C. This state is called the first state.

[0019] Also, light L2 from a point light source 91 located at a position ze2 400 mm away from the optical system 22 forms an image in front of the image sensor 23, and becomes a blurred image that spreads across the entire light receiving surface of the image sensor 23 located at position z1. This state is called the second state.

[0020] On the other hand, when the environmental temperature becomes 30° C. and the relative position of the image sensor 23 from the optical system 22 moves to position z2, the light L1 from the point light source 91 at position ze1 becomes a blurred image that spreads across the entire light receiving surface of the image sensor 23, which is at position z2 when the environmental temperature is 30° C. This state is called the third state.

[0021] From the viewpoint of the imaging element 23, the third state when the environmental temperature is 30° C. is considered to be substantially the same as the second state when the environmental temperature is 25° C. Therefore, when the environmental temperature is 25° C. is taken as a reference, the image of the point light source 91 at the position ze1 that is 300 mm away from the optical system 22 is regarded as the image of the point light source 91 at the position ze2 that is 400 mm away from the optical system 22, and the estimated depth value of the point light source 91 is erroneously measured as being 400 mm away from the optical system 22. In other words, when the environmental temperature changes from 25° C. to 30° C., the estimated depth value of 300 mm from the optical system 22 shifts to the estimated depth value of 400 mm from the optical system 22.

[0022] In view of the above circumstances, there is a demand for a DFD technique that can estimate the depth of a subject with higher accuracy regardless of the environmental temperature of the imaging device.

[0023] In view of the above circumstances, the inventors have invented the present invention as a result of intensive research. Hereinafter, an embodiment of the present invention will be described. Note that the embodiment described below is an example for carrying out the present invention, and does not limit the technical scope of the present invention. In the following embodiments, components having the same functions are given the same reference numerals, and repeated explanations thereof will be omitted unless particularly necessary.

[0024] (Embodiment 1) An imaging system according to the first embodiment of the present application will be described. The imaging system according to the first embodiment of the present application is an imaging system including an imaging device having an optical system and an imaging element, and encoding and imaging an object using the optical system and the imaging element, a temperature specifying unit specifying an environmental temperature of the imaging device, a function specifying unit specifying a point spread function to be used for decoding the captured image based on the specified environmental temperature, and a depth estimating unit decoding the image captured by the imaging device using the specified point spread function and estimating the depth from the optical system at each position of the object. Details of this imaging system are as follows.

[0025] 1 is a diagram illustrating an example of installation of an imaging system 1 according to embodiment 1. The z direction in the figure is the traveling direction of an automobile 100 on the forward side.

[0026] 1, an imaging system 1 is installed in a vehicle, that is, an automobile 100. The imaging system 1 is configured to capture an image of a subject 90 in the traveling direction of the automobile 100, that is, in front of the automobile 100, in an encoded form.

[0027] Example of the configuration of an imaging system according to the first embodiment 2 is a diagram showing an example of the configuration of the imaging system 1. As shown in FIG.

[0028] The imaging device 2 is installed in an automobile 100, which is a vehicle, as part of the imaging system 1. The imaging device 2 has a mask 21, an optical system 22, an imaging element 23, and a temperature sensor 24.

[0029] The mask 21 has a specific geometrical aperture pattern and functions as a filter for light that enters the optical system 22 from the subject 90 and reaches the image sensor 23. The mask 21 may be placed between the optical system 22 and the image sensor 23. Furthermore, when the optical system 22 is a compound lens, the mask 21 may be placed between the lenses. The mask 21 is also called a coded aperture.

[0030] The optical system 22 collects light incident from the subject 90 and forms an image on the light receiving surface 23a of the image sensor 23. The optical system 22 is, for example, a lens. The lens may be a single lens or a compound lens, and may be a fixed focal length lens or a zoom lens.

[0031] The imaging element 23 has a light receiving surface 23a, which is composed of a plurality of photoelectric conversion elements arranged two-dimensionally. The imaging element 23 converts the light L received by the light receiving surface 23a into an electrical signal according to its intensity, and outputs image data based on the electrical signal to the arithmetic control processing device 3. The imaging element 23 may output the photoelectrically converted electrical signal to the arithmetic control processing device 3, which may obtain image data based on the electrical signal. The imaging element 23 is also called an image sensor.

[0032] The temperature sensor 24 is disposed near the optical system 22 and the imaging element 23. The temperature sensor 24 is installed, for example, in a housing (not shown) that contains the optical system 22 and the imaging element 23. The temperature sensor 24 outputs a signal Td according to the environmental temperature of the imaging device 2. The environmental temperature of the imaging device 2 can be identified based on this output signal Td. The temperature sensor 24 is composed of, for example, a thermocouple, a resistance temperature detector, a thermistor, or the like.

[0033] The imaging device 2 has a point spread function specific to the imaging device 2. The point spread function is a function that determines how a point image on a subject appears blurred in a captured image, and is determined by a combination of the mask 21, the optical system 22, and the imaging element 23. As described above, the point spread function includes an element that depends on the positional relationship between the optical system 22 and the imaging element 23. In addition, the positional relationship between the optical system 22 and the imaging element 23 may change depending on the environmental temperature. For example, the positional relationship may change due to deformation due to temperature change of a member that supports the optical system 22 and the imaging element 23, temperature characteristics of a device that controls the positional relationship between the optical system 22 and the imaging element 23, etc. In this embodiment, it is assumed that the positional relationship between the optical system 22 and the imaging element 23 changes due to a change in the environmental temperature of the imaging device 2, and the point spread function also changes.

[0034] The arithmetic control processing device 3 identifies the environmental temperature of the imaging device 2 based on the output signal of the temperature sensor 24. The arithmetic control processing device 3 identifies a point spread function for the encoded imaging of the imaging device 2 based on the identified environmental temperature. The arithmetic control processing device 3 performs a decoding process on the captured image P1 obtained by the encoded imaging using the identified point spread function. By performing this decoding process, the arithmetic control processing device 3 obtains a decoded image in which the blur of the subject 90 has been improved, and a depth estimate value from the optical system at each position of the subject 90. The arithmetic control processing device 3 generates a depth map DM by associating the decoded image with the depth estimate value, and outputs the depth map DM to an external device 4 or the like.

[0035] The external device 4 executes various processes using the depth map DM received from the arithmetic control processing device 3. The external device 4 is, for example, an automobile driving assistance system, and uses the received depth map DM to control the steering, accelerator, brakes, etc., and to detect and report dangerous conditions.

[0036] 3 is a diagram showing an example of the configuration of the arithmetic control processing device 3. In this embodiment, the arithmetic control processing device 3 is a computer, and has a processor 301, a memory 302, an interface 303, and a communication bus 304. The processor 301, the memory 302, and the interface 303 are communicatively connected to each other via the communication bus 304. At least a part of the arithmetic control processing device 3 may be configured by semiconductor devices such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0037] The processor 301 is, for example, a central processing unit (CPU), a micro processing unit (MPU), a micro controller unit (MCU), a graphics processing unit (GPU), etc. The processor 301 executes various types of calculations and various types of processing.

[0038] The memory 302 is, for example, a semiconductor storage device, a hard disk drive (HDD), a solid state drive (SSD), etc. The memory 302 stores a program P. The processor 301 reads out and executes the program P stored in the memory 302, causing the computer to function as various functional blocks.

[0039] The interface 303 receives an electrical signal or image data representing a captured image from the image sensor 23, and outputs depth information of the subject 90, such as a depth map DM, to the outside.

[0040] <Functional blocks of the arithmetic control processing device> Fig. 4 is a diagram showing an example of functional blocks included in the arithmetic control processing device 3. As described above, the functional blocks are realized by the processor 301 executing the program P. As shown in Fig. 4, the arithmetic control processing device 3 has, as functional blocks, a control unit 31, a temperature specifying unit 32, a PSF specifying unit 33, a depth estimation unit 34, and a depth map generating unit 35.

[0041] The control unit 31 controls each unit so that the captured image P1 is received by encoded imaging, the captured image P1 is decoded to obtain a decoded image M1 and a subject depth estimate dr, the environmental temperature T is determined, the depth estimate dj is corrected, and a depth map DM is generated.

[0042] The temperature specifying unit 32 specifies the environmental temperature T, which is one of the parameters of the imaging device 2. In this embodiment, the environmental temperature T is specified based on an output signal of the temperature sensor 24. Note that the temperature specifying unit 32 may specify the environmental temperature T based on information other than that of the temperature sensor 24. For example, the temperature specifying unit 32 may specify the environmental temperature T based on temperature information received from the external device 4.

[0043] The PSF specifying unit 33 determines and specifies a point spread function to be used for decoding by calculation based on the specified environmental temperature T. For example, the PSF specifying unit 33 determines the distance between the optical system 22 and the image sensor 23 by referring to the relationship between the environmental temperature T and the distance between the optical system 22 and the image sensor 23 based on the specified environmental temperature T. The relationship between the environmental temperature T and the distance between the optical system 22 and the image sensor 23 can be determined in advance by a prior experiment. The PSF specifying unit 33 determines and specifies a point spread function F by a predetermined calculation, i.e., a simulation, based on the determined distance between the optical system 22 and the image sensor 23.

[0044] The method of obtaining the point spread function F by simulation based on the distance between the optical system 22 and the image sensor 23 can be implemented, for example, by using a simulation method that is a publicly known technology. As the simulation method, for example, the method described in the non-patent document "Deep Optics for Monocular Depth Estimation and 3D Object Detection", Julie Chang, Gordon Wetzstein, arXiv:1904.08601v1 [cs.CV] 18 Apr 2019 is used. The method of obtaining the PSF is described, for example, in 3. Differentiable Image Formation Model of this non-patent document.

[0045] In this way, by calculating the point spread function F used for decoding, the point spread function according to the environmental temperature T can be calculated continuously, not discretely, and an appropriate point spread function F can be accurately calculated. As a result, the captured image is properly decoded as intended, and the depth estimate of the subject can be accurately calculated.

[0046] The depth estimation unit 34 receives the captured image P1 obtained by encoding imaging from the imaging element 23 in response to control from the control unit 31. The depth estimation unit 34 also performs a decoding process on the received captured image P1. The decoding process of the captured image P1 is a process by deconvolution based on a point spread function determined by the aperture pattern of the mask 21, the optical system 22, the imaging element 23, etc. The point spread function used in the decoding is the point spread function F specified by the PSF specification unit 33. When the decoding process of the captured image P1 is performed, a decoded image M1, which is an image of the subject 90 with reduced blur, and a depth estimate value dr of the subject 90 from the optical system 22 are obtained. Note that in this specification, the depth estimate value dr represents a set of depth estimate values ​​at each position of the subject 90.

[0047] The depth map generating unit 35 generates a depth map DM by mapping each position of the subject 90 included in the decoded image M1 in association with the corrected depth estimate value dr in accordance with control from the control unit 31. The depth map generating unit 35 also outputs the generated depth map DM to the external device 4 or the like.

[0048] <Flow of Operation in the Imaging System According to the First Embodiment> The flow of operations in the imaging system 1 will be described below.

[0049] FIG. 5 is a flow diagram showing an example of the flow of operations in the imaging system 1 according to the first embodiment.

[0050] 5, in step S1, a process of encoding and capturing images is performed. Specifically, the depth estimation unit 34 performs encoded imaging of the subject 90 by reading image data from the imaging element 23 in response to control from the control unit 31, and obtains a captured image.

[0051] In step S2, a process of identifying the environmental temperature is performed. Specifically, the temperature identifying unit 32 identifies the environmental temperature T of the imaging device 2 based on the output signal of the temperature sensor 24 and the like.

[0052] In step S3, a process of specifying a point spread function is performed. Specifically, the PSF specifying unit 33 specifies a point spread function to be used for decoding the captured image obtained by the encoded imaging based on the specified environmental temperature T. The point spread function F specified in step S3 is a function that takes into consideration a change in the positional relationship between the optical system 22 and the image sensor 23 that changes depending on the environmental temperature T. When the captured image is decoded using the point spread function F, the obtained subject depth estimate is an estimate in which the deviation of the estimate due to the change in the positional relationship is cancelled.

[0053] In step S4, the captured image is decoded. Specifically, the depth estimation unit 34 performs a decoding process based on the specified point spread function on the captured image P1 under the control of the control unit 31, and obtains a decoded image M1 of the subject 90 and a depth estimate dr at each position of the subject 90. As described above, the obtained depth estimate dr at each position of the subject 90 is an estimate in which the deviation of the estimate due to the change in the positional relationship between the optical system 22 and the image sensor 23 according to the environmental temperature T is cancelled.

[0054] In step S5, a process of generating a depth map using the obtained depth estimate is performed. Specifically, the depth map generating unit 35, under the control of the control unit 31, generates a depth map DM by associating each position of the decoded image M1 obtained by decoding with the depth estimate value dr at each position of the subject 90.

[0055] In step S6, a process of outputting the depth map is performed. Specifically, the depth map generating unit 35 outputs the generated depth map DM to the external device 4 or the like in response to control from the control unit 31. The external device 4 is, for example, a driving support device for the automobile 100. When the external device 4 is a driving support device, it can detect an obstacle ahead of the automobile 100, the distance to the preceding vehicle, and the like, and use the detection information for automatic braking, cruise control, and the like of the automobile 100.

[0056] In step S7, a process is performed to determine whether or not there is a reason to stop the process. Specifically, the control unit 31 determines whether or not there is a reason to stop the process (hereinafter also referred to as a reason to stop the process). For example, if a command to stop the process is input from the user or the external device 4, or if some error occurs, it is determined that there is a reason to stop the process. If there is no command or error, it is determined that there is no reason to stop the process. If it is determined that there is a reason to stop the process (S7: Yes), the control unit 31 ends the process, and if it is determined that there is no reason to stop the process (S7: No), the control unit 31 returns the process step to S1 to continue the process.

[0057] The environmental temperature is specified in step S2 before the captured image is decoded and may be specified at any time as long as the output signal of the temperature sensor 24 obtained close to the time of encoded imaging is used. The point spread function is specified in step S3 after the environmental temperature is specified and may be specified at any time as long as the captured image is decoded.

[0058] In this way, the imaging system 1 according to the first embodiment captures encoded images of the subject 90 using the imaging device 2. The imaging system 1 also identifies the environmental temperature T of the imaging device 2. The imaging system 1 determines by calculation the point spread function F used for decoding the captured image P1 based on the identified environmental temperature T. The imaging system 1 decodes the captured image P1 obtained by the encoded imaging using the identified point spread function F to obtain a decoded image M1 with reduced blur and depth estimates dr from the optical system to each position of the subject.

[0059] <Depth Estimation Value Correction Method as a Comparison with the First Embodiment> Next, a description will be given of another method for correcting deviations in subject depth estimation values ​​according to changes in environmental temperature, as a comparison with embodiment 1. Note that this method is one of the methods devised by the present inventors, and does not fall under the prior art of the present invention.

[0060] FIG. 6 is a process flow diagram of another method for correcting deviations in subject depth estimates caused by changes in environmental temperature.

[0061] As shown in FIG. 6, for example, in step J1, a point spread function is generated in advance based on the imaging system of an imaging device that performs coding imaging.

[0062] In step J2, the imaging device captures encoded images of the subject, thereby obtaining a captured image.

[0063] In step J3, the captured image is decoded using the point spread function previously generated in step J1, resulting in a depth estimate for each position of the object.

[0064] In step J4, the environmental temperature of the imaging device is determined based on, for example, an output signal from a temperature sensor or the like.

[0065] In step J5, the depth estimate value at each position of the subject, i.e., at each pixel, is corrected based on the identified environmental temperature. This correction is performed using correction data obtained in advance. The correction data is data in which the pre-correction depth estimate value and the post-correction depth estimate value are associated with each environmental temperature. This correction data can be obtained by experiment. That is, in step J5, a process is performed in which the depth estimate value at each position of the subject, i.e., at each pixel, is converted from a pre-correction value to a post-correction value by referring to data representing the pre-correction and post-correction depth estimate values ​​at the identified environmental temperature.

[0066] In step J6, a process is performed to generate a depth map obtained by associating the deblurred image obtained by decoding with the corrected depth estimate value.

[0067] It should be noted that the determination of the environmental temperature in step J4 may be performed at any time before the correction of the depth estimate and based on temperature information acquired close to the time of encoding imaging.

[0068] In the depth estimation correction method for comparison, a captured image is decoded using a predefined point spread function, and the obtained depth estimation value is corrected for each pixel according to the environmental temperature. This method requires correction for each pixel, so it takes a relatively long time to perform the correction and places a heavy load on the processor.

[0069] On the other hand, in the imaging system 1 according to the first embodiment, a point spread function is specified based on the identified environmental temperature of the imaging device, and the captured image is encoded using the point spread function to obtain an estimated depth value of the subject.

[0070] Therefore, compared to a method of correcting the depth estimate to each position of the subject based on the environmental temperature T of the imaging device 2, the imaging system 1 according to the first embodiment does not require correction for each pixel, and can significantly reduce the amount of calculation by the processor 301. As a result, the load on the processor 301 can be reduced, and the time required to calculate the depth estimate at each position of the subject can be shortened.

[0071] In the imaging system 1 according to the first embodiment, the point spread function is determined by calculation. Therefore, the point spread function can be determined as a solution that changes continuously, not as a solution that changes discretely, for the determined environmental temperature, and an accurate point spread function can be determined, making it possible to determine a depth estimate with higher accuracy.

[0072] By performing such processing in the imaging system 1, even if the environmental temperature T of the imaging device 2 changes, the corrected and appropriate depth estimate dr of the subject 90 can be obtained with higher accuracy, thereby realizing a more practical DFD technology.

[0073] (Modification of the first embodiment) A modified example of the imaging system 1 according to the first embodiment will be described. In this modified example, the function of specifying the point spread function in the arithmetic control processing device is different from that of the first embodiment. Specifically, the arithmetic control processing device does not calculate the point spread function used for decoding, but calculates it based on pre-stored data.

[0074] Fig. 7 is a functional block diagram of an arithmetic control processing device 3a according to a modification of embodiment 1. As shown in Fig. 7, the arithmetic control processing device 3a according to this modification is based on the arithmetic control processing device 3 of embodiment 1, and a PSF data storage unit 36 ​​is newly added.

[0075] The PSF data storage unit 36 ​​stores data indicating a correspondence relationship between the environmental temperature and a point spread function (PSF) applied at each of a plurality of environmental temperatures of the imaging device 2. This correspondence relationship can be obtained by a prior experiment or the like.

[0076] When the PSF specifying unit 33 specifies the point spread function F based on the environmental temperature T, the PSF specifying unit 33 specifies the point spread function F using data stored in advance in the PSF data storage unit 36, rather than by calculation.

[0077] FIG. 8 is a diagram showing a table TB indicating a correspondence relationship between an environmental temperature T and a point spread function F applied in the case of the environmental temperature T. As shown in FIG. 8, for example, in the table TB, the environmental temperature T0 is associated with the point spread function F(T0) applied in the case of the environmental temperature T0. Also, the environmental temperature T1 is associated with the point spread function F(T1) applied in the case of the environmental temperature T1. Similarly, the point spread functions F(T0), F(T1), F(T2), . . ., F(Tn), . . ., F(TM) are associated with the environmental temperatures T0, T1, T2, . . ., Tn, . . ., TM. The range of the environmental temperatures T0 to TM in the table TB is, for example, -10°C to +40°C, and the environmental temperatures T0, T1, . . . are set in increments of 0.1°C.

[0078] The PSF specifying unit 33 reads out the point spread function corresponding to the environmental temperature closest to the specified environmental temperature T among the environmental temperatures T0 to TM from the PSF data storage unit 36, and specifies it as the point spread function to be used for decoding. Note that the PSF specifying unit 33 may perform an interpolation process using two point spread functions corresponding to the top two environmental temperatures closest to the specified environmental temperature T among the environmental temperatures T0 to TM, to obtain the point spread function to be used for decoding.

[0079] According to this modification, the point spread function to be used in decoding is not determined by calculation based on the determined environmental temperature, but is determined using pre-stored data, which can reduce the load on the processor 301 and shorten the time required to determine the point spread function.

[0080] In the first embodiment, the point spread function is determined from the environmental temperature of the imaging device. However, the positional relationship between the optical system and the imaging element may be obtained, and the point spread function may be obtained from the positional relationship by simulation.

[0081] Other Embodiments Although the imaging system according to the first embodiment and its modified examples have been described above, a subject depth estimation method according to the processing flow in the imaging system 1 is also one embodiment of the present invention. That is, a subject depth estimation method that encodes and captures an image of a subject using an imaging device having an optical system and an image sensor, identifies an environmental temperature of the imaging device, identifies a point spread function to be used for decoding the captured image based on the identified environmental temperature, and decodes the image captured by the imaging device using the identified point spread function to estimate the depth from the optical system at each position of the subject is one embodiment of the present invention.

[0082] In addition, a program for causing a computer to function as the temperature determination unit 32, the PSF determination unit 33, and the depth estimation unit 34, and a tangible computer-readable storage medium that non-temporarily stores the program are also an embodiment of the present invention.

[0083] Although various embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments, and various modified examples are included. Furthermore, the above-mentioned embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these belong to the scope of the present invention. Furthermore, the numerical values ​​and the like included in the text and figures are merely examples, and the effect of the present invention is not impaired even if different ones are used.

[0084] For example, in the above embodiment, the imaging system 1 is installed in an automobile, but the imaging system 1 may be installed in vehicles other than automobiles, such as railway or monorail trains, motorcycles, bicycles, drones, etc. Even in such an installation example, the imaging system 1 can achieve the same effects as in the above embodiment, and can be used, for example, in driving assistance technology. [Explanation of symbols]

[0085] 1...imaging system, 2...imaging device, 3...arithmetic control processing device, 4...external device, 21...mask, 22...optical system, 23...imaging element, 23a...light receiving surface, 31...control unit, 32...temperature determination unit, 33...PSF determination unit, 34...depth estimation unit, 35...depth map generation unit, 36...PSF data storage unit, 90...subject, 91...point light source, 100...automobile, 301...processor, 302...memory, 303...interface, 304...communication bus, P...program

Claims

1. an imaging device having an optical system and an imaging element, the imaging device imaging an object using the optical system and the imaging element in an encoded form; a temperature determination unit that determines an environmental temperature of the imaging device; a function specifying unit that specifies a point spread function to be used in decoding a captured image based on the specified environmental temperature; a depth estimation unit that decodes an image captured by the imaging device using the specified point spread function and estimates a depth from the optical system at each position of the subject. Imaging system.

2. 2. The imaging system according to claim 1, the function specifying unit determines a point spread function to be used in the decoding by a calculation using the specified environmental temperature as a parameter. Imaging system.

3. 2. The imaging system according to claim 1, a storage unit configured to store a correspondence relationship between an environmental temperature and a point spread function for each of a plurality of environmental temperatures of the imaging device; the function specifying unit specifies a point spread function to be used for the decoding based on the specified environmental temperature and the stored correspondence relationship. Imaging system.

4. 2. The imaging system according to claim 1, The imaging device is installed in a vehicle. Imaging system.

5. 5. The imaging system according to claim 4, The vehicle is an automobile. Imaging system.

6. An image of a subject is captured using an imaging device having an optical system and an imaging element, Identifying an environmental temperature of the imaging device; determining a point spread function to be used in decoding the captured image based on the determined environmental temperature; Decoding an image captured by the imaging device using the specified point spread function to estimate a depth from the optical system at each position of the subject. Subject depth estimation method.

7. The subject depth estimation method according to claim 6, The point spread function used in the decoding is determined and specified by a calculation using the specified environmental temperature as a parameter. Subject depth estimation method.

8. The subject depth estimation method according to claim 6, storing a correspondence relationship between an environmental temperature and a point spread function for each of a plurality of environmental temperatures of the imaging device; a point spread function used in the decoding is identified based on the identified environmental temperature and the stored correspondence relationship; Subject depth estimation method.

9. The subject depth estimation method according to claim 6, The imaging device is installed in a vehicle. Subject depth estimation method.

10. The subject depth estimation method according to claim 9, The vehicle is an automobile. Subject depth estimation method.

Images