Imaging unit and imaging method
The imaging unit and method correct environmental temperature effects on focal length and blur by using a coded aperture and TOF sensor to ensure accurate depth information, addressing displacement issues and enhancing precision.
Patent Information
- Application Number
- JP2023078002
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-10
- Publication Date
- 2024-11-21
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The accuracy of distance measurement by imaging devices using encoded imaging methods is compromised by displacement of the optical system or image sensor due to changes in environmental temperature, affecting the focal length and blur function.
An imaging unit and method that incorporates a coded aperture pattern, a correction sensor to measure light reflection time, a depth information acquisition unit, and a temperature correction mechanism to adjust depth information based on environmental temperature, using a TOF sensor for accurate distance measurement.
Enables high-accuracy depth information acquisition regardless of environmental temperature fluctuations, combining the benefits of coded aperture imaging and TOF sensor measurements to enhance precision.
Smart Images

Figure 2024162469000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an imaging unit and an imaging method. [Background technology]
[0002] Patent Document 1 describes the coded imaging method, which describes that the coded imaging method uses a complex pattern mask as an aperture (coded aperture) to control the generalized blur function (PSF) and its frequency characteristics, thereby making it possible to use a technique called Depth From Defocus (DFD), which estimates the depth of a scene from the blur of an image. [Prior art documents] [Non-patent literature]
[0003] [Non-Patent Document 1] Hajime Nagahara: Coded Imaging, IPSJ Technical Report, Vol. 2010-CVIM-171, No. 14, pp. 1-9, 2010 Summary of the Invention [Problem to be solved by the invention]
[0004] Here, the accuracy of distance measurement by an imaging device using the coded imaging method depends on the distance between an optical system including a lens and an imaging element (image sensor), i.e., the accuracy of the focal length. Therefore, if the optical system or the imaging element is displaced, accurate distance measurement may not be possible. One of the factors that cause displacement of the optical system or the imaging element is a change in environmental temperature.
[0005] The present invention has been made in view of the above circumstances, and an object of the present invention is to improve distance measurement accuracy. [Means for solving the problem]
[0006] In order to solve the above problem, the imaging unit of the present application has an imaging device having an optical system, an aperture section in which a coded aperture pattern that blocks a portion of the light incident on the optical system is formed, and an imaging element that acquires image data based on the incident light that has passed through the optical system, a correction sensor that measures the time it takes for the emitted light to be reflected off the surface of an arbitrary imaging target and returned, a first depth information acquisition section that acquires first depth information for each of a plurality of pixels of the imaging element based on a blur function related to the coded aperture pattern, a second depth information acquisition section that acquires second depth information of the imaging target based on the measurement result of the correction sensor, a memory section that stores a relationship between the first depth information and the second depth information in association with an environmental temperature, a temperature acquisition section that acquires the environmental temperature based on the first depth information of the imaging target acquired by the first depth information acquisition section and the second depth information of the imaging target acquired using the second depth information, and a correction section that corrects the first depth information for each of the plurality of pixels based on the environmental temperature acquired by the temperature acquisition section.
[0007] In addition, in order to solve the above-mentioned problems, the imaging method of the present application includes the steps of acquiring image data based on incident light that has passed through an aperture section in which a coded aperture pattern that partially blocks light incident on an optical system is formed, and acquiring first depth information for each of a plurality of pixels of an imaging element; acquiring second depth information of an arbitrary imaging target based on a measurement result of a correction sensor that measures the time it takes for emitted light to be reflected off the surface of the imaging target and returned; acquiring an environmental temperature based on the first depth information and the second depth information using a memory section that stores the relationship between the first depth information and the second depth information in association with the environmental temperature; and correcting the first depth information for each of the plurality of pixels based on the acquired environmental temperature. [Brief description of the drawings]
[0008] [Figure 1] FIG. 2 is a schematic diagram showing an imaging unit according to the present embodiment. [Diagram 2]FIG. 2 is a functional block diagram showing an example of functions realized by a control unit in the present embodiment. [Diagram 3] 1 is a graph showing an example of a relationship between a distance measurement result by an imaging device and an environmental temperature. [Figure 4] FIG. 4 illustrates an example of a table stored in a storage unit. [Diagram 5] 11A and 11B are diagrams illustrating correction of depth information for each pixel. [Figure 6] FIG. 4 is a flowchart showing a correction process according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] In the present application, the drawings may be schematic in width, thickness, shape, etc. of each part compared to the actual embodiment in order to clarify the description, but these are merely examples and do not limit the interpretation of the present invention. In this specification and each drawing, elements having the same functions as those described in the previous drawings may be given the same reference numerals, and duplicate descriptions may be omitted.
[0010] [Overview of the overall configuration of the imaging unit 100] First, an overview of the overall configuration of an imaging unit 100 according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram showing an imaging unit according to this embodiment. The imaging unit 100 according to this embodiment includes an imaging device 10 and a TOF (Time Of Flight) sensor 20, and a housing H that houses them. The imaging unit 100 also includes a control unit 30 and a storage unit 40.
[0011] 1 shows a schematic diagram of an image captured by the imaging unit 100 showing an imaging target A at a distance DA from the imaging unit 100 and an imaging target B at a distance DB from the imaging unit 100. Note that the distance DA is preferably the distance from the surface of the imaging target A to the center of the lens 11, and the distance DB is preferably the distance from the surface of the imaging target B to the center of the lens 11.
[0012] The imaging device 10 is a camera capable of acquiring the distance in the depth direction of an imaging target by using a coding imaging method for observing blur. The imaging device 10 may be a digital camera or a camera built into a smartphone or the like.
[0013] The imaging device 10 includes an optical system including at least one lens 11, an aperture section 12 in which a coded aperture 12a that limits external light passing through the optical system with an aperture pattern is formed, and an imaging element 13. The imaging device 10 is not limited to the configuration shown in Fig. 1, and may have various configurations that are mounted on a general camera to realize an imaging function. For example, the imaging device 10 may have a shutter or the like that adjusts the amount of exposure of the imaging element 13 to external light.
[0014] The lens 11 may be a lens used in a general camera, and may be configured to have an adjustable focal length. Although a single lens 11 is shown in FIG. 1, the optical system may include a group of multiple lenses.
[0015] The aperture section 12 partially blocks external light incident on the lens 11. A coded aperture 12a is formed in the aperture section 12. In the imaging unit 100, the aperture section 12 in which the coded aperture 12a is formed is used as an aperture (coded aperture) to control a point spread function (hereinafter also simply referred to as PSF) and its frequency characteristics, thereby utilizing a method called Depth from defocus (DFD) in which the depth of a scene is estimated from the blur of an image. The PSF is sometimes also called a blur function. The aperture section 12 is a liquid crystal shutter, and may be configured to be able to change the opening pattern of the coded aperture 12a.
[0016] The imaging element 13 may be a CMOS (complementary metal oxide semiconductor) sensor or a CCD (charge-coupled device) used in general cameras. The image detected by the imaging element 13 may be a color image or a monochrome image.
[0017] The control unit 30 may be an information processing device including at least one processor. The control unit 30 acquires depth information of the captured image by processing image data obtained by the imaging element 13. The control unit 30 may be capable of acquiring depth information of each of the multiple pixels of the imaging element 13 by a restoration process based on a blur function related to the aperture pattern of the coded aperture 12a of the diaphragm unit 12.
[0018] Information for correcting the depth information is stored in the storage unit 40. The information stored in the storage unit 40 will be described later with reference to FIG.
[0019] The TOF sensor 20 is a sensor that detects depth information of an imaging target by measuring the time it takes for emitted light to reflect off the surface of the imaging target and return. The TOF sensor 20 may have a light emitting unit that emits emitted light and a light receiving unit that receives light reflected off the surface of any imaging target and returns. In this embodiment, the TOF sensor 20 is used as an example of a correction sensor, but the present invention is not limited to this, and other distance measuring sensors may be used. For example, an ultrasonic distance measuring sensor, an infrared distance measuring sensor, a millimeter wave radar, or the like may be used as the correction sensor.
[0020] [Functions realized by the control unit 30] Fig. 2 is a functional block diagram showing an example of functions realized by the control unit in this embodiment. Each function shown in Fig. 2 is realized by a computer executing a program. The program may be stored in a computer-readable information recording medium.
[0021] In the control unit 30, a distance measurement result acquisition unit 30a which is a first depth information acquisition unit, an actual distance acquisition unit 30b which is a second depth information acquisition unit, a temperature acquisition unit 30c, and a correction unit 30d are realized.
[0022] The distance measurement result acquisition unit 30a acquires the distance measurement result (first depth information) measured by the imaging device 10. The actual distance acquisition unit 30b acquires the distance measurement result (second depth information) measured by the TOF sensor 20. The temperature acquisition unit 30c acquires the environmental temperature T based on the distance measurement result acquired by the distance measurement result acquisition unit 30a and the distance measurement result acquired by the actual distance acquisition unit 30b. The correction unit 30d corrects the distance measurement result for each of the multiple pixels of the imaging element 13 based on the environmental temperature T acquired by the temperature acquisition unit 30c. Note that in this embodiment, the environmental temperature T is the temperature around the lens 11 and the imaging element 13, or the temperature of the lens 11 and the imaging element 13 that retains heat generated in response to the temperature or the operation of the device.
[0023] [Depth information correction] Next, correction of depth information in this embodiment will be described with reference to Fig. 1 to Fig. 5. Fig. 3 is a graph showing an example of the relationship between the distance measurement result by the imaging device and the environmental temperature. Fig. 4 is a diagram showing an example of a table stored in the storage unit. Fig. 5 is a diagram explaining correction of depth information for each pixel.
[0024] Here, the following formula (1) is shown in the above-mentioned Non-Patent Document 1. Formula (1) represents the magnitude b of blur.
[0025]
number
[0026] In formula (1), a is the size of the aperture. p is the distance between the image sensor 13 and the lens 11. v is the distance between the lens 11 and the point where the incident light is focused. That is, v is the distance from the lens 11 to the image. According to formula (1), the degree of blur is determined based on the size of the aperture a, the distance v from the lens 11 to the image, and the distance p between the lens 11 and the image sensor 13. If these parameters change, the degree of blur cannot be obtained accurately. That is, the accuracy of the depth information obtained by the image capture device 10 decreases. It is to be noted that according to formula (1), if v and p match, the magnitude b of the blur is 0. That is, an in-focus image is obtained.
[0027] Here, the results of experiments performed by the inventors of the present application have revealed that the imaging device 10, which obtains depth information by observing blur using the coded aperture 12a, may not be able to obtain accurate depth information depending on the environmental temperature T. Fig. 3 shows the results of measuring the distance to an object at an actual distance D by the imaging device 10 while changing the temperature.
[0028] As shown in FIG. 3, in an environment with an environmental temperature of T1, the distance measurement result is D, which is consistent with the actual distance D. In an environment with an environmental temperature of T2 (>T1), the distance measurement result is D' (>D), which is longer than the actual distance D. In an environment with an environmental temperature of T3 (>T1), the distance measurement result is D'' (>D'), which is even longer than the distance measurement result in an environment with an environmental temperature of T2. As described above, as the environmental temperature T increases, the distance measurement result by the imaging device 10 becomes longer than the actual distance. This is considered to be because the distance between the lens 11 and the imaging element 13 changes as the environmental temperature T increases. In other words, the environmental temperature T causes the lens 11 and the imaging element 13 to expand or contract, which changes the distance between them. Note that, as shown in the above formula (1), the magnitude of the blur depends on the distance p between the lens 11 and the imaging element 13.
[0029] Therefore, in this embodiment, a configuration is adopted in which the relationship between the depth information (distance measurement result) acquired by the imaging device 10 and the actual distance is stored in advance for each environmental temperature T as a table, and the environmental temperature T is estimated and the depth information is corrected based on the table. The table shown in FIG. 4 may be stored in the storage unit 40.
[0030] The actual distance is the actual distance from the imaging device 10 to the imaging target. Here, the TOF sensor 20 is for measuring the distance in the depth direction like the imaging device 10, but has a lens 11 and an imaging element 13 like the imaging device 10, and measures the distance using a method different from the distance measurement method using the blur caused by the coded aperture 12a. Therefore, the TOF sensor 20 can measure the distance in the depth direction with high accuracy regardless of the environmental temperature T. Therefore, in this embodiment, the depth information acquired by the TOF sensor 20 is regarded as the actual distance.
[0031] In this embodiment, distance measurement is performed in advance by the imaging device 10, and the relationship between the distance measurement result and the actual distance for each environmental temperature T is obtained to create the table shown in Fig. 3. The distance measurement is preferably performed in a laboratory where the room temperature can be changed as appropriate.
[0032] As shown in Fig. 4, in an environment with an environmental temperature of T1, the actual distance to the imaging target when the distance measurement result is d1 is D11. In addition, in an environment with an environmental temperature of T1, the actual distance to the imaging target when the distance measurement result is d2 is D12. In addition, in an environment with an environmental temperature of T1, the actual distance to the imaging target when the distance measurement result is d3 is D13. As shown in Fig. 4, it is advisable to similarly obtain the relationship between the distance measurement result and the actual distance at environmental temperatures T2 and T3.
[0033] An example of correction of depth information using a table stored in advance in storage unit 40 will be described below with reference to Fig. 5. Fig. 5 shows a group of pixels before and after correction of depth information. Fig. 5 also shows an example in which imaging target A is displayed in pixels 7, 8, 12, and 13, and imaging target B is displayed in the other pixels. That is, pixels 7, 8, 12, and 13 contain depth information of imaging target A, and the other pixels contain depth information of imaging target B.
[0034] Here, when imaging target A is imaged, if the distance measurement result by imaging device 10 is d1 and the actual distance acquired by TOF sensor 20 is D11, control unit 30 determines that the environmental temperature is T1 by referring to the table shown in Fig. 4. Then, control unit 30 corrects the depth information of the pixel where imaging target A is displayed to D11.
[0035] As shown in Fig. 5, the distance measurement result by the imaging device 10 for imaging target B, which is the background of imaging target A, is d2. Therefore, the control unit 30 corrects the depth information of imaging target B by referring to the table shown in Fig. 4. Specifically, the control unit 30 determines that the actual distance to imaging target B is D12 because the distance measurement result is d2 at environmental temperature T1. Note that similarly, when a pixel with a distance measurement result of d3 is included, the control unit 30 may determine that the actual distance at that pixel is D13.
[0036] [flowchart] Next, the correction process of the depth information in this embodiment will be described with reference to Fig. 6. Fig. 6 is a diagram showing a flowchart of the correction process in this embodiment.
[0037] First, the distance measurement result acquisition unit 30a acquires depth information, which is a distance measurement result, for each pixel included in an image acquired by the imaging device 10 (S1). The actual distance acquisition unit 30b acquires the actual distance of the imaging target A acquired by the TOF sensor 20 (S2). Note that the order of S1 and S2 may be reversed, or they may be performed at the same timing.
[0038] Furthermore, the temperature acquisition unit 30c refers to a table stored in the storage unit 40 and acquires the environmental temperature based on the depth information of the imaging target A at the multiple pixels acquired by the imaging device 10 and the actual distance of the imaging target A acquired by the TOF sensor 20 (S3). Next, the correction unit 30d corrects the depth information, which is the distance measurement result at each pixel included in the image acquired by the imaging device 10, based on the acquired environmental temperature (S4).
[0039] [summary] In the present embodiment described above, it is possible to acquire depth information with high accuracy regardless of the environmental temperature. In addition, in the present embodiment, it is possible to correct the depth information for all pixels included in the image acquired by the imaging device 10 based on the actual distance of the imaging target acquired by the TOF sensor 20, and the processing load of the correction is small. That is, the processing load is small compared to a configuration in which the TOF sensor 20 acquires the actual distance for each of a plurality of pixels included in the image acquired by the imaging device 10. In addition, the TOF sensor 20 has a feature that it can obtain a distance measurement result close to the actual distance regardless of the environmental temperature, but has a disadvantage that it is easily affected by external light and the measurable distance is short. In the present embodiment, it is possible to complement the disadvantages of distance measurement by the coding imaging method and distance measurement by the TOF sensor 20, and it is possible to acquire depth information with high accuracy.
[0040] In the present embodiment, the example in which the table shown in FIG. 4 is stored in the storage unit 40 has been described, but the present invention is not limited to this. For example, the storage unit 40 may store a function that represents the relationship between the distance measurement result and the actual distance. In this case, the temperature acquisition unit 30c may acquire the environmental temperature based on the function stored in the storage unit 40. [Explanation of symbols]
[0041] 10 imaging device, 11 lens, 12 aperture section, 12a coded aperture, 13 imaging element, 20 TOF sensor, 30 control section, 30a distance measurement result acquisition section, 30b actual distance acquisition section, 30c temperature acquisition section, 30d correction section, 40 memory section, 100 imaging unit.
Claims
1. an imaging device including an optical system, a diaphragm unit in which a coded aperture pattern for partially blocking light incident on the optical system is formed, and an imaging element for acquiring image data based on the incident light that has passed through the optical system; a correction sensor that measures the time it takes for the emitted light to reflect off the surface of an arbitrary imaging target and return; a first depth information acquisition unit that acquires first depth information for each of a plurality of pixels of the imaging element; a second depth information acquisition unit that acquires second depth information of the imaging target based on a measurement result of the correction sensor; a storage unit that stores a relationship between the first depth information and the second depth information in association with an environmental temperature; a temperature acquisition unit that acquires an environmental temperature based on the first depth information of the imaging target acquired by the first depth information acquisition unit and the second depth information of the imaging target acquired by the second depth information acquisition unit; a correction unit that corrects first depth information for each of the plurality of pixels based on the environmental temperature acquired by the temperature acquisition unit; An imaging unit having the above configuration.
2. acquiring image data based on incident light that has passed through a diaphragm portion in which a coded aperture pattern is formed, and acquiring first depth information for each of a plurality of pixels of an image sensor; acquiring second depth information of an arbitrary imaging target based on a measurement result of a correction sensor that measures a time taken for emitted light to be reflected by a surface of the imaging target and return; acquiring an environmental temperature based on the first depth information and the second depth information using a storage unit that stores a relationship between the first depth information and the second depth information in association with an environmental temperature; correcting first depth information for each of the plurality of pixels based on the acquired environmental temperature; The imaging method includes:
