Control device, image capturing device, control system, mobile device, control method, and program

Abstract

To provide an image capturing device capable of acquiring highly accurate ranging data by easily reducing the effect of the deviation of a focal position.SOLUTION: A control device for controlling an image capturing device comprising image capturing means for capturing image data and a lens unit including multiple lenses, has: acquisition means for acquiring information on a temperature of the lens unit along an optical axis direction; and generation means for generating ranging data from the image data according to the information on the temperature.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a control device capable of acquiring the distance to an imaging target.

Background Art

[0002] Conventionally, an imaging device mounted on a vehicle and imaging the outside of the vehicle has also been used as a distance measurement camera system for measuring the distance to obstacles around the vehicle. Since the focal position of the lens unit included in the imaging device moves due to temperature changes, it is necessary to correct the focal position in order to acquire high-precision distance measurement data. Patent Document 1 discloses a configuration in which a correction lens capable of adjusting the focal position is physically moved by a drive unit such as a motor.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] A distance measurement camera system mounted on a vehicle is used in a harsh environment that is subject to heat effects such as sunlight, the temperature inside and outside the vehicle, and the heat generated by the imaging device itself, vibration, and impact. A configuration for correcting the focal position by moving the lens using a drive unit as in Patent Document 1 has a complicated structure, and it is difficult to adopt it for a vehicle used in a harsh environment over a long period of time from the viewpoint of reliability.

[0005] An object of the present invention is to provide a control device capable of acquiring high-precision distance measurement data.

Means for Solving the Problems

[0006] A control device according to one aspect of the present invention is a control device for controlling an imaging device including an imaging unit that captures image data and a lens unit including a plurality of lenses, the control device having an acquisition unit that acquires information regarding the temperature in the optical axis direction of the lens unit, and a generation unit that generates distance measurement data from the image data according to the information regarding the temperature.

Advantages of the Invention

[0007] According to the present invention, it is possible to provide a control device capable of acquiring highly accurate distance measurement data.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and overlapping descriptions are omitted. <Arrangement of Imaging Device Inside Vehicle> FIG. 1 is a diagram showing the mounting position of an imaging device 1 in a vehicle 100 according to an embodiment of the present invention. Note that the imaging device 1 is an in-vehicle camera mounted on the vehicle 100 in the present embodiment, but it may be mounted on a moving body other than a vehicle.

[0010] FIG. 1(a) shows a state where the imaging device 1 is arranged near the upper part of the front windshield (front glass) in the vehicle interior of the vehicle 100. In FIG. 1(a), the imaging device 1 monitors the front when the vehicle 100 is running.

[0011] FIG. 2 is a diagram showing a method of mounting the imaging device 1 near the upper part of the front glass of the vehicle 100. The imaging device 1 is attached to the vehicle 100 via a bracket 101. The bracket 101 is adhesively attached to the upper side of the vehicle interior of the front windshield 102 to fix the imaging device 1. As a fixing method between the bracket 101 and the imaging device 1, a method using fastening parts (not shown) such as press-fitting or screws is used.

[0012] Note that the imaging device 1 can be installed at various locations on the vehicle 100. For example, as shown in FIG. 1(b), it may be installed on the upper part of the dashboard, or as shown in FIG. 1(c), it may be installed at the rear part of the vehicle 100. Also, it may be installed at various positions capable of imaging the front and rear sides. <Function of Imaging Device Inside Vehicle and Vehicle Control> FIG. 3 is a block diagram of the imaging device 1. Light from the imaging object is incident on the lens unit 8. The light from the imaging object reaches the imaging element 9 via the lens unit 8 and is received by the imaging element 9. The imaging element 9 outputs a signal corresponding to the received light. The imaging element 9 is a semiconductor image sensor element such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge-Coupled Device) image sensor. Further, the imaging element 9 has a function capable of specifying a phase difference (shift of the image signal) by dividing one pixel into a plurality of photodiodes, and the imaging device 1 can acquire the distance to an obstacle around the vehicle in the imaging surface phase difference method. In the imaging surface phase difference method, the camera control IC (control device) 11 described later estimates the distance using the shift of the image signals generated by the light incident on a plurality of pixels via the lens unit 8.

[0013] The imaging element substrate 10 is provided with a signal output circuit and a first temperature sensor (detection means) 51. The signal output circuit outputs an image signal indicating an image of the imaging object to the main body substrate 6 in response to a signal output from the imaging element 9 based on the light incident on the lens unit 8. The first temperature sensor 51 acquires (detects) the temperature (second temperature) on the side of the imaging element 9 of the imaging element 9 and the lens unit 8. The signal output circuit outputs the temperature acquired by the first temperature sensor 51 to the main body substrate 6.

[0014] In the present embodiment, the imaging element 9 and the imaging element substrate 10 function as an imaging unit (imaging means) for imaging image data.

[0015] The main body substrate 6 includes a camera control IC 11 and a second temperature sensor 52. The camera control IC 11 processes the image signal output from the signal output circuit and generates distance measurement data. The second temperature sensor 52 acquires (detects) the temperature (first temperature) on the side of the imaging object of the lens unit 8.

[0016] The camera control IC 11 determines a correction value from the information on the correction value stored in the storage device 16 based on the temperatures acquired by the first temperature sensor 51 and the second temperature sensor 52, and corrects the distance measurement data generated using the correction value. The image signal processed by the camera control IC 11 can be output from the main body board 6 to the information processing device 103 in the vehicle 100.

[0017] In the present embodiment, the camera control IC 11 functions as an acquisition means for acquiring information regarding the temperature in the optical axis direction of the lens unit 8, and also functions as a generation means for generating distance measurement data from the image data according to the information regarding the temperature. Further, although the camera control IC 11 is mounted in the imaging device 1 in the present embodiment, it may be mounted on the vehicle 100. <First Embodiment> FIG. 4(a) and FIG. 4(b) are respectively an external view and an exploded view of the imaging device 1 of the present embodiment. FIG. 5 is a cross-sectional view of the imaging device 1. FIG. 6 is a cross-sectional view of the image sensor unit 5.

[0018] As shown in FIG. 4, in the imaging device 1, the image sensor unit 5, the main body board 6, etc. are housed in a housing composed of an upper housing 2, a lower housing 3, and a rear housing 4.

[0019] The upper housing 2 is made of metal such as aluminum or resin and forms the outer shell of the imaging device 1 in the X+ direction. The upper housing 2 includes a front wall portion 21, an upper wall portion 22, a right side wall portion 23, a left side wall portion 24, a front side wall portion 25, and an upper side wall portion 26, and functions as an attachment portion to the vehicle 100. The front wall portion 21 is configured as a flat plate parallel to the imaging element 9 and is arranged to face the outside of the vehicle 100. A hole for inserting the imaging element unit 5 is formed in the front wall portion 21. The upper wall portion 22 intersects the front wall portion 21. The upper wall portion 22 extends from the end surface of the front wall portion 21 along the X+ direction to the side in the Z− direction and is configured as a flat plate. A hole connected to the hole in the front wall portion 21 is formed in the upper wall portion 22 so as not to interfere with the imaging element unit 5. An upper lid 27 is installed to close the hole in the upper wall portion 22. The upper lid 27 is made of metal such as aluminum or resin and is joined by fastening components (not shown) such as screws or snap fits after the imaging element unit 5 is inserted. Heat-radiating fins (not shown) may be arranged on the Z+ direction side of the upper wall portion 22. A main body substrate 6 on which a plurality of electronic components are mounted is attached to the Z− direction side of the upper wall portion 22.

[0020] The lower housing 3 is made of metal such as aluminum or resin and forms the outer shell of the imaging device 1 on the Z− direction side. The lower housing 3 is installed so as to close the Z− direction side of the upper housing 2. The flat surface portion 31 is configured as a flat plate parallel to the upper wall portion 22.

[0021] The rear housing 4 is made of metal such as aluminum or resin and forms the outer shell of the imaging device 1 on the X− direction side. The rear housing 4 includes a front wall portion 41 and a rear wall portion 42. The front wall portion 41 is arranged substantially parallel to the imaging element substrate 10 and is joined to the upper housing 2 by fastening components (not shown) such as screws. Heat-radiating fins (not shown) may be attached to the rear wall portion 42.

[0022] The imaging element unit 5 is composed of a lens unit 8, an imaging element 9, and an imaging element substrate 10. The imaging element unit 5 is inserted into the hole of the upper housing 2 and joined by screws, adhesives, etc. An imaging element heat conduction member 13 is laminated on the X-direction side of the imaging element unit 5 (the X-direction side surface of the imaging element substrate 10). The imaging element heat conduction member 13 is a rubber-like material such as silicone and is configured in a thin film shape. The imaging element heat conduction member 13 is in contact with the front wall portion 41 and conducts the heat of the imaging element 9. The heat received by the front wall portion 41 is radiated from the rear wall portion 42.

[0023] The lens unit 8 has an end portion attached to the imaging element substrate 10 at a substantially right angle via an adhesive or the like. In this embodiment, the lens unit 8 is directly attached to the imaging element substrate 10, but the present invention is not limited to this. The lens unit 8 may be indirectly attached to the imaging element substrate 10 via, for example, the upper housing 2 or other members.

[0024] The lens unit 8 includes a cylindrical lens barrel 81 formed of a metal such as aluminum or a resin. A plurality of lenses 82 and a plurality of spacers 83 are accommodated in the lens barrel 81. The plurality of lenses 82 are formed of a transmissive material such as glass or resin. A wide-angle lens 84 is disposed on the side closest to the imaging target among the plurality of lenses 82. The wide-angle lens 84 is held by a retainer ring 85. The plurality of spacers 83 and the retainer ring 85 are formed of a metal such as aluminum or a resin. The retainer ring 85 includes a protrusion (protrusion member) 86 that protrudes from the Z-direction end portion in a direction orthogonal to the optical axis direction. The protrusion 86 is close to the second temperature sensor 52.

[0025] In FIG. 6, the plurality of lenses 82 and the plurality of spacers 83 have the same shape and the same quantity and are alternately accommodated, but the present invention is not limited to this. They may have different shapes and different quantities, or the order of accommodation may be different from the order in FIG. 6.

[0026] The imaging element 9 is attached to the surface of the imaging element substrate 10 on the side of the lens unit 8 and is housed inside the lens barrel 81. The imaging element 9 is the main heat source of the imaging element unit 5.

[0027] The imaging element substrate 10 is configured in a flat plate shape extending in the Y direction and the Z direction, and controls the imaging element 9. A first temperature sensor 51 is attached to the surface of the imaging element substrate 10 on the side opposite to the lens unit 8.

[0028] The main body substrate 6 is configured in a flat plate shape extending in the X direction and the Y direction, and is joined to the boss portion of the upper wall portion 22 by fastening components such as screws 7. The main body substrate 6 may be joined to the lower housing 3. The main body substrate 6 and the imaging element substrate 10 are electrically connected via the wiring 12. A plurality of electronic components are provided on both the Z+ side and the Z− side of the main body substrate 6. The plurality of electronic components mounted on the main body substrate 6 include a second temperature sensor 52 and a camera control IC 11.

[0029] The second temperature sensor 52 is attached at a position close to the protrusion 86 and acquires the temperature on the imaging target side of the lens unit 8. An upper surface side heat conduction member 15 is laminated on the second temperature sensor 52. The upper surface side heat conduction member 15 is a rubber-like material such as silicone and is configured in a thin film shape. Since the second temperature sensor 52 is indirectly in contact with the protrusion 86 via the upper surface side heat conduction member 15 and is thermally connected, the temperature of the lens unit 8 can be acquired. However, if the temperature on the imaging target side of the lens unit 8 can be acquired even when there is an air layer due to the proximity of the second temperature sensor 52 and the protrusion 86, the upper surface side heat conduction member 15 may not be used.

[0030] The camera control IC 11 integrally controls the entire imaging device 1. The upper surface side heat conduction member 15 is laminated in contact with at least a part of a plurality of electronic components including the camera control IC 11. The upper surface side heat conduction member 15 is in contact with the Z− direction side of the upper wall portion 22 and conducts the heat of the electronic components. The upper wall portion 22 receives the heat conducted from the upper surface side heat conduction member 15 on the Z− direction side and dissipates the heat from the Z+ direction side.

[0031] On the Z-direction side of the main body substrate 6, a lower surface side heat conduction member 14 is laminated. The lower surface side heat conduction member 14 is a rubber-like material such as silicone and is configured in a thin film shape. The lower surface side heat conduction member 14 is in contact with at least a part of a plurality of electronic components. The lower surface side heat conduction member 14 is in contact with the Z+ direction side of the lower housing 3 and conducts the heat of the electronic components. The lower housing 3 receives the heat conducted from the lower surface side heat conduction member 14 on the Z+ direction side and dissipates the heat from the Z− direction side. Note that the main body substrate 6 and the upper wall portion 22 are arranged on the Z− direction side of the lens unit 8 in FIG. 4, but may be arranged on the Z+ direction side of the lens unit 8.

[0032] Note that the first temperature sensor 51 is provided on the surface of the imaging element substrate 10 opposite to the side of the lens unit 8 in the present embodiment, but the installation position (second position) is not limited to this as long as the temperature on the side of the imaging element 9 of the lens unit 8 can be acquired. The first temperature sensor 51 is preferably installed within a range of 15% or less of the length in the optical axis direction of the lens unit 8 from the end on the side of the imaging means of the lens unit 8.

[0033] Further, the second temperature sensor 52 is provided at a position close to the protrusion 86 in the present embodiment, but the installation position (first position) is not limited to this as long as the temperature on the side of the imaging target of the lens unit 8 can be acquired. The second temperature sensor 52 is preferably installed within a range of 15% or less of the length in the optical axis direction of the lens unit 8 from the end on the side of the imaging target of the lens unit 8.

[0034] Hereinafter, the operation and effect of the imaging device 1 of the present embodiment will be described. In the present embodiment, the camera control IC 11 can determine a correction value for the distance measurement data by estimating (acquiring) the temperature distribution of the lens unit 8 (information regarding the temperature in the optical axis direction of the lens unit 8).

[0035] FIG. 7 is a diagram showing the heat transfer path around the lens unit 8. The main heat sources are two: heat source A due to the heat generation of the imaging device 9 itself and heat source B due to sunlight or the like. Heat source B is a factor causing defocus. Since the two heat sources are at both ends of the lens unit 8, if the temperatures at both ends can be obtained, the temperature distribution of the lens unit 8 can be estimated.

[0036] Heat source A is heat-transferred from the imaging device substrate 10 to the lens unit 8 and from the imaging device heat conduction member 13 to the rear housing 4. Since the first temperature sensor 51 and the lens unit 8 are directly attached to the imaging device substrate 10, they are thermally connected. Therefore, the first temperature sensor 51 can obtain the temperature on the side of the imaging device 9 of the lens unit 8.

[0037] Heat source B is heat-transferred from the wide-angle lens 84 and the pressing ring 85 to the entire lens unit 8. The second temperature sensor 52 is close to the protrusion 86 and is thermally connected via the upper surface side heat conduction member 15, so the temperature on the side of the imaging target of the lens unit 8 can be obtained.

[0038] FIG. 8 is a graph showing the relationship between the temperature difference at both ends of the lens unit 8 and the increase amount of the phase difference. In FIG. 8, the horizontal axis is the value (temperature difference) obtained by subtracting the temperature on the side of the imaging device 9 of the lens unit 8 from the temperature on the side of the imaging target of the lens unit 8, and the vertical axis is the increase amount of the phase difference. The deviation of the image signal generated when the light passing through the lens unit 8 is incident on the imaging device 9, that is, the phase difference, can be measured in pixel units. Therefore, in FIG. 8, the increase amount of the phase difference is shown as a numerical value in pixel units. As shown in FIG. 8, when there is no heat source B, the temperature difference is -1 which is smaller than 0, so the increase amount of the phase difference is 0. When the temperature of heat source B increases, the temperature difference increases and the increase amount of the phase difference also increases.

[0039] FIG. 9 is a conceptual diagram showing the principle of acquiring distance measurement data. FIG. 9(a) shows a state where the heat source is only heat source A and the focus position is not shifted. FIG. 9(b) shows a state where there are two heat sources, heat source A and heat source B, and the focus position is shifted compared to FIG. 9(a). As described above, the imaging element 9 uses the light from the imaging target incident through the lens unit 8 to obtain a phase difference 91, which is a shift of the image signal for each pixel, by the imaging surface phase difference method. By multiplying the phase difference 91 by a coefficient set for each imaging device 1, the defocus amount 92 can be obtained. The defocus amount 92 can be converted into the distance 93 to the imaging target using the lens design values (focal length, focus distance) and the lens formula.

[0040] In FIG. 9(b), the lens unit 8 moves with respect to FIG. 9(a) due to the influence of the heat source B. The actual movement amount is in the unit of several microns. Due to the movement of the lens unit 8, the phase difference 91 and the defocus amount 92 increase. The distance 93 to the imaging target includes an error distance 94 with respect to the exact position of the imaging target. Therefore, by subtracting the increase amount of the phase difference obtained from the temperature difference between both ends of the lens unit 8 from the phase difference 91 in FIG. 9(b), the value of the phase difference 91 in FIG. 9(a) can be corrected. As a result, the distance 93 to the imaging target can also be corrected.

[0041] As described above, according to the configuration of the present embodiment, it is possible to grasp the temperature distribution of the lens unit 8, easily reduce the influence of the shift of the focus position due to the expansion and contraction of the lens barrel 81 and the lens, and acquire high-precision distance measurement data. <Second Embodiment> FIG. 10 is a cross-sectional view of the imaging device 1A of the present embodiment. The imaging device 1A is different from the imaging device 1 of the first embodiment in that the second temperature sensor 52 is installed on the flexible substrate 17 connected to the main body substrate 6 and attached to the upper wall portion 22. In the present embodiment, only the configuration different from the first embodiment will be described, and the description of the common configuration will be omitted.

[0042] The upper wall portion 22 is provided with a hole for passing the flexible substrate 17. The flexible substrate 17 is attached to the upper wall portion 22 using fastening components such as screws, double-sided tape, or an adhesive. The second temperature sensor 52 is attached in proximity to the pressing ring 85. The second temperature sensor 52 may have a heat conduction member laminated thereon and be connected to the pressing ring 85.

[0043] In the imaging device 1A, the first temperature sensor 51 can acquire the temperature on the side of the imaging element 9 of the lens unit 8. Since the second temperature sensor 52 is attached in proximity to the pressing ring 85, it can acquire the temperature on the side of the imaging target of the lens unit 8. An increase amount of the phase difference is calculated (acquired) from the difference (temperature difference) obtained by subtracting the temperature acquired by the second temperature sensor 52 from the temperature acquired by the first temperature sensor 51. The distance measurement data can be corrected by subtracting the increase amount of the phase difference calculated from the phase difference that is the basis of the distance measurement data.

[0044] As described above, according to the configuration of the present embodiment, it is possible to grasp the temperature distribution of the lens unit 8, easily reduce the influence of the shift of the focus position due to the expansion and contraction of the lens barrel 81 and the lens, and acquire highly accurate distance measurement data. <Third Embodiment> FIG. 11 is a cross-sectional view of the imaging device 1A of the present embodiment. The imaging device 1B is different from the imaging device 1 of the first embodiment in that the second temperature sensor 52 and the third temperature sensor 53 are installed on the flexible substrate 17 connected to the main body substrate 6 and attached to the upper wall portion 22. Also, it is different in that a protrusion 87 is installed on the outer shape portion of the lens barrel 81. In the present embodiment, only the configuration different from the first embodiment will be described, and the description of the common configuration will be omitted.

[0045] The upper wall portion 22 is provided with a hole for passing the flexible substrate 17. The flexible substrate 17 is attached to the upper wall portion 22 using fastening components such as screws, double-sided tape, or an adhesive. The second temperature sensor 52 is attached in proximity to the pressing ring 85. The second temperature sensor 52 may have a heat conduction member laminated thereon and be connected to the pressing ring 85. The third temperature sensor 53 is attached in proximity to the protrusion 87 provided on the outer peripheral portion of the lens barrel 81. The third temperature sensor 53 may have a heat conduction member laminated thereon and be connected to the lens barrel 81. The protrusion 87 is configured in a protruding shape to facilitate connection with the third temperature sensor 53, but the shape is not limited thereto. Also, the protrusion 87 may be formed as a separate member from the lens barrel 81. Further, in this embodiment, the position of the protrusion 87 is around the center in the optical axis direction of the lens unit 8, but the installation position is not limited thereto. The protrusion 87 may be installed at a position effective for correcting phase difference displacement, such as near an adjustment lens having a function of correcting focus shift due to temperature change or near a position where the uneven portion provided on the lens barrel 81 joins the housing. If there is a temperature change point due to the combination of the shape and material of the lens holding member, it may be installed so as to acquire a temperature close to the temperature change point.

[0046] In the imaging device 1B, the first temperature sensor 51 can acquire the temperature on the side of the imaging element 9 of the lens unit 8. Since the second temperature sensor 52 is attached in proximity to the pressing ring 85, it can acquire the temperature on the side of the imaging target of the lens unit 8. Since the third temperature sensor 53 is in proximity to the protrusion 87, it can acquire the temperature of the intermediate portion of the lens unit 8.

[0047] FIG. 12 is a graph showing the relationship between the temperature difference of the lens unit 8 and the increase amount of the phase difference. In FIG. 12(a), the horizontal axis represents the value obtained by subtracting the temperature acquired by the third temperature sensor 53 from the temperature acquired by the first temperature sensor 51 (temperature difference A), and the vertical axis represents the increase amount A of the phase difference. In FIG. 12(b), the horizontal axis represents the value obtained by subtracting the temperature acquired by the second temperature sensor 52 from the temperature acquired by the third temperature sensor 53 (temperature difference B), and the vertical axis represents the increase amount B of the phase difference. By subtracting the increase amount A of the phase difference or the increase amount B of the phase difference from the phase difference that is the basis of the distance measurement data, the distance measurement data can be corrected. Note that the temperature difference B may be the value obtained by subtracting the temperature acquired by the second temperature sensor 52 from the temperature acquired by the first temperature sensor 51.

[0048] As described above, according to the configuration of the present embodiment, it is possible to grasp the temperature distribution of the lens unit 8, easily reduce the influence of the shift of the focus position due to the expansion and contraction of the lens barrel 81 and the lens, and acquire highly accurate distance measurement data. <Others> The materials of the upper housing 2, the lower housing 3, the rear housing 4, the lens barrel 81, the spacer 83, and the retainer ring 85 are not particularly limited. Preferably, they are made of materials with high thermal conductivity, such as metals such as aluminum, copper, zinc, and iron, or alloys obtained by mixing a plurality of metals.

[0049] The materials of the imaging element heat conduction member 13, the lower surface side heat conduction member 14, and the upper surface side heat conduction member 15 are not particularly limited. For example, silicone, graphite, or the like may be used. Also, a curable material obtained by mixing two liquid materials, a non-curable material such as grease, or the like may be used. <In-vehicle system> FIG. 13 is a configuration diagram of an imaging device 1 and an in-vehicle system (control system, driving support device) 600 including the same according to the present embodiment. The in-vehicle system 600 is held by a movable body (mobile device) such as an automobile (vehicle), and is a system for assisting the driving (operation) of the vehicle based on image information around the vehicle acquired by the imaging device 1 which is an in-vehicle camera. FIG. 14 is a schematic diagram of a vehicle 700 as a mobile device including the in-vehicle system 600. In FIG. 14, a case where the imaging range 50 of the imaging device 1 is set in front of the vehicle 700 is shown, but the imaging range 50 may be set behind or on the side of the vehicle 700 or the like.

[0050] As shown in FIG. 13, the in-vehicle system 600 includes an imaging device 1, a vehicle information acquisition device 20, a control device (control unit, ECU: electronic control unit) 30, and a warning device (warning unit) 40. Further, the imaging device 1 includes an imaging unit 61, an image processing unit 62, a parallax calculation unit 63, a distance acquisition unit 64, and a collision determination unit 65. The collision determination unit 65 may be included in the control device 30. The image processing unit 62, the parallax calculation unit 63, the distance acquisition unit 64, and the collision determination unit 65 constitute a processing unit. The imaging unit 61 has an optical system and an imaging element according to any of the above-described embodiments.

[0051] FIG. 15 is a flowchart showing an operation example of the in-vehicle system 600 according to the present embodiment. Hereinafter, the operation of the in-vehicle system 600 will be described along this flowchart.

[0052] First, in step S1, the imaging unit 61 is used to image an object (subject) such as an obstacle or a pedestrian around the vehicle, and a plurality of image data (parallax image data) are acquired.

[0053] Also, in step S2, the vehicle information acquisition device 20 acquires vehicle information. The vehicle information is information including the vehicle speed, yaw rate, steering angle, etc. of the vehicle.

[0054] In step S3, image processing is performed on the plurality of image data acquired by the imaging unit 61 by the image processing unit 62. Specifically, image feature analysis is performed to analyze feature amounts such as the amount and direction of edges and density values in the image data. Here, the image feature analysis may be performed on each of the plurality of image data, or may be performed only on some of the plurality of image data.

[0055] In step S4, parallax (image displacement) information between the plurality of image data acquired by the imaging unit 61 is calculated by the parallax calculation unit 63. As a method for calculating the parallax information, known methods such as the SSDA method and the area correlation method can be used, and thus the description thereof is omitted in this embodiment. Note that steps S2, S3, and S4 may be performed in the above order, or may be processed in parallel with each other.

[0056] In step S5, distance information between the object imaged by the imaging unit 61 is acquired (calculated) by the distance acquisition unit 64. The distance information can be calculated based on the parallax information calculated by the parallax calculation unit 63, the internal parameters and external parameters of the imaging unit 61. Here, the distance information refers to information regarding the relative position of the object, such as the distance to the object, the defocus amount, and the image displacement amount, and may directly represent the distance value of the object in the image or may indirectly represent information corresponding to the distance value.

[0057] Then, in step S6, using the vehicle information acquired by the vehicle information acquisition device 20 and the distance information calculated by the distance acquisition unit 64, the collision determination unit 65 determines whether or not the distance to the object is within a preset distance range. Thereby, it can be determined whether or not an object exists within the preset distance around the vehicle, and the possibility of collision between the vehicle and the object can be determined. When an object exists within the preset distance, the collision determination unit 65 determines that "there is a possibility of collision" (step S7), and when no object exists within the preset distance, it determines that "there is no possibility of collision" (step S8).

[0058] Next, when the collision determination unit 65 determines that "there is a possibility of collision", it notifies (transmits) the determination result to the control device 30 and the warning device 40. At this time, the control device 30 controls the vehicle based on the determination result of the collision determination unit 65 (step S6), and the warning device 40 warns the user (driver, passenger) of the vehicle based on the determination result of the collision determination unit 65 (step S7). Note that the notification of the determination result may be made to at least one of the control device 30 and the warning device 40.

[0059] The control device 30 can control the movement of the vehicle by outputting a control signal to the drive unit (engine, motor, etc.) of the vehicle. For example, in the vehicle, it performs controls such as applying brakes, releasing the accelerator, turning the steering wheel, and generating a braking force for each wheel to generate a control signal to suppress the output of the engine or motor. Further, the warning device 40 warns the user, for example, by emitting a warning sound (alarm), displaying warning information on the screen of a car navigation system, or applying vibration to the seat belt or steering wheel.

[0060] As described above, according to the in-vehicle system 600 according to the present embodiment, the target object can be effectively detected by the above processing, and it is possible to avoid a collision between the vehicle and the target object. In particular, by applying the optical systems according to the above-described respective embodiments to the in-vehicle system 600, it is possible to miniaturize the entire imaging device 1, increase the degree of freedom of arrangement, and perform detection and collision determination of the target object over a wide angle of view.

[0061] Regarding the calculation of the distance information, various embodiments can be considered. As an example, the case where a pupil division type image sensor having a plurality of pixel portions regularly arranged in a two-dimensional array is adopted as the image sensor included in the imaging unit 61 will be described. In the pupil division type image sensor, one pixel portion is composed of a microlens and a plurality of photoelectric conversion portions, receives a pair of light beams passing through different regions in the pupil of the optical system, and can output paired image data from each photoelectric conversion portion.

[0062] Then, the image displacement amount of each region is calculated by the correlation operation between the paired image data, and the displacement map data representing the distribution of the image displacement amount is calculated by the distance acquisition unit 64. Alternatively, the distance acquisition unit 64 may further convert the image displacement amount into a defocus amount, and generate defocus map data representing the distribution of the defocus amount (distribution on the two-dimensional plane of the captured image). Further, the distance acquisition unit 64 may acquire distance map data of the distance to the object converted from the defocus amount.

[0063] In addition, in the event that the vehicle 700 collides with an obstacle, the in-vehicle system 600 and the vehicle 700 may be provided with a notification device (notification unit) for notifying the manufacturer of the in-vehicle system (manufacturer) and the dealer of the mobile device (dealer) of such. For example, as the notification device, one that transmits information regarding the collision between the vehicle 700 and the obstacle (collision information) to a preset external notification destination by e-mail or the like can be adopted.

[0064] In this way, by adopting a configuration in which the collision information is automatically notified by the notification device, it is possible to promptly take measures such as inspection and repair after a collision occurs. Note that the notification destination of the collision information may be an insurance company, a medical institution, the police, etc., or any one set by the user. Further, not limited to the collision information, the notification device may be configured to notify the notification destination of the failure information of each part and the consumption information of consumables. Regarding the detection of the presence or absence of a collision, it may be performed using the acquired distance information, or may be performed by another detection unit (sensor).

[0065] In this embodiment, the in-vehicle system 600 is applied to driving support (collision damage reduction), but not limited thereto, the in-vehicle system 600 may be applied to cruise control (including the full vehicle speed following function) or automatic driving. Further, the in-vehicle system 600 is not limited to vehicles such as automobiles, and can be applied to moving bodies such as ships, airplanes, and industrial robots. Further, not limited to moving bodies, it can be applied to various devices that utilize object recognition such as advanced road traffic systems (ITS). [Other Embodiments] The present invention can also be implemented by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. It can also be implemented by a circuit (for example, an ASIC) that realizes one or more functions.

[0066] The disclosure of this embodiment includes the following configurations and methods. (Configuration 1) A control device for controlling an imaging device including an imaging unit that captures image data and a lens unit including a plurality of lenses, an acquisition unit that acquires information regarding the temperature in the optical axis direction of the lens unit, and a generation unit that generates distance measurement data from the image data according to the information regarding the temperature. The control device is characterized by having these. (Configuration 2) The control device according to Configuration 1, wherein the generation unit generates the distance measurement data by correcting first distance measurement data generated from the image data according to the information regarding the temperature. (Configuration 3) The control device according to Configuration 2, wherein the generation unit determines a correction value according to the information regarding the temperature, and generates the distance measurement data by correcting the first distance measurement data using the correction value. (Configuration 4) The information regarding the temperature includes a first temperature at a first position on the imaging target side of the lens unit and a second temperature at a second position on the imaging unit side of the lens unit, and the control device according to Configuration 3, wherein the generation unit determines the correction value based on the difference between the first temperature and the second temperature. (Configuration 5) The information regarding the temperature includes a first temperature at a first position on the imaging target side of the lens unit, a second temperature at a second position on the imaging unit side of the lens unit, and a third temperature at a third position between the first position and the second position, The generation means determines the correction value based on the difference between the first temperature and the third temperature, and the difference between the second temperature and the first temperature or the third temperature, in the control device according to Configuration 3. (Configuration 6) The third position is any one of the center of the lens unit in the optical axis direction, the vicinity of the adjustment lens having a function of correcting defocus, and the vicinity of the temperature change point due to the combination of the shape and material of the lens holding member, in the control device according to Configuration 5. (Configuration 7) The first position is included in a range within 15% of the length of the lens unit in the optical axis direction from the end on the imaging target side of the lens unit. The second position is included in a range within 15% of the length of the lens unit in the optical axis direction from the end on the imaging means side of the lens unit, in the control device according to any one of Configurations 4 to 6. (Configuration 8) The control device according to any one of Configurations 1 to 7, Imaging means for imaging image data, An imaging device having a lens unit including a plurality of lenses. (Configuration 9) The imaging means has a function of measuring a phase difference by dividing one pixel into a plurality of photodiodes. The generation means generates the distance measurement data using the phase difference, in the imaging device according to Configuration 8. (Configuration 10) A main body substrate electrically connected to an image sensor substrate included in the imaging means, A protruding member extending in a direction orthogonal to the optical axis direction from the lens unit, Detection means provided on the main body substrate so as to be close to the protruding member and detecting the temperature at the position on the imaging target side of the lens unit, further included in the imaging device according to Configuration 8 or 9. (Configuration 11) A main body substrate electrically connected to an image sensor substrate included in the imaging means, A flexible substrate connected to the main body substrate, A housing that covers the main body substrate and is provided between the main body substrate and the lens unit, Detection means provided on the housing side of the flexible substrate between the housing and the lens unit for detecting the temperature at the position on the imaging target side of the lens unit, and the imaging device according to Configuration 8 or 9, characterized by further comprising the same. (Configuration 12) An imaging device according to any one of Configurations 8 to 11, and a determination unit that determines the possibility of collision with the object based on the distance information of the object acquired by the imaging device, and a control system characterized by comprising the same. (Configuration 13) A control system according to Configuration 12, characterized by comprising a control device that outputs a control signal for generating a braking force to a drive unit of the moving device when it is determined that there is a possibility of collision between the moving device and the object. (Configuration 14) A control system according to Configuration 12 or 13, characterized by comprising a warning device that warns a user of the moving device when it is determined that there is a possibility of collision between the moving device and the object. (Configuration 15) A control system according to any one of Configurations 12 to 14, characterized by comprising a notification unit that notifies external information regarding a collision between the moving device and the object. (Configuration 16) A moving device comprising an imaging device according to any one of Configurations 8 to 11, and being movable while holding the imaging device. (Configuration 17) A moving device according to Configuration 16, characterized by having a determination unit that determines the possibility of collision with the object based on the distance information of the object obtained by the imaging device. (Configuration 18) A moving device according to Configuration 17, characterized by comprising a control unit that outputs a control signal for controlling movement when it is determined that there is a possibility of collision with the object. (Configuration 19) The mobile device according to Configuration 17 or 18, further comprising a warning unit configured to issue a warning to a user of the mobile device when it is determined that there is a possibility of collision with the object. (Configuration 20) The mobile device according to any one of Configurations 17 to 19, further comprising a notification unit configured to notify external parties of information regarding a collision with the object. (Method 1) A control method for controlling an imaging device including an imaging unit configured to capture image data and a lens unit including a plurality of lenses, the method comprising: obtaining information regarding a temperature in an optical axis direction of the lens unit; and generating distance measurement data from the image data according to the information regarding the temperature. (Configuration 21) A program causing a computer to execute the control method according to Method 1.

[0067] As described above, the preferred embodiments of the present invention have been described. However, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof.

Description of Reference Numerals

[0068] 1 Imaging device 8 Lens unit 9 Image sensor (imaging unit) 10 Image sensor substrate (imaging unit) 11 Camera control IC (acquisition unit, generation unit)

Claims

1. A control device for controlling an imaging device including imaging means for imaging image data and a lens unit including a plurality of lenses, acquisition means for acquiring information regarding the temperature in the optical axis direction of the lens unit, generation means for generating distance measurement data from the image data according to the information regarding the temperature, characterized in that the control device has the generation means.

2. The control device according to claim 1, characterized in that the generation means generates the distance measurement data by correcting first distance measurement data generated from the image data according to the information regarding the temperature.

3. The control device according to claim 2, characterized in that the generation means determines a correction value according to the information regarding the temperature, and generates the distance measurement data by correcting the first distance measurement data using the correction value.

4. The information regarding the temperature includes a first temperature at a first position on the imaging target side of the lens unit and a second temperature at a second position on the imaging means side of the lens unit, The control device according to claim 3, characterized in that the generation means determines the correction value based on the difference between the first temperature and the second temperature.

5. The information regarding the temperature includes a first temperature at a first position on the imaging target side of the lens unit, a second temperature at a second position on the imaging means side of the lens unit, and a third temperature at a third position between the first position and the second position, The control device according to claim 3, characterized in that the generation means determines the correction value based on the difference between the first temperature and the third temperature and the difference between the second temperature and the first temperature or the third temperature.

6. The control device according to claim 5, characterized in that the third position is any one of the center of the lens unit in the optical axis direction, the vicinity of an adjustment lens having a function of correcting defocus, and the vicinity of a temperature change point due to a combination of the shape and material of the lens holding member.

7. The control device according to claim 4 or 5, characterized in that the first position is included in a range within 15% of the length of the lens unit in the optical axis direction from the end on the imaging target side of the lens unit, The control device according to claim 4 or 5, characterized in that the second position is included in a range within 15% of the length of the lens unit in the optical axis direction from the end on the imaging means side of the lens unit.

8. The control device according to any one of claims 1 to 6, imaging means for imaging image data, An imaging device comprising a lens unit including a plurality of lenses.

9. The imaging means has a function capable of measuring a phase difference by dividing one pixel into a plurality of photodiodes, The imaging device according to claim 8, wherein the generation means generates the distance measurement data using the phase difference.

10. A main body substrate electrically connected to an image sensor substrate included in the imaging means, A protruding member extending from the lens unit in a direction orthogonal to the optical axis direction, The imaging device according to claim 8, further comprising: a detection means provided on the main body substrate so as to be close to the protruding member and detecting a temperature at a position on the imaging target side of the lens unit.

11. A main body substrate electrically connected to an image sensor substrate included in the imaging means, A flexible substrate connected to the main body substrate, A housing covering the main body substrate and provided between the main body substrate and the lens unit, The imaging device according to claim 8, further comprising: a detection means provided on the housing side of the flexible substrate between the housing and the lens unit and detecting a temperature at a position on the imaging target side of the lens unit.

12. A control system comprising the imaging device according to claim 8 and a determination unit that determines a possibility of collision with the object based on distance information of the object acquired by the imaging device.

13. The control system according to claim 12, further comprising a control device that outputs a control signal for generating a braking force to a drive unit of the mobile device when it is determined that there is a possibility of collision between the mobile device and the object.

14. The control system according to claim 12, further comprising a warning device that warns a user of the mobile device when it is determined that there is a possibility of collision between the mobile device and the object.

15. The control system according to claim 12, further comprising a notification unit that notifies external information regarding a collision between the mobile device and the object.

16. A mobile device comprising the imaging device according to claim 8 and being movable while holding the imaging device.

17. The mobile device according to claim 16, further comprising a determination unit that determines a possibility of collision with the object based on distance information of the object obtained by the imaging device.

18. The mobile device according to claim 17, further comprising a control unit configured to output a control signal for controlling movement when it is determined that there is a possibility of collision with the object.

19. The mobile device according to claim 17, further comprising a warning unit configured to warn a user of the mobile device when it is determined that there is a possibility of collision with the object.

20. The mobile device according to claim 17, further comprising a notification unit configured to notify external parties of information regarding a collision with the object.

21. A control method for controlling an imaging device including an imaging unit configured to capture image data and a lens unit including a plurality of lenses, the method comprising: obtaining information regarding the temperature in the optical axis direction of the lens unit; generating distance measurement data from the image data according to the information regarding the temperature.

22. A program, characterized in that the computer executes the control method according to claim 21.

Images