Control apparatus, image pickup apparatus, control system, mobile apparatus, control method, storage medium, and computer program product

Abstract

The invention discloses a control apparatus, an image pickup apparatus, a control system, a mobile apparatus, a control method, a storage medium, and a computer program product. The control apparatus controls an imaging apparatus including an imaging unit configured to capture image data and a lens unit having a plurality of lenses. The control apparatus includes a processor and a memory storing a program that causes the processor to execute a plurality of tasks. The plurality of tasks includes an acquisition task configured to acquire information related to a temperature in an optical axis direction of the lens unit, and a generation task configured to generate distance measurement data from the image data in accordance with the information related to the temperature.

Description

Technical Field

[0001] The present disclosure relates to a control device capable of obtaining the distance to an imaging object. Background Art

[0002] An imaging device mounted on a vehicle for photographing an external image of the vehicle has also been used as a distance measurement camera system for measuring the distance to an obstacle around the vehicle. Since the focal position of the lens unit in the imaging device shifts due to temperature changes, it is preferable to correct the focal position to obtain high-precision distance measurement data. Japanese Patent No. 6488429 discloses a configuration in which a correction lens capable of adjusting the focal position is physically moved by a drive unit such as a motor.

[0003] The distance measurement camera system mounted on a vehicle is used in a harsh environment exposed to thermal effects such as sunlight, internal and external temperatures, heat generated from the imaging device itself, as well as vibration and shock. The configuration of correcting the focal position by moving the lens with a drive unit as in Japanese Patent No. 6488429 has a complex structure, and due to reliability considerations, it is challenging to implement such a configuration in a vehicle used for an extended period in such a demanding environment. Summary of the Invention

[0004] A control device according to some embodiments controls an imaging device including an imaging unit configured to capture image data and a lens unit having a plurality of lenses. The control device includes a processor and a memory storing a program that causes the processor to execute a plurality of tasks. The plurality of tasks include an acquisition task configured to acquire information related to the temperature in the optical axis direction of the lens unit, and a generation task configured to generate distance measurement data from the image data based on the temperature-related information.

[0005] Further features of various embodiments of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. Brief Description of the Drawings

[0006] Figure 1A 、 Figure 1B and Figure 1C are diagrams showing the mounting positions of an imaging device on a vehicle according to an embodiment of the present disclosure.

[0007] Figure 2 is a diagram showing a method for mounting an imaging device near the upper part of a windshield of a vehicle.

[0008] Figure 3 is a block diagram of the imaging device.

[0009] Figure 4A and Figure 4BIt is an explanatory diagram of the imaging device according to the first embodiment.

[0010] Figure 5 It is a cross-sectional view of the imaging device according to the first embodiment.

[0011] Figure 6 It is a cross-sectional view of the image sensor unit according to the first embodiment.

[0012] Figure 7 It is a diagram showing the heat transfer path around the lens unit according to the first embodiment.

[0013] Figure 8 It is a graph showing the relationship between the temperature difference at both ends of the lens unit and the increase in the phase difference according to the first embodiment.

[0014] Figure 9A and Figure 9B It is a conceptual diagram showing the principle of obtaining distance measurement data.

[0015] Figure 10 It is a cross-sectional view of the imaging device according to the second embodiment.

[0016] Figure 11 It is a cross-sectional view of the imaging device according to the third embodiment.

[0017] Figure 12A and Figure 12B It is a graph showing the relationship between the temperature difference and the increase in the phase difference in the lens unit according to the third embodiment.

[0018] Figure 13 It is a functional block diagram of the in-vehicle system according to the embodiment.

[0019] Figure 14 It is a schematic diagram of the main part of the vehicle according to the present embodiment.

[0020] Figure 15 It is a flowchart showing an operation example of the in-vehicle system according to the present embodiment. Detailed Description of the Invention

[0021] As used herein, the term "unit" may refer to a software context, a hardware context, or a combination of software and hardware contexts. In a software context, the term "unit" refers to a functionality, an application, a software module, a function, a routine, an instruction set, or a program executable by a programmable processor (such as a microprocessor, a central processing unit (CPU), or a specially designed programmable device or controller). The memory contains instructions or programs, which, when executed by the CPU, cause the CPU to perform operations corresponding to the unit or function. In a hardware context, the term "unit" refers to a hardware element, a circuit, an assembly, a physical structure, a system, a module, or a subsystem. According to a specific embodiment, the term "unit" may include mechanical, optical, or electronic components, or any combination thereof. The term "unit" may include active (e.g., transistors) or passive (e.g., capacitors) components. The term "unit" may include a semiconductor device having a substrate and other material layers, where the substrate and other material layers have various conductivity concentrations. The term "unit" may include a CPU or a programmable processor that can execute a program stored in the memory to perform a specific function. The term "unit" may include logic elements (e.g., AND, OR) implemented by transistor circuits or any other switching circuits. In a combination of software and hardware contexts, the term "unit" or "circuit" refers to any combination of the software and hardware contexts as described above. Additionally, the terms "element", "assembly", "component", or "device" may also refer to a "circuit" integrated with or without packaging materials.

[0022] Reference is now made to the accompanying drawings, and a detailed description of various exemplary embodiments, features, and aspects in accordance with the present disclosure will be given. Corresponding elements in the respective figures will be denoted by the same reference numerals, and repeated descriptions thereof will be omitted.

[0023] Arrangement of a camera device in a vehicle

[0024] Figure 1A 、 Figure 1B and Figure 1C are diagrams showing the mounting positions of a camera device 1 on a vehicle 100 according to an embodiment of the present disclosure. In this embodiment, the camera device 1 is an in-vehicle camera mounted on the vehicle 100, but it may also be mounted on other types of moving bodies in addition to vehicles.

[0025] Figure 1A shows the camera device 1 located near the upper part of the front windshield (front glass) of the passenger compartment in the vehicle 100. In Figure 1A while driving, the camera device 1 monitors the area in front of the vehicle 100.

[0026] Figure 2FIG. is a diagram showing a method of mounting the imaging device 1 near the upper part of the front windshield of the vehicle 100. The imaging device 1 is attached to the vehicle 100 through the bracket 101. The bracket 101 is adhered to the upper side of the front windshield 102 in the passenger compartment to fix the imaging device 1. As a fixing method between the bracket 101 and the imaging device 1, a press-fit method or a method using fastening components (not shown) such as screws is used.

[0027] The imaging device 1 can also be installed at various positions on the vehicle 100. For example, as Figure 1B shown, the imaging device 1 can be mounted on the upper instrument panel, or as Figure 1C shown, located at the rear of the vehicle 100. In addition, the imaging device 1 can be installed at various positions to capture side or front / rear images.

[0028] Functions of the imaging device and vehicle control in the vehicle

[0029] Figure 3 FIG. is a block diagram of the imaging device 1. Light from the imaging object enters the lens unit 8. The light passes through the lens unit 8 and reaches the image sensor 9, and is received by the image sensor 9. The image sensor 9 outputs a signal corresponding to the received light. The image sensor 9 is a semiconductor image sensor element such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor. In addition, the image sensor 9 is capable of measuring the phase difference (displacement in the image signal) by dividing a single pixel into a plurality of photodiodes, and the imaging device 1 is capable of using the imaging plane phase difference method to obtain the distance to the obstacles around the vehicle. In the imaging plane phase difference method, the camera control integrated circuit (IC) (control device) 11 described later uses the displacement in the image signal generated when light enters a plurality of pixels through the lens unit 8 to estimate the distance.

[0030] The image sensor substrate 10 is equipped with a signal output circuit and a first temperature sensor (detector) 51. The signal output circuit outputs an image signal representing the image of the imaging object to the main body substrate 6 according to the signal output from the image sensor 9 based on the light incident on the lens unit 8. The first temperature sensor 51 acquires (detects) the temperature (second temperature) of the image sensor 9 and the lens unit 8 on the image sensor 9 side. The signal output circuit outputs the temperature obtained by the first temperature sensor 51 to the main body substrate 6.

[0031] In the present embodiment, the image sensor 9 and the image sensor substrate 10 serve as an imaging unit (image capturing unit) for capturing image data.

[0032] The main body substrate 6 is provided with a camera control IC 11 and a second temperature sensor 52. The camera control IC 11 processes the image signal output from the signal output circuit and generates distance measurement data. The second temperature sensor 52 acquires (detects) the temperature (first temperature) of the lens unit 8 on the imaging object side.

[0033] Based on the temperatures obtained by the first temperature sensor 51 and the second temperature sensor 52, the camera control IC 11 determines a correction value according to the correction value information stored in the storage device 16, and uses the correction value to correct the generated distance measurement data. The image signal processed by the camera control IC 11 can be output from the main body substrate 6 to the information processing device 103 in the vehicle 100.

[0034] In the present embodiment, the camera control IC 11 serves as an acquisition unit for acquiring information related to the temperature in the optical axis direction of the lens unit 8, and serves as a generation unit for generating distance measurement data from the image data according to the temperature-related information. In addition, the camera control IC 11 is mounted in the imaging device 1 in the present embodiment, but it may also be mounted on the vehicle 100.

[0035] First Embodiment

[0036] Figure 4A and Figure 4B are respectively an external view and an exploded view of the imaging device 1 according to the present embodiment. Figure 5 is a cross-sectional view of the imaging device 1. Figure 6 is a cross-sectional view of the image sensor unit 5.

[0037] As Figure 4A and Figure 4B shown, in the imaging device 1, the image sensor unit 5 and the main body substrate 6 are accommodated in a housing composed of an upper housing 2, a lower housing 3, and a rear housing 4.

[0038] The upper housing 2 is made of metal (such as aluminum, etc.) or resin, and forms the outer shell of the imaging device 1 on the X+ direction side. The upper housing 2 includes a front wall 21, an upper wall 22, a right side wall 23, a left side wall 24, a front side wall 25, and an upper side wall 26, and serves as a mounting portion for attaching to the vehicle 100. The front wall 21 is configured as a flat plate parallel to the image sensor 9 and is positioned to face the outside of the vehicle 100. The front wall 21 has a hole for inserting the image sensor unit 5. The upper wall 22 intersects the front wall 21. The upper wall 22 extends in the X+ direction from the end face on the Z- direction side of the front wall 21 and is configured as a flat plate. The upper wall 22 has a hole connected to the hole in the front wall 21 so as not to interfere with the image sensor unit 5. The upper cover 27 is installed to seal the hole in the upper wall 22. The upper cover 27 is made of metal or resin such as aluminum, etc., and is joined using fastening components (not shown) such as screws or snap fits after the image sensor unit 5 is inserted. A heat sink (not shown) can be arranged on the Z+ direction side of the upper wall 22. On the Z- direction side of the upper wall 22, the main body substrate 6 carrying a plurality of electronic components is attached.

[0039] The lower housing 3 is made of metal (such as aluminum, etc.) or resin, and forms the outer shell of the imaging device 1 on the Z- direction side. The lower housing 3 is installed to cover the upper housing 2 on the Z- direction side. The flat portion 31 is configured as a flat plate parallel to the upper wall 22.

[0040] The rear housing 4 is made of metal (such as aluminum, etc.) or resin, and forms the outer shell of the imaging device 1 on the X- direction side. The rear housing 4 includes a front wall 41 and a rear wall 42. The front wall 41 is positioned approximately parallel to the image sensor substrate 10 and is joined to the upper housing 2 using fastening components (not shown) such as screws. A heat sink (not shown) can be attached to the rear wall 42.

[0041] The image sensor unit 5 is composed of a lens unit 8, an image sensor 9, and an image sensor substrate 10. The image sensor unit 5 is inserted into the hole of the upper housing 2 and fixed with screws or an adhesive. On the X- direction side of the image sensor unit 5 (the X- direction side surface of the image sensor substrate 10), an image sensor heat conducting member 13 is laminated. The image sensor heat conducting member 13 is a thin rubber-like material made of silicone or a similar material. The image sensor heat conducting member 13 contacts the front wall 41 and conducts the heat from the image sensor 9. The heat received by the front wall 41 is dissipated from the rear wall 42.

[0042] The lens unit 8 is attached to the image sensor substrate 10 at an approximate right angle when fixed at its end by an adhesive or the like. In this embodiment, the lens unit 8 is directly attached to the image sensor substrate 10, but the present disclosure is not limited to this configuration. For example, the lens unit 8 can be indirectly attached to the image sensor substrate 10 through the upper housing 2 or other components.

[0043] The lens unit 8 includes a cylindrical lens barrel 81 made of a metal such as aluminum or a resin. Inside the lens barrel 81, a plurality of lenses 82 and spacers 83 are accommodated. The plurality of lenses 82 are made of a transparent material such as glass or resin. Among the plurality of lenses 82, the wide-angle lens 84 is arranged closest to the imaging object. The wide-angle lens 84 is held by a snap ring 85. The plurality of spacers 83 and the snap ring 85 are made of a metal such as aluminum or a resin. The snap ring 85 has a protrusion (protrusion member) 86 extending in a direction perpendicular to the optical axis direction from the end portion in the Z-direction. The position of the protrusion 86 is close to the second temperature sensor 52.

[0044] In Figure 6 this, a plurality of lenses 82 and spacers 83 are alternately accommodated in the same shape and quantity, but the present disclosure is not limited to this configuration. The plurality of lenses 82 and spacers 83 may have different shapes or quantities, and the order in which they are accommodated may be different from Figure 6 the order shown.

[0045] The image sensor 9 is mounted on the surface of the image sensor substrate 10 on the lens unit 8 side and is accommodated in the lens barrel 81. The image sensor 9 serves as the main heat source of the image sensor unit 5.

[0046] The image sensor substrate 10 is configured as a flat plate extending in the Y and Z directions and controls the image sensor 9. The first temperature sensor 51 is mounted on the surface of the image sensor substrate 10 opposite to the lens unit 8 side.

[0047] The main body substrate 6 is configured as a flat plate extending in the X and Y directions and is joined to the boss portion of the upper wall 22 using a fastening component such as a screw 7. The main body substrate 6 may be joined to the lower housing 3. The main body substrate 6 and the image sensor substrate 10 are electrically connected through a wiring 12. A plurality of electronic components are provided on both the Z+ side and the Z- side of the main body substrate 6. The plurality of electronic components mounted on the main body substrate 6 include the second temperature sensor 52 and the camera control IC 11.

[0048] The second temperature sensor 52 is mounted close to the protrusion 86 and acquires the temperature on the imaging object side of the lens unit 8. The upper side heat conducting member 15 is stacked on the second temperature sensor 52. The upper side heat conducting member 15 is made of a rubber-like material such as silicone or a similar material and is configured as a thin film. The second temperature sensor 52 is thermally connected to the protrusion 86 through the upper side heat conducting member 15, so that the temperature of the lens unit 8 can be acquired. However, if the second temperature sensor 52 is close enough to the protrusion 86 such that the temperature on the imaging object side of the lens unit 8 can still be acquired even with an air layer present, the upper side heat conducting member 15 may not be required.

[0049] The camera control IC 11 controls the entire imaging device 1 as a whole. The upper-side heat conductive member 15 is stacked in contact with at least some of the electronic components in the electronic component including the camera control IC 11. The upper-side heat conductive member 15 contacts the Z-direction side of the upper wall 22 and conducts heat from the electronic components. The upper wall 22 receives the heat conducted from the upper-side heat conductive member 15 on the Z-direction side and dissipates heat from the Z+ direction side.

[0050] The lower-side heat conductive member 14 is stacked on the Z-direction side of the main body substrate 6. The lower-side heat conductive member 14 is made of a rubber-like material such as silicone or a similar material and is configured as a thin film. The lower-side heat conductive member 14 contacts at least some of the electronic components. The lower-side heat conductive member 14 contacts the Z+ direction side of the lower housing 3 and conducts heat from the electronic components. The lower housing 3 receives the heat conducted from the lower-side heat conductive member 14 on the Z+ direction side and dissipates heat from the Z- direction side. In Figure 4A and Figure 4B , the main body substrate 6 and the upper wall 22 are located on the Z-direction side of the lens unit 8, but may also be located on the Z+ direction side of the lens unit 8.

[0051] The first temperature sensor 51 is provided on the surface of the image sensor substrate 10 opposite to the lens unit 8 side in the present embodiment, but the mounting position (second position) is not limited thereto as long as it can acquire the temperature on the image sensor 9 side of the lens unit 8. The first temperature sensor 51 is preferably mounted within 15% of the length in the optical axis direction of the lens unit 8 starting from the end on the imaging unit side of the lens unit 8.

[0052] In addition, the second temperature sensor 52 is provided at a position close to the protrusion 86 in the present embodiment, but the mounting position (first position) is not limited thereto as long as it can acquire the temperature on the imaging object side of the lens unit 8. The second temperature sensor 52 is preferably mounted within 15% of the length in the optical axis direction of the lens unit 8 starting from the end on the imaging object side of the lens unit 8.

[0053] Next, the effects of the imaging device 1 according to the present embodiment will be described. In the present embodiment, the camera control IC 11 can determine the correction value of the distance measurement data by estimating (acquiring) the temperature distribution of the lens unit 8 (information related to the temperature in the optical axis direction of the lens unit 8).

[0054] Figure 7 is a diagram showing the heat transfer path around the lens unit 8. The main heat sources are heat source A and heat source B. Heat source A is the heat generated by the image sensor 9 itself, and heat source B is caused by sunlight or similar factors. Heat source B may cause focus shift (defocus). Since the two heat sources are located at the two ends of the lens unit 8, the temperature distribution of the lens unit 8 can be estimated if the temperatures at the two ends can be acquired.

[0055] The heat source A transfers heat from the image sensor substrate 10 to the lens unit 8 and through the image sensor heat conducting member 13 to the rear housing 4. The first temperature sensor 51 and the lens unit 8 are directly mounted on the image sensor substrate 10, so they are thermally connected. Therefore, the first temperature sensor 51 can obtain the temperature on the image sensor 9 side of the lens unit 8.

[0056] The heat source B transfers heat from the wide-angle lens 84 and the snap ring 85 to the entire lens unit 8. The position of the second temperature sensor 52 is close to the protrusion 86 and is thermally connected through the upper side heat conducting member 15, so the temperature on the imaging object side of the lens unit 8 can be obtained.

[0057] Figure 8 is a graph showing the relationship between the temperature difference between the two ends of the lens unit 8 and the increase in the phase difference. In Figure 8 , the horizontal axis is the value (temperature difference) obtained by subtracting the temperature on the image sensor 9 side of the lens unit 8 from the temperature on the imaging object side, and the vertical axis is the increase in the phase difference. The phase difference, which is the offset of the image signal generated by the light entering the image sensor 9 through the lens unit 8, can be measured in pixels. Therefore, in Figure 8 , the increase in the phase difference is expressed as a value in pixels. As Figure 8 shown, when the heat source B does not exist, the temperature difference is -1 which is less than 0, so the increase in the phase difference is 0. When the temperature of the heat source B rises, the temperature difference increases, so the increase in the phase difference also increases.

[0058] Figure 9A and Figure 9B are conceptual diagrams showing the principle of obtaining distance measurement data. Figure 9A shows a state where only the heat source A exists and the focal position does not shift. Figure 9B shows a state where both the heat source A and B exist with the focal position shifted compared to Figure 9A . As described above, the image sensor 9 uses the light from the imaging object entering through the lens unit 8 and the imaging surface phase difference method to obtain the phase difference 91 which is the offset of the image signal for each pixel. By multiplying the phase difference 91 by the coefficient set for each imaging device 1, the defocus amount 92 can be obtained. Using lens design values (such as focal length and other related parameters) and the lens formula, the defocus amount 92 can be converted into the distance 93 to the imaging object.

[0059] In Figure 9B , due to the influence of the heat source B, the lens unit 8 relative to its state in Figure 9AThe position offset in it. The actual movement amount is about several micrometers. The movement of the lens unit 8 causes an increase in the phase difference 91 and the defocus amount 92. The distance 93 to the imaging object includes an error distance 94 from the true position of the imaging object. Therefore, by subtracting the increase in the phase difference calculated based on the temperature difference between the two ends of the lens unit 8 from the phase difference 91 in Figure 9B it, the phase difference 91 in Figure 9A it can be corrected. As a result, the distance 93 to the imaging object can also be corrected.

[0060] As described above, the configuration of this embodiment realizes the accurate acquisition of distance measurement data by identifying the temperature distribution of the lens unit 8 and easily reducing the influence of the focus position offset caused by the expansion and contraction of the lens barrel 81 and the lens.

[0061] Second Embodiment

[0062] Figure 10 FIG. 14 is a cross-sectional view of the imaging device 1A according to this embodiment. The imaging device 1A is different from the imaging device 1 of the first embodiment in that: the second temperature sensor 52 is mounted on the flexible substrate 17 connected to the main body substrate 6 and mounted on the upper wall 22. In this embodiment, only the configuration different from the first embodiment will be described, and the common configuration will be omitted.

[0063] The upper wall 22 is provided with a hole for passing through the flexible substrate 17. The flexible substrate 17 is attached to the upper wall 22 using a fastening component such as a screw, double-sided tape, or an adhesive. The second temperature sensor 52 is mounted close to the buckle 85. The second temperature sensor 52 can be connected to the buckle 85 by laminating a heat-conducting member.

[0064] In the imaging device 1A, the first temperature sensor 51 can acquire the temperature on the image sensor 9 side of the lens unit 8. The second temperature sensor 52 is mounted close to the buckle 85, so the temperature on the imaging object side of the lens unit 8 can be acquired. Based on the temperature difference obtained by subtracting the temperature acquired by the second temperature sensor 52 from the temperature acquired by the first temperature sensor 51, the increase in the phase difference is calculated (acquired). The distance measurement data can be corrected by subtracting the calculated increase in the phase difference from the phase difference on which the distance measurement data is based.

[0065] As described above, the configuration of this embodiment realizes the accurate acquisition of distance measurement data by identifying the temperature distribution of the lens unit 8 and easily reducing the influence of the focus position offset caused by the expansion and contraction of the lens barrel 81 and the lens.

[0066] Third Embodiment

[0067] Figure 11It is a cross-sectional view of the imaging device 1A according to the present embodiment. The imaging device 1B is different from the imaging device 1 of the first embodiment in that: the second temperature sensor 52 and the third temperature sensor 53 are mounted on the flexible substrate 17 connected to the main body substrate 6 and attached to the upper wall 22. Further, it is different in that the protrusion 87 is mounted on the outer shape of the lens barrel 81. In the present embodiment, only the configurations different from those of the first embodiment will be described, and the common configurations will be omitted.

[0068] The upper wall 22 is provided with a hole for passing through the flexible substrate 17. The flexible substrate 17 is attached to the upper wall 22 using a fastening component such as a screw, double-sided tape, or an adhesive. The second temperature sensor 52 is mounted close to the buckle 85. The second temperature sensor 52 can be connected to the buckle 85 by laminating a heat-conducting member. The third temperature sensor 53 is mounted close to the protrusion 87 provided on the outer shape of the lens barrel 81. The third temperature sensor 53 can be connected to the lens barrel 81 by laminating a heat-conducting member. The protrusion 87 is configured in a protruding shape to facilitate connection with the third temperature sensor 53, but the shape is not limited thereto. Further, the protrusion 87 can be made of a member separated from the lens barrel 81. In the present embodiment, the position of the protrusion 87 is near the center in the optical axis direction of the lens unit 8, but the mounting position is not limited thereto. The protrusion 87 should be mounted at a position effective for correcting the phase difference shift, such as near an adjustment lens for correcting the focus shift caused by temperature change or near an area where the protrusions and depressions on the lens barrel 81 are engaged with the housing. If there is a temperature change point due to the combination of the shape and material of the lens holding member, the protrusion 87 can be mounted to detect the temperature near the change point.

[0069] In the imaging device 1B, the first temperature sensor 51 can acquire the temperature on the image sensor 9 side of the lens unit 8. The second temperature sensor 52 is mounted close to the buckle 85, so the temperature on the imaging object side of the lens unit 8 can be acquired. The third temperature sensor 53 is close to the protrusion 87, so the temperature of the middle part of the lens unit 8 can be acquired.

[0070] Figure 12A and Figure 12B is a graph showing the relationship between the temperature difference in the lens unit 8 and the increase amount of the phase difference. In Figure 12A the horizontal axis is a value (temperature difference A) obtained by subtracting the temperature acquired by the third temperature sensor 53 from the temperature acquired by the first temperature sensor 51, and the vertical axis is the increase amount A of the phase difference. In Figure 12BIn this case, the horizontal axis is a value (temperature difference B) obtained by subtracting the temperature acquired by the second temperature sensor 52 from the temperature acquired by the third temperature sensor 53, and the vertical axis is the increase amount B of the phase difference. The distance measurement data can be corrected by subtracting the increase amount A of the phase difference or the increase amount B of the phase difference from the phase difference based on the distance measurement data. The temperature difference B can be a value obtained by subtracting the temperature acquired by the second temperature sensor 52 from the temperature acquired by the first temperature sensor 51.

[0071] As described above, the configuration of the present embodiment realizes the accurate acquisition of distance measurement data by identifying the temperature distribution of the lens unit 8 and easily reducing the influence of the focus position shift caused by the expansion and contraction of the lens barrel 81 and the lens.

[0072] Other

[0073] The materials of the upper housing 2, the lower housing 3, the rear housing 4, the lens barrel 81, the spacer 83, and the snap ring 85 are not particularly limited. Preferably, they are made of materials having high thermal conductivity (such as metals such as aluminum, copper, zinc, and iron, or alloys made by mixing multiple metals, etc.).

[0074] The materials of the image sensor heat conducting member 13, the lower side heat conducting member 14, and the upper side heat conducting member 15 are not particularly limited. For example, silicone or graphite can be used. In addition, a curable material made by mixing two-component materials or a non-curable material such as grease can also be used.

[0075] Vehicle-mounted system (in-vehicle system)

[0076] Figure 13 The configurations of the imaging device 1 according to the present embodiment and the vehicle-mounted (in-vehicle) system (control system, driving support device) 600 having the imaging device 1 are shown. The vehicle-mounted system 600 is a system that is held by a movable device (movable apparatus) such as an automobile (vehicle), and is configured to support the driving (operation) of the vehicle based on the image information around the vehicle acquired by the imaging device as a vehicle-mounted camera. Figure 14 The vehicle 700 as a movable device having the vehicle-mounted system 600 is shown. Figure 14 The case where the imaging range 50 of the imaging device 1 is set in front of the vehicle 700 is shown, but the imaging range 50 can be set behind or on the side of the vehicle 700, for example.

[0077] As Figure 13As shown, the vehicle-mounted system 600 includes a camera device 1, a vehicle information acquisition device 20, a control device (control unit, ECU: Electronic Control Unit) 30, and a warning device (warning unit) 40. The camera device 1 includes a camera unit 61, an image processing unit 62, a parallax calculator 63, a distance calculator 64, and a collision determination unit 65. The collision determination unit 65 may be included in the control device 30. The image processing unit 62, the parallax calculator 63, the distance calculator 64, and the collision determination unit 65 constitute a processing unit. The camera unit 61 includes an optical system and an image sensor according to any of the above embodiments.

[0078] Figure 15 is a flowchart showing an operation example of the vehicle-mounted system 600 according to an embodiment. The operation of the vehicle-mounted system 600 will be described below based on this flowchart.

[0079] First, in step S1, the camera unit 61 captures an image of an object such as an obstacle or a pedestrian around the vehicle, and acquires a plurality of image data (parallax image data).

[0080] In step S2, the vehicle information acquisition device 20 acquires vehicle information. The vehicle information includes the vehicle speed, yaw rate, steering angle, etc. of the vehicle.

[0081] In step S3, the image processing unit 62 performs image processing on the plurality of image data acquired by the camera unit 61. Specifically, image feature analysis is performed to analyze characteristics such as the amount and direction of edges and density values within the image data. Here, image feature analysis can be performed on each of the plurality of image data or only a part of the plurality of image data.

[0082] In step S4, the parallax calculator 63 calculates the parallax (image shift) information between the plurality of image data acquired by the camera unit 61. Known methods such as the SSDA method and the area correlation method can be used as the parallax information calculation method, and their detailed descriptions will be omitted. Steps S2, S3, and S4 can be performed in the above order, or can be performed in parallel with each other.

[0083] In step S5, the distance calculator 64 acquires (calculates) the distance information to the object photographed by the camera unit 61. The distance information can be calculated based on the parallax information calculated by the parallax calculator 63 and the internal and external parameters of the camera unit 61. The distance information here refers to information related to the position relative to the object such as the distance to the object, defocus amount, and image shift amount, and can directly represent the distance value of the object in the image or indirectly represent the information corresponding to the distance value.

[0084] In step S6, the collision determination unit 65 determines whether the distance to an object is within a preset distance range, using the vehicle information acquired by the vehicle information acquisition device 20 and the distance information calculated by the distance calculator 64. Thereby, it is possible to determine whether there is an object within the set distance around the vehicle, and to determine the possibility of a collision between the vehicle and the object. When an object exists within the set distance, the collision determination unit 65 determines that "there is a possibility of collision" (step S7), and when an object does not exist within the set distance, the collision determination unit 65 determines that "there is no possibility of collision" (step S8).

[0085] Next, when the collision determination unit 65 determines that there is a possibility of collision, the collision determination unit 65 notifies (transmits) the determination result to the control device 30 and the warning device 40. At this time, the control device 30 controls the vehicle based on the determination result of the collision determination unit 65 (step S6), and the warning device 40 warns the vehicle user (driver, passenger) based on the determination result of the collision determination unit 65 (step S7). The determination result can be notified to at least one of the control device 30 and the warning device 40.

[0086] The control device 30 can control the movement of the vehicle by outputting a control signal to a drive unit (engine, motor, etc.) of the vehicle. For example, the control device 30 performs controls such as applying brakes in the vehicle, releasing the accelerator, turning the steering wheel, generating control signals for generating braking forces in each wheel, and suppressing the output of the engine or motor. The warning device 40 warns the user, for example, by emitting a warning sound (warning alarm), displaying warning information on the screen of the car navigation system, or vibrating the seat belt or the steering wheel.

[0087] The in-vehicle system 600 according to the present embodiment can effectively perform object detection and avoid a collision between the vehicle and an object. Specifically, by applying the optical systems according to the respective embodiments in the foregoing embodiments to the in-vehicle system 600, it becomes possible to detect an object and perform collision determination at a wide viewing angle while reducing the overall size of the imaging device 1 and increasing its placement flexibility.

[0088] The distance information can be calculated by other methods. For example, it is assumed that a pupil division type image sensor having a plurality of pixel units arranged regularly in a two-dimensional array is applied to the image sensor in the imaging unit 61. In the pupil division type image sensor, one pixel unit includes a microlens and a plurality of photoelectric conversion units, receives a pair of light beams passing through different regions in the pupil of the optical system, and outputs a pair of image data from each photoelectric conversion unit.

[0089] Then, the image offset of each region is calculated by calculating the correlation between paired image data, and the image offset map data representing the distribution of the image offset is calculated by the distance calculator 64. Alternatively, the distance calculator 64 can also convert the image offset into a defocus amount and generate defocus map data representing the distribution of the defocus amount (distribution on the two-dimensional plane of the captured image). The distance calculator 64 can obtain distance map data of the distance to the object converted from the defocus amount.

[0090] The in-vehicle system 600 or the vehicle 700 may include a notification device (notification unit) configured to notify the fact to the in-vehicle system manufacturer and the dealer of the vehicle 700 in the event that the vehicle 700 collides with an obstacle. For example, the notification device may use a device for transmitting information related to the collision between the vehicle 700 and the obstacle (collision information) to a preset external notification destination via email or the like.

[0091] Therefore, the configuration of automatically notifying the collision information by the notification device can quickly respond to inspection and repair after a collision occurs. The notification destination of the collision information may be an insurance company, a medical institution, a police, or any destination set by the user. The notification device may be configured to not only notify the collision information to the notification destination, but also notify the failure information of each component and the consumption information of consumables. The presence or absence of a collision can be detected using the distance information obtained based on the output from the distance calculator 64 or other detectors (sensors).

[0092] In this embodiment, the in-vehicle system 600 is applied to driving support (collision damage reduction), but is not limited to this example. The in-vehicle system 600 can be used for cruise control (including an adaptive cruise control function) and autonomous driving. The in-vehicle system 600 can be applied not only to vehicles such as automobiles, but also to moving bodies such as ships, airplanes, and industrial robots. This embodiment is applicable not only to movable devices, but also to various devices such as intelligent transportation systems (ITS) and surveillance systems that use object recognition.

[0093] Other embodiments

[0094] Embodiments of the present invention can also be implemented by the following method, that is, a software (computer program product including computer programs / instructions) that executes the functions of the above embodiments is provided to a system or device via a network or various storage media, and a computer (central processing unit (CPU), microprocessing unit (MPU)) of the system or device reads and executes the computer programs / instructions.

[0095] Although the present disclosure describes exemplary embodiments, it should be understood that some embodiments are not limited to the disclosed embodiments. The scope of the appended claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

[0096] According to the present disclosure, a control device capable of obtaining high-precision distance measurement data can be provided.

Claims

1. A control device for controlling an imaging device, the imaging device including an imaging unit configured to capture image data and a lens unit having a plurality of lenses, the control device including: An acquisition unit configured to acquire information related to a temperature in a direction along an optical axis of the lens unit; And A generation unit configured to generate distance measurement data from the image data based on the information related to the temperature.

2. The control device according to claim 1, characterized in that, The generation unit generates the distance measurement data by correcting first distance measurement data generated from the image data based on the information related to the temperature.

3. The control device according to claim 2, wherein The generation unit determines a correction value based on the information related to the temperature, and generates the distance measurement data by correcting the first distance measurement data using the correction value.

4. The control device according to claim 3, wherein The information related to the temperature includes information related to a first temperature at a first position on an imaging object side of the lens unit and a second temperature at a second position on an imaging unit side of the lens unit, and The generation unit determines the correction value based on a difference between the first temperature and the second temperature.

5. The control device according to claim 3, wherein The information related to the temperature includes information related to the following temperatures: a first temperature at a first position on an imaging object side of the lens unit; a second temperature at a second position on an imaging unit side of the lens unit; and a third temperature at a third position between the first position and the second position, and The generation unit determines the correction value based on the following differences: a difference between the first temperature and the third temperature; And a difference between the second temperature and the first temperature or the third temperature.

6. The control device according to claim 5, characterized in that, The third position is any one of a center in a direction along the optical axis of the lens unit, near an adjustment lens for correcting focus shift, and near a temperature change point corresponding to a combination of the shape and material of a lens holding member.

7. The control device according to any one of claims 4 to 6, wherein The first position is included within 15% of a length in a direction along the optical axis of the lens unit starting from an end on the imaging object side of the lens unit, and The second position is included within 15% of a length in a direction along the optical axis of the lens unit starting from an end on the imaging unit side of the lens unit.

8. An imaging device, including: The control device according to any one of claims 1 to 7; An imaging unit configured to capture image data; And A lens unit including a plurality of lenses.

9. The imaging device according to claim 8, wherein The imaging unit is capable of measuring a phase difference by dividing a single pixel into a plurality of photodiodes, and The generation unit uses the phase difference to generate the distance measurement data.

10. The imaging device according to claim 8, further including: A main body substrate electrically connected to an image sensor substrate included in the imaging unit; A protruding member extending from the lens unit in a direction perpendicular to the optical axis direction; And A detector, which is disposed on the main body substrate closer to the protruding member and detects the temperature at the position on the imaging object side of the lens unit.

11. The imaging device according to any one of claims 8 to 10, further comprising: A main body substrate, which is electrically connected to the image sensor substrate included in the imaging unit; A flexible substrate, which is connected to the main body substrate; A housing, which is used to cover the main body substrate and is disposed between the main body substrate and the lens unit; And A detector, which is disposed on the housing side of the flexible substrate between the housing and the lens unit and detects the temperature at the position on the imaging object side of the lens unit.

12. A control system, comprising: The imaging device according to any one of claims 8 to 11; And A determination unit, which is configured to determine the possibility of collision with an object based on the distance information of the object acquired by the imaging device.

13. The control system according to claim 12, further comprising a control device, which is configured to output a control signal for generating a braking force to the drive unit of the mobile device when it is determined that there is a possibility of collision between the mobile device and an object.

14. The control system according to claim 12, further comprising a warning device, which is configured to warn the driver of the mobile device when it is determined that there is a possibility of collision between the mobile device and an object.

15. The control system according to any one of claims 12 to 14, further comprising a notification unit, which is configured to notify external parties of information related to the collision between the mobile device and an object.

16. A mobile device, comprising the imaging device according to any one of claims 8 to 11, Among them, The mobile device is configured to hold the imaging device and be capable of moving with the imaging device.

17. The mobile device according to claim 16, further comprising a determination unit, which is configured to determine the possibility of collision with an object based on the distance information of the object acquired by the imaging device.

18. The mobile device according to claim 17, further comprising a notification unit, which is configured to notify external parties of information related to the collision with an object.

19. A control method for controlling an imaging device, the imaging device comprising an imaging unit configured to capture image data and a lens unit having a plurality of lenses, the control method comprising the following steps: Obtaining information related to the temperature in the optical axis direction of the lens unit; And Generating distance measurement data from the image data based on the information related to the temperature.

20. A non-transitory computer-readable storage medium, which stores a program for causing a computer to execute the control method according to claim 19.

21. A computer program product, which includes a program for causing a computer to execute the control method according to claim 19.

Images