3D camera

Abstract

The purpose of this invention is to provide a stereo camera capable of identifying abnormal or malfunctioning parts without being affected by ambient temperature. The stereo camera of this invention is characterized by comprising: a first imaging element that captures images of the outside world and outputs first image data; a second imaging element that captures images of the outside world and outputs second image data; an image processing circuit that processes the first image data and the second image data to acquire external information; a first temperature sensor disposed near the first imaging element; a second temperature sensor disposed near the second imaging element; a third temperature sensor disposed near the image processing circuit; and a control microcomputer that determines an abnormality in the first imaging element, the second imaging element, or the image processing circuit based on the temperature difference measured by each temperature sensor.

Description

Technical Field

[0001] This invention relates to a stereo camera capable of identifying internal abnormalities. Background Technology

[0002] Vehicle-mounted devices, such as those in automobiles, require safety mechanisms to mitigate dangers arising from component malfunctions. One type of vehicle-mounted device is a vehicle camera that provides external information for driver assistance or autonomous driving to the vehicle control unit. A typical vehicle camera is equipped with an image sensor that captures images of the external environment, an image processing circuit that processes the captured image data to obtain external information, and a control microcomputer that transmits this information to the vehicle control unit. If a specific part of such a vehicle camera malfunctions and can no longer output normal external information, and the vehicle control unit receives this abnormal information and incorrectly controls the vehicle by performing actions such as steering, acceleration, or braking that do not correspond to the actual external conditions, a dangerous situation may occur.

[0003] Therefore, conventional vehicle-mounted cameras, such as those in Patent Document 1, have adopted the following approach: a temperature sensor is placed near the main components, and if the temperature measured by the temperature sensor exceeds a specified threshold, it is judged as an abnormality or malfunction, thereby cutting off the power supply to stop the function of the vehicle-mounted camera.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2001-88609 Summary of the Invention

[0007] The problem the invention aims to solve

[0008] However, in Patent Document 1, when the ambient temperature is low, even if an abnormal temperature rise is caused by a component failure, the abnormality or malfunction may not be detected if the temperature sensor's measured temperature is below the threshold. On the other hand, when the ambient temperature is high, even if no component failure has actually occurred, the abnormality or malfunction may be falsely detected if the temperature sensor's measured temperature exceeds the threshold.

[0009] Therefore, the object of the present invention is to provide a stereo camera that can identify parts that have malfunctioned or failed without being affected by ambient temperature.

[0010] Technical means to solve the problem

[0011] To solve the above problems, the stereo camera of the present invention comprises: a first imaging element that captures images of the outside world and outputs first image data; a second imaging element that captures images of the outside world and outputs second image data; an image processing circuit that processes the first image data and the second image data to acquire external information; a first temperature sensor disposed near the first imaging element; a second temperature sensor disposed near the second imaging element; a third temperature sensor disposed near the image processing circuit; and a control microcomputer that determines an abnormality of the first imaging element, the second imaging element, or the image processing circuit based on the temperature difference measured by each temperature sensor.

[0012] The effects of the invention

[0013] The stereo camera according to the present invention can identify parts that have malfunctioned or failed, regardless of ambient temperature. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the stereo camera in Example 1.

[0015] Figure 2A This is an example of the rise in internal temperature of a stereo camera under low ambient temperature conditions.

[0016] Figure 2B This is an example of the rise in internal temperature of a stereo camera under high ambient temperature conditions.

[0017] Figure 3 Example of anomaly detection threshold for temperature difference ΔT.

[0018] Figure 4 This is an example of a fault detection method that utilizes the temperature difference ΔT.

[0019] Figure 5 This is an example of Halt mode start / remove threshold based on temperature difference ΔT in Example 2.

[0020] Figure 6 This is an example of setting multiple temperature difference ΔT determination thresholds in Example 3.

[0021] Figure 7 This is an example of correcting the threshold based on past information of the temperature difference ΔT, as described in Example 4.

[0022] Figure 8 Example 5 illustrates how to predict future failures based on changes in temperature difference ΔT. Detailed Implementation

[0023] The embodiments of the stereo camera of the present invention will be described in detail with reference to the accompanying drawings. Furthermore, the present invention is not limited to the following embodiments, and various modifications or applications are also included within the scope of the technical concept of the present invention.

[0024] Example 1

[0025] First, use Figures 1 to 4 The stereo camera 1 of Embodiment 1 of the present invention will be described below. Furthermore, in this embodiment, a stereo camera 1 used for driving assistance or autonomous driving in automobiles will be used as an example for description; however, the stereo camera 1 may also be a stereo camera mounted in an autonomous mobile robot or industrial robot.

[0026] like Figure 1 As shown, the stereo camera 1 in this embodiment is equipped with a left camera element 2L, a right camera element 2R, an image processing circuit 3, a left temperature sensor 4L, a right temperature sensor 4R, a main temperature sensor 4M, a control microcomputer 5, a power supply circuit 6, and a storage unit 7.

[0027] The left and right camera elements 2L and 2R are CCD image sensors that capture images of the outside world and output image data. The image processing circuit 3 processes the image data to obtain information about the outside world in front of the vehicle (such as vehicles ahead, pedestrians, white lines, traffic lights, etc.).

[0028] The left temperature sensor 4L is located near the left camera element 2L, and measures the temperature T around the left temperature sensor 4L. L The right temperature sensor 4R is positioned near the right imaging element 2R to measure the temperature T around the right temperature sensor 4R. R The main temperature sensor 4M is located near the image processing circuit 3 to measure the temperature T around the image processing circuit 3. M .

[0029] The control microcomputer 5 sends the external information processed by the image processing circuit 3 to the vehicle control device 10, or sends the information based on the temperature T measured by each temperature sensor. L T R T M This is used to determine if there is any abnormality in stereo camera 1, or to identify any abnormal parts. Details of the processing here will be described later.

[0030] The power supply circuit 6 supplies power to the left camera element 2L, the right camera element 2R, the image processing circuit 3, etc., according to instructions from the control microcomputer 5, or temporarily stops supplying power to each part. Furthermore, the function of partially stopping the power supply will be referred to as "Halt" below.

[0031] The storage unit 7 contains EEPROMs and other storage devices that store past statuses of the stereo camera 1 as determined by the control microcomputer 5 and measured temperatures T from various temperature sensors.

[0032] The vehicle control unit 10 is a control device that controls the vehicle's steering system, drive system, and braking system based on external information sent from the stereo camera 1, thereby realizing driving assistance or automatic driving. It is usually called an ECU (Electronic Control Unit).

[0033] <Example of internal temperature rise in stereo camera 1>

[0034] Next, use Figure 2A and Figure 2B The example of the temperature rise inside the stereo camera 1 will be explained. Furthermore, the configuration of the heat-generating components around each temperature sensor is different. Therefore, if there is no abnormality or malfunction, the sequence of measured temperatures T of each temperature sensor will be fixed after a certain period of time since startup, maintaining a constant state, for example, always maintaining T. L >T R >T M The relationship.

[0035] Figure 2A This is an example of the internal temperature rise when the stereo camera 1 is started normally at an ambient temperature of low temperature T1. Figure 2B This is an example of the increase in internal temperature when the same stereo camera 1 is activated at an ambient temperature of high temperature T2. As is clear from the comparison of the two figures, when the ambient temperature increases, the temperature T... L (dashed line), T R (solid line), T M (The dashed line) rises while maintaining the relative relationship. This means that the temperature T is affected by the ambient temperature. L T R T M It will be difficult to detect abnormalities or malfunctions in the left camera element 2L, the right camera element 2R, and the image processing circuit 3.

[0036] Here, the temperature T L With temperature T R The difference is set as temperature difference ΔT LR Temperature T R With temperature T M The difference is set as temperature difference ΔT MR Temperature T L With temperature T M The difference is set as temperature difference ΔT MLIn this case, as is evident from the comparison of the two figures, the temperature differences ΔT remain approximately constant regardless of the ambient temperature. Therefore, in the temperature difference ΔT... LR ΔT MR ΔT ML When an anomaly occurs, it can be inferred that some kind of anomaly has occurred in stereo camera 1.

[0037] <Anomaly Detection Method Based on Control Microcomputer 5>

[0038] Next, use Figure 3 The method for determining anomalies in each temperature difference ΔT is explained. Figure 3 (a) shows the temperature difference ΔT LR The calculated value (white circle) and the upper and lower thresholds (black circles) used for anomaly detection. Figure 3 (b) shows the temperature difference ΔT MR The calculated value (white circle) and the upper and lower thresholds (black circles) used for anomaly detection. Figure 3 (c) shows the temperature difference ΔT ML The calculated values ​​(white circles) and the upper and lower thresholds (black circles) used for anomaly detection. Furthermore, each temperature difference ΔT is calculated by the control microcomputer 5 based on the measured temperature T of each temperature sensor, and each threshold is a threshold registered in the storage unit 7 by the designer and others considering the expected temperature difference of the measured temperature T of each temperature sensor.

[0039] The microcomputer 5 will calculate the temperature difference ΔT LR ΔT MR ΔT ML The system compares the data with upper and lower thresholds set for each individual. If the data falls within the normal range between the upper and lower thresholds, it is considered normal. If the data remains within the abnormal range above the upper threshold or below the lower threshold for a specified period of time, it is considered abnormal.

[0040] exist Figure 3 In the example, the temperature difference ΔT LR The calculated value (e.g., 1.3℃) and temperature difference ΔT MR The calculated value (e.g., 1.2℃) is within the normal range, so the control microcomputer 5 determines the temperature difference ΔT. LR and temperature difference ΔT MR Normal. On the other hand, the temperature difference ΔT ML The calculated value (e.g., 1.8℃) is within the abnormal region, so the temperature difference ΔT ML When the abnormal area stays for a specified time, the control microcomputer 5 determines the temperature difference ΔT. ML abnormal.

[0041] After determining whether there are any abnormalities in each temperature difference ΔT using the above methods, the control microcomputer 5 determines the fault location of the stereo camera 1 based on the combination of normal and abnormal conditions. Figure 4 This is an example of a fault detection method.

[0042] picture Figure 4 As shown in (a), under normal conditions with all temperature differences ΔT, the control microcomputer 5 determines that the stereo camera 1 is fault-free. On the other hand, like Figure 4 As shown in (e) to (g), in the case of any two abnormal temperature differences ΔT, the control microcomputer 5 determines the abnormal or faulty part of the stereo camera 1 based on the combination of abnormal temperature differences ΔT. Furthermore, in the case of any one abnormal temperature difference ΔT ( Figure 4 (b) to (d)) or all temperature differences ΔT are abnormal ( Figure 4 Under (h) conditions, although the occurrence of the abnormality or malfunction can be determined, its location cannot be determined.

[0043] When the control microcomputer 5 detects an abnormality or malfunction, it controls the power circuit 6 according to its condition, either by stopping the power supply to the abnormal part in Halt mode or by notifying the vehicle control device 10 of the abnormal condition.

[0044] For example, like Figure 4 As shown in (e) and (f), if it can be determined that one of the left camera element 2L or the right camera element 2R is normal and the other is abnormal, external information can continue to be acquired based on the image data of the normal camera element. Therefore, as long as the control microcomputer 5 notifies the vehicle control device 10 that the stereo camera 1 is being used as a monocular camera, the vehicle control device 10 can continue driving assistance or automatic driving based on recognizing a certain degree of degradation of external information.

[0045] The stereo camera of this embodiment, as described above, can identify parts that have malfunctioned or failed, regardless of ambient temperature. As a result, control corresponding to the nature of the malfunction or failure can continue within the stereo camera or vehicle control device.

[0046] Example 2

[0047] Next, use Figure 5 The stereo camera 1 of Embodiment 2 of the present invention will be described below. Furthermore, the commonalities with Embodiment 1 will be omitted from the description.

[0048] In Embodiment 1, the control microcomputer 5 will immediately stop the power supply to the identified abnormal part when it determines that the temperature difference ΔT is abnormal (Halt mode), and will immediately restore the power supply when it determines that the temperature difference ΔT is normal. However, when the temperature difference ΔT is near the threshold, the determination of the control microcomputer 5 will switch continuously in a short period of time, which may fail to maintain the continuity of processing in the stereo camera 1 or the vehicle control device 10.

[0049] Therefore, as Figure 5 As shown, in this embodiment, a hysteresis is set for the Halt start threshold and Halt release threshold to suppress fluctuations where Halt start / release occurs repeatedly. Therefore, the stereo camera 1 can switch to Halt mode only if the temperature difference ΔT exceeds the Halt start threshold for a certain period of time, and can deactivate Halt mode only if time has passed since the Halt mode state and the temperature difference ΔT has decreased to within the Halt release threshold. Furthermore, Figure 5 Only the temperature difference ΔT is shown in the example. LR For temperature difference ΔT MR or temperature difference ΔT ML It can also be used in the same way.

[0050] Example 3

[0051] Next, use Figure 6 The stereo camera 1 of Embodiment 3 of the present invention will be described below. Furthermore, commonalities with the above embodiments will be omitted from repeated descriptions.

[0052] In Example 1, when the temperature difference ΔT is within the abnormal range, it is always judged as abnormal or faulty, but the degree of abnormality or fault cannot be determined.

[0053] Therefore, like Figure 6 As shown, in this embodiment, multiple fault zones are set for each temperature difference ΔT to determine the degree of abnormality or fault. Therefore, if the temperature difference ΔT remains within the minor fault zone for a certain period, the control microcomputer 5 can notify the vehicle control device 10 of the occurrence of a minor fault. Conversely, if the temperature difference ΔT remains within the major fault zone for a certain period, the control microcomputer 5 can disconnect the power circuit 6 and notify the vehicle control device 10 of the occurrence of a major fault. Furthermore, Figure 6 Only the temperature difference ΔT is shown in the example. LR For temperature difference ΔT MR or temperature difference ΔT ML It can also be used in the same way.

[0054] Example 4

[0055] Next, use Figure 7The stereo camera 1 of Embodiment 4 of the present invention will be described below. Furthermore, commonalities with the above embodiments will be omitted from repeated descriptions.

[0056] In Embodiment 1, the designer of the stereo camera 1 must register an appropriate threshold in the storage unit 7, but may also consider that the threshold registered by the designer may be inappropriate, or that the appropriate threshold may change due to the deterioration of the stereo camera 1 over the years.

[0057] Therefore, in this embodiment, the thresholds can be corrected based on the actually measured temperature difference ΔT. For example, if the storage unit 7 in this embodiment stores multiple past data of temperature differences ΔT at times elapsed since the stereo camera was started, the control microcomputer 5 can correct the thresholds by narrowing the range compared to the initial threshold when the maximum and minimum values ​​of the past temperature differences ΔT are within the range of the initial threshold, thereby improving the sensitivity of anomaly detection. Furthermore, Figure 7 Only the temperature difference ΔT is shown in the example. LR For temperature difference ΔT MR or temperature difference ΔT ML It can also be used in the same way.

[0058] Example 5

[0059] Next, use Figure 8 The stereo camera 1 of Embodiment 5 of the present invention will be described below. Furthermore, commonalities with the above embodiments will be omitted from repeated descriptions.

[0060] In Example 1, the presence or absence of an anomaly is determined based on the current temperature difference ΔT. However, in this example, the timing of an anomaly can be predicted based on changes in the temperature difference ΔT from the past to the present. For instance, if the storage unit 7 in this example stores past data on the temperature difference ΔT over a certain period of time since the stereo camera was started, the control microcomputer 5 can predict future temperature differences ΔT based on changes in past temperature differences ΔT. Thus, even if the temperature difference ΔT at the current point in time is within the normal range, if a future anomaly or malfunction is predicted based on the prediction curve, the vehicle control device 10 can be notified in advance of the occurrence of the future anomaly or malfunction.

[0061] Symbol Explanation

[0062] 1… Stereo camera, 2L… Left camera element, 2R… Right camera element, 3… Image processing circuit, 4L… Left temperature sensor, 4R… Right temperature sensor, 4M… Main temperature sensor, 5… Control microcomputer, 6… Power supply circuit, 7… Storage unit, 10… Vehicle control device.

Claims

1. A stereoscopic camera, characterized in that, have: The first camera element captures images of the outside world and outputs first image data; The second camera element captures images of the outside world and outputs second image data; An image processing circuit processes the first image data and the second image data to obtain external information; A first temperature sensor is disposed near the first camera element; A second temperature sensor is located near the second camera element; A third temperature sensor is located near the image processing circuit. as well as Control microcomputer, The control microcomputer calculates the difference between the measured temperature of the first temperature sensor and the measured temperature of the second temperature sensor (i.e., the first temperature difference), the difference between the measured temperature of the second temperature sensor and the measured temperature of the third temperature sensor (i.e., the second temperature difference), and the difference between the measured temperature of the first temperature sensor and the measured temperature of the third temperature sensor (i.e., the third temperature difference). Determine whether there are any abnormalities in the first temperature difference, the second temperature difference, and the third temperature difference. Based on the combination of the abnormality determination results of the first temperature difference, the second temperature difference, and the third temperature difference, the abnormal part of the first camera element, the second camera element, or the image processing circuit is determined.

2. The stereoscopic camera according to claim 1, characterized in that, If the temperature difference is above or below the upper threshold and the state has lasted for a specified time, the control microcomputer determines that the temperature difference is abnormal.

3. The stereo camera according to claim 2, characterized in that, The upper threshold is composed of two values: the Halt start threshold and the Halt release threshold, which define the hysteresis region. The lower threshold is composed of two values: the Halt start threshold and the Halt release threshold, which define the hysteresis region.

4. The stereoscopic camera according to claim 2, characterized in that, The upper threshold is composed of two values: the first upper threshold, which represents the boundary between the normal region and the minor fault region, and the second upper threshold, which represents the boundary between the minor fault region and the major fault region. The lower threshold is composed of two values: the first lower threshold, which is the boundary between the normal region and the minor fault region, and the second lower threshold, which is the boundary between the minor fault region and the major fault region.

5. The stereo camera according to claim 1, characterized in that, If the control microcomputer determines that one camera element is normal and the other camera element is abnormal, The image processing circuit only processes image data from normal camera elements to obtain the external information.

6. The stereoscopic camera according to claim 2, characterized in that, It also includes a storage unit that stores multiple sets of past data on the temperature difference after a certain period of time since startup. The control microcomputer adjusts the upper threshold and the lower threshold based on the maximum and minimum values ​​of the past temperature difference data stored in the storage unit.

7. The stereoscopic camera according to claim 1, characterized in that, It also includes a storage unit that stores past data on the temperature difference over a certain period of time, starting from the time of startup. The control microcomputer predicts future temperature difference changes based on past temperature difference changes stored in the storage unit, thereby predicting the occurrence of anomalies.

Images