Image processing device, image processing method and storage medium

Abstract

The image processing apparatus (100) includes a saturation region detection unit (140), a dynamic range control unit (150), and a shutter control unit (170). The saturation region detection unit (140) acquires imaging data from an infrared imaging element that captures thermal images of the surroundings of a moving object. The dynamic range control unit (150) sets the dynamic range of the infrared imaging element to a first temperature range or at least a second temperature range where the upper limit temperature is higher than the first temperature range, based on the saturation region detection result. The shutter control unit (170) controls the opening and closing of the shutter to protect the infrared imaging element based on the saturation region detection result and the dynamic range setting.

Description

Technical Field

[0001] The invention relates to an image processing device, an image processing method and a storage medium. Background Art

[0002] Many object detection systems using infrared cameras that detect the heat of an object have been developed. In addition, it is expected that such object detection systems will improve the safety of, for example, automobiles. On the other hand, since infrared cameras have the characteristic of detecting the heat of an object, microbolometers used as imaging elements may malfunction when imaging sunlight. Therefore, a technology for suppressing the influence of sunlight is required.

[0003] For example, the far-infrared imaging device described in Patent Document 1 determines whether the amount of far-infrared rays detected by the far-infrared detection pixels is greater than a first amount, and closes the shutter if there is a far-infrared detection pixel that detects a far-infrared ray greater than the first amount. On the other hand, after closing the shutter, for each far-infrared detection pixel, it is determined whether the amount of far-infrared rays detected is less than a second amount, and the shutter is opened if the amount of far-infrared rays detected by all far-infrared detection pixels is less than the second amount, and the second amount is less than the first amount.

[0004] Prior Art Literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2009-005120. Summary of the invention

[0007] In order to improve the resolution of object detection, the microbolometer of the infrared camera is used with a narrow dynamic range. In this case, the detection signal will be saturated even in the temperature range where the microbolometer does not have abnormalities. Therefore, the infrared camera may close the shutter excessively to protect the microbolometer. However, the object detection function cannot be used while the shutter is closed, so a technology for appropriately closing the shutter when necessary is expected.

[0008] The present embodiment has been made to solve such a problem, and an object of the present embodiment is to provide an image processing device or the like that can appropriately detect sunlight and protect an infrared sensor from sunlight.

[0009] The image processing device involved in this embodiment includes a saturation area detection unit, a dynamic range control unit, and a shutter control unit. The saturation area detection unit obtains shooting data from an infrared shooting element, and detects the presence of a saturation area in the shooting data. The infrared shooting element shoots a thermal image of the periphery of a moving object. The dynamic range control unit sets the dynamic range of the infrared shooting element to a first temperature range or a second temperature range whose upper limit side temperature is higher than the first temperature range based on the detection result of the saturation area. The shutter control unit controls the opening and closing of a shutter for protecting the infrared shooting element based on the detection result of the saturation area and the setting of the dynamic range.

[0010] The image processing method involved in this embodiment includes a saturation area detection step, a dynamic range control step, and a shutter control step. The saturation area detection step acquires shooting data from an infrared shooting element that shoots a thermal image of the periphery of a moving object, and detects the presence of a saturation area in the shooting data. The dynamic range control step sets the dynamic range of the infrared shooting element to a first temperature range or a second temperature range whose upper limit side temperature is higher than the first temperature range based on the detection result of the saturation area. The shutter control step controls the opening and closing of a shutter for protecting the infrared shooting element based on the detection result of the saturation area and the setting of the dynamic range.

[0011] The program stored in the computer-readable storage medium involved in this embodiment enables the computer to execute the following image processing method. The above-mentioned image processing method includes a saturation area detection step, a dynamic range control step, and a shutter control step. The saturation area detection step acquires shooting data from an infrared shooting element, and detects the presence of a saturation area in the shooting data. The infrared shooting element shoots a thermal image of the periphery of a moving body. The dynamic range control step sets the dynamic range of the infrared shooting element to a first temperature range or a second temperature range whose upper limit side temperature is higher than the first temperature range based on the detection result of the saturation area. The shutter control step controls the opening and closing of a shutter for protecting the infrared shooting element based on the detection result of the saturation area and the setting of the dynamic range.

[0012] According to the present embodiment, it is possible to provide an image processing device or the like which can appropriately detect sunlight and protect an infrared sensor from sunlight. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a block diagram of the image processing device involved in Embodiment 1.

[0014] Figure 2 This is a flowchart of the image processing device involved in Embodiment 1.

[0015] Figure 3 This is a diagram showing a first example of an image generated by the image processing device according to the first embodiment.

[0016] Figure 4 This is a diagram showing a second example of an image generated by the image processing device according to the first embodiment.

[0017] Figure 5 This is a diagram showing a third example of an image generated by the image processing device according to the first embodiment.

[0018] Figure 6 This is a diagram showing a fourth example of an image generated by the image processing device according to the first embodiment.

[0019] Figure 7 This is a block diagram of an image processing device according to the second embodiment.

[0020] Figure 8 This is a flowchart of the image processing device involved in embodiment 2.

[0021] Fig. 9 This is a diagram showing an example of an image generated by the image processing device according to the second embodiment. DETAILED DESCRIPTION

[0022] Hereinafter, the present invention will be described by the embodiments of the invention, but the invention involved in the claims is not limited to the following embodiments. In addition, all the structures described in the embodiments are not necessarily necessary as means for solving the problem. In order to clarify the description, the following description and the accompanying drawings are appropriately omitted and simplified. In addition, in each figure, the same elements are marked with the same symbols, and repeated descriptions are omitted as needed.

[0023] <Implementation Method 1>

[0024] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Figure 1 This is a block diagram of the image processing device involved in Embodiment 1. Figure 1 The image processing device 100 shown is a component of the thermal image display system 10 that captures a thermal image of the surrounding area of ​​the vehicle and displays it on a monitor. The thermal image display system 10 includes the image processing device 100, a microbolometer 900, a shutter 901, and a monitor 902 as main components.

[0025] The image processing device 100 is described below. The image processing device 100 has, for example, a CPU (Central Processing Unit), an MCU (Micro Controller Unit) and other computing devices. In addition to the computing devices, the image processing device 100 also has at least a non-volatile or volatile memory such as a flash memory or a DRAM (Dynamic Random Access Memory), and a control substrate composed of other circuits. The image processing device 100 incorporates a program in these computing devices, etc., and realizes the functions shown below through a combination of software or hardware.

[0026] The image processing device 100 acquires the captured data of the thermal image captured by the micro-bolometer 900, and performs a preset process on the acquired captured data. The image processing device 100 outputs the captured data subjected to the preset process to the monitor 902. As a main structure, the image processing device 100 includes a defective pixel correction unit 110, a pixel non-uniformity correction unit 120, an AGC processing unit 130 (AGC = Auto-Gain-Control, automatic gain control), a saturation area detection unit 140, a dynamic range control unit 150, a pixel repair determination unit 160, and a shutter control unit 170. The image processing device 100 may include a recognition processing unit for recognizing a person or the like based on the thermal image captured by the micro-bolometer 900.

[0027] The defective pixel correction unit 110 receives the captured data from the micro-bolometer 900 and corrects the data of the defective pixel included in the captured data. The defective pixel correction unit 110 detects the coordinates of the defective pixel, for example, in a state where the shutter is closed, and stores the detected coordinates of the defective pixel. In addition, the defective pixel correction unit 110 also corrects the stored data of the defective pixel by generating interpolation data from the data of adjacent pixels. The defective pixel correction unit 110 provides the corrected captured data to the pixel non-uniformity correction unit 120 and the saturation area detection unit 140, respectively.

[0028] The pixel non-uniformity correction unit 120 corrects the non-uniformity state generated in each pixel of the captured data received from the defective pixel correction unit 110. The non-uniformity state of each pixel is generated due to the deviation of the characteristics of each pixel. The pixel non-uniformity correction unit 120 adjusts the gain and offset of each pixel using a method called NUC (Non-Uniformity Correction) to correct the captured data. The pixel non-uniformity correction unit 120 provides the corrected captured data to the AGC processing unit 130 and the pixel restoration determination unit 160 respectively.

[0029] The AGC processing unit 130 receives the captured data from the pixel non-uniformity correction unit 120 and adjusts the contrast of the captured data. By adjusting the contrast of the captured data, the AGC processing unit 130 generates an image that is easy for the user to see when displaying the thermal image on the monitor 902. In addition, by adjusting the contrast of the captured data, the AGC processing unit 130 can output a thermal image suitable for recognition processing when performing object recognition processing on the thermal image. For the captured data, the AGC processing unit 130 uses a histogram equalization method such as contrast-limited adaptive histogram equalization.

[0030] The saturated region detection unit 140 receives the captured data from the defective pixel correction unit 110, and detects a saturated region from the received captured data. The saturated region is a region where saturated pixels or substantially saturated pixels exist. Saturated pixels refer to pixels whose pixel data is in a state of an upper limit value. For example, when the brightness level of each pixel of the captured data is represented by 8 bits from zero to 255, the pixel whose coordinates have a brightness level of 255 is called a saturated pixel.

[0031] The saturated region detection unit 140 determines that a saturated region exists when, for example, it is detected that four or more adjacent pixels are saturated pixels. Alternatively, the saturated region detection unit 140 determines that a saturated region exists when, for example, nine or more adjacent pixels are 98% or more of the upper limit of the brightness value. When the saturated region detection unit 140 detects that a saturated region exists in the received captured data, it provides a signal indicating the detection result of the saturated region to the dynamic range control unit 150.

[0032] In addition, the saturation region detection unit 140 obtains information related to the setting of the dynamic range from the dynamic range control unit 150. The information related to the setting of the dynamic range includes: information indicating whether the setting of the dynamic range set in the microbolometer 900 is the first temperature range or the second temperature range. The first temperature range is, for example, a range set in such a way that the range of the temperature detected by the microbolometer is 10 degrees to 50 degrees. The second temperature range is a range in which the upper limit value is set to be higher than the first temperature range, for example, a range set in such a way that the range of the temperature detected by the microbolometer is 10 degrees to 200 degrees. When the dynamic range is set to the second temperature range, the saturation region detection unit 140 provides a signal for instructing the shutter to be closed when the above-mentioned saturation region is detected in the acquired shooting data.

[0033] The dynamic range control unit 150 cooperates with the saturation region detection unit 140 to set the dynamic range of the microbolometer to a first temperature range or a second temperature range whose upper limit temperature is higher than the first temperature range according to the detection result of the saturation region. The dynamic range control unit 150 is connected to the saturation region detection unit 140 and receives a signal indicating that a saturation region has been detected. In addition, the dynamic range control unit 150 is connected to the microbolometer 900 and provides a signal for indicating the setting of the dynamic range. In addition, the dynamic range in this embodiment can be renamed as a "scene dynamic range".

[0034] More specifically, for example, when the dynamic range is set to the first temperature range, the dynamic range control unit 150 changes the setting of the dynamic range from the first temperature range to the second temperature range when the saturated region detection unit 140 detects the saturated region. Also, for example, when the dynamic range is set to the second temperature range, the dynamic range control unit 150 changes the setting of the dynamic range from the second temperature range to the first temperature range when the saturated region detection unit 140 does not detect the saturated region.

[0035] In addition, when the dynamic range control unit 150 performs the process of setting the dynamic range to the first temperature range and the second temperature range, the dynamic range control unit 150, for example, changes the setting of the integration time of the microbolometer 900. By changing the setting of the integration time, the microbolometer 900 changes the exposure time. If the integration time becomes relatively shorter, the dynamic range of the signal output by the microbolometer 900 becomes relatively wider. That is, when the dynamic range is changed from the first temperature range to the second temperature range, the dynamic range control unit 150 sends an instruction to the microbolometer 900 to shorten the integration time.

[0036] In addition, the instruction of the dynamic range may be setting of the gain instead of the integration time. In the case of setting the gain, when the dynamic range is changed from the first temperature range to the second temperature range, the dynamic range control unit 150 sends an instruction to the microbolometer 900 to reduce the gain.

[0037] The pixel restoration determination unit 160 receives information on the brightness value of the captured data from the pixel unevenness correction unit 120, and detects the presence or absence of an abnormal region from the received information. The abnormal region refers to a region containing pixel data in an abnormal state.

[0038] Regarding the detection of the presence or absence of an abnormal area, for example, when the shutter is closed, that is, when photographing a thermally uniform surface, pixel data is compared with a frame average value, and pixels having a value different from the frame average value by a predetermined value or more are detected as abnormal pixels. An area with multiple abnormal pixels is set as an abnormal area.

[0039] Here, the situation where the pixel data becomes abnormal is described. When the micro-bolometer 900 captures the sun, the brightness value of the pixel in the area where the sun is captured reaches the upper limit and becomes saturated. Usually, the upper limit of the dynamic range of the micro-bolometer 900 is designed to be 200 degrees, for example. In this case, even if the dynamic range is set to 200 degrees as the upper limit, the pixel capturing the sun will be saturated. At this time, the imaging element that captured the sun becomes an abnormal state in which it cannot output a signal correctly. In addition, it is known that after the imaging element that has become abnormal is protected from sunlight by closing the shutter, it also takes time to return to a normal state.

[0040] Therefore, the pixel restoration determination unit 160 detects the presence or absence of an abnormal area with respect to the photographic data in the shutter-closed state, and provides the shutter control unit 170 with information on the presence or absence of the abnormal area.

[0041] The shutter control unit 170 controls the opening and closing of the shutter for protecting the infrared imaging element based on the detection result of the saturated area and the setting of the dynamic range. For example, when the dynamic range is set to the second temperature range and the saturated area detection unit detects the saturated area, the shutter control unit 170 closes the shutter. More specifically, the shutter control unit 170 is connected to the saturated area detection unit 140 and receives an instruction to close the shutter 901 from the saturated area detection unit 140. When receiving an instruction to close the shutter 901 from the saturated area detection unit 140, the shutter control unit 170 closes the shutter 901 according to the instruction.

[0042] In addition, the shutter control unit 170 is connected to the pixel repair determination unit 160, and receives information related to whether there is an abnormal area in the shooting data in the state where the shutter 901 is closed. When the shutter control unit 170 receives information indicating that there is an abnormal area in the shooting data in the state where the shutter 901 is closed, the shutter control unit 170 maintains the closed state of the shutter 901. On the other hand, when the shutter control unit 170 receives information indicating that there is no abnormal area in the shooting data in the state where the shutter 901 is closed, it performs a process of opening the shutter 901 from the closed state. The shutter control unit 170 also performs control to periodically close the shutter 901 in order to periodically perform temperature calibration of the micro-bolometer 900.

[0043] The image processing device 100 has been described above. Next, each configuration connected to the image processing device 100 will be briefly described.

[0044] The micro-bolometer 900 is an embodiment of an infrared imaging element. The micro-bolometer 900 is composed of infrared detection elements arranged in a matrix, and the infrared detection elements detect far infrared rays. In addition, the micro-bolometer 900 performs photoelectric conversion on the detected far infrared rays to generate imaging data, and provides the generated imaging data to the defective pixel correction unit 110 of the image processing device 100.

[0045] The micro-bolometer 900 is installed on the moving object to capture the thermal image of the surrounding of the moving object. When the moving object is a car, it is installed facing the front of the car to capture the moving direction of the car, but it can also be installed facing other directions.

[0046] The shutter 901 allows or inhibits external light from entering the microbolometer 900. The shutter 901 is controlled by the shutter control unit 170. When the shutter 901 is open, the shutter 901 allows external light to pass through and the microbolometer 900 receives light. When the shutter 901 is closed, the shutter 901 blocks external light and protects the microbolometer 900 from external light. In addition, the shutter 901 has a function of performing temperature correction of the microbolometer 900.

[0047] The monitor 902 is a display device that is provided in a manner capable of presenting information to the user, and includes, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. The monitor 902 is connected to the AGC processing unit 130 of the image processing device 100, receives the captured data from the AGC processing unit 130, and displays the received captured data. In addition to the captured data, an image representing a recognized person may also be displayed on the monitor 902.

[0048] Next, the processing performed by the image processing device 100 according to the first embodiment will be described. Figure 2 This is a flowchart of the image processing device involved in Embodiment 1. Figure 2 The flowchart shown in FIG. 1 starts when the image processing device 100 receives the captured data from the micro-bolometer 900. Figure 2 During the processing shown, the dynamic range of the micro-bolometer 900 is set to a first temperature range (eg, 10 degrees to 50 degrees). Figure 2 The flowchart shown in the figure mainly describes the processing of the saturation area detection unit 140, the dynamic range control unit 150, the pixel restoration judgment unit 160 and the shutter control unit 170, and the processing of other structures is appropriately omitted. Figure 1 As described.

[0049] First, when the image processing device 100 receives the imaging data from the microbolometer 900, the saturated region detection unit 140 starts detecting the saturated region (step S10). Next, the image processing device 100 determines whether a saturated region is detected in the imaging data (step S11).

[0050] If it is determined that a saturated area is not detected in the captured data (No in step S11), the image processing apparatus 100 repeats step S11. That is, the image processing apparatus 100 sequentially detects a saturated area on the acquired captured data.

[0051] When determining that a saturated area is detected in the captured data (YES in step S11 ), the image processing apparatus 100 changes the dynamic range from the first temperature range to the second temperature range (for example, from 10 degrees to 200 degrees) (step S12 ).

[0052] Next, the image processing device 100 determines whether a saturated region is detected for the captured data generated after the dynamic range is changed (step S13). The reason for performing such processing is that even if a saturated region is detected within the first temperature range, the detected saturated region may not be caused by sunlight. That is, by detecting a saturated region for the captured data whose dynamic range is expanded from the first temperature range to the second temperature range, the image processing device 100 determines whether the saturated region detected within the first temperature range is caused by sunlight.

[0053] When it is determined that a saturated area is not detected for the captured data generated after the dynamic range is changed to the second temperature range (No in step S13), the saturated area detected in the first temperature range is not caused by sunlight. Therefore, the image processing device 100 returns the setting of the dynamic range from the second temperature range to the first temperature range (step S18), and returns to step S11 again.

[0054] On the other hand, when it is determined that a saturated region is detected for the captured data generated after the dynamic range is changed to the second temperature range (Yes in step S13 ), the saturated region is caused by sunlight, and the image processing apparatus 100 closes the shutter 901 (step S14 ).

[0055] Next, the image processing device 100 returns the dynamic range from the second temperature range to the first temperature range again (step S15 ). Furthermore, the image processing device 100 determines whether an abnormal area is detected for the captured data generated with the shutter 901 closed (step S16 ).

[0056] When it is determined that an abnormal area is detected in the captured data generated in the state where the shutter 901 is closed ("Yes" in step S16), the image processing device 100 repeats step S16. That is, the image processing device 100 sequentially acquires the captured data in the state where the shutter 901 is closed, and determines whether an abnormal area is detected for the acquired captured data.

[0057] On the other hand, when it is determined that an abnormal area is not detected in the shooting data generated in the state where the shutter 901 is closed ("No" in step S16), the image processing device 100 opens the shutter 901 (step S17). Here, the image processing device 100 opens the shutter 901 because the abnormal state of the shooting element that has become abnormal due to sunlight is eliminated. After opening the shutter 901, the image processing device 100 repeats the processing from step S11 again. In addition, the order of step S15 and step S16 may be reversed.

[0058] Next, the difference in the setting of the dynamic range will be described with reference to an example of a thermal image. Figure 3 This is a diagram showing a first example of an image generated by the image processing device according to the first embodiment. Figure 3 This is a thermal image A11 generated by the image processing device 100 . Figure 3 The thermal image A11 shown is an example of taking a picture by setting the dynamic range of the micro-bolometer 900 to the first temperature range. The thermal image A11 is an image taken in the traveling direction of the vehicle equipped with the thermal image display system 10. The thermal image A11 includes another vehicle D11. In addition, the other vehicle D11 includes a muffler D12.

[0059] The thermal image A11 detects electromagnetic waves emitted by an object, so the portion with high heat is displayed brightly. Therefore, in the thermal image A11, the other vehicle D11 is displayed relatively brightly. In addition, the muffler D12 with a particularly high temperature in the other vehicle D11 is displayed in white. The brightness of the muffler D12 displayed in white in the thermal image A11 is in a saturated state.

[0060] Next, refer to Figure 4 , an image when the dynamic range is in the second temperature range is described. Figure 4 This is a diagram showing a second example of an image generated by the image processing device according to the first embodiment.

[0061] Figure 4 The thermal image A12 shown is an example of being captured by setting the dynamic range of the microbolometer 900 to a second temperature range wider than the first temperature range.

[0062] The thermal image A12 has a smaller overall bright range than the thermal image A11. The color tone of the other vehicle D11 is also displayed darker, and the muffler D12 is not in a saturated state.

[0063] Thus, the area that was saturated when shooting in the first temperature range is no longer saturated by changing the dynamic range to the second temperature range. Therefore, the image processing device 100 can return the dynamic range to the first temperature range without closing the shutter 901 and continue shooting.

[0064] Next, Figure 5 Provide explanation. Figure 5 This is a diagram showing a third example of an image generated by the image processing device according to the first embodiment. Figure 5 The thermal image A13 shown is an example of a shot taken by setting the dynamic range of the micro-bolometer 900 to the first temperature range. In addition to other vehicles D11, the thermal image A13 also includes the sun D13 in the upper left corner of the picture. Figure 5 As shown, the temperature of the sun D13 is high, so it is displayed in white. That is, the brightness of the portion displaying the sun D13 is in a saturated state.

[0065] Next, Figure 6 Provide explanation. Figure 6 This is a diagram showing a fourth example of an image generated by the image processing device according to the first embodiment. Figure 6 The thermal image A14 shown is an example of being captured by setting the dynamic range of the microbolometer 900 to the second temperature range.

[0066] The thermal image A14 has a smaller overall bright range than the thermal image A13. The color tone of the other vehicle D11 is also displayed darker, and the muffler D12 is not in a saturated state. On the other hand, the sun D13 has a slightly reduced saturated area, but the center portion is saturated.

[0067] Thus, when capturing the dynamic range in the first temperature range, sunlight in an area where the brightness is saturated will also be saturated even if the dynamic range is changed to the second temperature range. Therefore, the image processing device 100 appropriately detects the presence of sunlight and closes the shutter to protect the imaging element.

[0068] The first embodiment is described above. The image processing device 100 is not limited to the above-mentioned device. For example, in addition to the above-mentioned structure, the image processing device 100 may also include at least one of the micro-bolometer 900, the shutter 901, and the monitor 902. Figure 1Each structure described in the above description may be independently provided or may be constituted as a plurality of structures integrated into one. Figure 1 A part of the structure described in the above may also be set at a remote location and the above structure may be implemented through wireless communication.

[0069] As described above, when the image processing device 100 according to Embodiment 1 detects a saturated state of a pixel in a state where the dynamic range is in the first temperature range, the image processing device 100 can appropriately detect sunlight by changing the dynamic range to a second temperature range that is larger than the first temperature range. Therefore, according to Embodiment 1, it is possible to provide an image processing device or the like that can appropriately detect sunlight and protect an infrared sensor from sunlight.

[0070] <Implementation Method 2>

[0071] Next, embodiment 2 will be described. Figure 7 This is a block diagram of an image processing device according to the second embodiment. Figure 7 The thermal image display system 10 according to Embodiment 2 is shown. The thermal image display system 10 includes an image processing device 200 instead of the image processing device 100. The image processing device 200 is different from the image processing device 100 in that it includes a detection range setting unit 210.

[0072] The defective pixel correction section 110 receives the imaging data from the microbolometer 900 , and supplies the corrected imaging data to the pixel unevenness correction section 120 and the detection range setting section 210 .

[0073] The detection range setting unit 210 sets the range (detection range) in which the saturation area detection unit 140 performs saturation area detection. Specifically, for example, the detection range setting unit 210 receives the shooting data from the defective pixel correction unit 110, and intercepts the detection range as a set part of the received shooting data. The detection range may be a predetermined one, or may be changed according to a specified condition. For example, the detection range setting unit 210 may also set the detection range using at least one of the time, weather, vehicle posture, and vehicle travel direction as a parameter. When intercepting the detection range, the detection range setting unit 210 provides the intercepted shooting data to the saturation area detection unit 140. Therefore, the saturation area detection unit 140 detects the saturation area from the received shooting data in the same manner as in Embodiment 1.

[0074] Next, the processing performed by the image processing device 200 according to the second embodiment will be described. Figure 8 This is a flowchart of the image processing device involved in embodiment 2.

[0075] First, the image processing device 200 sets a detection range (step S20 ). After setting the detection range, the image processing device 200 then starts detecting a saturated region in the detection range (step S10 ).

[0076] Next, the image processing device 200 determines whether a saturated area is detected within the detection range of the captured data (step S21). If it is not determined that a saturated area is detected within the detection range ("No" in step S21), the image processing device 200 repeats step S21. That is, the image processing device 100 sequentially detects a saturated area for the detection range on the acquired captured data. If it is determined that a saturated area is detected within the detection range ("Yes" in step S21), the image processing device 100 changes the dynamic range from the first temperature range to the second temperature range (for example, from 10 degrees to 200 degrees) (step S12).

[0077] Next, the image processing device 100 determines whether a saturated area is detected for the detection range of the captured data generated after the dynamic range is changed (step S23). If it is determined that a saturated area is not detected for the detection range of the captured data generated after the dynamic range is changed to the second temperature range ("No" in step S23), the saturated area detected in the first temperature range is not caused by sunlight. Therefore, the image processing device 200 returns the setting of the dynamic range from the second temperature range to the first temperature range (step S18), and returns to step S21 again.

[0078] On the other hand, when it is determined that a saturated area is detected for the captured data generated after the dynamic range is changed to the second temperature range ("Yes" in step S23), the saturated area is caused by sunlight. Therefore, the image processing device 100 closes the shutter 901 (step S14). The processing from step S14 to step S17 is the same as the processing described in embodiment 1. Therefore, the description is omitted here.

[0079] Through the above-described processing, the image processing device 200 according to Embodiment 2 detects the saturated area for the set detection range. Therefore, the image processing device 200 can perform the processing at a high speed. In addition, the processing of step S20 can also be performed before the image processing device 200 obtains the captured data from the microbolometer 900.

[0080] Next, the setting of the detection range is further described with reference to an example of a thermal image. Fig. 9 This is a diagram showing an example of an image detected by the image processing device according to the second embodiment. Fig. 9 It is a thermal image A21 generated by the image processing device 200. Fig. 9The thermal image A21 shown is an example of a photo taken by setting the dynamic range of the micro-bolometer 900 to the first temperature range. The thermal image A21 includes other vehicles D11, a muffler D12, and the sun D13. Fig. 9 As shown, since the temperature of the muffler D12 and the sun D13 is relatively high, they are displayed in white. That is, the brightness of the portion displaying the muffler D12 and the sun D13 is in a saturated state.

[0081] Fig. 9 The detection range R21 is shown. Fig. 9 In FIG. 1 , the rectangle indicated by the one-dot chain line indicates the boundary line defining the detection range R21. That is, the detection range R21 is the range inside the rectangle indicated by the one-dot chain line. Fig. 9 The detection range R21 shown is the range of about 40% of the upper side in the vertical direction of the thermal image A21. That is, the image processing device 200 performs saturation area detection on the detection range R21 cut out from the shooting data. Therefore, the image processing device 200 does not detect the silencer D12 as a saturation area, but performs the above-mentioned processing on the area where the sun D13 is photographed.

[0082] In addition, in the above-mentioned detection range setting, the detection range setting unit 210 can set a state without a detection range (i.e., a state in which the detection range is not intercepted). In addition, in the above-mentioned detection range setting, the detection range setting unit 210 can also set the detection range using at least one of the time, weather, vehicle posture, and vehicle travel direction as a parameter.

[0083] For example, the detection range setting unit 210 may also set a state without a detection range when the sun is not out. In addition, the detection range setting unit 210 may also obtain weather information and set a state without a detection range in the case of rainy days. In addition, the detection range setting unit 210 may also expand the detection range to the lower side of the thermal image A21 in the morning and evening when the position of the sun is relatively low, and narrow the detection range to the upper side of the thermal image A21 during the day when the position of the sun is relatively high. In addition, the detection range setting unit 210 may also detect the posture of the vehicle, expand the detection range when climbing a slope, and narrow the detection range when going downhill. In addition, the detection range setting unit 210 may also detect time information and the direction of travel of the vehicle, for example, when driving toward the east in the morning, the detection range is set to be larger, and when driving toward the west in the morning, the detection range is set to be non-detectable.

[0084] As described above, when the image processing device 200 according to Embodiment 2 detects the saturation state of the pixel in the state where the dynamic range is in the first temperature range, the image processing device 200 can appropriately detect sunlight by changing the dynamic range to the second temperature range that is larger than the first temperature range. In addition, the image processing device 200 can efficiently detect sunlight by setting the detection range. Therefore, according to Embodiment 2, it is possible to provide an image processing device or the like that can appropriately detect sunlight while protecting an infrared sensor from sunlight.

[0085] In addition, the above-mentioned program can be stored using various types of non-transitory computer-readable media and provided to the computer. Non-transitory computer-readable media include recording media having various types of entities. Examples of non-transitory computer-readable media include magnetic recording media (such as floppy disks, magnetic tapes, hard disk drives), optical magnetic recording media (such as optical magnetic disks), CD-ROM (Read Only Memory: Read-only memory) CD-R, CD-R / W, semiconductor memory (such as mask ROM, PROM (Programmable ROM: Programmable ROM), EPROM (Erasable PROM: Erasable PROM), flash ROM, RAM (Random Access Memory: Random Access Memory)). In addition, the program can also be provided to the computer via various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable media can provide the program to the computer via wired communication paths such as wires and optical fibers or wireless communication paths.

[0086] In addition, the present invention is not limited to the above-mentioned embodiments, and can be appropriately modified without departing from the scope of the present invention. For example, the image processing device is not limited to automobiles, but can also be applied to mobile objects such as motorcycles, ships, airplanes, mobile robots, and drones.

[0087] This application claims the benefit of priority based on Japanese patent application No. 2020-000410, filed on January 6, 2020, the disclosure of which is incorporated herein in its entirety.

[0088] Industrial Applicability

[0089] The present disclosure can be applied to an image processing device mounted on a mobile object such as a car, a motorcycle, a ship, an airplane, a mobile robot, or a drone.

[0090] Explanation of symbols

[0091] 10: Thermal image display system;

[0092] 100, 200: image processing device;

[0093] 110: defective pixel correction unit;

[0094] 120: pixel non-uniformity correction unit;

[0095] 130: AGC processing unit;

[0096] 140: saturation region detection unit;

[0097] 150: dynamic range control unit;

[0098] 160: pixel restoration judgment unit;

[0099] 170: shutter control unit;

[0100] 210: detection range setting unit;

[0101] 900: Microbolometer;

[0102] 901: Shutter;

[0103] 902: monitor;

[0104] A11, A12, A13, A14, A21: thermal images;

[0105] D11: other vehicles;

[0106] D12: silencer;

[0107] D13: Sun;

[0108] R21: Detection range.

Claims

1. An image processing device, comprising: a saturation region detection unit that acquires imaging data from an infrared imaging element that captures a thermal image of the periphery of the moving object and detects the presence of a saturation region in the imaging data; a dynamic range control unit that sets a dynamic range to a first temperature range or a second temperature range based on a detection result of the saturated region, the dynamic range being a temperature range detected by the infrared imaging element, the first temperature range being a temperature range in which an upper limit side of the temperature of other vehicles becomes saturated, and the second temperature range being a temperature range in which at least an upper limit side temperature is higher than the first temperature range and in which the temperature of other vehicles does not become saturated; as well as a shutter control unit that controls the opening and closing of a shutter for protecting the infrared imaging element based on the detection result of the saturated area and the setting of the dynamic range, When the saturated region detection unit detects the saturated region when the dynamic range is set to the first temperature range, the dynamic range control unit changes the setting of the dynamic range from the first temperature range to the second temperature range, and when the saturated region detection unit does not detect the saturated region when the dynamic range is set to the second temperature range, the dynamic range control unit changes the setting of the dynamic range from the second temperature range to the first temperature range. The shutter control section closes the shutter when the dynamic range is set to the second temperature range and the saturated region detection section detects the saturated region.

2. The image processing device according to claim 1, wherein: The dynamic range control unit sets an upper limit of a temperature range detected by the infrared imaging element to 50 degrees as the first temperature range, and sets an upper limit of a temperature range detected by the infrared imaging element to 200 degrees as the second temperature range.

3. The image processing device according to claim 1, wherein: The saturated region detecting unit detects the saturated region with respect to a range on the upper side in the vertical direction of the thermal image as a part of the imaging data.

4. The image processing device according to any one of claims 1 to 3, further comprising: a detection range setting unit, which sets a detection range for detecting the saturated area in the captured data, The detection range setting unit sets the detection range using at least one of time, weather, the posture of the moving object, and the moving direction of the moving object as a parameter.

5. An image processing method, comprising: A photographing data acquisition step of acquiring photographing data from an infrared photographing element that photographs a thermal image of the periphery of the moving object; a saturated region detection step, detecting the presence of a saturated region in the captured data; a dynamic range control step of setting a dynamic range to a first temperature range or a second temperature range according to the detection result of the saturated area, the dynamic range being a temperature range detected by the infrared imaging element, the first temperature range being a temperature range in which the upper limit side of the temperature of other vehicles becomes saturated, and the second temperature range being a temperature range in which at least the upper limit side temperature is higher than the first temperature range and the temperature of other vehicles does not become saturated; as well as a shutter control step of controlling the opening and closing of a shutter for protecting the infrared imaging element based on the detection result of the saturated area and the setting of the dynamic range, In the dynamic range control step, when the saturated region is detected in the saturated region detection step when the dynamic range is set to the first temperature range, the setting of the dynamic range is changed from the first temperature range to the second temperature range; when the saturated region is not detected in the saturated region detection step when the dynamic range is set to the second temperature range, the setting of the dynamic range is changed from the second temperature range to the first temperature range; In the shutter control step, the shutter is closed when the saturated region is detected in the saturated region detection step while the dynamic range is set to the second temperature range.

6. A computer-readable storage medium storing a program, wherein the program causes a computer to execute an image processing method, the image processing method comprising: A photographing data acquisition step of acquiring photographing data from an infrared photographing element that photographs a thermal image of the periphery of the moving object; a saturated region detection step, detecting the presence of a saturated region in the captured data; a dynamic range control step of setting a dynamic range to a first temperature range or a second temperature range according to the detection result of the saturated area, the dynamic range being a temperature range detected by the infrared imaging element, the first temperature range being a temperature range in which the upper limit side of the temperature of other vehicles becomes saturated, and the second temperature range being a temperature range in which at least the upper limit side temperature is higher than the first temperature range and the temperature of other vehicles does not become saturated; as well as a shutter control step of controlling the opening and closing of a shutter for protecting the infrared imaging element based on the detection result of the saturated area and the setting of the dynamic range, In the dynamic range control step, when the saturated region is detected in the saturated region detection step when the dynamic range is set to the first temperature range, the setting of the dynamic range is changed from the first temperature range to the second temperature range; when the saturated region is not detected in the saturated region detection step when the dynamic range is set to the second temperature range, the setting of the dynamic range is changed from the second temperature range to the first temperature range; In the shutter control step, the shutter is closed when the saturated region is detected in the saturated region detection step while the dynamic range is set to the second temperature range.

Images