Image processing apparatus and sensor protection method

Abstract

To provide an image processing apparatus capable of appropriately protecting a sensor for imaging a thermal image.SOLUTION: The image processing device includes an image data acquisition unit, a filter operation control unit, a saturation region detection unit, and a sensor protection control unit. The image data acquisition unit acquires a thermal image captured by an infrared sensor that captures a thermal image. The filter operation control unit controls an insertion operation and a retraction operation of an attenuation filter that attenuates infrared rays into and from an optical path through which the infrared rays enter the infrared sensor. The saturated region detection unit detects the presence of a saturated region satisfying a predetermined condition in the thermal image. The sensor protection controller causes the filter operation controller to insert the attenuation filter when the saturated region detector detects a saturated region satisfying the predetermined condition in the thermal image, and causes the filter operation controller to retract the attenuation filter when a maximum pixel value of the thermal image in a state in which the attenuation filter is inserted is less than a predetermined threshold. The sensor protection controller changes the predetermined threshold on the basis of temperature information of the infrared sensor.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device and a sensor protection method. [Background technology]

[0002] A system has been proposed that uses a far-infrared camera to detect pedestrians and other objects. Since a far-infrared camera captures a thermal image of the imaging range, if it captures an object with a temperature above the upper limit of the set dynamic range, the pixels will become saturated, and if the sun enters the imaging range, the pixels in the sun's area will be damaged. For this reason, technology has been proposed to protect the far-infrared sensor from capturing images of the sun.

[0003] For example, Patent Document 1 discloses a technique for protecting a sensor that acquires a thermal image by closing a shutter when a saturated pixel that may be the sun is detected. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-110552 Summary of the Invention [Problem to be solved by the invention]

[0005] The technology described in Patent Document 1 makes it possible to protect the sensor that captures thermal images using a shutter when the sun is included in the shooting range, but it becomes impossible to detect objects such as pedestrians while the shutter is closed to protect the sensor.

[0006] For this reason, there is a need to develop technology that can adequately protect the sensors that capture thermal images. [Means for solving the problem]

[0007] The image processing device according to the present disclosure comprises an image data acquisition unit that acquires a thermal image captured by an infrared sensor that captures thermal images; a filter operation control unit that controls the insertion and retraction of an attenuation filter that attenuates infrared rays into the optical path through which the infrared rays enter the infrared sensor; a saturated area detection unit that detects the presence of a saturated area that meets predetermined conditions in the thermal image; and a sensor protection control unit that, when the saturated area detection unit detects a saturated area that meets the predetermined conditions in the thermal image, causes the filter operation control unit to insert the attenuation filter, and, when the maximum pixel value of the thermal image with the attenuation filter inserted becomes less than a predetermined threshold, causes the filter operation control unit to retract the attenuation filter, and the sensor protection control unit changes the predetermined threshold based on temperature information of the infrared sensor.

[0008] The sensor protection method disclosed herein involves an image processing device acquiring a thermal image captured by an infrared sensor that captures thermal images, detecting the presence of a saturated area that meets predetermined conditions in the thermal image, and, if a saturated area that meets the predetermined conditions is detected in the thermal image, inserting an attenuation filter that attenuates infrared rays into the optical path through which the infrared rays enter the infrared sensor, and retracting the attenuation filter when the maximum pixel value of the thermal image with the attenuation filter inserted becomes less than a predetermined threshold that is changed based on temperature information from the infrared sensor. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to provide an image processing device and a sensor protection method that are capable of appropriately protecting a sensor that captures a thermal image. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 2 is a control block diagram showing an example of the configuration of an imaging system. [Figure 2] FIG. 2 is a control block diagram schematically illustrating an example of the configuration of the infrared camera in FIG. [Figure 3] 2 is a control block diagram showing an example of the configuration of an image processing unit in FIG. 1. FIG. [Figure 4]FIG. 10 is a schematic diagram showing an example of a thermal image captured in an imaging range including the sun. [Figure 5] FIG. 10 is a schematic diagram illustrating an example of a thermal image captured in an imaging range including the sun reflected by a vehicle. [Figure 6] FIG. 10 is a diagram illustrating an example of a signal transfer function of an infrared sensor. [Figure 7] FIG. 10 is a schematic diagram showing the manner in which attenuation is performed by an attenuation filter. [Figure 8] FIG. 10 is a flowchart illustrating an example of a sensor protection process in the image processing device. [Figure 9] FIG. 10 is a flowchart illustrating another example of the sensor protection process in the image processing device. [Figure 10] 1 is a graph showing profile data for an arbitrary line in a thermal image acquired by an image processing device. [Figure 11] 1 is a graph showing profile data for an arbitrary line in a thermal image acquired by an image processing device. [Figure 12] 1 is a schematic diagram for explaining the position of the sun and the angle of view of an infrared camera. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0011] The following describes embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems. For clarity of explanation, the following description and drawings have been omitted and simplified as appropriate.

[0012] Embodiment 1 An image processing device according to this embodiment and an imaging system including the image processing device will be described with reference to Fig. 1. Fig. 1 is a control block diagram showing an example configuration of the imaging system 1. The imaging system 1 includes an image processing device 100, an infrared camera 200, and a display device 300. The imaging system 1 is, for example, an imaging system for vehicles that is used in a vehicle to acquire thermal images of the surroundings, or an imaging system used for surveillance purposes.

[0013] Here, the imaging system 1 will be described as an in-vehicle system installed in a vehicle. By installing the infrared camera 200 in a vehicle, safe driving support can be provided by detecting heat source objects such as people. The infrared camera 200 is mainly used to support visual safety confirmation at night. The infrared camera 200 is also effective when the amount of ambient light is low, such as around sunrise and sunset or inside a tunnel, when it is not nighttime.

[0014] The infrared camera 200 captures thermal images (far-infrared images). The infrared camera 200 captures thermal images at, for example, 15 to 30 frames per second. For example, the infrared camera 200 captures thermal images by detecting far-infrared rays from a subject. The infrared camera 200 is a far-infrared (FIR) camera that can visualize far-infrared rays emitted from an object. The data of the thermal image captured by the infrared camera 200 is also called thermal imaging data (or infrared image data). The thermal imaging data indicates the temperature distribution in the imaging range of the infrared camera 200.

[0015] The infrared camera 200 captures images of the area in front of or around the vehicle. The infrared camera 200 outputs thermal image data to the image processing device 100. The thermal image data input from the infrared camera 200 to the image processing device 100 is also referred to as input thermal image data. The thermal image data refers to thermal image data captured by the infrared camera. The configuration of the infrared camera 200 will be described later.

[0016] The image processing device 100 generates display image data by performing image processing on the input thermal imaging data. For example, the image processing device 100 performs various processes such as correcting pixel variations of the infrared camera 200 and removing noise caused by changes in characteristics over time. Furthermore, the image processing device 100 functions as a control device for controlling the infrared camera 200.

[0017] The image processing device 100 outputs the display image data that has been subjected to image processing to the display device 300. The display image data that has been subjected to image processing by the image processing device 100 and output to the display device 300 is also referred to as output thermal image data. In other words, the input thermal image data is data before processing by the image processing device 100, and the output thermal image data is data after processing by the image processing device 100.

[0018] The display device 300 includes a liquid crystal display, an organic EL (Electro Luminescence) display, or the like for displaying display image data. The display device 300 displays display image data that has been subjected to image processing by the image processing device 100. This allows the display device 300 to display a more accurate temperature distribution.

[0019] The output thermal imaging data output by the image processing device 100 is not limited to being output to the display device 300, but may also be output to a recognition processing device that recognizes people, etc. based on the output thermal imaging data. The recognition results by the recognition processing device may be displayed on the display device 300, or may be output to a warning device that issues a warning based on the recognition results.

[0020] The image processing device 100 includes a control IF (interface) 110, a ROM 120, a RAM 130, a system control unit 140, a thermal image data acquisition unit 160, an image processing unit 170, and a display image data output unit 180. Note that ROM stands for Read Only Memory, and RAM stands for Random Access Memory.

[0021] The thermal image data acquiring unit 160 acquires thermal image data captured by the infrared camera 200. The thermal image data acquiring unit 160 is provided with, for example, an IF (interface) for inputting thermal image data from the infrared camera 200. The interface between the thermal image data acquiring unit 160 and the infrared camera 200 may be a wired connection or a wireless connection. In other words, the thermal image data acquiring unit 160 is an example of an image data acquiring unit that acquires thermal images captured by an infrared sensor that captures thermal images.

[0022] The ROM 120 stores control programs and various parameters for controlling the image processing device 100. For example, the ROM 120 stores an image processing program executed by the image processing unit 170. The ROM 120 also stores a system control program executed by the system control unit 140. This system control program includes a sensor protection control program executed by the sensor protection control unit 141, which will be described later.

[0023] The RAM 130 stores various programs and parameters used for executing the programs, etc. Furthermore, it stores calculation data for the image processing unit 170, etc.

[0024] The system control unit 140 controls the entire imaging system 1. The system control unit 140 includes a processor such as a CPU (Central Processing Unit). The processor of the system control unit 140 executes a control program, thereby controlling the entire system.

[0025] The system control unit 140 includes a sensor protection control unit 141 that controls protection of the infrared camera 200, and a filter operation control unit 142 that controls insertion and retraction of an attenuation filter for protection.

[0026] The sensor protection control unit 141 determines whether or not to protect the infrared camera 200, and issues an instruction to the filter operation control unit 142. This determination will be described later.

[0027] The filter operation control unit 142 outputs an instruction to insert an attenuation filter between the lens and the sensor in the infrared camera 200. The filter operation control unit 142 also outputs an instruction to retract the attenuation filter from between the lens and the sensor in the infrared camera 200. In other words, the filter operation control unit 142 outputs a control signal to insert or retract the attenuation filter.

[0028] The control IF 110 is an interface for controlling the infrared camera 200. For example, the filter operation control unit 142 outputs a control signal for inserting and retracting an attenuation filter in the infrared camera 200 via the control IF 110. The image processing device 100 can also transmit various signals and data to the infrared camera 200 via the control IF 110. Furthermore, the control IF 110 can receive various signals and data from the infrared camera 200. As one of the received data, the control IF 110 receives temperature information indicating the ambient temperature measured by a temperature sensor of the infrared camera 200.

[0029] The image processing unit 170 performs predetermined image processing on the thermal imaging data acquired by the thermal imaging data acquisition unit 160. The image processing unit 170 may include a processor for executing an image processing program. The processor of the image processing unit 170 may be a general-purpose processor or a dedicated processor. The processor of the image processing unit 170 and the processor of the system control unit 140 may be the same. The processing in the image processing unit 170 will be described later. The image processing unit 170 generates display image data by performing image processing on the input thermal imaging data.

[0030] The display image data output unit 180 is an interface that receives display image data from the image processing unit 170 and outputs the display image data to the display device 300. This allows a user, such as a vehicle driver, to check the display image data on the display device 300. This allows the user to recognize people and the like around the vehicle even in dark conditions, such as at night.

[0031] Next, an example configuration of the infrared camera 200 in Fig. 1 will be described using Fig. 2. Fig. 2 is a control block diagram schematically showing an example configuration of the infrared camera 200. The infrared camera 200 includes a lens 210, an attenuation filter 220, a sensor 230, a transmission device 240, and a temperature sensor 250.

[0032] The lens 210 forms an image of far-infrared light from the subject on the light receiving surface of the sensor 230. The lens 210 has at least one lens. For example, the lens 210 may include multiple lenses such as a zoom lens and a focus lens.

[0033] The sensor 230 includes a plurality of pixels. Each pixel of the sensor 230 receives infrared light from the subject. This allows a thermal image of the subject to be captured. For example, the sensor 230 includes a microbolometer for detecting far-infrared rays. The sensor 230 includes a plurality of pixels arranged in a two-dimensional array. The detection values ​​(detection signals) of each pixel form a thermal image of the subject.

[0034] The transmission device 240 serves as an interface for transmitting various signals and data to the image processing device 100. The transmission device 240 transmits thermal imaging data captured by the sensor 230 to the image processing device 100. The transmission device 240 also receives control signals from the image processing device 100. Furthermore, the transmission device 240 may transmit lens information related to the zoom and focus of the lens 210.

[0035] The temperature sensor 250 measures the temperature in the environment in which the infrared camera 200 is used, that is, the environmental temperature which is the temperature of the sensor 230. The transmission device 240 transmits temperature information indicating the environmental temperature measured by the temperature sensor 250 to the image processing device 100. The environmental temperature measured by the temperature sensor 250 is used to control the insertion and retraction operations of the attenuation filter 220. More specifically, the environmental temperature measured by the temperature sensor 250 is used to change the threshold value used for that control. The environmental temperature measured by the temperature sensor 250 can also be used to correct output variations between pixels.

[0036] The attenuation filter 220 is disposed on the front side of the sensor 230 in order to attenuate infrared light. When inserted, the attenuation filter 220 is disposed in the optical path between the lens 210 and the sensor 230, and attenuates infrared light incident on the sensor 230 from the outside. When retracted, the attenuation filter 220 is disposed outside the optical path between the lens 210 and the sensor 230. The infrared camera 200 is provided with a mechanism that can insert and retract the attenuation filter 220.

[0037] The attenuation filter 220 performs an insertion operation or a retraction operation in accordance with a control signal received from the filter operation control unit 142 via the control IF 110 and the transmission device 240. When the attenuation filter 220 is inserted (hereinafter also simply referred to as the inserted state), infrared light from the lens 210 is attenuated by the attenuation filter 220 and enters the sensor 230. That is, in the inserted state, the attenuated infrared light that has passed through the attenuation filter 220 enters the sensor 230. When the attenuation filter 220 is retracted (hereinafter also simply referred to as the retracted state), the infrared light from the lens 210 enters the sensor 230 without passing through the attenuation filter 220. Here, the description is based on the assumption that the attenuation filter 220 is disposed between the lens 210 and the sensor 230 in the inserted state. However, the attenuation filter 220 may be disposed on the front side of the lens 210 in the inserted state.

[0038] Furthermore, the attenuation filter 220 does not necessarily need to be a filter that requires a mechanical insertion / retraction mechanism, but may be a filter that is always disposed in front of the sensor 230 and electronically switches between an inserted state and a retracted state. For example, the attenuation filter 220 may be a filter that applies technology for changing the transmittance of infrared light, such as that used in liquid crystal shutters. Such a filter may be electronically controlled so that when attenuating, it is essentially in an inserted state to attenuate infrared light without blocking it, and when not attenuating, it is essentially in a retracted state to transmit almost all of the infrared light.

[0039] The configuration and processing of the image processing unit 170 will be mainly described together with the processing of the system control unit 140 using Figures 3 to 5. Figure 3 is a control block diagram showing an example of the configuration of the image processing unit 170 in Figure 1. Figure 4 is a schematic diagram showing an example of a thermal image captured within an imaging range including the sun. Figure 5 is a schematic diagram showing an example of a thermal image captured within an imaging range including the sun reflected by a vehicle.

[0040] The image processing unit 170 includes a defective pixel correction unit 171, a NUC (Non-Uniformity Correction) unit 172, and a saturated region detection unit 174. The image processing unit 170 cooperates with the system control unit 140 and the RAM 130 to execute predetermined processing.

[0041] The defective pixel correction unit 171 performs an interpolation process on the pixel value of the defective pixel of the sensor 230, which is stored in advance in the RAM 130, using the pixel values ​​of the pixels surrounding the defective pixel. The defective pixel correction unit 171 performs the above-mentioned interpolation process on the input thermal imaging data acquired by the thermal imaging data acquisition unit 160, and outputs the image data after the interpolation process to the NUC unit 172 and the saturated region detection unit 174.

[0042] The NUC (Non-Uniformity Correction) unit 172 performs processing to correct output variations between pixels in the sensor 230. For example, the NUC unit 172 corrects the individual pixel values ​​of the image data based on preset gain and offset values ​​corresponding to the characteristics of each pixel of the sensor 230.

[0043] The saturated area detection unit 174 detects saturation of pixels in the thermal imaging data. The saturated area detection unit 174 can also be called a saturated pixel detection unit. The detected saturation includes saturation due to direct sunlight, such as the saturated area H11 in the thermal image A11 of FIG. 4. Furthermore, the detected saturation may also include saturation due to reflected sunlight, such as the saturated area H12 in the thermal image A12 of FIG. 5. The saturated area H12 is an area where sunlight is reflected by the forward vehicle V01. In addition, FIGS. 4 and 5 show an example in which the thermal images A11 and A12 also contain a saturated area H13 due to the muffler of the forward vehicle V01. The saturated areas H12 and H13 are areas that cause less damage to pixels than the saturated area H11.

[0044] The saturated area detection unit 174 identifies the pixel addresses of saturated pixels as a detection result. This allows the saturated area detection unit 174 to identify saturated areas in the thermal imaging data where pixels are saturated. A saturated area is an area where saturated pixels or substantially saturated pixels exist. A saturated pixel is a pixel whose pixel data has reached its upper limit value. For example, if the brightness level of each pixel in the imaging data is expressed in 14 bits from 0 to 16383, a pixel at a coordinate where the brightness level is 16383 is called a saturated pixel.

[0045] In particular, the saturated region detection unit 174 detects the presence of a saturated region that meets a predetermined condition in the thermal image. A saturated region that meets a predetermined condition can be a region that can be considered saturated due to direct sunlight or reflected sunlight. For example, a saturated region that meets a predetermined condition can be a region where there are multiple consecutive pixels whose pixel data reaches an upper limit value, and the size of the pixel group consisting of these multiple pixels is equal to or greater than a predetermined size, or a region where the shape of the pixel group is equal to or greater than a predetermined shape, such as a circle or ellipse. For example, a saturated region that meets a predetermined condition can be a region where there are a predetermined number of consecutive saturated pixels, such as 5 pixels x 5 pixels or more in the horizontal or vertical direction. Alternatively, a saturated region that meets a predetermined condition can be a region where the size of the pixel group is equal to or greater than a predetermined size and the shape of the pixel group is equal to or greater than a predetermined shape, such as a circle or ellipse. Alternatively, a saturated region that meets a predetermined condition can be a region where there are a predetermined number of consecutive saturated pixels, such as three pixels, in one horizontal or vertical line.

[0046] When the saturated area detection unit 174 detects a saturated area that satisfies a predetermined condition in the thermal image, the sensor protection control unit 141 instructs the filter operation control unit 142 to insert the attenuation filter 220 .

[0047] As described above, the filter operation control unit 142 controls the insertion operation and retraction operation of the attenuation filter 220 into the optical path through which infrared rays are incident on the infrared sensor 230. In the example of Fig. 1, the filter operation control unit 142 instructs the infrared camera 200, via the control IF 110, to perform an insertion operation to insert the attenuation filter 220 into the optical path through which infrared rays are incident on the sensor 230, or to perform a retraction operation to retract the attenuation filter 220 from this optical path.

[0048] Furthermore, the saturated region detection unit 174 detects the maximum pixel value of the thermal image when the attenuation filter 220 is inserted. The insertion state of the attenuation filter 220 can be determined by receiving attenuation filter insertion / removal information from the infrared camera 200, which indicates whether the attenuation filter 220 is inserted or retracted. Based on this attenuation filter insertion / removal information, the saturated region detection unit 174 can determine the detection timing for the maximum pixel value of the thermal image and perform the detection. Specifically, the saturated region detection unit 174 may periodically detect the maximum pixel value of the thermal image when the attenuation filter 220 is inserted.

[0049] Here, the saturated region detection unit 174 may detect saturated pixels for detecting saturated regions that satisfy predetermined conditions, or may detect the above-described maximum pixel value, in pixel block units such as 3x3 pixels or 5x5 pixels. Detecting the maximum pixel value may involve detecting the maximum pixel value in pixel block units, and then detecting the maximum value among the maximum pixel values ​​in each pixel block. Alternatively, detecting the maximum pixel value may involve detecting a group of pixel values ​​that has the maximum total or average pixel value in pixel block units, for example.

[0050] By employing the above-described pixel block unit detection process, it is possible to avoid situations where detection is performed on a pixel-by-pixel basis and is affected by defective pixels, etc. Of course, even in a configuration such as the example in Figure 1 where the image processing unit 170 is provided with the defective pixel correction unit 171, there is still some influence from defective pixels, so it can be said that performing detection on a pixel block unit is beneficial.

[0051] The image processing unit 170 outputs to the system control unit 140 saturation detection data related to saturation, indicating the presence or absence of saturation, the address of the saturated pixel, the saturated region, etc., and maximum pixel value detection data indicating the maximum pixel value detected when the attenuation filter 220 is in the inserted state. The saturation detection data and maximum pixel value detection data may be output separately or simultaneously. When output simultaneously, for example, when the attenuation filter 220 is in the retracted state, there is essentially no maximum pixel value detection data.

[0052] The sensor protection control unit 141 also receives the maximum pixel value detection data, and when the maximum pixel value of the thermal image with the attenuation filter 220 inserted is less than a predetermined threshold, causes the filter operation control unit 142 to retract the attenuation filter 220. Here, the sensor protection control unit 141 determines whether the maximum pixel value is less than the predetermined threshold when the attenuation filter 220 is inserted, and when it determines that the maximum pixel value is less than the predetermined threshold, sends an instruction to the filter operation control unit 142 to retract the attenuation filter 220. Then, the filter operation control unit 142 outputs a control signal to the infrared camera 200 via the control IF 110 to retract the attenuation filter 220 from the optical path through which infrared rays enter the sensor 230.

[0053] Furthermore, the sensor protection control unit 141 changes the above-mentioned predetermined threshold value based on the temperature information of the infrared sensor. That is, the sensor protection control unit 141 determines the condition for retracting the attenuation filter 220 based on the temperature information indicating the environmental temperature of the sensor 230 obtained by the temperature sensor 250.

[0054] Next, an example of changing the predetermined threshold value based on temperature information will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a diagram showing an example of a signal transfer function (SiTF) of an infrared sensor. Fig. 7 is a schematic diagram showing attenuation by the attenuation filter 220.

[0055] As shown in Figure 6, the SiTF of an infrared sensor, that is, the amount of change in digital output per 1°C of the infrared sensor, increases as the ambient temperature of the infrared sensor (hereinafter referred to as sensor temperature) increases. In other words, the SiTF of an infrared sensor increases as the sensor temperature increases. Sensor 230 is an infrared sensor, and the SiTF exhibits a similar tendency.

[0056] Therefore, the higher the sensor temperature of sensor 230, the narrower the temperature range (dynamic range) that can be measured within the measurement width. The measurement width refers to 0 to 16383 in the case of 14 bits, for example. And, since the dynamic range becomes narrower as the sensor temperature of sensor 230 becomes higher, the pixel value width up to the predetermined threshold that allows attenuation filter 220 to be retracted becomes smaller.

[0057] Therefore, the higher the sensor temperature of the sensor 230, the higher the "predetermined threshold" that is the condition for retracting the attenuation filter 220 when the maximum pixel value of the thermal image falls below the predetermined threshold should be set to. In other words, for example, the higher the sensor temperature of the sensor 230, the higher the predetermined threshold should be set by the sensor protection control unit 141.

[0058] By adopting such settings, it is possible to more accurately determine when sunlight incident on the sensor 230 as direct light or reflected light has gone outside the angle of view, and when the sunlight has gone outside the angle of view, it is possible to appropriately retract the attenuation filter 220. Furthermore, even when the attenuation filter 220 is inserted, it does not block all incident light like a shutter, so it is also possible to detect objects such as pedestrians.

[0059] Fig. 7 shows a schematic diagram of the sensor characteristics of the sensor 230 and the attenuation caused by the attenuation filter 220. The sensor characteristics shown in Fig. 7 show the characteristics of the measurement value (raw digital level) versus the sensor temperature when images of constant-temperature objects with temperatures of 10°C and 50°C are captured by the sensor 230. Here, the results are shown for the case where the measurement range of the sensor 220 is the above-mentioned range of 0 to 16383.

[0060] 7 also shows a sensor saturation region 70, which is a schematic representation of the measurement value when measurements of 16383 or more, which is the upper limit of the measurement value, are possible, and shows the change in output when sunlight is incident on the sensor in a simplified linear fashion, i.e., a straight line. For convenience, the change in output is depicted as a simplified linear fashion, but in reality, it is expected to be nonlinear, i.e., a curve, as in the example characteristics of images of objects at constant temperatures of 10°C and 50°C.

[0061] In reality, solar energy is a high thermal energy, and is therefore expected to have a value equal to or greater than the saturated pixel value. Therefore, as shown by the range indicated by the double-headed arrow in Fig. 7 where the attenuation caused by attenuation filter 220 from the assumed solar output value is exemplified, it is expected that, particularly when the sensor temperature is high, the value may not necessarily become less than the saturated pixel value even if attenuation filter 220 is inserted.

[0062] Therefore, when the attenuation filter 220 is inserted, that is, when the attenuation filter 220 provides a solar light attenuation barrier, it is advisable to divide the sensor temperature into sections and set a linear filter retraction threshold for each section. In this example, the filter retraction threshold is set to a different value for each of the three sections: when the temperature is below temperature Ta, when the temperature is equal to or higher than temperature Ta but lower than temperature Tb, and when the temperature is equal to or higher than temperature Tb. Each filter retraction threshold is an example of the predetermined threshold described above. In other words, the filter retraction threshold for each section illustrated in FIG. 7 is an example of a case where the predetermined threshold is set to a higher value as the sensor temperature of the sensor 230 is higher.

[0063] Furthermore, by setting a predetermined threshold value, for example, in finer divisions or nonlinearly, based on experimental values ​​obtained from experiments using the attenuation filter 220 and sensor 230 that are actually installed, it is possible to perform the retraction of the attenuation filter 220 at a more appropriate time.

[0064] Next, an example of sensor protection processing relating to the insertion / retraction operation of the attenuation filter 220 in the image processing device 100 will be described with reference to Fig. 8. Fig. 8 is a flow chart for explaining an example of sensor protection processing in the image processing device 100.

[0065] The saturated area detection unit 174 of the image processing unit 170 starts detecting whether a saturated area meeting a predetermined condition exists in the thermal image captured by the sensor 230 and acquired by the thermal image data acquisition unit 160 (step S10). As described above, a saturated area meeting a predetermined condition is an area where pixels are saturated due to direct sunlight or reflected sunlight. Then, the saturated area detection unit 174 determines whether a saturated area meeting a predetermined condition has been detected (step S11). If the saturated area detection unit 174 detects a predetermined saturated area, that is, if the answer to step S11 is YES, it transmits saturation detection data to the system control unit 140. Here, the processing is described from the case where the attenuation filter 220 is in the retracted state. However, if the attenuation filter 220 is in the inserted state, maximum pixel value detection data, which is the result of detecting the maximum pixel value, is also transmitted to the system control unit 140.

[0066] The sensor protection control unit 141 receives saturation detection data output from the image processing unit 170 as a result of detecting a saturated region that satisfies predetermined conditions, and instructs the filter operation control unit 142 to insert the attenuation filter 220. In response to this instruction, the filter operation control unit 142 instructs the infrared camera 200 via the control IF 110 to perform an insertion operation to insert the attenuation filter 220 into the optical path through which infrared light enters the sensor 230 (step S12). As a result, the attenuation filter 220 is inserted into the optical path.

[0067] Although not shown, after the insertion of the attenuation filter 220, the saturated region detection unit 174 detects the maximum pixel value in the thermal image. The saturated region detection unit 174 receives attenuation filter insertion / removal information from the infrared camera 200, and when the attenuation filter 220 is inserted, it detects the maximum pixel value in the thermal image.

[0068] Next, the sensor protection control unit 141 determines a predetermined threshold value based on the temperature information received from the infrared camera 200 via the control IF 110 (step S13).

[0069] Next, the sensor protection control unit 141 determines whether the maximum pixel value of the thermal image with the attenuation filter 220 inserted is less than a predetermined threshold value (step S14), and if NO, returns to the processing of step S13. If YES in step S14, the sensor protection control unit 141 instructs the filter operation control unit 142 to perform a retraction operation of the attenuation filter 220. In response to this instruction, the filter operation control unit 142 instructs the infrared camera 200 via the control IF 110 to perform a retraction operation to retract the attenuation filter 220 from the optical path (step S15). As a result, the attenuation filter 220 is retracted from the optical path.

[0070] Next, the system control unit 140 determines whether or not to end the process related to the insertion / retraction operation of the attenuation filter 220 (step S16). Whether or not to end the process related to the insertion / retraction operation may be determined based on whether or not the capture of thermal images is continuing. If the process related to the insertion / retraction operation is to be continued, that is, if the answer is NO in step S16, the system control unit 140 returns to the process of step S11 and causes the saturated region detection unit 174 to continue detecting saturated regions that meet predetermined conditions. If the process related to the insertion / retraction operation is to be ended, that is, if the answer is YES in step S16, the system control unit 140 ends this process as is. Also, if the answer is NO in step S11, the system control unit 140 makes the determination in step S16.

[0071] As described above, according to this embodiment, by controlling the insertion / retraction operation of the attenuation filter 220, the time for which the attenuation filter 220 is inserted can be shortened, thereby shortening the time required to obtain an attenuated thermal image, while protecting the sensor 230. Of course, according to this embodiment, the attenuation filter 220 is used instead of a shutter that blocks incident light to the sensor 230, so there is no period when the shutter is closed to protect the sensor, and thermal images can be obtained at all times. Therefore, even when applied to an object detection system that detects objects from thermal images, there is no risk of an inability to detect objects such as pedestrians. In this way, according to this embodiment, it is possible to appropriately protect the sensor that captures thermal images.

[0072] Embodiment 2 An image processing device according to a second embodiment will be described. In this embodiment, the setting of the predetermined threshold in the first embodiment is changed depending on whether the saturated region of the predetermined condition is saturated due to reflected sunlight or saturated due to direct light. In this embodiment, the various configuration examples and application examples described in the first embodiment can also be applied. Hereinafter, the image processing device according to this embodiment will be described using the image processing device 100 of FIG. 1 as an example, focusing on the differences from the first embodiment, and a description of the similarities will be omitted.

[0073] The saturated region detection unit 174 in this embodiment also determines whether the saturated region of a predetermined condition is saturated due to direct sunlight or saturated due to reflected light. Next, a first example of this determination method and a processing example of this embodiment based on this first example will be described.

[0074] (Example 1) When the saturation region that satisfies the above-mentioned predetermined conditions is detected with the attenuation filter 220 retracted, both the reflected light and the direct light are saturated, and therefore there is a good chance that there will be no difference in the output values ​​of the sensor 230 between the reflected light and the direct light. On the other hand, after the attenuation filter 220 is inserted, that is, with the attenuation filter 220 inserted, the output value of the sensor 230 for the attenuated reflected light is lower than that of the attenuated direct light.

[0075] Therefore, the saturated region detection unit 174 may determine whether, for example, with the attenuation filter 220 inserted, a statistical value of pixel values ​​in a region corresponding to a detected saturated region of a predetermined condition is equal to or greater than a predetermined incident light determination threshold. If the statistical value is equal to or greater than the predetermined incident light determination threshold, the saturated region detection unit 174 can determine that the detected saturated region of a predetermined condition is saturated due to direct light. On the other hand, if the statistical value is less than the predetermined incident light determination threshold, the saturated region detection unit 174 can determine that the detected saturated region of a predetermined condition is saturated due to indirect light. The statistical value may be, for example, a maximum value, a median value, an average value, or the like. The statistical value may also be calculated on a pixel block basis.

[0076] Furthermore, since imaging itself is possible even when the attenuation filter 220 is inserted, it is possible to determine whether the cause of saturation is direct light or indirect light without stopping functions such as obtaining thermal images and outputting display image data.

[0077] Then, the sensor protection control unit 141 in this embodiment receives a judgment result based on a predetermined incident light judgment threshold from the saturated area detection unit 174, and sets the predetermined threshold to a lower pixel value when the saturated area under the predetermined conditions is saturated due to reflected light than when it is saturated due to direct light.

[0078] Next, an example of sensor protection processing related to the insertion / retraction operation of the attenuation filter 220 in the image processing device 100 according to this embodiment will be described with reference to Fig. 9. Fig. 9 is a flow chart for explaining another example of sensor protection processing in the image processing device 100.

[0079] First, the same processes as steps S10 to S12 in Fig. 8 are executed (steps S20 to S22). At step S22, the attenuation filter 220 is inserted.

[0080] After the attenuation filter 220 is inserted, the saturated region detection unit 174 (not shown) detects the maximum pixel value in the thermal image and calculates the above statistical values ​​for the region corresponding to the detected saturated region of the predetermined condition. The saturated region detection unit 174 receives attenuation filter insertion / retraction information from the infrared camera 200, and when the attenuation filter 220 is inserted, it detects the maximum pixel value in the thermal image and calculates the above statistical values.

[0081] Next, the saturated region detection unit 174 determines whether the saturation threshold is due to direct light or indirect light (reflected light) by determining whether the statistical value is equal to or greater than a predetermined incident light determination threshold (step S23). The saturated region detection unit 174 passes the determination result to the sensor protection control unit 141.

[0082] If the answer is YES in step S23, the sensor protection control unit 141 determines a threshold value for direct light as a predetermined threshold value based on the temperature information received from the infrared camera 200 via the control IF 110 (step S24).

[0083] Next, the sensor protection control unit 141 determines whether the maximum pixel value of the thermal image with the attenuation filter 220 inserted is less than the threshold for direct light (step S25), and if NO, returns to the processing of step S24. If YES in step S25, the sensor protection control unit 141 instructs the filter operation control unit 142 to perform a retraction operation of the attenuation filter 220. In response to this instruction, the filter operation control unit 142 instructs the infrared camera 200 via the control IF 110 to perform a retraction operation to retract the attenuation filter 220 from the above-mentioned optical path (step S26). As a result, the attenuation filter 220 is retracted from the above-mentioned optical path.

[0084] Next, similarly to step S16, the system control unit 140 determines whether or not to end the process related to the insertion / removal operation of the attenuation filter 220 (step S27). If NO in step S27, the process returns to step S21, and the saturated region detection unit 174 continues detecting saturated regions that satisfy predetermined conditions. If YES in step S27, the system control unit 140 ends this process as is. Also, if NO in step S21, the determination in step S27 is performed.

[0085] On the other hand, if the result of step S23 is NO, the sensor protection control unit 141 determines a threshold value for indirect light as a predetermined threshold value based on the temperature information received from the infrared camera 200 via the control IF 110 (step S28). Here, the threshold value for indirect light is determined to be a pixel value lower than the threshold value for direct light.

[0086] Next, the sensor protection control unit 141 determines whether the maximum pixel value of the thermal image with the attenuation filter 220 inserted is less than the threshold value for indirect light (step S29), and if NO, the process returns to step S28. If YES in step S29, the process proceeds to step S26, where the attenuation filter 220 is retracted from the optical path.

[0087] As described above, in this embodiment, the determination of whether sunlight is direct light or reflected light is reflected in a predetermined threshold value to control the insertion / retraction operation of the attenuation filter 220. Therefore, according to this embodiment, the time for which the attenuation filter 220 is inserted is shortened compared to the first embodiment, thereby shortening the time required to obtain an attenuated thermal image and protecting the sensor 230.

[0088] (Example 2) Next, a second example will be given as another example of the method for determining the cause of saturation in saturated region detection unit 174. In the second example, the cause of saturation is determined based on a profile for pixels surrounding a saturated region that meets predetermined conditions. This determination method will be described below with reference to FIGS. 10 and 11, giving an example of a method for calculating a profile for pixels surrounding a saturated region that meets predetermined conditions.

[0089] FIG. 10 is a graph showing profile data for an arbitrary line in the thermal image A11 of FIG. 4 and the thermal image A12 of FIG. 5. In FIG. 10, the horizontal axis represents pixel address, and the vertical axis represents pixel output value. A horizontal or vertical line including the center of the saturated region of the predetermined condition is selected here. Of course, the profile data is not limited to a horizontal or vertical line, and may be a line of profile data in any inclination direction. As shown in the upper graph of FIG. 10, the thermal image A11 has a broader base around the saturated region of the predetermined condition. As shown in the lower graph of FIG. 10, the thermal image A12 has a narrower base around the saturated region of the predetermined condition. In other words, when saturation occurs due to direct sunlight, the profile rises gently at the pixels around the saturated region of the predetermined condition. When saturation occurs due to reflected sunlight, the profile rises sharply at the pixels around the saturated region of the predetermined condition. Therefore, the saturated region detection unit 174 in the second example uses the difference in profile shape around the saturated region of the predetermined condition to determine whether saturation occurs due to direct sunlight or reflected light. More specifically, the saturated region detection unit 174 in the second example can make this determination based on the tendency of spread of pixel output values ​​of peripheral pixels of a saturated region that satisfies a predetermined condition.

[0090] As shown in FIG. 10, the number of saturated pixels in one line is defined as A. The number of pixels in one line that are equal to or greater than a set value L is defined as the number of surrounding pixels B. Here, the number of saturated pixels A can be defined as the number of consecutive saturated pixels. In other words, the number of saturated pixels A is the number of pixels in one line that are included in a saturated region that meets a predetermined condition. The number of surrounding pixels B can be defined as the number of consecutive pixels in one line that exceed the set value L. The set value L is set to 80% of the maximum pixel output value. Here, for example, in thermal image A11, A=8 and B=16, and in thermal image A12, A=8 and B=9. The saturated region detection unit 174 counts the number of saturated pixels A and the number of surrounding pixels B in one line. Then, the saturated region detection unit 174 calculates a difference value C by subtracting the number of saturated pixels A from the number of surrounding pixels B. In the above example, C=8 in thermal image A11 and C=1 in thermal image A12.

[0091] The saturated region detection unit 174 determines whether the difference value C between the number of surrounding pixels B and the number of saturated pixels A is equal to or greater than a threshold value TH1. If the difference value C is equal to or greater than the threshold value TH1, the saturated region detection unit 174 determines that saturation is due to direct sunlight. If the difference value C is less than the threshold value TH1, the saturated region detection unit 174 determines that saturation is due to reflected sunlight. Here, TH1=4. Note that the threshold value TH1 and the set value L are not limited to the above values, and appropriate values ​​can be set depending on the number of pixels and characteristics of the infrared camera 200, etc.

[0092] In this way, the saturated region detection unit 174 determines whether the saturated region of the predetermined condition is saturated due to the incidence of direct light or the incidence of reflected light based on the tendency of spread of pixel output values ​​of peripheral pixels of the saturated region of the predetermined condition. Specifically, the saturated region detection unit 174 calculates, in a profile in an arbitrary direction, the number A of saturated pixels included in the saturated region of the predetermined condition and the number B of peripheral pixels whose pixel output values ​​are equal to or greater than a set value L. If the difference value C between the number A of saturated pixels and the number B of peripheral pixels is equal to or greater than a threshold value TH1, the saturated region detection unit 174 determines that the saturated region of the predetermined condition is saturated due to the incidence of direct light. If the difference value C between the number A of saturated pixels and the number B of peripheral pixels is less than the threshold value TH1, the saturated region detection unit 174 determines that the saturated region of the predetermined condition is saturated due to the incidence of reflected light.

[0093] Another example of the determination using a profile will be described with reference to Fig. 11. In Fig. 11, the upper and lower graphs show profile data for an arbitrary line in the thermal image A11 of Fig. 4 and the thermal image A12 of Fig. 5, respectively. In this example, the saturated region detection unit 174 performs a determination in accordance with the gradient of adjacent pixels outside the saturated regions R1 and R2 of predetermined conditions in the profile of one line.

[0094] First, the saturated region detection unit 174 extracts one line passing through saturated regions R1 and R2 that meet the specified conditions. Then, on both sides of the saturated region that meets the specified conditions, the saturated region detection unit 174 finds the slopes X and Y of the pixel output values. The slope X is the slope of the pixel output values ​​in the adjacent region to the left of the saturated region that meets the specified conditions, and the slope Y is the slope of the pixel output values ​​in the adjacent region to the right of the saturated region that meets the specified conditions. It is assumed that the slope X is a positive value and the slope Y is a negative value. The saturated region detection unit 174 makes a determination by comparing the absolute values ​​of the slopes X and Y with a threshold value TH2.

[0095] The saturated region detection unit 174 calculates the slopes X and Y based on the pixel output values ​​of pixels at the edge of the saturated region R1 of the predetermined conditions and the pixel output values ​​of pixels located a predetermined number of pixels outside the edge of the saturated region R1 of the predetermined conditions. Specifically, the slope calculated using the pixels from the second pixel to the seventh pixel from the edge of the saturated region R1 of the predetermined conditions is appropriate. Of course, the pixels for which the slope is calculated are not limited to the second pixel to the seventh pixel from the edge, and the appropriate range varies depending on the number of pixels in the sensor 230. The pixels are defined as those located an arbitrary predetermined number t of pixels outside the edge of the saturated region R1 of the predetermined conditions, from the second pixel to the seventh pixel.

[0096] The saturated region detection unit 174 finds the slopes X and Y by drawing a straight line from the pixel output value of the edge pixel of the saturated region R1 that satisfies the predetermined conditions to the pixel output value of the t-th pixel. For example, let R1L be the pixel output value of the pixel at the left edge of the saturated region R1 that satisfies the predetermined conditions, and let DtL be the pixel output value of the pixel that is the t-th pixel from the pixel at the left edge of the saturated region R1 that satisfies the predetermined conditions. In this case, the slope X is expressed as (R1L - DtL) / t.

[0097] Similarly, the saturated region detection unit 174 calculates the slope Y. For example, let R1R be the pixel output value of the pixel at the right end of a saturated region R1 under predetermined conditions, and let DtR be the pixel output value of the t-th pixel from the right end pixel in the saturated region R1 under predetermined conditions. In this case, the slope Y is expressed as Y=(DtL-R1R) / t. The saturated region detection unit 174 may also calculate the slope from multiple pixel output values ​​included in adjacent regions using the least squares method or the like. In other words, the saturated region detection unit 174 can also calculate the slope from an approximation line that approximates the pixel output values ​​of three or more pixels included in adjacent regions.

[0098] For example, in thermal image A11, the slope X is 7.98 and the slope Y is -11.67. In thermal image A12, the slope X is 13.76 and the slope Y is -13.25. Therefore, the threshold TH2 is set to 9. If at least one of the absolute values ​​of the slopes X and Y is less than the threshold TH2, the saturated region detection unit 174 determines that saturation is due to direct light. If both the absolute values ​​of the slopes X and Y are equal to or greater than the threshold TH2, the saturated region detection unit 174 determines that saturation is due to reflected light. In the case of saturation due to reflected light, the slope becomes steeper. In other words, in the case of saturation due to reflected light, the absolute values ​​of the slopes X and Y are larger than in the case of saturation due to direct light incidence. Therefore, the saturated region detection unit 174 can make an appropriate determination by comparing the absolute values ​​of the slopes X and Y with the threshold TH2.

[0099] In this way, the saturated region detection unit 174 can determine whether a saturated region of a predetermined condition is saturated due to the incidence of direct light or the incidence of reflected light, based on the tendency of the gradient of pixel output values ​​in an adjacent region outside the saturated region of the predetermined condition. Specifically, the saturated region detection unit 174 calculates the gradient of pixel output values ​​in an adjacent region based on the pixel output values ​​of pixels at the edge of the saturated region of the predetermined condition and the pixel output values ​​of pixels located a predetermined number of pixels outside the edge of the saturated region of the predetermined condition, in a profile in an arbitrary direction. Then, the saturated region detection unit 174 compares the absolute value of the gradient with a threshold value TH2.

[0100] If the absolute value of the slope is less than the threshold TH2, the saturated region detection unit 174 determines that the saturated region under the predetermined conditions is saturated due to the incidence of direct light.If the absolute value of the slope is equal to or greater than the threshold TH2, the saturated region detection unit 174 determines that the saturated region under the predetermined conditions is saturated due to the incidence of reflected light.

[0101] Two examples of the determination using the profile have been given. By performing the determination as shown, the saturated region detection unit 174 can appropriately determine whether the saturation is due to the incidence of direct sunlight or the incidence of reflected light. Therefore, it is possible to set different predetermined thresholds depending on whether direct light is incident or reflected light is incident.

[0102] In the second example, as in the first example, the time for which the attenuation filter 220 is inserted is shortened compared to the first embodiment, thereby shortening the time required to obtain a thermal image with attenuation, thereby protecting the sensor 230. Also, unlike the first example, the determination using the profile can be performed not only when the attenuation filter 220 is inserted but also when it is retracted. The second example can be used instead of the first example, but by combining it with the first example, the determination of the cause of saturation can be performed more accurately.

[0103] (Example 3) Next, a third example will be given as another example of the method for determining the cause of saturation in saturated region detection section 174.

[0104] In the third example, the saturation factor is determined based on the shooting direction and shooting range (angle of view) identified from the traveling direction of the vehicle equipped with at least the infrared camera 200 of the imaging system 1. Hereinafter, this determination method will be described with an example of a method for calculating the shooting direction and angle of view identified from the traveling direction of the vehicle equipped with at least the infrared camera 200 of the imaging system 1.

[0105] The image processing device 100 in the third example includes a position information acquisition unit (not shown) so that its own position information can be input to the system control unit 140, for example. The position information acquisition unit acquires position information related to the shooting position of the infrared camera 200. The position information acquisition unit sequentially acquires position information based on positioning signals such as GPS data received by a positioning information receiving unit (not shown). GPS stands for Global Positioning System. The position information acquisition unit acquires time-series data of the position information. Information related to the shooting direction of the infrared camera 200 may be acquired as the position information. The mounting position and mounting direction of the infrared camera 200 relative to the vehicle are known. Therefore, the shooting direction of the infrared camera 200 can be determined from the traveling direction of the vehicle.

[0106] In the third example below, the saturated area detection unit 174 also calculates the shooting direction and angle of view to determine whether the saturation is due to direct light or reflected light, but of course, a separate calculation unit may be provided within the image processing device 100 for this calculation.

[0107] The saturated area detection unit 174 calculates the traveling direction of the vehicle. For example, the saturated area detection unit 174 calculates the traveling direction of the vehicle from a history of GPS data. The saturated area detection unit 174 calculates the traveling direction of the vehicle by comparing continuously acquired position information. Alternatively, the saturated area detection unit 174 may calculate the traveling direction using a gyro sensor or the like mounted on the vehicle or the infrared camera 200. Then, the saturated area detection unit 174 identifies the shooting direction and angle of view of the infrared camera 200 from the traveling direction of the vehicle.

[0108] Furthermore, the saturated region detection unit 174 acquires the conditions of solar irradiation at the shooting position. The saturated region detection unit 174 identifies the position and direction of the sun at the shooting position based on the shooting date and time. The saturated region detection unit 174 can calculate the direction of the sun from the shooting position as the conditions of solar irradiation based on the calendar. The shooting position is acquired by the position information acquisition unit described above.

[0109] Then, the saturated region detection unit 174 determines whether the position of the sun is included in the shooting direction and angle of view of the infrared camera 200. In this way, when a saturated region that meets predetermined conditions exists, the saturated region detection unit 174 determines whether the cause of saturation is direct light or reflected light.

[0110] Next, this determination method will be explained using Fig. 12. Fig. 12 is a schematic diagram for explaining the position of the sun and the shooting direction and angle of view of the infrared camera 200. Fig. 12 schematically shows the traveling direction D of the vehicle V and the position of the sun S, as well as a captured thermal image A13.

[0111] In FIG. 12, the range DR indicated by the shooting direction and angle of view of the infrared camera 200 is the range sandwiched between two dashed lines starting from the vehicle V. The saturated area detection unit 174 can determine whether the sun S is directly irradiated by determining whether the position of the sun S is within this range DR. In the example of FIG. 12, since the sun S is within the range DR, the saturated area detection unit 174 can determine that the sun S is included in the thermal image A13 shown in FIG. 12, and if a saturated area meeting the predetermined conditions exists, the saturated area detection unit 174 can determine that the cause is direct light from the sun S. On the other hand, if the sun S is not within the range DR, the saturated area detection unit 174 can determine that the sun S is not included in the thermal image, and if a saturated area meeting the predetermined conditions exists, the saturated area detection unit 174 can determine that the cause is reflected light from the sun S reflected by another object.

[0112] In the third example, as in the first example, the time for which the attenuation filter 220 is inserted is shortened compared to the first embodiment, thereby shortening the time required to obtain a thermal image with attenuation, thereby protecting the sensor 230. Furthermore, unlike the first example, the determination using the imaging direction, etc., described in the third example can be performed not only when the attenuation filter 220 is inserted, but also when it is retracted. The third example can be used instead of the first example, but by combining it with at least one of the first and second examples, it becomes possible to more accurately determine the cause of saturation. In other words, by combining two or more of the first, second, and third examples, it becomes possible to more accurately determine the cause of saturation.

[0113] (Alternative examples, etc.) Some or all of the processing in the image processing device 100 described above can be implemented as a computer program. The program described above includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0114] The present disclosure is not limited to the above-described embodiment, and can be modified as appropriate within the scope of the present disclosure.

[0115] For example, as has been partially explained regarding the monitoring application, the object on which the infrared camera 200 is mounted is not limited to a vehicle. In other words, the infrared camera 200 is not limited to an in-vehicle camera, but may be any infrared camera that is subject to sunlight incident from outdoors. For example, the infrared camera 200 may be a monitoring camera installed in various facilities. One example of such a facility is a smart pole. Note that when the object on which the infrared camera 200 is mounted is a vehicle, the vehicle is not limited to an automobile, but may be any of various types of moving objects such as a motorcycle, a ship, an airplane, a robot, or a drone.

[0116] Furthermore, in the above embodiment, the imaging system 1 including the display device 300 has been given as an example of incorporating an image processing device, but the image processing device can also be incorporated into a system that does not include a display device. As briefly explained in the first embodiment, the image processing device can be applied to, for example, an object detection system that detects objects from thermal images. In this case, the image processing device may be provided with an image data output unit that outputs thermal images as a component equivalent to the display image data output unit. The object detection system may also be provided with a recognition processing unit that recognizes (detects) objects such as people that meet predetermined conditions, such as shape and movement, based on the thermal images output from the image processing device, and a notification unit that issues a notification when an object is detected.

[0117] The present invention has been specifically described above based on the embodiments, but it goes without saying that the present invention is not limited to the above embodiments and can be modified in various ways without departing from the spirit of the invention. Two or more of the above embodiments can also be combined as appropriate. [Explanation of symbols]

[0118] 1. Imaging system 100 Image processing device 110 Control IF 120 ROM 130 RAM 140 System control section 141 Sensor protection control unit 142 Filter operation control section 160 Thermal image data acquisition unit 170 Image processing section 171 Defective pixel correction unit 172 NUC Department 174 Saturation area detection unit 180 Display image data output unit 200 Infrared Camera 210 Lens 220 Attenuation Filter 230 Sensors 240 Transmission Devices 250 Temperature Sensor 300 indicates device

Claims

1. an image data acquisition unit that acquires a thermal image captured by an infrared sensor that captures a thermal image; a filter operation control unit that controls an insertion operation and a retraction operation of an attenuation filter that attenuates infrared light into an optical path through which infrared light is incident on the infrared sensor; a saturated area detection unit that detects the presence of a saturated area that meets predetermined conditions in the thermal image; a sensor protection control unit that causes the filter operation control unit to insert the attenuation filter when the saturated area detection unit detects a saturated area that satisfies the predetermined condition in the thermal image, and causes the filter operation control unit to retract the attenuation filter when a maximum pixel value of the thermal image with the attenuation filter inserted becomes less than a predetermined threshold; Equipped with the sensor protection control unit changes the predetermined threshold value based on temperature information of the infrared sensor. Image processing device.

2. The sensor protection control unit sets the predetermined threshold to a higher value as the temperature of the infrared sensor increases. The image processing device according to claim 1 .

3. the saturated region detection unit determines whether the saturated region of the predetermined condition is saturated due to direct sunlight or saturated due to reflected light, the sensor protection control unit sets the predetermined threshold to a lower pixel value when the saturation region of the predetermined condition is saturation due to reflected light than when the saturation region is saturation due to direct light.

3. The image processing device according to claim 1 or 2.

4. The infrared sensor captures a thermal image, and the captured thermal image is acquired. Detecting the presence of a saturated area that satisfies a predetermined condition in the thermal image; When a saturated region that satisfies the predetermined condition is detected in the thermal image, an attenuation filter that attenuates the infrared light is inserted into the optical path through which the infrared light is incident on the infrared sensor, and when the maximum pixel value of the thermal image with the attenuation filter inserted becomes less than a predetermined threshold value that is changed based on the temperature information of the infrared sensor, the attenuation filter is retracted. A sensor protection method executed by an image processing device.

Images