Image processing device and image processing method

Abstract

To provide an image processing device and an image processing method that are able to prevent interruption of imaging that results from protection of a sensor by a shutter.SOLUTION: An image processing device according to an embodiment includes: a saturation detection unit 174 configured to detect saturation of pixels in thermal image data; a determination unit 175 configured to determine, based on a tendency of pixel output values of peripheral pixels around a saturated pixel region in which a predetermined number or more of saturated pixels are continuous, whether the saturated pixel region is saturated due to incidence of direct light of sunlight or due to incidence of reflected light of the sunlight; and a shutter control unit 141 configured to control a shutter such that the shutter 220 for blocking the incidence of the light on the sensor is closed when the determination unit 175 determines that the saturation is caused by the incidence of the direct light, and the shutter 220 for blocking the incidence of the light on the sensor is not closed when the determination unit determines that the saturation is caused by the incidence of the reflected light.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an image processing device and an image processing method. [Background technology]

[0002] Patent Document 1 discloses an image processing device that processes imaging data from a far-infrared sensor. This device detects saturated areas where pixels are saturated. The image processing device controls the dynamic range according to the detection result of the saturated area. Then, the image processing device controls the opening and closing of the shutter based on the detection result of the saturated area and the setting of the dynamic range. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-110552 A Summary of the Invention [Problem to be solved by the invention]

[0004] Infrared sensors such as microbolometers used in infrared imaging devices have a problem in that the sensor elements deteriorate when high-intensity light is incident on them. For example, if sunlight is directly incident on an infrared sensor, the pixels will burn in. In this case, the sensitivity of the burned-in pixels will change, causing degradation of image quality.

[0005] In Patent Document 1, the sensor element is protected by closing the shutter. However, there is a problem in that imaging is not possible while the shutter is closed.

[0006] The present disclosure has been made in consideration of the above-mentioned points, and provides an image processing device and an image processing method that can suppress interruption of image capture due to protecting a sensor with a shutter. [Means for solving the problem]

[0007] The image processing device of this embodiment includes a thermal image data acquisition unit that acquires thermal image data captured by a sensor, a saturation detection unit that detects saturation of pixels in the thermal image data, a determination unit that determines whether the saturated pixel area is saturated due to direct incidence of sunlight or due to incidence of reflected sunlight based on the tendency of pixel output values ​​of surrounding pixels surrounding a saturated pixel area where a predetermined number of consecutive saturated pixels are present, and a shutter control unit that controls the shutter so that, when the determination unit determines that the saturation is due to the incidence of direct light, the shutter that blocks light from entering the sensor is closed, and when the determination unit determines that the saturation is due to the incidence of reflected light, the shutter that blocks light from entering the sensor is not closed.

[0008] The image processing method according to the present embodiment includes the steps of acquiring thermal imaging data captured by a sensor, detecting pixel saturation in the thermal imaging data, determining whether the saturated pixel region is saturated due to direct incidence of sunlight or due to reflected incidence of sunlight based on the tendency of pixel output values ​​of surrounding pixels surrounding a saturated pixel region where a predetermined number of consecutive saturated pixels are present, and controlling a shutter to close a shutter that blocks light from entering the sensor when it is determined that the saturation is due to the direct incidence of sunlight, and to not close the shutter that blocks light from entering the sensor when it is determined that the saturation is due to the reflected light. Effect of the Invention

[0009] According to the present disclosure, it is an object to provide an image processing device and an image processing method that can suppress interruption of image capture due to protecting a sensor with a shutter. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a control block diagram showing a configuration of the imaging system. [Diagram 2] FIG. 2 is a control block diagram showing a configuration of an infrared camera. [Diagram 3]FIG. 2 is a control block diagram showing a configuration of an image processing unit. [Figure 4] FIG. 13 is a diagram showing thermal imaging data when direct sunlight is incident; [Diagram 5] FIG. 13 is a diagram showing thermal imaging data when reflected sunlight is incident; [Figure 6] 11 is a graph showing profile data of pixel output values. [Figure 7] 4 is a flowchart showing an image processing method according to the first embodiment. [Figure 8] 11 is a graph showing profile data of pixel output values. [Figure 9] 10 is a flowchart showing an image processing method according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. However, the present disclosure is not limited to the following embodiments. In addition, the following description and drawings are appropriately simplified for clarity of explanation.

[0012] First embodiment An image processing device and an imaging system according to the present embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing a configuration of an imaging system 1. The imaging system 1 includes an image processing device 100, an infrared camera 200, and a display device 300. The imaging system 1 is, for example, an imaging system for vehicles that is used in a vehicle to acquire a thermal image of the surroundings, or an imaging system used for surveillance purposes.

[0013] Here, the imaging system 1 will be described as an in-vehicle system mounted on a vehicle. By mounting the infrared camera 200 on a vehicle, safe driving support can be performed by detecting a person or the like that is a heat source object. The infrared camera 200 is mainly used to support visual safety confirmation at night, etc. The infrared camera 200 is also effective when the amount of ambient light is small, such as around sunrise and sunset or inside a tunnel, other than at night.

[0014] The infrared camera 200 captures a thermal image (infrared image). The infrared camera 200 captures thermal images at, for example, 15 to 30 frames per second. For example, the infrared camera 200 captures a thermal image by detecting far infrared rays from a subject. The infrared camera 200 is a far infrared ray (FIR) camera that can visualize far infrared rays emitted from an object. Data of a thermal image captured by the infrared camera 200 is also called thermal imaging data (or infrared image data). The thermal imaging data indicates the temperature distribution in the imaging range of the infrared camera 200.

[0015] The infrared camera 200 captures an image of the area in front of the vehicle or the periphery of the vehicle. The infrared camera 200 outputs thermal imaging data to the image processing device 100. The thermal imaging data input from the infrared camera 200 to the image processing device 100 is also referred to as input thermal imaging data. The thermal imaging data refers to thermal imaging data captured by the infrared camera. The configuration of the infrared camera 200 will be described later.

[0016] The image processing device 100 performs image processing on the input thermal imaging data to generate display image data or image data for object recognition. For example, the image processing device 100 performs various processes such as correction of pixel variations of the infrared camera 200 and removal of noise caused by changes in characteristics over time. Furthermore, the image processing device 100 functions as a control device for controlling the infrared camera 200. For example, the image processing device 100 controls the shutter of the infrared camera 200. Specifically, when sunlight is incident on the infrared camera 200, the image processing device 100 closes the shutter to protect the infrared camera 200. Furthermore, the image processing device 100 closes the shutter when a sensor 230 (described later) performs calibration by capturing an image of the shutter.

[0017] The image processing device 100 outputs the display image data that has been subjected to image processing to the display device 300. The display image data that has been subjected to image processing by the image processing device 100 and output to the display device 300 is also referred to as output thermal image data. In other words, the input thermal image data is data before processing by the image processing device 100, and the output thermal image data is data after processing by the image processing device 100.

[0018] The display device 300 includes a liquid crystal display or the like for displaying display image data. The display device 300 displays the display image data that has been subjected to image processing in the image processing device 100. This allows the display device 300 to display a more accurate temperature distribution.

[0019] The image processing device 100 performs object recognition processing on the image data that has been subjected to image processing. The image processing device 100 recognizes objects included in the image data using learning models of people, vehicles, etc., and outputs the recognition results to the display device 300 or the like. The recognition results may be output to the display device 300 so as to be superimposed on the output thermal image.

[0020] The image processing device 100 includes a control IF (interface) 110, a ROM (Read Only Memory) 120, a RAM (Random Access Memory) 130, a system control unit 140, a thermal imaging data acquisition unit 160, an image processing unit 170, and a display image data output unit 180.

[0021] The thermal imaging data acquisition unit 160 acquires thermal imaging data captured by the infrared camera 200. For example, it is provided with an IF (interface) for inputting the thermal imaging data from the infrared camera 200. The interface between the thermal imaging data acquisition unit 160 and the infrared camera 200 may be a wired connection or a wireless connection.

[0022] The ROM 120 stores a control program and various parameters for controlling the image processing device 100. For example, the ROM 120 stores an image processing program executed by the image processing unit 170. The ROM 120 also stores a system control program executed by the system control unit 140.

[0023] The RAM 130 stores various programs and parameters used for executing the programs, etc. Furthermore, it stores calculation data of the image processing unit 170, etc.

[0024] The system control unit 140 controls the entire imaging system 1. The system control unit 140 includes a processor such as a CPU (Central Processing Unit). The processor of the system control unit 140 executes a control program to control the entire system.

[0025] The system control unit 140 includes a shutter control unit 141. The shutter control unit 141 controls the opening and closing of the shutter of the infrared camera 200. The shutter control unit 141 outputs an instruction to close the shutter of the infrared camera 200. In other words, the shutter control unit 141 outputs a control signal to open and close the shutter.

[0026] The control IF 110 is an interface for controlling the infrared camera 200. For example, the shutter control unit 141 outputs a control signal for opening and closing the shutter of the infrared camera 200 via the control IF 110. Furthermore, the image processing device 100 can transmit various signals and data to the infrared camera 200 via the control IF 110. Furthermore, the control IF 110 can receive various signals and data from the infrared camera 200. For example, the control IF 110 acquires the environmental temperature measured by a temperature sensor of the infrared camera 200.

[0027] The image processing unit 170 performs a predetermined image processing on the thermal imaging data acquired by the thermal imaging data acquisition unit 160. The image processing unit 170 may include a processor for executing an image processing program. The processor of the image processing unit 170 may be a general-purpose processor or a dedicated processor. The processor of the image processing unit 170 and the processor of the system control unit 140 may be the same. The processing in the image processing unit 170 will be described later.

[0028] The image processing unit 170 performs image processing on the input thermal image data to generate display image data or image data for object recognition. The display image data output unit 180 is an interface that outputs the display image data to the display device 300. This allows a user such as a vehicle driver to check the display image data. Therefore, even in dark conditions such as at night, people around the vehicle can be recognized. In addition, by outputting the object recognition result together with the display image data to the display device 300, the user can check the recognition result of people around the vehicle.

[0029] 2 is a block diagram showing a schematic diagram of a detailed configuration of the infrared camera 200. The infrared camera 200 includes a lens 210, a shutter 220, a sensor 230, a transmission device 240, and a temperature sensor 250.

[0030] The lens 210 forms an image of far-infrared light from a subject on the light receiving surface of the sensor 230. The lens 210 has at least one lens. For example, the lens 210 may include multiple lenses such as a zoom lens and a focus lens.

[0031] The sensor 230 includes a plurality of pixels. Each pixel of the sensor 230 receives far-infrared light from the subject. This allows a thermal image of the subject to be captured. For example, the sensor 230 is a microbolometer for detecting far-infrared light. The sensor 230 includes a plurality of pixels arranged in a two-dimensional array. The detection values ​​(detection signals) of each pixel form a thermal image of the subject.

[0032] The transmission device 240 serves as an interface for transmitting various signals and data to the image processing device 100. The transmission device 240 transmits thermal image data captured by the sensor 230 to the image processing device 100. The transmission device 240 also receives a control signal from the image processing device 100. Furthermore, the transmission device 240 may transmit lens information related to the zoom and focus of the lens 210.

[0033] The temperature sensor 250 measures the temperature in the environment in which the infrared camera is used. The transmission device 240 transmits the environmental temperature measured by the temperature sensor 250 to the image processing device 100. The environmental temperature measured by the temperature sensor 250 is used for shutterless correction, which calibrates the sensor 230 without closing the shutter.

[0034] The shutter 220 is disposed on the front side of the sensor 230. The shutter 220 blocks far-infrared light incident on the sensor 230 from the outside. The shutter 220 has an opening and closing mechanism. The shutter 220 opens and closes by a control signal from the shutter control unit 141. When the shutter 220 is closed (hereinafter, also simply referred to as the closed state), external light from the lens 210 does not enter the sensor 230. That is, in the closed state, far-infrared light from the shutter 220 enters the sensor 230. When the shutter 220 is open (hereinafter, also simply referred to as the open state), far-infrared light from the lens 210 enters the sensor 230. The shutter 220 can make the imaging range of the sensor 230 a thermally uniform surface. For example, the shutter 220 may be something like a barrier or cover disposed on the front side of the lens 210, or may be disposed between the lens 210 and the sensor 230. Closing the shutter can prevent the sensor 230 from burning out.

[0035] The configuration of the image processing unit 170 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing a detailed configuration of the image processing unit 170. The image processing unit 170 includes a defective pixel correction unit 171, a NUC (Non-Uniformity Correction) unit 172, a saturation detection unit 174, and a determination unit 175. The image processing unit 170 in Fig. 3 may include an object recognition unit for image data output from the NUC unit 172. The image processing unit 170 cooperates with the system control unit 140 and the RAM 130 to execute predetermined processing.

[0036] The defective pixel correction unit 171 performs an interpolation process on the pixel value of a defective pixel of the sensor 230 pre-stored in the RAM 130 using pixel values ​​of pixels surrounding the defective pixel. The defective pixel correction unit 171 performs the above-mentioned interpolation process on the input thermal imaging data acquired by the thermal imaging data acquisition unit 160, and outputs the image data after the interpolation process to the NUC unit 172 and the saturation detection unit 174.

[0037] The NUC unit 172 performs processing to correct output variations between pixels. For example, the NUC unit 172 performs shutterless correction based on the environmental temperature detected by the temperature sensor 250. A known method can be used for the shutterless correction in the NUC unit 172, so a description thereof will be omitted. The image processing unit 170 outputs the thermal image data that has been subjected to the NUC correction to the display image data output unit 180 as output thermal image data. The image processing unit 170 may perform AGC processing or the like on the thermal image data that has been subjected to the NUC correction.

[0038] The saturation detection unit 174 detects saturation of pixels in the thermal imaging data. The saturation detection unit 174 identifies pixel addresses of saturated pixels. A saturated pixel is a pixel whose pixel output value (called pixel data) is an upper limit value. For example, when the brightness level of each pixel in the imaging data is expressed in 14 bits from zero to 255, a pixel at a coordinate whose pixel output value is 16383 is called a saturated pixel. Of course, the pixel output value of a saturated pixel is not limited to the upper limit value, and may be a value indicating a predetermined level or higher. This allows the image processing unit 170 to identify a saturated pixel area in the thermal imaging data where pixels are saturated. A saturated pixel area is an area where saturated pixels or substantially saturated pixels exist consecutively.

[0039] The determination unit 175 performs a determination for controlling the opening and closing of the shutter. Specifically, the determination unit 175 determines whether the saturation of a pixel is caused by direct sunlight or reflected sunlight. Direct sunlight refers to sunlight from the sun that is directly incident on the infrared camera 200. Reflected sunlight refers to sunlight from the sun that is reflected by a structure such as a wall or glass and then incident on the infrared camera 200. The determination unit 175 performs a determination based on the tendency of pixel output values ​​of surrounding pixels around the saturated pixel region. The determination process of the determination unit 175 will be described later.

[0040] When direct sunlight is incident on the infrared camera 200, the amount of light incident on the infrared camera 200 becomes extremely high. Therefore, burn-in of the pixels (microbolometers) of the infrared camera 200 occurs. When sunlight is reflected by a window glass and incident on the infrared camera 200, the amount of light incident on the infrared camera 200 becomes lower than that of direct light. Therefore, when pixel saturation occurs due to reflected sunlight, burn-in of the pixels of the infrared camera 200 does not occur. Then, the image processing unit 170, the determination unit 175, determines whether the pixel saturation is due to direct sunlight or reflected sunlight. The image processing unit 170 outputs a determination signal indicating the determination result to the system control unit 140.

[0041] The shutter control unit 141 controls the opening and closing of the shutter 220 in response to a determination signal indicating the determination result. When the determination unit 175 determines that saturation has occurred due to the incidence of direct light, the shutter control unit 141 outputs a control signal to close the shutter 220. This makes it possible to block light from entering the infrared camera 200. When the determination unit 175 determines that saturation has occurred due to the incidence of reflected light, the shutter control unit 141 outputs a control signal not to close the shutter. This makes it possible to continue capturing infrared images of the periphery of the vehicle.

[0042] A method for determining whether pixel saturation in a saturated pixel region is due to direct light or reflected light will be described. Fig. 4 is a schematic diagram showing an image P1 in which the sun is included in the angle of view of the infrared camera 200, and Fig. 5 is a schematic diagram showing an image P2 in which the sun is not included in the angle of view of the infrared camera 200.

[0043] In FIG. 4, the sun S is included in the angle of view, so the sun S and its surrounding area are the saturated pixel area R1. In FIG. 5, the sun is not included in the angle of view, but sunlight is reflected by the window glass of the building O and enters the infrared camera 200. Therefore, the area corresponding to the window glass of the building O is the saturated pixel area R2. In the saturated pixel areas R1 and R2, a predetermined number or more of saturated pixels are present in succession. That is, the saturation detection unit 174 detects an area including a predetermined number or more of consecutive saturated pixels as the saturated pixel areas R1 and R2. For example, when three or more pixels of saturated pixels are consecutive in one horizontal line or one vertical line, the saturation detection unit 174 detects a group of consecutive saturated pixels as a saturated pixel area. Alternatively, when 5 pixels x 5 or more pixels of saturated pixels are consecutive in the horizontal and vertical directions, the group of consecutive saturated pixels is detected as a saturated pixel area. The saturated pixel region is a set of saturated pixels having a number of pixels based on the imaging angle of view of the sun S or the imaging angle of the reflected image of the sun, relative to the imaging angle of view of the infrared camera 200.

[0044] FIG. 6 is a graph showing profile data for an arbitrary line in images P1 and P2. In FIG. 6, the horizontal axis is the pixel address, and the vertical axis is the pixel output value. Here, a horizontal line or a vertical line including the center of the saturated pixel region is selected. Of course, it is not limited to a horizontal line or a vertical line, and it may be profile data for one line in an arbitrary inclination direction. In image P1, the base spreads around the saturated pixel, and in image P2, the base narrows around the saturated pixel. That is, in the case of saturation due to direct sunlight, the profile rises gently in the pixels around the saturated pixel. In the case of saturation due to reflected sunlight, the profile rises sharply in the pixels around the saturated pixel. Therefore, the determination unit 175 uses the difference in the profile shape around the saturated pixel to determine whether the saturation is due to direct sunlight or reflected light. In this embodiment, the determination unit 175 makes the determination based on the spread tendency of the pixel output values ​​of the surrounding pixels of the saturated pixel region.

[0045] As shown in FIG. 6, the number of saturated pixels in one line is A. The number of pixels in one line that is equal to or greater than the set value L is B, which is the number of surrounding pixels. Here, the number of saturated pixels A can be the number of pixels that are continuously saturated. That is, the number of saturated pixels A is the number of pixels included in the saturated pixel area in one line. The number of surrounding pixels B can be the number of pixels that are continuously greater than the set value L in one line. The set value L is 80% of the maximum pixel output value. Here, in image P1, A=8, B=16, and in image P2, A=8, B=9. The determination unit 175 counts the number of saturated pixels A and the number of surrounding pixels B in one line. Then, the determination unit 175 calculates a difference value C by subtracting the number of saturated pixels A from the number of surrounding pixels B. In image P1, C=8, and in image P2, C=1.

[0046] The determination unit 175 determines whether or not the difference value C between the number of surrounding pixels B and the number of saturated pixels A is equal to or greater than a threshold value TH1. If the difference value C is equal to or greater than the threshold value TH1, the determination unit 175 determines that saturation is due to direct sunlight. If the difference value C is less than the threshold value TH1, the determination unit 175 determines that saturation is due to reflected sunlight. Here, TH1=4. Note that the threshold value TH1 and the set value L are not limited to the above values, and appropriate values ​​can be set according to the characteristics of the infrared camera 200, etc.

[0047] In this way, the determination unit 175 determines whether the saturated pixel region is saturated due to the incidence of direct light or the incidence of reflected light, based on the tendency of spread of pixel output values ​​of the neighboring pixels of the saturated pixel region. Specifically, the determination unit 175 obtains the number A of saturated pixels included in the saturated pixel region and the number B of neighboring pixels whose pixel output values ​​are equal to or greater than the set value L, in a profile in an arbitrary direction. If the difference value C between the number A of saturated pixels and the number B of neighboring pixels is equal to or greater than the threshold value TH1, the determination unit 175 determines that the saturated pixel region is saturated due to the incidence of direct light. If the difference value C between the number A of saturated pixels and the number B of neighboring pixels is less than the threshold value TH1, the determination unit 175 determines that the saturated pixel region is saturated due to the incidence of reflected light.

[0048] In this way, the determination unit 175 can appropriately determine whether the saturation is due to the incidence of direct sunlight or the incidence of reflected light. Therefore, when direct light is incident, the shutter 220 can be closed. Since the infrared camera 200 can be protected by the shutter 220, pixel burn-in can be prevented. When no direct light is incident, the shutter 220 can be kept open. Therefore, even when the infrared camera 200 is protected by the shutter 220, thermal imaging data can be appropriately acquired. In other words, the period during which thermal imaging data cannot be acquired can be shortened.

[0049] Next, the image processing method according to the present embodiment will be described. First, the thermal image data acquisition unit 160 acquires thermal image data from the infrared camera 200 (step S101). The saturation detection unit 174 detects saturated pixels in the thermal image data (step S102). Then, the saturation detection unit 174 judges whether or not the saturated pixels are continuous for a predetermined number of pixels or more, in other words, whether or not there is a saturated pixel area based on the sun or a reflected image of the sun (step S103). If the saturated pixels are not continuous for a predetermined number of pixels or more (NO in step S103), the shutter 220 is opened (step S104). That is, the shutter control unit 141 does not close the shutter 220 and maintains the open state.

[0050] If saturated pixels continue for a predetermined number of pixels or more (YES in step S103), the determination unit 175 extracts pixel output values ​​of one line in the saturated pixel region (step S105). That is, if saturated pixels continue for a predetermined number of pixels or more, the saturation detection unit 174 identifies the region including that point as a saturated pixel region. When the determination unit 175 selects one line passing through the saturated pixel region, it is possible to extract a profile of pixel output values ​​as shown in FIG. 5. For example, the determination unit 175 extracts one line having the largest number of consecutive saturated pixels from among the lines passing through the saturated pixel region. The one line to be extracted may be a line along the horizontal or vertical direction of the image, or may be a line tilted from the horizontal and vertical directions.

[0051] Next, the determination unit 175 counts the number A of saturated pixels in one line (step S106). The determination unit 175 counts the number B of surrounding pixels in one line whose pixel output value is equal to or greater than a set value L (step S107). The determination unit 175 determines whether the difference value C (=BA) is equal to or greater than a threshold value TH1 (step S108).

[0052] If the difference value C (=BA) is equal to or greater than the threshold value TH1 (YES in step S108), it is determined that the light is direct sunlight, and the shutter 220 is closed (step S109). That is, the determination unit 175 determines that direct sunlight is incident on the infrared camera 200, and outputs a determination signal indicative of the determination result to the system control unit 140. The shutter control unit 141 outputs a control signal for closing the shutter 220 to the infrared camera 200.

[0053] If the difference value C(BA) is not equal to or greater than the threshold value TH1 (NO in step S108), it is determined that the light is reflected sunlight, that is, light of a reflected image of the sun, and the shutter 220 is maintained open (step S110). In other words, the determination unit 175 determines that reflected sunlight is incident on the infrared camera 200, and outputs a determination signal indicating the determination result to the system control unit 140. The shutter control unit 141 outputs a control signal to the infrared camera 200 to maintain the shutter 220 open rather than closed. Then, the above process is repeated for each frame or at regular intervals. In this manner, even when the infrared camera 200 is protected by the shutter 220, it is possible to appropriately acquire thermal imaging data.

[0054] Even if the shutter 220 is closed (step S109), the acquisition of thermal imaging data may continue. After the shutter 220 is closed, the determination unit 175 outputs an instruction to open the shutter 220 to the shutter control unit 141 after a predetermined period of time, for example, 5 to 10 seconds, has elapsed, to open the shutter 220, and returns to step S102. If the shutter is kept open (steps S104 and S110), the process returns to step S102.

[0055] Embodiment 2 The second embodiment will be described with reference to Fig. 8. Fig. 8 is a graph showing profiles of pixel output values ​​in images P1 and P2. In this embodiment, the determination unit 175 performs a determination in accordance with the gradient of adjacent pixels outside the saturated pixel regions R1 and R2 in a one-line profile. Note that, since the process other than that in the determination unit 175 is the same as that in the first embodiment, a description thereof will be omitted as appropriate.

[0056] The determination unit 175 extracts one line passing through the saturated pixel regions R1 and R2. Then, on both sides of the saturated pixel region, the determination unit 175 obtains the slopes X and Y of the pixel output values. The slope X is the slope of the pixel output values ​​in the adjacent region on the left side of the saturated pixel region, and the slope Y is the slope of the pixel output values ​​in the adjacent region on the right side of the saturated pixel region. It is assumed that the slope X is a positive value and the slope Y is a negative value. The determination unit 175 makes a determination by comparing the absolute values ​​of the slopes X and Y with a threshold value TH2.

[0057] The determination unit 175 calculates the slopes X and Y based on the pixel output values ​​of pixels at the edge of the saturated pixel region R1 and the pixel output values ​​of pixels located a predetermined number of pixels outside the edge of the saturated pixel region R1. Specifically, the slope calculated from the pixels 2 to 7 from the edge of the saturated pixel region R1 is appropriate. Of course, the pixels for which the slope is calculated are not limited to the pixels 2 to 7 from the edge, and the appropriate range changes depending on the number of pixels of the sensor 230. The pixels are assumed to be pixels located an arbitrary predetermined number of pixels t outside the edge of the saturated pixel region R1 from the 2nd pixel to the 7th pixel.

[0058] The determination unit 175 obtains the slopes X and Y by drawing a straight line from the pixel output value of the pixel at the edge of the saturated pixel region R1 to the pixel output value of the t-th pixel. For example, the pixel output value of the pixel at the left edge of the saturated pixel region R1 is R1L, and the pixel output value of the pixel that is the t-th pixel from the pixel at the left edge of the saturated pixel region R1 is DtL. In this case, the slope is expressed as X=(R1L-DtL) / t.

[0059] Similarly, the determination unit 175 calculates the slope Y. For example, the pixel output value of the pixel at the right end of the saturated pixel region R1 is R1R, and the pixel output value of the t-th pixel from the pixel at the right end of the saturated pixel region R1 is DtR. In this case, the slope Y is expressed as (DtL-R1R) / t. The determination unit 175 may also determine the slope from a plurality of pixel output values ​​included in the adjacent region using the least squares method or the like. In other words, the determination unit 175 can also determine the slope from an approximation line that approximates the pixel output values ​​of three or more pixels included in the adjacent region.

[0060] For example, in image P1, the slope X=7.98 and the slope Y=-11.67. In image P2, the slope X=13.76 and the slope Y=-13.25. Therefore, the threshold value TH2=9. When at least one of the absolute values ​​of the slopes X and Y is less than the threshold value TH2, the determination unit 175 determines that the saturation is due to direct light. When both the absolute values ​​of the slopes X and Y are equal to or greater than the threshold value TH2, the determination unit 175 determines that the saturation is due to reflected light. In the case of saturation due to reflected light, the slope becomes steep. That is, in the case of saturation due to reflected light, the absolute values ​​of the slopes X and Y are larger than in the case of saturation due to direct incidence of light. Therefore, the determination unit 175 can make an appropriate determination by comparing the absolute values ​​of the slopes X and Y with the threshold value TH2.

[0061] In this way, the determination unit 175 determines whether the saturated pixel region is saturated due to the incidence of direct light or the incidence of reflected light, based on the tendency of the gradient of the pixel output values ​​in the adjacent region outside the saturated pixel region. Specifically, the determination unit 175 calculates the gradient of the pixel output values ​​in the adjacent region based on the pixel output values ​​of the pixels at the edge of the saturated pixel region and the pixel output values ​​of the pixels located a predetermined number of pixels outside the edge of the saturated pixel region, in a profile in an arbitrary direction. Then, the determination unit 175 compares the absolute value of the gradient with the threshold value TH2.

[0062] If the absolute value of the slope is less than the threshold TH2, the determination unit 175 determines that the saturated pixel region is saturated due to the incidence of direct light, and if the absolute value of the slope is equal to or greater than the threshold TH2, the determination unit 175 determines that the saturated pixel region is saturated due to the incidence of reflected light.

[0063] Fig. 8 is a flowchart showing an image processing method according to this embodiment. In Fig. 9, steps S106 to S110 in the flowchart of Fig. 7 are replaced with steps S206 to S209. Steps S101 to S105 are the same as those in the first embodiment, so a description thereof will be omitted.

[0064] The determination unit 175 calculates the slopes X and Y of the adjacent region outside the saturated pixel region (step S206). Then, the determination unit 175 determines whether the absolute values ​​of the slopes X and Y are less than a threshold value TH2 (step S207). Here, the determination unit 175 calculates the slopes X and Y by connecting the pixel output values ​​of the two pixels with a straight line.

[0065] If the absolute values ​​of the slopes X and Y are less than the threshold value TH2 (YES in step S207), it is determined that the light is direct light, and the shutter 220 is closed (step S208). That is, the determination unit 175 determines that direct sunlight is incident on the infrared camera 200, and outputs a determination signal indicative of the determination result to the system control unit 140. The shutter control unit 141 outputs a control signal for closing the shutter 220 to the infrared camera 200.

[0066] If the absolute values ​​of the tilts X and Y are not less than the threshold value TH2 (NO in step S207), it is determined that the light is reflected, and the shutter 220 is opened (step S209). In other words, if the absolute values ​​of the tilts X and Y are equal to or greater than the threshold value TH2, the determination unit 175 determines that reflected sunlight is incident on the infrared camera 200. The determination unit 175 outputs a determination signal indicating the determination result to the system control unit 140. The shutter control unit 141 outputs a control signal to the infrared camera 200 to keep the shutter 220 open. Then, the above process is repeated for each frame or at regular time intervals. In this way, even when the infrared camera 200 is protected by the shutter 220, it is possible to appropriately acquire thermal imaging data.

[0067] In this way, the elements of the infrared camera 200 can be protected more appropriately. Furthermore, even if pixel saturation is detected, the period during which imaging can be performed can be extended. Note that the object on which the infrared camera 200 is mounted is not limited to a vehicle. In other words, the infrared camera 200 is not limited to an in-vehicle camera. For example, the infrared camera 200 may be a surveillance camera installed in a facility.

[0068] Also, a part or all of the processing in the image processing device 100 described above can be realized as a computer program. Such a program can be stored using various types of non-transitory computer-readable media and supplied to a computer. The non-transitory computer-readable medium includes various types of tangible recording media. Examples of the non-transitory computer-readable medium include semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)). The program may also be supplied to a computer by various types of temporary computer-readable media. Examples of the temporary computer-readable medium include electric signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to a computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

[0069] Although the present invention has been specifically described based on the embodiments, it goes without saying that the present invention is not limited to the above-mentioned embodiments and can be modified in various ways without departing from the spirit of the invention. Two or more of the above-mentioned embodiments can also be appropriately combined. [Explanation of symbols]

[0070] 1. Imaging system 100 Image processing device 110 Control IF 120 ROM 130 RAM 140 System control section 141 Shutter control section 160 Thermal image data acquisition unit 170 Image Processing Unit 171 Defective pixel correction section 172 NUC Department 174 Saturation detector 175 Judgment section 200 Infrared Camera 210 Lens 220 Shutter 230 Sensors 240 Transmission Devices 250 Temperature Sensor 300 display device

Claims

1. a thermal image data acquisition unit for acquiring thermal image data captured by the sensor; a saturation detection unit for detecting saturation of pixels in the thermal imaging data; a determination unit that determines whether a saturated pixel region is saturated due to direct incidence of sunlight or due to incidence of reflected sunlight based on a tendency of pixel output values ​​of surrounding pixels around a saturated pixel region including a predetermined number or more of consecutive saturated pixels; an image processing device comprising: a shutter control unit that controls a shutter so as to close a shutter that blocks light from entering the sensor when the judgment unit judges that saturation is due to the incidence of direct light, and so as not to close the shutter that blocks light from entering the sensor when the judgment unit judges that saturation is due to the incidence of reflected light.

2. The image processing device according to claim 1 , wherein the determining unit determines whether the saturated pixel region is saturated due to incidence of the direct light or the reflected light based on a tendency of spread of pixel output values ​​of pixels surrounding the saturated pixel region.

3. The determination unit is In a profile in an arbitrary direction, a number of saturated pixels included in the saturated pixel region and a number of surrounding pixels having pixel output values ​​equal to or greater than a pixel output threshold are calculated; When a difference value obtained by subtracting the number of saturated pixels from the number of surrounding pixels is equal to or greater than a first threshold value, it is determined that the saturated pixel region is saturated due to the incidence of direct light; The image processing apparatus according to claim 2 , wherein when the difference value is less than a first threshold value, it is determined that the saturated pixel area is saturated due to incidence of the reflected light.

4. The image processing device according to claim 1 , wherein the determination unit determines whether the saturated pixel region is saturated due to the incidence of direct light or the incidence of reflected light based on a tendency of a gradient of pixel output values ​​in an adjacent region outside the saturated pixel region.

5. The determination unit is calculating a gradient of pixel output values ​​in the adjacent region based on pixel output values ​​of pixels at an edge of the saturated pixel region and pixel output values ​​of pixels located a predetermined number of pixels outside the edge of the saturated pixel region in a profile in an arbitrary direction; If the absolute value of the gradient is less than a second threshold, it is determined that the saturated pixel region is saturated due to the incidence of direct light; The image processing device according to claim 4 , wherein when the absolute value of the gradient is equal to or greater than a second threshold value, it is determined that the saturated pixel region is saturated due to incidence of reflected light.

6. acquiring thermal imaging data captured by a sensor; detecting pixel saturation in the thermal imaging data; determining whether a saturated pixel region is saturated due to direct incidence of sunlight or due to incidence of reflected sunlight based on a tendency of pixel output values ​​of surrounding pixels around a saturated pixel region including a predetermined number or more of consecutive saturated pixels; and controlling a shutter to close the shutter that blocks light from entering the sensor when it is determined that saturation is due to the incidence of direct light, and to not close the shutter that blocks light from entering the sensor when it is determined that saturation is due to the incidence of reflected light.

Images