Image processor, image processing method, and program

Abstract

To reduce time for closing a shutter when protecting a sensor for capturing a thermal image by closing the shutter.SOLUTION: An image processor 100 includes: a saturation detection part 140; a determination part 150; and a shutter control part 160 for controlling operation of a shutter 901 for shading incidence of light to a sensor for capturing a thermal image. The saturation detection part 140 detects a predetermined saturation pixel as a saturation pixel that satisfies a predetermined condition from thermal image data acquired by a sensor. The determination part 150 determines whether a damaged pixel exists within a range in which the saturation detection part 140 detects a predetermined saturation pixel from thermal image data acquired in a state in which incidence of light to the sensor is shaded by the shutter 901. The shutter control part 160 closes the shutter 901 when the saturation detection part 140 detects a predetermined saturation pixel, and performs control for opening the shutter 901 at timing based on a determination result of the determination part 150.SELECTED DRAWING: Figure 1

Description

Technical Field

[0006] , ,

[0005] , , ,

[0007] ,

[0001] The present disclosure relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] A system has been proposed in which an infrared camera is mounted on a vehicle to detect pedestrians and the like. Since the infrared camera captures a thermal image of the shooting range, when an object having a temperature above the upper limit of the set dynamic range is captured, the pixels are saturated, and when the sun enters the shooting range, the pixels in the range of the sun are damaged. For this reason, techniques for protecting the infrared sensor from capturing the sun have also been proposed.

[0003] For example, Patent Document 1 discloses a technique for protecting a sensor that closes a shutter to acquire a thermal image when saturated pixels that may be the sun are detected.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the technique described in Patent Document 1, it is possible to protect a sensor that captures a thermal image using a shutter when the sun is included in the shooting range. However, during the period when the shutter is closed for protecting the sensor, it is impossible to detect an object such as a pedestrian.

[0006] Therefore, a technique for minimizing the time for closing the shutter for protecting the sensor that captures the thermal image is required.

[0007] This disclosure was made to solve such problems and aims to provide an image processing device, an image processing method, and a program that can shorten the time it takes to close the shutter when protecting a sensor that captures thermal images by closing the shutter. [Means for solving the problem]

[0008] The image processing apparatus according to this disclosure includes a saturation detection unit that detects predetermined saturation pixels, which are saturation pixels that satisfy predetermined conditions, from thermal image data acquired by a sensor that captures thermal images, and a shutter control unit that controls the operation of a shutter that shields the sensor from incident light. The image processing apparatus includes a determination unit that determines whether or not damaged pixels exist in the range in which the saturation detection unit has detected the predetermined saturation pixels, from the thermal image data acquired while the shutter is shielding the sensor from incident light. The shutter control unit closes the shutter when the saturation detection unit detects the predetermined saturation pixels, and controls the shutter to open at a timing based on the determination result of the determination unit.

[0009] The image processing method according to this disclosure includes a saturation detection step for detecting a predetermined saturation pixel, which is a saturation pixel that satisfies predetermined conditions, from thermal image data acquired by a sensor that captures thermal images. The image processing method includes a step of controlling the closing of a shutter that blocks the incidence of light to the sensor when the predetermined saturation pixel is detected in the saturation detection step. The image processing method includes a determination step for determining whether or not a damaged pixel exists in the range in which the predetermined saturation pixel was detected in the saturation detection step, from the thermal image data acquired while the incidence of light to the sensor is blocked by the shutter. The image processing method includes a step of controlling the opening of the shutter at a timing based on the determination result in the determination step.

[0010] The program relating to this disclosure causes a computer to perform the following processes. The process includes a saturation detection step of detecting a predetermined saturation pixel that satisfies predetermined conditions from thermal image data acquired by a sensor that captures thermal images. The process includes a step of controlling the closing of a shutter that blocks the incidence of light to the sensor when the predetermined saturation pixel is detected in the saturation detection step. The process includes a determination step of determining whether or not there are damaged pixels in the range where the predetermined saturation pixel was detected in the saturation detection step, from the thermal image data acquired while the incidence of light to the sensor is blocked by the shutter. The process includes a step of controlling the opening of the shutter at a timing based on the determination result in the determination step. [Effects of the Invention]

[0011] According to this disclosure, it is possible to provide an image processing device, an image processing method, and a program that can shorten the time it takes to close the shutter when protecting a sensor that captures a thermal image by closing the shutter. [Brief explanation of the drawing]

[0012] [Figure 1] This is a block diagram showing an example configuration of a thermal image display system including an image processing device according to Embodiment 1. [Figure 2] This is a schematic diagram showing an example of imaging data acquired before and after shutter closing control in the image processing device shown in Figure 1. [Figure 3] Figure 1 is a schematic diagram showing an example of imaging data acquired after shutter closing control in the image processing device. [Figure 4] This is a flowchart illustrating an example of the process performed by the image processing device shown in Figure 1. [Figure 5] This is a flowchart illustrating an example of processing performed by the image processing device according to Embodiment 2. [Figure 6] This is a flowchart illustrating another example of processing performed by the image processing device according to Embodiment 2. [Modes for carrying out the invention]

[0013] The embodiments of the invention will be described below, but the invention claimed in the patent is not limited to these embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential for solving the problem. For clarity of explanation, the following descriptions and drawings have been omitted and simplified as appropriate.

[0014] <Embodiment 1> Embodiment 1 will now be described with reference to the drawings. Figure 1 is a block diagram showing an example configuration of a thermal image display system including an image processing device according to Embodiment 1. The image processing device 100 shown in Figure 1 is a component of a thermal image display system 10 that captures thermal images, such as thermal images of the surroundings of an automobile, and displays them on a monitor. The thermal image display system 10 can mainly consist of the image processing device 100 according to this embodiment, a microbolometer 900, a shutter 901, and a monitor 902.

[0015] First, we will explain the general overview of each component connected to the image processing device 100. The microbolometer 900 is an example of a sensor that captures thermal images, and is composed of infrared detection elements arranged in a matrix, which detect far-infrared rays. In other words, the microbolometer 900 is a type of far-infrared sensor. The microbolometer 900 also converts the detected far-infrared rays into photoelectric data to generate imaging data, and supplies the generated imaging data to the defective pixel correction unit 110 of the image processing device 100, which will be described later.

[0016] The microbolometer 900 can be installed on a moving object, for example, with its infrared detection element surface facing the object to capture a thermal image of the surrounding area. For example, if the moving object is a car, the microbolometer 900 is installed facing forward of the car so that it can capture images in the direction of the car's movement. However, the microbolometer 900 may be installed facing in other directions.

[0017] The shutter 901 includes a mechanism for blocking the incidence of light to the microbolometer 900, and allows or suppresses the incidence of external light to the microbolometer 900. The operation of the shutter 901 is controlled by a shutter control unit 160 described later. When the shutter 901 is open, the shutter 901 allows external light to pass through and be received by the microbolometer 900. When the shutter 901 is closed, the shutter 901 blocks external light and protects the microbolometer 900 from external light.

[0018] The monitor 902 is a display device installed to be able to present information to the user, and includes, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. The monitor 902 is connected to an AGC processing unit 130 described later of the image processing apparatus 100, receives imaging data from the AGC processing unit 130, and displays the received imaging data. The monitor 902 may display, in addition to the imaging data, an image indicating a recognized person. Note that the monitor 902 is not an essential component in the present embodiment.

[0019] The image processing apparatus 100 will be described below. The image processing apparatus 100 can include an arithmetic unit such as a CPU (Central Processing Unit) or an MCU (Micro Controller Unit). In addition to the above arithmetic unit, the image processing apparatus 100 can include a non-volatile or volatile memory such as a flash memory or a DRAM (Dynamic Random Access Memory), and a control board constituted by other electric circuits. A program is incorporated into these arithmetic units and the like, and the functions of each part described below can be realized by hardware or a combination of a program and hardware.

[0020] The image processing device 100 acquires thermal image data (hereinafter referred to as captured data), which is the data of the thermal image captured by the microbolometer 900, and performs a preset process on the acquired captured data. The microbolometer 900 is an example of a sensor that captures a thermal image. The image processing device 100 can include an image data acquisition unit that acquires captured data from a sensor that captures a thermal image in this way. Also, the image processing device 100 can output the captured data on which the preset process has been performed to the monitor 902.

[0021] The image processing device 100 can include a defective pixel correction unit 110, a pixel non-uniformity correction unit 120, an AGC processing unit 130 (AGC = Auto-Gain-Control), a saturation detection unit 140, a determination unit 150, and a shutter control unit 160. Also, the image processing device 100 can include other functions, such as a function of recognizing a person or the like based on the thermal image captured by the microbolometer 900.

[0022] The function of recognizing a person by the image processing device 100 can, for example, recognize the person captured as a thermal image by using a recognition model obtained by machine learning on the thermal images of people and animals for the acquired thermal image. These recognition results can be displayed on the monitor 902 together with the captured data based on the captured thermal image, thereby presenting the presence of a person to a driver of a vehicle or the like. The image processing device 100 can be configured to be able to recognize animals or the like in addition to people.

[0023] The defective pixel correction unit 110 receives captured data from the microbolometer 900 and corrects the data due to defective pixels included in the captured data. For example, the defective pixel correction unit 110 detects the coordinates of the defective pixels in a state where the shutter 901 is closed, and stores the detected coordinates of the defective pixels. In the case of this example, although not shown, it is preferable to configure the defective pixel correction unit 110 to receive a signal for controlling the opening and closing of the shutter 901 from the shutter control unit 160.

[0024] Furthermore, the defective pixel correction unit 110 corrects the stored data of defective pixels by generating interpolation data from the data of adjacent pixels. The defective pixel correction unit 110 supplies the corrected imaging data to the pixel non-uniformity correction unit 120 and the saturation detection unit 140, respectively. The correction method used by the defective pixel correction unit 110 is not limited, and the image processing device 100 can also be configured without the defective pixel correction unit 110.

[0025] The pixel non-uniformity correction unit 120 corrects the non-uniform state of pixels in the imaging data received from the defective pixel correction unit 110. The non-uniform state of each pixel occurs due to variations in the characteristics of each pixel, etc. The pixel non-uniformity correction unit 120 corrects the imaging data by adjusting the gain and offset of each pixel using a method called NUC (Non-Uniformity Correction), for example. The pixel non-uniformity correction unit 120 supplies the corrected imaging data to the AGC processing unit 130. The correction method used by the pixel non-uniformity correction unit 120 is not limited, and the image processing device 100 may not include the pixel non-uniformity correction unit 120.

[0026] The AGC processing unit 130 receives imaging data from the pixel non-uniformity correction unit 120 and adjusts the contrast of the imaging data. By adjusting the contrast of the imaging data, the AGC processing unit 130 generates an image that is easy for the user to view when displaying the thermal image on the monitor 902. Furthermore, by adjusting the contrast of the imaging data, the AGC processing unit 130 can output a thermal image that is appropriate for recognition processing when performing object recognition processing on the thermal image. The AGC processing unit 130 applies histogram equalization techniques to the imaging data, such as contrast-limited adaptive histogram equalization.

[0027] The saturation detection unit 140 detects predetermined saturated pixels from the imaging data acquired by the microbolometer 900. Here, predetermined saturated pixels are saturated pixels that satisfy predetermined conditions, and the predetermined conditions and saturated pixels will be described later. In the configuration example in Figure 1, the saturation detection unit 140 receives imaging data from the defective pixel correction unit 110 and detects predetermined saturated pixels from the received imaging data (corrected imaging data).

[0028] When the saturation detection unit 140 detects a predetermined saturated pixel, it supplies a signal indicating the detection result to the determination unit 150 and the shutter control unit 160. The signal supplied to the determination unit 150 includes information indicating the position of the predetermined saturated pixel, or the pixel value (luminance value) of the predetermined saturated pixel, so that the determination unit 150 can refer to it.

[0029] The shutter control unit 160 controls the operation of the shutter 901. As described above, the shutter 901 is equipped with a mechanism to block the incidence of light onto the microbolometer 900. The shutter control unit 160 then controls the shutter 901 to close when the saturation detection unit 140 detects a predetermined saturated pixel.

[0030] Next, we will explain saturated pixels. A saturated pixel can be defined as a pixel that is completely saturated, or as a pixel that is both completely saturated and substantially saturated. A completely saturated pixel is a pixel in which the pixel data has reached its upper limit, and is a pixel that exceeds the dynamic range in which the amount of far-infrared radiation emitted by the subject can be captured as a temperature difference.

[0031] For example, if the sensor that captures thermal images is the microbolometer 900 exemplified, the resistance of a pixel that receives far-infrared radiation changes according to the intensity of the far-infrared radiation, and this is converted into an electrical signal. By converting this electrical signal into a digital signal and performing signal processing, the temperature of the subject is visualized. The saturation detection unit 140 can detect pixels with a value greater than or equal to the maximum value of the digital signal during this process, for example, pixels with a value of 3FFFh or more in the case of 14 bits, as pixels that are completely saturated. However, the bit width of the digital signal varies depending on the system.

[0032] Next, the predetermined conditions and an example of the detection process of predetermined saturated pixels in the saturation detection unit 140 using the predetermined conditions will be explained with reference to Figure 2. Figure 2 is a schematic diagram showing an example of imaging data acquired before and after shutter closing control in the image processing device 100 of Figure 1.

[0033] The saturation detection unit 140 can, for example, detect saturated pixels from the imaging data received from the defective pixel correction unit 110, and extract saturated pixels as predetermined saturated pixels if there are N or more adjacent saturated pixels, for example, N being an integer of 4 or more. Specifically, if fewer than four pixels, such as one pixel, indicate a saturated value, the saturation of those pixels is treated as noise. The saturation detection unit 140 can extract a group of pixels consisting of multiple saturated pixels from the imaging data received from the defective pixel correction unit 110, and output the extracted group of pixels as predetermined saturated pixels.

[0034] The saturation detection unit 140 may use the position of a pixel in the imaging data as a predetermined condition for extracting saturated pixels. Taking the case where the thermal image display system 10 is mounted on an automobile as an example, here are some examples of conditions regarding the position of a target pixel in the imaging data that can be set as a predetermined condition or included as a predetermined condition.

[0035] In the in-vehicle microbolometer 900, sunlight is expected to often occur on the upper part of the infrared detection element surface. On the other hand, in the microbolometer 900, pixels may become saturated even by high-temperature areas other than sunlight, such as the muffler of the vehicle in front, which are unlikely to damage the pixels. As mentioned above, in this embodiment, the shutter 901 is controlled to close when a predetermined saturated pixel is detected, so if the shutter 901 is closed even by high-temperature areas other than sunlight, the frequency of closing will increase.

[0036] Therefore, it is desirable that the microbolometer 900 be protected by the closing control of the shutter 901 primarily when sunlight is detected. For this reason, the predetermined conditions can be such that only sunlight can be detected by some means, that is, conditions that exclude pixel saturation caused by factors other than sunlight.

[0037] For example, if the saturation detection unit 140 detects a saturated pixel in the upper region (the upper 1 / 3 of the entire region) of the infrared detection element surface which has been divided into three parts (upper, middle, and lower), it can estimate that the saturated pixel is a pixel resulting from sunlight incident on it.

[0038] As an example, let's consider the case where the imaging data acquired from the in-vehicle microbolometer 900 is as shown in thermal image A11 in Figure 2. The example thermal image A11 is a single frame image of imaging data taken in the direction of travel of a vehicle equipped with the thermal image display system 10.

[0039] Thermal image A11 includes images of another vehicle V01 and also contains a group of saturated pixels H13 in the upper left due to the sun being captured. Thermal image A11 detects electromagnetic waves emitted by objects, so areas with high temperatures are displayed brightly. Therefore, thermal image A11 includes a group of saturated pixels H12 due to the capture of the muffler, which is particularly hot among the other vehicle V01. Here, we will assume that the pixels corresponding to the muffler and the sun in thermal image A11 are saturated pixels, and will proceed with the following explanation.

[0040] In this case, as illustrated by the dashed lines in thermal image A11, the thermal image shown by the imaging data is divided into three regions: upper, central, and lower, with the upper region set as the area where sunlight may be incident. In other words, a predetermined condition is included, such as the pixel being included in the upper region.

[0041] Thus, when the image processing device 100 is applied to an automobile and the above assumptions are adopted, the predetermined condition can be that the saturated pixels are those included in the upper region of the thermal image indicated by the imaging data.

[0042] The saturation detection unit 140 can then detect saturated pixels from the imaging data received from the defective pixel correction unit 110, and extract and output pixels located in the upper region of the thermal image A11 as predetermined saturated pixels from among the detected saturated pixels. Alternatively, the saturation detection unit 140 can detect saturated pixels from the group of pixels in the upper region of the thermal image A11 indicated by the imaging data received from the defective pixel correction unit 110, and output the detected saturated pixels as predetermined saturated pixels.

[0043] As a result, in a vehicle equipped with the thermal image display system 10, the saturated pixel group H12 that may occur when imaging the muffler of the vehicle in front is not determined to be a predetermined saturated pixel, and the shutter 901 does not need to be closed simply because such saturated pixels are detected.

[0044] The predetermined conditions can be conditions unrelated to the degree of saturation, as illustrated here, but they can also be conditions related to the degree of saturation, or a combination of conditions related to the degree of saturation and conditions unrelated to the degree of saturation. In the following, for the sake of simplicity, we will explain the case where the condition related to the degree of saturation is the predetermined condition itself. However, if other conditions are also included in the predetermined conditions, it is best to determine that a pixel is saturated when all the conditions included in the predetermined conditions are met.

[0045] Furthermore, as a condition for the degree of saturation, a condition can be adopted in which M is an integer of 2 or more, and M or more adjacent pixels are substantially saturated. For example, M can be 9, and substantially saturated means that the pixel value has a brightness value of 98% or more of the upper limit. In this case, the saturation detection unit 140 can detect a group of pixels as predetermined saturated pixels when 9 or more adjacent pixels have a brightness value of 98 percent or more of the upper limit.

[0046] Next, the determination unit 150 will be described. The determination unit 150 determines whether or not damaged pixels exist in the range where the saturation detection unit 140 has detected predetermined saturated pixels, based on the imaging data acquired while the incidence of light to the microbolometer 900 is blocked by the shutter 901. This imaging data is the data of the shutter 901. The above range refers to the range in which saturation satisfying predetermined conditions is detected, and can refer to the same position as the predetermined saturated pixels, for example. Alternatively, the above range can refer to a region of a predetermined shape, such as a rectangle, surrounding a predetermined group of saturated pixels, or a region consisting of a predetermined group of saturated regions and the adjacent groups of pixels around it.

[0047] Let me add some information about the detection of damaged pixels. Since the shutter 901 is basically at a uniform temperature, if the infrared detection element is not damaged, the thermal image when the shutter 901 is closed will have a uniform output across the entire screen. On the other hand, for example, in thermal image A12 in Figure 2, the pixel value corresponding to the position of the solar saturation pixel group H13 in thermal image A11 is an abnormal value, indicating that it is a damaged pixel group D23. Thus, pixels corresponding to infrared detection elements that have been damaged to some extent will show abnormal values ​​(output levels), resulting in a difference when compared to the pixel values ​​of normal pixels, such as surrounding normal pixels.

[0048] Therefore, the larger the difference in pixel values, the greater the damage to the pixel can be determined. Using this, the determination unit 150 can determine whether a pixel is damaged by applying a threshold to the difference in pixel values ​​compared to a normal pixel. This thresholding determines whether the value exceeds a predetermined value, and if it does, it is determined to be a damaged pixel. Furthermore, by setting multiple predetermined values ​​as thresholds, the level of damage can also be determined.

[0049] If the determination unit 150 also receives the pixel values ​​of unsaturated pixels from the saturation detection unit 140, it can determine the pixel values ​​of the surrounding normal pixels to be compared from among the pixel values ​​of the unsaturated pixels. For example, the pixel values ​​to be compared can be determined by calculating statistical values ​​(mean, median, minimum, etc.) of the unsaturated pixels.

[0050] Alternatively, the pixel values ​​to be compared can be the statistical values ​​of all pixels included in the target thermal image (frame image of the captured data), i.e., the frame statistics. Alternatively, the pixel values ​​to be compared can be the statistical values ​​of at least one of the pixel groups, such as the central pixel group and the lower region pixel group, as exemplified in thermal image A11 in Figure 2, or the pixel values ​​of predetermined positions within the central and lower region pixel groups.

[0051] As a simpler example, the determination unit 150 can also determine that a predetermined saturated pixel to be determined has suffered some degree of damage if its pixel value is greater than or equal to a predetermined value.

[0052] The shutter control unit 160 then controls the shutter 901 to open at a timing based on the determination result of the determination unit 150. In other words, when the saturation detection unit 140 detects a saturated pixel (a predetermined saturated pixel) that satisfies the predetermined conditions and controls the shutter 901 to close, the shutter control unit 160 controls the shutter 901 to open at a timing based on the determination result of the determination unit 150. Thus, the image processing apparatus 100 according to this embodiment functions as a shutter control device that controls the shutter 901.

[0053] The determination result of the determination unit 150 is the result of the determination unit 150 determining the presence or absence of damaged pixels in the imaging data acquired with the shutter 901 closed. Therefore, the timing based on the determination result can be, for example, a timing corresponding to the presence or absence of damaged pixels.

[0054] The timing for opening the shutter based on the determination result by the determination unit 150 will be explained with reference to Figure 3, citing an example of damaged pixel recovery. Figure 3 is a schematic diagram showing an example of imaging data acquired after shutter closing control in the image processing device 100 of Figure 1, and is a schematic diagram showing an example of how damaged pixels are recovered in the imaging data acquired after shutter closing control.

[0055] For example, when thermal image A11 in Figure 2 is acquired and the pixel group corresponding to the sun is detected as a predetermined saturated pixel H13, the shutter control unit 160 controls the shutter 901 to close, and the shutter 901 is closed. Thermal image A12, captured with the shutter 901 closed in this state, includes the damaged pixel group D23. Note that in Figures 2 and 3, thermal image A12 is an example of an image acquired several frames after the acquisition of thermal image A11, which is the time required for detection.

[0056] As time progresses, thermal image A12, which includes the damaged pixel group D23, changes in a way that repairs the damage, as shown in Figure 3, resulting in thermal images A13 including the damaged pixel group D33, thermal image A14 including the damaged pixel group D43, and so on. In Figure 3, as an example, thermal images A13 and A14 are shown as thermal images acquired 20 seconds and 2 minutes after thermal image A12 was acquired, respectively.

[0057] In Figure 3, it can be seen that as time progresses, the amount of white color decreases in the damaged pixel group D23, then D33, and finally D43, meaning that the white dots gradually become lighter. Note that in Figure 3, damaged pixel group D43 is depicted brighter for convenience, but this indicates that the damage has almost recovered.

[0058] As illustrated in Figure 3, damage recovers over time. Therefore, the shutter control unit 160 should control the shutter 901 to open when there are damaged pixels and a sufficient amount of time has elapsed for their recovery to be expected.

[0059] For example, the shutter control unit 160 can control the shutter 901 to open at a timing when it is expected that the pixel damage will reach a state as shown by the damaged pixel group D33 in the thermal image A13 (20 seconds later in the example above). To elaborate, for example, the detection process of predetermined saturated pixels by the saturation detection unit 140, that is, the determination of whether or not sunlight is incident, can be performed over several frames of the imaging data, and the shutter 901 can be closed immediately afterward. Therefore, the time during which sunlight causes damage to the infrared detection element is limited to the time corresponding to those few frames, and the damage is not serious. For example, it is possible to recover to some extent by closing the shutter 901 for a few seconds or about 20 seconds. Note that some damage may not improve even after a week, but some recovery can be expected by closing the shutter 901 for about 20 seconds.

[0060] Alternatively, the shutter control unit 160 can control the shutter 901 to open at a time when it is expected that the pixel damage will reach a state as shown in the damaged pixel group D43 of thermal image A14 (2 minutes later in the example above). Alternatively, the shutter control unit 160 can control the shutter 901 to open at a time when it is expected that the damage will result in a completely black thermal image that is not shown.

[0061] In these examples, the expected timing can be predetermined as the elapsed time from the time the shutter 901 is closed or from the time it is closed and imaging data is acquired. The elapsed time can also be predetermined as the number of frames corresponding to that time. The timing at which the shutter control unit 160 opens the shutter 901 is not limited to these examples; it can be predetermined as the elapsed time from the time the shutter 901 is closed, etc. Furthermore, this elapsed time can also be predetermined as a time corresponding to the determined damage level.

[0062] In all the examples for each timing, the explanation was based on the assumption that damage pixels exist, but if no damage pixels exist, shutter 901 can be opened without considering damage recovery.

[0063] Thus, if the determination unit 150 determines that a damaged pixel exists, the shutter control unit 160 can perform control to open the shutter 901 after a longer period of time has elapsed compared to when no damaged pixel is determined to exist. This control prevents the shutter 901 from remaining closed for an unnecessarily long time when no damaged pixels exist.

[0064] Next, with reference to Figure 4, an example of processing performed by the image processing device 100 with the above configuration will be described. Figure 4 is a flowchart illustrating an example of processing performed by the image processing device 100 in Figure 1.

[0065] The flowchart shown in Figure 4 begins when the image processing device 100 receives imaging data from the microbolometer 900. In other words, when the image processing device 100 receives imaging data from the microbolometer 900, the saturation detection unit 140 starts detecting predetermined saturated pixels from the imaging data corrected by the defective pixel correction unit 110 (for example, thermal image A11 in Figure 2) (step S10).

[0066] Next, the image processing device 100 determines whether or not a predetermined saturated pixel has been detected in the captured data (step S11). If it is determined that no predetermined saturated pixels have been detected in the captured data (step S11: NO), the process proceeds to step S16.

[0067] If the image processing device 100 determines that a predetermined saturated pixel has been detected in the imaging data (step S11: YES), it closes the shutter 901 (step S12). Then, in this state, the image processing device 100 causes the microbolometer 900 to take an image with the shutter 901 open to acquire imaging data (step S13).

[0068] If the thermal image A11 in Figure 2 is the target for detection of predetermined saturation pixels, the imaging data acquired in step S13 will be the data shown in thermal image A12 in Figures 2 and 3, for example. Thermal image A12 includes a group of damaged pixels D23 based on the group of saturation pixels H13 that resulted from imaging the sun as shown in thermal image A11.

[0069] Following step S13, the image processing device 100 determines whether or not there are damaged pixels in the range where predetermined saturated pixels of the acquired imaging data have been detected (step S14). If thermal image A12 is acquired, the result in step S14 is YES, and if a uniform thermal image without damage is acquired, the result in step S14 is NO.

[0070] If it is determined that no damaged pixels exist within the above range (step S14: NO), the image processing device 100 opens the shutter 901 immediately or after a period t1 has elapsed since the shutter 901 was closed (step S15).

[0071] In step S15, "immediately" means that if NO is determined in step S14, the shutter 901 will be opened immediately. In step S15, "after a period t1 has elapsed" means that if NO is determined in step S14, the shutter 901 will be opened after a period t1 has elapsed from the time the shutter 901 was closed in step S12. The value of t1 can be predetermined as the maximum time required to detect damage, and can be set to, for example, about 0.1 seconds.

[0072] After step S15, the image processing device 100 determines whether to terminate the process by detecting whether it has stopped receiving imaging data from the microbolometer 900, such as by receiving an instruction to terminate imaging, or whether it is continuing to receive imaging data (step S16). If the result in step S16 is NO, the image processing device 100 returns to step S11 and repeats the same process, and terminates the process if the result in step S16 is YES.

[0073] On the other hand, if it is determined that damaged pixels exist within the above range (step S14: YES), the image processing device 100 opens the shutter 901 after a period t2 has elapsed from the time the shutter 901 was closed (step S17). Period t2 can be set in advance to be longer than period t1, for example, 20 seconds. Period t2 can be set based on experimental results showing that recovery occurs under certain seasonal or other environmental conditions. However, whether recovery will definitely occur within the set period t2 depends on environmental factors such as season, weather, and time of day. Therefore, period t2 can also be changed according to the sunlight incidence conditions, which vary depending on environmental factors such as season, weather, and time of day. After step S17, the image processing device 100 executes the process in step S16.

[0074] Embodiment 1 has been described above. In this embodiment, as described above, the shutter 901 is not closed in all cases where saturated pixels exist, but rather the shutter 901 is closed when a predetermined saturated pixel that satisfies predetermined conditions is detected. Furthermore, in this embodiment, as described above, the shutter 901 is opened at a timing based on the determination result of the determination unit 150 while the shutter 901 is closed.

[0075] Therefore, according to this embodiment, when protecting the microbolometer 900 by closing the shutter 901, it is possible to delay the timing of starting the control to close the shutter 901 and to advance the timing of the control to open the shutter 901. In other words, according to this embodiment, it is possible to shorten the time it takes to close the shutter 901.

[0076] For example, even when the thermal image display system 10 is installed in a car, closing the shutter 901 can reduce situations where the detection of objects such as pedestrians is not possible, thus hindering the safety assistance of the car. The microbolometer 900 can be used for monitoring and observation purposes in various situations other than in automobiles, and in any of these situations, the purpose cannot be fulfilled during the period when the shutter 901 is closed to protect the microbolometer 900. However, in this embodiment, this period can be shortened, so that the original purpose can be achieved as much as possible.

[0077] In particular, in this embodiment, by controlling the period t2 in Figure 4 to be longer than the period t1, it is possible to avoid unnecessarily keeping the shutter 901 closed for a longer period when no damaged pixels are present.

[0078] Finally, we will describe other examples of conditions related to the degree of saturation that can be set as predetermined conditions or included as one of the predetermined conditions. As mentioned above, if the infrared detection element is not damaged, the thermal image when the shutter 901 is closed will have a uniform output across the entire screen. On the other hand, pixels corresponding to infrared detection elements that have suffered even a little damage will show abnormal pixel values ​​(output levels), resulting in a difference compared to the pixel values ​​of normal pixels, such as surrounding normal pixels.

[0079] Therefore, the larger the difference in these pixel values, the greater the damage to the pixel can be judged. Thus, the determination unit 150 can distinguish between saturation caused by sunlight and saturation caused by other factors by setting a threshold for this difference in pixel values ​​and performing threshold processing. In this way, a condition such as the difference in pixel values ​​being greater than a predetermined value can be adopted as a condition for the degree of saturation. In that case, the saturation detection unit 140 obtains the determination result from the determination unit 150 and detects a predetermined saturated pixel.

[0080] Thus, the saturation detection unit 140 can detect predetermined saturated pixels by obtaining the result of the determination unit 150's determination of whether or not a pixel in the former thermal image is sunlight, based on the thermal image taken with the shutter 901 open and the thermal image taken with the shutter 901 closed. However, in this example, the saturation detection unit 140 detects predetermined saturated pixels by obtaining the result of the determination unit 150's determination of whether the saturation is due to sunlight or to high-temperature areas other than sunlight, according to the degree of damage, so the shutter 901 is closed at the time of detection.

[0081] Therefore, the shutter control unit 160 should control the shutter 901 to open immediately if no predetermined saturated pixels are detected by the threshold processing described above, and to keep the shutter 901 closed only if a saturated pixel is detected. With this control, the saturation detection unit 140 can shorten the time the shutter 901 is closed, even if it distinguishes between saturation caused by sunlight and saturation caused by high-temperature areas other than sunlight, depending on the degree of damage.

[0082] <Embodiment 2> Next, Embodiment 2 will be described. The image processing apparatus according to Embodiment 2 differs from the image processing apparatus 100 in Figure 1 in some processing steps, so this embodiment will also be described with reference to the image processing apparatus 100 in Figure 1. Although not explained here, various application examples described in Embodiment 1 can also be applied to this embodiment.

[0083] The shutter control unit 160 in this embodiment differs from that in Embodiment 1 in that it performs the following control. If the determination unit 150 determines that a damaged pixel exists, the shutter control unit 160 controls the shutter 901 to open at a timing based on the determination result of the determination unit 150, after a longer period of time has elapsed compared to when no damaged pixel is determined to exist.

[0084] In other words, in Embodiment 1, the timing for the shutter control unit 160 to open the shutter 901 is predetermined by the elapsed time from the time the shutter 901 was closed. In Embodiment 1, this elapsed time can also be predetermined as a time corresponding to the determined level of damage. In contrast, in this embodiment, the presence or level of damage is re-determined for pixels that indicate damage, and control is performed to open the shutter 901 at a timing based on the re-determination result. Of course, the damage re-determination can be performed multiple times.

[0085] Here, the timing based on the re-evaluation result can be set according to the presence or absence of damaged pixels. For example, similar to the timing based on the evaluation result in Embodiment 1, the timing can be set earlier if damaged pixels are present or absent as a result of the re-evaluation. As a more specific example, the timing based on the re-evaluation result can be set to the timing when the re-evaluation result indicates that the target pixel has returned to a normal state. In other words, in this embodiment, if damaged pixels are present, the shutter 901 is kept closed, multiple images are taken, and the shutter 901 is controlled to open when there are no more damaged pixels (when the target infrared detection element has recovered).

[0086] Alternatively, in this embodiment, if damaged pixels are present, the shutter 901 can be kept closed, multiple shots are taken, and the shutter 901 is controlled to open when the damage has decreased to a predetermined level. This allows the shutter 901 to be opened at a timing when it is determined that opening it will not cause any problems.

[0087] Thus, by taking advantage of the fact that damage recovers over time, the shutter control unit 160 can check the degree of damage recovery using the re-determination result from the determination unit 150, and control the opening of the shutter 901 at a timing corresponding to the degree of damage recovery.

[0088] As described in Embodiment 1, the determination unit 150 can determine whether a pixel is damaged or not using, for example, the difference value from the frame statistics (mean, median, minimum, etc.) when the shutter 901 was used to capture an image. Therefore, the determination unit 150 can also determine the degree of recovery when the pixel difference between the frame statistics when the shutter 901 was used to capture an image and the damaged pixel falls below a predetermined value. Furthermore, by setting multiple predetermined values, the determination unit 150 can also determine the extent to which the pixel has recovered by thresholding using these predetermined values.

[0089] In this embodiment as well, the determination unit 150 can use various comparison pixel values ​​to determine whether or not a pixel is damaged, as described in Embodiment 1. Similarly, the determination unit 150 can determine that a pixel has recovered when the difference with the comparison pixel value falls below a predetermined value, and by setting multiple predetermined values, it is also possible to determine to what extent the pixel has recovered.

[0090] Specific examples of the degree of recovery will be explained with reference to Figure 3. For example, the shutter control unit 160 can control the shutter 901 to open when, after reassessment, the damage becomes as shown by the damaged pixel group D33 in thermal image A13. Alternatively, the shutter control unit 160 can also control the shutter 901 to open when, after reassessment, the damage becomes as shown by the damaged pixel group D43 in thermal image A14. In these examples, it can be said that this is effective in ensuring that the camera can take pictures for as long as possible. In particular, when the image processing device 100 is used with safety in mind, such as in an in-vehicle camera, it is also effective in terms of safety. However, the shutter control unit 160 can also control the shutter 901 to open when, after reassessment, the damage becomes a completely black thermal image that is not shown.

[0091] Next, with reference to Figure 5, the processing performed by the image processing apparatus 100 according to this embodiment will be described. Figure 5 is a flowchart illustrating an example of image processing performed by the image processing apparatus according to this embodiment.

[0092] The flowchart shown in Figure 5 differs from the flowchart in Figure 4 in the processing after step S14 results in YES. Below, we will explain the process starting from the point where step S14 results in YES, focusing on the differences from the flowchart in Figure 4.

[0093] If the image processing device 100 determines that there are damaged pixels (step S14: YES), it determines whether a period t2 (t2 > t1) has elapsed since the shutter 901 was closed (step S21), and waits until the period has elapsed. In this embodiment as well, the period t2 can be changed according to the incident sunlight conditions, which vary depending on the season, weather, time of day, etc.

[0094] If period t2 has elapsed (step S21: YES), the image processing device 100 keeps the shutter 901 closed and causes the microbolometer 900 to take an image of the shutter 901 to acquire imaging data (step S22). Next, the image processing device 100 determines, similar to step S14, whether or not damage remains for at least the pixels in the area that was damaged in step S14 (step S23). This area can be the area that was determined in step S14.

[0095] If damaged pixels remain (step S23: YES), the image processing device 100 returns to step S22 and repeats the shooting and judgment until there are no damaged pixels remaining (until step S23 is NO). If the damaged pixels have recovered and no pixels remain (step S23: NO), the image processing device 100 immediately opens the shutter 901 (step S24). After step S24, the image processing device 100 executes the process in step S16.

[0096] Next, with reference to Figure 6, other examples of processing performed by the image processing apparatus 100 according to this embodiment will be described. Figure 6 is a flowchart illustrating other examples of image processing performed by the image processing apparatus according to this embodiment.

[0097] The flowchart shown in Figure 6 differs from the flowchart shown in Figure 5 in that the processing after step S22 is different. Below, we will explain the differences from the flowchart in Figure 5, starting from the point where step S21 is answered with YES.

[0098] If the period t2 has elapsed (step S21: YES), the image processing device 100 keeps the shutter 901 closed and causes the microbolometer 900 to take an image with the shutter 901 to acquire imaging data (step S22). Next, the image processing device 100 determines whether the damage has been reduced to a predetermined value or more for at least the pixels in the area that was damaged in step S14 (step S31).

[0099] If the number of damaged pixels has not been reduced by a predetermined value or more (step S31: NO), the image processing device 100 returns to step S22 and repeats the shooting and judgment until the number of damaged pixels has been reduced by a predetermined value or more (until step S31 becomes YES). If the number of damaged pixels has been reduced by a predetermined value or more (step S31: YES), the image processing device 100 immediately opens the shutter 901 (step S24). After step S24, the image processing device 100 executes the process in step S16.

[0100] The second embodiment has been described above. With the above configuration, the image processing device 100 according to this embodiment controls the opening of the shutter 901 after confirming that the damage has recovered, thus avoiding taking photographs for monitoring objects, etc., when the damage has not yet recovered.

[0101] (Alternative examples, etc.) The program described above, when loaded into a computer, includes a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments. The program may be stored on a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically, or otherwise propagating signals.

[0102] This disclosure is not limited to the embodiments described above, and may be modified as appropriate without departing from its spirit.

[0103] For example, instead of detecting predetermined saturated pixels in Embodiments 1 and 2, saturated pixels can be detected, and control can be performed to close the shutter 901 when saturated pixels are detected. In that case, it is possible to determine from the imaging data with the shutter 901 closed whether or not there are damaged pixels in the range where saturated pixels were detected, and control can be performed to open the shutter 901 at a timing based on the determination result. In this way, the image processing device 100 can adopt a configuration that performs detection that does not depend on predetermined conditions. Even with an image processing device that adopts such a configuration, when protecting the microbolometer 900 by closing the shutter 901, the timing of controlling to open the shutter 901 can be advanced, and as a result, the time it takes to close the shutter 901 can be shortened.

[0104] Furthermore, the various image processing devices described above are applicable not only to automobiles but also to various types of mobile objects such as motorcycles, ships, airplanes, robots, and drones. In addition, as mentioned above, image processing devices can be used for purposes other than automobiles. For example, image processing devices are applicable to various imaging devices capable of capturing thermal images, such as fixed-point observation cameras capable of capturing thermal images, and are not limited to the application of the imaging device. Fixed-point observation cameras capable of capturing thermal images can make beneficial use of their functions by equipping them with the image processing devices described above, that is, in the control of closing the shutter when pixel saturation is detected, the closing time can be shortened.

[0105] Furthermore, in the above embodiment, a thermal image display system 10 equipped with a monitor 902 was given as an example of incorporating an image processing device, but an image processing device can also be incorporated into a system without a monitor. For example, the image processing device can be incorporated into an object detection system that includes a recognition processing unit that recognizes (detects) objects such as people that meet predetermined conditions such as shape and movement based on an image output from the AGC processing unit 130, and a notification unit that notifies when an object is detected. [Explanation of symbols]

[0106] 10 Thermal imaging display system 100 Image Processing Devices 110 Defective Pixel Correction Unit 120-pixel non-uniformity correction section 130 AGC Processing Unit 140 Saturation detection unit 150 Judgment section 160 Shutter control unit 900 microbolometers 901 Shutter 902 Monitor A11, A12, A13, A14 Thermal Images V01 Other vehicles H12, H13 Saturated Pixel Group D13 Sun D23, D33, D43 Damage Pixel Group

Claims

1. a saturation detection unit that detects predetermined saturated pixels that satisfy a predetermined condition from thermal image data acquired by a sensor that captures a thermal image; a shutter control unit that controls an operation of a shutter that blocks light from entering the sensor; A determination unit that determines whether or not there is a damaged pixel whose pixel value indicates an abnormal value within a range where the saturation detection unit detects the predetermined saturated pixel from the thermal image data acquired by photographing the shutter at a uniform temperature in a state where the shutter blocks light from entering the sensor; Equipped with The shutter control unit closes the shutter when the saturation detection unit detects the predetermined saturated pixel, and when the determination unit determines that no damaged pixels exist in the thermal imaging data captured when the shutter is closed, controls to open the shutter immediately or after a predetermined period of time that is predetermined as the maximum time required for the detection process of the predetermined saturated pixel has elapsed, and when the determination unit determines that damaged pixels exist in the thermal imaging data captured when the shutter is closed, controls to open the shutter after a time that is expected to recover from the damaged pixel has elapsed. Image processing device.

2. The determination unit further determines a level of damage to the damaged pixel, the shutter control unit, when it is determined by the determination unit that a damaged pixel exists, performs control to open the shutter after a time during which the damaged pixel is expected to recover according to the determined level of damage has elapsed. The image processing device according to claim 1 .

3. When the determination unit determines that a damaged pixel exists, the determination unit determines whether or not a damaged pixel exists multiple times while the shutter is closed, The shutter control unit performs control to open the shutter when the damaged pixel is restored by a plurality of determinations made by the determination unit.

3. The image processing device according to claim 1 or 2.

4. a saturation detection step of detecting a predetermined saturated pixel, which is a saturated pixel that satisfies a predetermined condition, from thermal image data acquired by a sensor that captures a thermal image; a step of performing control to close a shutter that blocks light from entering the sensor when the predetermined saturated pixel is detected in the saturation detection step; a determination step of determining whether or not there is a damaged pixel having an abnormal pixel value within a range where the predetermined saturated pixel is detected in the saturation detection step, from the thermal image data acquired by photographing the shutter at a uniform temperature in a state where the shutter blocks light from entering the sensor; a step of controlling the shutter to open immediately or after a predetermined period of time, which is a maximum time required for the detection process of the predetermined saturated pixel, has elapsed if it is determined in the determination step that the thermal imaging data captured when the shutter is closed does not contain a damaged pixel, and controlling the shutter to open after a time during which the damaged pixel is expected to recover has elapsed if it is determined in the determination step that the thermal imaging data captured when the shutter is closed contains a damaged pixel; An image processing method comprising:

5. a saturation detection step of detecting a predetermined saturated pixel, which is a saturated pixel that satisfies a predetermined condition, from thermal image data acquired by a sensor that captures a thermal image; a step of performing control to close a shutter that blocks light from entering the sensor when the predetermined saturated pixel is detected in the saturation detection step; a determination step of determining whether or not there is a damaged pixel having an abnormal pixel value within a range where the predetermined saturated pixel is detected in the saturation detection step, from the thermal image data acquired by photographing the shutter at a uniform temperature in a state where the shutter blocks light from entering the sensor; a step of controlling the shutter to open immediately or after a predetermined period of time, which is a maximum time required for the detection process of the predetermined saturated pixel, has elapsed if it is determined in the determination step that the thermal imaging data captured when the shutter is closed does not contain a damaged pixel, and controlling the shutter to open after a time during which the damaged pixel is expected to recover has elapsed if it is determined in the determination step that the thermal imaging data captured when the shutter is closed contains a damaged pixel; The process includes: A program that a computer runs.

Images