Image processing apparatus and image processing method
The image processing device corrects thermal images by identifying and correcting abnormal pixels based on statistical values and movement trajectories, addressing the issue of shutter-induced detection gaps in infrared cameras.
Patent Information
- Application Number
- JP2023077948
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-10
- Publication Date
- 2024-11-21
AI Technical Summary
Infrared cameras can malfunction when capturing high-temperature objects like the sun, leading to abnormal pixels and deteriorated image quality due to burn-in, and existing solutions that close the shutter to prevent this result in periods where objects cannot be detected.
An image processing device and method that corrects thermal images without closing the shutter by identifying and correcting abnormal pixel areas based on statistical values and movement trajectories of high-temperature objects.
Enables accurate detection and correction of abnormal pixels in thermal images, maintaining continuous imaging without shutter closure, thereby ensuring consistent object detection.
Smart Images

Figure 2024162443000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an image processing device and an image processing method. [Background technology]
[0002] There is a known technology for detecting objects using infrared cameras. Infrared cameras detect heat in captured images, so they can detect people and objects in the captured images even in environments where there is no visible light, such as at night. In recent years, many products using infrared cameras have been developed for security and mobile applications.
[0003] On the other hand, due to the characteristic of infrared cameras that detect the heat of objects, there is a possibility that microbolometers used as imaging elements may become abnormal when capturing an image of a high-temperature object such as the sun. For example, when sunlight is directly incident on an infrared sensor, abnormal pixels are generated due to burn-in caused by the sunlight. The abnormal pixels change the sensitivity, degrading the quality of the captured image. For example, the abnormal pixels appear in the captured image as white spots (artifacts). There is a demand for technology to suppress the occurrence of such abnormal pixels.
[0004] As a related technique, Patent Document 1 discloses an image processing device including a saturated area detection unit that detects the presence of a saturated area in imaging data, and a shutter control unit that performs shutter closing control based on the detection result of the saturated area. In the image processing device disclosed in Patent Document 1, when a saturated area is detected by the saturated area detection unit, the shutter control unit closes the shutter to protect the infrared imaging element.
[0005] The image processing device disclosed in Patent Document 1 includes an abnormal pixel detection unit that detects whether or not an abnormal pixel is included in imaging data acquired with the shutter closed, and a selection unit that selects and outputs first or second correction data, which is corrected imaging data, according to the detection result. The first correction data is imaging data corrected by correction (so-called shutterless correction) that adjusts the output according to a preset algorithm without using a shutter. The second correction data is imaging data corrected by correction (so-called mechanical shutter correction) that adjusts the output of a microbolometer with a mechanical shutter, which is a black body, closed. Thus, the image processing device disclosed in Patent Document 1 outputs the selected correction data to a monitor according to the detection result of the abnormal pixel. [Prior art documents] [Patent documents]
[0006] [Patent Document 1] International Publication No. 2022 / 018900 Summary of the Invention [Problem to be solved by the invention]
[0007] When the shutter is closed to protect the infrared camera as in the technology disclosed in Patent Document 1, the infrared camera cannot capture images while the shutter is closed, which causes a problem in that, for example, in a security system, there are periods when people or objects cannot be detected.
[0008] In view of the above-mentioned problems, an object of the present disclosure is to provide an image processing device and an image processing method capable of appropriately correcting an image without closing the shutter of an infrared camera. [Means for solving the problem]
[0009] The image processing device according to the present disclosure includes: a thermal image data acquisition unit that acquires thermal image data from an infrared camera that captures thermal images; an object area acquisition unit that acquires an object area indicating an area of an object in the thermal image data; a trajectory acquisition unit for acquiring a movement trajectory of a high-temperature object corresponding to an imaging range of the thermal imaging data; A calculation unit that calculates an object region statistical value that is a statistical value of pixel values in at least a part of the object region; a specifying unit that specifies, among the object regions, a pixel region having a pixel value whose difference from the object region statistical value is equal to or greater than a predetermined value and that is included in the movement trajectory, as an abnormal pixel region; A correction unit corrects pixel values of the abnormal pixel region based on the object region statistics.
[0010] The image processing method according to the present disclosure includes: A thermal image data acquisition step of acquiring thermal image data from an infrared camera that captures a thermal image; an object area acquisition step of acquiring an object area indicating an area of an object in the thermal image data; a trajectory acquisition step of acquiring a movement trajectory of a high-temperature object corresponding to an imaging range of the thermal imaging data; A calculation step of calculating an object region statistic, which is a statistic of pixel values in at least a portion of the object region; a specifying step of specifying, as an abnormal pixel region, a pixel region among the object regions, the pixel region having a pixel value whose difference from the object region statistical value is equal to or greater than a predetermined value and which is included in the movement trajectory; and a correction step of correcting pixel values of the abnormal pixel region based on the object region statistics. Effect of the Invention
[0011] The image processing device and image processing method according to the present disclosure make it possible to appropriately correct an image without closing the shutter of an infrared camera. [Brief description of the drawings]
[0012] [Figure 1] 1 is a block diagram showing a configuration of an imaging system according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing a detailed configuration of the infrared camera according to the first embodiment. [Diagram 3] FIG. 2 is a block diagram showing a configuration of an image processing unit according to the first embodiment. [Figure 4] FIG. 13 is a diagram showing an example of thermal imaging data when sunlight is incident. [Diagram 5] FIG. 13 is a diagram showing an example of thermal imaging data when an abnormal pixel area occurs; [Figure 6] 4 is a flowchart showing a process performed by the image processing device according to the first embodiment. [Figure 7] FIG. 11 is a block diagram showing a configuration of an image processing unit according to a second embodiment. [Figure 8] 13 is a diagram illustrating the uniformity of pixel values in a surrounding area according to the second embodiment. FIG. [Figure 9] 11 is a flowchart showing a process performed by an image processing device according to a second embodiment. [Figure 10] FIG. 11 is a block diagram showing the configuration of an image processing unit according to a third embodiment. [Figure 11] FIG. 13 is a diagram showing a state before an abnormal pixel region is divided according to the third embodiment. [Figure 12] FIG. 13 is a diagram showing a state after an abnormal pixel region according to the third embodiment has been divided. [Figure 13] 11 is a flowchart showing a process performed by an image processing device according to a third embodiment. [Figure 14] FIG. 13 is a block diagram showing a configuration of an image processing unit according to a modified example of the third embodiment. [Figure 15] 13 is a flowchart showing a process performed by an image processing device according to a modified example of the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals. For clarity of explanation, duplicate explanations will be omitted as necessary.
[0014] <Embodiment 1> (Imaging System 1) An image capturing system 1 according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the image capturing system 1 according to the present embodiment.
[0015] 1, the imaging system 1 includes an image processing device 101, an infrared camera 200, and a display device 300. The imaging system 1 may be used in, for example, a vehicle. The imaging system 1 may be, for example, an imaging system for a vehicle that acquires a thermal image of the surroundings, or an imaging system used for surveillance purposes.
[0016] The infrared camera 200 captures a thermal image (infrared image). The infrared camera 200 has an infrared sensor. The infrared camera 200 captures a thermal image by detecting infrared rays from a subject with the infrared sensor. The infrared camera 200 is, for example, a far infrared (FIR) camera that can visualize far infrared rays emitted from an object. The data of the thermal image captured by the infrared camera 200 is also called thermal imaging data. The thermal imaging data is a moving image composed of multiple frames, for example, 15 to 60 frames per second.
[0017] Here, the object refers to a target imaged by the infrared camera 200. The object is, for example, a vehicle, a person, a building, etc. The object may also include the sky, the sun, the sea, etc.
[0018] The infrared camera 200 outputs the thermal imaging data to the image processing device 101. The thermal imaging data acquired by the image processing device 101 from the infrared camera 200 is also referred to as acquired thermal imaging data. The acquired thermal imaging data indicates data of the thermal image captured by the infrared camera 200.
[0019] (Infrared Camera 200) Here, the configuration of the infrared camera 200 will be described in detail with reference to Fig. 2. Fig. 2 is a block diagram showing the detailed configuration of the infrared camera 200. As shown in Fig. 2, the infrared camera 200 includes a lens 210, a shutter 220, an infrared sensor 230, a transmission device 240, and a temperature sensor 250.
[0020] The lens 210 forms an image of infrared light from the subject on the light receiving surface of the infrared sensor 230. The infrared sensor 230 includes a plurality of pixels. Each pixel of the infrared sensor 230 receives infrared light from the subject. This allows a thermal image of the subject to be captured. For example, the infrared sensor 230 may include a microbolometer for detecting far-infrared rays. The infrared sensor 230 includes a plurality of pixels arranged in a two-dimensional array. The detection values (detection signals) of each pixel form a thermal image of the subject.
[0021] The transmission device 240 is an interface for transmitting various signals and data to the image processing device 101. The transmission device 240 transmits thermal imaging data captured by the infrared sensor 230 to the image processing device 101. In addition, the transmission device 240 receives a control signal from the image processing device 101.
[0022] The temperature sensor 250 measures the temperature in the environment in which the infrared camera 200 is used. The transmission device 240 transmits the environmental temperature measured by the temperature sensor 250 to the image processing device 101. The environmental temperature measured by the temperature sensor 250 can be used for shutterless correction.
[0023] The shutter 220 is disposed on the front side of the infrared sensor 230. The shutter 220 blocks infrared light incident on the infrared sensor 230 from the outside. The shutter 220 has an openable / closable mechanism. When the shutter 220 is closed, infrared light from the lens 210 does not enter the infrared sensor 230. In this case, infrared light from the shutter 220, not from the lens 210, enters the infrared sensor 230. Also, when the shutter 220 is open, infrared light from the lens 210 enters the infrared sensor 230. For example, the shutter 220 may be something like a barrier or a cover disposed on the front side of the lens 210. The infrared camera 200 may be configured without the shutter 220.
[0024] (Display device 300) 1, the description will be continued. The display device 300 includes a display for displaying display image data, etc. The display device 300 displays the display image data that has been subjected to image processing in the image processing device 101.
[0025] (Image processing device 101) The image processing device 101 generates display image data by performing image processing on the acquired thermal imaging data. For example, the image processing device 101 performs various processes such as correction of pixel variations of the infrared camera 200.
[0026] The image processing device 101 outputs the display image data that has been subjected to image processing to the display device 300. The display image data that has been subjected to image processing by the image processing device 101 and output to the display device 300 is also referred to as output thermal image data. In other words, the acquired thermal image data is data before processing by the image processing device 101, and the output thermal image data is data after processing.
[0027] The image processing device 101 includes, as physical components, a control IF (interface) 110, a ROM (Read Only Memory) 120, and a RAM (Random Access Memory) 130. The image processing device 101 further includes, as functional blocks realized by a program as processing of a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), a system control unit 140, a thermal imaging data acquisition unit 150, a high-temperature object region acquisition unit 160, an image processing unit 170, and a display control unit 180. For convenience, these components are shown as being connected to each other so as to be able to communicate with each other appropriately via a bus.
[0028] The control IF 110 is an interface for controlling the infrared camera 200. The image processing device 101 transmits various signals and data to the infrared camera 200 via the control IF 110. In addition, the control IF 110 receives various signals, data, and the like from the infrared camera 200.
[0029] The ROM 120 stores a control program and various parameters for controlling the image processing device 101. For example, the ROM 120 stores an image processing program executed by the image processing unit 170. The ROM 120 also stores a system control program executed by the system control unit 140.
[0030] The RAM 130 stores various programs and parameters used for executing the programs, etc. The RAM 130 also stores calculation data of the image processing unit 170, etc.
[0031] The system control unit 140 controls the entire imaging system 1. The processor executes a control program to control the entire system. For example, the processor loads a computer program from the ROM 120 into the RAM 130 and executes the computer program. In this way, the processor realizes the functions of, for example, the thermal imaging data acquisition unit 150, the high-temperature object region acquisition unit 160, the image processing unit 170, and the display control unit 180. Note that the processor may be a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), a quantum processor (quantum computer control chip), or the like.
[0032] Furthermore, when some or all of the components of the image processing device 101 are realized by multiple information processing devices, circuits, etc., the multiple information processing devices, circuits, etc. may be arranged in a centralized manner or in a distributed manner.
[0033] Note that system control unit 140 may have functions other than these functions. For example, system control unit 140 may have a function of a shutter control unit that controls opening and closing of the shutter of infrared camera 200.
[0034] The thermal imaging data acquisition unit 150 acquires thermal imaging data from an infrared camera 200 that captures thermal images. The thermal imaging data may be composed of multiple frames. The thermal imaging data acquisition unit 150 is provided with, for example, an IF (interface) for acquiring the thermal imaging data from the infrared camera 200. The interface between the thermal imaging data acquisition unit 150 and the infrared camera 200 may be a wired connection or a wireless connection.
[0035] The high temperature object region acquisition unit 160 acquires a high temperature object region that indicates a region of a high temperature object corresponding to the imaging range of the thermal imaging data. Here, the high temperature object indicates an object with a high temperature that may cause abnormal pixels in the thermal imaging data.
[0036] The high-temperature object may be set in advance depending on the characteristics of the infrared sensor 230. For example, an object that is assumed to have a temperature equal to or higher than a predetermined temperature may be set as the high-temperature object. The predetermined temperature is, for example, 1000°C. In this case, examples of the high-temperature object include the sun and a blast furnace. When the infrared camera 200 captures an image of such a high-temperature object, pixels may become saturated in the thermal imaging data, and the saturated pixels may become abnormal pixels.
[0037] On the other hand, for example, the temperature of a vehicle muffler is usually less than 1000°C. In the thermal imaging data, the pixels in the muffler area are also saturated, but are unlikely to become abnormal pixels. Therefore, regardless of whether the pixels are saturated in the thermal imaging data, by setting in advance the high-temperature objects that may cause abnormal pixels, the high-temperature object area acquisition unit 160 can appropriately acquire information on the high-temperature objects that may cause abnormal pixels.
[0038] Moreover, an abnormal pixel indicates a pixel that has been damaged by extremely high intensity far-infrared light being incident on the infrared sensor 230 for a certain period of time. For example, assume that the infrared camera 200 captures an image with the shutter 220 closed. In this case, the infrared camera 200 acquires thermal imaging data capturing an image of the shutter 220, which is a black body. In this case, if all pixels are normal, the difference in pixel value between adjacent pixels in the thermal imaging data will be zero or close to zero.
[0039] However, when strong light such as sunlight is incident on the infrared sensor 230, the pixels exposed to the sunlight are damaged and become abnormal pixels. When an image is captured with the shutter 220 closed in a state where abnormal pixels have occurred, the difference between the pixel value of the abnormal pixel and the pixel value of a normal pixel in the acquired thermal imaging data will not be zero or close to zero.
[0040] In this case, the abnormal pixels will have a higher pixel value than normal. In this case, an abnormal pixel area formed by multiple abnormal pixels will appear as a white dot in the thermal imaging data. Abnormal pixels and methods for detecting abnormal pixels will be described later.
[0041] The high-temperature object region acquisition unit 160 detects a high-temperature object based on location information indicating the location of the infrared camera 200, image capture date and time information indicating the image capture date and time, and map information indicating a map corresponding to the image capture range (angle of view). The high-temperature object region acquisition unit 160 acquires a high-temperature object region that is an area corresponding to the thermal imaging data of the detected high-temperature object.
[0042] For example, the high temperature object region acquisition unit 160 acquires the result of positioning using technology such as GNSS (Global Navigation Satellite System) as the position information of the infrared camera 200. For example, the infrared camera 200 is equipped with a GPS (Global Positioning System) sensor (not shown) and performs positioning using the GPS sensor. The high temperature object region acquisition unit 160 acquires the positioning result from the GPS sensor to acquire the position information of the infrared camera 200. The high temperature object region acquisition unit 160 acquires time series data of the position information by sequentially acquiring the position information.
[0043] Furthermore, the high-temperature object region acquisition unit 160 acquires information on the shooting direction of the infrared camera 200 as position information of the infrared camera 200. For example, when the infrared camera 200 is mounted on a vehicle, the mounting position and mounting direction of the infrared camera 200 are known. Therefore, the high-temperature object region acquisition unit 160 can obtain the shooting direction of the infrared camera 200 from the traveling direction of the vehicle.
[0044] Furthermore, the high temperature object region acquiring unit 160 acquires the photographing date and time information associated with the thermal imaging data from the thermal imaging data. The high temperature object region acquiring unit 160 also acquires the location information and the map information associated with the photographing date and time information via a network (not shown). The network (not shown) is, for example, a wired or wireless communication line.
[0045] For example, the high temperature object region acquisition unit 160 acquires the region of the sun in the thermal imaging data as the high temperature object region. Specifically, the high temperature object region acquisition unit 160 first identifies the position of the sun at the shooting position based on the position information and the shooting date and time information. The position information includes the shooting direction of the infrared camera 200.
[0046] The high-temperature object region acquisition unit 160 may acquire a high-temperature object region corresponding to another high-temperature object, not limited to the sun. For example, the high-temperature object region acquisition unit 160 acquires a high-temperature object region corresponding to a high-temperature object such as a blast furnace. In this way, the high-temperature object region acquisition unit 160 can acquire various high-temperature object regions corresponding to the imaging range of the thermal imaging data. The high-temperature object region acquisition unit 160 stores the acquired data on the high-temperature object region in the ROM 120 or the like.
[0047] The high-temperature object region acquisition unit 160 may acquire the high-temperature object region using a method other than the above-mentioned method of acquiring the high-temperature object region based on the location information, the shooting date and time information, and the map information. For example, the high-temperature object region acquisition unit 160 may detect saturated pixels and acquire the high-temperature object region using the saturated pixels.
[0048] Here, a saturated pixel is a pixel whose pixel value is at the upper limit. For example, assume that the brightness level of each pixel in the thermal imaging data is expressed in 14 bits ranging from zero to 16383. In this case, a pixel at a coordinate having a brightness level value of 16383 can be regarded as a saturated pixel. For example, the high temperature object area acquisition unit 160 receives thermal imaging data from the defective pixel correction unit 171 of the image processing unit 170, which will be described later, and detects saturated pixels contained in the received thermal imaging data.
[0049] In addition, the saturated pixel may be defined using a predetermined threshold value, not limited to the upper limit value. For example, the saturated pixel may be a pixel whose pixel value is equal to or greater than a predetermined threshold value. This allows the high temperature object region acquisition unit 160 to detect a substantially saturated pixel as a saturated pixel.
[0050] Moreover, the high-temperature object region acquisition unit 160 detects a saturated region having a plurality of saturated pixels. The high-temperature object region acquisition unit 160 may detect a pixel region that includes a plurality of saturated pixels and satisfies a predetermined condition as a saturated region. The predetermined condition may be set in advance. For example, when the high-temperature object region acquisition unit 160 detects four or more adjacent saturated pixels, it detects the region of the four or more adjacent saturated pixels as a saturated region. Alternatively, the high-temperature object region acquisition unit 160 may detect the region of nine or more adjacent pixels as a saturated region when, for example, nine or more adjacent pixels have a pixel value of 98% or more of the upper limit value described above. In this way, the high-temperature object region acquisition unit 160 can acquire a high-temperature object region by a method different from the method using the position information described above.
[0051] The image processing unit 170 performs predetermined image processing on the thermal imaging data acquired by the thermal imaging data acquisition unit 150. The processor that executes the processing of the image processing unit 170 may be a general-purpose processor or a dedicated processor. The processor that executes the processing of the image processing unit 170 and the processor that executes the processing of the system control unit 140 may be a common processor.
[0052] For example, the image processor 170 corrects the acquired thermal imaging data by performing a predetermined process on the acquired thermal imaging data acquired by the infrared camera 200. The image processor 170 outputs output thermal imaging data, which is the corrected data, to the display device 300.
[0053] Here, the image processing unit 170 will be described in detail with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the image processing unit 170. The image processing unit 170 includes a defective pixel correction unit 171, a NUC (Non-Uniformity Correction) unit 172, an object area acquisition unit 173, a trajectory acquisition unit 174, a calculation unit 175, an identification unit 176, and a correction unit 177. Note that the one-way arrows shown in the figure simply indicate the flow of information (data, signals, etc.), and do not exclude the two-way nature of information. The same applies to the subsequent figures.
[0054] The defective pixel correction unit 171 performs an interpolation process on the pixel value of a defective pixel of the infrared sensor 230, which is pre-stored in the RAM 130, from the pixel values of pixels surrounding the defective pixel. The defective pixel correction unit 171 performs the above-mentioned interpolation process on the acquired thermal imaging data acquired by the thermal imaging data acquisition unit 150, and outputs the image data after the interpolation process to the NUC unit 172.
[0055] The NUC unit 172 performs processing for correcting output variations between pixels. For example, the NUC unit 172 performs shutterless correction based on the environmental temperature detected by the temperature sensor 250. A known method can be used for the shutterless correction in the NUC unit 172.
[0056] The object area acquisition unit 173 acquires an object area indicating an area of an object in the thermal imaging data. For example, the object area acquisition unit 173 acquires the object area by extracting an object included in the acquired thermal imaging data acquired by the thermal imaging data acquisition unit 150 using a known image recognition technique. Without being limited to this, the object area acquisition unit 173 may acquire, for example, an object area detected in a functional unit (not shown) in the image processing unit 170 or outside the image processing unit 170 from the functional unit.
[0057] The trajectory acquisition unit 174 acquires the movement trajectory of the high temperature object corresponding to the imaging range of the thermal imaging data. Specifically, the trajectory acquisition unit 174 acquires the movement trajectory of the high temperature object based on the time series data of the high temperature object area acquired by the high temperature object area acquisition unit 160. For example, the trajectory acquisition unit 174 acquires the movement trajectory of the sun in the sky area (hereinafter referred to as "sky area") of the thermal imaging data.
[0058] The calculation unit 175 calculates an object region statistical value, which is a statistical value of pixel values in at least a part of the object region. The pixel value indicates the output value of each pixel included in the object region. When the object region is a sky region, the calculation unit 175 calculates, for example, the average value of pixel values in the entire sky region as the object region statistical value (hereinafter referred to as the "sky region statistical value"). The calculation unit 175 may also calculate the average value of pixel values in a part of the sky region as the sky region statistical value. The statistical value is not limited to the average value, and may be a mode or a median. Therefore, the calculation unit 175 may calculate, for example, the mode or the median of pixel values in the entire or part of the sky region as the object region statistical value.
[0059] The identification unit 176 identifies, as an abnormal pixel region, a pixel region that has a pixel value that differs from the object region statistical value by a predetermined value or more and is included in the movement trajectory of a high-temperature object. Specifically, the identification unit 176 first detects, in the object region, an abnormal pixel that has a pixel value that differs from the object region statistical value by a predetermined value or more. When a plurality of abnormal pixels form an area having a size equal to or larger than a predetermined size, the identification unit 176 identifies the area as an abnormal pixel region. The identification unit 176 can determine the size using a preset threshold value. The threshold value may be fixed or may be changed as appropriate.
[0060] For example, in the case of the sky region described above, the identifying unit 176 identifies, as an abnormal pixel region, a pixel region in the sky region that has a pixel value that differs from the sky region statistical value by a predetermined value or more and is included in the movement trajectory of a high-temperature object. In this way, the identifying unit 176 can identify, as an abnormal pixel region, a pixel region that has a pixel value that is significantly different from the statistical value of the pixel values in the entire sky region.
[0061] In addition, when a pixel region having a pixel value whose difference from the sky region statistical value is a predetermined value or more is substantially included in the movement trajectory of the high-temperature object, the identifying unit 176 may identify the pixel region as an abnormal pixel region. For example, when a pixel region whose difference from the sky region statistical value is a predetermined value or more protrudes from the movement trajectory within a predetermined range, the pixel region of the protruding range may be included in the abnormal pixel region.
[0062] Here, the abnormal pixel region will be specifically described with reference to Fig. 4 and Fig. 5. Fig. 4 is a diagram showing an example of thermal imaging data when sunlight is incident, and Fig. 5 is a diagram showing an example of thermal imaging data when an abnormal pixel region occurs.
[0063] 4 includes the sun SU, the sky SK, the vehicle 5, and the muffler 51 provided on the vehicle 5. The sun SU, the sky SK, the vehicle 5, and the muffler 51 are each an example of the object region described above. Also, the region corresponding to the sky SK corresponds to the sky region described above.
[0064] Both the sun SU and the muffler 51 are high temperature objects having temperatures that saturate the pixels in the thermal imaging data D1. Therefore, in the thermal imaging data D1, the sun SU and the muffler 51 are saturated regions formed by saturated pixels. A saturated pixel is a pixel whose pixel value has reached its upper limit. In the thermal imaging data D1, both the sun SU and the muffler 51 appear white.
[0065] As described above, the sun SU and the muffler 51 are examples of high-temperature objects that may cause abnormal pixels. However, among high-temperature objects, there are high-temperature objects that cause abnormal pixels and high-temperature objects that are unlikely to cause abnormal pixels, depending on the characteristics and settings of the infrared sensor 230. For example, the examples of Figures 4 and 5 show an example in which the sun SU is a high-temperature object that causes abnormal pixels, and the muffler 51 is a high-temperature object that does not cause abnormal pixels.
[0066] When the infrared camera 200 captures the sun SU for a certain period of time, an abnormal pixel area A occurs in the sky SK as shown in the thermal imaging data D2 in Fig. 5. Since the sun SU moves with time, the abnormal pixel area A occurs on the movement trajectory of the sun SU. The muffler 51 also moves with time, but since it is a high-temperature object that does not generate abnormal pixels, as shown in the figure, the area of the muffler 51 is a saturated area, but no abnormal pixel area occurs in the surrounding area. The abnormal pixels that have occurred recover over time and return to normal pixels with no abnormalities, so the abnormal pixel area A shown in Fig. 5 disappears. However, image quality deteriorates until the pixel state is completely restored.
[0067] Due to the occurrence of the abnormal pixel region A, in the thermal imaging data D2, the temperature difference between the abnormal pixel region A and the surrounding regions cannot be accurately measured. Therefore, there is a risk that the temperature of a part or the whole of the object detected in the thermal imaging data D2 cannot be accurately measured. In this case, an obstacle occurs to object recognition using the thermal imaging data of the infrared camera 200. In the present disclosure, such a problem is solved by correcting the pixel values of the abnormal pixel region A in the correction unit 177 described later.
[0068] Furthermore, for example, the muffler 51 is generally at a high temperature while the vehicle 5 is traveling. Therefore, it is expected that the difference between the pixel statistics in the region of the muffler 51 and the object region statistics for the entire vehicle 5 will be equal to or greater than a predetermined value. Therefore, if the identifying unit 176 identifies an abnormal pixel region without using the movement trajectory of a high-temperature object, the region of the muffler 51 may be erroneously identified as an abnormal pixel region.
[0069] However, in the present disclosure, the identification unit 176 does not simply use the above-mentioned difference, but also uses the movement trajectory of the high-temperature object to identify the abnormal pixel region. As a result, the identification unit 176 does not identify the muffler 51 that is not on the trajectory of the high-temperature object as an abnormal pixel region.
[0070] The identifying unit 176 may also identify an abnormal pixel region within a predetermined region that has been set in advance. For example, the predetermined region is assumed to be a range of one-third from the upper end of the thermal imaging data. The identifying unit 176 acquires an abnormal pixel region within the predetermined region. In this way, the identifying unit 176 identifies an abnormal pixel region only in the region that is one-third from the upper end of the thermal imaging data, and does not identify the muffler 51, which is close to the lower end of the thermal imaging data, as an abnormal pixel region. In this way, the identifying unit 176 can identify an abnormal pixel region with high accuracy.
[0071] Returning to Fig. 3, the explanation will be continued. Correction unit 177 corrects pixel values of the abnormal pixel region based on the object region statistics. For example, in the above-mentioned example of the sky region, correction unit 177 calculates a correction value for each pixel for correcting the pixel values of the abnormal pixel region based on the sky region statistics. Correction unit 177 corrects the pixel values of the abnormal pixel region using the calculated correction value.
[0072] For example, in the example of FIG. 5, the correction unit 177 calculates a correction value based on the statistical value of the sky SK, and corrects the pixel value of each pixel in the abnormal pixel region A using the correction value. In this way, the correction unit 177 can correct the pixel values of the abnormal pixel region A so that the difference between the pixel values of the abnormal pixel region A and the pixel values of the sky SK becomes smaller than before the correction. In this way, the abnormal pixel region A changes from white to a color close to the sky SK (e.g., dark gray). In this way, the correction unit 177 can suppress deterioration of image quality by correcting the abnormal pixel region A using the statistical value of the sky SK.
[0073] As described above, the correction unit 177 may calculate the correction value based on the statistics of the entire sky SK, or may calculate the correction value using the statistics of a partial region of the sky SK. For example, the correction unit 177 calculates the statistics of a region excluding the abnormal pixel region A from the sky SK, and calculates the correction value using the calculated statistics. Since the pixel values are assumed to be approximately constant in the region excluding the abnormal pixel region A from the sky SK, the correction unit 177 can calculate an appropriate correction value.
[0074] Returning to Fig. 1, the description of the imaging system 1 will be continued. The display control unit 180 causes the display device 300 to display the thermal imaging data acquired by the thermal imaging data acquisition unit 150. Specifically, the display control unit 180 causes the display device 300 to display the acquired thermal imaging data acquired by the thermal imaging data acquisition unit 150 after the data is processed by the image processing unit 170. The display control unit 180 is an interface that outputs the output thermal imaging data to the display device 300.
[0075] (Processing of image processing device 101) Next, the process performed by the image processing device 101 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the process performed by the image processing device 101. The process shown in Fig. 6 is started, for example, when the infrared camera 200 starts capturing an image.
[0076] First, the thermal imaging data acquisition unit 150 (see FIG. 1) starts acquiring thermal imaging data from the infrared camera 200 (S11). The thermal imaging data acquisition unit 150 outputs the acquired thermal imaging data to the image processing unit 170.
[0077] The object area acquisition unit 173 (see FIG. 3) starts acquiring the object area in the thermal imaging data (S12). Note that the object area acquisition unit 173 may perform the above process using the thermal imaging data corrected by the defective pixel correction unit 171 and the NUC unit 172.
[0078] Next, the high-temperature object region acquisition unit 160 judges whether or not a high-temperature object corresponding to the imaging range of the thermal imaging data is detected (S13). Specifically, the high-temperature object region acquisition unit 160 detects the high-temperature object based on the location information indicating the location of the infrared camera 200, the shooting date and time information indicating the shooting date and time, and the map information indicating the map corresponding to the imaging range. The high-temperature object is, for example, the sun.
[0079] When it is determined that a high-temperature object has not been detected (NO in S13), the high-temperature object region acquisition unit 160 repeats the process of step S13. When it is determined that a high-temperature object has been detected (YES in S13), the trajectory acquisition unit 174 acquires the movement trajectory of the high-temperature object (S14). The trajectory acquisition unit 174 may store the acquired movement trajectory in the RAM 130 or the like.
[0080] The processes from step S11 to step S14 are continuously executed while imaging is being performed by the infrared camera 200. This allows the thermal imaging data, the object area, and the movement trajectory of the high-temperature object to be stored in chronological order.
[0081] Next, the calculation unit 175 calculates an object region statistical value, which is a statistical value of pixel values in at least a part of the object region (S15). The identification unit 176 determines whether or not there is a pixel region in the object region that has a pixel value that differs from the object region statistical value by a predetermined value or more (S16). If it is determined that there is no such pixel region (NO in S16), the process ends.
[0082] If it is determined that the pixel region exists (YES in S16), the identification unit 176 determines whether or not the pixel region is included in the movement trajectory of the high-temperature object (S17). If it is determined that the pixel region is not included in the movement trajectory of the high-temperature object (NO in S17), the processing ends. If it is determined that the pixel region is included in the movement trajectory of the high-temperature object (YES in S17), the identification unit 176 identifies the pixel region as an abnormal pixel region (S18).
[0083] Next, correction unit 177 calculates a correction value for correcting the pixel value of the abnormal pixel region based on the object region statistics (S19). Correction unit 177 calculates the correction value so that the difference between the pixel value of the abnormal pixel region and the pixel value of the object region becomes smaller than before correction.
[0084] Then, the correction unit 177 corrects the pixel value of each pixel in the abnormal pixel region using the calculated correction value (S20). In the thermal imaging data after correction, the white color of the abnormal pixel region changes to a color closer to the object region including the abnormal pixel region, compared to the thermal imaging data before correction. The correction unit 177 may output the thermal imaging data after correction to the display device 300. As a result, the display device 300 displays the thermal imaging data after correction.
[0085] As described above, according to the imaging system 1 of this embodiment, in the image processing device 101, the thermal imaging data acquisition section 150 acquires thermal imaging data from the infrared camera 200. In the image processing section 170, the object area acquisition section 173 acquires the object area in the thermal imaging data. In addition, the trajectory acquisition section 174 acquires the movement trajectory of the high-temperature object corresponding to the imaging range of the thermal imaging data. For example, the trajectory acquisition section 174 acquires the movement trajectory of the high-temperature object based on the position information, shooting date and time information, and map information of the infrared camera 200.
[0086] Furthermore, calculation unit 175 calculates an object region statistical value, which is a statistical value of pixel values in at least a part of the object region. Identification unit 176 identifies, as an abnormal pixel region, a pixel region in the object region that has a pixel value that differs from the object region statistical value by a predetermined value or more and is included in the movement trajectory. This allows identification unit 176 to identify the abnormal pixel region taking into account the movement trajectory of the high-temperature object. Furthermore, correction unit 177 corrects the pixel values of the abnormal pixel region based on the object region statistical value.
[0087] In this way, the imaging system 1 according to this embodiment can detect the actual position of the high-temperature object and obtain the movement trajectory of the high-temperature object using the position information and map information of the infrared camera 200. This allows the imaging system 1 to accurately identify the abnormal pixel area based on the movement trajectory. In this way, the imaging system 1 according to this embodiment can appropriately correct the image without closing the shutter 220 of the infrared camera 200.
[0088] <Embodiment 2> Next, a second embodiment will be described. In the first embodiment, the correction unit 177 of the image processing device 101 calculates a correction value using pixel values of the entire object region or the object region excluding the abnormal pixel region, and corrects the pixel values of the abnormal pixel region using the correction value. In this embodiment, the uniformity of a region of the object region to be used for calculating the correction value is taken into consideration. Below, differences from the first embodiment will be mainly described, and explanations of points that overlap with the first embodiment will be omitted as appropriate.
[0089] The image processing device 101a according to this embodiment includes an image processing unit 170a instead of the image processing unit 170 of the image processing device 101. The configuration of the image processing device 101a can be explained by replacing the image processing unit 170 of the image processing device 101 (see FIG. 1) of the first embodiment with the image processing unit 170a. Therefore, a block diagram showing the configuration of the image processing device 101a is omitted.
[0090] The configuration of the image processing unit 170a will be described with reference to Fig. 7. Fig. 7 is a block diagram showing the configuration of the image processing unit 170a. Unlike the image processing unit 170 of the first embodiment (see Fig. 3), the image processing unit 170a includes a determination unit 178. Furthermore, the calculation unit 175a and the correction unit 177a have different functions from the calculation unit 175 and the correction unit 177, respectively.
[0091] The determination unit 178 determines whether or not the pixel values of the surrounding region, which is a region located around the abnormal pixel region, are uniform in the object region acquired by the object region acquisition unit 173. For example, the determination unit 178 calculates a distribution of pixel values of the surrounding region. The determination unit 178 analyzes the distribution to determine whether or not the pixel values of the surrounding region are uniform. For example, the determination unit 178 calculates an index value (e.g., the variance of pixel values of the surrounding region) representing the variation of pixel values of the surrounding region using a known statistical method. The determination unit 178 compares the calculated index value with a predetermined threshold value to determine whether or not the pixel values of the surrounding region are uniform. The predetermined threshold value may be set in advance. The predetermined threshold value may be fixed or may be changed as appropriate.
[0092] Note that since judgment unit 178 judges whether or not the pixel values of the surrounding region are uniform by comparing with a predetermined threshold value, "the pixel values of the surrounding region are uniform" does not require that the pixel values of the surrounding region be strictly uniform.
[0093] The surrounding region is a region that indicates a predetermined range from the abnormal pixel region. The surrounding region is, for example, a region that is within a predetermined distance outward from the boundary line of the abnormal pixel region. The surrounding region may be, for example, a region that is within a predetermined distance from the top, bottom, left, and right ends of the abnormal pixel region. The shape of the surrounding region is, for example, rectangular, circular, or elliptical, but is not limited to these.
[0094] When it is determined that the pixel values of the surrounding area are uniform, the calculation unit 175a further calculates a surrounding area statistic, which is a statistical value of the pixel values in the surrounding area. As with the object area statistic, the statistical value is, for example, an average value, a mode value, or a median value. The surrounding area statistic can be used to calculate a correction value in the correction unit 177a.
[0095] The correction unit 177a corrects the pixel values of the abnormal pixel regions, which are determined to have uniform pixel values in the surrounding region, based on the surrounding region statistics. The correction unit 177a may perform correction using different correction values for multiple abnormal pixel regions included in the thermal imaging data. For example, the correction unit 177a may perform correction for a part of the thermal imaging data using a correction value based on the object region statistics, which are statistics for the entire object region, as described in the first embodiment. On the other hand, the correction unit 177a may perform correction for the other parts using a correction value based on the surrounding region statistics as described in this embodiment.
[0096] Here, the uniformity of pixel values in the surrounding area will be described with reference to FIG. 8. FIG. 8 is a diagram for explaining the uniformity of pixel values in the surrounding area. Thermal imaging data D3 shown in FIG. 8 shows a plurality of examples of positions where abnormal pixel areas A occur in the thermal imaging data D2 (see FIG. 5) of the first embodiment. FIG. 8 shows examples of a case where the abnormal pixel area A occurs in the upper left of the figure and a case where the abnormal pixel area A occurs in the upper right of the figure. In FIG. 8, the abnormal pixel area A occurring in the upper left of the figure is shown as abnormal pixel area A1, and the abnormal pixel area A occurring in the upper right of the figure is shown as abnormal pixel area A2. The abnormal pixel areas A1 and A2 are, for example, abnormal pixel areas occurring in thermal imaging data captured at different times. However, here, the same drawing will be used for convenience in the description.
[0097] The left side of Fig. 8 shows a surrounding area P1 of the abnormal pixel area A. The right side of Fig. 8 shows a surrounding area P2. The surrounding areas P1 and P2 are, for example, rectangular areas located within 10 pixels from the top, bottom, left, and right ends of the abnormal pixel area A. The surrounding areas P1 and P2 are part of the sky SK. The surrounding areas P1 and P2 are areas that do not include the abnormal pixel area A.
[0098] In the surrounding area P1, there are utility poles 7 and electric wires 8. On the other hand, in the surrounding area P2, there are no utility poles 7 or electric wires 8. Therefore, in this case, when comparing the surrounding area P1 and the surrounding area P2, it can be said that the surrounding area P2 has a higher uniformity of pixel values.
[0099] For example, the determination unit 178 determines that the pixel values of the surrounding region P1 are not uniform, and determines that the pixel values of the surrounding region P2 are uniform. In this case, the calculation unit 175a calculates a surrounding region statistic, which is a statistic of the pixel values in the surrounding region P2.
[0100] The correction unit 177a calculates, for each pixel, a correction value for correcting the pixel values of the abnormal pixel region A based on the surrounding region statistics of the surrounding region P2. The correction unit 177a corrects the pixel values of the abnormal pixel region A using the calculated correction value.
[0101] The calculation unit 175a may not calculate the surrounding area statistics of the surrounding area P1 where the pixel value is determined to be not uniform. In this case, the calculation unit 175a may correct the pixel value of the abnormal pixel area A using the surrounding area statistics of the surrounding area P2 for the thermal image data when the abnormal pixel area A is on the left side of the figure. For example, when the abnormal pixel area A is caused by sunlight, the pixel value of the surrounding area may become uniform as the sun moves from the left side to the right side of the figure. Therefore, the calculation unit 175a can correct the pixel value of the abnormal pixel area A using the surrounding area statistics of the thermal image data captured at different timings.
[0102] In this embodiment, the determination as to whether the surrounding area is uniform is performed to calculate the correction value, but the present invention is not limited to this. For example, the determination unit 176 may determine whether the surrounding area is uniform and identify the abnormal pixel area according to the determination result. For example, the determination unit 176 takes into account the determination result as to whether the pixel values of the surrounding area of the abnormal pixel area are uniform in the determination of the abnormal pixel area described in the first embodiment. For example, the determination unit 176 maintains the result of identifying the abnormal pixel area when the pixel values of the surrounding area are uniform. Furthermore, the determination unit 176 may discard the result of identifying the abnormal pixel area when the pixel values of the surrounding area are not uniform. In this way, the abnormal pixel area is identified only when the uniformity of the surrounding area is high, and the determination unit 176 can identify the abnormal pixel area with high accuracy.
[0103] (Processing of image processing device 101a) Next, the process performed by the image processing device 101a will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the process performed by the image processing device 101a. The process shown in Fig. 9 differs from the process of the image processing device 101 in the first embodiment (see Fig. 6) in that steps S31 and S32 are added after step S18. The processes of steps S11 to S18 are the same as those of the first embodiment, so their description will be omitted, and the process from step S31 onwards will be described below.
[0104] After the specification unit 176 specifies the pixel region as an abnormal pixel region in step S18, the determination unit 178 determines whether the pixel values of the surrounding regions located around the abnormal pixel region in the object region are uniform (S31).
[0105] If it is determined that the pixel values of the surrounding region are not uniform (NO in S31), the process ends. If it is determined that the pixel values of the surrounding region are uniform (YES in S31), the calculation unit 175a calculates a surrounding region statistical value, which is a statistical value of the pixel values in the surrounding region, and the correction unit 177a calculates a correction value based on the surrounding region statistical value (S32). Then, the correction unit 177a corrects the pixel values of each pixel in the abnormal pixel region using the calculated correction value (S20).
[0106] As described above, in the image processing device 101a according to this embodiment, the determination unit 178 determines whether or not the pixel values of the surrounding region located around the abnormal pixel region in the object region are uniform. Furthermore, when it is determined that the pixel values of the surrounding region are uniform, the calculation unit 175a further calculates a surrounding region statistic, which is a statistical value of the pixel values in the surrounding region. Then, the correction unit 177a corrects the pixel values of the abnormal pixel region, which is determined to have uniform pixel values, based on the surrounding region statistic.
[0107] With this configuration, the image processing device 101a can evaluate the uniformity of pixel values in the surrounding area of the abnormal pixel area and correct the abnormal pixel area according to the evaluation result. Since the image processing device 101a calculates the correction value using the statistics of the surrounding area with uniform pixel values, it is possible to correct the abnormal pixel area with higher accuracy.
[0108] <Embodiment 3> Next, a third embodiment will be described. In the first and second embodiments, the image processing device 101 or 101a corrects the pixel values of the abnormal pixel region using one correction value. In the image processing device 101b according to this embodiment, when the abnormal pixel region spans multiple object regions, the image processing device 101b corrects the pixel values of the abnormal pixel region using the object region statistics of each of the multiple object regions. In the following, differences from the first and second embodiments will be mainly described, and descriptions of points that overlap with the first and second embodiments will be omitted as appropriate.
[0109] The image processing device 101b according to this embodiment includes an image processing unit 170b instead of the image processing unit 170 of the image processing device 101. The configuration of the image processing device 101b can be explained by replacing the image processing unit 170 of the image processing device 101 (see FIG. 1) of the first embodiment with the image processing unit 170b. Therefore, a block diagram showing the configuration of the image processing device 101b is omitted.
[0110] The configuration of the image processing unit 170b will be described with reference to Fig. 10. Fig. 10 is a block diagram showing the configuration of the image processing unit 170b. Unlike the image processing unit 170 of the first embodiment (see Fig. 3), the image processing unit 170b includes a division unit 179. Furthermore, the calculation unit 175b and the correction unit 177b have different functions from the calculation unit 175 and the correction unit 177, respectively.
[0111] The division unit 179 divides the abnormal pixel region into a plurality of division regions based on an object region related to the abnormal pixel region. The object region related to the abnormal pixel region is, for example, an object region to which the abnormal pixel region belongs. When an abnormal pixel region occurs so as to straddle a plurality of object regions, the abnormal pixel region belongs to a plurality of object regions. Therefore, in this case, each of the plurality of object regions is an object region related to the abnormal pixel region. Furthermore, the object region related to the abnormal pixel region may be, for example, an object region close to the abnormal pixel region, or an object region adjacent to the abnormal pixel region.
[0112] Furthermore, the object region associated with the abnormal pixel region may be, for example, another object region having a pixel value similar to that of the object region including the abnormal pixel region. For example, in the thermal imaging data, if the region of building X includes an abnormal pixel region, the region of building Y having a pixel value similar to that of building X is an object region associated with the abnormal pixel region.
[0113] Calculation unit 175b calculates object region statistics of the object regions corresponding to each of the divided regions. Correction unit 177b corrects the pixel values of each of the divided regions based on the object region statistics of the corresponding object region.
[0114] Here, the correction using the divided regions according to this embodiment will be specifically described with reference to Fig. 11 and Fig. 12. Fig. 11 is a diagram showing the state before the abnormal pixel region A is divided. Fig. 12 is a diagram showing the state after the abnormal pixel region A is divided. In Fig. 11 and Fig. 12, the thermal imaging data D4 and D5 include a sky SK, a utility pole 7, and the abnormal pixel region A. In Fig. 11 and Fig. 12, the abnormal pixel region A is shown by hatching for the sake of explanation.
[0115] In the thermal imaging data D4 shown in Fig. 11, the abnormal pixel region A occurs so as to straddle the sky SK and the utility pole 7. A part of the abnormal pixel region A is included in the sky SK. Furthermore, the part of the abnormal pixel region A that is not included in the sky SK is included in the utility pole 7. In other words, a part of the abnormal pixel region A belongs to the sky SK, and another part belongs to the utility pole 7.
[0116] The dividing unit 179 divides the abnormal pixel region A into a plurality of divided regions based on the sky SK and the utility pole 7. As a result, the dividing unit 179 divides the abnormal pixel region A into three divided regions A11 to A13, as shown in the thermal imaging data D5 in Fig. 12.
[0117] The dividing unit 179 may divide the abnormal pixel region A based on the location information, the photographing date and time information, and the map information described in the first embodiment. For example, the dividing unit 179 acquires the actual position of the utility pole 7 based on the location information, the photographing date and time information, and the map information. This allows the dividing unit 179 to detect that the region hidden by the abnormal pixel region A in the thermal imaging data D4 shown in Fig. 11 is a part of the utility pole 7. The dividing unit 179 may also divide the abnormal pixel region A using a known image recognition technique.
[0118] The calculation unit 175b calculates object region statistics of the object regions corresponding to each of the divided regions A11 to A13. Since the divided regions A11 and A13 are part of the sky SK, the object regions corresponding to the divided regions A11 and A13 are the sky SK. The calculation unit 175b calculates the object region statistics of the sky SK corresponding to the divided regions A11 and A13.
[0119] Moreover, since the divided region A12 is a part of the utility pole 7, the object region corresponding to the divided region A12 is the utility pole 7. The calculation unit 175b calculates the object region statistics value of the utility pole 7 corresponding to the divided region A12.
[0120] Correction unit 177b corrects the pixel values of divided regions A11 to A13 based on the object region statistics of the corresponding object regions. Correction unit 177b calculates a correction value for correcting the pixel values of divided regions A11 and A13 using the object region statistics of sky SK. Correction unit 177b also calculates a correction value for correcting the pixel value of divided region A12 using the object region statistics of utility pole 7. Correction unit 177b corrects the pixel values of divided regions A11 to A13 using the calculated correction value.
[0121] In this way, the correction unit 177b can use the correction values corresponding to each of the divided regions A11 to A13 to correct the pixel values of the abnormal pixel region A. This makes it possible to correct the abnormal pixel region with high accuracy even if the abnormal pixel region occurs across a plurality of object regions.
[0122] 11 and 12, electric wires and the like are omitted, but the image processing device 101b may perform the above-mentioned processing by including electric wires and the like in the correction target, similar to the utility pole 7. Alternatively, the image processing device 101b may perform the above-mentioned processing by excluding object regions having a size smaller than a predetermined size from the correction target. In this way, the image processing device 101b can avoid performing correction on objects having a size smaller than a predetermined size, such as electric wires.
[0123] 11 and 12, an example in which an abnormal pixel region spans multiple object regions has been described, but here, a case in which an object region is hidden by an abnormal pixel region, for example, when the abnormal pixel region is large, will be described. In such a case, the object region does not appear on the thermal imaging data, so the dividing unit 179 cannot divide the abnormal pixel region using the above-mentioned method.
[0124] In this case, the dividing unit 179 estimates an object region of an object hidden by the abnormal pixel region based on, for example, location information, shooting date and time information, and map information. The dividing unit 179 divides the abnormal pixel region into a plurality of divided regions based on the estimated object region.
[0125] The correction unit 177b may calculate the correction value using a pixel value set in advance for each object. For example, when a building area is hidden by an abnormal pixel area, the correction unit 177b calculates the correction value using the pixel value of the building area set in advance.
[0126] Alternatively, the correction unit 177b may determine whether or not there is a similar object region around the object region hidden by the abnormal pixel region, and calculate the correction value according to the determination result. For example, assume that there is a similar area of building Y around the area of building X hidden by the abnormal pixel region. In this case, the correction unit 177b calculates the correction value of building X based on the pixel value of building Y. The correction unit 177b may calculate the correction value of building X taking into account the scale of building Y and the position of building Y.
[0127] (Processing of image processing device 101b) Next, the process performed by the image processing device 101b will be described with reference to Fig. 13. Fig. 13 is a flowchart showing the process performed by the image processing device 101b. The process shown in Fig. 13 differs from the process of the image processing device 101 in the first embodiment (see Fig. 6) in that steps S41 and S42 are added after step S18. The processes of steps S11 to S18 are the same as those of the first embodiment, so their description will be omitted, and the process from step S41 onwards will be described below.
[0128] After the identification unit 176 identifies the pixel region as an abnormal pixel region in step S18, the division unit 179 determines whether the abnormal pixel region belongs to multiple object regions (S41). If it is determined that the abnormal pixel region does not belong to multiple object regions (NO in S41), the process proceeds to step S19.
[0129] When it is determined that the abnormal pixel region belongs to a plurality of object regions (YES in S41), the division unit 179 divides the abnormal pixel region into a plurality of divided regions (S42). The correction unit 177b calculates a correction value based on the object region statistics (S19). Specifically, the calculation unit 175b calculates the object region statistics of the object regions corresponding to each of the plurality of divided regions. Furthermore, the correction unit 177b calculates the correction values of each of the plurality of divided regions based on the object region statistics of the corresponding object region. The correction unit 177b corrects the pixel values of the abnormal pixel region using the calculated correction values (S20). In this embodiment, the correction unit 177b corrects the pixel values of each of the plurality of divided regions that constitute the abnormal pixel region.
[0130] As described above, the image processing device 101b according to this embodiment further includes a dividing unit 179 that divides the abnormal pixel region into a plurality of divided regions based on an object region related to the abnormal pixel region. With this configuration, the image processing device 101b can calculate a correction value corresponding to each divided region, and therefore can correct the pixel values of the abnormal pixel region with higher accuracy.
[0131] <Modification of the third embodiment> Next, a modified example of the third embodiment will be described. The image processing device 101b2 according to this modified example includes an image processing unit 170b2 having a different configuration from the image processing unit 170b of the image processing device 101b in the third embodiment. The configuration of the image processing device 101b2 can be explained by replacing the image processing unit 170 of the image processing device 101 (see FIG. 1) in the first embodiment with the image processing unit 170b2. Therefore, a block diagram showing the configuration of the image processing device 101b2 will be omitted.
[0132] In addition to the functions of the image processing unit 170b described in the third embodiment, the image processing unit 170b2 in this modified example has a function of acquiring a change trend of pixel values in the object region and correcting the pixel values based on the change trend.
[0133] For example, suppose that the thermal imaging data includes an area of a building. The pixel values of multiple points in the area of the building may differ by a certain amount or more due to the influence of sunlight, etc. For example, the temperature of a part of the building that is exposed to sunlight will be higher than that of a shaded part that is not exposed to sunlight. Note that the influence of not only sunlight but also wind direction, etc. may also be assumed.
[0134] In such a case, it is assumed that the abnormal pixel region straddles an object region corresponding to a high temperature region and an object region corresponding to a low temperature region. In such a case, if the abnormal pixel region is corrected using one correction value, there is a possibility that the pixel values of the abnormal pixel region cannot be appropriately corrected. This modification solves such a problem.
[0135] The configuration of the image processing unit 170b2 according to this modified example will be described with reference to Fig. 14. Fig. 14 is a block diagram showing the configuration of the image processing unit 170b2 according to this modified example. The image processing unit 170b2 differs from the configuration of the image processing unit 170b (see Fig. 10) of the third embodiment in that it includes a trend acquisition unit 1791. Furthermore, the correction unit 177b2 has a different function from that of the correction unit 177b.
[0136] The trend acquiring unit 1791 acquires a change trend of pixel values of an object region related to an abnormal pixel region. The change trend of pixel values is information indicating a change trend of pixel values in at least a part of the object region.
[0137] The trend acquiring unit 1791 acquires, for example, a change trend of pixel values between a plurality of points in an object region. For example, the trend acquiring unit 1791 acquires a change trend of pixel values by analyzing pixel values between a first point and a second point in an object region. The trend acquiring unit 1791 acquires, for example, a change trend such as pixel values increasing from the first point to the second point, or pixel values decreasing from the first point to the second point.
[0138] The positional relationship between the first and second points is arbitrary. For example, the first and second points may be arranged above and below, or may be arranged left and right. The trend acquiring unit 1791 may use three or more points to acquire the change trend of pixel values between the points.
[0139] The correction unit 177b2 calculates a correction value based on the change trend acquired by the trend acquisition unit 1791. For example, the correction unit 177b2 calculates a plurality of correction values for correcting one abnormal pixel region based on the change trend of pixel values. The correction unit 177b2 may also calculate correction values corresponding to a plurality of divided regions based on the trend of pixel values of the object region.
[0140] The trend acquiring unit 1791 may acquire the change trend using all object regions included in the thermal imaging data, or may acquire the change trend using a part of the object regions included in the thermal imaging data. For example, the trend acquiring unit 1791 may acquire the change trend using the object regions of objects related to the divided regions among the object regions included in the image data.
[0141] The process in which the trend acquiring unit 1791 acquires the change trend using all object regions included in the thermal imaging data will be described. For example, assume that one object region is entirely hidden by an abnormal pixel region in the thermal imaging data. In such a case, the trend acquiring unit 1791 acquires the change trend using all object regions included in the thermal imaging data. For example, the trend acquiring unit 1791 acquires a change trend in which the pixel value on the right side is greater than the pixel value on the left side in a predetermined ratio or more of the object regions among the multiple object regions included in the thermal imaging data. In this case, the correction unit 177b2 calculates a correction value so that the pixel value on the right side of the abnormal pixel region is greater than the pixel value on the left side.
[0142] Specifically, the correction unit 177b2 calculates the difference between the pixel value at the right end and the pixel value at the left end in each of a plurality of object regions that show a change tendency in the thermal imaging data such that the pixel value at the right end is greater than the pixel value at the left end. The correction unit 177b2 calculates the correction value using the average value of the calculated differences. The correction unit 177b2 may calculate the correction value using the mode or median of the calculated differences.
[0143] For example, in a plurality of object regions showing a change tendency in which pixel values on the right side are greater than pixel values on the left side, the pixel values on the right edge tend to be 20 greater on average than the pixel values on the left edge. In such a case, the correction unit 177b2 calculates a correction value such that the pixel value on the right edge of the abnormal pixel region is 20 greater than the pixel value on the left edge. For example, the correction unit 177b2 calculates a correction value for each pixel such that the pixel values change with an equal change width from the right edge to the left edge of the abnormal pixel region. In this way, the correction unit 177b2 can calculate a correction value such that the pixel values decrease stepwise from the right edge to the left edge of the abnormal pixel region.
[0144] The following describes the process in which the trend acquiring unit 1791 acquires a change trend using a part of an object region included in thermal imaging data. For example, as shown in Fig. 12, the abnormal pixel region A is divided into multiple divided regions A11 to A13. In this case, the trend acquiring unit 1791 acquires the change trend of pixel values of the utility pole 7 to which a part of the abnormal pixel region A (divided region A12) belongs.
[0145] For example, it is assumed that the pixel value in the upper region of the divided region A12 in the region of the utility pole 7 is 20 greater than the pixel value in the lower region of the divided region A12. The correction unit 177b2 calculates a correction value for the divided region A12 such that the pixel value at the upper end of the divided region A12 is 20 greater than the pixel value at the lower end of the divided region A12.
[0146] Specifically, the correction unit 177b2 calculates a correction value for each pixel such that the pixel value changes with an equal change width from the top to the bottom of the divided region A12. In this way, the correction unit 177b2 can calculate the correction value such that the pixel value decreases stepwise from the top to the bottom of the divided region A12. Note that the correction unit 177b2 may calculate the correction value such that the pixel value at the top end and the pixel value at the bottom end of the divided region A12 are the same value or close to the pixel value of the adjacent utility pole 7.
[0147] The above-mentioned method of calculating the correction value is an example, and the correction unit 177b2 may calculate the correction value using another calculation method. Note that the correction unit 177b2 may calculate the correction value so that the pixel value changes with a non-uniform change width.
[0148] (Processing of image processing device 101b2) Next, the process performed by the image processing device 101b2 will be described with reference to Fig. 15. Fig. 15 is a flowchart showing the process performed by the image processing device 101b2. The process shown in Fig. 15 differs from the process of the image processing device 101b in the third embodiment (see Fig. 13) in that step S43 is added after step S42. The processes of steps S11 to S42 are the same as those of the third embodiment, so their description will be omitted, and the process from step S43 on will be described below.
[0149] In step S42, after the dividing unit 179 divides the abnormal pixel region into a plurality of divided regions, the trend acquiring unit 1791 acquires the change trend of the pixel values of the object region (S43). The correcting unit 177b2 calculates a correction value based on the object region statistics (S19). In this modification, the correcting unit 177b2 calculates a correction value based on the acquired change trend. For example, the correcting unit 177b2 calculates a correction value for each pixel so that the pixel value changes stepwise from one end to the other end of the abnormal pixel region. Then, the correcting unit 177b2 corrects the pixel values of the abnormal pixel region (S20).
[0150] As described above, in the image processing device 101b2 according to this modification, the trend acquiring unit 1791 acquires the change trend of the pixel values of the object region related to the abnormal pixel region. Furthermore, the correction unit 177b2 calculates a correction value based on the change trend, and corrects the pixel values of the abnormal pixel region using the correction value. In this way, the image processing device 101b2 according to this modification can correct the abnormal pixel region so that the pixel values of the abnormal pixel region change stepwise according to the pixel values of the surrounding region.
[0151] Each functional component of the image processing devices 101, 101a, 101b, and 101b2 described above may be realized by hardware (e.g., a hardwired electronic circuit, etc.) that realizes each functional component, or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). For example, as described above, the present disclosure can also realize any process by having a CPU execute a computer program.
[0152] The program includes a set of instructions (or software code) for making the computer perform one or more functions described in the embodiment when the program is loaded into the computer. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. As an example, but not limited to, the non-transitory computer-readable medium or the tangible storage medium includes a RAM, a ROM, a flash memory, a solid-state drive (SSD), or other memories or various storage devices. The program may also be transmitted on a temporary computer-readable medium or a communication medium.
[0153] The present disclosure is not limited to the above-described embodiments, and can be modified as appropriate without departing from the spirit and scope of the present disclosure. Moreover, the above-described first to third embodiments and the modified examples can be combined in any manner.
[0154] For example, in the description of the modified example of the third embodiment, the configuration of the image processing unit 170b2 has been described as an example in which the image processing unit 170b according to the third embodiment further includes the trend acquisition unit 1791. However, this is not limited to this, and the image processing unit 170 according to the first embodiment or the image processing unit 170a according to the second embodiment may include the configuration of the trend acquisition unit 1791. For example, it is assumed that the image processing unit 170 according to the first embodiment includes the trend acquisition unit 1791. It is also assumed that the abnormal pixel region is not divided. In this case, the correction unit 177 of the image processing unit 170 corrects the pixel values of the entire abnormal pixel region based on the change trend of the pixel values of the object region related to the abnormal pixel region. In this way, the correction unit 177 can correct the pixel values of the abnormal pixel region generated in, for example, a sky region based on the change trend of the pixel values of the sky region. [Explanation of symbols]
[0155] 1. Imaging system 5. Vehicles 7. Electric pole 8 Electric wire 51 Muffler 101, 101a, 101b, 101b2 Image processing device 110 Control IF 140 System control section 150 Thermal image data acquisition unit 160 High temperature object area acquisition part 170, 170a, 170b, 170b2 Image processing unit 171 Defective pixel correction section 172 NUC Department 173 Object area acquisition unit 174 Trajectory acquisition part 175, 175a, 175b Calculation section 176 Specific section 177, 177a, 177b, 177b2 Correction section 178 Judgment Department 179 Division 1791 Trend acquisition department 180 Display control unit 200 Infrared Camera 210 Lens 220 Shutter 230 Infrared Sensor 240 Transmission Devices 250 Temperature Sensor 300 display device A, A1, A2 abnormal pixel area A11~A13 divided area D1~D5 Thermal imaging data P1, P2 surrounding area SK sky SU sun
Claims
1. a thermal image data acquisition unit that acquires thermal image data from an infrared camera that captures thermal images; an object area acquisition unit that acquires an object area indicating an area of an object in the thermal image data; a trajectory acquisition unit for acquiring a movement trajectory of a high-temperature object corresponding to an imaging range of the thermal imaging data; A calculation unit that calculates an object region statistical value that is a statistical value of pixel values in at least a portion of the object region; a specifying unit that specifies, among the object regions, a pixel region having a pixel value whose difference from the object region statistical value is equal to or greater than a predetermined value and that is included in the movement trajectory, as an abnormal pixel region; a correction unit that corrects pixel values of the abnormal pixel region based on the object region statistics. Image processing device.
2. a determination unit that determines whether pixel values of a peripheral region located around the abnormal pixel region in the object region are uniform; When it is determined that the pixel values of the surrounding region are uniform, the calculation unit further calculates a surrounding region statistical value which is a statistical value of the pixel values in the surrounding region; The correction unit corrects pixel values of the abnormal pixel region in which pixel values of the surrounding region are determined to be uniform based on the surrounding region statistics. The image processing device according to claim 1 .
3. a dividing unit that divides the abnormal pixel region into a plurality of divided regions based on the object region associated with the abnormal pixel region; the calculation unit calculates the object region statistical value of an object region corresponding to each of the plurality of divided regions; The correction unit corrects pixel values of the plurality of divided regions based on the object region statistics of the corresponding object regions.
3. The image processing device according to claim 1 or 2.
4. a trend acquisition unit that acquires a trend of change in pixel values of the object region associated with the abnormal pixel region, The correction unit corrects pixel values of the abnormal pixel region further based on the change trend.
3. The image processing device according to claim 1 or 2.
5. A thermal image data acquisition step of acquiring thermal image data from an infrared camera that captures a thermal image; an object area acquisition step of acquiring an object area indicating an area of an object in the thermal image data; a trajectory acquisition step of acquiring a movement trajectory of a high-temperature object corresponding to an imaging range of the thermal imaging data; A calculation step of calculating an object region statistic, which is a statistic of pixel values in at least a portion of the object region; a specifying step of specifying, as an abnormal pixel region, a pixel region among the object regions, the pixel region having a pixel value whose difference from the object region statistical value is equal to or greater than a predetermined value and which is included in the movement trajectory; and a correction step of correcting pixel values of the abnormal pixel region based on the object region statistics. Image processing methods.
Citation Information
Patent Citations
Defective pixel correction apparatus, method for controlling the same, and program for causing computer to execute the method
JP2013183282A
Method and device for removing residual images in infrared images of changing scenes
JP2022530660A
Inpainting device and method using segmentation of reference region
US20180158178A1
Image processing device, image processing method, and program
WO2022018900A1
