Image processing device and image processing method

Abstract

To provide a novel technique for simultaneously providing a plurality of infrared images on which different gain control processes are executed.SOLUTION: An image processing device 20 comprises: an image acquisition unit 21 for acquiring infrared images from an infrared imaging device; a gain adjustment unit 24 which executes a plurality of different gain control processes on the acquired infrared images; and a transmission processing unit 25 which transmits an image signal in which signals showing the luminance of each pixel concerning each of the infrared images subjected to the different gain control processes are arranged serially by each pixel.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image processing device and an image processing method. [Background technology]

[0002] Far-infrared cameras are capable of imaging the heat of an object by converting the temperature change caused by the absorption of infrared rays into an electrical signal. Far-infrared cameras are used for measuring the temperature of objects and obtaining images that do not depend on a light source, such as for nighttime surveillance.

[0003] The far-infrared camera performs AGC (Auto Gain Control) to create an image signal optimized for each application, and transmits the image using a standard transmission protocol. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-15705 A Summary of the Invention [Problem to be solved by the invention]

[0005] As shown in Patent Document 1, in far-infrared cameras, the method of gain adjustment in the internal AGC needs to be changed depending on the application. For example, in temperature measurement applications, the gain is adjusted so that the temperature range around body temperature can be detected with high accuracy, and in monitoring applications, the gain is adjusted so that the contrast of the object is clear. As a result, in temperature measurement applications, an image with an unclear contrast of the object may be presented, and in monitoring applications, an image that does not properly show the absolute temperature of the object may be presented.

[0006] In view of the above, an object of the present disclosure is to provide a new technique that makes it possible to provide multiple infrared images in which different gain control processes have been performed. [Means for solving the problem]

[0007] In order to solve the above problems, an image processing device of one aspect of the present disclosure includes an image acquisition unit that acquires an infrared image from an infrared imaging device, a gain adjustment unit that performs a plurality of different gain control processes on the acquired infrared image, and a transmission processing unit that transmits a video signal in which a signal indicating the brightness of each pixel for each of the infrared images that has been subjected to the plurality of different gain control processes is arranged in serial for each pixel.

[0008] Another aspect of an image processing method of the present disclosure includes the steps of acquiring an infrared image from an infrared imaging device, performing a plurality of different gain control processes on the acquired infrared image, and transmitting a video signal in which signals indicating the brightness of each pixel for each of the infrared images that have been subjected to the plurality of different gain control processes are arranged in serial for each pixel.

[0009] Any combination of the above components, and conversion of the expression of this embodiment between a method, an apparatus, a system, a recording medium, a computer program, etc. are also valid as aspects of this embodiment. Effect of the Invention

[0010] According to the present disclosure, it is possible to provide a new technique for simultaneously providing multiple infrared images, each of which has been subjected to different gain control processing. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram showing a configuration of an infrared imaging system according to an embodiment; [Diagram 2] FIG. 4 is a functional block diagram of an automatic gain control unit according to the embodiment. [Diagram 3] FIG. 1 is a diagram illustrating an example of a data structure of a general video signal. [Figure 4] 3 is a diagram illustrating a data structure of a video signal according to the embodiment; [Diagram 5] 4 is a flowchart of a process performed by the image processing device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] FIG. 1 is a diagram showing a configuration of an infrared imaging system 1 according to an embodiment of the present disclosure. The infrared imaging system 1 according to the embodiment includes an infrared imaging device 10 and an infrared image processing device 20. In this embodiment, a far-infrared image imaging device that captures far-infrared rays of 8 μm to 14 μm is assumed as the infrared imaging device 10. Unlike near-infrared rays, far-infrared rays can be captured without providing a light source. For this reason, in this embodiment, an infrared image processing device will be described as an example of an image processing device.

[0013] The infrared imaging device 10 includes a microbolometer 11, an amplifier (AMP) 12, an AD converter (ADC) 13, a thermistor 14, and an AD converter (ADC) 15. In this embodiment, the microbolometer 11 is assumed to be the far-infrared detection element.

[0014] The microbolometer 11 is a non-cooled thermal detection element. The microbolometer 11 includes an FPA (Focal Plane Array) in which a plurality of bolometers are arranged in a two-dimensional lattice (for example, 640×480, 1024×768). Each bolometer constituting one pixel has an absorption layer made of amorphous silicon or vanadium oxide. When far-infrared rays are incident on the absorption layer, the temperature of the absorption layer changes, and the resistance value of the absorption layer changes. By applying a predetermined voltage to two electrodes of the absorption layer, the change in the resistance value of the absorption layer is detected as a change in current value. The current value of each bolometer is converted into a voltage and output as a luminance signal of each pixel defined by an electric signal.

[0015] The amplifier 12 amplifies the analog voltage of each pixel with a predetermined gain. The AD converter 13 converts the analog voltage of each pixel amplified by the amplifier 12 into a digital value at a predetermined sampling period. In this embodiment, the digital value is converted into a 14-bit digital value.

[0016] The infrared imaging device 10 outputs an infrared thermal image, in which each pixel is defined by a 14-bit luminance signal, to the infrared image processing device 20. If a higher resolution thermal image is to be generated, the analog voltage of each pixel may be converted to a digital value of 16 bits or more. Conversely, if a lower cost configuration is desired, the analog voltage may be converted to a digital value of 12 bits or less.

[0017] The thermistor 14 is disposed near the microbolometer 11 and is a temperature detection element for measuring the environmental temperature around the microbolometer 11. The temperature around the microbolometer 11 is affected by the temperature of the microbolometer itself as well as the temperature of a lens (not shown) disposed in front of the microbolometer.

[0018] The thermistor 14 is connected in series with a resistor element and detects a change in the environmental temperature as a change in the divided voltage. The ADC 15 converts an analog voltage corresponding to the environmental temperature into a digital value. The infrared imaging device 10 outputs the measured environmental temperature to the infrared image processing device 20.

[0019] The thermistor 14 does not necessarily have to be installed inside the infrared imaging device 10, and may be installed outside the infrared imaging device 10 at a position where it can estimate the ambient temperature of the microbolometer 11. The thermistor 14 is an example of a temperature detection element, and other temperature detection elements such as a thermocouple may also be used.

[0020] The infrared image processing device 20 includes an image acquisition unit 21, a defective pixel correction unit 22, a non-uniformity correction unit 23, an auto gain control unit 24, a transmission processing unit 25, a temperature acquisition unit 26, and a sensitivity characteristic correction unit 27. These components can be realized by a combination of hardware resources and software resources, or by hardware resources alone. As hardware resources, a CPU, ROM, RAM, GPU (Graphics Processing Unit), DSP (Digital Signal Processor), ISP (Image Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and other LSIs can be used. As software resources, programs such as firmware can be used.

[0021] The image acquisition unit 21 acquires an infrared image from the infrared imaging device 10. The defective pixel correction unit 22 corrects the luminance signal of a defective pixel in which an abnormality occurs in the FPA, using interpolation data generated from normal pixels surrounding the defective pixel.

[0022] In a typical microbolometer 11, the gain and offset are non-uniform for each pixel. The non-uniformity correction unit 23 corrects the non-uniformity of the gain and offset between pixels. For example, with external incident light blocked, the microbolometer 11 is uniformly irradiated with far-infrared light using a blackbody shutter, and the gain and offset of each pixel are derived to generate a correction table so that the output of all pixels in the microbolometer 11 is constant. The non-uniformity correction unit 23 adjusts the gain and offset of each pixel by referring to the correction table.

[0023] The auto gain control (AGC) unit 24 executes a gain control process on the infrared image input from the non-uniformity correction unit 23. The gain control process is a process of determining a gain value and multiplying the determined gain value by the luminance signals of all pixels of the infrared image to adjust the sensitivity of the infrared image. The auto gain control unit 24 grasps the overall state of the infrared image input from the non-uniformity correction unit 23 and adjusts the gain according to the intended use. For example, the auto gain control unit 24 adjusts the sensitivity of the infrared image by adjusting the gain of the luminance signal of each pixel based on the sensitivity characteristics of the microbolometer 11 to the environmental temperature. The auto gain control unit 24 in this embodiment is an example of a gain adjustment unit.

[0024] The auto gain control unit 24 obtains a gain value by applying the environmental temperature acquired by the temperature acquisition unit 26 to the sensitivity characteristics described in a table or function created by a designer (described below), and multiplies the obtained gain value by the luminance signals of all pixels of the infrared image to adjust the sensitivity of the infrared image.

[0025] The transmission processing unit 25 converts the image created by the auto gain control unit 24 into a standard image format and outputs it as an image signal. The transmission processing unit 25 converts the digital signal format of the infrared image output from the infrared imaging device 10 into an image signal format for output. Many video devices use 8-bit (256 gradations) digital signal formats. The resolution of luminance that the human eye can distinguish is around 7-bit (128 gradations). In contrast, in this embodiment, 14-bit (16,384 gradations) is used as the digital signal format of the infrared image output from the infrared imaging device 10.

[0026] In this embodiment, the transmission processing unit 25 converts the 14-bit grayscale infrared image output from the infrared imaging device 10 into an 8-bit grayscale infrared image. Specifically, the transmission processing unit 25 maps the 14-bit luminance signal to an 8-bit luminance signal using a linear curve or a histogram flattening curve.

[0027] When the contrast of the infrared image output from the infrared imaging device 10 is low, the brightness histogram is concentrated in a specific range. In this case, the transmission processing unit 25 flattens the brightness histogram by lowering the peak of the brightness histogram and widening it on both sides. The transmission processing unit 25 maps the flattened 14-bit brightness signal to an 8-bit brightness signal. This improves the contrast of the infrared image.

[0028] Furthermore, when the contrast of the infrared image output from the infrared imaging device 10 is high and a flat area occurs in the brightness histogram, the transmission processing unit 25 maps the 14-bit brightness signal to an 8-bit brightness signal, excluding the flat area. For example, in a scene in which a small high-temperature object exists on a flat background, the brightness histogram will have a flat shape in the center. In this case, the contrast of the infrared image can be improved by mapping excluding the flat center area.

[0029] The infrared image converted to 8-bit grayscale by the transmission processing unit 25 is output to a monitor (not shown) for display. The converted infrared image may be output to a recording medium (not shown) and recorded on the recording medium. When recorded on the recording medium, the amount of data may be further compressed and recorded according to a predetermined compression encoding standard. The converted infrared image may be output to an image recognition engine (not shown), and a process of detecting a predetermined object (e.g., a person) from within the infrared image may be performed.

[0030] The temperature acquisition unit 26 acquires the environmental temperature of the microbolometer 11 from the infrared imaging device 10 .

[0031] The designer derives in advance a sensitivity characteristic indicating the relationship between the environmental temperature of the microbolometer 11 and the luminance signal output by the microbolometer 11 based on an experiment or a simulation. The sensitivity characteristic is a conversion characteristic for converting an infrared image actually acquired by the microbolometer 11 into a pseudo infrared image in which the environmental temperature of the microbolometer 11 is considered to be at a constant standard temperature. The designer sets a value of the luminance signal when the environmental temperature of the microbolometer 11 is the standard temperature. The designer determines a gain value to be multiplied by the luminance signal at each environmental temperature so that the value of the luminance signal becomes the value at the standard temperature regardless of the environmental temperature of the microbolometer 11. In the generated sensitivity characteristic, for example, the gain value to be multiplied by the luminance signal at each environmental temperature may be described as a table or as a function.

[0032] While explaining an example of the gain control process of the auto gain control unit 24, the problem of the gain control process will be explained. Generally, a transmitted image is quantized in 8 bits. In this case, it is necessary to adjust the image so that the entire screen falls within the range of 8 bits. For example, depending on the application, the gain of pixels outside the range of 8 bits is adjusted or saturated so that they become 8 bits. In general, if the temperature range (dynamic range) that can be imaged is set wide, the contrast of the object to be detected may be reduced, which may cause detection problems. In addition, if the temperature range (dynamic range) that can be imaged is set narrow, the number of saturated pixels and zero pixels may increase, which may make it difficult to grasp the situation.

[0033] For example, when measuring the temperature of an object within a certain temperature range, it is more suitable for the purpose of measuring the temperature to determine the upper and lower limits of the temperature on the screen so that the accuracy of the temperature measurement within the temperature range is maximized, while saturating the area outside the temperature range. For example, it is necessary to measure accurately around 37°C, which is close to body temperature, but it is not a problem for the purpose of measuring the temperature even if it is saturated around 30°C or 50°C, which are clearly far from body temperature. An image showing the temperature in a specific temperature range in detail for such temperature measurement purposes can accurately grasp the temperature of the specific temperature range, but the brightness of all other areas is the same, so the image tends to have a low overall resolution.

[0034] Furthermore, for surveillance purposes, it is more useful to have an image with high contrast so that objects can be easily recognized, rather than temperature accuracy. Dividing the image into small blocks and changing the gain for each block or object to emphasize the edges of the image makes it easier to recognize, rather than displaying the entire screen with the same gain. In this case, since the edges of each block tend to look unnatural, it is preferable to adjust the brightness based on the brightness of the edges of adjacent blocks. Such high-contrast images for surveillance purposes are useful for surveillance and recognition because they make the contours of objects clear, but they make it difficult to clearly grasp the temperature of the object.

[0035] In this way, the gain adjustment of far-infrared cameras depends on the purpose of use. Therefore, different gain control processes are set depending on the purpose or the object to be detected, and the image itself changes significantly.

[0036] On the other hand, there are cases where the far-infrared camera is used for multiple purposes, such as when it is used for monitoring purposes but also requires a certain degree of temperature measurement accuracy. In this case, it may be necessary to use a method such as switching the gain control process in the auto gain control unit 24 between monitoring purposes and temperature measurement purposes, using multiple infrared imaging systems 1, or outputting raw data from the infrared imaging device 10 to a monitor (not shown) downstream of the infrared image processing device 20 and performing auto gain control on the monitor.

[0037] When switching the processing of the auto gain control unit 24, it is not possible to process the same image at the same time, so it is not possible to display an image with the gain adjusted for monitoring and an image with the gain adjusted for temperature measurement in parallel for the same image. Therefore, it is necessary to consider the effort and timing of switching.

[0038] When multiple infrared imaging systems 1 are used, the angles of view of the images captured by each infrared imaging system 1 are not the same, so it is necessary to create an image by combining the images, which increases the processing load for the combination process. In addition, it is necessary to prepare multiple cameras, which increases costs.

[0039] When the raw data of the infrared imaging device 10 is processed by a monitor or the like downstream of the infrared image processing device 20, a dedicated interface is required for connection to the downstream. In addition, it is necessary to set the downstream processing in accordance with the processing performed on the infrared imaging system 1 side, which is troublesome.

[0040] Taking into consideration the above-mentioned problems with the gain control process, the present embodiment will be described in detail. Fig. 2 is a functional block diagram of the auto gain control unit 24 of the present embodiment. The auto gain control unit 24 of the present embodiment includes a first adjustment unit 31, a second adjustment unit 32, and a third adjustment unit 33. The auto gain control unit 24 executes a plurality of different gain control processes.

[0041] The first to third adjustment units 31 to 33 execute three different gain control processes. For example, the first adjustment unit 31 executes a first gain control process for measuring the temperature in a temperature range of about 37° C., which is close to body temperature. For example, the second adjustment unit 32 executes a second gain control process for measuring the temperature in a temperature range higher than body temperature. For example, the third adjustment unit 33 executes a third gain control process for monitoring purposes to generate a high-contrast image for human detection purposes. The first to third gain control processes are different processes. The above uses are examples, and the gain control processes executed by the first to third adjustment units 31 to 33 can be appropriately changed depending on the purpose. The infrared image on which the gain control process has been executed in each of the first to third adjustment units 31 to 33 is output to the transmission processing unit 25.

[0042] The transmission processing unit 25 of this embodiment combines the input infrared images into one and transmits it as a standard video signal. The video signal of this embodiment is a signal in the YCrCb color space represented by a luminance Y and two color differences Cr and Cb.

[0043] Here, FIG. 3 illustrates the data structure of a general video signal. As shown in FIG. 3, in a general video signal, three signals indicating luminance Y, color difference Cb, and color difference Cr are serially arranged for each pixel. For example, in the example of the video signal in FIG. 3, a luminance signal Y0, a first color difference signal Cb0, and a second color difference signal Cr0 for pixel P0, a luminance signal Y1, a first color difference signal Cb1, and a second color difference signal Cr1 for pixel P1, and a luminance signal Y2, a first color difference signal Cb2, and a second color difference signal Cr2 for pixel P2 are serially arranged. The luminance signal Y, the first color difference signal Cb, and the second color difference signal Cr are serially transmitted for each pixel. However, among these luminance signal Y, first color difference signal Cb, and second color difference signal Cr, only the luminance signal Y is useful information when displaying an infrared image captured by the infrared imaging device 10 as a black and white image, and the first color difference signal Cb and the second color difference signal Cr are fixed values.

[0044] 4 illustrates the data structure of the video signal of this embodiment. In the video signal of this embodiment, the first luminance signal Ya generated by the first adjustment unit 31, the second luminance signal Yb generated by the second adjustment unit 32, and the third luminance signal Yc generated by the third adjustment unit 33 are serially arranged. For example, in the video signal of this embodiment, the first luminance signal Ya0, the second luminance signal Yb0, and the third luminance signal Yc0 for pixel P0, the first luminance signal Ya1, the second luminance signal Yb1, and the third luminance signal Yc1 for pixel P1, and the first luminance signal Ya2, the second luminance signal Yb2, and the third luminance signal Yc2 for pixel P2 are serially arranged. That is, the video signal of this embodiment includes the second luminance signal Yb and the third luminance signal Yc instead of the first color difference signal Cb2 and the second color difference signal Cr2 of FIG. 3, which are fixed values. In this way, the transmission processing unit 25 of this embodiment can transmit a video signal in which the first to third luminance signals Ya to Yc are arranged serially for each pixel.

[0045] When the video signal is displayed on a monitor downstream of the transmission processing unit 25, the monitor is set to a black and white mode that is normally provided on monitors. In this embodiment, the transmission of the infrared image itself is the same as a standard method, so that it is possible to display each image based on the first to third luminance signals Ya to Yc contained in the video signal transmitted from the transmission processing unit 25 in parallel on the monitor without providing a dedicated device.

[0046] FIG. 5 is a flow chart illustrating a process S100 of the infrared image processing device 20 of this embodiment.

[0047] In step S101, the image acquisition unit 21 acquires an infrared image from the infrared imaging device 10.

[0048] In step S102, the defective pixel correction unit 22 corrects the luminance signal of a defective pixel in the infrared image using the interpolated data.

[0049] In step S103, the non-uniformity corrector 23 corrects non-uniformity in gain and offset between pixels for the infrared image in which the defective pixels have been corrected.

[0050] In step S104, the first to third adjusters 31 to 33 execute first to third gain control processes, respectively, on the infrared image whose non-uniformity has been corrected.

[0051] In step S105, the transmission processing unit 25 transmits a video signal in which the luminance signals of the pixels of the infrared images that have been subjected to the first to third gain control processes are arranged serially for each pixel.

[0052] After step S105, the process S100 ends.

[0053] In this embodiment, the auto gain control unit 24 executes a plurality of different gain control processes on the acquired infrared image, and the transmission processing unit 25 transmits a video signal in which a signal indicating the luminance of each pixel for each of the infrared images that have been subjected to the plurality of different gain control processes is arranged in series for each pixel. With this configuration, it is possible to display a plurality of infrared images that have been subjected to the different gain control processes in parallel on a monitor using a simple and low-cost method.

[0054] In this embodiment, the auto gain control unit 24 executes three different gain control processes on the acquired infrared image, and the transmission processing unit 25 transmits a video signal in which signals indicating the luminance of each pixel are arranged in series, three for each pixel. With this configuration, the transmission of the infrared image itself is the same as a standard method, so it is possible to display three infrared images that have been subjected to different gain control processes in parallel on a monitor without providing any dedicated equipment.

[0055] In this embodiment, the video signal is a signal in a color space represented by luminance Y and color differences Cr and Cb, and the video signal includes a signal indicating luminance instead of signals indicating color differences Cr and Cb for each pixel of the infrared images that have been subjected to three different gain control processes. With this configuration, it is possible to display black-and-white images based on the infrared images that have been subjected to three different gain control processes in parallel on a monitor.

[0056] In the present embodiment, a plurality of different gain control processes are set according to the intended use of the infrared image, making it possible to execute an appropriate gain control process according to the intended use.

[0057] The present disclosure has been described above based on the embodiments. These embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present disclosure.

[0058] In the embodiment, an example has been described in which the microbolometer 11 is used as a thermal type far-infrared detection element. In this regard, the correction control of the sensitivity characteristics according to the embodiment can also be applied to a far-infrared detection element using a thermopile or a focusing element.

[0059] In the embodiment, three different gain control processes are executed, but the present invention is not limited to this, and two different gain control processes may be executed. In this case, for example, the auto gain control unit 24 may include a first adjustment unit 31 and a second adjustment unit 32, and the transmission processing unit 25 may transmit a video signal in which a first luminance signal generated by the first adjustment unit 31, a second luminance signal generated by the second adjustment unit 32, and a signal indicating color difference Cr or Cb by a fixed value are serially arranged. In addition, when the transmission processing unit 25 transmits a video signal in which four or more signals are serially arranged for each pixel, the auto gain control unit 24 may execute four or more different gain control processes.

[0060] In the embodiment, the video signal is a signal in the YCrCb color space represented by luminance Y and two color differences Cr and Cb, but it may also be a signal in the RGB color space represented by the three primary colors red, green, and blue.

[0061] In the embodiment, the gain control process is set according to the use of the infrared image, but the present invention is not limited to this and may be set according to the object to be detected in the infrared image. For example, if the object to be detected is a human, the gain control process may be set so that the temperature of the human body can be accurately detected at about 37°C, which is close to body temperature. [Explanation of symbols]

[0062] 1 infrared imaging system, 10 infrared imaging device, 11 microbolometer, 12 amplifier, 13 AD converter, 14 thermistor, 15 AD converter, 20 infrared image processing device, 21 image acquisition section, 22 defective pixel correction section, 23 non-uniformity correction section, 24 auto gain control section, 25 transmission processing section, 26 temperature acquisition section, 31 first adjustment section, 32 second adjustment section, 33 third adjustment section.

Claims

1. an image acquisition unit that acquires an infrared image from an infrared imaging device; a gain adjustment unit that performs a plurality of different gain control processes on the acquired infrared image; a transmission processing unit that transmits a video signal in which a signal indicating the luminance of each pixel for each of the infrared images that have been subjected to a plurality of different gain control processes is serially arranged for each pixel; An image processing device comprising:

2. the plurality of different gain control processes are two or three different gain control processes; the transmission processing unit transmits the video signal in which signals indicating the luminance of each pixel are arranged in series in groups of three for each pixel. The image processing device according to claim 1 .

3. The video signal is a signal in a color space represented by luminance Y and color differences Cr and Cb, the video signal includes a signal indicating the luminance instead of signals indicating color differences Cr and Cb for each pixel of the infrared image that has been subjected to the two or three different gain control processes; The image processing device according to claim 2 .

4. The plurality of different gain control processes are set according to at least one of a purpose of the infrared image and an object to be detected. The image processing device according to claim 1 .

5. acquiring an infrared image from an infrared imaging device; performing a plurality of different gain control operations on the captured infrared image; transmitting a video signal in which a signal indicating the brightness of each pixel for each of the infrared images that have been subjected to a plurality of different gain control processes is serially arranged for each pixel; An image processing method comprising:

Images