Method for processing an infrared image by fusing a plurality of infrared image colorimetric processing operations
Patent Information
- Application Number
- EP2024712116
- Authority / Receiving Office
- EP · EP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-02-23
- Filing Date
- 2024-02-21
- Publication Date
- 2025-12-31
AI Technical Summary
Existing infrared image processing techniques struggle to balance the detection of areas of interest, such as hot spots, with the identification of scene details, as global and local histogram stretching methods either enhance contrast at the expense of detail or vice versa, limiting overall performance in surveillance and machine vision applications.
A method involving duplicating an infrared image into two grayscale images, applying local processing for detail enhancement and global processing for hot spot detection, followed by colorization and merging using an image fusion module to create a weighted final image that combines the strengths of both approaches, allowing for effective detection and identification of areas of interest and scene details.
This method enables the creation of an infrared image that maximizes both detection and identification capabilities, facilitating the recognition of areas of interest through colorimetric information while maintaining detailed scene information, thereby improving performance in surveillance and machine vision applications.
Smart Images
Figure FR2024050233_29082024_PF_FP_ABST
Abstract
Description
[0001] DESCRIPTION
[0002] TITLE: Method for processing an infrared image by merging several colorimetric treatments of the infrared image
[0003] Technical field
[0004] The present invention relates to the processing of infrared images captured by a thermal camera.
[0005] In particular, the present invention relates to a processing method comprising a fusion of at least two processing operations of the same infrared image so as to bring out several types of information from the scene captured in the form of an infrared image.
[0006] In general, the invention applies to any type of image capable of providing different types of information depending on the image processing applied.
[0007] Previous techniques
[0008] Existing imaging devices include, for example, a camera comprising a sensor and an optical system, as well as an image processing module for respectively capturing an image of a scene and processing said image by applying a processing function.
[0009] In particular, the image processing module comprises, for example, a module for optimizing the histogram of the captured images. The histogram of an image is a representation of the intensity distribution of the pixels of said image. The histogram optimization module is also called AHC for Automatic Histogram Control.
[0010] The histogram optimization module is intended to be able to analyze and optimize the processing of the image captured by the camera, for example by stretching the histogram globally and / or locally. Generally speaking, stretching a histogram consists, for example, in better distributing the intensity distribution of the pixels of the image, and more generally in giving another form of distribution of the intensities of the pixels. In particular, stretching the histogram globally is carried out by taking into account the value of the intensity of all the pixels of the image while stretching the histogram locally is carried out by taking into account the value of the intensity of pixels on one or more specific areas of the image, for example on areas having a particular spatial frequency.A local stretching of the image histogram is for example carried out by stretching the histogram for each spatial frequency of the image independently of the others, the superposition of these stretched histograms making it possible to obtain the locally processed image.
[0011] One advantage of globally stretching a histogram of an infrared image obtained with an infrared sensor, also called a thermal sensor, is that it makes it easier for a computer or operator to detect areas of interest in the infrared image, for example, areas emitting heat. Indeed, global stretching improves the contrast of the image and allows better detection of areas of interest at the expense of a loss of detail. This advantage is useful for both day and night use in surveillance, defense, and machine vision applications.
[0012] One advantage of locally stretching an infrared image histogram is that it helps identify details in the scene observed by the infrared camera. However, the amount of detail makes the human eye lose its ability to recognize and detect areas of interest, especially hot spots.
[0013] The histogram optimization module also allows you to stretch the histogram of an image in a mixed way, namely by applying a weighting between a global stretch and a local stretch.
[0014] However, choosing a mixed compromise between local stretching and global stretching does not allow obtaining the best performances in each of the two cases, to both detect areas of interest (for example hot spots within the image) and to identify in a fine way details of the scene represented in the image.
[0015] Imaging devices according to the state of the art comprising thermal cameras also comprise a light processing path in addition to the histogram optimization module in the image processing module. This processing path makes it possible to stretch the histogram of the image globally, or not to apply any histogram stretching processing. This processing path is in particular usually used to transmit tracking information or to recover a stream of unprocessed raw images.
[0016] In addition, imaging devices according to the prior art may comprise an image fusion module, allowing imaging devices comprising an infrared sensor and a sensor in the visible wavelengths to combine thermal and visible information on a single image output from the imaging device.
[0017] Statement of the invention
[0018] The present invention therefore aims to overcome the aforementioned drawbacks and to provide a method for processing an infrared image forming an image offering the best possible performance in both detection and identification.
[0019] The present invention relates to a method for processing an infrared image captured by an infrared camera, the method comprising the following steps:
[0020] - a step of duplicating the infrared image into a first grayscale image and a second grayscale image;
[0021] - a step of applying local processing to highlight the details on the first image for each spatial frequency by an image processing module;
[0022] - a step of applying a global processing to the second image by the image processing module highlighting hot spots of interest in the second image;
[0023] - a step of colorizing at least one area of the second image by means of a correspondence table between gray levels and a computer color coding format; - a step of converting the first image from a gray level format to a computer color coding format; and
[0024] - a step of merging the first image with the second image by an image merging module into a final image comprising for each pixel of the final image a weighted merging of the equivalent pixels respectively of the first image and of the area of the second image.
[0025] Thus, using pre-existing modules, in particular a pre-existing image processing module, the present invention makes it possible to form an infrared image combining the information from two different processes to the maximum of their capacity and thus to allow an operator or a computer to easily detect an area of interest, for example via colorimetric information contained in the final image, while being able to identify the details of the scene observed, for example via information from the local processing applied.
[0026] In one embodiment, the method further comprises a step of displaying the final image in real time by a display screen.
[0027] Advantageously, the step of applying local processing comprises stretching the histogram for each spatial frequency of the first image independently of the other spatial frequencies, and comprises superimposing these stretched histograms so as to obtain the first locally processed image.
[0028] Advantageously, the step of applying a global processing to the second image comprises stretching the histogram of the second image by taking into account the intensity value of all the pixels of the second image.
[0029] In a particular embodiment, the method comprises a step of automatic or manual selection of the area of the second image.
[0030] Advantageously, the step of selecting the area of the second image comprises a step of automatic detection of the area as a function of the temperature and / or the movement and / or the shape of said area.
[0031] In particular implementations, the computer color encoding format is an RGB or YCbCr format.
[0032] Advantageously, the coloring and / or blending steps are applied to the entire second image or only to one or more areas of the second image.
[0033] In one embodiment, the steps of applying local processing to the first image and applying global processing to the second image are performed simultaneously.
[0034] The present invention also relates to a computer program comprising program code instructions for executing the steps of the method as defined above, when said program is executed on a computer.
[0035] The present invention also relates to a computer-readable recording medium comprising instructions which, when executed by a computer, cause the latter to implement the steps of the method as defined above.
[0036] The present invention also relates to an imaging device comprising an image processing module configured to implement a method for processing an infrared image captured by an infrared camera, as defined above.
[0037] Brief description of the drawings
[0038] Other aims, characteristics and advantages of the invention will appear on reading the following description, given solely by way of non-limiting example, and made with reference to the appended drawings in which:
[0039] [Fig 1] is a schematic view of an imaging device intended to implement the method according to the invention; and
[0040] [Fig 2] is a schematic illustration of the different steps of the infrared image processing method according to the invention. Detailed description of at least one embodiment
[0041] Figure 1 schematically shows an imaging device 2 intended to implement the method described below.
[0042] The imaging device 2 comprises an infrared camera comprising an optical system and an infrared sensor 4 as well as an image processing module 6 which can be similar to a computer program embedded in the camera. The image processing module 6 comprises on the one hand a module for local optimization of the histogram 8 of images captured by the infrared sensor 4, and on the other hand a lightweight module for global stretching 10 of images captured by the infrared sensor 4.
[0043] The image processing module 6 also comprises a fusion module 12 of an output image from the local histogram optimization module 8 with an output image from the lightweight global stretching module 10.
[0044] In particular, the local histogram optimization module 8 allows stretching of the histogram of an image locally. The lightweight global stretching module 10 allows stretching of the histogram of an image globally.
[0045] The imaging device 2 optionally comprises a display screen 14 of an image. For example, the displayed image is the image obtained at the output of the fusion module 12.
[0046] Figure 2 schematically represents the different steps of a method for processing an infrared image 16 captured by an infrared sensor 4 in one embodiment. The infrared sensor is for example the infrared sensor 4 of the imaging device 2 illustrated in Figure 1 and the method is for example implemented by the imaging device 2 illustrated in Figure 1.
[0047] To implement the processing method, a step 18 of duplicating the infrared image 16 captured by the infrared sensor 4 is first carried out. The infrared image 16 is for example in grayscale. The duplication thus generates a first image 20 and a second image 22, identical to the captured infrared image 16, and in grayscale. In particular, the infrared image 16 represents an indoor or outdoor scene. The duplication step 18 can be carried out more particularly with a duplication module (not shown) included in the imaging device.
[0048] The duplication step 18 can also be carried out by duplicating the infrared image 16 into a number N of images. This makes it possible to subsequently apply a number N of different image processing operations respectively to said N images. Preferably, N is between 2 and 4 duplicated images. The present invention is therefore not limited to the duplication, processing and merging of two processing operations of an image.
[0049] A step 24 is then carried out for applying local processing to highlight the details in the first image 20. The details correspond to the high spatial frequencies of an image. This step 24 is for example implemented with the image processing module 6 by the local histogram optimization module 8.
[0050] In particular, step 24 of applying local processing comprises, for example, stretching the histogram for each spatial frequency of the first image independently of the others, and comprises superimposing these stretched histograms so as to obtain the first locally processed image. In other words, the first image is processed spatial frequency by spatial frequency, the local processing enhances the high spatial frequencies and lowers the low spatial frequencies for different resolutions of the image. This then makes it possible to study the different spatial components of the image. Thus, the intensity value of the pixels for a given spatial frequency is modified and stretched over the entire spectrum of possible intensities.The histogram stretching is performed for each spatial frequency of the first image, and all the virtual sub-images are combined into a first processed image. We obtain a first image with many intensity variations, and therefore a greater number of details. In other words, the local processing will further highlight the spatial information of the first image by enhancing the contours at different scales of said first image.
[0051] A step 26 of applying a global processing to the second image 22 is also carried out. This step 26 is for example implemented with the image processing module 6 and more particularly with the lightweight global stretching module 10. In one embodiment, this step 26 comprises the stretching of the histogram of the second image. In other words, the intensity value of all the pixels of the second image is modified so that the histogram of these values is stretched over all the possible intensity values of the pixels. More precisely, the stretching of the histogram is carried out by applying an intensity gain to each of the pixels so as to completely fill the intensity dynamics of the image, then by carrying out a recalibration of the histogram around the average value of said intensity dynamics of the image.Optionally, it is possible to customize this processing, for example by choosing the number of pixels that must be at a saturation value, by increasing or limiting the gain, in other words the contrast, or by applying an offset to the values of the pixels of the second image, in other words by increasing or decreasing the brightness. Obviously, the overall processing does not modify the contrast ratios between the different light intensities of an image, in particular in order to maintain a "thermal" view of said image.
[0052] Advantageously, the steps 24 and 26 of applying a local processing and applying a global processing are carried out simultaneously. This is made possible by the fact that the first image 20 and the second image 22 are processed independently, for example respectively by the local histogram optimization module 8 on the one hand and by the light global stretching module 10 on the other hand.
[0053] Alternatively, N images are duplicated, for example four images, steps 24 and 26 being applied to two images and different processing steps being applied to the two remaining images to modify them, for example by additional processing bricks. A step 28 of colorization of at least one area of the second processed image 22 is then carried out. This step 28 is carried out by means of a correspondence table between gray levels and a computer color coding format.
[0054] The computer color coding format is for example RGB or YCbCr format.
[0055] Thus, each gray level corresponds to a precise color combination defined in the correspondence table. This correspondence makes it possible to better highlight areas of interest, in particular hot spots in the infrared image. An advantage is that the colorimetric information obtained highlights for the observer the elements of interest in the scene, for example hot spots, while being superimposable with another type of information, such as variations in intensity illustrating the details of the image.
[0056] The coloring step 28 is applied for example to the entirety of the second processed image 22.
[0057] Alternatively, before the colorization step 28, a step 30 of automatic or manual selection of one or more zones of said second image 22, processed or unprocessed, is carried out. The colorization step can then be applied only to one or more selected zones, the unselected zones still being converted into a computer color coding format, without however changing their visual rendering. Thus, in a computer color coding format of the RGB type, namely with three red, green and blue components, the unselected zones retain a grayscale appearance while having their RGB components such that the intensity of the red component is equal, for each pixel of an unselected zone, to the intensity of the green component, as well as to the intensity of the blue component, noted R=G=B.
[0058] The selection is for example manual and is carried out by an operator who selects, for example on a display screen, the areas of interest of the second image 20.
[0059] Optionally, the step of selecting an area of the second image 22 comprises a step 32 of automatic detection of the area as a function of the temperature and / or the movement and / or the shape of said area. The detection is carried out automatically by a computer and / or an on-board electronic card as a function of temperature criteria, and / or movement and / or shape of the area to be selected.
[0060] Thus, detecting a hot spot of interest then allows it to be colored to make it more visible.
[0061] A step 34 of converting the first image 20 from its grayscale format to a computer color coding format is carried out in parallel. This step 34 is similar to the conversion of the unselected areas mentioned above. In preparation for merging the first and second images 20 and 22, the first image 20 is converted into a computer color coding format while maintaining the same visual appearance of grayscale. Thus, in a computer color coding format of the RGB type, the intensity of the red component is equal, for each pixel of the first image, to the intensity of the green component, as well as to the intensity of the blue component, noted R=G=B.
[0062] A step 36 is then carried out for merging the first image 20 modified after step 34 with the second image 22 modified after step 28 into a final image 38, the first and second images being in the same computer color coding format. If necessary, the merging step 36 makes it possible to merge the N modified images. The merging step 36 is for example carried out by the image merging module 12 of the imaging device 2.
[0063] The final image 38 more particularly comprises for each pixel of said final image 38 a weighted fusion of the equivalent pixels respectively of the first image 20 and of the second image 22. Equivalent pixels are understood to mean the pixels of the first and second images which correspond to the same pixel in the captured infrared image.
[0064] According to an example of weighted fusion, by denoting x and y the coordinates of each pixel of the final image 38, a the weighting factor between 0 and 1, R, G, B the intensities of the colors for an RGB color computer coding format, we obtain the RGB intensities for each pixel of the final image 38, such that:
[0065] The weighting factor can be modified by an operator. When it is equal to 0, only the second modified image 22 is visible on the final image 38. When it is equal to 1, only the first modified image 20 is visible on the final image 38.
[0066] Optionally, the weighting factor is different depending on the x and y coordinates, for example for coordinates included in selected areas.
[0067] The step 36 of merging the first modified image 20 with the second modified image 22 into a final image 38 makes it possible to combine the respective interests of local processing and global processing, and therefore to facilitate respectively the identification and the detection on the same final image 38.
[0068] Optionally, a step 40 of displaying the final image 38 is carried out by the display screen 14 of the imaging device 2, the display step 40 preferably being carried out in real time so that the display step 40 is carried out at most one second after the infrared image 16 has been duplicated, for example in a time corresponding to three processing frames, or 60 milliseconds for an acquisition frequency of 50 Hz.
Claims
CLAIMS 1. Method for processing an infrared image (16) captured by an infrared camera, characterized in that it comprises the following steps: a step of duplicating (18) the infrared image (16) into a first image (20) in gray levels and a second image (22) in gray levels; a step of applying a local processing (24) highlighting the details on the first image (20) for each spatial frequency by an image processing module (6); a step of applying a global processing (26) on the second image (22) by the image processing module (6) highlighting hot spots of interest in the second image (22); a step of colorizing (28) at least one area of the second image (22) by means of a correspondence table between gray levels and a computer color coding format; a step of converting (34) the first image (20) from a grayscale format to a color computer encoding format;and a step of merging (36) the first image (20) with the second image (22) by an image merging module (12), the merging resulting in a final image (38) comprising for each pixel of the final image (38) a weighted merging of the equivalent pixels respectively of the first image (20) and of the area of the second image (22).; 2. Method according to claim 1, further comprising a step of displaying (40) in real time the final image (38) by a display screen (14).
3. Method according to any one of claims 1 and 2, in which the step of applying a local processing (24) comprises stretching the histogram for each spatial frequency of the first image (20) independently of the other spatial frequencies, and includes the superposition of these stretched histograms so as to obtain the first locally processed image (20).
4. Method according to any one of claims 1 to 3, in which the step of applying a global processing (26) on the second image (22) comprises stretching the histogram of the second image (22) by taking into account the value of the intensity of all the pixels of the second image (22).
5. Method according to any one of claims 1 to 4, comprising a step of automatic or manual selection (30) of the area of the second image (22).
6. Method according to claim 5, in which the step of selecting (30) the area of the second image (22) comprises a step of automatically detecting (32) the area as a function of the temperature and / or the movement and / or the shape of said area.
7. A method according to any one of claims 1 to 6, wherein the computer color coding format is an RGB or YCbCr format.
8. Method according to any one of claims 1 to 7, in which the coloring (28) and / or merging (36) steps are applied to the entire second image (22) or only to one or more areas of the second image (22).
9. Method according to any one of claims 1 to 8, in which the steps of applying a local processing (24) to the first image (20) and of applying a global processing (26) to the second image (22) are carried out simultaneously.
10. A computer program comprising program code instructions for carrying out the steps of the method according to any one of claims 1 to 9, when said program is executed on a computer.