Method for full-resolution estimation of temperature emissivity by multispectral thermal infrared imaging

Abstract

The invention relates to a method for processing TIR images, which comprises the steps of: generating training data (MDD) from a physical model of a bolometric sensor with a plurality of spectral channels, the physical model providing a radiance measurement according to temperature and emissivity values; training a regression module (TRD) estimating temperature images (TMP) and emissivity images (EMSi) for each spectral channel (i), on the basis of the training data; training a rendering module (MRIR) configured to produce a full-resolution multispectral radiance image; training a processing chain comprising the rendering and regression modules, and a refinement module (PRTR, PSTR); acquiring a radiance image (LM[1-C]) for each spectral channel; and processing the image acquired by the trained processing chain, in order to obtain a temperature image and an emissivity image for each spectral channel.

Description

[0001] DESCRIPTION

[0002] TITLE: Full-resolution method for estimating temperature emissivity using multispectral thermal infrared imaging

[0003] technical field

[0004] The present invention relates to infrared thermal imaging (IRT). This process generates an image of a scene by measuring the infrared radiation emitted by that scene. Since the amount of thermal radiation depends on the temperature, it is possible to observe temperature variations within the captured scene.

[0005] State of the art

[0006] Infrared radiation can be measured in different ways, depending on the type of sensor. For example, a bolometer measures radiant heat using a material whose electrical resistance depends on temperature. More precisely, the incident IR flux causes a change in electrical resistance, from which radiation parameters can be deduced.

[0007] Physically, radiation is determined by initially unknown variables, namely a temperature variable and emissivity variables of the observed object, representing the efficiency with which an object emits thermal radiation. To study IR radiation and utilize this radiation in an application system, it can be useful to precisely determine the temperature and emissivity from the measured radiation. However, the problem of temperature-emissivity separation (TES) is complex, particularly due to the non-linear relationship between temperature and radiance. For the sake of simplification, most existing work assumes that the emissivity is approximately constant in each spectral channel i considered.

[0008] In this hypothesis, for a given temperature T, the radiance L mes (T) measured in spectral channel i can be calculated using the following simplified model:

[0009] Lmes i (T)=E i LcN,i(T)+(1-E i )L C N,i(Te) (1) in which E represents the average emissivity of the object in channel i and T e represents the ambient temperature. The radiance emitted directly by the target L CN ,(T) and the radiance of the environment reflected by the target L CN (T e ) can be calculated using Planck's equation: the spectral response of a filter covering the wavelength band extending from A m in to Amax, defining spectral channel i.

[0010] It turns out that multispectral sensors with M spectral channels Δj=i...M can only provide M TES measurements for M+1 variables (M emissivities ej=i...M and a temperature T). The problem therefore appears ill-posed because the number of equations linking the variables to be determined is less than the number of these variables. Indeed, there is no one-to-one correspondence between T, ej=i...M and L mes,i (T). Therefore, there are an infinite number of predecessors of Àj=i...M. To circumvent this lack of equations, most existing TES algorithms inherently add constraints. Due to their iterative and non-deterministic nature, they are slow and difficult to deploy in end-of-chain devices that can integrate sensors. Furthermore, these algorithms process each pixel of the scene separately, that is, without taking into account information from the surrounding region. Consequently, they cannot guarantee the production of images free from artifacts or aliasing of a visually pleasing quality for the human eye (i.e., with sharp edges and uniformity within the same object).

[0011] Temperature-Emissivity Separation (TES) is a crucial step when using TIR sensors to separate the object's surface temperature from its emissivity. As mentioned earlier, in the context of a multispectral TIR image with M spectral measurement channels, we always end up with an underdetermined system of equations with only M equations for M+1 unknowns (one temperature equation and M emissivity equations over M spectral channels, based on the assumption of constant emissivity per band). Other assumptions could be made here regarding the variation of emissivity with wavelength (other than piecewise or bandwise constant). For example, this variation could be modeled by the linear combination of reference curves (which can therefore be analyzed by a transform).These reference "curves" could be defined either manually or by learning based on a database of typical emissivity of the materials considered.

[0012] Several approaches have been proposed to solve this problem, which can be divided into two groups: dataless iterative algorithms and data-driven learning algorithms. Dataless iterative algorithms introduce an additional constraint (or prior assumption) to eliminate underdetermination. This additional constraint typically applies to emissivity, that is, the range of possible emissivity values ​​given the wavelength range. Some of these algorithms apply the normalized emissivity (NEM) method. In particular, [1] uses an empirical relationship between a range of emissivities and a minimum value for a multichannel observation.Starting with a predetermined emissivity value, these algorithms estimate a relative emissivity from the measured radiation and then calculate a minimum emissivity value using an empirical relationship established from laboratory experiments. This yields an emissivity value for each spectral channel and a temperature value. This process is then repeated iteratively with a new predetermined emissivity value until the resulting values ​​converge.

[0013] Subsequently, new iterative algorithms were developed within the framework of maximum entropy [2]. These involve calculating the probability density distribution (PDF: "Probability Density Distribution") of the random variables, such as temperature and emissivities, in order to derive the estimated values ​​of these variables.

[0014] Data-driven learning algorithms primarily rely on neural networks and a predetermined training dataset to estimate temperature and emissivity from radiance measurements. These algorithms mainly focus on extracting Earth surface temperature using hyperspectral measurements. Aires et al. [3] propose a single-hidden-layer multilayer perceptron (MLP) to extract surface or atmospheric temperature from a large number of radiance measurements. Chen et al. [4] use principal component analysis (PCA) to compress and remove noise from hyperspectral TIR data before feeding it into a single-hidden-layer MLP.More recently, Ye et al. [5] exploit a deep neural network structure comprising a bidirectional recurrent network with long short-term memory (LSTM) combined with multiple 1D convolutional layers and fully connected layers to exploit correlations between hyperspectral TIR channels. Finally, [6] presents a heat-assisted detection and ranging (HADAR) system that allows for the estimation of temperature, emissivity, and a texture image from a hyperspectral thermal radiation cube.

[0015] Recently, infrared (IR) images have received increased attention as an alternative to RGB images for improving the performance of perception algorithms under adverse weather and lighting conditions. Several studies have focused on rendering tasks such as denoising and superresolution to enhance the visual quality of IR images. In particular, Saragadam et al. [7] propose an image processing chain for thermal cameras that combines a physical model of an uncooled bolometric camera with a deep convolutional neural network to address denoising and superresolution issues. This processing chain relies on capturing multiple images of the scene and estimating the scene's radiance flux using a deep regularized neural network.Shyam et al. [8] propose a lightweight super-resolution network to increase the resolution of the thermal image, which is then fused with an RGB image to improve the performance of the object detection algorithm in adverse weather conditions. In this invention, the radiance image rendering module can apply various operations (e.g., denoising, demosaicing, or super-resolution) to the input multispectral TIR images, with the aim of producing correct full-frame radiance images ready for the TES operation. [9] proposes a multi-level feature attention network that is antagonistically trained to denoise infrared images. It is worth mentioning that existing work mainly proposes thermal image rendering networks, either standalone or as a component of a multimodal perception algorithm, to process a single input thermal image.

[0016] There are also blind image quality assessment (BIQA) algorithms designed to automatically evaluate the perceived quality of a single image without any reference, that is, to assess the image quality without comparing it to a known, high-quality reference image. By determining the quality of an image, it is possible to make informed decisions about its use or suitability for specific tasks, or to adjust a processing chain to improve the output image quality. Most BIQA algorithms are initially proposed for RGB images. However, some work focuses on evaluating the quality of thermal images. Agaian et al.

[0010] introduce new measures for evaluating the quality of thermal images based on the human visual system, the information system, and distribution.Teutsch et al.

[0011] propose quality measures covering exposure, contrast, noise visibility, and temporal inconsistency attributes for tone mapping of thermal infrared videos. Erlenbusch et al.

[0012] use an existing BIQA algorithm in a fog-aware density estimator (FADE: "Fog Aware Density Evaluator") to denoise TIR images.

[0017] It is therefore desirable to improve existing multispectral image processing algorithms by developing a method capable of rapidly producing high-quality images (pleasant to human observers) without requiring significant computing power. It is also desirable to avoid being limited to a specific context such as the Earth's surface or the atmosphere, as is the case with some of the methods described previously. Furthermore, it is also desirable to directly produce absolute thermographic images.

[0018] Résumé

[0019] Embodiments relate to a method for processing infrared thermal images (IRT), comprising steps of: generating training data from a physical model of a bolometric sensor comprising several spectral channels, the physical model providing for each spectral channel a radiance measurement as a function of temperature and emissivity values ​​in the spectral channel or as a function of coefficients of a linear combination of several spectral response curves; training a regression module comprising a forward-predictive neural network to estimate a temperature image and an emissivity image for each spectral channel or coefficient, from the training data; training an image rendering module configured to produce a full-resolution multispectral radiance image having appropriate visual quality in a luminance range, from a database of real images;train a processing chain comprising the rendering module, the regression module, and at least one refinement module disposed between the rendering module and the regression module and / or at the output of the regression module, the rendering module and the regression module being frozen or adjusted by regularization during training; acquire by a processor, a radiance image provided by a multispectral bolometric sensor, the radiance image comprising for each pixel a radiance value for each spectral channel or each coefficient;and process the image acquired by the trained processing chain, implemented by the processor, to obtain a temperature image and an emissivity image per spectral channel or per coefficient. According to one embodiment, the regression module includes a neural network for estimating a temperature image which is trained, then used to train a neural network for estimating an emissivity image for each spectral channel or for each coefficient.

[0020] According to one embodiment, the processing chain includes a first refinement module located between the rendering module and the regression module and a second refinement module located at the output of the regression module, the first and second refinement modules being trained at the same time during the training of the processing chain.

[0021] According to one embodiment, the radiance image rendering module is configured to separately process each radiance image for a spectral channel supplied by the sensor and produce a full-resolution radiance image for the spectral channel with appropriate visual quality in the luminance domain.

[0022] According to one embodiment, the refinement module is configured to process each pixel of the radiance images by taking into account neighboring pixels, to remove any artifacts, and improve the visual quality of the temperature / emissivity images.

[0023] According to one embodiment, the image rendering module performs at least one of the following processing steps: demosaicing, denoising, resolution upscaling, and spectral channel realignment.

[0024] According to one embodiment, the training of the processing chain is carried out in a self-supervised manner or without using reference temperature and emissivity images.

[0025] According to one embodiment, the training of the processing chain is carried out using a fidelity or loss of similarity function to constrain the behavior of the refinement module not to remove global structures in the images compared to unrefined images provided by the regression module.

[0026] According to one embodiment, the refinement module is configured to perform smoothing and / or denoising in uniform areas and edge sharpening of temperature and emissivity images provided by the regression module, by constraining the output images of the processing chain to get closer to the output images of the regression module.

[0027] According to one embodiment, the processing chain includes: a panchromatic estimation process producing a panchromatic radiance image from an input or output image of the regression module, and / or a false-color thermal image estimation process receiving the input and / or output image of the regression module, and / or output image of the refinement module.

[0028] According to one embodiment, the process includes a step of merging the temperature image and / or emissivity images with images from an image sensor in the visible domain.

[0029] According to one embodiment, the processing chain includes a perception module receiving images from the refinement module, and which provides an inference decision.

[0030] Embodiments may also relate to a device for generating multispectral temperature and emissivity images, comprising: a multispectral radiance sensor, and a processor configured to implement the steps of the process as defined above.

[0031] According to one embodiment, the sensor is of the infrared thermal bolometer type, TIR, comprising a filter disposed in a focal plane of the sensor and having a mosaic pattern of 2 x 2 filter elements, each filter element of the pattern corresponding respectively to one of the spectral channels.

[0032] Embodiments may also relate to a computer-readable recording medium comprising instructions which, when executed by a computer, cause the computer to carry out the process as defined above.

[0033] Brief description of the figures

[0034] The present invention will be better understood with the aid of the following description of exemplary embodiments with reference to the accompanying figures, in which identical reference numerals correspond to structurally and / or functionally identical or similar elements, and in which:

[0035] [Fig 1] Figure 1 schematically represents a device for acquiring and processing multispectral TIR images, according to one embodiment,

[0036] [Fig 2] Figure 2 schematically represents a module of the acquisition device, producing a temperature estimate from multispectral TIR image data, according to one embodiment,

[0037] [Fig 3] Figure 3 schematically represents a module producing an emissivity estimate from multispectral TIR image data, according to one embodiment, [Fig 4] Figure 4 schematically represents a module of the acquisition device, performing demosaicing of multispectral TIR images, according to one embodiment,

[0038] [Fig 5] Figure 5 schematically represents an image rendering module of the acquisition device, according to one embodiment,

[0039] [Fig 6] Figure 6 schematically represents part of the image editing module, according to one embodiment,

[0040] [Fig 7] Figure 7 schematically represents part of the image editing module, according to one embodiment,

[0041] [Fig 8] Figure 8 schematically represents a refinement module of the acquisition device, according to one embodiment,

[0042] [Fig 9] Figure 9 schematically represents a multispectral TIR image processing chain, according to another embodiment,

[0043] [Fig 10] Figure 10 schematically represents a multispectral TIR image processing chain, according to another embodiment,

[0044] [Fig 11] Figure 11 schematically represents a multispectral TIR image processing chain, according to another embodiment,

[0045] [Fig 12] Figure 12 schematically represents, in perspective, a multispectral radiance image sensor, according to one embodiment,

[0046] [Fig 13] Figure 13 schematically represents the device for acquiring and processing multispectral TIR images, according to one embodiment.

[0047] Detailed description

[0048] Figure 1 depicts a device for acquiring and processing multispectral radiance images, according to one embodiment. The device comprises a PRC processor and a C-channel multispectral radiance SNS image sensor for LM[1-C] images, connected to the processor. The PRC processor implements a processing method based on deep neural networks generated by self-supervised learning, dedicated to rendering full-resolution TM P temperature and EMS emissivity images from the LM[1-C] images. The processing method is configured to estimate both a TM P temperature image and an EMSi emissivity image per spectral channel i, assuming constant emissivity in each spectral channel.The processing method relies on the core concept of deep learning, utilizing various neural networks and learning strategies (supervised and self-supervised) for absolute thermography applications. It also incorporates image quality assessment to produce images that are pleasing to human observers.

[0049] The PRC processor comprises a processing chain with four modules: a low-level radiance image rendering MRIR module, a regression neural network TRG module, and two refinement modules, PRTR and PSTR, which can be placed before or after the TRG regression model. In practice, a processing chain may contain only one refinement module, and the role of the MRIR module is configurable, depending on the nature of the LM[1-C] multispectral data provided by the SNS sensor.

[0050] In one embodiment, the low-level radiance image rendering (MRIR) module is configured to process each LMi radiance image for a spectral channel i provided by the SNS sensor separately and produce a full-resolution radiance image for the spectral channel with appropriate visual quality in the luminance domain. Subsequently, the regression TRG module aims to transition from the RDM domain of radiance measurements to the TEDM domain of temperature and emissivity. In one embodiment, the TRG module implements not an iterative algorithm as in the prior art, but a feed-forward neural network to estimate the temperature and emissivity of a scene in real time. The regression TRG module contains only pixel-level operations, since the temperature and emissivity of each pixel are derived solely from the measured multispectral radiances.Therefore, the output image from the C+1-value-per-pixel regression TRG module may contain artifacts that are not visually pleasing to the human eye. The refinement modules PRTR and PSTR are designed to correct these artifacts before and / or after processing by the TRG module, ensuring that the final temperature and emissivity images have acceptable visual quality. It should be noted that these artifacts are primarily or mostly due to inconsistencies or discrepancies between the physical acquisition model and the one used during the initial supervised model training.

[0051] Furthermore, in prior art iterative TES methods, the temperature and emissivity estimates for each pixel are performed independently, without considering the values ​​of adjacent pixels. Consequently, if the TES algorithm fails at a single pixel, discontinuity artifacts are generated in the output temperature / emissivity images. In contrast, the PRTR and / or PSTR refinement modules process each pixel while taking neighboring pixels into account, thereby removing potential artifacts and improving the visual quality of the temperature / emissivity images.

[0052] Furthermore, prior art iterative TES methods are not well-suited to parallel processing and are time-consuming. In contrast, the proposed processing chain, based on a predictive or direct inference neural network, is perfectly compatible with parallel processing and therefore suitable for real-time deployment. If each module is designed compactly, the entire processing chain can be implemented in peripheral devices to perform a first real-time estimation of the temperature / emissivity of a scene. It can also enable high-quality scene display, i.e., presenting panchromatic or false-color images, providing clearer scene information.

[0053] The proposed processing chain also enables various rendering tasks (e.g., demosaicing, denoising, super-resolution), while maintaining knowledge of the visual quality of the output temperature / emissivity images. It can therefore be used in different image capture contexts, such as an IR camera with a repeating pattern of mosaicked multispectral filters applied to the camera's pixel array, or a low-spatial-resolution multispectral camera.

[0054] It is also worth noting that collecting a dataset of images for supervised learning using temperature / emissivity ground truth images of different scenes is expensive and practically impractical. In contrast, the proposed processing chain enables absolute thermography from thermal radiation measurements in the absence of such a dataset. This can be achieved using a semi-supervised learning approach, which is both supervised (using the physical model and an image database for demosaicing) and self-supervised (using data from the sensor itself).

[0055] The proposed processing chain also offers direct output of temperature and emissivity via spectral channel. It can therefore be combined with a decision-making module located at the output of the processing chain to perform real-time applications such as monitoring or anomaly detection.

[0056] According to one embodiment, the processing chain learning procedure comprises three learning steps, ST1, ST2, and ST3. The first learning step, ST1, is performed only for the regression TRG module. Temperature and emissivity are estimated using a physical model of the SNS sensor. This physical model can be that defined by equations (1) and (2). For this purpose, a training dataset (MDD) is generated in the form of a table. Alternatively, the physical model is incorporated into a data augmentation scheme to generate the training data incrementally. Each sample of the generated data includes, as input, measured radiance values ​​(C), and as output, a temperature value (TMP) and emissivity values ​​(EMS[1-C]).The TMP temperature and EMSi emissivity for each spectral channel i can be randomly sampled within a given range and then used to calculate the C radiance measurements using the physical model defined by equations (1) and (2) and integrating the spectral response of each filter from a set of C filters. The regression neural network model of the TRG module receives as input the LM[1-C] radiations extracted from the MDD training data and learns to determine the TMP temperature and EMS[1-C] emissivities by minimizing an RLS quality loss function (e.g., the mean squared error function or L2 distance, or the mean absolute error function MAE or L1 distance).

[0057] The second learning stage, ST2, concerns only the MRIR module for radiance image rendering. This stage is configurable, as it depends on the nature of the multispectral data provided by the SNS sensor and therefore on the processing performed by the MRIR module. In one example, the MRIR module performs demosaicing of the received LM[1-C] multispectral images when these images are matrixed in a manner similar to the matrixing observed in images provided by cameras in the visible range. The LM[1-C] multispectral images are thus provided by a TIR-type SNS sensor (such as a microbolometer) with an array of different filters integrated in the focal plane, allowing a spectral channel to be associated with each pixel of the sensor. In another example, the MRIR module performs denoising to eliminate noise from the received images.In cases where the input multispectral images are small, the MRIR module can include a super-resolution array to increase image resolution. The MRIR module can also perform channel realignment processing to address spatial shifts caused by using a camera with a filter wheel to sequentially capture multispectral images. Furthermore, if radiance is measured by C filters, the MRIR module includes simultaneous reconstruction of C full-frame images. The MRIR module can be trained in a supervised manner with available ground truth annotations. From a set of high-resolution, high-quality images, a transformation (e.g., matrixing, noise reduction, or downsampling) is applied to these images to obtain images corresponding to multispectral data provided by the SNS sensor.The MRI R module learns to reverse this transformation to recover the original IMD images by minimizing a loss of fidelity FLS function (e.g., of type L1 or L2).

[0058] The third learning stage, ST3, involves the PRTR and / or PSTR refinement modules, depending on the processing chain configuration, using either one or both of these modules. The IMD training data used in ST3 can be IMD images from the SNS sensor, combined with QLS visual quality data. Following the training of the TRG and MRIR modules, these modules are locked (i.e., their parameters are fixed). One or both of the PRTR and PSTR refinement modules, which also include neural networks, are incorporated into the MRIR and TRG modules to form the complete processing chain that undergoes the learning process.Unlike the previous steps, a self-supervised learning strategy is implemented to optimize both the PRTR and PSTR refinement modules, as reference temperature and emissivity images are unavailable. The learning objective of this third step is to improve the visual quality (QLS) of the TMP temperature and EMS[1-C] emissivity images provided by the TRG module (e.g., improving smoothness in uniform areas and edge sharpness), while constraining the refined output images to be close to the output of the TRG module without the PRTR and PSTR modules. The visual quality (QLS) is determined by a regularization term during the learning phase at the output of the PSTR module (or, if absent, at the output of the TRG module), in order to guide the behavior of the PRTR and / or PSTR modules towards producing pleasing temperature / emissivity images (with improved visual quality).

[0059] For the sake of simplicity and clarity, the temperature and emissivity estimates are separated and performed using two distinct models. Figure 2 illustrates an embodiment of the TRG module's temperature regression TMPM neural network in a case involving a C-channel spectral TES model. The temperature regression neural network comprises several branches BR1-BR5 with fully connected (or "dense") layers DRLn and DRLm of n and m units, respectively, including a Rectified Linear Unit (ReLU) activation function to set all negative values ​​to zero. In one embodiment, n and m are equal to 16 when C equals 4. At the beginning of the MRIR module, the minimum (MN), maximum (MX), and average (MY) values ​​of the C-wave LMi radiation measurements are calculated.These values ​​are concatenated with the radiation measurements using a CT concatenation operation to obtain a C+3 component VC vector. The VC vector is supplied to three branches, BR1, BR2, and BR3, each comprising an n-unit DRLn dense layer and an m-unit DRLm dense layer. In branches BR1 and BR3, the m-unit DRLm dense layers are in series with the n-unit DRLn dense layers. Branch BR2 includes a multiplication operator, M1, which receives the output of the BR2 branch's DRLn dense layer and the output of the BR1 branch's DRLm dense layer. The output of operator M1 is supplied to the input of the BR2 branch's DRLm dense layer, as well as to the input of the BR4 branch's DRLn dense layer. The BR4 branch's DRLn dense layer is followed by an m-unit sigmoid DSGm dense layer.The DSGm dense layer applies the sigmoid activation function (sigmoid(x) = 1 / (1 + exp(-x)), which returns a value very close to zero for values ​​less than -5 and close to one for values ​​greater than 5. The output of the DRLm dense layer of branch BR3 is fed to an X1 incrementing operator of one. The output of the DRLm dense layer of branch BR2 is divided by the output of the X1 operator using a D1 division operator. The output of the D1 operator is fed to the DRLn dense layer of branch BR5. The DRLn dense layer of branch BR5 is followed by a DRLm dense layer. The output of the DSGm layer of branch BR4 and the output of the DRLm dense layer of branch BR5 are fed to the input of an A1 addition operator. The output of the A1 operator is fed to the input of a linear DL1 dense layer of one. which provides a temperature image TM P.The loss function to be minimized during the training of the neural network in Figure 2 can be the mean absolute error function (L1).

[0060] Figure 3 shows an embodiment of the TRG module's emissivity regression EMSM neural network, also in the case implementing a C-channel spectral TES model. This neural network comprises two branches, BR11 and BR12. Each branch BR11 and BR12 includes two DRLnl dense layers in series, with n1 and n2 units respectively, each DRLnl, DRLn2 dense layer including a ReLU-type activation function. Branch BR12 includes a TM PM temperature regression model as a layer, in series with one DRLnl layer and one DRLn2 layer. The output of the DRLn2 layer of branch BR12 is fed to an X2 incrementing operator of 1, whose output forms the output of branch BR12.The output of the DRLn2 layer from branch BR11 and the output of branch BR12 are combined using a D2 splitting operator before being converted into C-dimensional EMS emissivity by a dense C-unit DCRL layer. This DCRL layer includes a clipped ReLU activation function to maintain emissivity values ​​within the appropriate range (typically between 0.1 and 1.0). The dense DCRL layer provides the C-component EMS[1-C] emissivity image. The mean absolute error function L1 can be used as the loss-of-fidelity function for the emissivity model.

[0061] The TRG module learning procedure (ST1 step) comprises two steps, namely: firstly, the training of the temperature neural network, then the training of the emissivity neural network with the TMPM temperature regression model, initialized from the trained temperature neural network.

[0062] As previously mentioned, one possible scenario involves measuring multispectral radiance using a TIR bolometer-type SNS sensor with a filter in its focal plane featuring a 2 x 2 mosaic pattern, each element covering one sensor pixel. Thus, if the images provided by the SNS sensor consist of H x W pixels, the pixels for a spectral channel are spaced one pixel apart in both directions of the image. The images for each of the spectral channels i therefore consist of H / 2 x W / 2 pixels. In this case, the MRIR module performs a demosaicing operation to provide full-frame multispectral images of size H x W for each channel i.Due to the complete absence of strong physical assumptions (particularly regarding inter-channel correlation) and the spatial misalignment between the available TIR filters, demosaicing can only be performed independently of the channels; that is, each spectral channel is reconstructed without using information from the other channels. This problem is equivalent to that of image retouching, in which three missing pixel values ​​must be determined for each 2 x 2 pixel window in each of the LM radiance images provided by the SNS sensor.

[0063] Figure 4 shows a CSDM demosaicing network, according to one embodiment. This demosaicing network comprises C (= 4) processing channels to reconstruct a radiance image in each of the spectral channels independently, from a matrixed TIR image. The input image (from the sensor), of size H x W, is represented in Figure 4 by a 2 x 2 pixel IWI image window containing the pixels corresponding to the four spectral channels. The input image from the MRIR module is duplicated into four H x W images, each containing only the pixels relating to a respective channel of the input image. These four matrixed images are each represented by a 2 x 2 pixel window DW1, DW2, DW3, DW4, each containing one respective pixel from the IWI window, the other three pixels being undefined.Each image, symbolized by one of the DW1-DW4 windows, is processed separately by the CSDM network, which implements a unique demosaicing model to determine the indeterminate pixels of each of the input images. The four resulting images, corresponding to the four channels and symbolized by the spectral windows RW1, RW2, RW3, and RW4, are then processed by a CCT concatenation operator to obtain a complete DTC frame of size H x W x C pixels of the multispectral radiation.

[0064] Figure 5 shows in more detail how the raster image of each channel i can be super-resolved in a coloring-like manner. The raster image, symbolized in Figure 5 by a 4 x 4 pixel DDWI window, is processed by a DI PF deep retouching module to obtain a full-frame intermediate image, symbolized by a 4 x 4 pixel 11M window, in which all pixels are determined. The intermediate image is then combined by a multiplication operator M3 with a binary mask image BIM, and then combined by an addition operator A3 with the raster image in such a way as to preserve the determined pixels of the input raster image (pixels labeled 11, 12, 13, 14 in Figure 5). Thus, the determined pixels of the DDWI matrix image are assigned to the same positions in the intermediate image I IM while retaining the other pixels of the image 11 M to obtain a reconstructed channel image symbolized by the RIM window.

[0065] An example of the DIPF module architecture is shown in Figure 6. The DIPF module comprises a fully convolutional neural network. To mitigate boundary artifacts, before each convolution, one pixel on each spatial side of the data is mirrored. Then, a valid convolution is performed to maintain the data dimensionality. The model's expressiveness can be enhanced by utilizing residual connections. At the end of the model, a local rearrangement of the channel pixel data is used to double the horizontal and vertical spatial dimensions and obtain a full-frame image (i.e., at the sensor's spatial resolution). For this purpose, the DIPF function includes an SSMP subsampling module, an LKRn layer receiving the image output from the SSMP module, and three serial RBLK modules receiving the output from the LKRn layer.The SSMP module receives the image symbolized by the DDWI window, corresponding to spectral channel i, and removes the undetermined pixels from the image to obtain an image symbolized by a 2 x 2 pixel W2 window containing only the determined pixels from the DDWI image. The DIPF module also includes an A4 addition operator combining the output of the LKRn layer with the output of the series of three RBLK modules. The output of the addition operator is provided to the input of another LKRp layer in series with a DPSP data reorganization module and an RPRp layer. The DPSP module reorganizes the pixel data distributed in H x W x D arrays using a 2 x 2 pixel window, merging the data planes along the depth D into D / 4 planes of size 2H x 2W.

[0066] The LKRn module applies a one-pixel image upscaling function, followed by a valid 3x3 convolution with n units, and finally, a "Leaky ReLU" activation function that is transparent for positive values ​​and attenuates negative values ​​with a linear function. The one-pixel upscaling function of the LKRn module creates a one-pixel-wide reflected frame around a window, for example, 4x4 pixels. Thus, the left edge of the upscaling window receives the pixels from the second column of the original window, the right edge of the upscaling window receives the pixels from the second-to-last column of the original window, the top edge of the upscaling window receives the pixels from the second row of the original window, and the bottom edge of the upscaling window receives the pixels from the second-to-last row of the original window.Furthermore, the pixel in the upper-left corner of the enlarged window receives the pixel in the second row and second column of the original window; the pixel in the upper-right corner receives the pixel in the second row and second-to-last column of the original window; the pixel in the lower-left corner of the enlarged window receives the pixel in the second-to-last row and second column of the original window; and the pixel in the lower-right corner receives the pixel in the second-to-last row and second-to-last column of the original window. The RPRp layers apply a one-pixel image upscaling function, a valid 3x3 convolution with p units, and a ReLU-type activation function. In one example implementation, n = 16 and p = 64.

[0067] Figure 7 shows an example implementation of the RBLK module. The RBLK module comprises an LKRn layer in series with an LKRq layer, the input of the LKRn module being added to the output of the LKRq module by an addition operator A5. The output of the A5 operator constitutes the output of the RBLK module. In one example implementation, q = 8.

[0068] In one embodiment, the MRIR module is trained using a set of single-channel infrared images corresponding to the IMD image set in Figure 1. To this end, small 128 x 128 pixels are randomly extracted from each image in the set. Then, one pixel from each 2 x 2 window in each thumbnail is extracted to generate the DDWI rasterized image as input to the MRIR module. The loss of fidelity function used can be the mean absolute error (L1) function.

[0069] Figure 8 illustrates the architecture of the PRTR refinement module, according to an example embodiment. In this example embodiment, the PRTR module comprises a feature branch BR21 for improving the quality of the DTC image, and an attention branch BR22, with branches BR21 and BR22 receiving the DTC image. Branch BR22 adaptively combines the enhanced image with the DTC image to ensure that the output images preserve the original overall structure. Branch BR21 includes an LKRn layer in series with three RBLK modules (Figure 7) in series. The output of the LKRn module is combined by an A6 addition operator with the output of the third RBLK module in the RBLK series. The output of the A6 addition operator is fed to an RPRr layer with r units, whose output forms the output of branch BR21. The output of the RPRr layer is fed to an M6 multiplication operator.

[0070] Branch BR21 includes an LKRn layer in series with an RPSG layer. The RPSG layer applies a valid 3 x 3 convolution to r units with a sigmoid activation function. The output of the RPSG layer is fed to the input of an X6 operator applying f(x) = 1 - x and an M7 multiplication operator. The output of the X6 operator is fed to the M6 ​​multiplication operator. An A7 addition operator combines the output of the M6 ​​operator with the output of the M7 operator and provides the enhanced DTCC image. The DTCC image has a size of H x W pixels with B channels, where B is not necessarily equal to the initial number C of spectral channels. In one embodiment, as described in Figure 8, r = 4 and B = C.

[0071] The PRTR module can be trained with images acquired from a matrix filter bolometric sensor with C filters corresponding to C spectral channels, used to train the TES regression TRG module.

[0072] As mentioned previously, the processing chain can contain both the PRTR and PSTR refinement modules, or only one of them. In an embodiment illustrated by Figure 9, only one PRTR refinement module is implemented. In this embodiment, the processing chain comprises a TNW processing chain without refinement, designated as the "teacher network," and a complete SNW processing chain with a PRTR refinement module, designated as the "student network." The TNW and SNW networks share the same MRIR demosaicing and TRG TES regression modules trained in the preceding steps. During the training of the SNW student network, the parameters of the MRIR and TRG modules can be fixed or adjusted by regularization to maintain the outputs of these modules close to their values ​​prior to this adjustment.The SNW and TNW arrays take as input the IWI matrix image captured by the camera, then reconstruct four full-frame radiation images RW1-RW4 using the MRIR module (performing demosaicing). These arrays produce C+1 full-frame images including a temperature image TM P and C emissivity images EMS.

[0073] According to one embodiment, the PSTR refinement module has an architecture identical or similar to that shown in Figure 8.

[0074] The main objective of the PRTR and PSTR refinement modules is to improve the quality of the TMP thermometric image, for example, by sharpening edges while ensuring a uniform intensity level within a region originating from the same object in the scene. To this end, Total Variation (TV) regularization is applied to the output Ys images of the SNW student array (LTV = |VYs|). This LTV regularization term is used as a quality metric to drive the optimization of the PRTR and / or PSTR refinement module. Furthermore, processing can be implemented to ensure that the refinement module does not alter the overall detail of the output image; that is, the Ys output remains close to the Yt output of the TNW teacher array.Therefore, a loss of similarity function (e.g., L1) can also be used to constrain the behavior of the PRTR and / or PSTR module to not remove global image structures compared to unrefined images. This similarity function can be focused on specific regions and images chosen either by an operator or automatically by an algorithmic process. According to an example embodiment in which the MRIR and TRG modules are fixed, the objective learning function of the PRTR and / or PSTR module is as follows:

[0075] L = a L TV + (1 - a) L sim (3) in which a = 9.5*10 -7 is a hyperparameter controlling the effect of the PRTR refinement module, and L Sjmis a similarity or fidelity cost function between the output and input images of the refinement module. If the MRIR and TRG modules need to be adjusted, additional similarity losses can be applied between the outputs of these modules of the teacher TNW and student SNW networks. According to another embodiment, the distance (e.g., L2) between the adjusted parameters of the student SNW network and the original parameters (of the teacher TNW network) is regularized.

[0076] Figure 12 shows one embodiment of a multispectral radiance imaging SNS sensor. The SNS sensor comprises an SNA array of radiance RSE detector elements, for example, of the microbolometer type, surmounted by an FTA array of spectral filter FTE elements arranged in the focal plane of the SNA array. Some FTE elements are not shown in Figure 12 for clarity, to better illustrate the SNA array and its radiance detection RSE elements. In the example shown in Figure 12, the FTA array comprises four different types of FTE filter elements arranged in a mosaic pattern of four filter elements, corresponding to the IWI window in Figure 4, to obtain radiance images in four spectral channels. The respective bandwidths of the four types of FTE filter elements may partially overlap or be completely separate.CSR elements are sensitive to infrared radiation, for example in the wavelength band extending between 7.5 pm and 14 pm.

[0077] Figure 13 illustrates components of a PRD image acquisition and processing device. The PRD device comprises at least the PRC processor, and functionally coupled to the processor, the SNS sensor, MEM memories, and NIT circuits for wired and / or wireless transmission or communication. The MEM memories store an operating system, and in particular, an application dedicated to processing images from the SNS sensor. To this end, the application implements the processing chain described previously, including the MRIR, TRD, and PRTR and / or PSTR modules. The NIT communication circuits allow the PRC processor to be functionally coupled to an NT communication network such as the Internet, notably to transmit the images output from the SNS sensor and / or the images provided by the dedicated application.

[0078] It will be readily apparent to those skilled in the art that the present invention is susceptible to various embodiments and applications. In particular, the invention is not limited to the module architectures described above. Thus, the main function of the MRIR module is to provide full-resolution multispectral radiance images usable in an absolute thermography process. Depending on the nature of the camera, the MRIR module can perform various functions such as demosaicing, denoising, super-resolution, and realignment to convert the raw sensor measurements into suitable full-resolution multispectral images. These operations can be performed channel by channel (i.e., without information exchange between spectral channels) or by correlating the channels with each other (i.e., similarly to RGB images).

[0079] The main function of the TES regression TRG module is to provide an estimate of the TMP temperature and EMS emissivity for each pixel of the input image. The emissivity can be estimated based on the assumption of a constant value in each spectral channel, as previously described.

[0080] According to one embodiment, the emissivity is estimated based on the more general assumption that it varies with wavelength and that it can be decomposed into a linear combination of K basis functions gj, in the following way:

[0081] E(À) = Zj K =ia jgj (To) (4)

[0082] Under this assumption, the radiance measured for each spectral channel i extending between wavelengths Amin and Amax can be calculated using the following equation:

[0083] Lmes,i(T)= J À Àmax [e(A)R BB (YOUR) + (1- e (A))RBB (T e ,A))] fi (A)dA (5) where e(A) is the emissivity of the target object at wavelength X, fi(A) is the spectral response of a filter covering the band of wavelengths extending between the wavelengths Amin → Amax, and RBB is the spectral radiance density emitted directly by a black body at temperature T. The spectral radiance density RBB can be calculated using the following Planck equation:

[0084] Equations (4), (5), (6) are used as a physical sensor model to generate the MDD training dataset. Instead of providing an emissivity image EMSi per spectral channel i, the method provides an image of the K coefficients Oj of the spectral emissivity response curve.

[0085] In another embodiment, the TES module is configured to also provide a confidence level for each predicted value, or even a range of possible values ​​for each estimate. With this information, the refinement module (particularly the PSTR module) can refine the image more easily to produce pleasing thermographic images.

[0086] As previously described, the learning procedure comprises three steps that can be performed simultaneously, without necessarily requiring a sequencing of learning from one step to the next. In the case of training the MRIR and TRG modules, other supervised fidelity loss functions can be used, such as the mean absolute error (L1), the mean squared error (L2), etc. In particular, quality regularization can be introduced during the MRIR module training. In the case of the refinement modules, in addition to the total variation (TV), existing BIQA measurements can be considered as an alternative regularization of image quality. Furthermore, the similarity loss can be any supervised fidelity loss. The MRI R and TRG modules can be frozen or adjusted by regularization during the training of the PRTR and PSTR refinement modules.Alternatively, adaptive loss with TV regularization can be applied only to regions around edges, and similarity loss can be applied only to uniform areas. An adversarial framework can also be used for training the PRTR and PSTR refinement modules. This training can involve using a discriminator in parallel with the proposed processing chain (considered the generator in the domain semantics). With a database of real, high-quality temperature / emissivity images available, the processing chain creates images, while the discriminator evaluates the quality of these images (classifying them as real and of high visual quality). The two networks can train in a loop, allowing for the improvement of their respective performance.

[0087] Furthermore, as previously described, the processing chain comprises three elements: the MRIR module, the TES regression TRG module, and the refinement processing. The refinement processing can be performed by two refinement modules, PRTR and PSTR, or by a single module, to improve the visual quality of the TMP thermal image and EMS emissivity images. However, other modules dedicated to specific applications can be integrated. For example, additional processing can be implemented to estimate a panchromatic radiance image or a false-color thermal image from multispectral images.Thus, Figure 10 represents a processing chain that differs from that of Figure 1 in that it includes a panchromatic estimation PCE module producing a panchromatic radiance PIM image, and a false-color FCIM thermal estimation FCE module receiving the pre-processed image output from the PRTR module. According to another embodiment, the panchromatic estimation PCE module is located at the output of the TRG or PSTR module to provide false-color temperature and / or emissivity images.

[0088] Each of the previously described MRIR, TRG, PRTR, and PSTR modules can adopt several types of neural network architectures. In the case of the TES regression TRG module, only per-pixel calculations are considered, as the temperature / emissivity estimation at each spatial position is performed independently. Consequently, the model used can be built solely on 1 x 1 convolutions, i.e., fully connected layers. For the MRIR, PRTR, and PSTR modules, a convolutional neural network can be implemented with a different kernel size, larger than 1 x 1.In addition, if the multispectral measurements are in the form of a video (i.e., in the form of a continuous acquisition of images as a function of time), other model structures can be used for the MRIR, PRTR and PSTR modules, to obtain spatiotemporal features, such as 3D convolutions, or recurrent neural networks (RNNs).

[0089] The present invention paves the way for estimating TMP temperature and EMS emissivity images of a scene, using a neural network architecture suitable for real-time deployment on a wide variety of hardware platforms. It provides a unified framework for estimating the temperature and emissivity of each pixel in a scene while taking into account the visual quality of the resulting temperature and emissivity images. It offers a model learning strategy compatible with the current lack of suitable training databases, thus enabling fully supervised learning. Furthermore, the processing chain is versatile, being independent of the acquisition method for the multispectral radiance measurements.

[0090] Furthermore, the use of TIR images in addition to visible RGB images is now a popular topic, as it offers more comprehensive perception regardless of lighting or weather conditions. The resulting TMP temperature and EMS emissivity images can be fused with images from a visible-domain image sensor to improve the algorithmic performance of the inference task, particularly when the visible image is of poor quality due to adverse weather or lighting conditions. Beyond the direct use of the previously described processing chain to obtain a real-time absolute thermographic image of a scene, other applications can be considered in the fields of surveillance, security, or anomaly detection, based on temperature and emissivity images.To this end, it is possible to add in series to the previously described processing chain, a suitable inference module (i.e. allowing the understanding of the scene, and whose output can typically correspond to a pixel segmentation of the image content), with the possibility of training the latter separately or jointly with the PRTR, PSTR refinement modules.

[0091] It may also be possible to integrate other inference applications derived from the pre / post-TES regression model (Figure 11). Thus, Figure 11 represents a processing chain that differs from that of Figure 1 in that it includes a PMD perception module that can receive images from the PRTR or PSTR module and provides an INFD inference decision. For example, the PMD module could be a neural network module configured to detect areas of anomalies, estimate materials in the scene by pixel, or detect a specific event of interest.

[0092] References Cited

[0093] [1] A. Gillespie, S. Rokugawa, T. Matsunaga, JS Cothern, S. Hook, and AB Kahle, “A temperature and emissivity separation algorithm for Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) images,” IEEE Trans. Geosci. Remote Sens., vol. 36, no. 4, p. 1113-1126, July 1998, doi: 10.1109 / 36.700995.

[0094] [2] A. Barducci, D. Guzzi, C. Lastri, P. Marcoionni, V. Nardino, et I. Pippi, « Emissivity and Temperature Assessment Using a Maximum Entropy Estimator: Structure and Performance of the MaxEnTES Algorithm », IEEE Trans. Geosci. Remote Sens., vol. 53, n° 2, p. 738-751 , févr. 2015, doi: 10.1109 / TGRS.2014.2327218.

[0095] [3] F. Aires, A. Chédin, N. A. Scott, et W. B. Rossow, « A Regularized Neural Net Approach for Retrieval of Atmospheric and Surface Temperatures with the IASI Instrument », J. Appl. Meteorol. Climatol., vol. 41 , n° 2, p. 144-159, févr. 2002, doi: 10.1175 / 1520-0450(2002)041 <0144:ARNNAF>2.0.CO;2.

[0096] [4] M. Chen, L. Ni, X. Jiang, Z. Li, et H. Wu, « Retrieval of Atmospheric and Land Surface Parameters from Satellite-Based Thermal Infrared Hyperspectral Data Using an Artificial Neural Network Technique », in IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium, Valencia: IEEE, juill. 2018, p. 2745-2748. doi: 10.1109 / IGARSS.2018.8518131.

[0097] [5] X. Ye et al., « Simultaneous Estimation of Land Surface and Atmospheric Parameters From Thermal Hyperspectral Data Using a LSTM-CNN Combined Deep Neural Network », IEEE Geosci. Remote Sens. Lett., vol. 19, p. 1-5, 2022, doi: 10.1109 / LGRS.2021 .3104501 .

[0098] [6] F. Bao et al., « Heat-assisted detection and ranging », Nature, vol. 619, n° 7971 , Art. n° 7971 , juill. 2023, doi: 10.1038 / s41586-023-06174-6.

[0099] [7] V. Saragadam, A. Dave, A. Veeraraghavan, et R. G. Baraniuk, « Thermal Image Processing via Physics-Inspired Deep Networks », in 2021 IEEE / CVF International Conference on Computer Vision Workshops (ICCVW), Montreal, BC, Canada: IEEE, oct. 2021 , p. 4040-4048. doi: 10.1109 / ICCVW54120.2021.00451.

[0100] [8] P. Shyam et H. Yoo, « Lightweight Thermal Super-Resolution and Object Detection for Robust Perception in Adverse Weather Conditions ».

[0101] [9] P. Yang, H. Wu, L. Cheng, et S. Luo, « Infrared image denoising via adversarial learning with multi-level feature attention network », Infrared Phys. Technol., vol. 128, p. 104527, janv. 2023, doi: 10.1016 / j. infra red.2022.104527.

[0102]

[0010] S. Agaian, M. Roopaei, et D. Akopian, « Thermal-image quality measurements », in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy: IEEE, mai 2014, p. 2779-2783. doi: 10.1109 / ICASSP.2014.6854106.

[0103]

[0011] M. Teutsch, S. Sedelmaier, S. Moosbauer, G. Eilertsen, et T. Walter, « An Evaluation of Objective Image Quality Assessment for Thermal Infrared Video Tone Mapping », in 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition

[0104] Workshops (CVPRW), Seattle, WA, USA: IEEE, juin 2020, p. 488-497. doi: 10.1109 / CVPRW50498.2020.00062.

[0105]

[0012] F. Erlenbusch et al., « Thermal Infrared Single Image Dehazing and Blind Image Quality Assessment », in 2023 IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Vancouver, BC, Canada: IEEE, juin 2023, p.

[0106] 459-469. doi: 10.1109 / CVPRW59228.2023.00052.

Claims

26 DEMANDS 1. A method for processing infrared thermal images (IRT), comprising steps of: generating training data (MDD) from a physical model of a bolometric sensor comprising several spectral channels, the physical model providing for each spectral channel a radiance measurement as a function of temperature and emissivity values ​​in the spectral channel or as a function of coefficients of a linear combination of several spectral response curves; training a regression module (TRD) comprising a forward-predicting neural network (TMPM) to estimate a temperature image (TMP) and an emissivity image (EMSi) for each spectral channel (i) or coefficient, from the training data; training an image rendering module (MRIR) configured to produce a full-resolution multispectral radiance image having appropriate visual quality in a luminance domain, from a database (IMD) of real images;train a processing chain comprising the rendering module, the regression module, and at least one refinement module (PRTR, PSTR) disposed between the rendering module and the regression module and / or at the output of the regression module, the rendering module and the regression module being fixed or adjusted by regularization during training; acquire by a processor (PRC), a radiance image (LM[1-C]) provided by a multispectral bolometric sensor (SNS), the radiance image comprising for each pixel a radiance value for each spectral channel or each coefficient; and process the image acquired by the trained processing chain, implemented by the processor, to obtain a temperature image and an emissivity image per spectral channel or per coefficient.

2. A method according to claim 1, wherein the regression module (TRG) comprises a temperature image estimation (TMP) neural network which is trained and then used to train an emissivity image estimation (EMSi) neural network for each spectral channel or for each coefficient.

3. A method according to claim 1 or 2, wherein the processing chain comprises a first refinement module (PRTR) disposed between the rendering module (MRIR) and the regression module (TRG) and a second refinement module (PSTR) disposed at the output of the regression module, the first and second modules of refinement being trained at the same time during the training of the processing chain.

4. A method according to any one of claims 1 to 3, wherein the radiance image rendering module (MRIR) (LM[1-C]) is configured to separately process each radiance image (LMi) for a spectral channel (i) supplied by the sensor (SNS) and produce a full-resolution radiance image for the spectral channel with appropriate visual quality in the luminance domain.

5. A method according to any one of claims 1 to 4, wherein the refinement module (PRTR, PSTR) is configured to process each pixel of the radiance images (LM[1-C]) taking into account neighboring pixels, to remove possible artifacts, and improve the visual quality of the temperature / emissivity images.

6. A method according to any one of claims 1 to 5, wherein the image rendering module (MRIR) performs at least one of the following processing: a demosaicing process, a denoising process, a resolution upscaling process, and a spectral channel realignment process.

7. A method according to any one of claims 1 to 6, wherein the processing chain is driven in a self-supervised manner or without using reference temperature and emissivity images.

8. A method according to any one of claims 1 to 7, wherein the training of the processing chain is carried out using a fidelity or loss of similarity function to constrain the behavior of the refinement module (PRTR, PSTR) not to remove global structures in the images compared to unrefined images provided by the regression module (TRG).

9. A method according to any one of claims 1 to 8, wherein the refinement module (PRTR, PSTR) is configured to perform smoothing and / or denoising processing in uniform areas and edge sharpening of temperature (TMP) and emissivity (EMS[1-C]) images provided by the regression module (TRG), by forcing the output images of the processing chain to approach the output images of the regression module.

10. A method according to any one of claims 1 to 9, wherein the processing chain comprises: a panchromatic estimation processing (PCE) producing a panchromatic radiance image (PIM) from an input or output image of the regression module (TRG), and / or a false-color thermal image estimation (FCIM) processing receiving the input and / or output image of the regression module (TRG), and / or output image of the refinement module (PSTR).

11. A method according to any one of claims 1 to 10, comprising a step of fusing the temperature image (TMP) and / or emissivity images (EMS[1-C]) with images from an image sensor in the visible range.

12. A method according to any one of claims 1 to 11, wherein the processing chain comprises a perception module (PMD) receiving images from the refinement module (PRTR, PSTR), and which provides an inference decision (INFD).

13. Device for generating multispectral temperature and emissivity images, comprising: a multispectral radiance sensor (SNS), and a processor (PRC) configured to implement the steps of the process according to any one of claims 1 to 12.

14. Device according to claim 13, wherein the sensor (SNS) is of the infrared thermal bolometer type, TIR, comprising a filter disposed in a focal plane of the sensor and having a mosaic pattern of 2 x 2 filter elements, each filter element of the pattern corresponding respectively to one of the spectral channels.

15. Computer-readable recording medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to any one of claims 1 to 12.

Images