A method for fusion of visible light image and infrared image based on radiation characteristic inversion

Abstract

A visible light image and infrared image fusion method based on radiation characteristic inversion belongs to the field of optical image processing technology, and specifically includes the following steps: Step 1: Utilize multi-spectral visible light image inversion to calculate the full-spectrum reflection characteristics of objects in the scene; Step 2: Utilize infrared image inversion to calculate the thermal radiation characteristics of objects in the scene; Step 3: Fusion of full-spectrum reflection characteristics and thermal radiation characteristics of objects to obtain a high-resolution infrared scene, and reconstruct infrared images through full-link simulation; Step 4: Establish a degradation relationship between a real high-resolution infrared image and a measured low-resolution infrared image, and use the measured infrared image to correct the reconstructed simulation result to obtain a high-fidelity high-resolution infrared image. The present invention can provide high-resolution image data for surface analysis, environmental monitoring, disaster prevention and control, and at the same time provide characteristic priors and data input for the study of related detection and recognition methods.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical image processing, and in particular relates to a visible light image and infrared image fusion method based on radiation characteristic inversion. Background Art

[0002] High-resolution satellite-borne infrared imaging systems can obtain information on global objects around the clock, reflect surface temperature patterns and their relationship with surface features, and therefore can accurately describe urban land cover types, estimate surface emissivity, soil moisture content, and vegetation coverage, etc. They are widely used in many fields such as urban heat island effect assessment, river temperature monitoring, forest fire and volcanic evolution detection, and surface oil slick monitoring. In addition, due to the atmospheric transmission characteristics of the infrared band, it has stronger penetration than the visible light band under low visibility conditions such as clouds, fog, and haze, so imaging can be repeated day and night and at fixed periods without airspace restrictions.

[0003] High-resolution infrared remote sensing images are of great significance to the development of both military and civilian fields. However, due to the limitations of space-based imaging conditions, the resolution of infrared remote sensing images is relatively low, and algorithm processing is needed to improve the resolution capability.

[0004] The energy that constitutes the infrared image information mainly consists of two parts: sunlight reflection and self-thermal radiation. On the one hand, objects on the ground (i.e., targets), such as the atmosphere, land, water, vegetation, and artificial structures, all have the characteristics of thermal radiation when the temperature is above absolute zero. On the other hand, when sunlight shines on the surface of the earth, objects on the ground will absorb and reflect the sunlight. The radiation energy reflects the characteristics of the objects in the image to a large extent; other characterization quantities of the scene characteristics, such as the shape, size, hue, texture, contrast, variance of the radiation brightness of the objects, and the average spatial details of the objects, are also basically determined by the radiation characteristics of the scene.

[0005] Both the reflective characteristics and the spontaneous radiation characteristics are closely related to the wavelength. The existing methods directly fuse infrared and visible light images at the pixel level to obtain infrared image super-resolution results, which are difficult to reflect the real infrared characteristics of the scene and difficult to provide reliable data input for applications such as urban heat island effect assessment, river temperature monitoring, forest fire and volcanic evolution detection, and surface oil floating monitoring. Therefore, it is necessary to fuse and super-resolution infrared images based on physical mechanisms. Summary of the invention

[0006] In order to solve the technical problem of low resolution of infrared remote sensing images in the prior art, the present invention provides a visible light image and infrared image fusion method based on radiation characteristic inversion, which is based on the optical remote sensing imaging mechanism, and respectively utilizes the measured visible light and infrared images to invert and calculate the spectral reflectance characteristics and temperature characteristics of the ground object scene, and then physically fuses the visible light and infrared information at the radiation characteristic level, and then obtains a high-fidelity infrared image fusion super-resolution result through imaging reconstruction and error correction, which more realistically reflects the high-resolution infrared characteristics.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0008] A visible light image and infrared image fusion method based on radiation characteristic inversion includes the following steps:

[0009] Step 1: Use multi-spectral visible light images to invert and calculate the full-spectrum reflectance characteristics of the objects in the scene;

[0010] Step 2: Use infrared images to invert and calculate the thermal radiation characteristics of objects in the scene;

[0011] Step 3: The full-spectrum reflectance characteristics and thermal radiation characteristics of the ground objects are integrated to obtain a high-resolution infrared scene, and the infrared image is reconstructed through full-link simulation;

[0012] Step 4: Establish the degradation relationship between the real high-resolution infrared image and the measured low-resolution infrared image, and use the measured infrared image to correct the reconstruction result to obtain a high-fidelity high-resolution infrared image.

[0013] The beneficial effects of the present invention relative to the prior art are:

[0014] The present invention is based on the mechanism of optical remote sensing imaging, and respectively utilizes the measured visible light and infrared images to invert and calculate the spectral reflectance characteristics and temperature characteristics of the ground scene, and then physically fuses the visible light and infrared information at the radiation characteristic level. Then, through imaging reconstruction and error correction, a high-fidelity infrared image fusion super-resolution result is obtained, which more realistically reflects the high-resolution infrared characteristics, and can provide high-resolution image data for surface analysis, environmental monitoring, disaster prevention and control, etc. At the same time, it provides characteristic priors and data input for the research of related detection and recognition methods.

[0015] The specific innovations are as follows:

[0016] (1) Starting from the physical mechanism, the reflectivity characteristics and thermal radiation characteristics are obtained by using visible light multispectral and infrared image inversion respectively, and then the complete radiation information of the ground object scene is obtained by fusion, which can more realistically reflect the infrared characteristics and provide high-confidence input for image reconstruction;

[0017] (2) Based on the optical remote sensing imaging mechanism, a full-link reconstruction method from the target scene to the digital image is proposed. Then, according to the correlation between the high-resolution and low-resolution images, the reconstructed image is refined and corrected to achieve high-confidence infrared image fusion super-resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 : Infrared image fusion super-resolution flow chart;

[0019] Figure 2 :Image reconstruction flow chart based on the full link of optical remote sensing;

[0020] Figure 3 : Display of infrared image fusion super-resolution effect: before processing (a), after processing (b). DETAILED DESCRIPTION

[0021] The following is combined with Figure 1-3 The present invention is described in detail with reference to the accompanying drawings and specific embodiments. Specific implementation method one:

[0023] Step 1: Use multi-spectral visible light images to invert and calculate the full-spectrum reflectance characteristics of the objects in the scene.

[0024] The radiation brightness at the entrance pupil of the satellite sensor in the visible light band mainly comes from three transmission paths: solar radiation reflected from the ground surface reaches the sensor L reflected , solar radiation is scattered by the atmosphere multiple times before reaching the surface of the earth to the sensor L path , the reflected radiation from the ground is scattered multiple times by the atmosphere and enters the sensor L adjacency .

[0025] L sensor =L path +L reflected +L adjacency (1)

[0026] Assuming that the ground is a Lambertian reflector, the radiation received by the satellite sensor can be expressed as:

[0027]

[0028] Where, L sensor is the radiation brightness observed by the satellite, L path is the atmospheric path radiation, ρ is the surface reflectivity, E g is the total downward radiation flux, including the direct solar radiation and the diffuse radiation flux from the sky, T is the atmospheric transmittance, and S is the atmospheric hemispherical albedo.

[0029] E g The product of T is regarded as the new unknown quantity P, and the intermediate variable but:

[0030]

[0031] Based on the above formula, the aerosol optical thickness values ​​of each band, the on-board radiation brightness values ​​obtained after radiation calibration, and the established characterization model of the radiation brightness received by the sensor can be used to realize the pixel-by-pixel surface reflectance inversion of the measured multi-spectral visible light image. The spectral feature point set extracted from the measured spectral reflectance database can then be used to guide the subdivision and interpolation of the spectral reflectance of the objects. By comparing the differences between the interpolation results and the subdivided spectra guiding the interpolation, the classification of the objects in the original image scene and the acquisition of the full-spectrum reflectance information can be achieved.

[0032] Step 2: Use low-resolution infrared images to invert and calculate the thermal radiation characteristics of objects in the scene;

[0033] The radiation energy L received by the satellite in the infrared band infrared Thermal radiation from the ground object itself L(T s ), atmospheric upward thermal radiation L a ↑ and the downward thermal radiation of the atmosphere L a ↓The part that is reflected by the ground and then returns to the infrared sensor:

[0034] L infrared =τεL(T s )+τ(1-ε)L a ↓+L a ↑ (4)

[0035] In the formula, ε represents the ground object emissivity, and τ represents the atmospheric transmittance.

[0036] According to Planck's blackbody radiation formula, the thermal radiation of the object itself L(T s ) and ground temperature T s And the relationship between the wavelength λ is:

[0037]

[0038] In the formula, h represents Planck's constant, c is the speed of light, k represents the Boltzmann constant, K 1 =2hc 2 / λ 5 , e is a natural constant. Performing a first-order Taylor expansion on the temperature, we obtain a single-window algorithm for temperature inversion for infrared band data of remote sensing images:

[0039] T s =[K 2 (C+D)B(T s )+(1-CD)B(T s ) 2 -K 2 DTa ] / K 2 C (6)

[0040]

[0041] In the formula, C=ετ, D=(1-τ)[1+(1-ε)τ], K 2 =1321.08, B(T s ) is the temperature T s The radiation brightness of the black body received by the infrared sensor, T a is the average atmospheric temperature, gain is gain, offset is compensation, and N is the image grayscale value. It can be seen that the key to temperature inversion is to obtain the values ​​of surface emissivity and atmospheric transmittance.

[0042] The atmospheric transmittance τ can be obtained from the MODTRAN classic atmospheric model, and the surface emissivity ε is obtained using the surface vegetation coverage P v Perform the calculation:

[0043]

[0044] In the formula, NDVI represents the normalized difference vegetation index, s NDVI is the normalized difference vegetation index value of bare soil or building surface. v is the normalized vegetation index value of the vegetation area. Then:

[0045]

[0046] In the formula, ε water , ε building , ε natural Represent the spontaneous emission rates of water, buildings and vegetation respectively.

[0047] Step 3: The full-spectrum reflectance characteristics and thermal radiation characteristics of the ground objects are integrated to obtain a high-resolution infrared scene, and the infrared image is reconstructed through full-link simulation;

[0048] To reconstruct an infrared image, we must first obtain the surface radiation brightness L land , mainly composed of solar radiation reflected by the ground objects and thermal radiation of the ground objects themselves L(T s ), and the downward thermal radiation of the atmosphere L a ↓The part reflected by the ground objects.

[0049]

[0050] The imaging of the space-based optical earth observation system is a multi-link integrated process. The imaging link starts with the scene target and ends with the image. Based on the reflection characteristics and thermal radiation characteristics of the ground objects inverted in steps one and two, the radiation characteristics of the target scene can be constructed. The radiation light wave is transmitted through the atmosphere, enters the optical system of the satellite platform, and is imaged on the detector. After photoelectric conversion and quantification, a digital image is obtained.

[0051] The influence of atmosphere on remote sensing imaging is mainly reflected in absorption and scattering, which causes the energy of light waves to attenuate. Let T(λ) be the atmospheric spectral transmittance function under absorption and scattering, then:

[0052] L a (λ)=T(λ)L land (λ) (11)

[0053] Where, L a (λ) is the spectral radiation brightness after atmospheric transmission, and λ is the wavelength. After the ground object's own radiation and the radiation reflected from the sun are transmitted through the atmosphere, they are collected by the optical system and converged onto the detector for imaging. The imaging process of the optical system satisfies the property of linear superposition, so the incident object radiation brightness L a (x', y') forms the image plane radiation brightness distribution L after passing through the imaging system op (x,y) can be expressed in the following integral form:

[0054]

[0055] Where (x', y') and (x, y) are the coordinates of the incident plane and image plane respectively. is the impulse response of the corresponding coordinate point; the optical system is usually considered to be a linear invariant system, and the imaging process of the optical system is expressed in the form of convolution:

[0056] L op (x,y)=∫∫h r (x-x',y-y')L a (x',y')dx'dy'=h r *L a (13)

[0057] In the formula, h r Represents the point spread function of the imaging system. In the remote sensing system, it is composed of optical aberrations, platform vibrations and other aspects.

[0058] The detector pixel samples and integrates the energy of the target scene image, and converts the integrated photons into electrons through photoelectric conversion. When the electrons in the pixel accumulate to a certain number, they are converted into current or voltage. 1 ~λ 2 The number of electrons generated by one pixel of the inner detector is Se It is expressed as follows:

[0059]

[0060] In the formula, A d is the detector pixel area; τ 0 (λ) is the transmittance of the optical system (including the transmittance of the filter); T int is the detector integration time; η(λ) is the quantum efficiency of the device; F is the F number of the optical system; L op (λ) is the spectral radiation brightness received by the detector; h is Planck's constant and c is the speed of light.

[0061] Finally, the electronics system amplifies, clamps, reduces noise, and digitizes the signal from the detector. sa is the detector well depth (in electrons), Ln is the number of quantization bits, L sb is the minimum quantization unit or quantization interval. If the A / D converter matches the output of the amplifier, the highest output voltage corresponds to the well depth of the detector. For uniform quantization, L sb As shown in the following formula.

[0062]

[0063] The quantitative result is:

[0064]

[0065] Where x and y are the coordinates of the detector pixel, S e (x,y) represents the number of electrons generated by the pixel at position (x,y), g is the reconstruction result of the infrared image, and Floor(·) represents the rounding operation. The above steps can be used to calculate the reconstruction result of the infrared remote sensing image.

[0066] Step 4: Establish the degradation relationship between the real high-resolution infrared image and the measured low-resolution infrared image, and use the measured infrared image to correct the reconstruction result to obtain a high-fidelity high-resolution infrared image.

[0067] The above steps obtain images based on the inversion of ground object characteristics and full-link imaging reconstruction, which can basically reflect the infrared characteristics of the scene. However, errors are inevitable in the process of characteristic inversion and image reconstruction, which affects the fidelity of the infrared image fusion super-resolution results. To address this problem, this step establishes the degradation relationship between the real high-resolution infrared image and the measured low-resolution infrared image, proposes an image error correction method based on the imaging model, and uses the low-resolution infrared image obtained by actual observation for refined correction to achieve high-confidence fusion super-resolution of infrared images, where high resolution and low resolution are relative and there is no need to limit specific values.

[0068] Combined with the relevant theories of the above steps, the low-resolution measured images of a given scene are considered to be generated by the real high-resolution images through the full-link degradation process of optical remote sensing:

[0069]

[0070] Where: (x 2 ,y 2 ) is the low-resolution image coordinate, (x 1 ,y 1 ) represents the high-resolution image coordinates, g l represents a low-resolution image, g h represents the actual high-resolution image, n is the noise; h d (x 1 ,y 1 ;x 2 ,y 2 ) represents the weight of the point spread function, that is, the high-resolution image at (x 1 ,y 1 ) position of the pixel in the low-resolution image at (x 2 ,y 2 )Contribution of pixel grayscale at position .

[0071] According to the mechanism of optical remote sensing imaging, the degradation process can be abstractly decomposed into three processes: optical blur, subsampling, and random noise, so as to establish the imaging model as follows:

[0072] g l =D hl Hg h +n (18)

[0073] Where D hl represents the resolution reduction operator; H represents the fuzzy kernel operator, which is equivalent to the system transfer function. The error function E is defined according to the least squares principle:

[0074] E=g l -(D hl Hg h ) (19)

[0075] Through iterative optimization, the mean square error between the high-resolution estimation solution of the infrared image and the actual observed image after degradation is minimized. At the same time, in order to protect the texture details of the high-resolution image during the iteration process, the total variation constraint term is introduced based on the image scene Laplace prior. The image correction objective function is:

[0076]

[0077] In the formula, represents the final high-resolution infrared image calculation result, represents a bidirectional differential filter, ||·|| 1 、||·|| 2 Respectively represent the 1-norm and 2-norm calculation symbols. The model can be solved according to the gradient descent method or the iterative shrinkage threshold method, and the final high-fidelity infrared image fusion super-resolution result can be obtained after iterative calculation.

[0078] The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A visible light image and infrared image fusion method based on radiation characteristic inversion, characterized in that: The following steps are involved: Step 1: Use multi-spectral visible light images to invert and calculate the full-spectrum reflectance characteristics of the objects in the scene; Step 2: Use infrared images to invert and calculate the thermal radiation characteristics of objects in the scene; Step 3: The full-spectrum reflectance characteristics and thermal radiation characteristics of the ground objects are integrated to obtain a high-resolution infrared scene, and the infrared image is reconstructed through full-link simulation; Step 4: Establish the degradation relationship between the real high-resolution infrared image and the measured low-resolution infrared image, and use the measured infrared image to correct the reconstruction result to obtain a high-fidelity high-resolution infrared image; In step 1, assuming that the ground is a Lambertian surface reflector, the radiation received by the satellite sensor in the visible light band can be expressed as: Where, L sensor is the radiation brightness observed by the satellite, L path is the atmospheric path radiation, ρ is the surface reflectivity, E g is the total downward radiation flux, including the direct solar radiation and the diffuse radiation flux from the sky, T is the atmospheric transmittance, S is the atmospheric hemispherical albedo, E g The product of T is regarded as the new unknown quantity P, and the intermediate variable but: Based on the above formula, the aerosol optical thickness values ​​of each band, the on-board radiance values ​​obtained after radiation calibration, and the established representation model of the radiance received by the sensor can be used to realize the pixel-by-pixel surface reflectance inversion of the measured multi-spectral visible light image. Then, the spectral feature point set extracted from the measured spectral reflectance database can be used to guide the subdivision and interpolation of the spectral reflectance of the ground objects. By comparing the difference between the interpolation result and the subdivision spectrum guiding the interpolation, the original image scene ground object classification and the acquisition of the full-spectrum reflectance information can be realized. In step 2, the thermal radiation L(T s ), atmospheric upward thermal radiation L a ↑ and the downward thermal radiation of the atmosphere L a ↓Radiant energy L reflected by the ground and then returned to the infrared sensor infrared for: L infrared =τεL(T s )+τ(1-ε)L a ↓+L a ↑ (4) In the formula, ε represents the ground object emissivity, and τ represents the atmospheric transmittance; According to Planck's blackbody radiation formula, the thermal radiation of the object itself L(T s ) and ground temperature T s And the relationship between the wavelength λ is: In the formula, h represents Planck's constant, c is the speed of light, k represents the Boltzmann constant, K1 = 2hc 2 / λ 5 , e is a natural constant; by performing a first-order Taylor expansion on the temperature, we can obtain a single-window algorithm for temperature inversion for infrared band data of remote sensing images: T s =[K2(C+D)B(T s )+(1-C-D)B(T s ) 2 -K2DT a ] / K2C (6) In the formula, C=ετ, D=(1-τ)[1+(1-ε)τ], K2=1321.08, B(T s ) is the temperature T s The radiation brightness of the black body received by the infrared sensor, T a is the average atmospheric temperature, gain is gain, offset is compensation, and N is the image grayscale value. It can be seen that the key to temperature inversion is to obtain the surface emissivity and atmospheric transmittance values; The atmospheric transmittance τ is obtained by the MODTRAN classical atmospheric model, and the surface emissivity ε is obtained by the surface vegetation coverage P v Perform the calculation: In the formula, NDVI represents the normalized difference vegetation index, s NDVI is the normalized difference vegetation index value of bare soil or building surface. v is the normalized vegetation index value of the vegetation area, then: In the formula, ε water , ε building , ε natural Represent the spontaneous emission rates of water, buildings, and vegetation respectively; In step 3, to reconstruct the infrared image, we first need to obtain the surface radiation brightness L land , The imaging of the space-based optical earth observation system is a multi-link integrated process. The imaging link starts with the scene target and ends with the image. Based on the reflection characteristics and thermal radiation characteristics of the ground objects obtained by step one and step two, the radiation characteristics of the target scene are constructed. The radiation light wave is transmitted through the atmosphere and enters the optical system of the satellite platform to be imaged on the detector. After photoelectric conversion and quantification, a digital image is obtained. The influence of atmosphere on remote sensing imaging is mainly reflected in absorption and scattering, which causes the light wave energy to attenuate. Let T(λ) be the atmospheric spectral transmittance function under absorption and scattering, then: L a (λ)=T(λ)L land (l) (11) Where, L a (λ) is the spectral radiation brightness after being transmitted through the atmosphere; the radiation of the ground object itself and the radiation reflected from the sun are transmitted through the atmosphere, collected by the optical system and focused on the detector for imaging. The imaging process of the optical system satisfies the property of linear superposition, so the incident object radiation brightness L a (x', y') forms the image plane radiation brightness distribution L after passing through the imaging system op (x,y) can be expressed in the following integral form: Where (x', y') and (x, y) are the coordinates of the incident plane and image plane respectively. is the impulse response of the corresponding coordinate point; the optical system is usually considered to be a linear invariant system, and the imaging process of the optical system is expressed in the form of convolution: L op (x,y)=∫h r (x-x',y-y')L a (x',y')dx'dy'=h r *L a (13) In the formula, h r represents the point spread function of the imaging system; The detector pixel samples and integrates the energy of the target scene image, and converts the integrated photons into electrons through photoelectric conversion. The number of electrons generated by a pixel of the detector within a spectral range of λ1~λ2 is S e It is expressed as follows: In the formula, A d is the detector pixel area; τ(λ) is the transmittance of the optical system; T int is the detector integration time; η(λ) is the quantum efficiency of the device; F is the F number of the optical system; L op (λ) is the spectral radiation brightness received by the detector; h represents Planck's constant, c is the speed of light; Finally, the electronic system amplifies, clamps, reduces noise, and digitizes the signal from the detector; let S sa is the detector well depth, Ln is the number of quantization bits, L sb is the minimum quantization unit or quantization interval. If the A / D converter matches the output of the amplifier, the highest output voltage corresponds to the well depth of the detector. For uniform quantization, L sb As shown below: The quantitative result is: In the formula, x and y are the coordinates of the detector pixel, g is the reconstruction result of the infrared image, and Floor(·) represents the rounding operation; S e (x,y) represents the number of electrons generated by the pixel at the (x,y) position. The infrared remote sensing image reconstruction result can be calculated through the above steps.

2. The visible light image and infrared image fusion method based on radiation characteristic inversion according to claim 1 is characterized by: In step 4, the low-resolution measured images of the given scene are considered to be generated by the real high-resolution images through the full-link degradation process of optical remote sensing: Where: (x2, y2) low-resolution image coordinates, (x1, y1) represents high-resolution image coordinates, g l represents a low-resolution image, g h represents a high-resolution image, n is noise; h d (x1, y1; x2, y2) represents the weight of the point spread function, that is, the contribution of the pixel at the position (x1, y1) of the high-resolution image to the grayscale of the pixel at the position (x2, y2) of the low-resolution image; According to the mechanism of optical remote sensing imaging, the degradation process can be abstractly decomposed into three processes: optical blur, subsampling, and random noise, so as to establish the imaging model as follows: g l =D hl Hg h +n (18) Where D hl represents the resolution reduction operator; H represents the fuzzy kernel operator, which is equivalent to the system transfer function. The error function E is defined according to the least squares principle: E=g l -(D hl Hg h ) (19) Through iterative optimization, the mean square error between the high-resolution estimation solution of the infrared image and the actual observed image after degradation is minimized. At the same time, in order to protect the texture details of the high-resolution image during the iteration process, the total variation constraint term is introduced based on the image scene Laplace prior. The image correction objective function is: In the formula, represents the final high-resolution infrared image calculation result, represents a bidirectional differential filter, ||·||1 and ||·||2 represent the 1-norm and 2-norm calculation symbols respectively. The model can be solved according to the gradient descent method or the iterative shrinkage threshold method. After iterative calculation, the final high-fidelity infrared image fusion super-resolution result can be obtained.

Images