Coding aperture snapshot type hyperspectral imaging perception modeling method in small airborne equipment

Abstract

The invention relates to a coded aperture snapshot type hyperspectral imaging perception modeling method in small airborne equipment, which comprises the following steps of: collecting light source information by using an imaging spectrometer in hyperspectral imaging, decomposing the light source information into various wavelengths or spectral bands, modulating the light source information from a spatial dimension and a spectral dimension by using a coded aperture, a grating or a scattering prism optical device, and finally, the captured information is projected to a sensor. And then modeling analysis is carried out through coded aperture snapshot spectral imaging in combination with optical information modulation and digital image processing. The problem that space and time are wasted due to storage and calculation of a large amount of data in development and application of current unmanned aerial vehicle-mounted hyperspectral imaging equipment is solved, the dynamic scene spectral information can be rapidly captured, and the information capturing efficiency is greatly improved.

Description

Technical Field

[0001] The present invention relates to computational imaging, high-dimensional multi-channel signal processing, computer science and technology, and in particular to a coded aperture snapshot hyperspectral imaging perception modeling method in a small airborne device. Background Art

[0002] Hyperspectral imagers collect three-dimensional data by collecting the spectral radiation of the scene, which is called a data cube. It adds one-dimensional spectral information to ordinary two-dimensional spatial information, and the image band range it obtains covers the range from visible light to infrared and even thermal infrared. Compared with ordinary RGB spatial image information, hyperspectral images can obtain diagnostic spectral information such as material composition type and thermal radiation characteristics. The carrier platform of hyperspectral imagers can be divided into: drones, aircraft, local workstations, and satellites according to their size. Hyperspectral imagers based on different carrier platforms have their own advantages and disadvantages. Specifically, aerial hyperspectral imagers have a large width and a wide coverage area, but low spatial resolution and poor timeliness; ground spectral imagers are expensive, but they are also inflexible, which will cause a huge waste of human and material resources during the test process. With the improvement of the process level of optical devices, the portability of equipment, and the development of hardware integration technology, hyperspectral imagers based on small drone carrier platforms have developed rapidly. These imaging devices are suitable for various ground test tasks due to their advantages of flexibility and timeliness, including medical diagnosis, mineral identification, water quality monitoring, etc.

[0003] UAV imaging is to capture and analyze the data of the target scene by carrying different types of imagers on the drone. At present, the imaging methods of the hyperspectral imager on the drone include: push-broom scanning imaging, built-in push-broom scanning imaging, and frame imaging. Among them, compared with the other two types of scanning, push-broom scanning imaging has the advantages of easy implementation, high resolution, high system sensitivity, small size, and light weight, becoming the mainstream imaging method for small UAV spectral imaging platforms. However, this imaging method is limited by the size of the detector and has problems such as difficulty in correction. Built-in push-broom imaging has a smaller volume and structure, but it needs to pause in space. Although frame imaging has high imaging efficiency, when processing massive and high-dimensional data, it will cause low image spatial resolution and few spectral bands due to the large amount of data. In summary, although the scanning-based method can collect complete hyperspectral data through multiple exposures, this type of scanning method is at the expense of time resolution and cannot scan the spectral information of the target scene in a dynamic form. Summary of the invention

[0004] Aiming at the problem of space and time waste caused by large amounts of data storage and calculation in the development and application of current unmanned aerial vehicle-mounted hyperspectral imaging equipment, a coded aperture snapshot hyperspectral imaging perception modeling method in small airborne equipment is proposed. Coded aperture snapshot spectral imaging combines optical information modulation and digital image processing, and has the ability to quickly capture spectral information of dynamic scenes, greatly improving the efficiency of information capture.

[0005] The technical solution of the present invention is:

[0006] A coded aperture snapshot hyperspectral imaging perception modeling method in a small airborne device. Hyperspectral imaging uses an imaging spectrometer to collect light source information, decomposes it into individual wavelengths or spectral bands, and uses coded aperture, grating or scattering prism optical devices to modulate the light source information from the spatial dimension and spectral dimension, and finally projects the captured information onto the sensor; the mathematical process of encoding and decoding is as follows:

[0007] Assume that the spectral density function of the target is f 0 (x, y, λ), when the target light passes through the coded aperture template for spatial information modulation, the spectral density process is expressed as:

[0008] f 1 (x,y,λ)=f 0 (x,y,λ)T(x,y) (1)

[0009] Where T(x,y) is the time-varying coded aperture pattern; after the light splitting effect of the scattering prism, the spatial information image corresponding to each band is deviated; then, the compressed measurement value integrated and projected onto the focusing plane array is expressed as:

[0010] f 2 (x,y,λ)=∫∫dx′dy′δ(x′-[x-α(λ)(λ-λ c )])δ(y′-y)f 1 (x′,y′,λ)

[0011] =T(x-α(λ)(λ-λ c ),y)f 0 (x-α(λ)(λ-λ c ),y,λ) (2)

[0012] Among them, δ(x′-[x-α(λ)(λ-λ c )]), δ(y′-y) represents the unit impulse response in the coded aperture spectral imaging system, α(λ) represents the dispersion coefficient, λ c Represents the central band; considering that the light source information intensity within the wavelength range is captured on the focusing plane array, the light source information intensity can be expressed as g(x,y):

[0013] g(x,y)=∫f 0 (x-α(λ)(λ-λ c ),y,λ)T(x-α(λ)(λ-λ c ),y)dλ (3)

[0014] In actual scenarios, the sensor cannot obtain the light field energy in a continuous form and needs to be discretized. Assume that the size of the sensor pixel is Δ, and assume that the size of each aperture in the coded aperture pattern is equal to the sensor pixel size. In this way, the transfer function T(x,y) of the coded aperture pattern at position (i,j) is expressed as:

[0015]

[0016] (i,j) represents any spatial position on the coding template, t i,j represents the transmittance corresponding to the position (i, j), and rect() represents the rectangular pulse response;

[0017] Therefore, the discretization process of the compressed measurement value at the sensor (i, j) position is expressed as:

[0018]

[0019] where S(λ) = α(λ)(λ-λ c ); Assuming the spectral intensity f 0 The discretization form of (x,y,λ) is f i,j,k , then the compression measurement on the sensor is expressed as:

[0020] g i,j =∑ k f i+k,j,k t i+k,j +e i,j (6)

[0021] Among them, g m,n represents the compression measurement value at the position (m,n), t m,n represents the transmittance at that point, e m,n represents the system noise, k represents the spectral deviation, which is proportional to the number of spectral bands; for the convenience of mathematical modeling, the compression measurement process in discretized form is expressed in the form of vector-matrix product:

[0022] g=Ψf+e (7)

[0023] Among them, R represents the size of the space dimension, g∈R N(M+L-1) represents the compressed measurement value in vectorized form on the sensor; f∈R MNL represents the vectorized form of f m,n,k, e represents the system noise in vectorized form; Ψ∈R N(M+L-1)×NML As a forward imaging model operator, it includes the coded aperture pattern in the spatial domain and the scattering process in the spectral domain; wherein the scattering process is realized by the spectroscopic effect of the scattering prism;

[0024] According to the theory of compressed sensing, it can be known that the process of decoding the hyperspectral data cube f from the linear measurement value g is equivalent to the process of solving the underdetermined equations, indicating that the solution of the equation is not unique; then, it is necessary to use the sparse representation in the theory to decompose f, f = Ψθ, that is, the measured signal is sparse or sparse in a certain domain; due to the high correlation characteristics of f in the spatial domain and the spectral domain, The symbol represents the Kronecker product, represents a two-dimensional discrete cosine basis in the spatial domain, represents the one-dimensional wavelet basis in the spectral domain, Ψ 3D represents a joint sparse basis;

[0025] Assuming that the measured signal has a sparse representation, the perception process represented by formula (7) can be rewritten as:

[0026] g=Aθ+e (8)

[0027] Among them, A represents the equivalent perception matrix; in order to solve the image reconstruction problem, it is necessary to construct a sparse constrained optimization problem to solve the coefficient θ:

[0028]

[0029] in, is a sparse constrained optimization function; β represents the upper bound of the noise; the above nonlinear optimization problem can be equivalent to:

[0030]

[0031] in, represents the reconstructed image;

[0032] Among them, τ>0 indicates the adjustment parameter, Represents the bi-norm, which is used to weigh the error between the reconstruction result and the observed value, ‖θ‖ 1 represents a norm, which is limited by the sparsity of the reconstructed solution. The above optimization problem can be solved by, for example, gradient projection sparse reconstruction, two-step iterative shrinkage threshold algorithm, and alternating factor product.

[0033] Furthermore, hyperspectral imaging divides the spectrum acquisition process into two stages:

[0034] First, the scene information is collected by "reducing the dimension" to obtain the observed image, and then the observed image is "upgraded" through calculation and correction steps to reconstruct the original spectral image; the former represents the optical imaging system, namely the "hardware encoder", and the latter represents the software reconstruction algorithm, namely the "software decoder"; among them, the "hardware encoding" of coded aperture imaging can be decomposed into the following steps:

[0035] Step 1. The scene light signal first passes through the objective lens and is projected onto the coding template plane several times, and is spatially modulated along the spatial dimension by the binary coding template;

[0036] Step 2. The modulated light signal is re-projected to the focal plane array of the camera through the relay lens. During the projection process, the scattering prism performs light splitting modulation along the spectral dimension.

[0037] Step 3. The light signal is received by the sensor to form an observation image.

[0038] Preferably, the data set is collected as follows: hyperspectral data structure, the spatial dimension of the image data is 610×340 pixels, with 115 continuous bands; in addition, a total of 12 noise bands need to be removed from the original 115 bands, and the remaining 103 bands are retained; a specific area is selected for the test data, the size is 140×80 pixels, and it contains 8 major landmark categories.

[0039] The beneficial effects of the present invention are:

[0040] Snapshot spectral imaging can capture a complete three-dimensional data cube within a single or several exposure times. Compared with the imaging methods of time series point scanning, line scanning or spectrum scanning carried on small unmanned equipment, the snapshot-based imaging method can dynamically capture the target scene information. Without sacrificing the spatial resolution of the image, the snapshot spectral imaging system has the characteristics of small size, small mass, high degree of hardware simplicity, and no need for special design of optical devices. It can be widely integrated and applied to small unmanned equipment platforms. On this basis, the present invention analyzes the technical principles of snapshot spectral imaging technology, and based on this, conducts detailed mathematical modeling of its system forward process, providing theoretical guidance for coding aperture design, perception process modeling, and reconstruction algorithm design. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a hyperspectral data cube diagram of the present invention;

[0042] Figure 2 This is a schematic diagram of the coded aperture snapshot spectral imaging system of the present invention;

[0043] Figure 3 This is a diagram of the spatial-spectral modulation process of the coded aperture hyperspectral imaging system of the present invention;

[0044] Figure 4 Different types of coded aperture diagrams of the present invention;

[0045] Figure 5 It is a sparse representation graph of the image data in a specific domain of the present invention. DETAILED DESCRIPTION

[0046] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0047] like Figure 1 As shown in the figure, hyperspectral imaging is a multi-dimensional information technology that combines imaging technology with spectral technology. It can simultaneously capture the two-dimensional spatial information and one-dimensional spectral information of the target scene, achieving "image-spectrum integration". These spectral data can be structurally viewed as a spectral data cube, in which each pixel represents the corresponding light intensity. Figure 1 Represents the structure of hyperspectral data. The dataset was acquired by an airborne reflectance optical spectroscopic imager made in Germany over the target area in Pavia, Italy in 2003. The spatial dimension of the image data is 610×340 pixels, with 115 continuous bands. In addition, a total of 12 noisy bands need to be removed from the original 115 bands, leaving the remaining 103 bands. A specific area is selected for the test data, which is 140×80 pixels in size and contains 8 main landmark categories.

[0048] like Figure 2 As shown in the figure, hyperspectral imaging uses an imaging spectrometer to collect light source information, decomposes it into various wavelengths or spectral bands, and uses optical devices such as coded apertures, gratings, and scattering prisms to modulate the light source information from the spatial dimension and the spectral dimension, and finally projects the captured information onto the sensor. Specifically, this technology divides the spectrum acquisition process into two stages: first, the scene information is "reduced" to obtain the observed image, and then the observed image is "upgraded" to reconstruct the original spectral image through calculation, correction and other steps. The former represents the optical imaging system, namely the "hardware encoder", and the latter represents the software reconstruction algorithm, namely the "software decoder". Among them, the "hardware encoding" of coded aperture imaging can be decomposed into the following steps: 1. The scene light signal first passes through the objective lens and is projected several times onto the coding template plane, and is spatially modulated along the spatial dimension by the binary coding template; 2. The modulated light signal passes through the relay lens and is reprojected to the focal plane array of the camera. During the projection process, it is split and modulated along the spectral dimension by the scattering prism; 3. The light signal is received by the sensor to form an observed image.

[0049] The above paragraph describes the hardware optical architecture and process of the coded aperture spectral imager. Next, we will begin to describe the mathematical process of encoding and decoding.

[0050] like Figure 3 As shown, assuming that the spectral density function of the target is f 0 (x, y, λ), when the target light passes through the coded aperture template for spatial information modulation, the spectral density process is expressed as:

[0051] f 1 (x,y,λ)=f 0 (x,y,λ)T(x,y) (1)

[0052] where T(x,y) is the time-varying coded aperture pattern ( Figure 4 Coded aperture patterns representing different spatial distributions, from left to right: Gaussian random coding, Bernoulli coding, blue noise coding). After the light splitting effect of the scattering prism, the spatial information image corresponding to each band has deviated. Then, the compressed measurement value integrated and projected onto the focusing plane array (i.e., the detector, used to collect the compressed image information) is expressed as:

[0053] f 2 (x,y,λ)=∫∫dx ′ dy ′ δ(x ′ -[x-α(λ)(λ-λ c )])δ(y ′ -y)f 1 (x ′ ,y ′ ,λ)

[0054] =T(x-α(λ)(λ-λ c ),y)f 0 (x-α(λ)(λ-λ c ),y,λ) (2)

[0055] Among them, δ(x′-[x-α(λ)(λ-λ c )]), δ(y′-y) represents the unit impulse response in the coded aperture spectral imaging system, α(λ) represents the dispersion coefficient, λ c Represents the central band. Considering that the light source information intensity captured on the focusing plane array is within the wavelength range, the light source information intensity can be expressed as g(x,y):

[0056] g(x,y)=∫f 0 (x-α(λ)(λ-λ c ),y,λ)T(x-α(λ)(λ-λ c ),y)dλ (3)

[0057] In actual scenarios, the sensor cannot obtain the light field energy in a continuous form and needs to be discretized. Assume that the size of the sensor pixel is Δ, and assume that the size of each aperture in the coded aperture pattern is equal to the size of the sensor pixel. In this way, the transfer function T(x,y) of the coded aperture pattern at position (i,j) is expressed as:

[0058]

[0059] (i,j) represents any spatial position on the coding template, t i,j It represents the transmittance corresponding to the position (i, j), and rect() represents the rectangular pulse response.

[0060] Therefore, the discretization process of the compressed measurement value at the sensor (i, j) position is expressed as:

[0061]

[0062] where S(λ) = α(λ)(λ-λ c ). Assume that the spectral intensity f 0 The discretization form of (x,y,λ) is f i,j,k , then the compression measurement on the sensor is expressed as:

[0063] g i,j =∑ k f i+k,j,k t i+k,j +e i,j (6)

[0064] Among them, g m,n represents the compression measurement value at the position (m,n), t m,n represents the transmittance at that point, e m,n represents the system noise, and k represents the spectral deviation, which is proportional to the number of spectral bands. In order to facilitate mathematical modeling, the compression measurement process in discretized form is expressed in the form of vector-matrix product:

[0065] g=Ψf+e (7)

[0066] Among them, R represents the size of the space dimension, g∈R N(M+L-1) represents the compressed measurement value in vectorized form on the sensor. f∈R MNL represents the vectorized form of f m,n,k , e represents the system noise in vectorized form. Ψ∈R N(M+L-1)×NML The forward imaging model operator (perception matrix) includes the coded aperture pattern in the spatial domain and the scattering process in the spectral domain. The scattering process is realized by the spectroscopic effect of the scattering prism.

[0067] According to the theory of compressed sensing, it can be seen that the process of decoding the hyperspectral data cube f from the linear measurement value g is equivalent to the process of solving the underdetermined equations, indicating that the solution of the equation is not unique. Then, it is necessary to use the sparse representation (prior knowledge) in the theory to decompose f, f = Ψθ (such as Figure 5 As shown in Figure 2, the measured signal is sparse or sparse in a certain domain. This is due to the high correlation characteristics of f in the spatial domain and the spectral domain. The symbol represents the Kronecker product, represents a two-dimensional discrete cosine basis in the spatial domain, represents the one-dimensional wavelet basis in the spectral domain, Ψ 3D Represents a joint sparse basis.

[0068] Assuming that the measured signal has a sparse representation, the perception process represented by formula (7) can be rewritten as:

[0069] g=Aθ+e (8)

[0070] Where A represents the equivalent perception matrix. In order to solve the image reconstruction problem, it is necessary to construct a sparse constrained optimization problem to solve the coefficient θ:

[0071]

[0072] in, is the sparse constraint optimization function. β represents the upper bound of the noise. The above nonlinear optimization problem can be equivalent to:

[0073]

[0074] in, Represents the reconstructed image.

[0075] Among them, τ>0 indicates the adjustment parameter, Represents the bi-norm, which is used to weigh the error between the reconstruction result and the observed value, ‖θ‖ 1 represents a norm, which is limited by the sparsity of the reconstructed solution. The above optimization problem can be solved by, for example, gradient projection sparse reconstruction (GPSR), two-part iterative shrinkage threshold (TwIST) algorithm, and alternating factor multiplication (ADMM).

[0076] The above-mentioned embodiment only expresses one implementation mode of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A coded aperture snapshot hyperspectral imaging perception modeling method in a small airborne device, characterized in that: Hyperspectral imaging uses an imaging spectrometer to collect light source information, decompose it into individual wavelengths or spectral bands, modulate the light source information from the spatial and spectral dimensions using coded apertures, gratings, or scattering prism optical devices, and finally project the captured information onto the sensor; the mathematical process of encoding and decoding is as follows: Assuming that the spectral density function of the target is f0(x, y, λ), when the target light passes through the coded aperture template for spatial information modulation, the spectral density process is expressed as: f1(x,y,λ)=f0(x,y,λ)T(x,y) (1) Where T(x,y) is the time-varying coded aperture pattern; after the light splitting effect of the scattering prism, the spatial information image corresponding to each band is deviated; then, the compressed measurement value integrated and projected onto the focusing plane array is expressed as: f2(x,y,λ)=∫∫dx ′ day ′ d(x ′ -[x-a(λ)(λ-λ c )])δ(y ′ -y)f1(x ′ ,y ′ ,l) =T(x-α(λ)(λ-λ) c ),y)f0(x-α(λ)(λ-λ c ),y,λ) (2) Among them, δ(x ′ -[x-α(λ)(λ-λ c )])、δ(y ′ -y) represents the unit impulse response in the coded aperture spectral imaging system, α(λ) represents the dispersion coefficient, and λ c Represents the central band; considering that the light source information intensity within the wavelength range is captured on the focusing plane array, the light source information intensity can be expressed as g(x,y): g(x,y)=∫f0(x-α(λ)(λ-λ c ),y,λ)T(x-α(λ)(λ-λ c ),y)dλ (3) In actual scenarios, the sensor cannot obtain the light field energy in a continuous form and needs to be discretized. Assume that the size of the sensor pixel is Δ, and assume that the size of each aperture in the coded aperture pattern is equal to the sensor pixel size. In this way, the transfer function T(x,y) of the coded aperture pattern at position (i,j) is expressed as: (i,j) represents any spatial position on the coding template, t i,j represents the transmittance corresponding to the position (i, j), and rect() represents the rectangular pulse response; Therefore, the discretization process of the compressed measurement value at the sensor (i, j) position is expressed as: where S(λ) = α(λ)(λ-λ c ); Assume that the discretization form of the spectral intensity f0(x,y,λ) is f i,j,k , then the compression measurement on the sensor is expressed as: g i,j =∑ k f i+k,j,k t i+k,j +e i,j (6) Among them, g m,n represents the compression measurement value at the position (m,n), t m,n represents the transmittance at that point, e m,n represents the system noise, k represents the spectral deviation, which is proportional to the number of spectral bands; for the convenience of mathematical modeling, the compression measurement process in discretized form is expressed in the form of vector-matrix product: g=Ψf+e (7) Among them, R represents the size of the space dimension, g∈R N(M+L-1) represents the compressed measurement value in vectorized form on the sensor; f∈R MNL represents the vectorized form of f m,n,k , e represents the system noise in vectorized form; Ψ∈R N(M+L-1)×NML As a forward imaging model operator, it includes the coded aperture pattern in the spatial domain and the scattering process in the spectral domain; wherein the scattering process is realized by the spectroscopic effect of the scattering prism; According to the theory of compressed sensing, it can be known that the process of decoding the hyperspectral data cube f from the linear measurement value g is equivalent to the process of solving the underdetermined equations, indicating that the solution of the equation is not unique; then, it is necessary to use the sparse representation in the theory to decompose f, f = Ψθ, that is, the measured signal is sparse or sparse in a certain domain; due to the high correlation characteristics of f in the spatial domain and the spectral domain, The symbol represents the Kronecker product, represents a two-dimensional discrete cosine basis in the spatial domain, represents the one-dimensional wavelet basis in the spectral domain, Ψ 3D represents a joint sparse basis; Assuming that the measured signal has a sparse representation, the perception process represented by formula (7) can be rewritten as: g=Aθ+e (8) Among them, A represents the equivalent perception matrix; in order to solve the image reconstruction problem, it is necessary to construct a sparse constrained optimization problem to solve the coefficient θ: in, is a sparse constrained optimization function; β represents the upper bound of the noise; the above nonlinear optimization problem can be equivalent to: in, represents the reconstructed image; Among them, τ>0 indicates the adjustment parameter, represents the second norm, which is used to weigh the error between the reconstruction result and the observation value, and ‖θ‖1 represents the first norm, which is limited by the sparsity of the reconstruction solution. The above optimization problem can be solved by, for example, gradient projection sparse reconstruction, two-step iterative shrinkage threshold algorithm, and alternating factor product.

2. The coded aperture snapshot hyperspectral imaging perception modeling method in a small airborne device according to claim 1, characterized in that: Hyperspectral imaging divides the spectrum acquisition process into two stages: First, the scene information is "reduced in dimension" to obtain the observed image, and then the observed image is "upgraded in dimension" through calculation and correction steps to reconstruct the original spectral image; the former represents the optical imaging system, namely the "hardware encoder", and the latter represents the software reconstruction algorithm, namely the "software decoder"; among them, the "hardware encoding" of coded aperture imaging can be decomposed into the following steps: Step 1. The scene light signal first passes through the objective lens and is projected onto the coding template plane several times, and is spatially modulated along the spatial dimension by the binary coding template; Step 2. The modulated light signal is re-projected to the focal plane array of the camera through the relay lens. During the projection process, the scattering prism performs light splitting modulation along the spectral dimension. Step 3. The light signal is received by the sensor to form an observation image.

3. The coded aperture snapshot hyperspectral imaging perception modeling method in a small airborne device according to claim 1, characterized in that: The dataset is collected as follows: hyperspectral data structure, the spatial dimension of the image data is 610×340 pixels, with 115 continuous bands; in addition, a total of 12 noisy bands need to be removed from the original 115 bands, and the remaining 103 bands are retained; a specific area is selected for the test data, with a size of 140×80 pixels, containing 8 major landmark categories.

Images