Airborne spectrum video reconstruction method and system based on global spectrum coding

Abstract

The invention discloses an airborne spectrum video reconstruction method and system based on global spectrum coding, and the method comprises the steps: building a differentiable mathematical model based on global spectrum coding, building a spectrum reconstruction network model based on autoregressive deep convolution, carrying out the combined training of the differentiable mathematical model and the spectrum reconstruction network model, and carrying out the reconstruction of an airborne spectrum video. The method comprises the following steps: respectively obtaining a multispectral data acquisition model and a multispectral data reconstruction model, obtaining spectral information of a target scene based on the multispectral data acquisition model, carrying out global spectral coding to obtain a multispectral compressed image, and inputting the multispectral compressed image into the multispectral data reconstruction model to carry out decoding reconstruction. And outputting a multispectral reconstruction image of the target scene. Through light-weight, low-bandwidth and low-cost global spectrum coding and a high-precision and low-time-delay spectrum reconstruction network, acquisition and reconstruction of multispectral data can be efficiently completed, an imaging view field can be effectively widened, the size, weight and cost of the system are reduced, and airborne working conditions are met.

Description

Technical Field

[0001] The present invention belongs to the technical field of computational spectral imaging, and in particular relates to an airborne spectral video reconstruction method and system based on global spectral coding. Background Art

[0002] Multispectral imaging technology aims to collect high-dimensional spectral information of a scene. Airborne spectral video imaging systems have very wide application needs in the fields of agricultural and forestry monitoring, remote sensing surveys, pollution detection, etc. However, due to factors such as the limited payload volume and data bandwidth of the airborne platform, efficient airborne spectral video acquisition is still difficult. Among various multispectral imaging schemes, computational spectral imaging technology can more efficiently and conveniently achieve large field of view, lightweight, and low-cost multispectral video data acquisition by combining optical system design and reconstruction algorithm design. It also has the significant advantages of low bandwidth and high throughput, and has huge application potential and good development prospects in the field of airborne spectral imaging. Existing spectral imaging solutions are mainly divided into three categories. Among them, although scanning spectral imaging technology can achieve relatively high spectral sampling accuracy, it requires precise mechanical scanning, so the design and implementation cost of the optical system is high, and the acquisition speed is slow and the stability is poor in application, which is not suitable for rapid spectral acquisition in airborne dynamic scenes; snapshot spectral imaging technology can obtain multi-spectral data in one shot with the help of bandpass filter groups, and the acquisition speed is fast, but it is limited by the number of filters, and the number of spectral bands it can collect is small, and the image signal-to-noise ratio is low; another type of compressed computational spectral imaging technology can significantly reduce the difficulty of optical system design by encoding and compressing the scene spectral information and reconstructing it through algorithm decoding, and realize low-cost, lightweight, portable system construction, and can efficiently complete high-throughput, low-bandwidth, large-field-of-view, high-speed spectral video acquisition. This type of solution has a very broad application prospect in airborne scenarios with limited payload volume and weight and limited data transmission bandwidth. The existing compressed computational spectral imaging technology mainly relies on the combination of two-dimensional random masks and dispersion prisms to achieve pixel-level encoding and acquisition of spectra (i.e., CASS I system). Its overall optical path structure is still long and the field of view is small, making it difficult to achieve extreme miniaturization and lightness. The pixel-level encoding accuracy requirement also makes the system very sensitive to external disturbances, vibrations and other environmental factors, making it difficult to adapt to the work requirements on airborne platforms. In addition, the speed and accuracy of existing spectral reconstruction algorithms are difficult to meet the application requirements in complex scenarios. Summary of the invention

[0003] To solve the above problems, the present invention provides an airborne spectral video reconstruction method and system based on global spectral coding, so as to solve the problems of existing multispectral imaging methods, such as complex system, poor stability, insufficient lightness, large data transmission bandwidth occupancy, small imaging field of view, difficulty in fast dynamic spectral video acquisition, and difficulty in meeting application requirements in airborne dynamic scenes.

[0004] An airborne spectral video reconstruction method based on global spectral coding comprises:

[0005] Establish a differentiable mathematical model based on global spectral encoding, and

[0006] Establish a spectral reconstruction network model based on autoregressive deep convolution;

[0007] The differentiable mathematical model and the spectral reconstruction network model are jointly trained to obtain a multispectral data acquisition model and a multispectral data reconstruction model respectively;

[0008] Based on the multispectral data acquisition model, the spectral information of the target scene is acquired and global spectral encoding is performed to obtain a multispectral compressed image;

[0009] The multispectral compressed image is input into the multispectral data reconstruction model for decoding and reconstruction, and a multispectral reconstructed image of the target scene is output.

[0010] According to a specific embodiment of the present invention, establishing a differentiable mathematical model based on global spectral coding includes:

[0011] According to the sampling range and number of sampling points of the scene spectral information, a multispectral data linear integral equation based on global spectral encoding is established. The multispectral data linear integral equation is expressed as:

[0012]

[0013] Where X(λ) is the spectral image of the scene at wavelength λ, T(λ) is the spectral response of the coding filter at wavelength λ, Y is the spectral compression image collected under global spectral coding, and λ s and λ e All are sampling bands of the spectrum;

[0014] The parameters and variables in the linear integral equation of multispectral data are discretized to obtain a differentiable mathematical model based on global spectral encoding:

[0015]

[0016] In the formula, X n is the nth spectral image obtained after discretization, T nis the spectral response encoding matrix of the nth spectral image, n is the spectral image sequence number, and N is the total number of spectral images.

[0017] According to a specific embodiment of the present invention, establishing a spectral reconstruction network model based on autoregressive deep convolution includes:

[0018] A deep learning network framework is used to build a spectral reconstruction network model based on autoregressive deep convolution. The spectral reconstruction network model includes a feature extraction module, a feature reconstruction module, a data fusion module and an output module.

[0019] The feature extraction module is used to extract features from the input multi-spectral compression image to obtain feature data of the multi-spectral compression image;

[0020] The feature reconstruction module is used to perform feature reconstruction on the feature data to obtain feature reconstruction data of the multi-spectral compressed image;

[0021] The data fusion module is used to fuse the feature reconstruction data of the previous frame of multi-spectral compression image with the feature data of the original multi-spectral compression image;

[0022] The output module is used to output the multispectral data reconstruction results of the multispectral compressed image.

[0023] According to a specific embodiment of the present invention, the deep learning network framework includes PyTorch and / or Tensorflow.

[0024] According to a specific embodiment of the present invention, the differentiable mathematical model and the spectral reconstruction network model are jointly trained to obtain a multispectral data acquisition model and a multispectral data reconstruction model, respectively, including:

[0025] Based on the deep learning end-to-end joint training technology, the differentiable mathematical model and the spectral reconstruction network model are jointly trained with the objective function of minimizing the spectral reconstruction error to obtain the optimal spectral encoding mode and reconstruction network parameters respectively.

[0026] A band-stop filter group or a single coated filter with a corresponding spectral response curve is constructed based on the spectral coding mode and placed between camera sensors to obtain a multispectral data acquisition model;

[0027] A multispectral data reconstruction model is constructed based on the reconstruction network parameters.

[0028] According to a specific embodiment of the present invention, the spectral information of the target scene is acquired based on the multispectral data acquisition model and global spectral encoding is performed to obtain a multispectral compressed image, which includes:

[0029] Collect multispectral image information of the target scene at different wavelengths based on the multispectral data acquisition model;

[0030] A band-stop filter group or a single coated filter is used to perform global spectral encoding on the multispectral image information to obtain a multispectral compressed image.

[0031] An airborne spectral video reconstruction system based on global spectral coding, comprising:

[0032] Model building module, used to build a differentiable mathematical model based on global spectral encoding and a spectral reconstruction network model based on autoregressive deep convolution;

[0033] A model training module is used to jointly train the differentiable mathematical model and the spectral reconstruction network model to obtain a multispectral data acquisition model and a multispectral data reconstruction model respectively;

[0034] A data acquisition module is used to acquire the spectral information of the target scene based on a multispectral data acquisition model and perform global spectral encoding to obtain a multispectral compressed image;

[0035] The data reconstruction module is used to input the multispectral compressed image into the multispectral data reconstruction model for decoding and reconstruction, and output the multispectral reconstructed image of the target scene.

[0036] According to a specific embodiment of the present invention, the data acquisition module further includes:

[0037] A spectral data acquisition module is used to collect multispectral image information of the target scene at different wavelengths based on a multispectral data acquisition model;

[0038] The spectral encoding module is used to perform global spectral encoding on multispectral image information by using a band-stop filter group or a single coated filter to obtain a multispectral compressed image.

[0039] A computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned airborne spectral video reconstruction method based on global spectral coding.

[0040] A computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the above-mentioned airborne spectral video reconstruction method based on global spectral coding.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] The present invention provides an airborne spectral video reconstruction method and system based on global spectral coding. The method uses a bandpass filter group or a single specific coated filter to perform spectral coding and compression acquisition, which can effectively widen the imaging field of view, reduce the volume, weight and cost of the system, increase the stability of the system, and reduce the demand for data transmission bandwidth, thereby meeting the requirements under airborne working conditions. In addition, the method uses a multi-spectral data reconstruction model based on autoregressive deep convolution to decode and reconstruct the collected multi-spectral compressed image, which can achieve multi-spectral data reconstruction with higher accuracy and lower latency. The present invention can efficiently complete the acquisition and reconstruction of multi-spectral data through lightweight, low-bandwidth, low-cost, easy-to-implement global spectral coding and high-precision, low-latency spectral reconstruction network. The system is based on global spectral coding technology, which can be realized by introducing a filter before the sensor, without the need for complex optical path design. The system is simple and compact, lightweight and portable, low cost, can achieve a larger field of view, and the coding and compression acquisition can reduce the data transmission bandwidth requirement, which is suitable for working as an airborne payload. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 It is a flow chart of an airborne spectral video reconstruction method based on global spectral coding according to an embodiment of the present invention.

[0045] Figure 2 It is a flow chart of a method for establishing a differentiable mathematical model based on global spectral coding according to an embodiment of the present invention.

[0046] Figure 3 It is a flow chart of a method for jointly training a differentiable mathematical model and a spectral reconstruction network model provided according to an embodiment of the present invention.

[0047] Figure 4 It is a flow chart of a method for acquiring and processing spectral information of a target scene based on a multispectral data acquisition model provided according to an embodiment of the present invention.

[0048] Figure 5 It is a schematic diagram of the structure of a spectral reconstruction network model based on autoregressive deep convolution according to an embodiment of the present invention.

[0049] Figure 6 It is a schematic diagram of the structure of an airborne spectral video reconstruction system based on global spectral coding according to an embodiment of the present invention.

[0050] Figure 7 It is a structural diagram of a data acquisition module provided according to an embodiment of the present invention.

[0051] Figure 8 It is a schematic diagram of the structure of a computer device provided according to an embodiment of the present invention.

[0052] Reference numerals:

[0053] 01-model building module; 02-model training module; 03-data acquisition module; 04-data reconstruction module;

[0054] 031- Spectral data acquisition module; 032- Spectral encoding module. DETAILED DESCRIPTION

[0055] In order to make those skilled in the art understand the concept and thought of the present invention more clearly, the present invention is described in detail below in conjunction with specific embodiment.It should be understood that the embodiment provided herein is only a part of all embodiments that the present invention may have.Those skilled in the art, after reading the specification of the application, have the ability to make improvements, transformations, or replacements to part or the whole of the following embodiments, and these improvements, transformations, or replacements are also included in the scope of the present invention.

[0056] In this article, the terms "first", "second" and other similar words are not intended to imply any order, quantity and importance, but are only used to distinguish different elements. In this article, the terms "one", "an" and other similar words are not intended to indicate that there is only one thing, but to indicate that the relevant description is only for one of the things, and the thing may have one or more. In this article, the terms "comprise", "include" and other similar words are intended to indicate logical mutual relations, and cannot be regarded as indicating spatial structural relations. For example, "A includes B" is intended to indicate that B belongs to A logically, but does not mean that B is located inside A in space. In addition, the meanings of the terms "comprise", "include" and other similar words should be regarded as open, rather than closed. For example, "A includes B" is intended to indicate that B belongs to A, but B does not necessarily constitute the whole of A, and A may also include other elements such as C, D, and E.

[0057] In this document, the terms "embodiment", "this embodiment", "one embodiment", and "an embodiment" do not mean that the relevant description is only applicable to a specific embodiment, but rather that the description may also be applicable to one or more other embodiments. Those skilled in the art should understand that in this document, any description of a certain embodiment can be replaced, combined, or combined in other ways with the relevant descriptions in one or more other embodiments, and the new embodiments generated by the replacement, combination, or combination in other ways are easily conceivable by those skilled in the art and fall within the scope of protection of the present invention.

[0058] Example 1

[0059] Additional aspects and advantages of embodiments of the present invention will be described in part in the following description, and will become apparent from the following description, or will be learned through practice of embodiments of the present invention. Figure 1-Figure 5 The embodiment of the present invention provides an airborne spectral video reconstruction method based on global spectral coding, comprising:

[0060] S1: Establish a differentiable mathematical model based on global spectral encoding.

[0061] S2: Establish a spectral reconstruction network model based on autoregressive deep convolution.

[0062] S3: The differentiable mathematical model and the spectral reconstruction network model are jointly trained to obtain a multispectral data acquisition model and a multispectral data reconstruction model, respectively.

[0063] S4: Based on the multispectral data acquisition model, the spectral information of the target scene is acquired and global spectral encoding is performed to obtain a multispectral compressed image.

[0064] S5: Input the multispectral compressed image into the multispectral data reconstruction model for decoding and reconstruction, and output the multispectral reconstructed image of the target scene.

[0065] The present invention establishes a differentiable mathematical model based on global spectral coding and a spectral reconstruction network model based on autoregressive deep convolution, and uses deep learning end-to-end joint training technology to design and optimize model parameters, thereby obtaining a multispectral data acquisition model and a multispectral data reconstruction model. The multispectral data acquisition model is used to perform global spectral coding on the spectral information of the target scene to obtain a multispectral compressed image, and the multispectral data reconstruction model is used to decode and reconstruct the input multispectral compressed image to obtain a multispectral reconstruction graph of the target scene. The present invention can efficiently complete the acquisition and reconstruction of multispectral data through lightweight, low-bandwidth, low-cost, easy-to-implement global spectral coding and high-precision, low-latency spectral reconstruction network.

[0066] Specifically, step S1 of establishing a differentiable mathematical model based on global spectral encoding includes:

[0067] S11: A multispectral data linear integral equation based on global spectral coding is established according to the sampling range and number of sampling points of the scene spectral information. The multispectral data linear integral equation is expressed as:

[0068]

[0069] Where X(λ) is the spectral image of the scene at wavelength λ, T(λ) is the spectral response of the coding filter at wavelength λ, Y is the spectral compression image collected under global spectral coding, and λ s and λ e These are the sampling bands of the spectrum.

[0070] S12: Discretize the parameters and variables in the linear integral equation of multispectral data to obtain a differentiable mathematical model based on global spectral encoding:

[0071]

[0072] In the formula, X n is the nth spectral image obtained after discretization, T n is the spectral response encoding matrix of the nth spectral image, n is the spectral image sequence number, and N is the total number of spectral images.

[0073] In an embodiment of the present invention, the differentiable mathematical model based on global spectral coding established in step S1 is essentially a multi-spectral data linear integration model weighted by the spectral response function of the filter or filter group. By establishing the differentiable mathematical model, spectral compression data acquisition of global spectral coding can be realized.

[0074] Specifically, step S2 establishes a spectral reconstruction network model based on autoregressive deep convolution, including:

[0075] Use PyTorch and / or Tensorflow's deep learning network framework to build a spectral reconstruction network model based on autoregressive deep convolution, such as Figure 4 As shown, the spectral reconstruction network model includes a feature extraction module, a feature reconstruction module, a data fusion module and an output module, each module is composed of a convolution layer and a residual block, wherein the feature extraction module is used to extract features from the input multi-spectral compression image to obtain feature data of the multi-spectral compression image. The feature reconstruction module is used to perform feature reconstruction on the feature data to obtain feature reconstruction data of the multi-spectral compression image. The data fusion module is used to perform data fusion on the feature reconstruction data of the previous frame of multi-spectral compression image with the feature data of the original multi-spectral compression image. The output module is used to output the multi-spectral data reconstruction result of the multi-spectral compression image.

[0076] Autoregressive deep convolution is a technology that combines autoregressive models and convolutional neural networks. It is mainly used to process time series data or image data. PyTorch and Tensorflow are both open source machine learning frameworks for building neural network models. In an embodiment of the present invention, the input spectral compression image first undergoes feature extraction through a feature extraction module, and then undergoes reconstruction of the first spectral image through a feature reconstruction module and an output module. Starting from the second spectral image, when reconstructing the spectral image, the feature reconstruction result of the previous frame and the feature data of the original spectral compression image are first fused, and then subsequent reconstruction and output steps are performed. When reconstructing different spectral images, the parameters of each network module can be shared. The spectral reconstruction network model based on autoregressive deep convolution constructed by the present invention can provide better spectral reconstruction accuracy and speed, and can achieve higher accuracy and lower latency multi-spectral data reconstruction.

[0077] Specifically, step S3 jointly trains the differentiable mathematical model and the spectral reconstruction network model to obtain a multispectral data acquisition model and a multispectral data reconstruction model, respectively, including:

[0078] S31: Based on deep learning end-to-end joint training technology, the differentiable mathematical model and the spectral reconstruction network model are jointly trained with the objective function of minimizing the spectral reconstruction error to obtain the optimal spectral coding mode and reconstruction network parameters respectively.

[0079] S32: construct a band-stop filter group or a single coated filter of a corresponding spectral corresponding curve based on the spectral encoding mode and place it between camera sensors to obtain a multi-spectral data acquisition model.

[0080] S33: Construct a multispectral data reconstruction model based on the reconstruction network parameters.

[0081] End-to-end joint training of deep learning refers to training multiple related tasks in one model at the same time, usually involving multiple neural network models. These models cooperate with each other during the training process to jointly optimize an objective function. End-to-end joint training can reduce artificial interference, and the model can automatically learn features from the original data, thereby providing the accuracy and robustness of the model. The embodiment of the present invention first takes minimizing the spectral reconstruction error as the objective function, and obtains the optimal spectral encoding mode and reconstruction network parameters respectively by jointly training the differentiable mathematical model and the spectral reconstruction network model, and then constructs a band-stop filter group or a single coated filter of the corresponding spectral response curve according to the optimal spectral encoding mode, and places the band-stop filter group or the single coated filter in front of the camera sensor, that is, a multi-spectral data acquisition model is obtained. The band-stop filter is a filter that only blocks a certain narrow band of light. The band-stop filter group refers to a group of band-stop filters, which can be equivalently directly replaced by a coated filter with a specific spectral response curve. By constructing a multi-spectral data acquisition model, the global encoding of the spectral information of the target scene can be realized to obtain a multi-spectral compressed image. A multispectral data reconstruction model can be constructed according to the optimal reconstruction network parameters, and the decoding and reconstruction of the multispectral compressed image can be realized by constructing the multispectral data reconstruction model.

[0082] Specifically, step S4 acquires and processes the spectral information of the target scene based on the multispectral data acquisition model to obtain a multispectral compressed image, including:

[0083] S41: Collect multispectral image information of the target scene at different wavelengths based on the multispectral data acquisition model.

[0084] S42: globally spectrally encode the multispectral image information using a band-stop filter group or a single coated filter to obtain a multispectral compressed image.

[0085] In an embodiment of the present invention, by placing a band-stop filter group or a single coated filter in front of the sensor of the camera, the multi-spectral image information of the collected target scene can be globally spectrally encoded. Global spectral encoding is a technology for converting optical signals into digital signals. In optical sensors, the spectral information of optical signals can be encoded into digital signals for subsequent processing and analysis. For spectral imaging, multi-spectral imaging can be achieved by encoding the light source, that is, collecting image information of the scene at different wavelengths. In an embodiment of the present invention, the multi-spectral data acquisition model uses a group of band-stop filters placed in front of the sensor or a single coated filter with a specific spectral response to perform global spectral encoding on the scene spectral information. The encoded spectral information is coupled and collected by the grayscale sensor, that is, a multi-spectral compressed image is obtained. The present invention performs global spectral encoding on the scene spectral information, which can be achieved by placing a filter in front of the sensor, without the need for complex optical path design. The system is simple, compact, lightweight and portable, and has low cost. It can effectively widen the imaging field of view, reduce the volume, weight and cost of the system, increase the stability of the system, and reduce the demand for data transmission bandwidth, thereby meeting the requirements under airborne working conditions.

[0086] Specifically, step S5 inputs the multispectral compressed image into the multispectral data reconstruction model for decoding and reconstruction, and outputs the multispectral reconstructed image of the target scene. The embodiment of the present invention loads the optimal reconstruction network parameters obtained in step S3 into the multispectral data reconstruction model, and inputs the collected spectral compression image, so as to output the corresponding multispectral reconstructed image.

[0087] Example 2

[0088] Based on the above method, the embodiment of the present invention also provides an airborne spectral video reconstruction system based on global spectral coding, such as Figure 6 and Figure 7 As shown, including:

[0089] Model building module 01 is used to establish a differentiable mathematical model based on global spectral encoding and a spectral reconstruction network model based on autoregressive deep convolution.

[0090] The model training module 02 is used to jointly train the differentiable mathematical model and the spectral reconstruction network model to obtain a multispectral data acquisition model and a multispectral data reconstruction model respectively.

[0091] The data acquisition module 03 is used to acquire the spectral information of the target scene based on the multispectral data acquisition model and perform global spectral encoding to obtain a multispectral compressed image.

[0092] The data reconstruction module 04 is used to input the multispectral compressed image into the multispectral data reconstruction model for decoding and reconstruction, and output a multispectral reconstructed image of the target scene.

[0093] The present invention establishes a differentiable mathematical model based on global spectral coding and a spectral reconstruction network model based on autoregressive deep convolution through a model building module 01. The model parameters are designed and optimized through a model training module 02 to obtain a multispectral data acquisition model and a multispectral data reconstruction model respectively. The multispectral data acquisition model is used to perform global spectral coding on the spectral information of the target scene to obtain a multispectral compressed image, and the multispectral data reconstruction model is used to decode and reconstruct the input multispectral compressed image to obtain a multispectral reconstruction graph of the target scene. The spectral information of the target scene is acquired through a data acquisition module 03 and global spectral coding is performed to obtain a multispectral compressed image. The multispectral compressed image is input into the multispectral data reconstruction model through a data reconstruction module 04 for decoding and reconstruction, and a multispectral reconstructed image of the target scene is output. The present invention can efficiently complete the acquisition and reconstruction of multispectral data through lightweight, low-bandwidth, low-cost, easy-to-implement global spectral coding and a high-precision, low-latency spectral reconstruction network. The differentiable mathematical model is essentially a linear integral model of multi-spectral data weighted by the spectral response function of the filter or filter group. By establishing the differentiable mathematical model, spectral compression data acquisition with global spectral encoding can be realized.

[0094] Among them, the differentiable mathematical model based on global spectral coding is essentially a multi-spectral data linear integration model weighted by the spectral response function of the filter or filter group. By establishing a differentiable mathematical model, spectral compression data acquisition of global spectral coding can be realized. The spectral reconstruction network model based on autoregressive deep convolution is a spectral reconstruction network model built using the deep learning network framework of PyTorch and / or Tensorflow, such as Figure 4As shown, the spectral reconstruction network model includes a feature extraction module, a feature reconstruction module, a data fusion module and an output module, each module is composed of a convolution layer and a residual block, wherein the feature extraction module is used to extract features from the input multi-spectral compression image to obtain feature data of the multi-spectral compression image. The feature reconstruction module is used to perform feature reconstruction on the feature data to obtain feature reconstruction data of the multi-spectral compression image. The data fusion module is used to perform data fusion on the feature reconstruction data of the previous frame of the multi-spectral compression image and the feature data of the original multi-spectral compression image. The output module is used to output the multi-spectral data reconstruction result of the multi-spectral compression image. In an embodiment of the present invention, the input spectral compression image first passes through the feature extraction module to complete feature extraction, and then passes through the feature reconstruction module and the output module to complete the reconstruction of the first spectral image. Starting from the second spectral image, when reconstructing the spectral image, the feature reconstruction result of the previous frame and the feature data of the original spectral compression image will be fused first, and then the subsequent reconstruction and output steps will be performed. When reconstructing different spectral images, the parameters of each network module can be shared. The spectral reconstruction network model based on autoregressive deep convolution constructed in the present invention can provide better spectral reconstruction accuracy and speed, and can achieve multi-spectral data reconstruction with higher accuracy and lower latency.

[0095] Specifically, the model training module 02 firstly uses the deep learning end-to-end joint training technology to jointly train the differentiable mathematical model and the spectral reconstruction network model with the objective function of minimizing the spectral reconstruction error, and obtains the optimal spectral encoding mode and reconstruction network parameters respectively. Then, a band-stop filter group or a single coated filter of the corresponding spectral corresponding curve is constructed based on the spectral encoding mode and placed between the camera sensors to obtain a multispectral data acquisition model. Finally, a multispectral data reconstruction model is constructed based on the reconstruction network parameters.

[0096] Specifically, the data acquisition module 03 also includes:

[0097] The spectral data acquisition module 031 is used to collect multispectral image information of the target scene at different wavelengths based on the multispectral data acquisition model.

[0098] The spectral encoding module 032 is used to perform global spectral encoding on the multi-spectral image information by using a band-stop filter group or a single coated filter to obtain a multi-spectral compressed image.

[0099] In an embodiment of the present invention, multispectral image information of the target scene at different wavelengths is collected by the spectral data acquisition module 031, and then the multispectral image information is globally spectrally encoded by the spectral encoding module 032 to obtain a multispectral compressed image. The multispectral data acquisition model uses a set of band-stop filters placed in front of the sensor or a single coated filter with a specific spectral response to perform global spectral encoding on the scene spectral information. The encoded spectral information is coupled and collected by the grayscale sensor to obtain a multispectral compressed image. The present invention performs global spectral encoding on the scene spectral information, which can be achieved by placing a filter in front of the sensor without the need for complex optical path design. The system is simple, compact, lightweight, portable, and low-cost. It can effectively widen the imaging field of view, reduce the volume, weight, and cost of the system, increase the stability of the system, and reduce the demand for data transmission bandwidth, thereby meeting the requirements under airborne working conditions.

[0100] Example 3

[0101] like Figure 8 As shown, an embodiment of the present invention further provides a computer device, the computer device includes a processor and a memory, a computer program is stored in the memory, the computer program is loaded and executed by the processor to implement the above-mentioned airborne spectral video reconstruction method based on global spectral coding, and the device in the present invention can be a server, PC, PAD, mobile phone, etc.

[0102] Furthermore, an embodiment of the present invention also provides a computer-readable storage medium, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned airborne spectral video reconstruction method based on global spectral coding.

[0103] Furthermore, an embodiment of the present invention also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that initializes the steps of the airborne spectral video reconstruction method based on global spectral coding as described above.

[0104] In summary, the present invention has the following advantages:

[0105] The present invention provides an airborne spectral video reconstruction method and system based on global spectral coding. The method uses a bandpass filter group or a single specific coated filter to perform spectral coding and compression acquisition, which can effectively widen the imaging field of view, reduce the volume, weight and cost of the system, increase the stability of the system, and reduce the demand for data transmission bandwidth, thereby meeting the requirements under airborne working conditions. In addition, the method uses a multi-spectral data reconstruction model based on autoregressive deep convolution to decode and reconstruct the collected multi-spectral compressed image, which can achieve multi-spectral data reconstruction with higher accuracy and lower latency. The present invention can efficiently complete the acquisition and reconstruction of multi-spectral data through lightweight, low-bandwidth, low-cost, easy-to-implement global spectral coding and high-precision, low-latency spectral reconstruction network. The system is based on global spectral coding technology, which can be realized by introducing a filter before the sensor, without the need for complex optical path design. The system is simple and compact, lightweight and portable, low cost, can achieve a larger field of view, and the coding and compression acquisition can reduce the data transmission bandwidth requirement, which is suitable for working as an airborne payload.

[0106] The above describes in detail the concepts, principles and ideas of the present invention in conjunction with specific implementation methods (including embodiments and examples). Those skilled in the art should understand that the implementation methods of the present invention are not limited to the forms given above. After reading this application document, those skilled in the art can make any possible improvements, substitutions and equivalent forms to the steps, methods, systems and components in the above implementation methods. These improvements, substitutions and equivalent forms should be deemed to fall within the scope of the present invention, and the protection scope of the present invention is subject only to the claims.

Claims

1. An airborne spectral video reconstruction method based on global spectral coding, characterized in that: include: Establish a differentiable mathematical model based on global spectral encoding, and Establish a spectral reconstruction network model based on autoregressive deep convolution; Jointly training the differentiable mathematical model and the spectral reconstruction network model to obtain a multispectral data acquisition model and a multispectral data reconstruction model respectively; Based on the multispectral data acquisition model, the spectral information of the target scene is acquired and global spectral encoding is performed to obtain a multispectral compressed image; The multispectral compressed image is input into the multispectral data reconstruction model for decoding and reconstruction, and a multispectral reconstructed image of the target scene is output.

2. The method for airborne spectral video reconstruction based on global spectral coding according to claim 1, characterized in that: The establishment of a differentiable mathematical model based on global spectral encoding includes: According to the sampling range and the number of sampling points of the scene spectral information, a multispectral data linear integral equation based on global spectral coding is established. The multispectral data linear integral equation is expressed as: Where X(λ) is the spectral image of the scene at wavelength λ, T(λ) is the spectral response of the coding filter at wavelength λ, Y is the spectral compression image collected under global spectral coding, and λ s and λ e All are sampling bands of the spectrum; The parameters and variables in the multispectral data linear integral equation are discretized to obtain a differentiable mathematical model based on global spectral encoding: In the formula, X n is the nth spectral image obtained after discretization, T n is the spectral response encoding matrix of the nth spectral image, n is the spectral image sequence number, and N is the total number of spectral images.

3. The method for airborne spectral video reconstruction based on global spectral coding according to claim 1, characterized in that: The establishment of a spectrum reconstruction network model based on autoregressive deep convolution includes: A deep learning network framework is used to build a spectral reconstruction network model based on autoregressive deep convolution. The spectral reconstruction network model includes a feature extraction module, a feature reconstruction module, a data fusion module and an output module. The feature extraction module is used to extract features from the input multi-spectral compression image to obtain feature data of the multi-spectral compression image; The feature reconstruction module is used to perform feature reconstruction on the feature data to obtain feature reconstruction data of the multi-spectral compression image; The data fusion module is used to fuse the feature reconstruction data of the previous frame of multi-spectral compression image with the feature data of the original multi-spectral compression image; The output module is used to output the multispectral data reconstruction result of the multispectral compression image.

4. The method for airborne spectral video reconstruction based on global spectral coding according to claim 3, characterized in that: The deep learning network framework includes PyTorch and / or Tensorflow.

5. The method for airborne spectral video reconstruction based on global spectral coding according to claim 1, characterized in that: The method of jointly training the differentiable mathematical model and the spectral reconstruction network model to obtain a multispectral data acquisition model and a multispectral data reconstruction model respectively comprises: Based on the deep learning end-to-end joint training technology, the differentiable mathematical model and the spectral reconstruction network model are jointly trained with minimizing the spectral reconstruction error as the objective function to obtain the optimal spectral encoding mode and reconstruction network parameters respectively; A band-stop filter group or a single coated filter of a corresponding spectral corresponding curve is constructed based on the spectral encoding mode and placed between camera sensors to obtain a multi-spectral data acquisition model; A multispectral data reconstruction model is constructed based on the reconstruction network parameters.

6. The method for airborne spectral video reconstruction based on global spectral coding according to claim 5, characterized in that: The acquiring of the target scene spectral information based on the multispectral data acquisition model and performing global spectral encoding to obtain a multispectral compressed image comprises: Collect multispectral image information of the target scene at different wavelengths based on the multispectral data acquisition model; A band-stop filter group or a single coated filter is used to perform global spectral encoding on the multispectral image information to obtain a multispectral compressed image.

7. An airborne spectral video reconstruction system based on global spectral coding, characterized in that: include: Model building module, used to build a differentiable mathematical model based on global spectral encoding and a spectral reconstruction network model based on autoregressive deep convolution; A model training module, used for jointly training the differentiable mathematical model and the spectral reconstruction network model to obtain a multispectral data acquisition model and a multispectral data reconstruction model respectively; A data acquisition module, used to acquire the spectral information of the target scene based on the multispectral data acquisition model and perform global spectral encoding to obtain a multispectral compressed image; The data reconstruction module is used to input the multispectral compressed image into the multispectral data reconstruction model for decoding and reconstruction, and output a multispectral reconstructed image of the target scene.

8. The airborne spectral video reconstruction system based on global spectral coding according to claim 7, characterized in that: The data acquisition module also includes: A spectral data acquisition module, used for acquiring multispectral image information of a target scene at different wavelengths based on the multispectral data acquisition model; The spectral encoding module is used to perform global spectral encoding on the multi-spectral image information by using a band-stop filter group or a single coated filter to obtain a multi-spectral compressed image.

9. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the airborne spectral video reconstruction method based on global spectral coding as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the airborne spectral video reconstruction method based on global spectral coding as described in any one of claims 1 to 7.

Images