A multispectral image reconstruction method and apparatus

Abstract

The application belongs to the technical field of multispectral imaging and computational spectral reconstruction, and particularly relates to a multispectral image reconstruction method and device. The method comprises: obtaining multiple RAW original images by switching a narrow-band filter assembly, and constructing a joint observation vector; constructing a joint forward matrix based on a mapping relationship from a target waveband space to a joint observation space; establishing an expression of a joint forward model; constructing a unified joint optimization objective of the model; solving a target spectral vector and a bias vector in the joint optimization objective by using an alternating optimization method until convergence is reached; fixing the bias vector when solving the target spectral vector, and fixing the target spectral vector when solving the bias vector; after the alternating solving is completed, constructing an additional loss function by using a reference whiteboard continuity constraint and a consistency constraint, optimizing and adjusting the target spectral vector, and outputting a reconstructed target spectral vector corresponding to each target waveband. The method can improve the multi-waveband recovery quality.

Description

Technical Field

[0001] This application belongs to the field of multispectral imaging and computational spectral reconstruction technology, specifically relating to a multispectral image reconstruction method and apparatus. Background Technology

[0002] Multispectral imaging technology can acquire spectral response information of a target scene across multiple discrete bands, and has significant application value in fields such as agricultural monitoring, vegetation index calculation, environmental remote sensing, precision agriculture, and ecological analysis. Existing multispectral and hyperspectral imaging technologies can be mainly divided into two categories: dedicated multispectral imaging systems and low-cost multiband reconstruction systems.

[0003] Dedicated multispectral or hyperspectral imaging systems typically acquire multiple narrow-band image data in the following ways: (1) Multi-camera collaborative imaging: Multiple independent cameras are configured with different band filters to acquire images of different bands at the same time, but strict optical alignment and geometric registration are required, and the system complexity and cost are high; (2) Filter wheel switching structure: Multi-band acquisition is achieved by mechanically switching different filters. This type of system can obtain high spectral resolution and reconstruction accuracy, but it usually has problems such as high hardware cost, complex system structure, large size and high deployment threshold; (3) Mosaic filter array method: A pixel-level filter array is integrated on the surface of the image sensor to achieve acquisition of multi-band information in a single exposure, but there are problems such as spatial resolution loss, spectral aliasing and assembly error; (4) Prism or grating beam splitting structure: The incident light is separated into multiple spectral channels by beam splitting elements and received by multiple detectors, which requires complex optical design and precise system integration.

[0004] While the aforementioned dedicated systems can achieve high spectral resolution and reconstruction accuracy, they generally suffer from high hardware costs, complex system structures, large size, and high deployment barriers, making it difficult to meet the application requirements of low cost, lightweight, and edge deployment.

[0005] To reduce system costs, low-cost multi-band imaging schemes based on ordinary color image sensors have emerged in recent years. These schemes typically use ordinary RGB image sensors in conjunction with external narrowband filter components to obtain a limited number of narrowband observation data in a single acquisition. Then, methods such as matrix inversion, linear regression, or empirical models are used to estimate the target band and further calculate vegetation indices.

[0006] Most existing low-cost solutions are designed for the recovery of a small number of bands under a single set of filter components, and their goal is usually to recover about three target bands. Related technologies usually adopt a fixed filter structure, do not involve the switching of multiple sets of filter components, and rarely involve joint modeling and unified optimization of multiple acquisition results. For low-cost multi-band imaging systems based on ordinary RGB image sensors and external filter components, existing technologies mainly have the following problems: (1) There is a lack of joint modeling mechanism for multiple filter acquisition results. Existing technologies usually solve different filter acquisition results independently, that is, the multi-band recovery problem is broken down into multiple local low-dimensional inverse problem. This processing method is difficult to make full use of the joint constraint relationship between multiple observations and is more sensitive to observation noise, out-of-band light leakage, scale drift and system error. In practical applications, it is easy to lead to the following problems: inconsistent scale between different filter acquisition results, poor spectral continuity of the same ground object in different bands, and distortion of the response relationship between bands. (2) The stability of traditional inverse methods is insufficient. Existing technologies generally use direct matrix inversion, linear regression or local empirical models to recover target bands. However, in actual low-cost systems, due to the obvious response coupling between RGB channels, the out-of-band light leakage of the filter components, and the scale mismatch of the forward response matrix, the system usually has a certain degree of ill-conditioning. In the absence of physical non-negativity constraints, spectral continuity constraints, and stabilization constraints, problems such as abnormal negative values ​​in the target band, amplification of noise in the weak response band, and local zero-value collapse are likely to occur. Especially on typical targets such as green plants and soil, some visible light bands may be abnormally suppressed, while some near-infrared bands may be abnormally raised, resulting in the recovery results deviating from the spectral characteristics of the real ground objects. (3) The system background bias and dynamic range compression problems are difficult to express effectively. Existing technologies usually use a fixed background subtraction method to deal with system background errors. However, the background error sources in actual systems are diverse, including: dark bias of the image sensor, light leakage background of the filter components, low-frequency background terms of the system, and scale mismatch of the joint forward model. The above errors usually have cross-channel differences and dynamic change characteristics, which are difficult to express accurately by a fixed background subtraction method. Furthermore, background bias can easily lead to distortion of the dynamic range of high and low reflectance regions and absolute scale drift between bands, thereby affecting the accuracy of subsequent reflectance recovery results.

[0007] In summary, there is an urgent need for a joint multi-band reconstruction method suitable for low-cost multiple-filter acquisition systems to improve the quality of multi-band reconstruction and thus meet the practical needs of low-cost edge remote sensing systems in applications such as agricultural monitoring, vegetation analysis, and ecological environment assessment. Summary of the Invention

[0008] To improve the quality of multi-band image reconstruction, firstly, embodiments of this application provide a multispectral image reconstruction method, the method comprising: Using the same Bayer color image sensor, while maintaining consistent exposure parameters, gain parameters, sensor operating mode, and imaging geometry, the narrowband filter components are switched electronically to sequentially acquire multiple narrowband response observation groups, resulting in multiple corresponding RAW raw images. Different narrowband response observation groups correspond to different target band sets. Using each Bayer primitive unit as the processing basis, each color response channel is extracted from the original pixel array of the RAW image to construct a joint observation vector. ; Based on the mapping relationship between the target band space and the joint observation space, a joint forward matrix is ​​constructed. Joint forward matrix The behavior color observation channel, the columns of the joint forward matrix are the target bands, and the elements in the joint forward matrix represent the response coefficients of the target bands to the color observation channel; Establish the expression for the joint forward model ,in, For the target spectral vector, The bias vector is the bias vector. It is with the joint observation vector The multidimensional bias parameters corresponding to the dimension are used to characterize the background residual components in the system observations; Weighted constraints based on observation credibility Spectral continuity constraint Stabilization energy constraint And with the non-negativity constraint of spectral physics, a unified joint optimization objective is constructed: S≥0, D A first-order difference operator constructed along the target wavelength direction; For bias vector Initialization, the target spectral vector is solved using an alternating optimization approach. and bias vector Until convergence is achieved, when the target spectral vector is solved. Fixed bias vector When solving for the bias vector Fixed target spectral vector ; After the alternating solution is completed, an additional loss function is constructed by referencing the whiteboard spectral continuity constraint and the scaling consistency constraint, and applied to the target spectral vector. Optimize and adjust the target spectral vector. Substitute the bias vectors into the unified joint optimization objective and solve for the bias vectors again. Output the reconstructed target spectral vector corresponding to each target band.

[0009] In one possible implementation, the process uses each Bayer primitive unit as the basis for processing, extracting each color response channel from the raw pixel array of the RAW image, and constructing a joint observation vector. ,include: Based on each Bayer primitive unit, red response pixels are extracted from the original pixel array of the RAW image to construct the R channel, two green response pixels in the original pixel array of the RAW image are fused by mean to construct the G channel, and blue response pixels are extracted from the original pixel array of the RAW image to construct the B channel. A joint observation vector is constructed based on the R, G, and B channels of multiple RAW original images. .

[0010] In one possible implementation, the bias vector Initialization, the target spectral vector is solved using an alternating optimization approach. and bias vector Until convergence is achieved, when the target spectral vector is solved. Fixed bias vector When solving for the bias vector Fixed target spectral vector ,include: bias vector Initialize the target spectral vector to a zero vector and use an alternating optimization approach. and bias vector Perform joint inversion solutions until convergence is achieved; During the k-th iteration, the current bias vector is fixed. The joint optimization objective is degraded to a value related to the target spectral vector. Sub-optimization problem: ; The target spectral vector is updated using a constrained iterative solution method to obtain the updated target spectral vector. ; With a fixed target spectral vector Under these conditions, the joint optimization objective degenerates into a function about the bias vector. Sub-optimization problem: ; For cross-observation channel difference operators; This is the bias amplitude constraint coefficient; These are the bias continuity constraint coefficients; The bias vector is updated using a constrained iterative solution method to obtain the updated bias vector. The updated bias vector Used for the next round of target spectral vector solving process.

[0011] In one possible implementation, the additional loss function is: + ; To reference the whiteboard spectral continuity constraint weights, To reference the whiteboard ratio consistency constraint weight, For reference, the whiteboard spectral continuity loss function, Use the whiteboard ratio consistency loss function as a reference. ; This represents the target spectral vector corresponding to a reference whiteboard with a reflectance of a%. This represents the target spectral vector corresponding to a reference whiteboard with a reflectance of b%. .

[0012] In one possible implementation, the additional loss function is constructed by referencing the whiteboard spectral continuity constraint and the scaling consistency constraint, and then applied to the target spectral vector. Optimize and adjust the target spectral vector. Substitute the bias vectors into the unified joint optimization objective and solve for the bias vectors again. Output the reconstructed target spectral vectors corresponding to each target band, including: The target spectral vector obtained by applying an additional loss function is then processed. Optimize; The target spectral vector obtained by the solution Substitute into the joint objective function and correct the bias vector in reverse. ; Joint objective function = + + ; .

[0013] In one possible implementation, the method further includes: Obtain the reconstructed target spectral vectors of a reference whiteboard with reflectivity of a% in each target band. The reconstructed target spectral vectors of a reference whiteboard with a reflectance of b% in each target band. ; Establish a linear reflectivity mapping relationship for the target band: ; This represents the reconstructed target spectral vectors in each target band. This represents the reflectivity of the target at the λ band.

[0014] Secondly, embodiments of this application provide a multispectral image reconstruction apparatus, the apparatus comprising: The acquisition module is used to acquire multiple narrowband response observation groups in sequence by electronically switching the narrowband filter components under the condition of keeping the exposure parameters, gain parameters, sensor working mode and imaging geometry consistent, using the same Bayer color image sensor, and obtaining multiple corresponding RAW raw images. Different narrowband response observation groups correspond to different target band sets. The joint observation vector construction module is used to extract each color response channel from the raw pixel array of the RAW image, based on each Bayer primitive unit, and construct a joint observation vector. ; The joint forward matrix construction module is used to construct the joint forward matrix based on the mapping relationship between the target band space and the joint observation space. Joint forward matrix The behavior color observation channel, the columns of the joint forward matrix are the target bands, and the elements in the joint forward matrix represent the response coefficients of the target bands to the color observation channel; The joint forward model building module is used to construct the expression of the joint forward model. ,in, For the target spectral vector, The bias vector is the bias vector. It is with the joint observation vector The multidimensional bias parameters corresponding to the dimension are used to characterize the background residual components in the system observations; A unified joint optimization objective construction module is used to impose weighted constraints based on observation confidence. Spectral continuity constraint Stabilization energy constraint And with the non-negativity constraint of spectral physics, a unified joint optimization objective is constructed: S≥0, D A first-order difference operator constructed along the target wavelength direction; The alternating solver module is used to solve the bias vector. Initialization, the target spectral vector is solved using an alternating optimization approach. and bias vector Until convergence is achieved, when the target spectral vector is solved. Fixed bias vector When solving for the bias vector Fixed target spectral vector ; The whiteboard constraint module is used to construct an additional loss function for the target spectral vector after the alternating solution is completed, by referring to the whiteboard spectral continuity constraint and scaling consistency constraint. Optimize and adjust the target spectral vector. Substitute the bias vectors into the unified joint optimization objective and solve for the bias vectors again. Output the reconstructed target spectral vector corresponding to each target band.

[0015] In one possible implementation, the joint observation vector construction module is specifically used for: Based on each Bayer primitive unit, red response pixels are extracted from the original pixel array of the RAW image to construct the R channel, two green response pixels in the original pixel array of the RAW image are fused by mean to construct the G channel, and blue response pixels are extracted from the original pixel array of the RAW image to construct the B channel. A joint observation vector is constructed based on the R, G, and B channels of multiple RAW original images. .

[0016] In one possible implementation, the device further includes a reflectivity calculation module for: Obtain the reconstructed target spectral vectors of a reference whiteboard with reflectivity of a% in each target band. The reconstructed target spectral vectors of a reference whiteboard with a reflectance of b% in each target band. ; Establish a linear reflectivity mapping relationship for the target band: ; This represents the reconstructed target spectral vectors in each target band. This represents the reflectivity of the target at the λ band.

[0017] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the aforementioned multispectral image reconstruction methods.

[0018] This application provides a multispectral image reconstruction method that uses the same Bayer color image sensor. While maintaining consistent exposure parameters, gain parameters, sensor operating mode, and imaging geometry, it electronically switches narrowband filter components to sequentially acquire multiple narrowband response observation groups, resulting in multiple corresponding RAW raw images. Different narrowband response observation groups correspond to different target band sets. Using each Bayer primitive unit as the processing basis, it extracts each color response channel from the original pixel array of the RAW raw image to construct a joint observation vector. Based on the mapping relationship between the target band space and the joint observation space, a joint forward matrix is ​​constructed. Joint forward matrix The behavior color observation channel is defined, with the columns of the joint forward matrix representing the target band, and the elements in the joint forward matrix representing the response coefficients of the target band to the color observation channel; the expression for the joint forward model is established. ,in, For the target spectral vector, The bias vector is the bias vector. It is with the joint observation vector The multidimensional bias parameters corresponding to the dimensions are used to characterize the background residual components in the system observations; weighted constraints are applied based on observation confidence. Spectral continuity constraint Stabilization energy constraint And with the non-negativity constraint of spectral physics, a unified joint optimization objective is constructed: S≥0, D A first-order difference operator constructed along the target wavelength direction; for the bias vector Initialization, the target spectral vector is solved using an alternating optimization approach. and bias vector Until convergence is achieved, when the target spectral vector is solved. Fixed bias vector When solving for the bias vector Fixed target spectral vector After the alternating solution is completed, an additional loss function is constructed by referencing the whiteboard spectral continuity constraint and the scaling consistency constraint, and applied to the target spectral vector. Optimize and adjust the target spectral vector. Substitute the bias vectors into the unified joint optimization objective and solve for the bias vectors again. The method outputs the reconstructed target spectral vector corresponding to each target band. This method can improve the quality of multi-band reconstruction. Attached Figure Description

[0019] Figure 1 A schematic flowchart of a multispectral image reconstruction method provided in an embodiment of this application; Figure 2 A schematic diagram for constructing joint observation vectors; Figure 3 A schematic diagram for constructing the joint forward matrix; Figure 4 A schematic diagram of the alternating optimization method; Figure 5 RGB images captured by a color sensor; Figure 6 for Figure 5The reconstructed target spectral response corresponding to each target band obtained by inversion after using the multispectral image reconstruction method of this application; Figure 7 This is a vegetation index map; Figure 8 This is another schematic flowchart of the multispectral image reconstruction method provided in the embodiments of this application. Detailed Implementation

[0020] Firstly, see [the following] Figure 1 This application provides a multispectral image reconstruction method, the method comprising: S101 uses the same Bayer color image sensor. Under the condition of keeping the exposure parameters, gain parameters, sensor working mode and imaging geometry consistent, it switches the narrowband filter components by electronic control to acquire multiple narrowband response observation groups in sequence, and obtains multiple corresponding RAW raw images.

[0021] In this application, different narrowband response observation groups correspond to different target band sets. The same Bayer color image sensor is used as a unified imaging detector. While maintaining consistent exposure time, analog gain, digital gain, sensor operating mode, and imaging geometry, multiple narrowband response observations are sequentially acquired using an externally mounted switchable narrowband filter assembly. The switchable narrowband filter assembly employs a multi-position narrowband filter switching mechanism, switching filters electronically. Different narrowband filters correspond to different target band sets, forming multiple narrowband response observation groups.

[0022] The three sets of narrowband response observations correspond to the following target band sets: the first observation group corresponds to the 450nm, 650nm, and 800nm ​​bands; the second observation group corresponds to the 530nm, 670nm, and 840nm bands; and the third observation group corresponds to the 555nm, 730nm, and 960nm bands. Each observation group was acquired using the same imaging optical path, the same image sensor, and consistent acquisition parameters, ensuring a unified radiative response basis and joint modeling conditions across different observation groups. Since the system uses a Bayer color image sensor for raw response acquisition, each observation outputs unprocessed RAW raw data to preserve the original spectral response information of the sensor as much as possible and reduce subsequent interpolation errors. After completing the acquisition of multiple sets of narrowband responses, multiple corresponding RAW raw images were obtained, denoted as RAW1, RAW2, and RAW3, where RAW1 corresponds to the first narrowband response observation group, RAW2 corresponds to the second narrowband response observation group, and RAW3 corresponds to the third narrowband response observation group.

[0023] The core objective of this step is to provide a unified observational basis for subsequent joint forward matrix modeling, intrinsic estimation of bias terms, and joint optimization solution of multiple narrowband response observations under a unified imaging link.

[0024] S102, using each Bayer primitive unit as the processing basis, extracts each color response channel from the original pixel array of the RAW image and constructs a joint observation vector.

[0025] Specifically, this may include: Based on each Bayer primitive unit, red response pixels are extracted from the original pixel array of the RAW image to construct the R channel, two green response pixels in the original pixel array of the RAW image are fused by mean to construct the G channel, and blue response pixels are extracted from the original pixel array of the RAW image to construct the B channel. A joint observation vector is constructed based on the R, G, and B channels of multiple RAW original images. .

[0026] Each group of RAW images is analyzed separately. Since the input data is the raw response data output from a Bayer color image sensor, this scheme does not perform traditional demosaic interpolation. Instead, it directly extracts each color response channel from the original pixel array according to the Bayer Pattern arrangement rules. Using each Bayer primitive unit as the processing basis, different color response positions in the original pixel array are extracted and recombined to obtain spatially registered raw color response channels. Specifically: red response pixels are extracted to construct the R channel, green response pixels are combined to construct the G channel, and blue response pixels are extracted to construct the B channel. The green channel preferably uses two green response pixels from the same Bayer primitive unit for mean fusion to reduce single-pixel noise fluctuations and improve green response stability. In this way, corresponding raw color response observation channels can be constructed from each RAW image without performing cross-pixel spatial interpolation. Compared to traditional demosaicing, this scheme directly constructs channels based on the raw Bayer response, offering the following advantages: 1. It avoids spatial artifacts and edge errors caused by demosaicing interpolation; 2. It avoids additional spectral aliasing between color channels; 3. It preserves the original narrowband response relationship of the image sensor. After RAW data parsing, each observation group forms a corresponding color response channel: First observation group: R1, G1, B1; Second observation group: R2, G2, B2; Third observation group: R3, G3, B3. Subsequently, multiple color response channels are jointly organized according to a preset order to construct a unified joint observation vector. .in, The nine-dimensional observation vector in the joint observation space corresponds to a color response observation channel for each dimension, and different observation groups share the same spatial coordinates and the same imaging link.

[0027] By constructing the joint observation method described above, this scheme no longer treats multiple sets of observations as independent local inverse problems, but instead maps them uniformly to the same joint observation space, providing a unified observation basis for subsequent joint forward matrix modeling, endogenous estimation of bias terms, and joint optimization solution of multiple bands.

[0028] See Figure 2 , is the joint observation vector The schematic diagram shows that this application acquires three sets of RAW images (RAW1, RAW2, RAW3) through three narrowband filters. Different filter sets correspond to different target bands, but each set is in Bayer RAW format (containing one red response pixel, one blue response pixel, and two green response pixels). Each set of RAW images is arranged in Bayer to extract compact R, G, and B channels. The nine channels obtained from the three acquisitions are stacked in a fixed order to form a 9×1 joint observation vector Y.

[0029] S103, Based on the mapping relationship between the target band space and the joint observation space, construct the joint forward matrix.

[0030] Joint forward matrix The behavior of the color observation channel is represented by the columns of the joint forward matrix, which are the target bands. The elements in the joint forward matrix represent the response coefficients of the target bands to the color observation channel.

[0031] After constructing the joint observation vector, this scheme further establishes a joint forward matrix to describe the mapping relationship between the joint observation space and the target band space. Define the target spectral components corresponding to the nine target bands as follows: S 450 This represents the target response component of the target scene in the 450nm band; S 530 This represents the target response component of the target scene in the 530nm band; the definition method is the same for other bands. Target spectral vector. Used to characterize the joint spectral response of a target scene across multiple discrete target bands.

[0032] Based on the spectral transmission characteristics of different narrowband response observation groups and the color response characteristics of image sensors, a joint forward matrix is ​​constructed. This describes the mapping relationship between the target band space and the joint observation space. Joint forward matrix. It has an approximately blocky sparse structure, and its construction principle is: different narrowband response observation groups mainly correspond to different target band sets, and form corresponding response sub-matrices in the joint matrix.

[0033] In this scheme, the first observation group mainly corresponds to the 450nm, 650nm, and 800nm ​​band responses; the second observation group mainly corresponds to the 530nm, 670nm, and 840nm band responses; and the third observation group mainly corresponds to the 555nm, 730nm, and 960nm band responses. Therefore, the joint forward matrix... It can be represented as a joint mapping matrix composed of multiple response sub-matrices. Each observation group forms the main response region at its corresponding target band position, while forming a weak response or low coupling response at non-corresponding band positions.

[0034] Using the above method, the forward matrix is ​​combined. It can uniformly describe: the spectral transmission characteristics of narrowband filter components, the color response characteristics of Bayer color image sensors, the joint coupling relationship between multiple sets of observations, and the forward mapping relationship between the target band and the observation channel.

[0035] See Figure 3 This is a schematic diagram of the joint forward matrix construction and the joint observation vector. The nine channels formed constitute the rows of the joint forward matrix, and the nine target bands constitute the columns of the joint forward matrix. Each element a of the joint forward matrix... i,j The response coefficient of target band j to observation channel i can be obtained by measuring the response of each filter + sensor combination at each target wavelength using a monochromator (or tunable light source) + spectrometer. Joint forward matrix This approach, which pre-establishes a response matrix for the system calibration phase, has a clear engineering basis. Specifically, after the equipment is assembled, a set of standard samples with known reflectivity characteristics (such as standard reflectors, standard color charts, or calibration samples with known spectral reflectivity) can be used to collect data under different target wavelength conditions, establishing the response relationship between the target wavelength and each color observation channel. Based on the known spectral reflectivity of the calibration samples and the corresponding RAW observation data, methods such as least squares fitting, linear regression, or pseudo-inverse solving can be used to estimate the response coefficients of each target wavelength to each observation channel, thereby obtaining the joint forward matrix. . Obtained These parameters serve as the system calibration parameters for the equipment and are consistently used in subsequent actual measurements, being incorporated into the joint optimization model for target spectrum inversion. Without changes to the equipment hardware (camera, filter components, optical system), recalibration is unnecessary for each acquisition. Recalibration should only be performed when equipment is replaced, filter components are replaced, or the optical system undergoes significant changes. Therefore, The fixed response matrix obtained from the offline calibration of the system, rather than the unknown parameters estimated in real time during each reconstruction, makes this implementation method clearly feasible in engineering.

[0036] S104, Establish the expression for the joint forward model. .

[0037] in, For the target spectral vector, The bias vector is the bias vector. It is with the joint observation vector The multidimensional bias parameter corresponding to the dimension is used to characterize the background residual component in the system observation.

[0038] Traditional multi-band unmixing methods typically employ fixed dark field subtraction, fixed background compensation, or preset basis correction to uniformly correct the background of the input observation channels. However, in practical low-cost multi-band imaging systems, factors such as out-of-band light leakage from filter components, dark bias of image sensors, low-frequency background in the system, and scale mismatch in the forward model often prevent the system observations from meeting the ideal zero-intercept linear response model. Directly using fixed background subtraction can lead to the following problems: 1. Inaccurate description of non-uniform bias differences between different observation channels; 2. Amplification of observation noise in weak response bands; 3. Disruption of uniform optimization consistency in the joint observation space; 4. Difficulty in dynamically adapting to background changes in different bands; 5. Potential for compression of the reference whiteboard's dynamic range and scaling distortion. Therefore, this scheme does not perform fixed background correction on the joint observation vector at the input observation end, but instead explicitly introduces a bias vector within the joint forward model. b 9, and use it as the variable to be estimated in the joint optimization process.

[0039] Background residual components in the system observations include: background response caused by out-of-band leakage from the narrowband filter assembly, dark bias and fixed mode offset of the image sensor, residual components of the forward matrix scaling error in the observation space, low-frequency background terms of the system, and slowly varying global offset terms between channels. The bias vector in this scheme... b 9 is the multidimensional bias parameter corresponding to the dimension of the joint observation vector, which allows different observation channels to have different bias responses, thereby improving the joint model's ability to express actual system errors.

[0040] S105, using observation confidence weighting constraints Spectral continuity constraint Stabilization energy constraint And with the non-negativity constraint of spectral physics, a unified joint optimization objective is constructed: , S≥0.

[0041] After establishing the joint forward model, this scheme further constructs a multi-constraint joint optimization model based on the joint observation space, with the target spectral vector as the target. With joint bias vector As joint variables to be estimated, stable inversion of the nine-band target response is achieved.

[0042] Due to factors such as out-of-band light leakage, color response coupling, system scale mismatch, and observation noise among multiple narrowband response observations, the joint forward model typically exhibits a certain degree of ill-posedness. Directly employing matrix inversion or locally independent solution methods can easily lead to amplification of noise in weak response bands, abnormal peaks and spectral oscillations in multi-band results, collapse of local negative and zero values, and scale drift between different observation groups. Therefore, this scheme further introduces the following constraints on the joint forward model: observation confidence weighting, spectral continuity constraints, stabilization energy constraints, and spectral physical nonnegativity constraints.

[0043] Nonnegativity constraints are used to ensure that the response of the target band satisfies the physical nonnegativity of the actual spectral reflection or radiation response, thereby avoiding large-area negative value anomalies that occur in the traditional inverse solution process.

[0044] Since different observation channels vary in terms of light leakage, noise level, dynamic range, and response stability, this scheme introduces an observation reliability weight matrix. Different observation channels are weighted differently. In one embodiment, the observation weights can be constructed based on at least one of the following: reference whiteboard consistency error, the proportion of negative values ​​in a channel, the proportion of zero-value collapse in a channel, joint residual statistical characteristics, and channel noise estimation results. Observation channels with higher reliability have higher weights, while observation channels with poor stability or large errors have lower weights. This approach reduces the bias of outlier observations on the joint inversion results, improving the stability and robustness of the joint optimization process.

[0045] Since natural features typically exhibit continuously varying spectral response characteristics between adjacent bands, this scheme introduces a first-order difference constraint along the wavelength direction to suppress anomalous spikes and high-spectral oscillations in multi-band results. In this embodiment, the difference operator... D This is used to describe the variation relationship between adjacent target bands, thereby constraining the continuity of the target spectral vector in the wavelength dimension. Through this constraint, local abnormal peaks can be effectively suppressed, the continuity of multi-band spectral shapes can be improved, spectral oscillations caused by out-of-band light leakage can be reduced, and the physical rationality of the vegetation and whiteboard spectral shapes can be improved.

[0046] Furthermore, this scheme limits the overall energy amplitude of the target spectral vector through stabilization energy constraints to reduce the instability caused by ill-conditioned matrices during joint inversion. This constraint can improve the stability of joint solutions, reduce noise amplification in weak response bands, enhance the convergence of joint optimization, and suppress abnormally large solutions.

[0047] By combining the above-mentioned observation confidence weighting, spectral continuity constraints, stabilization energy constraints, and physical nonnegativity constraints in a joint modeling approach, this scheme can integrate multiple observation groups into the same joint optimization framework for stable inversion, thereby improving the spectral continuity, scale consistency, and physical interpretability of the nine-band results.

[0048] S106, for the bias vector Initialization, the target spectral vector is solved using an alternating optimization approach. and bias vector Until convergence is achieved, when the target spectral vector is solved. Fixed bias vector When solving for the bias vector Fixed target spectral direction .

[0049] Specifically, this can include: adjusting the bias vector Initialize the target spectral vector to a zero vector and use an alternating optimization approach. and bias vector Perform joint inversion solutions until convergence is achieved; During the k-th iteration, the current bias vector is fixed. The joint optimization objective is degraded to a value related to the target spectral vector. Sub-optimization problem: ; The target spectral vector is updated using a constrained iterative solution method to obtain the updated target spectral vector. ; With a fixed target spectral vector Under these conditions, the joint optimization objective degenerates into a function about the bias vector. Sub-optimization problem: ; The bias vector is updated using a constrained iterative solution method to obtain the updated bias vector. The updated bias vector Used for the next round of target spectral vector solving process.

[0050] After establishing the joint optimization model, this invention further employs an alternating optimization strategy for the target spectral vector. With joint bias vector Perform a joint inversion to solve the problem.

[0051] Since the target spectral component and the background bias component act simultaneously on the joint observation vector The two are strongly coupled in the observation space. If all unknown variables are solved jointly at once, problems such as background bias absorbing the target spectral energy, misfitting the target spectrum with background information, instability in the optimization process, and difficulty in convergence under ill-conditioned conditions are likely to occur.

[0052] Therefore, this invention decomposes the joint optimization problem into a target spectrum estimation subproblem and a background bias estimation subproblem. By fixing one set of variables and optimizing another set of variables, the problem is iteratively updated alternately, thereby gradually approaching the optimal solution of the joint optimization objective and improving the stability, robustness and physical consistency of the inversion process.

[0053] (1) Initialization phase In a preferred embodiment, a joint optimization objective with observation confidence weighting and spectral continuity constraints is first constructed: ; Because the objective function simultaneously includes and Since there are two sets of unknown variables, an alternating optimization approach is used to solve the problem. In the initialization phase, the joint bias vector is first initialized. Initialization; Joint forward matrix The process includes: setting spectral continuity constraint parameters; setting stabilization constraint parameters; setting bias constraint parameters; setting the maximum number of iterations and convergence threshold. The joint bias vector is also included. It can be initialized as a zero vector, a background estimate based on the joint observation mean, or a statistical background estimate for dark areas.

[0054] (2) Fixed bias term Solve for the target spectral vector

[0055] During the k-th iteration, when the current joint bias vector is fixed... At that time, the joint optimization objective degenerates into a function of the target spectral vector. Sub-optimization problem: And satisfy the physical nonnegativity constraint: .

[0056] The first term measures the matching error between the current spectral estimate and the joint observation results; the second term constrains the target spectrum to remain continuously varied between adjacent wavelengths; and the third term suppresses the energy divergence problem generated during ill-conditioned inversion and improves the stability of the solution.

[0057] In one embodiment, the target spectral vector can be updated using the projection gradient method, the accelerated gradient method FISTA, the nonnegative least squares method NNLS, the conjugate gradient method, or other constrained iterative optimization methods to obtain the updated target spectral vector. .

[0058] Furthermore, after each iteration update, a non-negative projection constraint is applied to the target spectral vector. This ensures that the inversion result satisfies the physical property of non-negative spectral reflectance in reality, and avoids non-physically negative solutions during the joint optimization process.

[0059] The core objective of this stage is to obtain the joint observation vector under the current background bias estimation conditions. The most consistent target spectral estimation result that simultaneously satisfies spectral continuity and physical constraints is used as the input for the next round of joint bias optimization.

[0060] (3) Fixed target spectral vector Update joint bias vector

[0061] To complete the current target spectral vector After the update, the spectral estimation result is fixed, and the joint optimization objective is transformed into an objective about the joint bias vector. This is a sub-optimization problem. Given a fixed target spectral vector... Under the given conditions, the joint optimization objective degenerates into a function of the joint bias vector. The optimization problem. In a preferred embodiment, the optimization objective of the bias term can be expressed as: ; in: For cross-observation channel difference operators; This is the bias amplitude constraint coefficient; This represents the bias continuity constraint coefficient.

[0062] To avoid abnormal amplification, cross-channel oscillation, and noise overfitting during bias estimation, this invention further applies the following constraints to the joint bias vector: bias amplitude constraint, limiting the bias compensation amount to a reasonable physical range; and bias continuity constraint, ensuring smooth bias changes between adjacent channels. In one embodiment, the joint bias vector can be updated using gradient descent, conjugate gradient, least squares optimization, proximal gradient optimization, or other constrained iterative solution methods. ,get: .

[0063] The updated joint bias vector is used in the next round of target spectrum solution process, thereby gradually approaching the optimal solution of the joint optimization objective through alternating optimization, and realizing the synchronous separation and joint estimation of target spectrum and background bias.

[0064] (4) Alternating Iteration Convergence Complete the target spectral vector Update and joint bias vector After the update, repeat the above alternating joint estimation process.

[0065] The process continues until at least one of the following conditions is met: the maximum number of iterations is reached; the joint residual converges; the spectral change is lower than a set threshold; or the bias update is lower than a set threshold.

[0066] Through the above alternating optimization process, the target spectral vector With joint bias vector In practice, alternating condition optimization is performed under the same joint optimization objective function, and eventually converges synchronously to a stable solution that satisfies joint observation consistency, spectral continuity, reasonable background bias, and multi-band scale consistency.

[0067] Compared with traditional one-time inverse kinematics methods, this invention can more effectively distinguish between target spectral components and background bias components, and achieve synchronous separation and joint estimation of target spectrum and background response, thereby significantly improving the stability, physical consistency and reliability of the nine-band joint inversion results and subsequent vegetation index calculation.

[0068] See Figure 4 This is a schematic diagram of the alternating optimization process, with the input observation vector. Joint forward matrix Observation weight matrix Maximum number of iterations Convergence threshold, hyperparameters: , , After (representing the bias constraint coefficient), you can choose to initialize the target spectral vector or bias vector to a zero vector and calculate the current residual. Then, the alternating optimization process begins: during the k-th iteration, the current bias vector is fixed. ,optimization This is obtained through nonnegative least squares and smoothing regularization. ;fixed use optimization The closed-form solution obtained by adding Tikhonov regularization (where I is the identity matrix) is obtained. After each iteration, the function value is calculated, and convergence is determined based on the convergence threshold or the number of iterations. If convergence is achieved, the current function value is... and If the result is the optimal estimate, then continue iterative execution; otherwise, proceed with the iterative process. and Alternating optimization process.

[0069] S107, after the alternating solution is completed, an additional loss function is constructed by referencing the whiteboard spectral continuity constraint and the scaling consistency constraint, for the target spectral vector. Optimize and adjust the target spectral vector. Substitute the bias vectors into the unified joint optimization objective and solve for the bias vectors again. Output the reconstructed target spectral vector corresponding to each target band.

[0070] In one example, the additional loss function is + ; To reference the whiteboard spectral continuity constraint weights, To reference the whiteboard ratio consistency constraint weight, For reference, the whiteboard spectral continuity loss function, Use the whiteboard ratio consistency loss function as a reference. ; This represents the target spectral vector corresponding to a reference whiteboard with a reflectance of a%. This represents the target spectral vector corresponding to a reference whiteboard with a reflectance of b%. .

[0071] The target spectral vector obtained by applying an additional loss function is then processed. Optimize; The target spectral vector obtained by the solution Substitute into the joint objective function and correct the bias vector in reverse. ; Joint objective function = + + ; .

[0072] In traditional multi-band reconstruction methods, the reference whiteboard is typically used only for reflectance normalization or scaling correction in the post-processing stage, its role mainly limited to output calibration. However, under conditions such as out-of-band light leakage, background bias, dynamic range compression, and scale drift, simply relying on post-processing whiteboard normalization is insufficient to suppress joint inversion errors from within the model. Therefore, this scheme extends the reference whiteboard from a traditional post-processing calibration tool to a source of physical constraints within the joint optimization model, using reference reflector information to inversely constrain the joint forward model, bias estimation, and target spectral recovery process.

[0073] In this implementation, multiple reference diffuse reflection regions with different reflectivity levels are set within the imaging scene, preferably including a 30% (a) reference white plate and a 90% (b) reference white plate. These reference white plates participate in imaging simultaneously with the target area and are under the same illumination environment, thereby providing a stable and reliable physical reference for the joint inversion process.

[0074] Furthermore, the joint reconstruction results of the corresponding target bands are extracted from each reference whiteboard area, and a joint constraint model based on the reference reflector information is constructed.

[0075] (1) Reference whiteboard spectral continuity constraint Since a uniform diffuse reflectance reference whiteboard typically exhibits continuously varying spectral response characteristics between adjacent target bands, this scheme applies a spectral continuity constraint to the target spectral results corresponding to the reference whiteboard region. The reference whiteboard constraint term is expressed as follows: ; This represents the target spectral vector corresponding to the 30% reference whiteboard; This represents the target spectral vector corresponding to the 90% reference whiteboard.

[0076] (2) Consistency constraint of multi-level reference whiteboard ratio Since reference whiteboards of different reflectivity levels should maintain a stable proportional relationship under the same illumination conditions, this scheme further introduces proportional consistency constraints based on multi-level reference reflectors.

[0077] In a preferred embodiment, the theoretical reflectivity ratio between the 90% reference white panel and the 30% reference white panel is close to 3.0, therefore the following proportional consistency constraint is established: .

[0078] Furthermore, the scaling consistency constraint preferably adopts a penalized soft constraint form rather than a strict hard constraint form, thereby allowing the system to maintain a certain degree of freedom under actual observation noise, light leakage background, and non-ideal sensor response conditions. Through the above scaling consistency constraint, the absolute scale drift in the joint inversion process can be effectively suppressed, the dynamic range compression caused by background bias can be suppressed, and the scaling consistency between different target bands can be improved.

[0079] (3) Reference whiteboard-driven model internal constraint propagation Furthermore, the reference whiteboard constraint in this scheme not only affects the final output but also serves as an additional loss term embedded in the joint optimization objective function, continuously participating in model parameter updates during the joint inversion process. Specifically, the reference whiteboard continuity constraint and the scaling consistency constraint are jointly introduced into the joint objective function.

[0080] During the joint iterative optimization process, the aforementioned reference whiteboard constraint participates in the target spectral vector through the gradient propagation mechanism. and joint bias vector Synchronous optimization.

[0081] When scaling compression or scale shift occurs between reference whiteboards, the scaling consistency error increases and, through the joint objective function, inversely affects the spectral reconstruction process, thereby driving the system to correct the overall reflectivity scale. When local anomalous fluctuations occur in the spectrum reconstructed from the reference whiteboards, the continuity constraint error increases, thus strengthening the constraint on the smoothness between adjacent target bands and suppressing local spectral oscillations and anomalous spikes. Furthermore, during the joint optimization process, the reference whiteboard constraints and observation residual terms jointly participate in parameter updates, enabling the model to satisfy both observation consistency and the physical consistency requirements corresponding to the reference reflector.

[0082] Therefore, in this scheme, the reference whiteboard is no longer merely used as a scaling benchmark in the post-processing stage, but rather as a physical reference anchor point in the joint model, participating in the entire multi-band joint optimization process. Through the aforementioned constraint propagation mechanism, the following can be further achieved: unified absolute scale across multiple bands, suppression of background bias, restoration of the whiteboard's dynamic range, enhanced spectral continuity, and improved physical consistency of multi-band results. This improves the stability of the nine-band joint reconstruction results and the reliability of subsequent vegetation index calculations.

[0083] This application provides a multispectral image reconstruction method that uses the same Bayer color image sensor. While maintaining consistent exposure parameters, gain parameters, sensor operating mode, and imaging geometry, it electronically switches narrowband filter components to sequentially acquire multiple narrowband response observation groups, resulting in multiple corresponding RAW raw images. Different narrowband response observation groups correspond to different target band sets. Using each Bayer primitive unit as the processing basis, it extracts each color response channel from the original pixel array of the RAW raw image to construct a joint observation vector. Based on the mapping relationship between the target band space and the joint observation space, a joint forward matrix is ​​constructed. Joint forward matrix The behavior color observation channel is defined, with the columns of the joint forward matrix representing the target band, and the elements in the joint forward matrix representing the response coefficients of the target band to the color observation channel; the expression for the joint forward model is established. ,in, For the target spectral vector, The bias vector is the bias vector. It is with the joint observation vector The multidimensional bias parameters corresponding to the dimensions are used to characterize the background residual components in the system observations; weighted constraints are applied based on observation confidence. Spectral continuity constraint Stabilization energy constraint And with the non-negativity constraint of spectral physics, a unified joint optimization objective is constructed: S≥0, D is a first-order difference operator constructed along the target wavelength direction; for the bias vector Initialization, the target spectral vector is solved using an alternating optimization approach. and bias vector Until convergence is achieved, when the target spectral vector is solved. Fixed bias vector When solving for the bias vector Fixed target spectral vector After the alternating solution is completed, an additional loss function is constructed by referencing the whiteboard spectral continuity constraint and the scaling consistency constraint, and applied to the target spectral vector. Optimize and adjust the target spectral vector. Substitute the bias vectors into the unified joint optimization objective and solve for the bias vectors again. The method outputs the reconstructed target spectral vector corresponding to each target band. This method can improve the quality of multi-band reconstruction.

[0084] In one example, the method further includes: Obtain the reconstructed target spectral vectors of a reference whiteboard with reflectivity of a% in each target band. The reconstructed target spectral vectors of a reference whiteboard with a reflectance of b% in each target band. ; Establish a linear reflectivity mapping relationship for the target band: ; This represents the reconstructed target spectral vectors in each target band. This represents the reflectivity of the target at the λ band.

[0085] After completing the joint forward inversion, joint bias estimation, and reference whiteboard constraint optimization, the reconstruction results for the nine target bands are obtained, preferably represented as E450, E530, E555, E650, E670, E730, E800, E840, and E960. These results correspond to the joint spectral response estimation results of the target scene in each target band.

[0086] Furthermore, to improve the consistency of results across different scenarios, devices, and acquisition conditions, reflectance recovery processing can be performed on the nine-band results to obtain physically meaningful target reflectance results. Unlike traditional multi-band systems that rely solely on post-processing whiteboard normalization, the reflectance calculation in this scheme is based on a joint model that has already completed joint scale constraints, background bias suppression, spectral continuity optimization, and multi-level reference whiteboard consistency correction. Therefore, the subsequent reflectance calculation process has better stability and physical consistency.

[0087] Furthermore, a two-point affine correction model can be constructed using reference whiteboards with different reflectivity levels. In a preferred embodiment, a first reference whiteboard with a known reflectivity of 30% (a is 30) and a second reference whiteboard with a known reflectivity of 90% (b is 90) are respectively set up, and the radiation response values ​​of the two sets of reference whiteboards under each target band are acquired simultaneously. For any target band Let the spectral response value of the first reference whiteboard be... The spectral response value of the second reference whiteboard is The spectral response value of the target to be measured is Then a linear reflectivity mapping relationship for the target band can be established: ;in This represents the reflectivity of the target at the λ band.

[0088] Through the above two-point correction, the system gain error, background offset error and dynamic range compression error can be compensated simultaneously, so that the joint inversion result is mapped to the standard reflectivity space.

[0089] Since the 30% and 90% reference whiteboards correspond to the low-end and high-end anchor points of the reflectivity scale, respectively, this dual-point correction method can effectively constrain the slope and intercept of the system response curve. Compared with the traditional single-point whiteboard normalization method, it can significantly improve the reflectivity consistency between different devices and under long-term operating conditions.

[0090] Furthermore, when the system is configured with multiple reference whiteboards of known reflectivity levels, a nonlinear response correction model can be established using a multi-point fitting method to compensate for the nonlinear response error that may exist in the sensor in the high-brightness or low-brightness areas, thereby further improving the reflectivity recovery accuracy.

[0091] Therefore, the reflectance recovery process in this invention is not a simple post-processing normalization of the original image data, but rather a physical scale mapping and fine correction performed on the basis of joint optimization results, thereby obtaining a nine-band reflectance output result with higher stability, higher repeatability and stronger physical interpretability.

[0092] Based on the above nine-band reflectance results or normalized target spectral results, this scheme can further calculate at least one vegetation index for vegetation growth status analysis, chlorophyll estimation, stress detection, and ecological remote sensing analysis. In a preferred embodiment, the following vegetation indices can be calculated, including but not limited to: NDVI, GNDVI, NDRE, EVI, OSAVI, LCI, YI, and DVI. Furthermore, since this scheme has jointly suppressed negative anomalies, band crosstalk, background bias, zero-value collapse, spectral oscillation, and multi-band scale drift during the joint optimization stage, the vegetation indices calculated based on the nine-band results can maintain better numerical stability and physical consistency. Compared with the traditional three-band independent unmixing method, the vegetation indices calculated by this scheme have the following advantages: 1. Higher scale consistency between bands; 2. Stronger stability of denominator-sensitive indices; 3. Lower noise amplification effect in weak reflectance areas; 4. More stable separation ability between vegetation and background features; 5. Better spatial distribution continuity of indices; 6. Less prone to local abnormally high or low values. Furthermore, this scheme enhances the physical interpretability of the nine-band results, making it more suitable for: crop growth monitoring, nitrogen estimation, disease identification, water stress analysis, ecological environment monitoring, and deployment of low-cost edge remote sensing systems.

[0093] In addition, this application is also applicable to multi-band reflectance feature analysis of typical land features such as river beaches, bare land, and roads; and multi-band image reconstruction and exponential output systems on low-cost edge devices.

[0094] In one example, Figure 5 The image shown is an RGB observation image captured by a color image sensor. Figure 6 The corresponding nine-band image is obtained by inversion using the multispectral image reconstruction method of this application, corresponding to nine target bands: 450nm, 530nm, 555nm, 650nm, 670nm, 730nm, 800nm, 840nm, and 960nm. Figure 7 This is an NVDI vegetation index map calculated from the reconstructed target spectral vector obtained by the multispectral image reconstruction method of this application.

[0095] Figure 8 Another flowchart illustrating the multispectral image reconstruction method provided in this application involves: firstly, multiple narrowband filtered RAW data acquisitions; secondly, parsing the acquired Bayer RAW data to extract compact RGB channels and construct a joint observation vector; and finally, establishing a joint forward matrix. A 9. And for the bias term b9. Joint modeling: Joint optimization solution with constraints (non-negativity constraint, spectral continuity constraint, observation weighting constraint, and stabilization energy constraint), and further optimization solution using reference whiteboard continuity and consistency constraints, to obtain nine-band restoration results. Reflectance restoration is performed based on the nine-band restoration results, and finally vegetation index is output based on reflectance.

[0096] This invention proposes a nine-band reconstruction method based on joint observation modeling, which integrates multiple filtered acquisition results into a unified joint optimization framework. Compared to the traditional method of independent unmixing of three bands in groups, this method effectively improves the scale consistency and spectral continuity between different bands. By explicitly introducing a bias vector into the joint forward model, this invention achieves unified modeling of system background bias, out-of-band leakage, and dark response residuals. Compared to the traditional fixed background subtraction method, this reduces the error amplification problem caused by model mismatch and improves the stability of joint inversion. In the joint optimization target model, by introducing observation confidence weighting, spectral continuity constraints, and physical nonnegativity constraints, this method effectively suppresses negative value anomalies, spectral oscillations, local overshoot, and zero-value collapse phenomena during multi-band reconstruction, thereby improving the physical rationality of the nine-band results. Furthermore, this invention extends the reference whiteboard from a traditional post-processing calibration tool to a source of internal constraints within the joint model. By using the spectral continuity and multi-level scaling consistency of the reference whiteboard to inversely constrain the joint model parameters, this method helps to improve the dynamic range compression and scale drift problems of the reference whiteboard, thus improving the overall consistency of the system.

[0097] This invention can maintain good multi-band spectral consistency and reflectance stability in green plants, soil, roads and other typical ground features, which is beneficial for subsequent vegetation index calculation, reflectance analysis and low-cost multi-band remote sensing applications.

[0098] Secondly, embodiments of this application provide a multispectral image reconstruction apparatus, the apparatus comprising: The acquisition module is used to acquire multiple narrowband response observation groups in sequence by electronically switching the narrowband filter components under the condition of keeping the exposure parameters, gain parameters, sensor working mode and imaging geometry consistent, using the same Bayer color image sensor, and obtaining multiple corresponding RAW raw images. Different narrowband response observation groups correspond to different target band sets. The joint observation vector construction module is used to extract each color response channel from the raw pixel array of the RAW image, based on each Bayer primitive unit, and construct a joint observation vector. ; The joint forward matrix construction module is used to construct the joint forward matrix based on the mapping relationship between the target band space and the joint observation space. Joint forward matrix The behavior color observation channel, the columns of the joint forward matrix are the target bands, and the elements in the joint forward matrix represent the response coefficients of the target bands to the color observation channel; The joint forward model building module is used to construct the expression of the joint forward model. ,in, For the target spectral vector, The bias vector is the bias vector. It is with the joint observation vector Y The multidimensional bias parameters corresponding to the dimension are used to characterize the background residual components in the system observations; A unified joint optimization objective construction module is used to impose weighted constraints based on observation confidence. Spectral continuity constraint Stabilization energy constraint And with the non-negativity constraint of spectral physics, a unified joint optimization objective is constructed: S≥0, D A first-order difference operator constructed along the target wavelength direction; The alternating solver module is used to solve the bias vector. Initialization, the target spectral vector is solved using an alternating optimization approach. and bias vector Until convergence is achieved, when the target spectral vector is solved. Fixed bias vector When solving for the bias vector Fixed target spectral vector ; The whiteboard constraint module is used to construct an additional loss function for the target spectral vector after the alternating solution is completed, by referring to the whiteboard spectral continuity constraint and scaling consistency constraint. Optimize and adjust the target spectral vector. Substitute the bias vectors into the unified joint optimization objective and solve for the bias vectors again. Output the reconstructed target spectral vector corresponding to each target band.

[0099] In one possible implementation, the joint observation vector construction module is specifically used for: Based on each Bayer primitive unit, red response pixels are extracted from the original pixel array of the RAW image to construct the R channel, two green response pixels in the original pixel array of the RAW image are fused by mean to construct the G channel, and blue response pixels are extracted from the original pixel array of the RAW image to construct the B channel. A joint observation vector is constructed based on the R, G, and B channels of multiple RAW original images. .

[0100] In one possible implementation, the device further includes a reflectivity calculation module for: Obtain the reconstructed target spectral vectors of a reference whiteboard with reflectivity of a% in each target band. The reconstructed target spectral vectors of a reference whiteboard with a reflectance of b% in each target band. ; Establish a linear reflectivity mapping relationship for the target band: ; This represents the reconstructed target spectral vectors in each target band. This represents the reflectivity of the target at the λ band.

[0101] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the aforementioned multispectral image reconstruction methods.

[0102] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0103] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0104] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are described simply because they are similar to the method embodiments; relevant parts can be referred to the descriptions of the method embodiments.

[0105] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A multispectral image reconstruction method, characterized in that, The method includes: Using the same Bayer color image sensor, while maintaining consistent exposure parameters, gain parameters, sensor operating mode, and imaging geometry, the narrowband filter components are switched electronically to sequentially acquire multiple narrowband response observation groups, resulting in multiple corresponding RAW raw images. Different narrowband response observation groups correspond to different target band sets. Using each Bayer primitive unit as the processing basis, each color response channel is extracted from the original pixel array of the RAW image to construct a joint observation vector. ; Based on the mapping relationship between the target band space and the joint observation space, a joint forward matrix is ​​constructed. Joint forward matrix The behavior color observation channel, the columns of the joint forward matrix are the target bands, and the elements in the joint forward matrix represent the response coefficients of the target bands to the color observation channel; Establish the expression for the joint forward model ,in, For the target spectral vector, The bias vector is the bias vector. It is with the joint observation vector The multidimensional bias parameters corresponding to the dimension are used to characterize the background residual components in the system observations; Weighted constraints based on observation credibility Spectral continuity constraint Stabilization energy constraint And with the non-negativity constraint of spectral physics, a unified joint optimization objective is constructed: S≥0, D A first-order difference operator constructed along the target wavelength direction; For bias vector Initialization, the target spectral vector is solved using an alternating optimization approach. and bias vector Until convergence is achieved, when the target spectral vector is solved. Fixed bias vector When solving for the bias vector Fixed target spectral vector ; After the alternating solution is completed, an additional loss function is constructed by referencing the whiteboard spectral continuity constraint and the scaling consistency constraint, and applied to the target spectral vector. Optimize and adjust the target spectral vector. Substitute the bias vectors into the unified joint optimization objective and solve for the bias vectors again. Output the reconstructed target spectral vector corresponding to each target band; The additional loss function is: + ; To reference the whiteboard spectral continuity constraint weights, To reference the whiteboard ratio consistency constraint weight, For reference, the whiteboard spectral continuity loss function, Use the whiteboard ratio consistency loss function as a reference. ； This represents the target spectral vector corresponding to a reference whiteboard with a reflectance of a%. This represents the target spectral vector corresponding to a reference whiteboard with a reflectance of b%; a is 30, and b is 90. ； The additional loss function is constructed by referencing the whiteboard spectral continuity constraint and the proportional consistency constraint, and is applied to the target spectral vector. Optimize and adjust the target spectral vector. Substitute the bias vectors into the unified joint optimization objective and solve for the bias vectors again. Output the reconstructed target spectral vectors corresponding to each target band, including: The target spectral vector obtained by applying an additional loss function is then processed. Optimize; The target spectral vector obtained by the solution Substitute into the joint objective function and correct the bias vector in reverse. ; Joint objective function = + + ; .

2. The method according to claim 1, characterized in that, The process uses each Bayer primitive unit as the basis for processing, extracting each color response channel from the original pixel array of the RAW image, and constructing a joint observation vector. ,include: Based on each Bayer primitive unit, red response pixels are extracted from the original pixel array of the RAW image to construct the R channel, two green response pixels in the original pixel array of the RAW image are fused by mean to construct the G channel, and blue response pixels are extracted from the original pixel array of the RAW image to construct the B channel. A joint observation vector is constructed based on the R, G, and B channels of multiple RAW original images. .

3. The method according to claim 1, characterized in that, The bias vector Initialization, the target spectral vector is solved using an alternating optimization approach. and bias vector Until convergence is achieved, when the target spectral vector is solved. Fixed bias vector When solving for the bias vector Fixed target spectral vector ,include: bias vector Initialize the target spectral vector to a zero vector and use an alternating optimization approach. and bias vector Perform joint inversion solutions until convergence is achieved; During the k-th iteration, the current bias vector is fixed. The joint optimization objective is degraded to a value related to the target spectral vector. Sub-optimization problem: ； The target spectral vector is updated using a constrained iterative solution method to obtain the updated target spectral vector. ; With a fixed target spectral vector Under these conditions, the joint optimization objective degenerates into a function about the bias vector. Sub-optimization problem: ； For cross-observation channel difference operators; This is the bias amplitude constraint coefficient; These are the bias continuity constraint coefficients; The bias vector is updated using a constrained iterative solution method to obtain the updated bias vector. The updated bias vector Used for the next round of target spectral vector solving process.

4. The method according to claim 1, characterized in that, The method further includes: Obtain the reconstructed target spectral vectors of a reference whiteboard with reflectivity of a% in each target band. The reconstructed target spectral vectors of a reference whiteboard with a reflectance of b% in each target band. ; Establish a linear reflectivity mapping relationship for the target band: ; This represents the reconstructed target spectral vectors in each target band. This represents the reflectivity of the target at the λ band.

5. A multispectral image reconstruction device, characterized in that, The device includes: The acquisition module is used to acquire multiple narrowband response observation groups in sequence by electronically switching the narrowband filter components under the condition of keeping the exposure parameters, gain parameters, sensor working mode and imaging geometry consistent, using the same Bayer color image sensor, and obtaining multiple corresponding RAW raw images. Different narrowband response observation groups correspond to different target band sets. The joint observation vector construction module is used to extract each color response channel from the raw pixel array of the RAW image, based on each Bayer primitive unit, and construct a joint observation vector. ; The joint forward matrix construction module is used to construct the joint forward matrix based on the mapping relationship between the target band space and the joint observation space. Joint forward matrix The behavior color observation channel, the columns of the joint forward matrix are the target bands, and the elements in the joint forward matrix represent the response coefficients of the target bands to the color observation channel; The joint forward model building module is used to construct the expression of the joint forward model. ,in, For the target spectral vector, The bias vector is the bias vector. It is with the joint observation vector The multidimensional bias parameters corresponding to the dimension are used to characterize the background residual components in the system observations; A unified joint optimization objective construction module is used to impose weighted constraints based on observation confidence. Spectral continuity constraint Stabilization energy constraint And with the non-negativity constraint of spectral physics, a unified joint optimization objective is constructed: S≥0, D A first-order difference operator constructed along the target wavelength direction; The alternating solver module is used to solve the bias vector. Initialization, the target spectral vector is solved using an alternating optimization approach. and bias vector Until convergence is achieved, when the target spectral vector is solved. Fixed bias vector When solving for the bias vector Fixed target spectral vector ; The whiteboard constraint module is used to construct an additional loss function for the target spectral vector after the alternating solution is completed, by referring to the whiteboard spectral continuity constraint and scaling consistency constraint. Optimize and adjust the target spectral vector. Substitute the bias vectors into the unified joint optimization objective and solve for the bias vectors again. Output the reconstructed target spectral vector corresponding to each target band; The additional loss function is: + ; To reference the whiteboard spectral continuity constraint weights, To reference the whiteboard ratio consistency constraint weight, For reference, the whiteboard spectral continuity loss function, Use the whiteboard ratio consistency loss function as a reference. ； This represents the target spectral vector corresponding to a reference whiteboard with a reflectance of a%. This represents the target spectral vector corresponding to a reference whiteboard with a reflectance of b%; a is 30, and b is 90. ； The whiteboard constraint module is specifically used for: The target spectral vector obtained by applying an additional loss function is then processed. Optimize; The target spectral vector obtained by the solution Substitute into the joint objective function and correct the bias vector in reverse. ; Joint objective function = + + ; .

6. The apparatus according to claim 5, characterized in that, The joint observation vector construction module is specifically used for: Based on each Bayer primitive unit, red response pixels are extracted from the original pixel array of the RAW image to construct the R channel, two green response pixels in the original pixel array of the RAW image are fused by mean to construct the G channel, and blue response pixels are extracted from the original pixel array of the RAW image to construct the B channel. A joint observation vector is constructed based on the R, G, and B channels of multiple RAW original images. .

7. The apparatus according to claim 5, characterized in that, The device also includes a reflectivity calculation module for: Obtain the reconstructed target spectral vectors of a reference whiteboard with reflectivity of a% in each target band. The reconstructed target spectral vectors of a reference whiteboard with a reflectance of b% in each target band. ; Establish a linear reflectivity mapping relationship for the target band: ; This represents the reconstructed target spectral vectors in each target band. This represents the reflectivity of the target at the λ band.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-4.

Images