Multi-spectral image demosaicing method and system based on multiple prior calibration networks

Abstract

The invention provides a multispectral image demosaicing method and system based on multiple prior calibration networks, and the method comprises the steps: collecting a multispectral image, and forming a training sample pair; generating a pseudo panchromatic image through a pseudo panchromatic image interpolation network, performing initial reconstruction on the Raw image in combination with weighted bilinear interpolation to obtain an initial multispectral image, and performing multiple prior calibration on the initial multispectral image to obtain a multiple prior calibration network model; calculating a loss function of the multiple prior calibration network model, optimizing network parameters through back propagation until loss convergence, and obtaining a trained multiple prior calibration network model; and inputting a test Raw image into the trained multiple prior calibration network model, and obtaining a complete demosaicing multispectral image through forward propagation. According to the method, a multi-spectral image reconstruction process is accurately simulated by fusing a plurality of priors, and spectral correlation between long-distance spectral segments is mined by using global information, so that spectral aliasing caused by low cross correlation between the long-distance spectral segments is effectively avoided.

Description

Technical Field

[0001] This invention belongs to the field of mosaic removal technology, and specifically relates to a method and system for multispectral image demosaic based on multiple prior calibration networks. Background Technology

[0002] Multispectral images (MSI) consist of multiple narrow bands in the electromagnetic spectrum, offering higher spectral resolution compared to color RGB images. This characteristic enables multispectral imaging technology to provide richer and more accurate spatial and spectral information, thus more precisely describing the characteristics of objects in a scene.

[0003] Multispectral Filter Array (MSFA) sensors are a promising technology that enables the simultaneous acquisition of high-quality, accurately aligned multispectral images in a single exposure. This technology extends single-lens imaging systems (such as ordinary RGB cameras) into multispectral imaging systems by replacing traditional color filter arrays (CFAs) with MSFAs and spatially undersampling three or more spectral bands. An MSFA consists of multiple bandpass filters arranged in a spatially repeating pattern on the surface of a monochrome imaging sensor; the acquired raw data is called the raw image. Because the MSFA spatially downsamples all bands, each pixel can only record sparse information from a specific band. Therefore, to reconstruct a full-resolution, high-fidelity multispectral image, all missing spectral values ​​must be estimated from this sparsely sampled data. This crucial reconstruction process, often referred to as multispectral de-mosaicing, is an indispensable core component of high-quality MSFA single-sensor imaging systems.

[0004] Traditional multispectral demosaicing methods rely on manually designed image priors. However, these prior models focus on the two-dimensional spatial correlation of spectral images, and their representational capabilities are limited. Inaccurate modeling can lead to spectral distortion or blurred edges in the spatial domain, making it difficult to reconstruct fine image details. Deep learning-based multispectral demosaicing methods, whether heuristically designed or model-guided, learn deterministic priors directly from the dataset through deep networks, ignoring the differences in difficulty and accuracy in learning different priors for spectral images. Furthermore, the low sampling rate in spectral imaging systems causes high-frequency components in the real signal to appear as low-frequency components in the sampled signal, easily leading to spectral aliasing and periodic artifacts. Specifically, firstly, the increase in spectral bands and the absence of main channels reduce the spatial sampling rate, weakening the spatial correlation between adjacent pixels sampled on the same spectrum. Sparsely sampled raw data reduces the spatial accuracy of the reconstructed multispectral image. Secondly, as the spectrum is further subdivided and reduced, the spectral cross-correlation between adjacent bands is stronger than that between bands that are far apart, causing aliasing problems between different channels in the reconstructed multispectral image.

[0005] In summary, multispectral demosaicing reconstruction involves problems such as spectral aliasing and periodic artifacts caused by low sampling rates and multiple spectral bands, posing a significant challenge to achieving high-quality reconstruction of the model. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for demosaicing multispectral images based on multiple prior calibration networks, so as to solve the above-mentioned problems.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a multispectral image demosaic method based on a multiple prior calibration network, comprising: Multispectral images are acquired from a hyperspectral database. After spectral sampling, a high-resolution reference multispectral image GT is obtained. Simulation sampling is performed on the image GT to generate a Raw image. The GT and Raw images are cropped into multiple sub-image patches to form training sample pairs. A pseudo-panchromatic image is generated by a pseudo-panchromatic image interpolation network, and an initial multispectral image is obtained by combining weighted bilinear interpolation with the raw image. Multiple prior calibrations are then performed on the initial multispectral image to obtain a multiple prior calibration network model. Calculate the loss function of the multiple prior calibration network model, optimize the network parameters through backpropagation until the loss converges, and obtain the trained multiple prior calibration network model. The test raw image is input into the trained multiple prior calibration network model, and the complete demosaic multispectral image is obtained through forward propagation.

[0008] Furthermore, the process involves acquiring multispectral images from a hyperspectral database, obtaining a high-resolution reference multispectral image GT after spectral sampling, performing simulated sampling on the GT image to generate a Raw image, and cropping the GT and Raw images into multiple sub-image patches to form training sample pairs, including: Multispectral images are acquired from publicly available hyperspectral databases. Spectral bands are sampled from each selected image according to the required number of bands to obtain high-resolution reference multispectral data GT. First, [the data is then processed]. Defined as a multispectral image with C channels and full resolution, the multispectral data is simulated according to the actual spectral filter array structure to obtain the corresponding Raw image; the Raw image is considered as a ground truth image based on pixel coordinates. The result of spectral sampling is shown in equation (1): (1) In equation (1), mod represents the remainder; Data preprocessing is performed on the obtained ground truth (GT) and raw images; the GT is cropped into several sub-image patches. Each sub-image block The size is The corresponding sub-image blocks are obtained according to formula (1). The size of the raw image is Each sub-image block The size is A total of T training sample pairs were generated. .

[0009] Furthermore, the process involves generating a pseudo-panchromatic image through a pseudo-panchromatic image interpolation network, and performing initial reconstruction of the raw image using weighted bilinear interpolation to obtain an initial multispectral image. Multiple prior calibrations are then performed on the initial multispectral image to obtain a multiple prior calibration network model, including: The multiple prior calibration network model includes a coarse demosaic stage and a multiple prior calibration reconstruction stage: The coarse demosaic stage employs a pseudo-panchromatic image interpolation network and weighted bilinear interpolation to perform interpolation reconstruction; the raw image... The input is fed into a pseudo-panchromatic image interpolation network based on residual learning, and then passes through a filter. Generate pseudo-panchromatic image Simultaneously, the raw image passes through four convolutional layers with different kernel sizes, and then the output features of the convolutional layers are combined with... Adding them together gives : (2) In equation (2), , , and It consists of four convolutional layers. Represents the convolution operator; The coefficients are the sum of all filter elements; The raw image is segmented directly into different spectral bands using a segmentation method, and the missing spectral information is left blank to obtain a sparse image. Then, weighted bilinear interpolation convolution operations are used to estimate the multispectral image. ,Right now (4) in, For filters; The multiple prior calibration reconstruction stage recalibrates the initial prediction by combining different prior knowledge; and As input to the multiple prior calibration reconstruction stage, a multiple prior calibration module is used to correct pixels and improve the multispectral image. After the multiple prior calibration reconstruction stage, the final output multispectral image is obtained. : (6) In equation (6), , and These represent the outputs of the spectral response function module, the global periodic attention module, and the edge aggregation module, respectively. This indicates a multiple prior calibration module.

[0010] Furthermore, the spectral response function module, the global periodic attention module, and the edge aggregation module include: The input to the spectral response function module is a sparse image. and interpolated images spliced ​​images The output is the spectral sensitivity function and content prior information. First, shallow features are extracted using six convolutional layers with symmetrical residual connections; each convolutional layer uses a kernel of the same size. The shallow features are then divided into four parts, and the gradient information of each part is calculated and then concatenated. Next, the features are input into three parallel convolutional layers, and their outputs are concatenated into a unified feature representation. At the same time, the pixel position information of the concatenated features is converted into a weight matrix, which is the estimated spectral sensitivity function. Finally, the weight matrix and the output image of the spectral response function module are resized and multiplied to obtain the final content prior information. The global periodic attention module consists of a feature extraction layer A residual block A global multi-head attention module and periodic attention blocks Composition; Output of the global cycle module Represented as (7) In equation (7), This represents the input feature map of the global periodic attention module; make As input to the global multi-head module, it is first reshaped into a tensor. Then, Linearly projected onto the query matrix Key matrix Value matrix and two auxiliary tensors and ; (8) In equation (8), , , , and This represents the learnable parameters, i.e., the projection matrix of each component; Then , , , and Divided along the spectral channel dimension Size: , , , and The dimensions of each head are: In the attention graph, each spectral representation is treated as a token, and calculations are performed. Self-attention: (9) (10) (11) In equation (9), express The transpose of the matrix; Introduce a learnable parameter Through the Internal matrix multiplication and Reweighting to adjust self-attention ; then, indivual The outputs are connected together and linearly projected; finally, the reshape is performed. The output feature map is obtained. ; (12) In the periodic attention block, adaptive average pooling is first used to aggregate the multispectral filter array. Feature points at the same relative position on each channel: (13) In the formula, and . It is the size of the multispectral filter array; Then, the Sigmoid function is used to obtain the mosaic attention map. : (14) Repeating in spatial dimensions To obtain aggregation features : (15) Finally passed Reweighting Obtain the output features of the global periodic attention module ; (16) The edge aggregation module consists of two branches, whose inputs are the low-level representation mosaic image and the low-level representation mosaic image, respectively. The output image of the pseudo-panchromatic interpolation network To increase the number of channels, two convolutional layers with a kernel size of 3×3 are used for each of the two inputs, as shown below: (17) (18) In equation (17), It is the ReLU activation function. and Representing the convolutional layer, the weights are respectively and ; After multiplying the outputs of the two branches and performing a convolution, we obtain selectively weighted attention. Adaptively learn the weights of different branches: (19) By weighting the features, more attention is focused on important edge regions; residual aggregation and splicing operations are used to fuse the features from the two branches, and then a convolutional layer is applied to obtain the edge prior value. : (20) In the formula, This indicates a convolution operation with a kernel size of 3×3 and weights of size . ; Indicates a splicing operation; Finally, by using a 1×1 convolutional layer Using the Sobel operator to obtain edge features .

[0011] Furthermore, the calculation of the loss function of the multiple prior calibration network model, and the optimization of network parameters through backpropagation until the loss converges, to obtain the trained multiple prior calibration network model, includes: Calculate the spatial consistency loss, mean squared error loss (L2 loss), and marginal loss of the multiple prior calibration network model, and then weight and combine them into a total loss function: Calculate the spatial consistency loss of a multi-prior calibration network model. The expression is as follows: (twenty four) In the formula, Represents the spectral sensitivity function. The parameters representing the spectral sensitivity function, Indicates the first The first image A vector representation of the true spectral reflectance at each pixel location; It is a nonlinear function for estimating spectral reflectance; where It is the first The number of pixels in an image. This refers to the number of images used. Calculate the multiple prior calibration network model The loss is expressed as follows: (25) In equation (25), It is the number of training samples. Indicates a reference image. This represents the multispectral demosaiced image estimated by the method of the present invention; Calculate the edge loss of the multiple prior calibration network model The expression is as follows: Using real edge images and estimated edge image The root mean square error loss between them is used as the marginal loss: (26) In the formula, This represents a true pseudo-panchromatic interpolated image. This represents the estimated pseudo-panchromatic interpolated image; Will loss, Loss and The losses are weighted and combined to obtain the total loss of the multi-prior calibration network model. The expression is as follows: (27); Establish the original Image and reference multispectral image The nonlinear mapping relationship between them is determined by training a multi-prior calibration network model until the loss value reaches its minimum. The structure and parameters of each layer of the trained multi-prior calibration network model are then saved to obtain the established nonlinear mapping relationship.

[0012] Furthermore, the step of inputting the test raw image into the trained multiple prior calibration network model and obtaining a complete demosaicized multispectral image through forward propagation includes: The test raw image is input into the trained multiple prior calibration network model, and multispectral reconstruction is performed on the multispectral mosaic images of the test set to obtain the complete de-mosaic multispectral image.

[0013] Furthermore, peak signal-to-noise ratio, structural similarity, spectral angular distance, root mean square error relative parameter estimation standard deviation, and visualization effect of reconstructed image are used as evaluation criteria to evaluate performance.

[0014] Secondly, the present invention provides a multispectral image demosaic system based on a multiple priori calibration network, comprising: The data acquisition module is used to acquire multispectral images from the hyperspectral database. After spectral sampling, a high-resolution reference multispectral image GT is obtained. The image GT is simulated and sampled to generate a Raw image. The GT and Raw images are cropped into multiple sub-image patches to form training sample pairs. The model building module is used to generate a pseudo-panchromatic image through a pseudo-panchromatic image interpolation network, and to perform initial reconstruction of the raw image by combining weighted bilinear interpolation to obtain an initial multispectral image. Multiple prior calibrations are then performed on the initial multispectral image to obtain a multiple prior calibration network model. The training module is used to calculate the loss function of the multiple prior calibration network model, optimize the network parameters through backpropagation until the loss converges, and obtain the trained multiple prior calibration network model. The output module is used to input the test Raw image into the trained multiple prior calibration network model and obtain the complete demosaic multispectral image through forward propagation.

[0015] Thirdly, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the multispectral image demosaic method based on a multiple prior calibration network as described in any one of claims 1 to 7.

[0016] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the multispectral image demosaic method based on a multiple prior calibration network.

[0017] Compared with the prior art, the present invention has the following technical effects: This invention enables multi-faceted modeling of content, global, and edge information dependencies during the demosaicing reconstruction process. A spectral response function module is proposed, using the spectral response function as supervisory information to guide the proposed method in learning fine spectral details from real spectral channels, thereby effectively reconstructing multispectral information. A global periodicity module is proposed, combining global periodicity modeling and MSFA periodicity modeling to maintain the complete spatial structure of the original image. An edge aggregation module is proposed to obtain initial edge priors, effectively alleviating the boundary blurring problem during reconstruction. This invention can be widely applied in fields such as military reconnaissance, remote sensing monitoring, medical imaging, and industrial inspection. Utilizing the global perception, detail reconstruction, and edge preservation capabilities of the proposed architecture, it effectively improves image quality, significantly enhancing visual recognition and the reliability of downstream task analysis while accurately recovering spectral information. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the method of the present invention; Figure 2 This is a schematic diagram of a 4×4 filter array on the sensor surface used in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the multiple prior calibration network model constructed by the method of the present invention; Figure 4 This is a schematic diagram of the structure of the pseudo-panchromatic interpolation network constructed by the method of this invention; Figure 5 This is a schematic diagram of the structure of the spectral response function module constructed by the method of the present invention; Figure 6 This is a schematic diagram of the structure of the global periodic attention module constructed by the method of the present invention; Figure 7 This is a schematic diagram of the edge aggregation module constructed by the method of the present invention; Figure 8 This is a schematic diagram of the structure of the multiple prior calibration module used in the method of the present invention; Figure 9 This is a comparison chart of the experimental results of Embodiment 1 of the present invention on the Tokyo31 dataset; Figure 10 This is a comparison chart of residual results on the Tokyo31 dataset in Embodiment 2 of the present invention. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings: Example 1, please refer to Figure 1 This invention provides a multispectral image demosaic method based on a multiple prior calibration network, comprising: Multispectral images are acquired from a hyperspectral database. After spectral sampling, a high-resolution reference multispectral image GT is obtained. Simulation sampling is performed on the image GT to generate a Raw image. The GT and Raw images are cropped into multiple sub-image patches to form training sample pairs. A pseudo-panchromatic image is generated by a pseudo-panchromatic image interpolation network, and an initial multispectral image is obtained by combining weighted bilinear interpolation with the raw image. Multiple prior calibrations are then performed on the initial multispectral image to obtain a multiple prior calibration network model. Calculate the loss function of the multiple prior calibration network model, optimize the network parameters through backpropagation until the loss converges, and obtain the trained multiple prior calibration network model. The test raw image is input into the trained multiple prior calibration network model, and the complete demosaic multispectral image is obtained through forward propagation.

[0020] This invention can be widely applied in fields such as military reconnaissance, remote sensing monitoring, medical imaging, and industrial inspection. By utilizing the global perception, detail reconstruction, and edge preservation capabilities of the proposed architecture, it can effectively improve image quality, accurately recover spectral information, and significantly enhance visual recognition and the reliability of downstream task analysis.

[0021] Example 2: This invention provides a multispectral image demosaic method based on a multiple prior calibration network, comprising: Step 1: Data preprocessing First, multispectral images are acquired from existing publicly available hyperspectral databases. Then, spectral bands are sampled from each selected image according to the required number of bands to obtain high-resolution reference multispectral data (GT), which is used for comparison and evaluation. Defined as a multispectral image with C channels and full resolution, and then based on the actual spectral filtering array structure, such as... Figure 2 As shown, multispectral data were simulated using a 4×4 (C=16) multispectral filter array to obtain the corresponding Raw image; In a raw image, each pixel contains information only for a specific spectral band; values ​​from other channels are unavailable. Assuming the spatial dimensions of the raw image are H×W, based on the design principles of multispectral filter arrays, a single spectral band... With each pixel Related, , Treating Raw images as ground truth based on pixel coordinates The result of spectral sampling is shown in equation (1): (1) In equation (1), mod represents the remainder. It is 16.

[0022] To facilitate network training and modeling, data preprocessing is performed based on the obtained ground truth (GT) and raw images. Assume the size of the GT is... The GT is cropped into several sub-image blocks. Each sub-image block The size is In actual training process, the best option is selected. The corresponding sub-image blocks are obtained according to formula (1). The size of the raw image is Each sub-image block The size is A total of T training sample pairs were generated. .

[0023] Step 2, see Figure 3 We constructed a multi-prior calibration network model to realize the transformation of raw images into complete multispectral images. The mapping, The multiple prior calibration network model includes a coarse demosaic stage and a multiple prior calibration reconstruction stage. Firstly, the coarse demosaic stage employs a pseudo-panchromatic image interpolation network and weighted bilinear interpolation to perform interpolation reconstruction. The pseudo-panchromatic image is similar to a grayscale version of a panchromatic image, improving image resolution while preserving the spectral information of the multispectral image. For example... Figure 3 As shown, first, the Raw image The input is fed into a pseudo-panchromatic image interpolation network based on residual learning, and then passes through a filter. Generate pseudo-panchromatic image Simultaneously, the raw image is passed through four convolutional layers with different kernel sizes. The output features of these convolutional layers are then combined with... Adding them together gives .

[0024] (2) In equation (2), , , and The kernel sizes are 9×9, 7×7, 5×5 and 5×5, respectively. This represents the convolution operator. The coefficients are the sum of all filter elements, therefore each spatial pixel is crucial for generating a pseudo-panchromatic image. When using 4×4 (i.e., When using a multispectral filter array, for (3) The raw image is segmented directly into different spectral bands using a segmentation method, and the missing spectral information is left blank, thus obtaining a sparse image. Then, weighted bilinear interpolation convolution operations are used to estimate the multispectral image. ,Right now (4) Among them, the filter for (5) Weighted bilinear interpolation assigns a weight to each pixel. These weights are typically inversely proportional to the distance from the data point to the interpolation point; that is, the closer a data point is to the interpolation point, the greater its weight and the greater its influence on the interpolation result.

[0025] Secondly, the multiple prior calibration reconstruction stage recalibrates the initial prediction by combining different prior knowledge. and As input to the multiple prior calibration reconstruction stage, and using a multiple prior calibration module to correct pixels and improve the multispectral image, the final output multispectral image is obtained after the multiple prior calibration reconstruction stage. : (6) In equation (6), , and These represent the outputs of the spectral response function module, the global periodic attention module, and the edge aggregation module, respectively. This indicates a multiple prior calibration module.

[0026] Third, in order to learn the mapping between the input image and its optimal spectral sensitivity function, a spectral response function module is proposed. Figure 5 The detailed network structure of the spectral response function module is shown. The input to the spectral response function module is a sparse image. and interpolated images spliced ​​images The output is the spectral sensitivity function and content prior information. First, shallow features are extracted using six convolutional layers with symmetrical residual connections. Each convolutional layer uses a kernel of the same size. The extracted shallow features are then divided into four parts, and the gradient information of each part is calculated and concatenated. Next, the features are input into three parallel convolutional layers, and their outputs are concatenated into a unified feature representation. Simultaneously, the pixel position information of the concatenated features is converted into a weight matrix, which is the estimated spectral sensitivity function. Finally, this weight matrix is ​​resized and multiplied with the output image of the spectral response function module to obtain the final content prior information.

[0027] Fourth, in multispectral filter array imaging, spatial-spectral ambiguity arises because the spectral information of each pixel is sampled and compressed, and then the nearest neighbor pixels with similar spectral information are periodically arranged according to MSFA. To address this issue, a global multi-head attention module and a periodic attention block are proposed in the global periodic attention module to solve the periodic artifact problem caused by spatial-spectral ambiguity. For example... Figure 6 As shown, the global periodic attention module consists of a feature extraction layer. A residual block A global multi-head attention module and periodic attention blocks Composition. Output of the global cycle module. It can be represented as (7) In equation (7), This represents the input feature map of the global periodic attention module.

[0028] like Figure 6 As shown, the global multi-head attention module extracts information from different scales through multi-scale feature extraction, feature transformation, and attention mechanisms, further enhancing artifact removal performance. Let As input to the global multi-head module, it is first reshaped into a tensor. Then, Linearly projected onto the query matrix Key matrix Value matrix and two auxiliary tensors and .

[0029] (8) In equation (8), , , , and This represents the learnable parameters, i.e., the projection matrix of each component.

[0030] Then , , , and Divided along the spectral channel dimension Size: , , , and The dimensions of each head are... In the attention graph, each spectral representation is treated as a token, and calculations are performed. Self-attention: (9) (10) (11) In equation (9), express The transpose of .

[0031] Since the spectral density varies significantly with wavelength, a learnable parameter is introduced. Through the Internal matrix multiplication and Reweighting to adjust self-attention Subsequently, indivual The outputs are connected and linearly projected. Finally, the images are reshaped. The output feature map is obtained. .

[0032] (12) The method of this invention performs a forward propagation process on the segmented spectral cube. To avoid periodic mosaic distortion, a periodic attention block is proposed. In the multispectral filter array, feature points with the same relative position are treated as a group. This strategy takes into account the periodic spectral mosaic variations in the raw image and avoids spatial destruction caused by the sparse arrangement of the raw image. In the periodic attention block, adaptive average pooling is first applied to aggregate the features against the multispectral filter array. Feature points at the same relative position on each channel: (13) In the formula, and . It is the size of the multispectral filter array.

[0033] Then, a sigmoid function is used to obtain the mosaic attention map. : (14) Repeating in spatial dimensions To obtain aggregation features : (15) It can be used to reduce mosaic distortion within each channel and highlight important information. Finally, through Reweighting Obtain the output features of the global periodic attention module .

[0034] (16) Fifth, to better reconstruct edge information, the method of this invention uses explicit edge priors as additional auxiliary information. Low-level representations can serve as supplementary information for the multiple prior recalibration module, helping it reconstruct finer edge details. However, directly fusing low-level features into high-level features may lead to redundancy and inconsistency issues. To address this problem, the method of this invention designs an edge aggregation module that adaptively extracts mutual representations from the input before fusing low-level and high-level features, which helps learn more edge information with explicit constraints and enhanced feature details.

[0035] like Figure 7 As shown, the edge aggregation module consists of two branches, whose inputs are the low-level representation mosaic image. The output image of the pseudo-panchromatic interpolation network To increase the number of channels by using two convolutional layers with kernel size of 3×3 for each of the two inputs, it can be represented as follows: (17) (18) In equation (17), It is the ReLU activation function. and Representing the convolutional layer, the weights are respectively and .

[0036] After multiplying the outputs of the two branches and performing a convolution, we obtain selectively weighted attention. It can adaptively learn the weights of different branches: (19) By weighting the features, more attention is focused on important edge regions. To reduce information loss in deep networks, residual aggregation and concatenation operations are used to fuse the features of the two branches, which are then passed through a convolutional layer to obtain the edge prior values. : (20) In the formula, This indicates a convolution operation with a kernel size of 3×3 and weights of size . . This indicates a splicing operation.

[0037] Finally, by using a 1×1 convolutional layer Using the Sobel operator to obtain edge features .

[0038] Sixth, the method of this invention designs a multi-prior calibration module (MPCM) that learns from multiple perspectives, jointly focusing on low-level and high-level features, and readjusting semantic priors in a cascaded manner. Simultaneously, the method of this invention further explores the relationship between edge-aware details, semantic information, and spectral sensitivity functions to accurately reconstruct lost information.

[0039] like Figure 8 As shown on the left, the three prior maps—the content prior map output by the spectral response function module, the global prior map output by the global periodic attention module, and the edge prior feature map output by the edge aggregation module—are input into the multi-priority calibration module. The calibration process of the multi-priority calibration module can be summarized in the following two steps: For content priors First, the content prior values ​​are normalized to [0, 1] using the standard Sigmoid function. Then, the normalized features are multiplied by the global prior values ​​to obtain the fused features. : (twenty one) Edge-aware details are provided in multiple prior calibrations. Edge priors supplement the visual details learned from the edge aggregation module during recalibration. Figure 7 (Edge information in the data). In the multi-calibration module, the edge information and fused features are first concatenated and then passed to the first 3×3 convolutional layer. It learns additional low-level feature information while considering spatial neighborhood effects. The two middle 3×3 convolution operations ( and ) and ReLU activation function Used to select the next spectrum from redundant features for optimization. The last 1×1 convolution operation. This approach enables pixel-by-pixel fine-tuning of each spectral vector, significantly improving the reconstruction results of low-level features and further facilitating the mapping learning of the multi-prior calibration module. Furthermore, skip connections are employed in the multi-prior calibration module to accelerate network computation while simultaneously forcing the network to pay closer attention to changing details. After a series of convolutional operations, the reconstructed multispectral image is finally obtained. : (twenty two) (twenty three) Step 3: Update the parameters of the multiple prior calibration network model. Calculate the spatial consistency loss of a multi-prior calibration network model. The expression is as follows: (twenty four) In the formula, Represents the spectral sensitivity function. The parameters representing the spectral sensitivity function, Indicates the first The first image The vector representation of the true spectral reflectance at each pixel location. It is a nonlinear function used to estimate spectral reflectance using the model of this invention. It is the first The number of pixels in an image. This is the number of images used.

[0040] Calculate the multiple prior calibration network model The loss is expressed as follows: (25) In equation (25), It is the number of training samples. Indicates a reference image. This represents the multispectral demosaiced image estimated by the method of the present invention.

[0041] Calculate the edge loss of the multiple prior calibration network model The expression is as follows: This invention uses real edge images and estimated edge image The root mean square error loss between them is used as the marginal loss: (26) In the formula, This represents a true pseudo-panchromatic interpolated image. This represents the estimated pseudo-panchromatic interpolated image.

[0042] Will loss, Loss and The losses are weighted and combined to obtain the total loss of the multi-prior calibration network model. The expression is as follows: (27) Step 4: Establish the original Image and reference multispectral image The nonlinear mapping relationship between them is determined by training a multi-prior calibration network model until the loss value reaches its minimum. The structure and parameters of each layer of the trained multi-prior calibration network model are then saved to obtain the established nonlinear mapping relationship.

[0043] Step 5: Obtain the fully defined multispectral demosaic image, input the test data into the trained multiple prior calibration network model, and obtain the final multispectral image.

[0044] The optimized multi-prior calibration network model from step 4 is used to perform multispectral reconstruction on the multispectral mosaic images of the test set. Peak signal-to-noise ratio (PSNR), structure similarity index measure (SSIM), spectral angular distance (SAM), error relative global accuracy synthesis (ERGAS), and the visualization effect of the reconstructed image are used as evaluation criteria. The performance of four existing reconstruction methods is compared. The three existing deep learning-based multispectral de-mosaic methods are Splitting-based Network (SPNet), Mosaic Convolution Attention Network (MCAN), and Deep Demosaicking Network (DDM-Net).

[0045] Table 1 Comparison of objective indicators for different demosaic methods

[0046] Table 1 lists the results of four evaluation metrics when using the IMEC16 filter response sampling operator on the Tokyo31 dataset. Specifically, the method of the present invention achieves a high PSNR value, indicating a high similarity between the reconstructed image and the reference image. Furthermore, the ERGAS and SAM metrics show minimal difference between the reconstructed image and the reference image, highlighting its effectiveness in reconstructing spectral information. In addition, the method of the present invention has a high SSIM score, indicating strong structural similarity between the generated image and the reference image in the Tokyo31 dataset.

[0047] Example 3 is implemented using a portion of the Tokyo31 dataset, following the steps and procedures described above.

[0048] See Figure 9 The paper demonstrates the visualization effects of the proposed method and several existing methods on different multispectral images for demosaicing on the Tokyo31 dataset. From the comparison of the visual effects of the reconstructed images, it can be seen that SPNet and MCAN methods introduce obvious artifacts and discontinuities at the edges of objects, while DDM-Net method loses some image details. The proposed method can effectively capture the high-frequency and smooth parts of the image, and the reconstructed image edges are clearer, which is closest to the ground truth effect of the real image.

[0049] Example 4 The present invention is implemented using a portion of the Tokyo31 dataset, following the steps outlined above.

[0050] See Figure 10 This illustrates the residual plots of the method of the present invention and different comparison methods. The residual plots are the average relative absolute errors at 415 nm between the reference image GT and the multispectral images recovered by different comparison methods. Figure 10 As can be seen, the residual value of the method of the present invention has the smallest error compared with the real image GT, and the best effect is achieved in detail reconstruction.

[0051] This invention proposes a multispectral image demosaicing method based on a multiple prior calibration network model, combining physical observation models. By fusing multiple priors, it accurately simulates the multispectral image reconstruction process and utilizes global information to mine spectral correlations between distant spectral bands, effectively avoiding spectral aliasing caused by low cross-correlation between distant spectral bands. Simultaneously, considering the periodic mosaic information changes in the raw image, it introduces content priors to model the spatial correlation between adjacent pixels. Furthermore, guided by the image after initial interpolation, an edge aggregation mechanism based on edge priors is proposed, which helps to restore edge details of the reconstructed image. This invention can achieve high-precision multispectral image demosaicing and reconstruction, and can be widely applied in fields such as military reconnaissance, environmental protection, agricultural monitoring, and medical imaging.

[0052] In another embodiment of the present invention, a multispectral image demosaic system based on a multiple prior calibration network is provided, which can be used to implement the above-mentioned multispectral image demosaic method based on a multiple prior calibration network. Specifically, the system includes: The data acquisition module is used to acquire multispectral images from the hyperspectral database. After spectral sampling, a high-resolution reference multispectral image GT is obtained. The image GT is simulated and sampled to generate a Raw image. The GT and Raw images are cropped into multiple sub-image patches to form training sample pairs. The model building module is used to generate a pseudo-panchromatic image through a pseudo-panchromatic image interpolation network, and to perform initial reconstruction of the raw image by combining weighted bilinear interpolation to obtain an initial multispectral image. Multiple prior calibrations are then performed on the initial multispectral image to obtain a multiple prior calibration network model. The training module is used to calculate the loss function of the multiple prior calibration network model, optimize the network parameters through backpropagation until the loss converges, and obtain the trained multiple prior calibration network model. The output module is used to input the test Raw image into the trained multiple prior calibration network model and obtain the complete demosaic multispectral image through forward propagation.

[0053] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0054] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or function. The processor described in this embodiment of the present invention can be used for the operation of a multispectral image demosaic method based on a multiple prior calibration network.

[0055] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the multispectral image demosaic method based on multiple prior calibration networks in the above embodiments.

[0056] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0057] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0058] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0059] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multispectral image demosaicking method based on multiple prior calibration networks, characterized in that, The method comprises the following steps: Collecting a multispectral image from a hyperspectral database, obtaining a high-resolution reference multispectral image GT after spectral sampling, simulating sampling on the image GT to generate a Raw image, and cutting the GT and the Raw image into multiple sub-image blocks to form a training sample pair; Generating a pseudo-panchromatic image through a pseudo-panchromatic image interpolation network, and combining weighted bilinear interpolation to perform initial reconstruction on the Raw image to obtain an initial multispectral image, and performing multiple prior calibration on the initial multispectral image to obtain a multiple prior calibration network model; Calculating the loss function of the multiple prior calibration network model, optimizing the network parameters through back propagation until the loss converges, and obtaining the trained multiple prior calibration network model; Inputting a test Raw image into the trained multiple prior calibration network model to obtain a complete demosaiced multispectral image through forward propagation.

2. The multi-spectral image demosaicking method based on multiple priori calibration network according to claim 1, characterized in that, In formula (1), mod represents the remainder. The multispectral images are collected from a public hyperspectral database, and each selected spectral image is sampled according to the required number of spectral bands to obtain high-resolution reference multispectral data GT; first, the multispectral image with C channels and full resolution is defined as GT, and then according to the actual spectral filter array structure, the multispectral data is simulated according to the multispectral filter array to obtain the corresponding Raw image; the Raw image is regarded as GT according to the pixel coordinates The multispectral image with C channels and full resolution is defined as GT, and then according to the actual spectral filter array structure, the multispectral data is simulated according to the multispectral filter array to obtain the corresponding Raw image; the Raw image is regarded as GT according to the pixel coordinates The result of spectral sampling is shown in formula (1): （1） The method comprises the following steps: According to the obtained GT and Raw images, data preprocessing is performed; the GT is cropped into a plurality of sub-image blocks , and the size of each sub-image block is ; according to formula (1), the corresponding sub-image block of the Raw image is obtained, the size of the Raw image is , the size of each sub-image block is , and T training sample pairs are generated in total.

3. The multi-spectral image demosaicking method based on multiple priori calibration network according to claim 1, characterized in that, The multiple prior calibration network model comprises a coarse demosaicing stage and a multiple prior calibration reconstruction stage: The spectral response function module, the global periodic attention module and the edge aggregation module comprise: The coarse demosaicking stage adopts a pseudo-panchromatic image interpolation network and weighted bilinear interpolation to perform interpolation reconstruction; the Raw image is input into a pseudo-panchromatic image interpolation network based on residual learning, filtered to generate a pseudo-panchromatic image ; meanwhile, the Raw image is filtered through four convolutional layers with different kernel sizes, and the output features of the convolutional layers are added together to obtain : （2） In formula (2), , , and is a four-layer convolution, denotes a convolution operator; the coefficient of is the sum of all filter elements; The different spectral bands are directly segmented using a segmentation method on the Raw image, and the missing spectral information is left blank to obtain a sparse image ; and then a convolution operation of weighted bilinear interpolation is used to estimate the multispectral image , i.e. （4） wherein is a filter; The multiple prior calibration reconstruction stage recalibrates the initial prediction by combining different prior knowledge; and and As the input of the multiple prior calibration reconstruction stage, the multiple prior calibration module is used to correct pixels and perfect the multispectral image, and the final output multispectral image is obtained after the multiple prior calibration reconstruction stage : （6） In formula (6), , and respectively represent the output of the spectral response function module, the global periodic attention module, and the edge aggregation module, represents the multiple prior calibration module.

4. The multi-spectral image demosaicking method based on multiple priori calibration network according to claim 3, characterized in that, The method comprises the following steps: The input to the spectral response function module is a sparse image. and interpolated images spliced ​​images The output is the spectral sensitivity function and content prior information. First, shallow features are extracted using six convolutional layers with symmetrical residual connections; each convolutional layer uses a kernel of the same size. The shallow features are then divided into four parts, and the gradient information of each part is calculated and then concatenated. Next, the features are input into three parallel convolutional layers, and their outputs are concatenated into a unified feature representation. At the same time, the pixel position information of the concatenated features is converted into a weight matrix, which is the estimated spectral sensitivity function. Finally, the weight matrix and the output image of the spectral response function module are resized and multiplied to obtain the final content prior information. The global periodic attention module consists of a feature extraction layer A residual block A global multi-head attention module and periodic attention blocks Composition; Output of the global cycle module Represented as （7） In formula (7), represents the input feature map of the global periodic attention module; Let As input to the global multi-head module, it is first reshaped into a tensor ; then, is linearly projected to the query matrix , the key matrix , the value matrix , and two auxiliary tensors and ; （8） In formula (8), , , , and represent learnable parameters, i.e., projection matrices for each component; Subsequently, the , , , and are divided along the spectral channel dimension into heads: , , , and ; each head has a dimension of ; each spectral representation in the attention map is considered as a token and the self-attention of is calculated: （9） （10） （11） In formula (9), denotes the transpose matrix of Introduce a learnable parameter Through the Internal matrix multiplication and Reweighting to adjust self-attention ; then, indivual The outputs are connected together and linearly projected; finally, the reshape is performed. The output feature map is obtained. ; （12） In the cycle attention block, first, an adaptive average pooling is adopted to aggregate Feature points at the same relative position on each channel: （13） wherein and ; is the size of the multispectral filter array; Then the Sigmoid function is used to obtain the mosaic attention map : （14） repeated in the spatial dimension , obtaining aggregated features : （15） Finally, by reweighting obtaining output features of the global periodic attention module ; （16） The edge aggregation module consists of two branches, whose inputs are the low-level representation mosaic image and the output image of the pseudo-panchromatic interpolation network Two convolution layers with kernel size 3x3 are used to increase the channel number for each input, denoted as: （17） （18） In formula (17), is a ReLU activation function, and represents a convolutional layer, and the weights are and ; After the outputs of the two branches are multiplied for convolution, selective weighted attention is obtained adaptively learns the weights of different branches: （19） By mutual weighting, more attention is focused on the important edge region; the two branch features are fused using residual aggregation and splicing operations, and then a convolution layer is used to obtain the edge prior value : （20） In the formula, represents a convolution operation with a kernel size of 3x3, and the weight size is ; represents a splicing operation; Finally, edge features are obtained by using a 1x1 convolutional layer and the Sobel operator .

5. The multi-spectral image demosaicking method based on multiple priori calibration network according to claim 1, characterized in that, The method comprises the following steps: The spatial consistency loss, the mean square error loss L2 loss and the edge loss of the multiple prior calibration network model are calculated, and the three are combined as a total loss function by weighting: Computing spatial consistency loss for a multi-prior calibration network model , as follows: （24） wherein denotes the spectral sensitivity function, denotes a parameter of the spectral sensitivity function, denotes the first vector representation of the true spectral reflectance at the pixel position of the is a non-linear function estimating the spectral reflectance; wherein is the first number of pixels of the is the number of images used; computing a multi-prior calibration network model loss, expressed as follows: （25） In formula (25), is the number of training samples, denotes the reference image, denotes the multispectral demosaicked image estimated by the method of the present application; Computing marginal loss for a multiple prior calibration network model , as follows: Adopting a root mean square error loss between a real edge image and an estimated edge image as an edge loss: （26） wherein denotes the real pseudo-panchromatic interpolated image, denotes the estimated pseudo-panchromatic interpolated image; combining the losses losses, losses and losses are combined to obtain a total loss of the multiple prior calibration network model , which is expressed as follows: （27）； Establishing original Nonlinear mapping relationship between the image and the reference multispectral image A multiple prior calibration network model is trained until the loss value reaches the minimum, and the structure and layer parameters of the trained multiple prior calibration network model are saved to obtain the established nonlinear mapping relationship.

6. The multi-spectral image demosaicking method based on multiple priori calibration network according to claim 1, characterized in that, The method comprises the following steps: The test Raw image is input into the trained multiple prior calibration network model, and the multispectral image of the test set is reconstructed to obtain a complete demosaiced multispectral image.

7. The multi-spectral image demosaicking method based on multiple priori calibration network according to claim 6, characterized in that, The performance is evaluated by using the peak signal-to-noise ratio, the structural similarity, the spectral angle distance, the root mean square error, the relative parameter estimation standard deviation and the visual effect of the reconstructed image as evaluation criteria.

8. A multispectral image demosaicking system based on multiple prior calibration networks, characterized in that, The method comprises the following steps: The data acquisition module is used to collect a multispectral image from a hyperspectral database, obtain a high-resolution reference multispectral image GT after spectral sampling, simulate sampling on the image GT to generate a Raw image, and cut the GT and the Raw image into multiple sub-image blocks to form a training sample pair; The model construction module is used to generate a pseudo-panchromatic image through a pseudo-panchromatic image interpolation network, and combine weighted bilinear interpolation to perform initial reconstruction on the Raw image to obtain an initial multispectral image, and perform multiple prior calibration on the initial multispectral image to obtain a multiple prior calibration network model; The training module is configured to calculate a loss function of the multiple-priori calibration network model, optimize network parameters through back propagation until the loss converges, and obtain the trained multiple-priori calibration network model. The output module is configured to input a test Raw image into the trained multiple-priori calibration network model and obtain a complete demosaiced multispectral image through forward propagation.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the steps of the multiple-priori calibration network-based multispectral image demosaicing method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the steps of the multiple-priori calibration network-based multispectral image demosaicing method according to any one of claims 1 to 7.

Images