Image reconstruction methods, apparatus, equipment, and media based on multispectral pixel decomposition.

Abstract

This invention provides an image reconstruction method, apparatus, device, and medium based on multispectral pixel decomposition, relating to the field of remote sensing image processing technology. The invention acquires multispectral images, performs dimensionality reduction and feature extraction on the images, and outputs a feature map. Then, the feature map is divided into several patches and flattened into sequence vectors. A prior knowledge bias matrix is ​​introduced to optimize attention weight calculation, capturing the global context and geoscientific prior association of the sequence vectors, outputting an enhanced global feature sequence. The global feature sequence is reconstructed to its original resolution, and an activation function is applied to the output channel to obtain an abundance map. An unbiased linear decoder is used to reconstruct the image from the abundance map, obtaining a reconstructed image. This application can integrate multi-source geographic environment priors and global spectral spatial context, improving unmixing accuracy and physical interpretability, and solving the uncertainty problem of endmember selection caused by spectral variability.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and more specifically, to an image reconstruction method, apparatus, device, and medium based on multispectral pixel decomposition. Background Technology

[0002] Mixed pixel decomposition is a key technology in multispectral and hyperspectral remote sensing image processing. Its core objective is to invert the endmember spectra and corresponding abundance proportions of an observed pixel. Due to the limitations of spatial resolution of remote sensing sensors and the complexity and diversity of land cover types, a single pixel often contains multiple land cover components, forming so-called mixed pixels. Traditional methods are mainly based on linear spectral mixture models (LSMM) or nonlinear mixture models, extracting endmembers and estimating abundance through constrained optimization. LSMM assumes that the pixel spectrum is a linear weighted sum of several endmember spectra and solves under physical constraints such as nonnegativity of abundance and a sum of one. However, these methods usually rely on the strong assumption of "pure" endmembers, which is difficult to satisfy in complex real-world scenarios. Furthermore, their ability to model nonlinear mixing effects is limited, and the endmember extraction process is susceptible to noise, atmospheric interference, and image quality, resulting in a lack of spatial continuity and physical consistency in the abundance map.

[0003] In recent years, with the development of deep learning, models such as autoencoders and convolutional autoencoders based on convolutional neural networks (CNNs) have been widely applied to hybrid pixel decomposition tasks. These models can automatically learn endmember features and abundance distributions from data, alleviating the dependence of traditional methods on prior assumptions to some extent. However, existing deep learning methods still have significant limitations: on the one hand, the inherent local receptive field of CNN structures makes it difficult to effectively model long-distance spatial dependencies, especially when dealing with large homogeneous areas or complex feature boundaries; on the other hand, although recent studies have attempted to introduce the Transformer architecture into demixing tasks to utilize its global modeling capabilities, these models only focus on the spectral-spatial information of the image itself, ignoring the inherent semantic relationships and geographical distribution patterns between feature categories, such as the prior knowledge that "roads" are often adjacent to "buildings" and that "water bodies" and "vegetation" have ecological coupling. This neglect of geoscientific semantic information makes the models prone to confusion when faced with features that are spectrally similar but have different geographical attributes, limiting their deep understanding and interpretability of the physical causes of hybrid pixels.

[0004] In view of the above, this application is hereby submitted. Summary of the Invention

[0005] The present invention aims to provide an image reconstruction method, device, equipment and medium based on multispectral pixel decomposition to solve the three major technical defects of existing mixed pixel decomposition methods in complex surface scenes: (1) Traditional linear unmixing models are highly dependent on the endmember purity assumption and are difficult to adapt to spectral variability; (2) Deep learning methods based on convolutional neural networks (CNN) are limited by local receptive fields and cannot effectively model long-distance spatial dependencies, resulting in insufficient unmixing accuracy for large-area mixed regions or complex boundaries; (3) Although the existing Transformer unmixing model has global modeling capabilities, it only uses the spectral-spatial information of the image itself and does not introduce the semantic association between land cover categories and the prior knowledge of the geographical environment, resulting in the output results lacking physical interpretability and geoscientific rationality.

[0006] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0007] An image reconstruction method based on multispectral pixel decomposition includes:

[0008] S1, acquire multispectral images;

[0009] S2, perform dimensionality reduction and feature extraction on the multispectral image, and output a feature map;

[0010] S3, the feature map is divided into several patches and flattened into sequence vectors. Then, a prior knowledge bias matrix is ​​introduced to optimize the attention weight calculation in order to capture the global context of the sequence vector and the association with geoscience priors, and output the enhanced global feature sequence.

[0011] S4, reshape the global feature sequence to the original resolution, and apply an activation function to the output channel to obtain an abundance map;

[0012] S5, the abundance map is reconstructed using a biasless linear decoder to obtain a reconstructed image.

[0013] Preferably, when performing dimensionality reduction and feature extraction on the multispectral image, a CNN encoder with three 1×1 convolutional layers is used to obtain the feature map after dimensionality reduction of the feature channels.

[0014] Preferably, the steps for constructing the prior knowledge bias matrix are as follows:

[0015] Constructing a geoscience knowledge graph ;

[0016] in, It is a collection of entities, including land cover type and geoscientific attributes; For a set of relations, using triples Formal representation of knowledge association; , These are the head entity and the tail entity, respectively; For relationship;

[0017] Group discrete sampled data according to land cover type;

[0018] A Gaussian model was used to fit the numerical distribution of the geoscientific attributes corresponding to each type of endmember;

[0019] Initialize the embedding vectors of all entities and the embedding vectors of all relations;

[0020] For any triplet in the knowledge graph The embedding vector must satisfy the following constraints:

[0021] ;

[0022] in, For the head entity embedding vector; For relation embedding vectors; For tail entity embedding vectors;

[0023] The training is optimized by minimizing the distance between positive triples and maximizing the difference between positive and negative distances to obtain the embedded vectors of entities and relations after training.

[0024] For any pixel i in the multispectral image, the corresponding geoscientific auxiliary data is extracted, and an attribute embedding vector of the pixel is constructed based on the Gaussian probability distribution and the trained entity and relation embedding vectors. ;

[0025] Calculate the attribute embedding vector of pixel i With the Standard embedding vectors of endmembers Euclidean distance between The formula is:

[0026] ;

[0027] in, This is the Euclidean distance, i.e., the L2 norm; The larger the value, the more likely pixel i belongs to the endmember category. The lower the probability;

[0028] Euclidean distance Mapped to pixel i and endmember similarity probability ;

[0029] Based on similarity probability Calculate the penalty weights of pixel endmembers The formula is:

[0030] ;

[0031] in, This is a balancing parameter used to adjust the intensity of the penalty. This represents the maximum probability of endmember similarity across all pixels. This is a local minimum value, used to avoid the denominator being 0;

[0032] The penalty weights of all pixels are integrated according to the image spatial resolution and the total number of endmember categories to generate a prior knowledge penalty weight matrix. ;

[0033] Then, through dimensional transformation, it is reshaped into a pixel endmember matrix. ,Right now ,in, For pixel i, the pixel endmember; Let i be the two-dimensional coordinates of pixel i;

[0034] Based on pixel endmember matrix For any two pixels m and n, calculate their weight similarity in the endmember dimension, and use it as an element of the bias matrix.

[0035] Normalize the bias matrix to obtain the prior knowledge bias matrix.

[0036] Preferably, the formula for optimizing attention weights by introducing a prior knowledge bias matrix is ​​as follows:

[0037] ;

[0038] ;

[0039] in, Attention weights; For activation functions; , These are the query matrix and the key matrix, respectively, obtained by linear transformation of sequence vectors. , These are the query and key projection weight matrices for the i-th head, respectively; The dimension of the attention head is used for scaling to prevent gradient explosion; This is a hyperparameter used to control the strength of the influence of prior knowledge; This is the prior knowledge bias matrix; for Element; , These are the prior weight vectors for pixels m and n, respectively; This is the transpose symbol.

[0040] Preferably, the abundance map is calculated by processing each pixel in the reshaped global feature sequence along R endmember channel dimensions. Furthermore, it strictly satisfies the abundance sum to one constraint and the abundance non-negativity constraint, and its expression is:

[0041] ;

[0042] ;

[0043] ;

[0044] in, The position of the i-th pixel corresponds to the first pixel. Abundance values ​​of each endmember; Abundance plot; The total number of end-member categories; The reshaped global feature sequence The i-th pixel in Each terminal element; , This is an index for the end-member sequence number.

[0045] Preferably, the abundance map is reconstructed using a biasless linear decoder, specifically as follows:

[0046] For any pixel i, its reconstructed spectrum is obtained by a linear combination of the endmember matrix and the abundance vector at the corresponding position, as shown in the formula:

[0047] ;

[0048] in, Let be the reconstructed spectral vector of the i-th pixel; This is the endmember matrix, which is the linear convolutional layer weight matrix of the decoder; Let be the abundance vector of pixel i;

[0049] By iterating through all pixel locations, a complete reconstructed image is obtained.

[0050] Preferably, it further includes: designing a composite total loss function, embedding geoscience prior knowledge into the Transformer unmixing network in the form of a differentiable regularization term for training;

[0051] The composite total loss function is composed of a weighted average of reconstruction loss, spectral angular distance loss, and knowledge graph prior constraint loss, and the formula is as follows:

[0052] ;

[0053] ;

[0054] ;

[0055] ;

[0056] in, This is the composite total loss function; , These are the weighting coefficients; The reconstruction loss is used to constrain the difference between the reconstructed spectrum and the observed spectrum; For spectral angular distance loss; For prior constraint loss of knowledge graph; The total number of pixels in the image; , The height and width of the image; Let be the observed spectral vector of the i-th pixel; Let be the reconstructed spectral vector of the i-th pixel; This is the Euclidean distance, i.e., the L2 norm; The total number of end-member categories; For the first Initial endmember spectral vector of the class; The first learned by the Transformer decoder Initial endmember spectral vector of the class; Prior knowledge penalty weight matrix The i-th pixel corresponds to the i-th pixel. Penalty weight for class-endmembers; To predict the position of the i-th pixel in the abundance map corresponding to the i-th pixel position Abundance values ​​of each endmember; It is the inverse cosine function;

[0057] During training, initialize the Transformer encoder and decoder weights and endmember matrices. Weighting coefficients and Then, forward and backward propagation calculations are performed until the total loss converges, yielding the optimal unmixed model.

[0058] The present invention also provides an image reconstruction apparatus based on multispectral pixel decomposition, comprising:

[0059] Acquisition unit, used to acquire multispectral images;

[0060] The feature extraction unit is used to perform dimensionality reduction and feature extraction on the multispectral image and output a feature map.

[0061] The prior perception unit is used to divide the feature map into several patches and flatten them into sequence vectors. Then, a prior knowledge bias matrix is ​​introduced to optimize the attention weight calculation in order to capture the global context of the sequence vectors and the association with geoscience priors, and output the enhanced global feature sequence.

[0062] The abundance calculation unit is used to reshape the global feature sequence to the original resolution and apply an activation function to the output channel to obtain an abundance map;

[0063] The reconstruction unit is used to reconstruct the abundance map using a biasless linear decoder to obtain a reconstructed image.

[0064] The present invention also provides an image reconstruction device based on multispectral pixel decomposition, including a processor and a memory, wherein the memory stores a computer program that can be executed by the processor to implement the image reconstruction method based on multispectral pixel decomposition as described above.

[0065] The present invention also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor of the device on which the computer-readable storage medium is located, implement an image reconstruction method based on multispectral pixel decomposition as described above.

[0066] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0067] This invention constructs a knowledge-aware self-attention mechanism (KASA) by embedding a geoscience knowledge graph into a Transformer unmixing architecture. This mechanism enables the model to consider not only spectral similarity when calculating attention weights between pixels, but also semantic similarity bias derived from geoscience attributes such as elevation, slope, and NDVI. This allows for the priority aggregation of non-local pixels with consistent geoscience contexts globally, effectively mitigating the interference of heterospectral phenomena on unmixing and solving the uncertainty problem of endmember selection caused by spectral variability.

[0068] After reshaping the feature sequence, this invention directly applies the Softmax activation function along the endmember channel, and the generated abundance map naturally satisfies two core physical constraints, eliminating post-processing steps and ensuring the accuracy and rigor of abundance estimation.

[0069] This invention employs a lightweight CNN encoder to reduce the spectral dimension during the preprocessing stage, significantly reducing the number of parameters required for subsequent attention calculations. At the same time, it uses a biasless linear decoder to avoid the accuracy loss caused by complex nonlinear transformations, achieving a balance between "low computational cost and high reconstruction accuracy".

[0070] This invention designs a knowledge-guided composite total loss function (reconstruction loss + spectral angular distance loss + knowledge prior loss) to transform multi-source geographic environmental information into differentiable weighted sparse regularization terms. During training, gradient penalties are applied to abundance predictions that violate geoscientific logic, so that the model output has clear physical interpretability and reasonable geographic distribution.

[0071] This invention enables the fusion and probabilistic representation of heterogeneous geoscientific data, enhancing the model's adaptability to spectral variability and complex surface environments.

[0072] In summary, this invention provides a novel hybrid pixel decomposition paradigm that integrates knowledge-driven and data-driven approaches, offering a technical solution with both high accuracy and high reliability for quantitative interpretation of remote sensing images. Attached Figure Description

[0073] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0074] Figure 1 This is a schematic diagram of an image reconstruction method based on multispectral pixel decomposition provided in Example 1.

[0075] Figure 2 This is a schematic diagram of the overall architecture of an image reconstruction method based on multispectral pixel decomposition, provided in Example 1.

[0076] Figure 3 This is a schematic diagram illustrating the principle of the Knowledge Awareness Self-Attention (KASA) mechanism provided in Example 1.

[0077] Figure 4 The diagram illustrates the principle of the knowledge-guided loss function provided in Example 1.

[0078] Figure 5 This is a schematic diagram of an image reconstruction device based on multispectral pixel decomposition provided in Embodiment 2.

[0079] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0081] Example 1

[0082] Embodiment 1 of the present invention provides an image reconstruction method based on multispectral pixel decomposition, which can be implemented by an image reconstruction device based on multispectral pixel decomposition (hereinafter referred to as image reconstruction device), specifically, executed by one or more processors within the image reconstruction device.

[0083] In this embodiment, the image reconstruction device may be an electronic device equipped with a processor, the processor having a computer program for the image reconstruction method based on multispectral pixel decomposition and the computer program being executable, such as a computer, smartphone, smart tablet, workstation, etc., without limitation.

[0084] This invention aims to overcome the shortcomings of existing technologies by designing an image reconstruction model based on multispectral pixel decomposition that combines knowledge graphs and Transformer networks. By introducing ground feature knowledge graph embedding, the semantic relationships between ground features are integrated into the decomposition process, enhancing the physical interpretability of the model. Simultaneously, the self-attention mechanism of the Transformer is used to globally model the spatial-spectral context information of the image, improving the decomposition accuracy for complex mixed pixels. Ultimately, high-precision and robust inversion of endmember spectra and abundance maps is achieved.

[0085] like Figures 1-2 As shown, an image reconstruction method based on multispectral pixel decomposition includes steps S1 to S5.

[0086] S1, acquire multispectral images.

[0087] Acquire or read multispectral images to be processed, and then perform preprocessing operations such as radiometric calibration and atmospheric correction on the original images to eliminate noise and systematic errors, ensuring the accuracy of subsequent feature extraction.

[0088] S2, dimensionality reduction and feature extraction are performed on the multispectral image, and a feature map is output.

[0089] This step uses a CNN encoder to reduce the dimensionality of the preprocessed image, extract high-level semantic features, and reduce computational complexity.

[0090] When performing dimensionality reduction and feature extraction on the multispectral image, a lightweight CNN encoder with three 1×1 convolutional layers is used to obtain the feature map after dimensionality reduction of the feature channels, so as to significantly reduce the number of parameters in subsequent attention calculation while retaining key semantic features.

[0091] S3, the feature map is divided into several patches and flattened into sequence vectors. Then, a prior knowledge bias matrix is ​​introduced to optimize the attention weight calculation in order to capture the global context of the sequence vectors and the association with geoscience priors, and output the enhanced global feature sequence.

[0092] This step transforms feature maps into sequence vectors and introduces a bias matrix of geoscientific prior knowledge to optimize attention weights, thereby achieving dual feature enhancement based on global context and prior association.

[0093] First, the prior knowledge bias matrix is ​​constructed, following these steps:

[0094] Constructing a geoscience knowledge graph ;

[0095] in, It is a collection of entities, including land cover type and geoscientific attributes; For a set of relations, using triples Formal representation of knowledge association; , These are the head entity and the tail entity, respectively; For the relationship.

[0096] Group discrete sampled data according to land cover type;

[0097] A Gaussian model was used to fit the numerical distribution of the geoscientific attributes corresponding to each type of endmember;

[0098] Initialize the embedding vectors for all entities and the embedding vectors for all relations.

[0099] For continuous geoscientific data such as NDVI values ​​and elevation, we use a Gaussian model to fit the distribution of attribute values ​​for different land cover types, transforming discrete sampled data into a continuous probability distribution expression, thereby solving the problem of inconsistent data dimensions.

[0100] The TransE model is used to map entities and relations to a low-dimensional, continuous vector space. For any triple in the knowledge graph... The embedding vector must satisfy the following constraints:

[0101] ;

[0102] in, For the head entity embedding vector; For relation embedding vectors; This is the tail entity embedding vector.

[0103] The training is optimized by minimizing the distance between positive triples and maximizing the difference between the distances between positive and negative examples, resulting in the embedded vectors of entities and relations after training.

[0104] For any pixel i in the multispectral image, the corresponding geoscientific auxiliary data is extracted, and an attribute embedding vector of the pixel is constructed based on the Gaussian probability distribution and the trained entity and relation embedding vectors. .

[0105] Calculate the attribute embedding vector of pixel i With the Standard embedding vectors of endmembers Euclidean distance between The formula is:

[0106] ;

[0107] in, This is the Euclidean distance, i.e., the L2 norm; The larger the value, the more likely pixel i belongs to the endmember category. The lower the probability, the less likely it is.

[0108] Euclidean distance Mapped to pixel i and endmember similarity probability The smaller the distance, the higher the probability of similarity. For example, mapping can be performed using exponential functions, normalized reciprocals, or linear normalized mapping functions.

[0109] Based on similarity probability Calculate the penalty weights of pixel endmembers The formula is:

[0110] ;

[0111] in, This is a balancing parameter used to adjust the intensity of the penalty. This represents the maximum probability of endmember similarity across all pixels. This is a local minimum value, used to avoid a denominator of 0.

[0112] The penalty weights of all pixels are integrated according to the image spatial resolution and the total number of endmember categories to generate a prior knowledge penalty weight matrix. .

[0113] Then, through dimensional transformation, it is reshaped into a pixel endmember matrix. ,Right now ,in, For pixel i, the pixel endmember; Let i be the two-dimensional coordinates of pixel i.

[0114] Based on pixel endmember matrix For any two pixels m and n, calculate their weight similarity in the endmember dimension, and use it as an element of the bias matrix.

[0115] Normalize the bias matrix to obtain the prior knowledge bias matrix.

[0116] After dividing the feature map into fixed-size patches and flattening them into one-dimensional sequence vectors, in order to capture global contextual information, such as... Figure 3 As shown, we introduce the proposed Knowledge-Aware Self-Attention (KASA) method, which incorporates a bias matrix constructed from prior knowledge when calculating attention weights. The query matrix Q and key matrix K are generated based on sequence vectors, and the attention weights are calculated in conjunction with the prior bias matrix.

[0117] The formula for optimizing attention weights by introducing a prior knowledge bias matrix is ​​as follows:

[0118] ;

[0119] ;

[0120] in, Attention weights; For activation functions; , These are the query matrix and the key matrix, respectively, obtained by linear transformation of sequence vectors. , These are the query and key projection weight matrices for the i-th head, respectively; The dimension of the attention head is used for scaling to prevent gradient explosion; This is a hyperparameter used to control the strength of the influence of prior knowledge; This is the prior knowledge bias matrix; for Element; , These are the prior weight vectors for pixels m and n, respectively (i.e., the weight similarity between the two in the endmember dimension, which includes the embedding information of the geographic attributes of the pixel). This is the transpose symbol.

[0121] Attention weights are applied to the value matrix, and the output is an enhanced global feature sequence.

[0122] This mechanism enables the model to capture spectral similarity while paying more attention to pixel pairs that are more likely to belong to homogeneous regions in terms of geoscientific priors. Prior bias implicitly guides the model to select endmember combinations that conform to spatial distribution patterns by enhancing the attention weights between corresponding pixels of semantically related endmembers (such as the strong association between building and road pixels in urban areas). For example, for pixels in coastal cities, It can strengthen its association with typical urban end-members (such as high-illuminance land features) and water end-members, while weakening the false connections with vegetation end-members.

[0123] S4, reshape the global feature sequence to the original resolution, and apply an activation function to the output channel to obtain an abundance map.

[0124] This step restores the enhanced feature sequence to the original spatial resolution and constrains the abundance values ​​through activation functions to meet the physical constraints of pixel decomposition.

[0125] The one-dimensional global feature sequence is reshaped into a three-dimensional spatial feature map. If there is a downsampling operation in the encoding stage, it needs to be upsampled to the original resolution through interpolation (such as bilinear interpolation).

[0126] The abundance map is calculated by processing each pixel in the reshaped global feature sequence along R endmember channel dimensions. And it strictly satisfies the abundance sum-of-ones constraint (ASC) and the abundance nonnegativity constraint (ANC), as expressed in:

[0127] ;

[0128] ;

[0129] ;

[0130] in, The position of the i-th pixel corresponds to the first pixel. Abundance values ​​of each endmember; Abundance plot; The total number of end-member categories; The reshaped global feature sequence The i-th pixel in Each terminal element.

[0131] S5, the abundance map is reconstructed using a biasless linear decoder to obtain a reconstructed image.

[0132] This step takes the abundance map as input and completes the reconstruction of the multispectral image through an unbiased linear decoder. The decoder weights are the endmember matrix, and each column of the decoder corresponds to the spectral response vector of a class of endmembers.

[0133] Image reconstruction of the abundance map is performed using an unbiased linear decoder, specifically as follows:

[0134] For any pixel i, its reconstructed spectrum is obtained by a linear combination of the endmember matrix and the abundance vector at the corresponding position, as shown in the formula:

[0135] ;

[0136] in, Let be the reconstructed spectral vector of the i-th pixel; This is the endmember matrix, which is the linear convolutional layer weight matrix of the decoder; Let be the abundance vector of pixel i;

[0137] By traversing all pixel locations, a complete reconstructed image with the same resolution and number of bands as the original image is obtained.

[0138] like Figure 4 As shown, in a preferred embodiment, the method further includes: designing a composite total loss function, embedding geoscience prior knowledge into the Transformer unmixing network in the form of a differentiable regularization term for training.

[0139] Wherein, the composite total loss function The loss is composed of a weighted average of reconstruction loss, spectral angular distance loss, and knowledge graph prior constraint loss, and the formula is as follows:

[0140] ;

[0141] ;

[0142] ;

[0143] ;

[0144] in, This is the composite total loss function; , These are the weighting coefficients; The reconstruction loss is used to constrain the difference between the reconstructed spectrum and the observed spectrum; For spectral angular distance loss; For prior constraint loss of knowledge graph; The total number of pixels in the image; , The height and width of the image; Let be the observed spectral vector of the i-th pixel; Let be the reconstructed spectral vector of the i-th pixel; This is the Euclidean distance, i.e., the L2 norm; The total number of end-member categories; For the first Initial endmember spectral vector of the class; The first learned by the Transformer decoder Initial endmember spectral vector of the class; Prior knowledge penalty weight matrix The i-th pixel corresponds to the i-th pixel. Penalty weight for class-endmembers; To predict the position of the i-th pixel in the abundance map corresponding to the i-th pixel position Abundance values ​​of each endmember; It is the inverse cosine function;

[0145] During training, initialize the Transformer encoder and decoder weights and endmember matrices. Weighting coefficients and Then, forward and backward propagation calculations are performed until the total loss converges, yielding the optimal unmixed model.

[0146] If knowledge graph reasoning indicates pixels Belongs to end element The probability is very low (leading to a low penalty weight). (Very large), and the network predicted a high abundance value. This would result in a significant loss. Through backpropagation, this constraint forces the Transformer network to suppress the abundance of endmembers that are physically inconsistent with prior geoscience knowledge, thereby achieving knowledge-guided precise unmixing. By minimizing... The network can simultaneously optimize spectral reconstruction accuracy, endmember spectral fidelity, and consistency with geoscience knowledge.

[0147] In another preferred embodiment, in order to verify the superiority of the present invention, experiments were conducted using real data.

[0148] The dataset consists of Sentinel-2 imagery acquired on January 27, 2016, using Level-2A imagery, atmospherically corrected, with a spatial resolution of 10m. 10m. Select an image containing an island and part of the land area. Crop to 5600. 5600-pixel image. Landsat 8 imagery acquired on February 15, 2021, atmospherically corrected, selected from images containing an island and part of the land area, with a spatial resolution of 30m. 30m.

[0149] First, based on the constructed geoscience knowledge graph triples The TransE model is used for embedding training. Low-dimensional embedding vectors for entities and relations are obtained by minimizing the L2 norm-based scoring function. Subsequently, for the multispectral image to be demixed, the Euclidean distance between the attribute embedding vector and the standard vector of each endmember is calculated based on the auxiliary attribute data (such as NDVI, elevation, etc.) of each pixel, generating a global prior penalty matrix. This matrix remains frozen during subsequent network training, serving as a static hard constraint mask.

[0150] To accelerate the convergence of the Transformer network and avoid local optima, Vertex Component Analysis (VCA) is used to initialize the weights of the decoder. Initial endmember spectra are extracted from the original multispectral image using VCA and assigned to the weights of the linear convolutional layers in the decoder. This initialization strategy provides the network with a physically clear starting point, allowing the Transformer encoder to focus more on refined abundance regression.

[0151] During the training phase, we synchronously input multispectral image patches and their corresponding prior weight matrix patches into the network. This is then processed using a composite total loss function. The update is performed using the Adam optimizer with stochastic gradient descent. During backpropagation, the loss is reconstructed. The driving network learns spectral features, and the SAD loss is used. Constrained endmember shape, while prior knowledge loss. Then according to The instructions apply a penalty gradient to abundance predictions that do not conform to geoscientific laws, forcing the network output to meet both physical and geographical constraints.

[0152] The training process employs a dynamic learning rate adjustment strategy, with the initial learning rate set at [value missing]. The regularization parameter gradually decreases with increasing epoch to ensure stability in later training stages. and The parameters are set according to sensitivity analysis to balance the weights of data fidelity terms and prior constraint terms.

[0153] Experiments show that in the water-mangrove transition zone, traditional CNN methods misclassify some water bodies as vegetation due to local receptive field limitations. This invention, however, correctly identifies low-elevation, high-NDVI areas as mangroves and high-elevation, low-NDVI areas as building clusters by combining elevation and NDVI priors through the KASA mechanism. In densely built-up areas, due to severe spectral confusion, pure Transformer models often mix rooftops with roads. This invention utilizes the adjacency relationship between "buildings" and "roads" in the knowledge graph, strengthening their association in the prior bias matrix Ω, resulting in a reasonable spatial distribution of the abundance map.

[0154] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0155] This invention significantly improves unmixing accuracy under spectral variability conditions. Existing techniques (such as CNNs) are limited by local receptive fields and struggle to handle spectral variability. This invention utilizes a Transformer structure to acquire global features and innovatively designs a knowledge-aware self-attention mechanism (KASA). This mechanism, by introducing a bias matrix constructed from prior knowledge, forces the network to focus on pixel pairs that are "spectrally similar and have consistent geoscientific properties." This enables the model to accurately distinguish endmembers with similar spectral features based on the geographic context (e.g., distinguishing the same vegetation at different altitudes), effectively reducing confusion errors.

[0156] This invention overcomes the blindness and uncertainty of purely data-driven methods. Existing deep learning demixing methods typically perform end-to-end mapping as a "black box," lacking physical constraints. This invention constructs a knowledge-guided loss function, applying pixel-level penalty weights inferred from the knowledge graph. This directly affects the gradient update process of the network. This is equivalent to introducing a "geoscience expert" into the neural network, which corrects predictions that do not conform to geoscientific logic in real time during training, thereby ensuring the rationality of the mixed abundance map in terms of physical meaning and geographical distribution.

[0157] This invention improves the model's adaptability to complex surface environments. It utilizes a knowledge graph (TransE) to process multi-source heterogeneous data (such as elevation, temperature, and slope) and transforms it into a unified embedded representation. This means the model is not merely "seeing" the spectrum, but "understanding" the environment. This multi-source information fusion mechanism enables this invention to exhibit stronger robustness and generalization ability than single-spectrum-dependent methods in multispectral image unmixing tasks involving complex terrain and diverse land cover types.

[0158] Example 2

[0159] like Figure 5 As shown, the second embodiment of the present invention also provides an image reconstruction apparatus based on multispectral pixel decomposition, comprising:

[0160] Acquisition unit, used to acquire multispectral images;

[0161] The feature extraction unit is used to perform dimensionality reduction and feature extraction on the multispectral image and output a feature map.

[0162] The prior perception unit is used to divide the feature map into several patches and flatten them into sequence vectors. Then, a prior knowledge bias matrix is ​​introduced to optimize the attention weight calculation in order to capture the global context of the sequence vectors and the association with geoscience priors, and output the enhanced global feature sequence.

[0163] The abundance calculation unit is used to reshape the global feature sequence to the original resolution and apply an activation function to the output channel to obtain an abundance map;

[0164] The reconstruction unit is used to reconstruct the abundance map using a biasless linear decoder to obtain a reconstructed image.

[0165] Example 3

[0166] The third embodiment of the present invention also provides an image reconstruction device based on multispectral pixel decomposition, which includes a memory and a processor. The memory stores a computer program that can be executed by the processor to implement the image reconstruction method based on multispectral pixel decomposition as described above.

[0167] Example 4

[0168] The fourth embodiment of the present invention also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor of the device on which the computer-readable storage medium is located, implement the image reconstruction method based on multispectral pixel decomposition as described above.

[0169] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image reconstruction method based on multispectral pixel decomposition, characterized in that, include: Acquire multispectral images; The multispectral image is subjected to dimensionality reduction and feature extraction to output a feature map; The feature map is divided into several patches and flattened into sequence vectors. Then, a prior knowledge bias matrix is ​​introduced to optimize the attention weight calculation in order to capture the global context of the sequence vectors and the association with geoscience priors, and output the enhanced global feature sequence. The global feature sequence is reshaped to its original resolution, and an activation function is applied to the output channel to obtain an abundance map. The abundance map is reconstructed using an unbiased linear decoder to obtain the reconstructed image; In the process of dimensionality reduction and feature extraction of the multispectral image, a CNN encoder with three 1×1 convolutional layers is used to obtain the feature map after dimensionality reduction of the feature channels. The formula for optimizing attention weights by introducing a prior knowledge bias matrix is ​​as follows: ； ； in, Attention weights; For activation functions; , These are the query matrix and the key matrix, respectively, obtained by linear transformation of sequence vectors. , These are the query and key projection weight matrices for the i-th head, respectively; The dimension of the attention head is used for scaling to prevent gradient explosion; This is a hyperparameter used to control the strength of the influence of prior knowledge; This is the prior knowledge bias matrix; for Element; , These are the prior weight vectors for pixels m and n, respectively; It is the transpose symbol; The abundance map is calculated by processing each pixel in the reshaped global feature sequence along R endmember channel dimensions. Furthermore, it strictly satisfies the abundance sum to one constraint and the abundance non-negativity constraint, and its expression is: ； ； ； in, The position of the i-th pixel corresponds to the first pixel. Abundance values ​​of each endmember; Abundance plot; The total number of end-member categories; The reshaped global feature sequence The i-th pixel in Each terminal element; , Indexed by end-member sequence number; Image reconstruction of the abundance map is performed using an unbiased linear decoder, specifically as follows: For any pixel i, its reconstructed spectrum is obtained by a linear combination of the endmember matrix and the abundance vector at the corresponding position, as shown in the formula: ； in, Let be the reconstructed spectral vector of the i-th pixel; This is the endmember matrix, which is the linear convolutional layer weight matrix of the decoder; Let be the abundance vector of pixel i; By iterating through all pixel locations, a complete reconstructed image is obtained.

2. The image reconstruction method based on multispectral pixel decomposition according to claim 1, characterized in that... The steps for constructing the prior knowledge bias matrix are as follows: Constructing a geoscience knowledge graph ; in, It is a collection of entities, including land cover type and geoscientific attributes; For a set of relations, using triples Formal representation of knowledge association; , These are the head entity and the tail entity, respectively; For relationship; Group discrete sampled data according to land cover type; A Gaussian model was used to fit the numerical distribution of the geoscientific attributes corresponding to each type of endmember; Initialize the embedding vectors of all entities and the embedding vectors of all relations; For any triplet in the knowledge graph The embedding vector must satisfy the following constraints: ； in, For the head entity embedding vector; For relation embedding vectors; For tail entity embedding vectors; The training is optimized by minimizing the distance between positive triples and maximizing the difference between positive and negative distances to obtain the embedded vectors of entities and relations after training. For any pixel i in the multispectral image, the corresponding geoscientific auxiliary data is extracted, and an attribute embedding vector of the pixel is constructed based on the Gaussian probability distribution and the trained entity and relation embedding vectors. ; Calculate the attribute embedding vector of pixel i With the Standard embedding vectors of endmembers Euclidean distance between The formula is: ； in, This is the Euclidean distance, i.e., the L2 norm; The larger the value, the more likely pixel i belongs to the endmember category. The lower the probability; Euclidean distance Mapped to pixel i and endmember similarity probability ; Based on similarity probability Calculate the penalty weights of pixel endmembers The formula is: ； in, This is a balancing parameter used to adjust the intensity of the penalty. This represents the maximum probability of endmember similarity across all pixels. This is a local minimum value, used to avoid the denominator being 0; The penalty weights of all pixels are integrated according to the image spatial resolution and the total number of endmember categories to generate a prior knowledge penalty weight matrix. ; Then, through dimensional transformation, it is reshaped into a pixel endmember matrix. ,Right now ,in, For pixel i, the pixel endmember; Let i be the two-dimensional coordinates of pixel i; Based on pixel endmember matrix For any two pixels m and n, calculate their weight similarity in the endmember dimension, and use it as an element of the bias matrix. Normalize the bias matrix to obtain the prior knowledge bias matrix.

3. The image reconstruction method based on multispectral pixel decomposition according to claim 1, characterized in that... It also includes: designing a composite total loss function, embedding geoscience prior knowledge into the Transformer unmixing network in the form of differentiable regularization terms for training; The composite total loss function is composed of a weighted average of reconstruction loss, spectral angular distance loss, and knowledge graph prior constraint loss, and the formula is as follows: ； ； ； ； in, This is the composite total loss function; , These are the weighting coefficients; The reconstruction loss is used to constrain the difference between the reconstructed spectrum and the observed spectrum; For spectral angular distance loss; For prior constraint loss of knowledge graph; The total number of pixels in the image; , The height and width of the image; Let be the observed spectral vector of the i-th pixel; Let be the reconstructed spectral vector of the i-th pixel; This is the Euclidean distance, i.e., the L2 norm; The total number of end-member categories; For the first Initial endmember spectral vector of the class; The first learned by the Transformer decoder Initial endmember spectral vector of the class; Prior knowledge penalty weight matrix The i-th pixel corresponds to the i-th pixel. Penalty weight for class-endmembers; To predict the position of the i-th pixel in the abundance map corresponding to the i-th pixel position Abundance values ​​of each endmember; It is the inverse cosine function; During training, initialize the Transformer encoder and decoder weights and endmember matrices. Weighting coefficients and Then, forward and backward propagation calculations are performed until the total loss converges, yielding the optimal unmixed model.

4. An image reconstruction apparatus based on multispectral pixel decomposition, used to implement the image reconstruction method based on multispectral pixel decomposition as described in any one of claims 1-3, characterized in that, include: Acquisition unit, used to acquire multispectral images; The feature extraction unit is used to perform dimensionality reduction and feature extraction on the multispectral image and output a feature map. The prior perception unit is used to divide the feature map into several patches and flatten them into sequence vectors. Then, a prior knowledge bias matrix is ​​introduced to optimize the attention weight calculation in order to capture the global context of the sequence vectors and the association with geoscience priors, and output the enhanced global feature sequence. The abundance calculation unit is used to reshape the global feature sequence to the original resolution and apply an activation function to the output channel to obtain an abundance map; The reconstruction unit is used to reconstruct the abundance map using a biasless linear decoder to obtain a reconstructed image.

5. An image reconstruction device based on multispectral pixel decomposition, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that can be executed by the processor to implement an image reconstruction method based on multispectral pixel decomposition as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor of the device on which the computer-readable storage medium resides, implement an image reconstruction method based on multispectral pixel decomposition as described in any one of claims 1-3.

Images