Thangka pigment distribution extraction method based on end-to-end hyperspectral unmixing
By constructing a dual-stream multi-branch CNN encoder and a Transformer encoder, combined with the spectral angular distance loss function, the problems of low accuracy and incomplete classification of Thangka pigment distribution in traditional methods are solved, and a more accurate pigment distribution extraction effect is achieved.
Patent Information
- Application Number
- CN202411557941.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-04
- Publication Date
- 2024-11-29
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, the Thangka pigment distribution extraction method based on the traditional end element extraction algorithm is limited by the convex geometry theory, and the pigment spectral extraction accuracy is limited, and there will be incomplete classification phenomenon simply by relying solely on spectral angular distance classification, making it difficult to effectively solve the mixed image element. question.
Using end-to-end hyperspectral demixing method, the spectral features and multi-scale spatial features of Thangka hyperspectral images are extracted by constructing a dual-stream multi-branch CNN encoder and a Transformer encoder, and the images are reconstructed through feature fusion and decoder. The extraction effect is optimized using the spectral angular distance loss function.
It significantly improves the accuracy of Thangka pigment distribution and extraction, especially the ability to identify Thangka with different pigment regions and strong heterogeneity distribution, improving the accuracy of feature fusion and abundance estimation.
Smart Images

Figure CN119048524A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image technology, and in particular to a thangka pigment distribution extraction method based on end-to-end hyperspectral unmixing. Background Art
[0002] Thangka is a cultural resource with special ethnic characteristics, but there are few studies on the protection and development of Thangka in combination with information technology. The existing technology is a method for extracting pigment distribution in Thangka based on hyperspectral remote sensing. First, the spectral information of different pigments is extracted based on the traditional end-member extraction algorithm, and then the spectral angle mapper (SAM) and other algorithms are used to classify the image of the pigment of the whole Thangka to obtain the corresponding pigment distribution map; however, the traditional end-member extraction algorithm is limited by convex geometry theory, and the accuracy of pigment spectrum extraction is limited. In addition, considering the problem of mixed pixels, incomplete classification will occur if classification is performed solely based on spectral angle distance; therefore, a method for extracting pigment distribution in Thangka based on end-to-end hyperspectral unmixing is urgently needed to solve the above problems. Summary of the invention
[0003] The purpose of the present invention is to provide a thangka pigment distribution extraction method based on end-to-end hyperspectral unmixing, which can effectively solve the problems existing in the above-mentioned prior art.
[0004] In order to solve the above technical problems, the present invention adopts the following technical solution: a thangka pigment distribution extraction method based on end-to-end hyperspectral unmixing, comprising the following steps: S1. Construct a dual-stream multi-branch CNN encoder, including a spectral stream composed of multiple modules and a spatial stream of several multi-scale spatial feature extraction branches, and simultaneously extract the spectral features and multi-scale spatial features of the input thangka hyperspectral image; S2, feature fusion of spectral features and multi-scale spatial features through Transformer encoder; S3, an abundance map obtained by using a decoder using a ReLU activation function and a normalization operation; and The abundance map is converted into reconstructed thangka hyperspectral image data through a convolutional layer.
[0005] Preferably, in step S1, the spectral stream includes three modules, each module includes a 1×1 two-dimensional convolution layer, which gradually compresses the input thangka hyperspectral image in the spectral dimension, and the three modules are batch normalized after the convolution layer to ensure the stability of training.
[0006] Preferably, a dropout layer is added after the batch normalization layer in the first module in the spectral stream to alleviate the gradient vanishing problem during network training, and before outputting the first two modules, the data is activated by a Leaky ReLU function to introduce nonlinearity.
[0007] Preferably, in step S1, the spatial stream includes three multi-scale spatial feature extraction branches, specifically: The main branch includes three modules, and the convolution kernel size of each module is adjusted to 5×5, 3×3 and 3×3. The main branch extracts large-scale spatial features; The upper branch receives the output of the second module of the main branch as input and extracts the mesoscale spatial features; The lower branch receives the output of the first module of the main branch as input and extracts small-scale spatial features.
[0008] Multi-scale information comes from different modules within the same branch. This block-by-block branching strategy not only ensures the correlation between multi-scale features and feature alignment, but also reduces the computational complexity of model training.
[0009] Preferably, the Transformer encoder comprises two sequentially stacked combinations, each of which comprises a multi-head self-scaling aggregation attention module and a scaled multilayer perceptron module; the scaled multilayer perceptron module comprises two 1×1 convolutional layers and a Gaussian error linear unit activation function.
[0010] Preferably, the multi-head self-scaling aggregation attention module is specifically: Receive multi-scale feature maps, treat the input scale features as word units, and use 1×1 convolutional layers instead of fully connected layers to obtain query Q, key K, and value V; Flatten the multi-scale feature maps in Q, K, and V to obtain , and ; according to , and The attention mechanism is used to calculate the self-scaled aggregate attention matrix, and the size of the self-scaled aggregate attention matrix is restored.
[0011] Preferably, , and The multi-scale feature map of is resized to , to obtain , and ; represents the number of attention heads, = ), is the spatial size of the hyperspectral image, is the number of endmembers contained in the hyperspectral image.
[0012] Preferably, the calculation of the self-scaled aggregate attention matrix is: calculate The dot product of and divided by a scaling factor; Use the softmax function to obtain attention weights; The obtained attention weight matrix is Multiply them together to get the attention matrix ; ; in, is the scaling factor.
[0013] Preferably, the abundance graph satisfies the non-negative constraint and the sum-to-one constraint: ; in, It is a normalization operation.
[0014] Preferably, based on the reconstructed thangka hyperspectral image data, the spectral angular distance between the original image pixels and the reconstructed thangka hyperspectral image pixels is calculated by a spectral angular distance loss function, and the loss function is minimized by a gradient descent method to narrow the gap between the reconstructed thangka hyperspectral image and the original image. The spectral angular distance loss function is: ; in, represents the number of pixels in the input HSI, Represent the original images and reconstruct the image Middle The spectral vector of pixels, T is the transpose.
[0015] Beneficial effects: The present invention uses a dual-stream, multi-branch CNN encoder to extract multi-scale spatial features and spectral features, and includes independent spectral streams and spatial streams. The spatial stream adopts a multi-branch architecture to extract information from three different receptive fields; in the main branch of the spatial stream, the present invention designs a block-by-block branching strategy to minimize the computational complexity while maintaining multi-scale feature alignment.
[0016] In the present invention, a multi-head self-scaling aggregation attention module is integrated in the Transformer encoder; by stacking the multi-head self-scaling aggregation attention modules, the encoder can continuously compare the correlation and autocorrelation between multi-scale features, and effectively capture the intrinsic properties of pigments in thangka hyperspectral images; this process enhances the scale perception ability of the network and improves the level of feature fusion; by adaptively adjusting feature weights, the Transformer encoder can optimize feature extraction, thereby supporting more accurate abundance estimation; it can significantly improve the accuracy of thangka pigment distribution extraction methods that are currently based only on traditional mathematical methods, especially thangkas with different pigment regions of different sizes and strong heterogeneous distribution can be identified. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.
[0018] In the attached picture: Figure 1 It is a schematic diagram of the overall structure of the present invention; Figure 2 It is the architecture diagram of the MssaA module of the present invention. DETAILED DESCRIPTION
[0019] The embodiments of the present invention are described below in conjunction with the drawings in the embodiments of the present invention. The terms used in the implementation mode of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention. The embodiments of the present application are described below in conjunction with the drawings.
[0020] Example: Figure 1 As shown in the figure, the thangka hyperspectral remote sensing image (HSI) is input into the convolutional neural network (CNN) encoder to extract multi-scale features (including spectral and spatial features), and then the Transformer encoder is used for effective feature fusion, and finally the decoder is used for decoding and output, realizing end-to-end, dynamic and adaptive use of multi-scale information, and providing accurate thangka pigment distribution extraction results.
[0021] by represents the input thangka hyperspectral image, where represents the spatial size of the image, Indicates the number of spectral bands of the image; encoder The task is to convert the input HSI into a low-dimensional feature representation : ; in, represents the CNN encoder, which aims to extract spectral information and multi-scale spatial information from the input image; Represents the Transformer encoder, which is used to aggregate multi-scale information: ; ; Represents multi-scale feature maps; decoder For low-dimensional representation Convert to reconstructed HSI data .
[0022] .
[0023] In a specific embodiment: Construct a dual-stream multi-branch CNN encoder, including a spectral stream composed of three modules and a spatial stream with three multi-scale spatial feature extraction branches, to simultaneously extract the spectral features and multi-scale spatial features of the input thangka hyperspectral image; The spectral stream consists of three modules, each of which contains a 1×1 two-dimensional convolutional layer to gradually compress the input HSI in the spectral dimension; all three modules use batch normalization (BN) after the convolutional layer to ensure the stability of training; a dropout layer is added after the batch normalization layer in the first module to alleviate the gradient vanishing problem during network training; before outputting the first two modules of the spectral stream, the data needs to pass through the Leaky ReLU activation function to introduce nonlinearity.
[0024] The spatial stream contains three multi-scale spatial feature extraction branches, which are used to extract large-scale, medium-scale and small-scale spatial features from HSI. In the spatial stream, the design of the main branch is similar to that of the spectral stream. In this embodiment, the convolution kernel sizes of the three modules contained in the main branch are adjusted to 5×5, 3×3 and 3×3 respectively; the main branch is responsible for extracting large-scale spatial features; the output of the second module of the main branch (representing the medium-scale spatial features) will be input into the upper branch; the upper branch consists of a single module, which is designed similarly to the last module of the spectral stream and is dedicated to reducing the number of bands of medium-scale spatial features; similarly, the lower branch receives the output from the first module of the main branch as input to obtain small-scale spatial features.
[0025] Multi-scale information comes from different modules in the same branch. This block-by-block branching strategy not only ensures the correlation between multi-scale features, but also reduces the computational complexity of model training. Finally, the feature matrix output by the four branches of the dual-stream multi-branch encoder will be stacked into a four-dimensional matrix. , as the output of the CNN encoder. The structure of the CNN encoder is shown in the following table:
[0026] Transformer encoder with multi-head self-scaling aggregate attention module: The Transformer encoder consists of two sequentially stacked combinations, each of which contains a Multihead Self-scale-aggregation Attention (MSsaA) module and a Scale-wise Multilayered Perceptron (SMLP) module; layer normalization (Layernorm) is applied before each module, and residual connections are set after each module; the SMLP module contains two 1×1 convolutional layers and a Gaussian Error Linear Unit (GELU) activation function to fine-tune multi-scale features; the overall structure of the Transformer encoder is as follows: ; Among them, mean() means calculating the mean of the input matrix along the first dimension.
[0027] refer to Figure 2 As shown in the figure, it is the architecture of the MSsaA module. The MSsaA module is specifically: Step 1: The multi-scale feature map T is directly used as Input into the first MSsaA module; Different from other Transformer-based unmixing networks, this embodiment does not require patch embedding, position embedding, and adding class tokens; 1×1 convolutional layers are used instead of fully connected layers to obtain queries Q, keys K, and values V, solving the problem of a single token being too large: ; in , and They represent the convolutional layers of two MSsaA modules respectively; , , and , , They represent the weights and biases of the convolutional layers respectively.
[0028] Step 2: Flatten the multi-scale feature maps in the query Q, key K, and value V to obtain , and , their sizes are ,in equal ), is the number of end members contained in the hyperspectral image, and the number of end members is the number of types of raw materials used to make thangka pigments; for example, the pigments of a thangka include lapis lazuli, realgar, cinnabar and other raw materials; , and Resize to , to obtain the corresponding , and ,in Indicates the number of attention heads; enhances the feature expression capability of MSsaA and implements a multi-head self-scaling aggregation attention mechanism for simultaneously learning feature information in multiple subspaces; Step 3: In this step, according to , and Using the attention mechanism to calculate the self-scaled aggregate attention matrix :Calculate first and The dot product of is divided by a scaling factor, and then the softmax function is used to obtain the attention score; finally, the obtained attention weight matrix is combined with Multiply them together to get the attention matrix ; The process can be expressed as: ; in is the scaling factor; In order to maintain the consistency of feature dimensions, the self-scaled aggregate attention matrix Resize to the size of ( ) to obtain .
[0029] The decoder performs the decoding operation, and the output of the Transformer encoder is , calculate the mean along the first dimension to obtain a low-dimensional representation of the same size as the target abundance map ; In the decoder, the ReLU activation function and the Sum-To-One (STO) operation are first used to ensure that the abundance map obtained Satisfy the non-negativity constraint and the sum-to-one constraint: ; It is a normalization operation. The specific calculation method is as follows: ; ; in, is a minimum value, Representation of low dimensionality Included in The feature map corresponding to the end members.
[0030] Finally, the abundance map Transformed through a convolutional layer to reconstruct the thangka HSI data ; The convolutional layer has no bias, and its weights are initialized by vertex component analysis; after training, the weights of the convolutional layer are used as the estimated end member matrix Output. Based on the reconstructed thangka hyperspectral image data, the spectral angle distance between the original image pixels and the reconstructed thangka hyperspectral image pixels is calculated through the spectral angle distance loss function (SpectralAngleDistance, SAD), which can measure the difference between the two at the pixel level, and minimize the loss function through the gradient descent method to narrow the gap between the reconstructed thangka hyperspectral image and the original image, which can effectively improve the accuracy of gradient update and ensure the accuracy of endmember extraction and abundance estimation; Apply the spectral angular distance loss function: ; in, represents the number of pixels in the input HSI, Represent the original images and reconstruct the image Middle The spectral vector of pixels, T is the transpose.
[0031] The embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments. For ordinary technicians in this technical field, after knowing the contents recorded in the present invention, they can make several equivalent changes and substitutions without departing from the principle of the present invention. These equivalent changes and substitutions should also be regarded as belonging to the protection scope of the present invention.
Claims
1. A method for extracting thangka pigment distribution based on end-to-end hyperspectral unmixing, characterized in that: The steps include: S1. Construct a dual-stream multi-branch CNN encoder, including a spectral stream composed of multiple modules and a spatial stream of several multi-scale spatial feature extraction branches, and simultaneously extract the spectral features and multi-scale spatial features of the input thangka hyperspectral image; S2, feature fusion of spectral features and multi-scale spatial features through Transformer encoder; S3, using the decoder to obtain an abundance map using the ReLU activation function and the unification operation; and converting the abundance map into reconstructed thangka hyperspectral image data through a convolutional layer.
2. The method for extracting thangka pigment distribution based on end-to-end hyperspectral unmixing according to claim 1 is characterized in that: In step S1, the spectral stream includes three modules, each of which includes a 1×1 two-dimensional convolution layer to gradually compress the input thangka hyperspectral image in the spectral dimension. The three modules are batch normalized after the convolution layer.
3. The method for extracting thangka pigment distribution based on end-to-end hyperspectral unmixing according to claim 2 is characterized in that: A dropout layer is added after the batch normalization layer in the first module in the spectral stream, and before the output of the first two modules, the data is activated by a Leaky ReLU function to introduce nonlinearity.
4. The method for extracting thangka pigment distribution based on end-to-end hyperspectral unmixing according to claim 1 is characterized in that: In step S1, the spatial stream includes three multi-scale spatial feature extraction branches, specifically: The main branch includes three modules, and the convolution kernel size of each module is adjusted to 5×5, 3×3 and 3×3. The main branch extracts large-scale spatial features; The upper branch receives the output of the second module of the main branch as input and extracts the mesoscale spatial features; The lower branch receives the output of the first module of the main branch as input and extracts small-scale spatial features.
5. The method for extracting thangka pigment distribution based on end-to-end hyperspectral unmixing according to claim 1 is characterized in that: The Transformer encoder includes two sequentially stacked combinations, each of which includes a multi-head self-scaling aggregation attention module and a scaled multi-layer perceptron module; the scaled multi-layer perceptron module includes two 1×1 convolutional layers and a Gaussian error linear unit activation function.
6. The method for extracting thangka pigment distribution based on end-to-end hyperspectral unmixing according to claim 5 is characterized in that: The multi-head self-scaling aggregation attention module is specifically: Receive multi-scale feature maps, treat the input scale features as word units, and use 1×1 convolutional layers instead of fully connected layers to obtain query Q, key K, and value V; Flatten the multi-scale feature maps in Q, K, and V to obtain , and , and , and Resize the multi-scale feature map to obtain , and ; according to , and The attention mechanism is used to calculate the self-scaled aggregate attention matrix, and the size of the self-scaled aggregate attention matrix is restored.
7. The method for extracting thangka pigment distribution based on end-to-end hyperspectral unmixing according to claim 6 is characterized in that: Will , and The multi-scale feature map of is resized to , to obtain , and ; represents the number of attention heads, = ), is the spatial size of the hyperspectral image, is the number of endmembers contained in the hyperspectral image.
8. The method for extracting thangka pigment distribution based on end-to-end hyperspectral unmixing according to claim 7 is characterized in that: Calculation of self-scaled aggregate attention matrix: calculate and The dot product of and divided by a scaling factor; Use the softmax function to obtain attention weights; The obtained attention weight matrix is Multiply them together to get the attention matrix ; ; in, is the scaling factor.
9. The method for extracting thangka pigment distribution based on end-to-end hyperspectral unmixing according to claim 1, characterized in that: The abundance graph satisfies the non-negative constraint and the sum-to-one constraint: ; in, It is a normalization operation.
10. The method for extracting thangka pigment distribution based on end-to-end hyperspectral unmixing according to claim 1, characterized in that: Based on the reconstructed thangka hyperspectral image data, the spectral angular distance between the original image pixels and the reconstructed thangka hyperspectral image pixels is calculated by the spectral angular distance loss function, and the loss function is minimized by the gradient descent method to narrow the gap between the reconstructed thangka hyperspectral image and the original image. The spectral angular distance loss function is: ; in, represents the number of pixels in the input HSI, Represent the original images and reconstruct the image Middle The spectral vector of pixels, T is the transpose.
Citation Information
Patent Citations
Hyper-spectral and panchromatic image fusion method for extracting multi-scale spatial-spectral characteristics based on AE
CN112634137A
Image human-object interaction positioning method and system based on Transform model
CN114663915A
Visual perception method and device, storage medium and electronic equipment
CN114882465A
ViT human body behavior recognition method based on double-flow network architecture
CN115797827A
Hyperspectral image unmixing method based on minimum simplex volume constraint and Transform structure
CN116012349A
Cited By
Thangka element detection method and system based on improved RT-DETR
CN120852936A
A non-destructive detection method for painting works based on hyperspectrum
CN122714949A
