A spectral image processing system based on deep sequential feature extraction
The spectral image processing system, which utilizes deep sequence feature extraction and adaptive parameter adjustment, solves the problems of incomplete feature extraction and rigid parameters in existing technologies, achieving efficient and accurate processing of spectral images and improving feature fusion effects and overall processing quality.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ANHUI OCCUPATIONAL COLLEGE OF CITY MANAGEMENT
- Filing Date
- 2026-03-18
- Publication Date
- 2026-06-05
AI Technical Summary
Existing spectral image processing techniques struggle to balance the sequential correlation of spectral bands with the spatial details of the image during feature extraction. This results in insufficient comprehensiveness and accuracy of feature representation, rigid parameter settings, poor preprocessing adaptability, and limited feature fusion effects, leading to unstable processing results.
A spectral image processing system based on deep sequence feature extraction is adopted, including modules for spectral image acquisition, preprocessing, deep sequence feature extraction, feature fusion, and adaptive parameter adjustment. Features are extracted through band grouping, bidirectional temporal modeling, and multi-scale convolution. Combined with adaptive filtering, feature point matching, and normalization processing, parameters are dynamically adjusted to adapt to different scenarios and complexities.
It achieves comprehensive and accurate representation of spectral image features, improves processing accuracy and efficiency, significantly enhances signal-to-noise ratio, spatial resolution and spectral consistency, and adapts to the complexity variations of different scenarios.
Smart Images
Figure CN122156840A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of spectral image processing technology, specifically to a spectral image processing system based on deep sequence feature extraction. Background Technology
[0002] Spectral image processing technology is one of the core technologies in the field of spectral analysis. With the continuous development of iterative upgrades of spectral imaging equipment, it has gradually expanded from simple filtering and enhancement processing of image spatial domain in the early stage to feature extraction and analysis combined with spectral band dimensions, and then to the introduction of deep learning related algorithms to achieve intelligent processing. The technology system has been continuously improved and the processing capability has been gradually enhanced. It has been widely used in many fields such as remote sensing monitoring, medical image analysis, and industrial inspection, adapting to the basic processing needs of spectral images in different scenarios.
[0003] Existing spectral image processing techniques still have many areas for improvement in practical applications. In the feature extraction stage, it's difficult to simultaneously consider the sequential correlation of spectral bands and capture the spatial details of the image, easily leading to missing feature information and insufficient comprehensiveness and accuracy of feature representation. Parameter settings often use fixed patterns, failing to dynamically adjust according to the actual feature complexity of the image. Under different working conditions such as noise interference and band differences, the processing effect is unstable, and manual parameter adjustment is costly. Preprocessing operations such as filtering and band alignment lack precise matching with the image's own characteristics, making it difficult to specifically eliminate noise, band shifts, and uneven pixel distribution, directly affecting the accuracy and efficiency of the overall processing flow. Feature fusion stages often use single-dimensional weight allocation methods, failing to fully utilize the complementarity of features at different levels, resulting in limited representational capabilities of fused features. Consequently, the improvement in signal-to-noise ratio, spatial resolution, and spectral consistency of the final output image is relatively limited. Summary of the Invention
[0004] This invention provides a spectral image processing system based on deep sequence feature extraction, which solves the problems of incomplete feature extraction, rigid parameter adjustment, poor preprocessing adaptability, and limited feature fusion effect in existing technologies. It realizes accurate extraction of multi-dimensional features of spectral images and adaptive parameter adjustment, thereby improving processing accuracy, adaptability, and efficiency.
[0005] To solve the above-mentioned technical problems, the technical solution provided by the present invention is as follows:
[0006] A spectral image processing system based on deep sequence feature extraction includes a spectral image acquisition module, a spectral image preprocessing module, a deep sequence feature extraction module, a feature fusion module, a parameter adaptive adjustment module, and a spectral image processing output module. The spectral image acquisition module acquires spectral image data and transmits it to the spectral image preprocessing module. The spectral image preprocessing module performs joint preprocessing on the spectral image data and then transmits it to the deep sequence feature extraction module. The deep sequence feature extraction module extracts features from the preprocessed spectral image data using a combination of band grouping and deep sequence modeling to obtain deep sequence feature vectors. The feature fusion module performs multi-scale fusion on the deep sequence feature vectors to obtain fused feature vectors. The parameter adaptive adjustment module dynamically adjusts the relevant parameters of the deep sequence feature extraction module and the feature fusion module according to the feature complexity of the deep sequence feature vectors. The spectral image processing output module reconstructs the features from the fused feature vectors and outputs the processed spectral image.
[0007] Furthermore, the spectral image preprocessing module sequentially performs noise suppression, band alignment, and normalization on the spectral image data. Noise suppression employs an adaptive filtering algorithm based on pixel gray-level differences. The filtering coefficients are adjusted by real-time calculation of the gray-level variance of the pixel neighborhood. The filtering coefficients are linearly positively correlated with the gray-level variance of the pixel neighborhood, and their correlation satisfies the following relationship:
[0008] ;
[0009] in, These are the filter coefficients. Based on the filter coefficients, The correlation coefficient, The grayscale variance of the pixel neighborhood;
[0010] Band alignment employs a feature point matching algorithm based on grayscale matching of feature points. This algorithm identifies corresponding feature points whose grayscale differences between bands fall within a preset threshold range. These feature points are then used as a reference to correct the spatial position of each band and eliminate band offset.
[0011] The preset threshold is linearly positively correlated with the grayscale dynamic range of the spectral image, and the correlation satisfies:
[0012] ;
[0013] in, For the preset threshold, Based on the threshold, The correlation coefficient, This refers to the grayscale dynamic range.
[0014] Normalization processes map pixel values of spectral image data to a normalization interval determined by the pixel value distribution characteristics of the spectral image data. The upper and lower limits of this interval are proportional to the maximum and minimum values of the pixel values, and the proportional relationship is adaptively set according to the dispersion of the pixel value distribution.
[0015] Furthermore, the deep sequence feature extraction module performs band grouping on the preprocessed spectral image data, dividing continuous spectral bands into several band groups. During grouping, an adaptive division is performed, considering both the total number of spectral bands and band correlation. The range of group sizes is linearly positively correlated with the total number of spectral bands, and the correlation satisfies the following relationship:
[0016] ;
[0017] in, This is the maximum number of groups. Number of basic groups The correlation coefficient, This represents the total number of spectral bands.
[0018] By combining the band correlation threshold to filter groups, the band correlation within the same group is higher than the correlation between groups; feature encoding is performed on each band group to obtain the group feature vector of each band group, and the group feature vectors of all band groups constitute the group feature vector sequence.
[0019] The deep sequence feature extraction module performs bidirectional temporal modeling on the group feature vector sequence, and simultaneously extracts spatial detail features from the group feature vector sequence through multi-scale convolutional layers. The bidirectional temporal modeling result is concatenated with the multi-scale convolutional extraction result to obtain the deep sequence feature vector; the feature encoding satisfies the formula:
[0020] ;
[0021] in, For a single band group, the group feature vector is... A pixel matrix of a spectral image for a single band group. The weight matrix for feature encoding. This is the bias vector for feature encoding. For activation functions;
[0022] Bidirectional timing modeling satisfies the following formula:
[0023] ;
[0024] ;
[0025] in, The band group number. To model the output vector corresponding to the t-th band group in the forward time series, To model the output vector corresponding to the (t-1)th band group in the forward time series, Let be the group feature vector of the t-th band group. The weight matrix for forward time series modeling. Bias vector for forward timing modeling To model the output vector corresponding to the t-th band group in the backward time series, To model the output vector corresponding to the (t+1)th band group in the backward time series, Weight matrix for backward time series modeling Bias vector for backward timing modeling It is the hyperbolic tangent activation function;
[0026] Multi-scale convolution extraction satisfies the following formula:
[0027] ;
[0028] in, This represents the spatial detail feature vector corresponding to the t-th band group extracted by multi-scale convolution. This is the first convolution operation. For the second convolution operation, This is the third convolution operation. This is a vector concatenation operation.
[0029] Furthermore, the band groups are divided according to the wavelength continuity and correlation of the spectral bands. Spectral bands within the same band group have continuous wavelengths and higher correlation than those between different band groups. The correlation is determined using a conventional correlation coefficient algorithm. A correlation coefficient threshold is set during the division, and the threshold is linearly positively correlated with the wavelength overlap of the spectral bands. The correlation relationship satisfies the following:
[0030] ;
[0031] in, For the correlation coefficient threshold, Based on the threshold, The correlation coefficient, The wavelength overlap is denoted as λ. The multi-scale convolutional layer contains three convolutional operations with kernel sizes increasing in gradient. Each kernel size corresponds to a different spatial receptive field, and the size of the spatial receptive field increases with the kernel size. The convolutional results of the three scales are concatenated to obtain the spatial detail feature vector.
[0032] Furthermore, the feature fusion module performs multi-scale fusion of shallow and deep features on the deep sequence feature vector, and the fusion process satisfies the formula:
[0033] ;
[0034] in, To fuse feature vectors, To integrate the weighting coefficients, This refers to the deep feature vector corresponding to the deep sequence feature vector. This is a shallow feature vector obtained by downsampling the feature vector of the deep sequence.
[0035] The downsampling process employs pooling operations, which are conventional in this field, and the operation type is adaptively selected based on the dimension of the depth sequence feature vector. The values are linearly positively correlated with feature similarity, and the correlation relationship satisfies:
[0036] ;
[0037] in, To integrate the weighting coefficients, Based on the weighting coefficient, The correlation coefficient, The feature similarity is calculated using a cosine similarity algorithm based on the inner product of vectors.
[0038] Furthermore, the parameter adaptive adjustment module obtains the feature complexity of the depth sequence feature vector in real time. The feature complexity is calculated using the feature entropy value, which satisfies the following formula:
[0039] ;
[0040] in, The feature entropy value, The characteristic component index, The total number of feature components in the feature vector of the depth sequence. Let be the probability distribution of the i-th feature component;
[0041] Feature entropy and feature complexity are linearly positively correlated, and their correlation satisfies:
[0042] ;
[0043] in, For feature complexity, Based on basic complexity, The correlation coefficient is used; the parameter adaptive adjustment module adjusts the feature encoding weight matrix, bidirectional temporal modeling weight matrix, and multi-scale convolution kernel parameters of the deep sequence feature extraction module according to the feature complexity, while simultaneously adjusting the fusion weight coefficient of the feature fusion module. As feature complexity increases, the parameter iteration frequency and... The values all increase synchronously with the feature complexity, and the parameter iteration frequency is linearly positively correlated with the feature complexity. This correlation satisfies the following:
[0044] ;
[0045] in, For the parameter iteration frequency, Based on the iteration frequency, The correlation coefficient, For feature complexity;
[0046] The value and feature complexity are linearly positively correlated, and the correlation satisfies:
[0047]
[0048] in, To integrate the weighting coefficients, Based on the weighting coefficient, The correlation coefficient, The feature complexity is denoted as .
[0049] Furthermore, the spectral image processing output module performs feature inversion and spatial reconstruction on the fused feature vector to obtain a processed spectral image with the same size as the original spectral image. Feature inversion is achieved through a deconvolutional network, which upsamples the fused feature vector to restore the feature dimension. Spatial reconstruction uses a feature mapping algorithm to map the deconvolution-processed feature vector to the pixel space of the original spectral image, outputting the processed spectral image. The number of layers and the upsampling ratio of the deconvolutional network are adaptively matched according to the dimension of the fused feature vector and the size of the original spectral image. The number of layers in the deconvolutional network is linearly positively correlated with the dimension of the feature vector, and their correlation satisfies the following:
[0050]
[0051] in, This represents the number of layers in the deconvolutional network. Based on the number of layers, The correlation coefficient, The dimension of the feature vector.
[0052] The advantages of this invention compared to the prior art are:
[0053] This invention overcomes the limitations of single feature extraction by combining band grouping with bidirectional temporal modeling and multi-scale convolution for deep sequence feature extraction. It simultaneously captures the sequence correlation and spatial detail features of spectral bands, effectively avoiding the omission of feature information, and achieving comprehensive and accurate representation of spectral image features. This provides high-quality data assurance for feature fusion and significantly improves the processing accuracy of complex spectral images.
[0054] This invention quantifies the complexity of deep sequence features by using feature entropy values, dynamically adjusts the core parameters of the feature extraction and fusion modules, abandons fixed parameter settings, and achieves adaptive adaptation of the system to spectral images of different scenes and complexities. This significantly reduces the cost of manual parameter adjustment and can still stably output optimized results under complex working conditions such as noise interference and band differences.
[0055] This invention addresses noise, band shift, and uneven pixel distribution in spectral images through adaptive filtering based on pixel grayscale differences, band alignment based on feature point matching, and adaptive normalization preprocessing. This achieves precise matching between preprocessing results and image characteristics, eliminates interference obstacles in high-precision spectral image processing, and directly improves the accuracy and efficiency of the overall processing flow.
[0056] This invention achieves complementary enhancement of features at different levels by multi-scale weighted fusion of shallow and deep features, and optimizes the fusion weight by combining feature similarity and complexity. This balances the global correlation and local detail information of features, improves the representational ability of fused features, and significantly enhances the signal-to-noise ratio, spatial resolution and spectral consistency of the output image. Attached Figure Description
[0057] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application.
[0058] In the attached diagram:
[0059] Figure 1 This is a schematic diagram of the overall framework of the spectral image processing system in Example 1.
[0060] Figure 2 This is a flowchart of the spectral image preprocessing module in Example 1.
[0061] Figure 3 This is a flowchart of the deep sequence feature extraction module in Example 1.
[0062] Figure 4 This is a flowchart of the feature fusion module in Example 1.
[0063] Figure 5 This is a flowchart of the parameter adaptive adjustment module in Example 1.
[0064] Figure 6 This is a flowchart of the spectral image processing output module in Example 1. Detailed Implementation
[0065] The following detailed description of the embodiments is intended to exemplify the principles of this application, but should not be used to limit the scope of this application. That is, the spectral image processing system based on depth sequence feature extraction of this application is not limited to the described embodiments.
[0066] The present invention will be further described below with reference to embodiments.
[0067] This embodiment demonstrates the practical implementation of a spectral image processing system based on deep sequence feature extraction. The core hardware utilizes a GaiaSky-mini2 pushbroom spectral camera, covering the 400-1000nm visible to near-infrared band with a pixel resolution of 1024×768. Equipped with a GigE industrial Ethernet interface, it can stably acquire multi-band spectral data. The software and hardware framework is built upon Python 3.8, OpenCV 4.8.0, and PyTorch 2.0, ensuring compatibility and no conflicts between versions. For computing power, an Intel Core i7-12700H processor and an NVIDIA RTX 3090 graphics card are used. The Intel Core i7-12700H processor has 14 cores and 20 threads with a base clock speed of 2.7GHz, while the NVIDIA RTX 3090 graphics card has 24GB of GDDR6X video memory and 10496 CUDA cores. The system's modules are connected in an orderly manner according to the data flow. It adopts modular programming and hardware deployment, and can be directly deployed on general hardware platforms such as industrial computers and servers. It is suitable for the entire process of spectral image processing in scenarios such as remote sensing monitoring, industrial inspection, and spectral imaging of medical pathological slides.
[0068] like Figure 1 As shown, a spectral image processing system based on deep sequence feature extraction includes a spectral image acquisition module, a spectral image preprocessing module, a deep sequence feature extraction module, a feature fusion module, a parameter adaptive adjustment module, and a spectral image processing output module. The spectral image acquisition module acquires spectral image data and transmits it to the spectral image preprocessing module. The spectral image preprocessing module performs joint preprocessing on the spectral image data and then transmits it to the deep sequence feature extraction module. The deep sequence feature extraction module extracts features from the preprocessed spectral image data using a combination of band grouping and deep sequence modeling to obtain a deep sequence feature vector. The feature fusion module performs multi-scale fusion on the deep sequence feature vector to obtain a fused feature vector. The parameter adaptive adjustment module dynamically adjusts the relevant parameters of the deep sequence feature extraction module and the feature fusion module according to the feature complexity of the deep sequence feature vector. The spectral image processing output module reconstructs the features from the fused feature vector and outputs the processed spectral image.
[0069] In a specific embodiment, the spectral image processing system based on deep sequence feature extraction includes, in sequence, a spectral image acquisition module, a spectral image preprocessing module, a deep sequence feature extraction module, a feature fusion module, a parameter adaptive adjustment module, and a spectral image processing output module. The spectral image acquisition module uses a GaiaSky-mini2 pushbroom spectral camera as its core acquisition device, acquiring multi-band spectral data and outputting raw data in a three-dimensional matrix format. This format is the default output format of the GaiaSky-mini2 camera and can be directly recognized by subsequent modules. The spectral image preprocessing module performs noise suppression, band alignment, and normalization on the raw data to improve data quality. The deep sequence feature extraction module extracts spectral sequence features and spatial detail features through band grouping, feature encoding, and time-series modeling. The feature fusion module performs multi-scale fusion of shallow and deep features to enhance feature representation. The parameter adaptive adjustment module dynamically adjusts relevant parameters according to feature complexity. The spectral image processing output module completes feature inversion and spatial reconstruction, outputting a processed image that meets the requirements. The system's hardware and software framework is built on Python 3.8, OpenCV 4.8.0, and PyTorch 2.0. The computing power is supported by an Intel Core i7-12700H processor and an NVIDIA RTX 3090 graphics card. It adopts modular programming and hardware deployment and can be directly deployed on general-purpose hardware platforms such as industrial computers and servers.
[0070] Furthermore, such as Figure 2 As shown, the spectral image preprocessing module sequentially performs noise suppression, band alignment, and normalization on the spectral image data. Noise suppression employs an adaptive filtering algorithm based on pixel gray-level differences. The filtering coefficients are adjusted by real-time calculation of the gray-level variance of the pixel neighborhood. The filtering coefficients are linearly positively correlated with the gray-level variance of the pixel neighborhood, and their correlation satisfies the following relationship:
[0071] ;
[0072] in, These are the filter coefficients. Based on the filter coefficients, The correlation coefficient, The grayscale variance of the pixel neighborhood;
[0073] Band alignment employs a feature point matching algorithm based on grayscale matching of feature points. This algorithm identifies corresponding feature points whose grayscale differences between bands fall within a preset threshold range. These feature points are then used as a reference to correct the spatial position of each band and eliminate band offset.
[0074] The preset threshold is linearly positively correlated with the grayscale dynamic range of the spectral image, and the correlation satisfies:
[0075] ;
[0076] in, For the preset threshold, Based on the threshold, The correlation coefficient, This refers to the grayscale dynamic range.
[0077] Normalization processes map pixel values of spectral image data to a normalization interval determined by the pixel value distribution characteristics of the spectral image data. The upper and lower limits of this interval are proportional to the maximum and minimum values of the pixel values, and the proportional relationship is adaptively set according to the dispersion of the pixel value distribution.
[0078] In a specific embodiment, the spectral image preprocessing module processes data in the order of noise suppression, band alignment, and normalization. First, it removes noise introduced by the device and environment, then corrects spatial positional deviations in different bands, and finally optimizes pixel value distribution. Noise suppression employs an adaptive filtering algorithm based on pixel grayscale differences. It calculates the neighborhood grayscale variance pixel by pixel to adjust the filtering coefficients. Combined with the 1024×768 pixel resolution of the GaiaSky-mini2 pushbroom spectral camera, a 3×3 rectangular neighborhood is selected to calculate the pixel neighborhood grayscale variance. The image values acquired by the GaiaSky-mini2 pushbroom spectral camera range from 0 to 10. This range is based on the statistical analysis of the grayscale variance of 1000 images acquired by the camera under normal lighting and industrial inspection scenarios. Accordingly, the basic filtering coefficient is set to 0.2, and the correlation coefficient is set to 0.1. Calculated according to the formula, the value range is 0.2–1.2. Band alignment employs the ORB feature point grayscale matching algorithm, a mature algorithm built into the OpenCV 4.8.0 framework that can be directly called to implement feature point matching. A preset threshold for grayscale difference is used to filter matching feature points, and the grayscale dynamic range is obtained from the average difference between the maximum and minimum grayscale values of each band. The image value range of the GaiaSky-mini2 pushbroom spectral camera is 0–200, based on the camera's 400–1000nm band photosensitive characteristics. The grayscale output range of its photosensitive element for each band is 0–255. After deducting invalid grayscale values, the average difference between the maximum and minimum grayscale values of each band is distributed within this range. Based on this, a base threshold of 8 and a correlation coefficient of 0.02 are set. According to the formula, the preset threshold range is 8–12. Using the center band as a reference, the spatial coordinates of other bands are corrected through affine transformation. The affine transformation can be implemented using the `warpAffine` function in the OpenCV 4.8.0 framework. Normalization maps pixel values of each band independently to an interval that suits its own distribution characteristics. First, the maximum and minimum values of pixels in a single band are calculated. Then, the ratio between the upper and lower limits of the interval and the maximum and minimum values is adjusted according to the degree of dispersion of the distribution. If the dispersion is large, the interval is widened; if the dispersion is small, the interval is narrowed. Finally, the values are uniformly mapped to the [0,1] interval, which meets the input requirements of the deep sequence feature extraction algorithm in the PyTorch 2.0 framework. The normalization operation can be assisted by the nn.BatchNorm2d function in the PyTorch 2.0 framework.
[0079] Furthermore, such as Figure 3 As shown, the deep sequence feature extraction module performs band grouping on the preprocessed spectral image data, dividing continuous spectral bands into several band groups. During grouping, an adaptive division is performed, considering both the total number of spectral bands and their correlation. The range of group sizes is linearly positively correlated with the total number of spectral bands, and the correlation satisfies the following relationship:
[0080] ;
[0081] in, This is the maximum number of groups. Number of basic groups The correlation coefficient, This represents the total number of spectral bands.
[0082] By combining the band correlation threshold to filter groups, the band correlation within the same group is higher than the correlation between groups; feature encoding is performed on each band group to obtain the group feature vector of each band group, and the group feature vectors of all band groups constitute the group feature vector sequence.
[0083] The deep sequence feature extraction module performs bidirectional temporal modeling on the group feature vector sequence, and simultaneously extracts spatial detail features from the group feature vector sequence through multi-scale convolutional layers. The bidirectional temporal modeling result is concatenated with the multi-scale convolutional extraction result to obtain the deep sequence feature vector; the feature encoding satisfies the formula:
[0084] ;
[0085] in, For a single band group, the group feature vector is... A pixel matrix of a spectral image for a single band group. The weight matrix for feature encoding. This is the bias vector for feature encoding. For activation functions;
[0086] Bidirectional timing modeling satisfies the following formula:
[0087] ;
[0088] ;
[0089] in, The band group number. To model the output vector corresponding to the t-th band group in the forward time series, To model the output vector corresponding to the (t-1)th band group in the forward time series, Let be the group feature vector of the t-th band group. The weight matrix for forward time series modeling. Bias vector for forward timing modeling To model the output vector corresponding to the t-th band group in the backward time series, To model the output vector corresponding to the (t+1)th band group in the backward time series, Weight matrix for backward time series modeling Bias vector for backward timing modeling It is the hyperbolic tangent activation function;
[0090] Multi-scale convolution extraction satisfies the following formula:
[0091] ;
[0092] in, This represents the spatial detail feature vector corresponding to the t-th band group extracted by multi-scale convolution. This is the first convolution operation. For the second convolution operation, This is the third convolution operation. This is a vector concatenation operation.
[0093] In a specific embodiment, in the deep sequence feature extraction module, after band grouping is completed, feature encoding and temporal modeling are performed on each band group one by one. The pixel matrix of the spectral image of a single band group is flattened into a one-dimensional vector, which serves as the basic data for feature encoding; the weight matrix realizes the dimensionality transformation from the pixel matrix to the feature vector, and its dimension matches the pixel dimension of the band group and the 64-dimensional feature vector. The 64-dimensional feature vector, combined with the 1024×768 pixel resolution of the GaiaSky-mini2 camera, balances feature condensation effect and computational efficiency, compresses the pixel matrix dimension, reduces the computational load of temporal modeling, and adapts to the input requirements of the LSTM network in the PyTorch 2.0 framework. The PyTorch 2.0 framework has a built-in nn.LSTM class that can directly implement temporal modeling. The bias vector dimension is consistent with the group feature vector, used to optimize the feature value distribution. The activation function chosen is LeakyReLU, a built-in activation function in the PyTorch 2.0 framework, specifically nn.LeakyReLU, which can be directly called and effectively solves the problem of dead neurons in the ReLU function, introducing non-linear characteristics into feature encoding to obtain the encoded group feature vector. All band groups are arranged in wavelength order to form a sequence of group feature vectors. A bidirectional LSTM network is used for time series modeling. The bidirectional LSTM can be implemented using the bidirectional=True parameter in the nn.LSTM class of the PyTorch 2.0 framework. The band group number ensures that time series modeling proceeds in wavelength order. The group feature vector of the i-th band group serves as the core input, with and being the output vectors for forward and backward time series modeling, respectively, and and being the output vectors for the previous and next time steps, respectively. The weight matrix and dimension design are 128×64. Since the bidirectional LSTM input is the concatenation result of the previous 64-dimensional output vector and the current 64-dimensional feature vector, the concatenated dimension is 128. This dimension design conforms to the input-output dimension matching rule of the LSTM network in the PyTorch 2.0 framework. The bias vector and optimized output feature values are achieved using the tanh activation function, which is a built-in activation function of the PyTorch 2.0 framework. This constrains the feature values to the interval [−1,1], ensuring the stability of temporal feature extraction and capturing the forward and backward sequence correlation of spectral bands.
[0094] Furthermore, the band groups are divided according to the wavelength continuity and correlation of the spectral bands. Spectral bands within the same band group have continuous wavelengths and higher correlation than those between different band groups. The correlation is determined using a conventional correlation coefficient algorithm. A correlation coefficient threshold is set during the division, and the threshold is linearly positively correlated with the wavelength overlap of the spectral bands. The correlation relationship satisfies the following:
[0095] ;
[0096] in, For the correlation coefficient threshold, Based on the threshold, The correlation coefficient, The wavelength overlap is denoted as λ. The multi-scale convolutional layer contains three convolutional operations with kernel sizes increasing in gradient. Each kernel size corresponds to a different spatial receptive field, and the size of the spatial receptive field increases with the kernel size. The convolutional results of the three scales are concatenated to obtain the spatial detail feature vector.
[0097] In a specific embodiment, the deep sequence feature extraction module first groups the preprocessed data by band, and adaptively divides the data by considering the total number of spectral bands, correlation, and wavelength continuity. Based on the 40-200 band coverage of the GaiaSky-mini2 pushbroom spectral camera, a basic group size of 5 and a correlation coefficient of 0.05 are set. According to the formula, the upper limit of the number of groups is between 5 and 15. This range is set according to the band count characteristics of the GaiaSky-mini2 camera; when the number of bands is 40, each group contains 8 bands, and when the number of bands is 200, each group contains 13-14 bands, balancing the continuity of spectral information within the group with feature extraction efficiency. During operation, the spectral bands are first arranged in ascending order of wavelength, and adjacent bands are divided sequentially according to their wavelength range. Then, the correlation of all pixel grayscale values within a band is calculated using the Pearson correlation coefficient algorithm. This algorithm can be implemented using the `scipy.stats.pearsonr` function in Python 3.8, and its computational efficiency and accuracy meet the system requirements. Wavelength overlap is obtained by dividing the intersection range of adjacent wavelength bands by the union range, with a value ranging from 0 to 1. This range represents the inherent interval of wavelength overlap, with 0 for no overlap and 1 for complete overlap, conforming to the physical distribution law of spectral bands. Based on this, a base threshold of 0.7 and a correlation coefficient of 0.2 are set. Calculated using the formula, the correlation coefficient threshold ranges from 0.7 to 0.9, ensuring that the correlation of bands within a group is greater than 0.7. Highly correlated adjacent bands are then selected and merged based on this threshold. Simultaneously with bidirectional temporal modeling, spatial detail features are extracted from the group feature vector sequence through a multi-scale convolutional layer. First, the group feature vector sequence is reconstructed into a two-dimensional feature map, then input into the multi-scale convolutional layer. This layer uses three gradient-increasing convolution operations, with kernel sizes of 3×3, 5×5, and 7×7 respectively. The kernel size conforms to the conventional design for spatial feature extraction of spectral images and can be implemented using the nn.Conv2d class in the PyTorch 2.0 framework. The dimensions are set based on the 64-dimensional feature map space: 3×3 captures fine edges, 5×5 captures mid-level textures, and 7×7 captures global spatial features. Computation is performed in parallel, utilizing the CUDA cores of the NVIDIA RTX 3090 graphics card. The CUDA cores can be called via the `cuda()` method in the PyTorch 2.0 framework, reducing the time consumption of multi-scale convolution to 1.2 times that of a single convolution operation. The feature maps output from each scale convolution are batch normalized, optimized with the ReLU activation function, and then fused using the `concat` vector concatenation operation. Batch normalization can be implemented using the `nn.BatchNorm2d` function in the PyTorch 2.0 framework, the ReLU activation function is a built-in function of the framework (specifically `nn.ReLU`), and the `concat` operation can be implemented using the `torch.cat` function. Finally, the sequence feature vector output from bidirectional temporal modeling is concatenated with the feature vector according to the feature dimensions to obtain a deep sequence feature vector containing both spectral sequence features and image spatial features.
[0098] Furthermore, such as Figure 4 As shown, the feature fusion module performs multi-scale fusion of shallow and deep features on the deep sequence feature vector. The fusion process satisfies the formula:
[0099] ;
[0100] in, To fuse feature vectors, To integrate the weighting coefficients, This refers to the deep feature vector corresponding to the deep sequence feature vector. This is a shallow feature vector obtained by downsampling the feature vector of the deep sequence.
[0101] The downsampling process employs pooling operations, which are conventional in this field, and the operation type is adaptively selected based on the dimension of the depth sequence feature vector. The values are linearly positively correlated with feature similarity, and the correlation relationship satisfies:
[0102] ;
[0103] in, To integrate the weighting coefficients, Based on the weighting coefficient, The correlation coefficient, The feature similarity is calculated using a cosine similarity algorithm based on the inner product of vectors.
[0104] In a specific embodiment, the feature fusion module performs multi-scale fusion operations on shallow and deep features for deep sequence feature vectors. Deep feature vectors, taken from the outputs of the last two layers of the feature extraction network, represent the global features and sequence correlation characteristics of the spectral image; shallow feature vectors, taken from the outputs of the first two layers of the network, reflect the local spatial details of the image. The fusion weight coefficients are calculated according to a formula, with the base weight coefficient set to 0.3 and the correlation coefficient to 0.5. Feature similarity is calculated using a cosine similarity algorithm based on vector inner product, which can be implemented using the `scipy.spatial.distance.cosine` function in Python 3.8. The resulting fused feature vector achieves complementary enhancement of deep and shallow features. Shallow features are obtained by downsampling deep features using an average pooling algorithm. Average pooling can be implemented using the `nn.AvgPool2d` class in the PyTorch 2.0 framework. The deep feature vector is 64-dimensional, and a 2×2 pooling kernel size is used. After downsampling, a 32-dimensional feature vector is obtained, which matches the inherent dimension of the shallow features output from the first two layers of the feature extraction network, allowing for direct weighted fusion calculation. The reference standard feature vector is obtained from 500 standard spectral images of the same type through the same feature extraction process. This sample size can cover spectral features with different lighting and scene complexity, ensuring that it can represent the mean spectral features of the corresponding scene. Standard spectral images can be obtained from public spectral databases, such as the USGS spectral library. The cosine similarity value ranges from 0 to 1, with higher feature matching degrees closer to 1. Based on this, the calculated value ranges from 0.3 to 0.8, allowing for dynamic adjustment and improving the accuracy of the fused feature representation.
[0105] Furthermore, such as Figure 5 As shown, the parameter adaptive adjustment module obtains the feature complexity of the deep sequence feature vector in real time. The feature complexity is calculated by the feature entropy value, which satisfies the formula:
[0106] ;
[0107] in, The feature entropy value, The characteristic component index, The total number of feature components in the feature vector of the depth sequence. Let be the probability distribution of the i-th feature component;
[0108] Feature entropy and feature complexity are linearly positively correlated, and their correlation satisfies:
[0109] ;
[0110] in, For feature complexity, Based on basic complexity, The correlation coefficient is used; the parameter adaptive adjustment module adjusts the feature encoding weight matrix, bidirectional temporal modeling weight matrix, and multi-scale convolution kernel parameters of the deep sequence feature extraction module according to the feature complexity, while simultaneously adjusting the fusion weight coefficient of the feature fusion module. As feature complexity increases, the parameter iteration frequency and... The values all increase synchronously with the feature complexity, and the parameter iteration frequency is linearly positively correlated with the feature complexity. This correlation satisfies the following:
[0111] ;
[0112] in, For the parameter iteration frequency, Based on the iteration frequency, The correlation coefficient, For feature complexity;
[0113] The value and feature complexity are linearly positively correlated, and the correlation satisfies:
[0114]
[0115] in, To integrate the weighting coefficients, Based on the weighting coefficient, The correlation coefficient, The feature complexity is denoted as .
[0116] In a specific embodiment, the parameter adaptive adjustment module dynamically adjusts the relevant parameters of the deep sequence feature extraction module and the feature fusion module based on the feature complexity of the deep sequence feature vector. The core of this module is to obtain the feature complexity through feature entropy calculation. The probability distribution of the i-th feature component is obtained by normalizing the feature components, i.e., the proportion of each feature component's pixel value to the total pixel value of all components. The feature component index and the total number of feature components define the entropy calculation range. The feature entropy quantifies the feature complexity; the more dispersed the feature component distribution, the larger the value. Entropy calculation can be implemented using the `scipy.stats.entropy` function in Python 3.8. The feature complexity is calculated according to the formula. The entropy value of the 64-dimensional deep sequence feature vector ranges from 0 to 3. This range is calculated based on the entropy values of 1000 sets of feature vectors from different scenes. Based on this, the base complexity is set to 0.2, the correlation coefficient to 0.4, and the calculated value range is 0.2–1.4. The parameter iteration frequency is calculated using a formula, with a base iteration frequency of 15Hz and a correlation coefficient of 8. The calculated value range is 15-26.2Hz. 15Hz matches the computing speed of an NVIDIA RTX 3090 graphics card, and the correlation coefficient of 8 allows the iteration frequency to adjust linearly with feature complexity. The iteration frequency can be controlled using the `time` module in Python 3.8. The fusion weight coefficient is dynamically adjusted, maintaining consistency with the value of the correlation coefficient when used with feature similarity. The parameter adjustments for feature extraction and fusion employ gradient descent for iterative updates. Gradient descent can be implemented using the `torch.optim.SGD` class in the PyTorch 2.0 framework, with a learning rate set to 0.001. This learning rate is based on the numerical distribution of the 64-dimensional feature vector, allowing the parameters to converge to their optimal values within 500 iterations. The adjusted parameters include the feature encoding weight matrix, the bidirectional temporal modeling weight matrix, and the multi-scale convolution kernel parameters. All parameter adjustments can be implemented using the parameter update mechanism of the PyTorch 2.0 framework.
[0117] Furthermore, such as Figure 6 As shown, the spectral image processing output module performs feature inversion and spatial reconstruction on the fused feature vector to obtain a processed spectral image with the same size as the original spectral image. Feature inversion is achieved through a deconvolutional network, which upsamples the fused feature vector to restore the feature dimension. Spatial reconstruction uses a feature mapping algorithm to map the deconvolution-processed feature vector to the pixel space of the original spectral image, outputting the processed spectral image. The number of layers and the upsampling ratio of the deconvolutional network are adaptively matched according to the dimension of the fused feature vector and the size of the original spectral image. The number of layers in the deconvolutional network is linearly positively correlated with the dimension of the feature vector, and their correlation satisfies:
[0118]
[0119] in, This represents the number of layers in the deconvolutional network. Based on the number of layers, The correlation coefficient, The dimension of the feature vector.
[0120] In a specific embodiment, the spectral image processing output module processes the fused feature vector in the order of feature inversion and spatial reconstruction, outputting a processed image that meets the requirements of spectral analysis. The fused feature vector is input into a deconvolutional network to perform feature inversion. This network is constructed by alternating transposed convolutional layers and batch normalization layers. The transposed convolution can be implemented using the nn.ConvTranspose2d class in the PyTorch 2.0 framework, and the batch normalization can be implemented using the nn.BatchNorm2d function. Upsampling converts the feature vector from the feature dimension to the pixel dimension. The fused feature vector has a dimension of 256, which is obtained by concatenating a 64-dimensional temporal feature vector and a 64-dimensional spatial feature vector and then expanding it through feature optimization. The 256-dimensional dimension is adapted to the original image's pixel resolution of 1024×768, ensuring that the pixel dimension after feature inversion matches it. The dimension adaptation conforms to the PyTorch 2.0 framework deconvolutional network design specifications. The number of deconvolutional network layers is calculated using a formula, with the fused feature vector ranging from 128 to 512. 128 is used for simple scenarios, and 512 for complex scenarios. A base layer count of 3 and a correlation coefficient of 0.04 are set, resulting in a calculated value range of 3 to 23.48. Integer layers are used for implementation, specifically 3 to 23 layers. This range is set based on upsampling requirements. 3 to 23 layers ensure smooth upsampling of the feature vector from the feature dimension to the pixel dimension, avoiding insufficient upsampling or image distortion. The upsampling ratio is adaptively matched to the number of deconvolutional layers, the dimension of the fused feature vector, and the original image size. The ratio calculation can be implemented using Python 3.8. After feature inversion, a bilinear interpolation feature mapping algorithm maps the deconvolution-processed feature vector to the 1024×768 pixel space of the original spectral image. Bilinear interpolation can be implemented using the `resize` function in the OpenCV 4.8.0 framework, establishing a one-to-one correspondence between the feature vector and pixel grayscale values, thus completing spatial reconstruction. The final output is a spectral image in TIFF and ENVI formats. These two formats are standard formats in the field of spectral analysis and can be directly read by commonly used spectral analysis software such as ENVI and ArcGIS. The size and number of bands are completely consistent with the original image, and the band and pixel spatial information of the spectral image are preserved. The image format output can be achieved by using the PIL library of Python 3.8 to achieve TIFF format output and by using the spectral library to achieve ENVI format output.
[0121] It should be noted that the combination of the technical features in this case is not limited to the combination methods described in the claims of this case or the combination methods described in the specific embodiments. All technical features described in this case can be freely combined or combined in any way, unless they contradict each other.
[0122] It should also be noted that the embodiments listed above are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments, and similar changes or modifications made thereto are those that can be directly derived or easily conceived by those skilled in the art from the content disclosed in the present invention, and should all fall within the protection scope of the present invention.
Claims
1. A spectral image processing system based on deep sequence feature extraction, characterized in that, The system includes a spectral image acquisition module, a spectral image preprocessing module, a deep sequence feature extraction module, a feature fusion module, a parameter adaptive adjustment module, and a spectral image processing output module. The spectral image acquisition module acquires spectral image data and transmits it to the spectral image preprocessing module. The spectral image preprocessing module performs joint preprocessing on the spectral image data and then transmits it to the deep sequence feature extraction module. The deep sequence feature extraction module extracts features from the preprocessed spectral image data using a combination of band grouping and deep sequence modeling to obtain a deep sequence feature vector. The feature fusion module performs multi-scale fusion on the deep sequence feature vector to obtain a fused feature vector. The parameter adaptive adjustment module dynamically adjusts the relevant parameters of the deep sequence feature extraction module and the feature fusion module according to the feature complexity of the deep sequence feature vector. The spectral image processing output module reconstructs the features from the fused feature vector and outputs the processed spectral image.
2. The spectral image processing system based on deep sequence feature extraction according to claim 1, characterized in that: The spectral image preprocessing module sequentially performs noise suppression, band alignment, and normalization on the spectral image data. The noise suppression employs an adaptive filtering algorithm based on pixel gray-level differences. This algorithm adjusts the filtering coefficients by real-time calculation of the gray-level variance of the pixel neighborhood. The filtering coefficients are linearly positively correlated with the gray-level variance of the pixel neighborhood, and their correlation satisfies the following relationship: ； in, These are the filter coefficients. Based on the filter coefficients, The correlation coefficient, The grayscale variance of the pixel neighborhood; The band alignment employs a feature point matching algorithm based on grayscale matching of feature points. This algorithm identifies corresponding feature points whose grayscale differences between bands fall within a preset threshold range. These feature points are then used as a reference to correct the spatial position of each band and eliminate band offset. The preset threshold is linearly positively correlated with the grayscale dynamic range of the spectral image, and the correlation satisfies the following: ； in, For the preset threshold, Based on the threshold, The correlation coefficient, This refers to the grayscale dynamic range. The normalization process maps the pixel values of the spectral image data to a normalization interval determined by the pixel value distribution characteristics of the spectral image data. The upper and lower limits of this interval are proportional to the maximum and minimum values of the pixel values, and the proportional relationship is adaptively set according to the dispersion of the pixel value distribution.
3. The spectral image processing system based on deep sequence feature extraction according to claim 2, characterized in that: The deep sequence feature extraction module performs band grouping on the preprocessed spectral image data, dividing continuous spectral bands into several band groups. During grouping, an adaptive division is performed, considering both the total number of spectral bands and band correlation. The number of groups is linearly positively correlated with the total number of spectral bands, and the correlation satisfies the following: ； in, This is the maximum number of groups. Number of basic groups The correlation coefficient, This represents the total number of spectral bands. By combining the band correlation threshold to filter groups, the band correlation within the same group is higher than the correlation between groups; feature encoding is performed on each band group to obtain the group feature vector of each band group, and the group feature vectors of all band groups constitute the group feature vector sequence. The deep sequence feature extraction module performs bidirectional temporal modeling on the group feature vector sequence, and simultaneously extracts spatial detail features from the group feature vector sequence through multi-scale convolutional layers. The bidirectional temporal modeling result and the multi-scale convolutional extraction result are concatenated to obtain the deep sequence feature vector; the feature encoding satisfies the formula: ； in, For a single band group, the group feature vector is... A pixel matrix of a spectral image for a single band group. The weight matrix for feature encoding. This is the bias vector for feature encoding. For activation functions; The bidirectional time series modeling satisfies the following formula: ； ； in, The band group number. To model the output vector corresponding to the t-th band group in the forward time series, To model the output vector corresponding to the (t-1)th band group in the forward time series, Let be the group feature vector of the t-th band group. The weight matrix for forward time series modeling. Bias vector for forward timing modeling To model the output vector corresponding to the t-th band group in the backward time series, To model the output vector corresponding to the (t+1)th band group in the backward time series, Weight matrix for backward time series modeling Bias vector for backward timing modeling It is the hyperbolic tangent activation function; The multi-scale convolution extraction satisfies the following formula: ； in, This represents the spatial detail feature vector corresponding to the t-th band group extracted by multi-scale convolution. This is the first convolution operation. For the second convolution operation, This is the third convolution operation. This is a vector concatenation operation.
4. The spectral image processing system based on deep sequence feature extraction according to claim 3, characterized in that: The band groups are divided according to the wavelength continuity and correlation of the spectral bands. Within the same band group, the wavelengths of the spectral bands are continuous and the correlation is higher than that between different band groups. The correlation is calculated and determined using a conventional correlation coefficient algorithm. A correlation coefficient threshold is set during the division, and this threshold is linearly positively correlated with the wavelength overlap of the spectral bands. The correlation relationship satisfies the following: ； in, For the correlation coefficient threshold, Based on the threshold, The correlation coefficient, The wavelength overlap is defined as follows: The multi-scale convolutional layer contains three convolutional operations with kernel sizes increasing in gradient. Each kernel size corresponds to a different spatial receptive field, and the size of the spatial receptive field increases with the kernel size. The convolutional results of the three scales are concatenated to obtain a spatial detail feature vector.
5. The spectral image processing system based on deep sequence feature extraction according to claim 4, characterized in that: The feature fusion module performs multi-scale fusion of shallow and deep features on the deep sequence feature vector, and the fusion process satisfies the formula: ； in, To fuse feature vectors, To integrate the weighting coefficients, This refers to the deep feature vector corresponding to the deep sequence feature vector. This is a shallow feature vector obtained by downsampling the feature vector of the deep sequence. The downsampling process employs pooling operations, which are conventional in the field, and the operation type is adaptively selected based on the dimension of the depth sequence feature vector. The values are linearly positively correlated with feature similarity, and the correlation relationship satisfies: ； in, To integrate the weighting coefficients, Based on the weighting coefficient, The correlation coefficient, The feature similarity is calculated using a cosine similarity algorithm based on the inner product of vectors.
6. The spectral image processing system based on deep sequence feature extraction according to claim 5, characterized in that: The parameter adaptive adjustment module acquires the feature complexity of the depth sequence feature vector in real time. The feature complexity is calculated using the feature entropy value, which satisfies the following formula: ； in, The feature entropy value, The characteristic component index, The total number of feature components in the feature vector of the depth sequence. Let be the probability distribution of the i-th feature component; The feature entropy value is linearly positively correlated with the feature complexity, and the correlation relationship satisfies: ； in, For feature complexity, Based on basic complexity, The correlation coefficient is used; the parameter adaptive adjustment module adjusts the feature encoding weight matrix, bidirectional temporal modeling weight matrix, and multi-scale convolution kernel parameters of the deep sequence feature extraction module according to the feature complexity, while simultaneously adjusting the fusion weight coefficient of the feature fusion module. As feature complexity increases, the parameter iteration frequency and... The values all increase synchronously with the feature complexity, and the iteration frequency of the parameter is linearly positively correlated with the feature complexity, satisfying the following relationship: ； in, For the parameter iteration frequency, Based on the iteration frequency, The correlation coefficient, For feature complexity; The value and feature complexity are linearly positively correlated, and the correlation satisfies: in, To integrate the weighting coefficients, Based on the weighting coefficient, The correlation coefficient, The feature complexity is denoted as .
7. The spectral image processing system based on deep sequence feature extraction according to claim 6, characterized in that: The spectral image processing output module performs feature inversion and spatial reconstruction on the fused feature vector to obtain a processed spectral image with the same size as the original spectral image. The feature inversion is achieved through a deconvolutional network, which upsamples the fused feature vector to restore the feature dimension. The spatial reconstruction maps the deconvolution-processed feature vectors to the pixel space of the original spectral image using a feature mapping algorithm, outputting the processed spectral image. The number of layers and the upsampling ratio of the deconvolution network are adaptively matched according to the dimension of the fused feature vectors and the size of the original spectral image. The number of layers in the deconvolution network is linearly positively correlated with the dimension of the feature vectors, and their correlation satisfies the following: in, This represents the number of layers in the deconvolutional network. Based on the number of layers, The correlation coefficient, The dimension of the feature vector.