Dual-mode fusion wavelet enhancement upper six pieces of hyperspectral classification method and system

Abstract

This invention discloses a dual-modal fusion wavelet-enhanced six-image hyperspectral classification method and system. The method includes: acquiring spectral and RGB image data of the six images; extracting features from the spectral and RGB image data respectively; enhancing the feature-extracted data using a learnable wavelet enhancement module; flattening and stitching the enhanced data; fusing the stitched feature sequences to obtain a fused feature sequence; embedding location information and category labels into the fused feature sequence, and feeding it into a Transformer encoder for global context modeling and classification. This invention's dual-modal fusion wavelet-enhanced six-image hyperspectral classification method and system achieves the fusion and utilization of hyperspectral imaging, near-infrared spectroscopy, and digital image features through feature-level and data-level fusion, fully utilizing the spatial-spectral information of multiple data sources, leveraging the advantages of various data, and effectively overcoming the limitations of single-modal methods.

Description

Technical Field

[0001] This invention relates to the field of tobacco leaf classification technology, and more specifically, to a dual-modal fusion wavelet-enhanced six-element hyperspectral classification method and system. Background Technology

[0002] The top six leaves, used as raw materials for Henan Tobacco's high-end "Golden Leaf" series, have extremely strict quality requirements. Currently, manual sorting is widely used, with professional technicians grading the tobacco leaves by sight, touch, and smell. This process is time-consuming, labor-intensive, and demands a high level of expertise from the technicians. The evaluation of the tobacco leaf appearance quality, in particular, presents a significant challenge to the technicians' skills due to the numerous appearance factors and grading levels involved.

[0003] Hyperspectral imaging (HSI) technology is based on image data from hundreds of narrow bands, combining the advantages of both imaging and spectral techniques to acquire two-dimensional geometric space and one-dimensional spectral information of the target. Leveraging its rich spatial-spectral information, HSI can accurately capture subtle differences in objects, playing a crucial role in fields such as remote sensing, biomedicine, food science, mineral exploration, agricultural monitoring, and materials science. Therefore, using hyperspectral technology to obtain the quality of the top six tobacco leaves and establishing a correlation between hyperspectral data and the appearance quality of these leaves enables rapid, non-destructive, and automated evaluation of tobacco leaf appearance quality, contributing to the industry's intelligent, automated, and digital transformation.

[0004] Therefore, there is an urgent need for a wavelet-enhanced six-image hyperspectral classification method and system with dual-modal fusion. Summary of the Invention

[0005] The purpose of this invention is to provide a wavelet-enhanced six-image hyperspectral classification method and system with dual-modal fusion to solve the problems in the prior art. It can realize the fusion and utilization of hyperspectral imaging, near-infrared spectroscopy and digital image features, make full use of the spatial-spectral information of multiple data, give full play to the advantages of multiple data, and effectively overcome the limitations of single-modal methods.

[0006] This invention provides a wavelet-enhanced six-image hyperspectral classification method with dual-modal fusion, comprising:

[0007] Acquire the spectral data and RGB image data of the six images above, wherein the spectral data includes hyperspectral data and near-infrared spectral data;

[0008] Feature extraction is performed on the spectral data and the RGB image data respectively;

[0009] The spectral data and the RGB image data after feature extraction are enhanced using a learnable wavelet enhancement module;

[0010] The enhanced spectral data and the RGB image data are flattened and stitched together to obtain a stitched feature sequence.

[0011] The spliced ​​feature sequence is subjected to feature fusion processing to obtain a fused feature sequence;

[0012] The location information and category label of the features are embedded into the fused feature sequence, and the fused feature sequence is fed into the Transformer encoder for global context modeling and classification.

[0013] The wavelet-enhanced six-image hyperspectral classification method with dual-modal fusion as described above, preferably, involves feature extraction from the spectral data and the RGB image data, respectively, including:

[0014] Dimensionality reduction processing is performed on the hyperspectral data and the RGB image data respectively;

[0015] The dimensionality-reduced hyperspectral and RGB image input data are divided into blocks to obtain spectral blocks and image blocks respectively.

[0016] Feature extraction is performed on the spectral block and the image block according to the modality alignment principle.

[0017] The wavelet-enhanced six-image hyperspectral classification method with dual-modal fusion as described above, preferably, involves extracting features from the spectral blocks and the image blocks according to modal alignment principles, including:

[0018] The spectral block is feature extracted using a heterogeneous convolutional dual-modal feature extraction module;

[0019] Feature extraction is performed on the image patch using a spatially aware multi-scale excitation module.

[0020] The wavelet-enhanced six-pane hyperspectral classification method using dual-modal fusion as described above, preferably, involves extracting features from the spectral blocks using a heterogeneous convolutional dual-modal feature extraction module, including:

[0021] The spectral-spatial joint features of the spectral block are mined using 3D convolutional layers;

[0022] Further extraction of spatial details is achieved using 2D convolution.

[0023] The feature extraction of the image patch using the spatially aware multi-scale excitation module includes:

[0024] Multi-scale, multi-class convolution is used to extract rich spatial information from the image patch, specifically including:

[0025] Use small convolution kernels to focus on local texture details;

[0026] Large convolutional kernels and dilated convolutions are used to expand the receptive field and extract rich spatial information;

[0027] The extracted multi-scale features are then concatenated.

[0028] Global average pooling is used to compress the concatenated multi-scale features;

[0029] Adaptive channel activation is performed through fully connected layers to enhance feature channels that are critical to the classification task and suppress redundant channels.

[0030] Feature fusion and dimensionality adjustment are performed using convolutional layers with a kernel size of 1×1.

[0031] The dual-modal fusion wavelet-enhanced six-image hyperspectral classification method described above preferably includes the following: The enhancement processing of the feature-extracted spectral data and the RGB image data using a learnable wavelet enhancement module includes:

[0032] Initialization is performed using Haar standard wavelets, and the optimal filter parameters are learned through backpropagation. Let the input feature map be... , This represents the feature map obtained through the feature extraction step, where, Indicates batch size, This indicates the number of channels, with a value of 64. Indicates the height of the feature map, The width of the feature map is represented by four components obtained through discrete wavelet transform, expressed by the following formula:

[0033] Where DWT represents Discrete Wavelet Transform, This represents the low-frequency component that contains global structural information about the features. This indicates the high-frequency components that capture horizontal texture information; Represents the high-frequency vertical components that encode vertical texture information; The diagonal high-frequency components represent edge and texture abrupt change information, and the spatial dimensions of all four components are... The number of channels is ;

[0034] Low frequency components Horizontal high-frequency components Vertical high frequency components Diagonal high frequency components These four components are concatenated along the channel dimension to obtain the fused feature, which is expressed by the following formula:

[0035]

[0036] in, Indicates fusion features, Indicates concatenation operators;

[0037] Using convolutional modules to fuse features The processing includes:

[0038] Fusion features The number of channels is expanded to 6C by passing through a convolutional layer with a kernel size of 1×1;

[0039] Fusion features A deep separable convolutional layer with a kernel size of 7×7 is used to extract cross-channel spatially relevant features;

[0040] Fusion features The GELU activation function is applied to increase nonlinearity and enhance feature representation capabilities.

[0041] Fusion features The convolutional module passes through a 1×1 convolutional layer to reduce the number of channels to 4C for feature reconstruction. The output is:

[0042]

[0043] in, Represents convolutional modules Output, ( ) indicates the operation of the convolution module;

[0044] Convolutional module Output Divide the channel into four groups evenly along the channel dimension, with each group containing [number of channels]. The four processed components are represented by the following formula:

[0045]

[0046] in, This indicates the corresponding low-frequency component. , , indicates the corresponding vertical high-frequency component. This indicates the corresponding high-frequency components at the diagonal. This indicates a uniform division operation;

[0047] The output feature map is reconstructed using inverse discrete wavelet transform, as expressed by the following formula:

[0048]

[0049] Where IDWT represents the inverse discrete wavelet transform. This represents the reconstructed features, which maintain the same spatial dimension H×W and number of channels C as the original input features.

[0050] The wavelet-enhanced six-image hyperspectral classification method with dual-modal fusion as described above, preferably, involves flattening and stitching the enhanced spectral data and the RGB image data to obtain a stitched feature sequence, including:

[0051] The enhanced hyperspectral data and RGB image data are flattened and then stitched together with near-infrared features along the feature dimension to obtain a stitched feature sequence.

[0052] The wavelet-enhanced six-image hyperspectral classification method with dual-modal fusion as described above, preferably, involves performing feature fusion processing on the spliced ​​feature sequence to obtain a fused feature sequence, including:

[0053] The concatenated feature sequence is fused using the feature fusion module of the residual tokenizer to obtain a fused feature sequence, specifically including:

[0054] The concatenated feature sequence is used as the input feature matrix X.

[0055] in, B represents the stitched feature tensor, H and W represent the size of the image patch, and C represents the number of feature channels.

[0056] Using attention weights Controlling the attention distribution of tokens and utilizing content weights Controlling the content expression of the token to achieve adaptive token generation.

[0057] Where L represents the preset number of tokens, with a value of 4, and C represents the preset content features, with a value of 64;

[0058] The feature fusion module combines the input feature matrix X with attention weights. Perform a transposed dot product to construct the original attention score, and introduce a learnable parameter T to adjust the sharpness of the attention distribution. Normalize the result using the Softmax function to obtain the attention distribution A, expressed by the following formula:

[0059]

[0060] in, Each element in the attention distribution A represents the attention weight of the Lth Token to the 2HWth spatial location of the input feature;

[0061] The feature fusion module combines the input feature matrix X with content weights. Perform a linear transformation to generate basic content features V, while preserving the channel information of the input features;

[0062] The token content representation T is obtained by weighted aggregation of the basic V using the attention distribution A.

[0063] By introducing residual connections, the mean of the basic content feature V across the spatial dimension (dim=1) is calculated to obtain the global content feature. Then, it is added to the aggregated T, as expressed by the following formula:

[0064]

[0065]

[0066]

[0067] in, , That is, each sample is compressed into L tokens with dimension C.

[0068] The wavelet-enhanced six-image hyperspectral classification method with dual-modal fusion as described above, preferably, involves embedding the feature location information and category labels into the fused feature sequence, and feeding the fused feature sequence into a Transformer encoder for global context modeling classification, including:

[0069] The Transformer module is used to globally model the fused features after Tokenizer compression, and a self-attention mechanism is employed to capture long-distance dependencies and complex spatial relationships between features. Specifically, this includes:

[0070] During the input phase, the model adds a learnable category label to the compressed token sequence and combines it with positional encoding to introduce sequence information;

[0071] The Transformer encoder first interacts with the token sequence through multi-head self-attention, so that each token can pay attention to the features of all other positions in the sequence;

[0072] The outputs of each attention point are concatenated and fused to enhance the expressive power of the features;

[0073] The fused sequence of each attention head is subjected to nonlinear transformation and feature mapping through a feedforward fully connected network;

[0074] The entire module uses residual connections and layer normalization to stabilize the training process and improve feature transfer efficiency;

[0075] The category label serves as an aggregated representation of the global features, and the classification result is output through a fully connected layer.

[0076] The dual-modal fusion wavelet-enhanced six-image hyperspectral classification method described above preferably further includes:

[0077] A loss function is constructed by employing a joint optimization strategy of cross-entropy loss and wavelet loss, specifically including:

[0078] The cross-entropy loss is calculated using the following formula:

[0079]

[0080] in, The cross-entropy loss value is represented by c, where c represents the category index and C represents the total number of categories in the tobacco classification task. This represents the true label value of the sample in class c, and is a variable between 0 and 1. This represents the predicted probability value of the model for a sample in class c;

[0081] The wavelet loss consists of two parts: perfect reconstruction loss and aliasing cancellation loss. The calculation process of the perfect reconstruction loss includes:

[0082] The perfect reconstruction constraint formula can be constructed using the following formula:

[0083]

[0084] in, Represents frequency domain variables, This represents the frequency domain expression of a decomposed low-pass filter. This represents the frequency domain expression of a decomposed high-pass filter. This represents the frequency domain expression of the reconstructed low-pass filter. This represents the frequency domain expression of the reconstructed high-pass filter;

[0085] The actual output P is obtained by polynomial multiplication of a one-dimensional convolutional analog filter. test And calculate P test The mean square error of the ideal output, as the perfect reconstruction loss, is expressed by the following formula:

[0086]

[0087] in, This represents the loss value for perfect reconstruction. Represents the unit impulse function;

[0088] The calculation process for aliasing elimination loss includes: constructing the aliasing elimination condition constraint formula using the following formula:

[0089]

[0090] Here, −z represents the inverse frequency variable, which is simulated by alternating the signs of the filter coefficients. The −z substitution is achieved by using the filter alternating signal [1,-1,1,-1,...] to construct the mask;

[0091] Decompose the frequency domain expression of the low-pass filter. Decompose the frequency domain expression of the high-pass filter. Multiply by the mask and then by the corresponding reconstruction filter , Perform a one-dimensional convolution, and sum the convolution results to obtain the actual output P. test Calculate P test The aliasing elimination loss is obtained by taking the mean square error between the ideal output all-zero tensor and the mean square error, and is expressed by the following formula:

[0092]

[0093] in, This represents the aliasing elimination loss value. This represents the ideal output value for aliasing elimination;

[0094] The total wavelet transform loss is calculated using the following formula:

[0095]

[0096] in, This represents the total loss of the wavelet transform;

[0097] Calculate the total loss using the following formula;

[0098]

[0099] in, This represents the total loss of the model.

[0100] The present invention also provides a wavelet-enhanced six-chip hyperspectral classification system employing the above-described method, comprising:

[0101] The data acquisition module is used to acquire spectral data and RGB image data of the six images above, wherein the spectral data includes hyperspectral data and near-infrared spectral data;

[0102] The feature extraction module is used to extract features from the spectral data and the RGB image data respectively;

[0103] The wavelet enhancement module is used to enhance the spectral data and the RGB image data after feature extraction using a learnable wavelet enhancement module;

[0104] The feature stitching module is used to flatten and stitch the enhanced spectral data and the RGB image data to obtain a stitched feature sequence.

[0105] The feature fusion module is used to perform feature fusion processing on the spliced ​​feature sequence to obtain a fused feature sequence.

[0106] The encoding and classification module is used to embed the location information and category label of the features into the fused feature sequence, and then feed the fused feature sequence into the Transformer encoder for global context modeling and classification.

[0107] This invention provides a dual-modal fusion wavelet-enhanced six-image hyperspectral classification method and system, proposing a WEMSTT three-branch architecture: through feature-level + data-level fusion, it achieves the fusion and utilization of features from hyperspectral imaging, near-infrared spectroscopy, and digital images, fully utilizing the spatial-spectral information of multiple data sources, leveraging the advantages of various data, and effectively overcoming the limitations of single-modal methods; a feature extraction module combining 3D convolution and 2D convolution is designed: first, 3D convolution is used to mine spectral-spatial joint features in hyperspectral data, and then 2D convolutional layers are used to further optimize spatial details; by narrowing the receptive field to focus on local texture and edge information, the discriminative power of features is enhanced; a learnable wavelet feature extraction module is designed: through backpropagation to learn wavelet filters, HSI and RGB data are adaptively processed in the spectral domain, effectively extracting key features under multimodal conditions, while reducing the interference of redundant information, thereby achieving better performance; an effective feature fusion is achieved through a residual tokenizer module: by leveraging the idea of ​​visual Transformer (ViT) feature compression, learnable attention, content mapping, and residual enhancement are fused to achieve efficient compression and information recombination of cross-modal high-dimensional fusion features. Attached Figure Description

[0108] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings, wherein:

[0109] Figure 1 A flowchart illustrating an embodiment of the wavelet-enhanced six-chip hyperspectral classification method with dual-modal fusion provided by the present invention;

[0110] Figure 2 This is an architecture diagram of the WEMSTT general model;

[0111] Figure 3This is an architecture diagram of the multimodal feature extraction module;

[0112] Figure 4 This is an architecture diagram of the enhanced learnable wavelet transform module.

[0113] Figure 5 This is an architecture diagram of the Tokenizer module;

[0114] Figure 6 Architecture diagram of the Transformer encoding module for multi-head self-attention;

[0115] Figure 7 This is a structural block diagram of an embodiment of the dual-modal fusion wavelet-enhanced six-chip hyperspectral classification system provided by the present invention. Detailed Implementation

[0116] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. The descriptions of the exemplary embodiments are merely illustrative and are in no way intended to limit the present disclosure or its application or use. The present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are provided so that the present disclosure will be thorough and complete, and will fully express the scope of the disclosure to those skilled in the art. It should be noted that, unless specifically stated otherwise, the relative arrangement of components and steps, the composition of materials, numerical expressions, and values ​​set forth in these embodiments should be interpreted as exemplary only and not as limiting.

[0117] The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different parts. Terms such as “including” or “contains” mean that the element preceding the term encompasses the element listed after it, and do not exclude the possibility of encompassing other elements as well. Terms such as “above” and “below” are used only to indicate relative positional relationships; when the absolute position of the described object changes, this relative positional relationship may also change accordingly.

[0118] In this disclosure, when a specific component is described as being located between a first component and a second component, an intermediary component may or may not be present between the specific component and the first or second component. When a specific component is described as connecting to other components, the specific component may be directly connected to the other components without having an intermediary component, or it may not be directly connected to the other components but may have an intermediary component.

[0119] All terms used in this disclosure (including technical or scientific terms) have the same meaning as understood by one of ordinary skill in the art to which this disclosure pertains, unless otherwise specifically defined. It should also be understood that terms defined in a general dictionary, such as a dictionary, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or highly formalized meaning, unless expressly defined herein.

[0120] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0121] Traditional hyperspectral classification tasks primarily rely on machine learning methods such as Support Vector Machines (SVM) and Random Forests. However, these methods exhibit limitations when handling high-dimensional, nonlinear features, failing to achieve satisfactory high-precision classification results and thus limiting their application in engineering practice. Convolutional Neural Networks (CNNs) have shone brightly in computer vision and image processing. By utilizing fixed-scale convolutional kernels, CNNs can effectively capture local information in images, and through the combined action of multiple kernels, they can achieve good image classification. However, due to the inherent scale of the convolutional kernels, the receptive field of a single kernel is limited, making it unsuitable for effectively describing global information.

[0122] In recent years, deep learning networks based on the Transformer architecture have emerged in the field of hyperspectral imaging (HSI) processing. Leveraging its self-attention mechanism, the Transformer effectively captures long-range dependencies, exhibiting superior global context modeling capabilities and powerful feature extraction performance. Although Transformer-based classification methods have achieved impressive performance across various tasks, some challenges remain, especially in the context of the rapid development of multimodal data fusion. Effectively utilizing the complementarity of different data modalities to further improve classification accuracy has become a key research focus.

[0123] like Figure 1 and Figure 2 As shown, the dual-modal fusion wavelet-enhanced six-image hyperspectral classification method provided in this embodiment includes the following steps in actual implementation:

[0124] Step S1: Obtain the spectral data and RGB image data of the six images above, wherein the spectral data includes hyperspectral data and near-infrared spectral data.

[0125] In one embodiment of the present invention, the acquired dataset of the six tobacco leaves includes spectral and RGB images, which are spatially corresponding. The hyperspectral image contains 480 bands, covering the wavelength from 396.7 nm to 1004.1 nm, and the image size is 856 × 1092 pixels. The near-infrared spectrum is obtained by grinding the corresponding type of tobacco leaf samples into powder form.

[0126] Step S2: Extract features from the spectral data and the RGB image data respectively.

[0127] In this invention, a multimodal feature extraction module (such as...) is used. Figure 3 (As shown) Feature extraction is performed on the spectral data and the RGB image data respectively. In one embodiment of the dual-modal fusion wavelet-enhanced six-image hyperspectral classification method of the present invention, step S2 may specifically include:

[0128] Step S21: Perform dimensionality reduction processing on the hyperspectral data and the RGB image data respectively.

[0129] Step S22: Perform block partitioning on the dimensionality-reduced hyperspectral and RGB image input data respectively to obtain spectral blocks and image blocks.

[0130] Step S23: According to the modal alignment principle, feature extraction is performed on the spectral block and the image block respectively.

[0131] In one embodiment of the wavelet-enhanced six-image hyperspectral classification method of the present invention, the step S23 may specifically include:

[0132] Step S231: Extract features from the spectral block using the heterogeneous convolutional dual-modal feature extraction module.

[0133] In one embodiment of the wavelet-enhanced six-chip hyperspectral classification method of the present invention, the step S231 may specifically include:

[0134] Step S2311: Mine the spectral-spatial joint features of the spectral block through a 3D convolutional layer.

[0135] By using step S2311, the three-dimensional structure (height × width × spectral dimension) of hyperspectral data can be preserved while avoiding the fragmentation of continuous information in the spectral dimension caused by traditional 2D convolution.

[0136] Step S2312: Further extract spatial details using 2D convolution.

[0137] By step S2312, the receptive field can be narrowed to focus on local texture and edge information, providing a more accurate spatial basis for the subsequent wavelet enhancement module.

[0138] Step S232: Use the spatially aware multi-scale excitation module to extract features from the image patch.

[0139] In this invention, multi-scale, multi-class convolution is used to extract rich spatial information from the image patch. In one embodiment of the dual-modal fusion wavelet-enhanced six-image hyperspectral classification method of this invention, step S232 may specifically include:

[0140] Step S2321: Use a small convolution kernel (e.g., a kernel size of 3×3) to focus on local texture details.

[0141] Step S2322: Expand the receptive field by using large convolution kernels (e.g., kernel size of 5×5) and dilated convolution (e.g., kernel size of 3×3) to extract rich spatial information.

[0142] Step S2323: Concatenate the extracted multi-scale features.

[0143] Step S2324: Perform feature compression on the spliced ​​multi-scale features using global average pooling.

[0144] Step S2325: Adaptive channel activation is performed through a fully connected layer to enhance the feature channels that are key to the classification task (such as the target texture channel) and suppress redundant channels (such as the background noise channel).

[0145] Step S2326: Perform feature fusion and dimension adjustment using a convolutional layer with a kernel size of 1×1.

[0146] By using steps S2321-S2326, the number of channels can be compressed to a dimension that matches the hyperspectral branch while integrating multi-scale calibration features, thus preparing for subsequent modal feature stitching.

[0147] Step S3: Utilize a learnable wavelet enhancement module (such as...) Figure 4 As shown, the spectral data and RGB image data after feature extraction are enhanced respectively.

[0148] In the learnable wavelet transform feature enhancement module, the mathematical properties of traditional wavelet transform are combined with the learnability of deep learning. Efficient feature extraction is achieved through wavelet multi-scale decomposition and dynamic feature selection, adaptively adjusting attention weights to enhance feature representation while maintaining perfect reconstruction characteristics. In one embodiment of the dual-modal fusion wavelet enhancement method for six-image hyperspectral classification of this invention, step S3 may specifically include:

[0149] Step S31: Initialize using Haar standard wavelets, learn the optimal filter parameters through backpropagation, and let the input feature map be... , This represents the feature map obtained through the feature extraction step, where, Indicates batch size, This indicates the number of channels, with a value of 64. Indicates the height of the feature map, The width of the feature map is represented by four components obtained through discrete wavelet transform, expressed by the following formula:

[0150] Where DWT represents Discrete Wavelet Transform, This represents the low-frequency component that contains global structural information about the features. This indicates the high-frequency components that capture horizontal texture information; Represents the high-frequency vertical components that encode vertical texture information; The diagonal high-frequency components represent edge and texture abrupt change information, and the spatial dimensions of all four components are... The number of channels is .

[0151] Step S32: Convert the low-frequency component Horizontal high-frequency components Vertical high frequency components Diagonal high frequency components These four components are concatenated along the channel dimension to obtain the fused feature, which is expressed by the following formula:

[0152]

[0153] in, Indicates fusion features, This indicates a concatenation operator.

[0154] Step S33: Utilize the convolution module ( ) on fusion features Process it.

[0155] In one embodiment of the wavelet-enhanced six-chip hyperspectral classification method of the present invention, the step S33 may specifically include:

[0156] Step S331, Feature Fusion The convolutional layer with a kernel size of 1×1 is used to expand the number of channels to 6C.

[0157] Step S332, Feature Fusion A deep separable convolutional layer with a kernel size of 7×7 is used to extract cross-channel spatially relevant features.

[0158] Step S333, Feature Fusion The GELU activation function is applied to increase nonlinearity and enhance feature representation capabilities.

[0159] Step S334, Feature Fusion The convolutional module passes through a 1×1 convolutional layer to reduce the number of channels to 4C for feature reconstruction. The output is:

[0160]

[0161] in, Represents convolutional modules Output, () indicates the operation of the convolution module.

[0162] Step S34: Convolution module Output The channel is uniformly divided into four chunks along the channel dimension, with each chunk containing [number of channels]. The four processed components are represented by the following formula:

[0163]

[0164] in, This indicates the corresponding low-frequency component. , , indicates the corresponding vertical high-frequency component. This indicates the corresponding high-frequency components at the diagonal. This indicates a uniform division operation.

[0165] Step S35: Reconstruct the output feature map using inverse discrete wavelet transform, expressed by the following formula:

[0166]

[0167] Where IDWT represents the inverse discrete wavelet transform. This represents the reconstructed features, which maintain the same spatial dimension H×W and number of channels C as the original input features.

[0168] In summary, the processing procedure for the hyperspectral block branch is as follows: the input data is sequentially passed through 3D convolutional layers and 2D convolutional layers, and then input into a learnable wavelet transform enhancement module to strengthen spectral features and further extract spatial features. For the RGB branch: the input data is processed through a multi-scale convolutional module, and then through a learnable wavelet enhancement module to focus on enhancing its spatial representation capabilities.

[0169] Step S4: Flatten and stitch the enhanced spectral data and the RGB image data to obtain a stitched feature sequence.

[0170] Specifically, the hyperspectral data and RGB image data after enhancement processing are flattened and stitched together with near-infrared features along the feature dimension to obtain a stitched feature sequence.

[0171] Step S5: Perform feature fusion processing on the spliced ​​feature sequence to obtain a fused feature sequence.

[0172] In this invention, the feature fusion module of the residual tokenizer (such as...) is utilized. Figure 5 (As shown) The spliced ​​feature sequence is subjected to feature fusion processing to obtain a fused feature sequence, thereby achieving effective fusion of modal information. The feature fusion module of the residual tokenizer is responsible for mapping and compressing the fused hyperspectral and RGB features from high-dimensional space into a compact token with strong expressive power. Its design is based on the tokenization idea of ​​Visual Transformer (ViT) and is specifically optimized for the high-dimensional characteristics of hyperspectral data and the rich spatial information of RGB data to better achieve efficient encoding of cross-modal features. In one embodiment of the wavelet-enhanced six-piece hyperspectral classification method of the present invention, step S5 may specifically include:

[0173] Step S51: Use the concatenated feature sequence as the input feature matrix X.

[0174] in, denoted by , B represents the batch size, H and W represent the size of the image patch, and C represents the number of feature channels.

[0175] Step S52: Utilize attention weights Controlling the attention distribution of tokens and utilizing content weights Controlling the content expression of the token to achieve adaptive token generation.

[0176] Where L represents the preset number of tokens, with a value of 4, and C represents the preset content features, with a value of 64. By introducing... and These two sets of core learnable parameters enable adaptive token generation.

[0177] Step S53: The feature fusion module combines the input feature matrix X with the attention weights. Perform a transposed dot product to construct the original attention score, and introduce a learnable parameter T to adjust the sharpness of the attention distribution. Normalize the result using the Softmax function to obtain the attention distribution A, expressed by the following formula:

[0178]

[0179] in, Each element in the attention distribution A represents the attention weight of the Lth Token to the 2HWth spatial location of the input feature.

[0180] Step S54: The feature fusion module processes the input feature matrix X and content weights. Perform a linear transformation to generate the basic content feature V, while preserving the channel information of the input feature.

[0181] Step S55: Use the attention distribution A to perform weighted aggregation on the basic V to obtain the Token content representation T.

[0182] Step S56: Introduce residual connections and calculate the mean of the basic content feature V in the spatial dimension (dim=1) to obtain the global content feature. Then, it is added to the aggregated T, as expressed by the following formula:

[0183]

[0184]

[0185]

[0186] in, , That is, each sample is compressed into L tokens with dimension C.

[0187] By introducing residual connections, global feature information can be avoided during token generation.

[0188] In summary, the feature fusion module of this invention achieves efficient compression and information recombination of cross-modal high-dimensional fusion features through a design of "learnable attention + content mapping + residual enhancement". Compared with traditional global pooling methods, it avoids the loss of discriminative information caused by "averaging" and can adaptively focus on key regions and channels. Compared with ViT's fixed patch partitioning, this module does not require preset segmentation rules and can dynamically select effective features according to data characteristics, thereby reducing redundancy. In addition, its compact output token can significantly reduce the input dimension of the subsequent Transformer encoder while retaining core discriminative information, balancing modeling efficiency and expressive power.

[0189] Step S6: Embed the location information and category label of the features into the fused feature sequence, and send the fused feature sequence into the Transformer encoder for global context modeling and classification.

[0190] The Transformer module is used to globally model the fused features after Tokenizer compression, and a self-attention mechanism is employed to capture long-distance dependencies and complex spatial relationships between features. Figure 6 An architecture diagram of a multi-head self-attention Transformer coding module is shown. In one embodiment of the dual-modal fusion wavelet-enhanced six-chip hyperspectral classification method of the present invention, step S6 may specifically include:

[0191] Step S61, Input Phase: The model adds a learnable category label to the front of the compressed token sequence and combines it with positional encoding to introduce sequence information.

[0192] In one embodiment of the present invention, a classification task is performed on the dataset obtained in step S1 using six tobacco leaf evaluation indicators: Color, Maturity, Structure, Body, Oiliness, and Chroma.

[0193] Table 1. Category information for the six classification tasks used for evaluation.

[0194]

[0195] Step S62: The Transformer encoder first interacts with the token sequence through multi-head self-attention so that each token can pay attention to the features of all other positions in the sequence.

[0196] Step S62 allows for the fusion of the global context.

[0197] Step S63: Concatenate and fuse the outputs of each attention head to enhance the expressive power of the features.

[0198] Step S64: The fused sequence of each attention head is subjected to nonlinear transformation and feature mapping through a feedforward fully connected network.

[0199] Step S65: The entire module uses residual connections and layer normalization to stabilize the training process and improve feature transfer efficiency.

[0200] Step S66: The category label is used as an aggregated representation of the global features, and the classification result is output through a fully connected layer.

[0201] Through steps S61-S66, the Transformer module can adaptively focus on the most discriminative feature region, achieving deep fusion of cross-modal information while maintaining low computational overhead and good generalization ability.

[0202] Furthermore, in one embodiment of the present invention, the wavelet-enhanced six-image hyperspectral classification method with dual-modal fusion further includes:

[0203] Step S7: Construct a loss function by employing a joint optimization strategy of cross-entropy loss and wavelet loss.

[0204] This invention guides the model's learning from two dimensions: category discrimination accuracy and wavelet transform performance. The core function of cross-entropy loss is to guide the model to learn accurate category discrimination features; its optimization logic is directly related to the category prediction probability. The higher the model's prediction probability of the sample's "true category," the smaller the loss value; conversely, if the model's prediction probability of the true category is too low (or the prediction probability of the incorrect category is too high), the loss value will increase significantly. Through backpropagation of this loss, the model parameters can be directly pushed towards the optimal direction of "improving category discrimination accuracy," making it the core loss term for optimizing category prediction performance in classification tasks.

[0205] Wavelet loss is the core constraint in the learnable wavelet transform feature enhancement module. It aims to optimize the wavelet filter bank through specific loss terms to ensure that it has excellent signal reconstruction and aliasing suppression capabilities. The core consists of two parts: perfect reconstruction loss and aliasing elimination loss.

[0206] In one embodiment of the wavelet-enhanced six-image hyperspectral classification method of the present invention, the step S7 may specifically include:

[0207] Step S71: Calculate the cross-entropy loss using the following formula:

[0208]

[0209] in, represents the cross-entropy loss value, used to measure the difference between the model's predicted probability and the true label; the smaller the value, the higher the prediction accuracy. 'c' represents the category index, a count variable used to iterate through all categories. 'C' represents the total number of categories in the tobacco classification task, corresponding to the total number of categories for the six evaluation metrics, including Color and Maturity. This represents the true label value of the sample in class c, and is a variable between 0 and 1 (1 if it belongs to this class, 0 otherwise). This represents the predicted probability value of the model for the sample in class c, with a value ranging from 0 to 1;

[0210] Step S72: The wavelet loss consists of two parts: perfect reconstruction loss and aliasing elimination loss.

[0211] In one embodiment of the wavelet-enhanced six-image hyperspectral classification method of the present invention, the step S72 may specifically include:

[0212] Step S721: Calculate the perfect reconstruction loss.

[0213] Perfect reconstruction loss, i.e., satisfying the requirement of distortion-free restoration after signal decomposition and reconstruction. In one embodiment of the wavelet enhancement method for six-image hyperspectral classification using dual-modal fusion of the present invention, step S721 may specifically include:

[0214] Step S7211: Construct the perfect reconstruction constraint formula using the following formula:

[0215]

[0216] in, Represents frequency domain variables, used to characterize the frequency domain characteristics of a filter. This represents the frequency domain expression of a decomposed low-pass filter. This represents the frequency domain expression of a decomposed high-pass filter. This represents the frequency domain expression of the reconstructed low-pass filter. This represents the frequency domain expression of the reconstructed high-pass filter.

[0217] Step S7212: Obtain the actual output P through polynomial multiplication of a one-dimensional convolutional analog filter. test And calculate P test The mean square error of the ideal output, as the perfect reconstruction loss, is expressed by the following formula:

[0218]

[0219] in, This represents the perfect reconstruction loss value, used to measure the degree of distortion-free signal restoration after wavelet decomposition and reconstruction. The smaller the value, the higher the reconstruction accuracy. This represents the unit impulse function, which takes the value 1 only when the independent variable is 0, and is 0 in other cases. It is used to calibrate the output reference of the ideal reconstruction.

[0220] Step S722: Calculate the aliasing elimination loss.

[0221] In one embodiment of the wavelet-enhanced six-chip hyperspectral classification method of the present invention, the step S722 may specifically include:

[0222] Step S7221: Construct the aliasing elimination condition constraint formula using the following formula:

[0223]

[0224] Here, −z represents the inverse frequency variable, which is simulated by alternating the signs of the filter coefficients. The −z substitution is achieved using the filter's alternating signals [1,-1,1,-1,...], thus constructing a mask. The inverse frequency variable, by alternating the signs of the filter coefficients, simulates the inverse frequency characteristics and is used to suppress frequency aliasing during signal decomposition.

[0225] Signal decomposition is prone to distortion due to frequency aliasing, and aliasing must be eliminated to remove the constraints.

[0226] Step S7222: Decompose the frequency domain expression of the low-pass filter. Decompose the frequency domain expression of the high-pass filter. Multiply by the mask and then by the corresponding reconstruction filter , Perform a one-dimensional convolution, and sum the convolution results to obtain the actual output P. test Calculate P test The aliasing elimination loss is obtained by taking the mean square error between the ideal output all-zero tensor and the mean square error, and is expressed by the following formula:

[0227]

[0228] in, This represents the aliasing reduction loss value, used to measure the effectiveness of frequency aliasing suppression. The smaller the value, the less severe the aliasing. This represents the ideal output value for aliasing elimination, i.e., the theoretical calculation result when there is no frequency aliasing.

[0229] Step S7223: Calculate the aliasing elimination loss using the following formula:

[0230]

[0231] in, This represents the total loss of the wavelet transform, used to comprehensively constrain the signal reconstruction capability and aliasing suppression of the wavelet filter.

[0232] Step S723: Calculate the total loss using the following formula:

[0233]

[0234] in, This represents the total loss of the model.

[0235] This invention proposes a hyperspectral classification model (WEMSTT) network architecture based on a wavelet-enhanced multi-modal spectral-spatial tokenized Transformer. This network consists of four modules: a depthwise convolution-based multimodal feature extraction module to extract rich multi-scale features from the original input; a wavelet feature enhancement module to optimize spectral feature representation; a feature fusion module to effectively fuse features from different data sources; and a Transformer encoding module to establish global feature dependencies. Verification shows that the collaborative multi-module design enables the network to effectively mine data information and achieve accurate classification of the appearance of the top six tobacco leaves. The total model loss is... It is the core optimization objective of the entire WEMSTT network, and its value is minimized through backpropagation to improve model performance.

[0236] Correspondingly, such as Figure 7 As shown, the present invention also provides a wavelet-enhanced six-chip hyperspectral classification system employing the above-described method, comprising:

[0237] The data acquisition module 11 is used to acquire the spectral data and RGB image data of the six images above, wherein the spectral data includes hyperspectral data and near-infrared spectral data;

[0238] Feature extraction module 12 is used to extract features from the spectral data and the RGB image data respectively;

[0239] Wavelet enhancement module 13 is used to enhance the spectral data and the RGB image data after feature extraction using a learnable wavelet enhancement module;

[0240] Feature stitching module 14 is used to flatten and stitch the enhanced spectral data and the RGB image data to obtain a stitched feature sequence.

[0241] Feature fusion module 15 is used to perform feature fusion processing on the spliced ​​feature sequence to obtain a fused feature sequence;

[0242] The encoding and classification module 16 is used to embed the location information and category label of the features into the fused feature sequence, and to send the fused feature sequence into the Transformer encoder for global context modeling and classification.

[0243] The dual-modal fusion wavelet-enhanced six-image hyperspectral classification method and system provided in this invention proposes a WEMSTT three-branch architecture: through feature-level + data-level fusion, it realizes the fusion and utilization of features from hyperspectral imaging, near-infrared spectroscopy, and digital images, making full use of the spatial-spectral information of multiple data, leveraging the advantages of multiple data, and effectively overcoming the limitations of single-modal methods; a feature extraction module combining 3D convolution and 2D convolution is designed: first, 3D convolution is used to mine the spectral-spatial joint features in hyperspectral data, and then 2D convolutional layers are used to further optimize spatial details; by narrowing the receptive field to focus on local texture and edge information, the discriminative power of features is enhanced; a learnable wavelet feature extraction module is designed: wavelet filters are learned through backpropagation, and HSI and RGB data are adaptively processed in the spectral domain to effectively extract key features under multimodal conditions, while reducing the interference of redundant information, thereby obtaining better performance; an effective feature fusion is achieved through a residual tokenizer module: by leveraging the idea of ​​visual Transformer (ViT) feature compression, learnable attention, content mapping, and residual enhancement are fused to achieve efficient compression and information recombination of cross-modal high-dimensional fusion features.

[0244] The embodiments of this disclosure have now been described in detail. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein based on the above description.

[0245] While specific embodiments of this disclosure have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments or equivalent substitutions can be made to some technical features without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A wavelet-enhanced six-image hyperspectral classification method using dual-modal fusion, characterized in that, include: Acquire the spectral data and RGB image data of the six images above, wherein the spectral data includes hyperspectral data and near-infrared spectral data; Feature extraction is performed on the spectral data and the RGB image data respectively; The spectral data and the RGB image data after feature extraction are enhanced using a learnable wavelet enhancement module; The enhanced spectral data and the RGB image data are flattened and stitched together to obtain a stitched feature sequence. The spliced ​​feature sequence is subjected to feature fusion processing to obtain a fused feature sequence; The location information and category label of the features are embedded into the fused feature sequence, and the fused feature sequence is fed into the Transformer encoder for global context modeling and classification.

2. The wavelet-enhanced six-image hyperspectral classification method based on dual-modal fusion according to claim 1, characterized in that, The feature extraction process for the spectral data and the RGB image data includes: Dimensionality reduction processing is performed on the hyperspectral data and the RGB image data respectively; The dimensionality-reduced hyperspectral and RGB image input data are divided into blocks to obtain spectral blocks and image blocks respectively. Feature extraction is performed on the spectral block and the image block according to the modality alignment principle.

3. The wavelet-enhanced six-image hyperspectral classification method based on dual-modal fusion according to claim 2, characterized in that, The step of extracting features from the spectral block and the image block according to the modality alignment principle includes: The spectral block is feature extracted using a heterogeneous convolutional dual-modal feature extraction module; Feature extraction is performed on the image patch using a spatially aware multi-scale excitation module.

4. The wavelet-enhanced six-image hyperspectral classification method based on dual-modal fusion according to claim 3, characterized in that, The feature extraction of the spectral block using the heterogeneous convolutional dual-modal feature extraction module includes: The spectral-spatial joint features of the spectral block are mined using 3D convolutional layers; Further extraction of spatial details is achieved using 2D convolution. The feature extraction of the image patch using the spatially aware multi-scale excitation module includes: Multi-scale, multi-class convolution is used to extract rich spatial information from the image patch, specifically including: Use small convolution kernels to focus on local texture details; Large convolutional kernels and dilated convolutions are used to expand the receptive field and extract rich spatial information; The extracted multi-scale features are then concatenated. Global average pooling is used to compress the concatenated multi-scale features; Adaptive channel activation is performed through fully connected layers to enhance feature channels that are key to the classification task and suppress redundant channels. Feature fusion and dimensionality adjustment are performed using convolutional layers with a kernel size of 1×1.

5. The wavelet-enhanced six-image hyperspectral classification method based on dual-modal fusion according to claim 1, characterized in that, The enhancement process using a learnable wavelet enhancement module to enhance the spectral data and RGB image data after feature extraction includes: Initialization is performed using Haar standard wavelets, and the optimal filter parameters are learned through backpropagation. Let the input feature map be... , This represents the feature map obtained through the feature extraction step, where, Indicates batch size, This indicates the number of channels, with a value of 64. Indicates the height of the feature map, The width of the feature map is represented by four components obtained through discrete wavelet transform, expressed by the following formula: Where DWT represents Discrete Wavelet Transform, This represents the low-frequency component that contains global structural information about the features. This represents the high-frequency components that capture horizontal texture information; Represents the high-frequency vertical components that encode vertical texture information; The diagonal high-frequency components represent edge and texture abrupt change information, and the spatial dimensions of all four components are... The number of channels is ; Low frequency components Horizontal high-frequency components Vertical high frequency components Diagonal high frequency components These four components are concatenated along the channel dimension to obtain the fused feature, which is expressed by the following formula: in, Indicates fusion features, Indicates concatenation operators; Using convolutional modules to fuse features The processing includes: Fusion features The number of channels is expanded to 6C by passing through a convolutional layer with a kernel size of 1×1; Fusion features A deep separable convolutional layer with a kernel size of 7×7 is used to extract cross-channel spatially relevant features; Fusion features The GELU activation function is applied to increase nonlinearity and enhance feature representation capabilities. Fusion features The convolutional module passes through a 1×1 convolutional layer to reduce the number of channels to 4C for feature reconstruction. The output is: in, Represents convolutional modules Output, ( ) indicates the operation of the convolution module; Convolutional module Output Divide the channel into four groups evenly along the channel dimension, with each group containing [number of channels]. The four processed components are represented by the following formula: in, This indicates the corresponding low-frequency component. , , indicates the corresponding vertical high-frequency component. This indicates the corresponding high-frequency components at the diagonal. This indicates a uniform division operation; The output feature map is reconstructed using inverse discrete wavelet transform, as expressed by the following formula: Where IDWT represents the inverse discrete wavelet transform. This represents the reconstructed features, which maintain the same spatial dimensions H×W and number of channels C as the original input features.

6. The wavelet-enhanced six-image hyperspectral classification method based on dual-modal fusion according to claim 1, characterized in that, The process of flattening and stitching the enhanced spectral data and the RGB image data to obtain a stitched feature sequence includes: The enhanced hyperspectral data and RGB image data are flattened and then stitched together with near-infrared features along the feature dimension to obtain a stitched feature sequence.

7. The wavelet-enhanced six-image hyperspectral classification method based on dual-modal fusion according to claim 1, characterized in that, The step of performing feature fusion processing on the spliced ​​feature sequence to obtain a fused feature sequence includes: The concatenated feature sequence is fused using the feature fusion module of the residual tokenizer to obtain a fused feature sequence, specifically including: The concatenated feature sequence is used as the input feature matrix X. in, B represents the stitched feature tensor, H and W represent the size of the image patch, and C represents the number of feature channels. Using attention weights Controlling the attention distribution of tokens and utilizing content weights Controlling the content expression of the token to achieve adaptive token generation. Where L represents the preset number of tokens, with a value of 4, and C represents the preset content features, with a value of 64; The feature fusion module combines the input feature matrix X with attention weights. Perform a transposed dot product to construct the original attention score, and introduce a learnable parameter T to adjust the sharpness of the attention distribution. Normalize the result using the Softmax function to obtain the attention distribution A, expressed by the following formula: in, Each element in the attention distribution A represents the attention weight of the Lth Token to the 2HWth spatial location of the input feature; The feature fusion module combines the input feature matrix X with content weights. Perform a linear transformation to generate basic content features V, while preserving the channel information of the input features; The token content representation T is obtained by weighted aggregation of the basic V using the attention distribution A. By introducing residual connections, the mean of the basic content feature V across the spatial dimension (dim=1) is calculated to obtain the global content feature. Then, it is added to the aggregated T, as expressed by the following formula: in, , That is, each sample is compressed into L tokens with dimension C.

8. The wavelet-enhanced six-image hyperspectral classification method based on dual-modal fusion according to claim 1, characterized in that, The step of embedding the location information and category label of the features into the fused feature sequence, and feeding the fused feature sequence into the Transformer encoder for global context modeling and classification, includes: The Transformer module is used to globally model the fused features after Tokenizer compression, and a self-attention mechanism is employed to capture long-distance dependencies and complex spatial relationships between features. Specifically, this includes: During the input phase, the model adds a learnable category label to the compressed token sequence and combines it with positional encoding to introduce sequence information; The Transformer encoder first interacts with the token sequence through multi-head self-attention, so that each token can pay attention to the features of all other positions in the sequence; The outputs of each attention point are concatenated and fused to enhance the expressive power of the features; The fused sequence of each attention head is subjected to nonlinear transformation and feature mapping through a feedforward fully connected network; The entire module uses residual connections and layer normalization to stabilize the training process and improve feature transfer efficiency; The category label serves as an aggregated representation of the global features, and the classification result is output through a fully connected layer.

9. The wavelet-enhanced six-image hyperspectral classification method based on dual-modal fusion according to claim 1, characterized in that, The dual-modal fusion wavelet-enhanced six-image hyperspectral classification method also includes: A loss function is constructed by employing a joint optimization strategy of cross-entropy loss and wavelet loss, specifically including: The cross-entropy loss is calculated using the following formula: in, The cross-entropy loss value is represented by c, where c represents the category index and C represents the total number of categories in the tobacco classification task. This represents the true label value of the sample in class c, and is a variable between 0 and 1. This represents the predicted probability value of the model for a sample in class c; The wavelet loss consists of two parts: perfect reconstruction loss and aliasing cancellation loss. The calculation process of the perfect reconstruction loss includes: The perfect reconstruction constraint formula can be constructed using the following formula: in, Represents frequency domain variables, This represents the frequency domain expression of a decomposed low-pass filter. This represents the frequency domain expression of a decomposed high-pass filter. This represents the frequency domain expression of the reconstructed low-pass filter. This represents the frequency domain expression of the reconstructed high-pass filter; The polynomial multiplication of the filter is simulated by a one-dimensional convolution to get the actual output P test and compute P test The mean squared error with the ideal output as the perfect reconstruction loss is expressed by the following equation: in, This represents the loss value for perfect reconstruction. Represents the unit impulse function; The calculation process for aliasing elimination loss includes: constructing the aliasing elimination condition constraint formula using the following formula: Here, −z represents the inverse frequency variable, which is simulated by alternating the signs of the filter coefficients. The −z substitution is achieved by using the filter alternating signal [1,-1,1,-1,...] to construct the mask; Decompose the frequency domain expression of the low-pass filter. Decompose the frequency domain expression of the high-pass filter. Multiply by the mask and then with the corresponding reconstruction filter , Perform a one-dimensional convolution, and sum the convolution results to obtain the actual output P. test Calculate P test The aliasing elimination loss is obtained by taking the mean square error between the ideal output all-zero tensor and the mean square error, and is expressed by the following formula: in, This represents the aliasing elimination loss value. This represents the ideal output value for aliasing elimination; The total wavelet transform loss is calculated using the following formula: in, This represents the total loss of the wavelet transform; Calculate the total loss using the following formula; in, This represents the total loss of the model.

10. A wavelet-enhanced six-chip hyperspectral classification system employing the method described in any one of claims 1-9, characterized in that, include: The data acquisition module is used to acquire spectral data and RGB image data of the six images above, wherein the spectral data includes hyperspectral data and near-infrared spectral data; The feature extraction module is used to extract features from the spectral data and the RGB image data respectively; The wavelet enhancement module is used to enhance the spectral data and the RGB image data after feature extraction using a learnable wavelet enhancement module; The feature stitching module is used to flatten and stitch the enhanced spectral data and the RGB image data to obtain a stitched feature sequence. The feature fusion module is used to perform feature fusion processing on the spliced ​​feature sequence to obtain a fused feature sequence. The encoding and classification module is used to embed the location information and category label of the features into the fused feature sequence, and then feed the fused feature sequence into the Transformer encoder for global context modeling and classification.

Images