Paper mildew spot detection method based on multi-modal feature fusion

Abstract

The invention relates to the technical field of mildew spot detection, and particularly discloses a paper mildew spot detection method based on multi-modal feature fusion, which innovatively integrates space-spectrum two-dimensional features: firstly, constructing an improved two-dimensional convolutional network, extracting multi-scale spatial features of RGB (Red, Green, Blue) wavebands through a spatial attention mechanism, and constructing an improved two-dimensional convolutional network; strengthening collaborative characterization of local details and global morphology of the mildew spot area; secondly, a SpectralFormer module is introduced, hyperspectral data characteristics are analyzed by using a spectral attention mechanism, and mildew specific response in a spectral interval of 400-1000 nm is accurately captured; and finally, designing a hybrid convolutional network to carry out cross-modal feature fusion, and realizing efficient integration and classification decision of multi-source information. The invention provides a high-precision and practical solution for mildew detection of paper cultural relics, and has important practical value for establishing a preventive protection system.

Description

Technical Field

[0001] The present invention relates to the technical field of mildew spot detection, and particularly to a paper mildew spot detection method based on multi-modal feature fusion. Background Art

[0002] Molds utilize the organic nature and high hygroscopicity of the fibers of paper cultural relics, not only damaging the structure but even causing irreversible chemical changes, seriously affecting the historical and artistic value of the cultural relics. Currently, the methods for killing and eliminating molds on cultural relics are mainly divided into three types: mechanical removal, physical methods, and biochemical methods. Mechanical methods include using tools such as brushes, scalpels, and vacuum cleaners to remove pathogenic microorganisms, but they cannot completely eliminate all microorganisms. Physical methods include using ultraviolet rays, laser cleaning technology, etc. to sterilize and kill insects on paper cultural relics, but they have little effect on most bacteria and black fungi. The more effective method is the biochemical method, but due to the diverse types of molds, a single strategy is difficult to achieve comprehensive killing and elimination. Precise repair requires identifying the types of molds to avoid secondary damage.

[0003] Therefore, the accurate identification of the types of molds is not only a prerequisite for the killing and elimination work but also a key step in protecting paper cultural relics from long-term mold damage. Currently, the traditional paper cultural relic mold detection techniques include morphological identification, molecular biology techniques, and biochemical detection methods. However, the above methods are only limited to using one of image or spectral techniques and cannot comprehensively identify molds, having certain limitations.

[0004] As an advanced detection method that integrates image and spectral information, hyperspectral imaging technology has shown broad application prospects in the field of mold detection due to its high sensitivity, multi-dimensional information acquisition ability, and non-destructive detection characteristics. Currently, this technology is mainly applied in fields such as food quality assessment and crop disease detection, and preliminary progress has also been made in the identification and assessment of mold contamination in the field of cultural relics protection. For example, Mengzhen Lu et al. proposed an automated labeling method based on hyperspectral images for the detection of mural mold diseases

Research on Labeling Method for Fungal Diseases in Murals Based on Hyperspectral Images[C] / / 2024 3rd International Conference on Image Processing and Media Computing(ICIPMC).IEEE,2024:93-100

Deep convolutional neural networks for hyperspectral image classification[J].Journal of Sensors,2015,2015(1):258619

[0005] The above research provides important technical support for the accurate detection and scientific treatment of mildew stains, further expanding the application potential of hyperspectral imaging technology in the field of cultural relics protection. However, neither of them has achieved the efficient detection of mildew stains on paper cultural relics. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a method for detecting paper mildew stains based on multi-modal feature fusion, which combines digital image information and hyperspectral image information, and uses multi-modal feature fusion technology and machine learning thinking to achieve the efficient detection of mildew stains on paper cultural relics.

[0007] To achieve the above purpose, the present invention provides a method for detecting paper mildew stains based on multi-modal feature fusion, including the following steps:

[0008] S1. Obtain the hyperspectral image of the paper mildew spot to be detected;

[0009] S2. Input the hyperspectral image obtained in S1 into the multi-modal feature fusion network to obtain the fusion features of the hyperspectral image;

[0010] S3. Output the detection result according to the fusion features obtained in S2;

[0011] Among them, step S2 includes:

[0012] S21. Extract the RGB three bands of the input hyperspectral image to generate an RGB digital image;

[0013] S22. For the RGB digital modality, capture the edge morphology and spatial characteristics of color changes of the pseudo-color image through two-dimensional convolution, then introduce a spatial attention module to enhance the spatial texture representation of the mildew spot area, filter out the interference of non-mildew spot areas, and extract spatial features;

[0014] S23. For the hyperspectral data modality, two paths are adopted for feature extraction: in terms of spectral feature extraction, first use a spectral transformer to capture the subtle spectral differences of the mildew spot in the hyperspectral band, and then introduce spectral attention to highlight the key band features to generate spectral characteristics; in terms of spatial-spectral feature extraction, use a hybrid convolution network to jointly analyze the spatial distribution and spectral changes of the mildew spot in the hyperspectral data;

[0015] S24. Deeply fuse the spatial features output by S22, the spectral features and spatial-spectral features output by S23 to obtain fusion features.

[0016] As a further improvement to the above technical solution, in step S22, the two-dimensional convolution realizes local feature extraction through convolution operations. The convolution kernel extracts edge, texture and structural features in the spatial local range in the form of parameter sharing, and gradually learns more advanced semantic information layer by layer.

[0017] As a further improvement to the above technical solution, in step S22, first extract the RGB three bands from X and flatten them, then pass through a three-layer two-dimensional convolution network to gradually extract the spatial features of the hyperspectral image and optimize the spatial features extracted by each layer of convolution to enhance the expression of key spatial positions. Finally, flatten the output and use a three-layer fully connected network to gradually reduce the dimension, and finally output the predicted category.

[0018] As a further improvement to the above technical solution, in step S22, the spatial attention module enhances the feature expression of key spatial positions by calculating the weights in space. The formula for calculating the attention energy is:

[0019] E = Q·K;

[0020] Wherein, Q is the query matrix and K is the key matrix;

[0021] Perform a Softmax operation on each row of E to obtain the attention weight A. The formula for calculating the attention weight is:

[0022]

[0023] Wherein, A ij represents the weight distribution of the pixel spatial position at the i-th row and j-th column.

[0024] As a further improvement to the above technical solution, in step S23, first introduce a spectral attention mechanism, perform global average pooling to obtain the global representation of the spectral channels, and weight the features of the spectral bands to highlight the information of important bands. Then, perform spectral embedding through linear projection, add classification tokens and positional embeddings, and subsequently use the Transformer module for sequence modeling, and finally output the classification.

[0025] As a further improvement to the above technical solution, in step S23, the input is the extracted spectral information X ∈ R B×C×N , where N is the number of features. The expression of the spectral attention process is:

[0026] X′ = X · softmax(W2 · σ(W1 · z T + b1)+ b2);

[0027] Wherein, d is the hidden dimension, W1 ∈ R d×C , W2 ∈ R C×d , σ is the sigmoid activation function, and b1 and b2 are bias parameters.

[0028] As a further improvement to the above technical solution, in step S23, the hybrid convolutional network uses multi-dimensional feature extraction, that is, 3D convolution to extract the spectral and spatial features of the hyperspectral data, and combines 2D convolution to further extract the spatial features; then layer by layer, the features are fused, that is, the high-dimensional spectral features are integrated into low-dimensional features, and 2D convolution operations are introduced to aggregate and optimize the spatial feature expression.

[0029] As a further improvement to the above technical solution, in step S24, a fusion method such as additive fusion, multiplicative fusion or splicing fusion is used for feature fusion.

[0030] As a further improvement to the above technical solution, the molds that form mildew spots include one or more of Paecilomyces lilacinus, Aspergillus niger, Alternaria alternata, Penicillium citrinum, Trichoderma longibrachiatum, and Cladosporium cladosporioides.

[0031] Compared with the prior art, the present invention has the following beneficial technical effects:

[0032] A method for detecting paper mildew spots based on multi-modal feature fusion provided by the present invention creatively proposes a multi-modal feature fusion network, which improves the accuracy and robustness of mildew spot detection through the efficient fusion of multi-modal features. The multi-modal feature fusion network introduces a multi-path structure and combines convolutional operations with different receptive fields, which can effectively capture multi-scale spatial and spectral features in hyperspectral images and enhance the expression ability of fine-grained and global features. In addition, the fusion network includes two complementary paths, which respectively focus on the spectral feature extraction and spatial feature extraction of hyperspectral data, enabling the model to simultaneously pay attention to the local spectral changes and spatial distribution patterns of hyperspectral images. After multi-path feature extraction, the fusion network realizes feature alignment and integration through a fusion strategy, thereby more comprehensively expressing the complex characteristics of hyperspectral data and improving the classification performance.

[0033] Advantages of additional aspects of the present invention will be partly given in the following description, partly will become obvious from the following description, or will be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The specification drawings forming a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute a limitation to the present invention.

[0035] Figure 1 is a flowchart of a method for detecting paper mildew spots based on multi-modal feature fusion in an embodiment of the present invention;

[0036] Figure 2 is a false color map of a simulated mildew-infected sample in an embodiment of the present invention;

[0037] Figure 3 is a true value map of a simulated mildew-infected sample in an embodiment of the invention;

[0038] Figure 4 is a network structure diagram of a multi-modal feature fusion network in an embodiment of the present invention;

[0039] Figure 5 is a network structure diagram of a spatial feature extraction model in an embodiment of the present invention;

[0040] Figure 6 is a network structure diagram of a spectral feature extraction model in an embodiment of the present invention;

[0041] Figure 7 is a network structure diagram of a spatial-spectral feature extraction model in an embodiment of the present invention;

[0042] Figure 8 is a comparison chart of OA of training samples with different ratios in an embodiment of the present invention;

[0043] Figure 9 This is the classification result graph of the dataset by different methods in the embodiments of the present invention. Detailed implementation manners

[0044] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] Embodiment

[0046] This embodiment provides a paper mildew spot detection method based on multi-modal feature fusion, including the following steps:

[0047] S1. Obtain the hyperspectral image of the paper mildew spot to be detected;

[0048] S2. Input the hyperspectral image obtained in S1 into the multi-modal feature fusion network to obtain the fusion features of the hyperspectral image;

[0049] S3. Output the detection result according to the fusion features obtained in S2.

[0050] Among them, step S2 includes:

[0051] S21. Extract the RGB three bands of the input hyperspectral image to generate an RGB digital image;

[0052] S22. For the RGB digital modality, capture the edge morphology and spatial characteristics of color changes of the pseudo-color image through two-dimensional convolution, then introduce a spatial attention module to enhance the spatial texture representation of the mildew spot area, filter the interference of non-mildew spot areas, and extract spatial features;

[0053] S23. For the hyperspectral data modality, two paths are adopted for feature extraction: in terms of spectral feature extraction, first use a spectral transformer to capture the subtle spectral differences of the mildew spot in the hyperspectral band, and then introduce spectral attention to highlight the key band features to generate spectral characteristics; in terms of spatial-spectral feature extraction, use a hybrid convolution network to jointly analyze the spatial distribution and spectral changes of the mildew spot in the hyperspectral data;

[0054] S24. Deeply fuse the spatial features output by S22 and the spectral features and spatial-spectral features output by S23 to obtain fusion features.

[0055] The detection method of this embodiment can capture both the spatial morphology (such as edge texture) and spectral characteristics (such as reflectance differences in specific bands) of mildew spots through the dual-modal analysis of hyperspectral data and RGB images. In a certain band range of hyperspectral, the spectral differences between mildew spots and healthy areas are significant, while the RGB image can intuitively reflect color changes, and the two are highly complementary. The spatial attention module helps improve edge sensitivity by enhancing the spatial texture of the mildew spot area and suppressing background interference. The spectral attention module dynamically assigns weights to key bands, which is beneficial to solving the problem of hyperspectral data redundancy. The spectral transformer is used to capture global spectral differences, and the hybrid convolutional network jointly analyzes spatial-spectral features, increasing the long-range dependence modeling ability of the transformer and making it more adaptable to complex mildew spot distributions.

[0056] In addition, in step S3, as one of the methods, an improved Feature Pyramid Network (FPN) can be used to perform multi-scale reconstruction on the fused features, generate P3 - P5 multi-scale feature layers through lateral connections, calculate the spectral deviation based on key bands (such as the mildew-sensitive band of 413 - 855nm), and normalize it to an auxiliary confidence index through the Sigmoid function. Finally, a spatial attention heatmap (indicating the spatial distribution of the mildew spot area) and a spectral response curve (showing key band differences) can be output. Referring to the existing cross-modal visualization method, the interpretability can be enhanced by superimposing hyperspectral features with RGB pseudo-color.

[0057] Specifically, the experimental content corresponding to the detection method shown in this embodiment is as follows.

[0058] 1. Experimental part

[0059] 1.1 Simulated mildew-infected samples

[0060] For the paper samples in the simulated mold-infected samples, cotton and linen raw rice paper was selected, and the cut size was 5*5 cm. To ensure the diversity and accuracy of the experimental data, in this embodiment, according to the relevant literature on the common mold species in the preservation environment of paper cultural relics in the past five years, six representative mold samples from different genera were determined, namely Aspergillus niger in the genus Aspergillus, Penicillium citrus in the genus Penicillium, Trichoderma longiformis in the genus Trichoderma, Alternaria alternata in the genus Alternaria, Paecilomyces lilacinus in the genus Paecilomyces, and Cladosporium cladosporioides in the genus Cladosporium as the experimental objects. These six molds belong to different genera, reflecting the diversity and ecological adaptability of molds. They are widely distributed in the natural environment and may have an impact on the preservation of cultural relics. Therefore, these different genera of molds need to be concerned in the detection and protection of cultural relics molds. The paper cultural relics and molds required for this experiment were all provided by the Key Scientific Research Base of the State Administration of Cultural Heritage for the Control of Pests in Cultural Relics Collections.

[0061] The physical diagram of the simulated mold-infected sample is as Figure 2 , Figure 3 shown; among them Figure 2 is a false color image (in the figure, a is Paecilomyces lilacinus, b is Aspergillus niger, c is Alternaria alternata, d is Penicillium citrus, e is Trichoderma longiformis, f is Cladosporium cladosporioides), Figure 3 is the true value image.

[0062] 1.2 Hyperspectral Paper Cultural Relics Mildew Spot Collection System

[0063] The hyperspectral paper cultural relics mildew spot image collection system built in this experiment consists of the iSpecHyper-VS1000 portable hyperspectral imager of Laissen Optics Company, 2 halogen light sources that can adjust brightness and color temperature, and a Canon RF24mm focal length lens, etc. After multiple shooting experiments, the final confirmed optimal parameters are a focal length of 4615 mm, a frame rate of 16 Hz·s -1 , and an integration time of 56338 μs. The images collected under the above parameters have excellent quality, clear edges and no obvious distortion.

[0064] 1.3 Experimental Settings

[0065] 1) Evaluation Index

[0066] To verify the effectiveness of the model, three commonly used classification performance evaluation metrics, namely Overall Accuracy (OA), Average Accuracy (AA), and Kappa coefficient (Kappa), are adopted in this experiment to quantitatively analyze the classification results of each model. Among them, OA reflects the overall classification accuracy of the model across all classes, AA represents the mean of the classification accuracies of each class, and the Kappa coefficient measures the consistency between the classification results and the random classification results, thus providing a more comprehensive evaluation of the model performance. Meanwhile, to further visually compare the classification capabilities of each model, this experiment also conducts qualitative analysis through visualization of classification maps to observe their spatial distribution characteristics and the accuracy of the classification boundaries, comprehensively evaluating the classification performance of the model from both quantitative and qualitative dimensions.

[0067] 2) Comparison of different models

[0068] In this paper, several currently advanced models are selected for comparative analysis with the proposed model. These models include Support Vector Machine (SVM), 1-D-CNN [for reference, see Hu W, Huang Y, Wei L, et al. Deep convolutional neural networks for hyperspectral image classification[J]. Journal of Sensors, 2015, 2015(1):258619], 2-D-CNN [for reference, see Zhao W, Du S. Spectral–spatial feature extraction for hyperspectral image classification: A dimension reduction and deep learning approach[J]. IEEE Transactions on Geoscience and Remote Sensing, 2016, 54(8):4544-4554], SSFTT [for reference, see Sun L, Zhao G, Zheng Y, et al. Spectral–spatial feature tokenization transformer for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:1-14], SpectralFormer [for reference, see Hong D, Han Z, Yao J, et al. SpectralFormer: Rethinking hyperspectral image classification with transformers[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60:1-15] and HybirdSN [for reference, see Roy S K, Krishna G, Dubey S R, et al. HybridSN: Exploring 3-D–2-D CNN feature hierarchy for hyperspectral image classification[J]. IEEE Geoscience and Remote Sensing Letters, 2019, 17(2):277-281].

[0069] And to ensure the unity and effectiveness of the experimental data, the experimental data are all processed by a unified data preprocessing method, that is, SG smoothing filtering is used to remove noise, and the PCA principal component analysis method is used to reduce the number of bands to 30 dimensions, so as to remove redundant information and highly similar features and increase the recognition accuracy.

[0070] 3) Experimental environment configuration

[0071] To verify the mildew hyperspectral image classification algorithm based on multi-modal feature fusion proposed in this embodiment, all experiments are carried out using the Python 3.12 programming language and implemented in the PyCharm 2024.3.1 integrated development environment. The experimental running environment is a high-performance personal computer configured with an Intel(R) Core i5-14600KF processor, an NVIDIA GeForce RTX 4070 graphics processor, and 32GB of memory.

[0072] During the experiment, all deep learning models use the cross-entropy loss function as the optimization objective, combine the Adam optimizer to update the parameters, and use the stochastic gradient descent method (SGD) to optimize the training of the model. The learning rate is uniformly set to 0.001, and the number of training epochs is fixed at 50 to ensure the stability and convergence of the model, so as to reliably evaluate the performance of each model.

[0073] 2. Related work

[0074] In this section, first, the multi-modal feature fusion network model (TPMFN model) proposed in this embodiment is introduced. Secondly, the feature extraction design logic under three different paths in the model and its role in multi-modal feature extraction and fusion are described in detail. Finally, the advantages and potential of the TPMFN model are summarized. Here, the hyperspectral image features input after data block division are X ∈ R B×H×W×C , where B represents the batch size, H and W represent the spatial dimensions, and C represents the number of channels.

[0075] 2.1 TPMFN model

[0076] The model aims to improve the accuracy and robustness of mildew detection through the efficient fusion of multi-modal features. The core design concept of the TPMFN model is to make full use of the characteristics of three different modal data, extract features from the spectral dimension of hyperspectral data, the spatial-spectral joint dimension, and the spatial dimension of RGB images respectively, and realize the joint expression of global features through a deep fusion mechanism. The specific network structure diagram is as Figure 4 shown.

[0077] In the TPMFN model, first, the input hyperspectral image is extracted into three RGB bands to generate an RGB digital image. Secondly, different methods are adopted for different modal features. For the RGB digital modality, two-dimensional convolution is used to capture spatial features such as the edge morphology and color changes of the pseudo-color image, and then a spatial attention module is introduced to enhance the spatial texture representation of the mildew spots and filter out the interference of non-mildew regions. For the hyperspectral data modality, two paths are used for feature extraction: in terms of spectral feature extraction, first, a Spectral Transformer is used to capture the subtle spectral differences of mildew spots in the hyperspectral bands, and then spectral attention is introduced to highlight the key band features and generate spectral characteristics. In terms of spatial-spectral feature extraction, a Hybrid Convolutional Network is used to jointly analyze the spatial distribution and spectral changes of mildew spots in the hyperspectral data.

[0078] 1) Spatial features

[0079] The 2D convolutional neural network is a deep learning model specialized in processing two-dimensional data. Its core feature is to extract local features through convolutional operations. The convolutional kernel extracts low-level features such as edges, textures, and structures in a spatially local range in the form of parameter sharing, and gradually learns higher-level semantic information layer by layer. In this embodiment, first, the three RGB bands are extracted from X and flattened, and then through a three-layer two-dimensional convolutional network, the spatial features of the hyperspectral image are gradually extracted and the spatial features extracted by each layer of convolution are optimized to enhance the expression of key spatial positions. Finally, by flattening the output, a three-layer fully connected network is used to gradually reduce the dimension, and the predicted category is finally output.

[0080] Among them, the adopted spatial attention mechanism enhances the feature expression of key spatial positions by calculating spatial weights. The formula for calculating the attention energy is as shown in Equation (1):

[0081] E = Q·K (1)

[0082] In Equation (1), Q is the Query matrix and K is the Key matrix.

[0083] Then, a Softmax operation is performed on each row of E to obtain the attention weight A. The formula for calculating the attention weight is as shown in Equation (2):

[0084]

[0085] In Equation (2), A ij represents the weight distribution of the pixel spatial position at the i-th row and j-th column.

[0086] The specific network structure diagram of the spatial feature extraction algorithm is as follows Figure 5 shown.

[0087] 2) Spectral features

[0088] SpectralFormer is improved based on Transformer and can perform local spectral representation from multiple adjacent bands at each encoding position instead of a single band. Due to the existence of group intelligent spectral embedding (GSE) and cross-layer adaptive fusion (CAF), it can respectively improve the detail capture ability of subtle spectral differences and enhance the inter-layer information transferability. In addition, a cross-layer skip connection is designed in SpectralFormer to gradually transfer memory-like components from the shallow layer to the deep layer by adaptively learning to fuse their "soft" residuals. Although SpectralFormer is good at capturing global sequence information, it performs poorly in modeling the local context information of hyperspectral data.

[0089] To address the above deficiencies, the spectral feature extraction model adopted in this embodiment is an improved structure based on SpectralFormer, as shown in Figure 6 shown. First, a spectral attention mechanism is introduced to perform global average pooling to obtain the global representation of the spectral channels, and the features of the spectral bands are weighted to highlight the information of important bands. Then, spectral embedding is performed through linear projection, classification tokens and position embeddings are added, and then the Transformer module is used for sequence modeling, and finally the classification is output. The input is the spectral information X∈R B×C×N , where N is the number of features. The expression of the spectral attention process is as shown in Equation (3):

[0090] X′ = X · softmax(W2 · σ(W1 · z T + b1)+ b2) (3)

[0091] In Equation (3), d is the hidden dimension, W1∈R d×C , W2∈R C×d , σ is the sigmoid activation function, and b1 and b2 are bias parameters.

[0092] 3) Spatial-spectral features

[0093] The hybrid convolutional network adopts multi-dimensional feature extraction, i.e., 3D convolution to extract the spectral and spatial features of hyperspectral data, and combines 2D convolution to further extract spatial features. Subsequently, the features are fused layer by layer, that is, the high-dimensional spectral features are integrated into low-dimensional features, and 2D convolution operations are introduced to aggregate and optimize the spatial feature representation. Its significant advantage lies in being able to effectively integrate the multi-dimensional feature information in hyperspectral data, thereby improving the accuracy and robustness of the classification task. At the same time, the utilization of computing resources is optimized through the hierarchical extraction strategy to cope with the challenges of the complexity and redundancy of hyperspectral data. The specific structure of the hybrid convolutional network model adopted in this embodiment is as Figure 7 shown.

[0094] 2.2 Advantages of the TPMFN Model

[0095] Aiming at the limitations of the existing technology, the TPMFN model proposed in this embodiment can have the following advantages:

[0096] First, by introducing a multi-path structure and combining convolution operations with different receptive fields, it can effectively capture multi-scale spatial and spectral features in hyperspectral images, enhancing the expression ability for fine-grained and global features;

[0097] Second, the model contains two complementary paths, which respectively focus on the spectral feature extraction and spatial feature extraction of hyperspectral data, enabling the model to simultaneously focus on the local spectral changes and spatial distribution patterns of hyperspectral images;

[0098] Third, after multi-path feature extraction, TPMFN achieves feature alignment and integration through a fusion strategy, thereby more comprehensively expressing the complex characteristics of hyperspectral data and improving the classification performance.

[0099] 3. Results and Discussion

[0100] In this section, first, the hyperspectral image dataset used in this experiment is described. Subsequently, ablation experiments are conducted to analyze the influence of three different modal extraction modules on the performance of this model. Finally, a comparative experiment is carried out between the model proposed in this embodiment and other mainstream models, and the mildew classification performance of different models is evaluated in both quantitative and visual ways.

[0101] 3.1 Dataset Introduction

[0102] The hyperspectral image dataset used in this embodiment covers the mildew spots generated after the above 6 kinds of mold infections. The sampling height is 40 cm, the size of the image is 521×364 pixels, there are a total of 19,951 effective pixels, and a total of 300 spectral bands. The dataset is divided into a 10% training set and a 90% test set. The class names and sample numbers used for training and testing in the classification task are shown in Table 1 (the mildew coverage categories in the dataset and the training and test samples of each category).

[0103] Table 1

[0104]

[0105] 3.2 Ablation Experiments

[0106] To fully verify the effectiveness of the method, ablation experiments with different combinations of components were conducted on the dataset mentioned in this embodiment. A total of 5 combinations were considered, and the impact of different components on the accuracy of the entire model was analyzed from the perspective of classification accuracy. All experimental results are listed in Table 2.

[0107] Specifically, the model was divided into 4 parts - 2-D-CNN+Spectral_Att, SpectralFormer, Spectral_Att, and Hybrid Network. The classification accuracy of the model output without the group of SpectralFormer and Spectral_Att (the first case) was the worst, with an accuracy rate of 90.31%. In the case without the 2-D-CNN+Spectral_Att layer (the fourth case), the classification accuracy of the model was slightly better than the previous group, with an accuracy rate of 91.01%. In the case without the HybridNetwork layer (the fifth case), relying solely on the separate processing of spatial and spectral features, the classification accuracy of the model was 93.45%. By comparing the performance of the two models with and without the introduction of Spectral_Att in SpectralFormer (corresponding to the third case and the sixth case), it can be observed that the accuracy rate was significantly improved to 97.12%.

[0108] This result indicates that Spectral_Att plays a positive role in spectral feature processing, thus effectively enhancing the classification accuracy of the model. The analysis of the proposed model on this dataset (non-optimal results) is shown in Table 2 below.

[0109] Table 2

[0110]

[0111] To better verify the effectiveness of this algorithm, this embodiment also conducted experiments on this dataset using different feature fusion methods. A total of 3 different fusion methods were used, namely additive fusion, multiplicative fusion, and splicing fusion. All experimental results are listed in Table 3.

[0112] Table 3

[0113]

[0114] It can be observed that the model after multiplicative fusion often has better classification performance than those after additive fusion and concatenation fusion, with an accuracy rate as high as 98.84%. This indicates that the multiplicative fusion method has better capabilities in feature interaction and information coupling, and can more fully explore the complementary relationships between multi-modal features, thus improving the classification performance.

[0115] 3.3 Quantitative Analysis

[0116] Table 4 shows the OA, AA, Kappa, and classification accuracy for each class of this dataset using all the classification methods in Section 2 above.

[0117] The evaluation data clearly shows that the mentioned TPMFN method has the best performance, obtaining the highest OA, AA, κ values, and some classification accuracy values. For example, in the class Trichoderma longibrachiatum, the classification effects of SVM, 1-D-CNN, 2-D-CNN, Spectralformer, and SSFTT are not very ideal, probably because the sample size of this class is small and the distribution is not concentrated, resulting in the failure to learn the correct features. In addition, random sampling is based on percentage sampling, which may lead to sample imbalance.

[0118] However, the classification accuracy of each class obtained by TPMFN is relatively consistent, all remaining above 96%. This indicates that the TPMFN model has strong class balance ability when dealing with hyperspectral image classification tasks. Whether it is the dominant class or the minor class, the model can achieve high-precision classification performance. This consistency may benefit from the model's multi-modal feature fusion mechanism and efficient feature extraction ability.

[0119] However, there is another situation. In the class Paecilomyces lilacinus, the classification accuracy of SSFTT is significantly higher than that of TPMFN. The main reason is that the sample distribution of this class has a high degree of convergence, showing a close feature close to a circle, while the sample distributions of other classes are more divergent. Therefore, the method in this embodiment does not have enough advantages in the classification of this class, but in data where the sample points are discrete and local, TPMFN has obvious advantages and can better capture local detailed information.

[0120] This embodiment also trains different models with different proportions of training samples, as Figure 8 shown. In the case of a small number of samples, TPMFN still has good performance. As the number of samples increases, the performance of SSFTT and HybirdSN is only slightly lower than this method.

[0121] Table 4

[0122]

[0123]

[0124] 3.4 Visual Evaluation

[0125] The classification result graphs of the above several comparison methods for this dataset are as shown in Figure 9 the figure. In the figure, (a) is the ground truth graph, (b) represents SVM (OA = 61.18%), (c) represents 1-D-CNN (OA = 70.90%), (d) represents 2-D-CNN (OA = 85.09%), (e) represents Spectralformer (OA = 91.71%), (f) represents SSFTT (OA = 94.39%), (g) represents HybirdSN (OA = 95.38%), and (h) represents TPMFN (OA = 98.52%).

[0126] It can be observed that the classification graph of TPMFN is the cleanest and closest to the ground truth graph. In contrast, methods such as SVM, 1-D-CNN, and 2-D-CNN can only capture limited spectral features or shallow features, resulting in more noise in the classification results and containing too much error information. This also indirectly indicates that these models cannot accurately identify object types and have poor performance. Although the classification accuracies of methods such as Spectralformer, SSFTT, and HybirdSN all reach over 90%, there are still a small number of misclassifications in some complex scenarios, such as in the areas of Paecilomyces lilacinus and Trichoderma longibrachiatum. And the method proposed in this embodiment correctly identifies these two mildew areas to a large extent, demonstrating the excellent performance of TPMFN.

[0127] 4. Conclusion

[0128] In this embodiment, aiming at the problems of accuracy and efficiency in the detection of mildew on paper cultural relics, a multi-modal feature fusion method TPMFN based on hyperspectral imaging technology is proposed. This method comprehensively utilizes spectral features, spatial features, and spatial-spectral joint features to achieve deep feature extraction and fusion with a multi-path structure.

[0129] Specifically, TPMFN analyzes spectral details through Spectral Transformer, combines 2-D-CNN with spatial attention mechanism to capture the spatial patterns of mildew, and simultaneously uses a hybrid convolutional network to integrate multi-dimensional features to optimize the feature expression ability and improve the calculation efficiency. To utilize this feature, this embodiment further studies several different fusion modules: additive, multiplicative, and concatenation. The experimental results show that the method after multiplicative fusion is significantly superior to traditional classification methods and single-path networks in terms of accurately capturing local feature changes of mildew, enhancing the expression of key information regions, and classification performance. The innovative design of TPMFN provides new ideas for hyperspectral image processing tasks and also provides an efficient and intelligent solution for the field of mildew detection on paper cultural relics.

[0130] Finally, it should be noted that specific examples are used in this article to illustrate the principles and implementation modes of the present invention. The description of the above embodiments is only used to help understand the core idea of the present invention. Without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A paper mildew spot detection method based on multi-modal feature fusion, characterized in that It includes the following steps: S1. Obtain the hyperspectral image of the paper mildew spot to be detected; S2. Input the hyperspectral image obtained in S1 into the multimodal feature fusion network to obtain the fusion features of the hyperspectral image; S3. Output the detection result according to the fusion features obtained in S2; Among them, step S2 includes: S21. Extract the RGB three bands of the input hyperspectral image to generate an RGB digital image; S22. For the RGB digital modality, capture the edge morphology and spatial characteristics of color changes of the pseudo-color image through two-dimensional convolution, then introduce a spatial attention module to enhance the spatial texture representation of the mildew spot area, filter the interference of non-mildew spot areas, and extract spatial features; S23. For the hyperspectral data modality, two paths are used for feature extraction: in terms of spectral feature extraction, first use a spectral transformer to capture the subtle spectral differences of the mildew spot in the hyperspectral band, and then introduce spectral attention to highlight the key band features to generate spectral characteristics; in terms of spatial-spectral feature extraction, use a hybrid convolution network to jointly analyze the spatial distribution and spectral changes of the mildew spot in the hyperspectral data; S24. Deeply fuse the spatial features output by S22, the spectral features and spatial-spectral features output by S23 to obtain fusion features.

2. A method for detecting paper mildew spots based on multimodal feature fusion according to claim 1, characterized in that: In step S22, the two-dimensional convolution realizes local feature extraction through convolution operations. The convolution kernel extracts edge, texture and structural features in the local spatial range in the form of parameter sharing, and gradually learns more advanced semantic information layer by layer.

3. A method for detecting paper mildew spots based on multimodal feature fusion according to claim 2, characterized in that: In step S22, first extract the RGB three bands from X and flatten them, then pass through a three-layer two-dimensional convolution network to gradually extract the spatial features of the hyperspectral image and optimize the spatial features extracted by each layer of convolution to enhance the expression of key spatial positions. Finally, flatten the output and use a three-layer fully connected network to gradually reduce the dimension, and finally output the predicted category.

4. A method for detecting paper mildew spots based on multimodal feature fusion according to claim 2, characterized in that: In step S22, the spatial attention module enhances the feature expression of key spatial positions by calculating the weights in space. The formula for calculating the attention energy is: E = Q·K; In the formula, Q is the query matrix and K is the key matrix; Perform a Softmax operation on each row of E to obtain the attention weight A. The formula for calculating the attention weight is: Where, A ij represents the weight distribution of the pixel spatial position in the i-th row and j-th column.

5. A method for detecting paper mildew spots based on multimodal feature fusion according to claim 1, characterized in that: In step S23, first introduce a spectral attention mechanism, perform global average pooling to obtain the global representation of the spectral channels, and weight the features of the spectral bands to highlight the information of important bands. Then perform spectral embedding through linear projection, add classification markers and position embeddings, and then use the Transformer module for sequence modeling, and finally output the classification.

6. A method for detecting paper mildew spots based on multi-modal feature fusion according to claim 5, characterized in that: In step S23, the input is the extracted spectral information X ∈ R B×C×N , where N is the number of features, and the expression of the spectral attention process is: X′ = X · softmax(W2 · σ(W1 · z T + b1) + b2); where d is the hidden dimension, W1 ∈ R d×C , W2 ∈ R C×d , σ is the sigmoid activation function, and b1 and b2 are bias parameters.

7. A method for detecting paper mildew spots based on multi-modal feature fusion according to claim 1, characterized in that: In step S23, the hybrid convolutional network uses multi-dimensional feature extraction, i.e., 3D convolution to extract the spectral and spatial features of hyperspectral data, and at the same time combines 2D convolution to further extract spatial features; subsequently, the features are fused layer by layer, that is, the high-dimensional spectral features are integrated into low-dimensional features, and 2D convolution operations are introduced to aggregate and optimize the spatial feature expression.

8. A method for detecting paper mildew spots based on multi-modal feature fusion according to claim 1, characterized in that: In step S24, a fusion method such as additive fusion, multiplicative fusion or splicing fusion is used for feature fusion.

9. A method for detecting paper mildew spots based on multi-modal feature fusion according to claim 1, characterized in that: The molds forming the mildew spots include one or more of Paecilomyces lilacinus, Aspergillus niger, Alternaria alternata, Penicillium citrinum, Trichoderma longibrachiatum and Cladosporium cladosporioides.

Images