Raman spectrum and near infrared spectrum feature fusion method, device and equipment

Abstract

The invention discloses a Raman spectrum and near infrared spectrum feature fusion method, device and equipment, and the method comprises the steps: respectively collecting spectral signals of a to-be-detected substance through a Raman spectrometer and a near infrared spectrometer, and obtaining a corresponding Raman spectrum and a corresponding near infrared spectrum; inputting the Raman spectrum into a position coding-residual convolution module for Raman spectrum feature extraction to obtain a plurality of Raman multi-scale features; inputting the near-infrared spectrum into a near-infrared feature extraction module for near-infrared spectrum feature extraction to obtain a plurality of near-infrared multi-scale features; inputting the Raman multi-scale features and the near-infrared multi-scale features into a two-way cross attention module for feature interaction to obtain a plurality of two-way features of different scales; and fusing the bidirectional features of the plurality of different scales through a multi-scale gating fusion module to obtain a fused feature. The feature information of the two spectrums on different scales can be fully mined and fused, and the sufficiency of cross-modal association mining is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of spectral data analysis, and in particular to a method, device and equipment for feature fusion of Raman spectroscopy and near-infrared spectroscopy. Background Art

[0002] Infrared spectroscopy forms characteristic absorption peaks by the vibration of polar chemical bonds in molecules absorbing infrared light of specific wavelengths, which can sensitively detect the types and contents of functional groups and perform well in identifying oxygen / nitrogen-containing organic compounds and gas molecules. However, it is easily interfered by the strong absorption of water molecules, making it difficult to analyze wet samples or chemicals in high-humidity environments, and it cannot penetrate packaging and dark materials. Raman spectroscopy is based on the frequency shift of inelastic scattered light caused by molecular vibration, which can accurately analyze the vibration of non-polar bonds and the lattice structure of inorganic substances, and has the advantages of being not interfered by water and being able to penetrate transparent packaging to detect internal samples. However, its signal intensity is low, it is easily interfered by the fluorescence background, and the detection efficiency is greatly affected by the sample morphology.

[0003] Nowadays, the advantages of Raman and near-infrared spectroscopy are increasingly combined, and the two are used in combination for spectral analysis from multiple dimensions. Deep learning technology, especially convolutional neural networks (CNNs), performs excellently in image recognition and classification tasks and can identify subtle differences in spectral data. Selecting a suitable network architecture is crucial for classification performance. Related research such as Resnet-based network architectures and transformer-based network models have all demonstrated the powerful ability of deep learning to process high-dimensional and complex data.

[0004] However, existing Raman and near-infrared spectral feature fusion technologies have problems such as low data fusion efficiency, lack of cross-modal interaction mechanisms resulting in insufficient semantic alignment, and poor adaptability to complex samples due to static weight allocation, which restricts their application effects in complex scenarios such as chemical detection / substance analysis. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to propose a method, device and equipment for feature fusion of Raman spectroscopy and near-infrared spectroscopy, aiming to solve the problems of insufficient flexibility and insufficient cross-modal association mining in existing spectral fusion methods.

[0006] To achieve the above purpose, the present invention provides a method for feature fusion of Raman spectroscopy and near-infrared spectroscopy, the method comprising: Using a Raman spectrometer and a near-infrared spectrometer to respectively collect spectral signals of a substance to be detected, obtaining corresponding Raman spectra and near-infrared spectra; Inputting the Raman spectrum into a position encoding-residual convolution module for Raman spectral feature extraction, obtaining a plurality of Raman multi-scale features; Input the near-infrared spectrum into a near-infrared feature extraction module for near-infrared spectrum feature extraction to obtain multiple near-infrared multi-scale features; Input the Raman multi-scale features and the near-infrared multi-scale features into a bidirectional cross-attention module for feature interaction to obtain multiple bidirectional features of different scales; Fuse the multiple bidirectional features of different scales through a multi-scale gated fusion module to obtain a fused feature.

[0007] Preferably, the position encoding-residual convolution module adopts a hierarchical progressive residual convolution architecture. The position encoding-residual convolution module includes a position encoding convolution layer and multiple downsampling layers; the step of inputting the Raman spectrum into the position encoding-residual convolution module for Raman spectrum feature extraction to obtain multiple Raman multi-scale features includes: Input the Raman spectrum into the position encoding convolution layer to obtain Raman position encoding features; Successively pass the Raman position encoding features through multiple downsampling layers to obtain Raman medium-scale features and Raman large-scale features. Among them, the downsampling layer includes two residual blocks and a shortcut connection. The residual block includes a first convolution layer, a batch normalization layer, a RELU activation function, and a second convolution layer connected in series in sequence. There is a shortcut connection between the input and output of each residual block; Use the Raman position encoding features, the Raman medium-scale features, and the Raman large-scale features as the multiple Raman multi-scale features.

[0008] Preferably, the step of inputting the Raman spectrum into the position encoding convolution layer includes: Embed a pre-trained Gaussian second derivative kernel function in the position encoding convolution layer, and perform convolution operation on the Raman spectrum through the Gaussian second derivative kernel function to obtain the Raman position encoding features.

[0009] Preferably, the near-infrared feature extraction module adopts a parallel multi-branch architecture. The near-infrared feature extraction module includes a high-frequency convolution layer branch, a medium-frequency convolution layer branch, and a low-frequency convolution layer branch; the step of inputting the near-infrared spectrum into the near-infrared feature extraction module for near-infrared spectrum feature extraction to obtain multiple Raman multi-scale features includes: Perform a convolution operation on the near-infrared spectrum through the high-frequency convolution layer branch to obtain near-infrared local features; Perform a convolution operation on the near-infrared spectrum through the medium-frequency convolution layer branch to obtain near-infrared group-level features, where the convolution kernel of the high-frequency convolution layer branch is smaller than the convolution kernel of the medium-frequency convolution layer branch; Performing a convolution operation on the near-infrared spectrum through the low-frequency convolution layer branch to obtain a near-infrared global feature, wherein the low-frequency convolution layer branch includes a plurality of fully-connected layers; Taking the near-infrared local feature, the near-infrared group-level feature, and the near-infrared global feature as the plurality of Raman multi-scale features.

[0010] Preferably, the step of inputting the Raman multi-scale features and the near-infrared multi-scale features into a bidirectional cross-attention module for feature interaction to obtain bidirectional features of multiple different scales includes: Respectively selecting the Raman multi-scale features and the near-infrared multi-scale features with the same length as a group to obtain multiple groups of same-scale features; Performing a bidirectional cross-attention mechanism including Raman-to-near-infrared attention and near-infrared-to-Raman attention on each group of the same-scale features to obtain Raman-to-near-infrared features and near-infrared-to-Raman features; Concatenating the Raman-to-near-infrared features and the near-infrared-to-Raman features in the same group to obtain the multiple bidirectional features.

[0011] Preferably, the step of performing a bidirectional cross-attention mechanism including Raman-to-near-infrared attention and near-infrared-to-Raman attention on each group of the same-scale features to obtain Raman-to-near-infrared features and near-infrared-to-Raman features includes: In the Raman-to-near-infrared attention, generating a Raman query vector according to the Raman multi-scale features, and generating a near-infrared key vector and a near-infrared value vector according to the near-infrared multi-scale features; Calculating the similarity between the Raman query vector and the near-infrared key vector, and obtaining the Raman-to-near-infrared attention weight through scaled dot product; Multiplying the Raman-to-near-infrared attention weight by the near-infrared value vector to obtain the Raman-to-near-infrared feature; In the near-infrared-to-Raman attention, generating a near-infrared query vector according to the near-infrared multi-scale features, and generating a Raman key vector and a Raman value vector according to the Raman multi-scale features; Calculating the similarity between the near-infrared query vector and the Raman key vector, and obtaining the near-infrared-to-Raman attention weight through scaled dot product; Multiplying the near-infrared-to-Raman attention weight by the Raman value vector to obtain the near-infrared-to-Raman feature.

[0012] Preferably, the step of fusing the bidirectional features of multiple different scales through a multi-scale gated fusion module to obtain a fusion feature includes: Assigning an importance score to the bidirectional feature of each scale through a scoring network, wherein the scoring network includes multiple levels of fully-connected layers; The bidirectional features of multiple different scales are respectively projected to the same length by different fully-connected layers, and the projected bidirectional features are weighted and summed element by element according to the corresponding importance scores to obtain the fused feature.

[0013] To achieve the above object, the present invention also provides a feature fusion device for Raman spectroscopy and near-infrared spectroscopy, and the device includes: A spectrum acquisition unit, configured to respectively acquire spectral signals of a substance to be detected by using a Raman spectrometer and a near-infrared spectrometer, so as to obtain corresponding Raman spectra and near-infrared spectra; A first extraction unit, configured to input the Raman spectrum into a position encoding-residual convolution module to extract Raman spectral features, so as to obtain a plurality of Raman multi-scale features; A second extraction unit, configured to input the near-infrared spectrum into a near-infrared feature extraction module to extract near-infrared spectral features, so as to obtain a plurality of near-infrared multi-scale features; A feature interaction unit, configured to input the Raman multi-scale features and the near-infrared multi-scale features into a bidirectional cross-attention module to perform feature interaction, so as to obtain a plurality of bidirectional features of different scales; A feature fusion unit, configured to fuse the bidirectional features of multiple different scales through a multi-scale gated fusion module to obtain a fused feature.

[0014] To achieve the above object, the present invention also proposes a feature fusion device for Raman spectroscopy and near-infrared spectroscopy, including a processor, a memory, and a computer program stored in the memory, and the computer program is executed by the processor to implement the steps of a feature fusion method for Raman spectroscopy and near-infrared spectroscopy as described in the above embodiment.

[0015] To achieve the above object, the present invention also proposes a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the steps of a feature fusion method for Raman spectroscopy and near-infrared spectroscopy as described in the above embodiment.

[0016] To achieve the above object, the present invention also proposes a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of a feature fusion method for Raman spectroscopy and near-infrared spectroscopy as described in the above embodiment are implemented.

[0017] Beneficial effects: In the above solution, by extracting multi-scale features from Raman spectroscopy and near-infrared spectroscopy, and then integrating them through the interaction of the bidirectional cross-attention module and the multi-scale gating fusion module, the characteristic information of the two spectra at different scales is fully explored and fused, enabling a more comprehensive and in-depth combination of their advantages. In particular, through the multi-scale pyramid fusion architecture, the Raman-near-infrared feature combinations at different scales are retained for feature fusion, and multi-level semantic alignment fusion such as global scale, local scale, and group scale is performed to fully explore cross-modal associations.

[0018] Adopting a hierarchical progressive residual convolution architecture, the position-encoded features of the Raman spectrum are first extracted through the position-encoding convolution layer. The subsequent downsampling layer contains residual blocks and shortcut connections, which can sequentially generate Raman mid-scale features and large-scale features, realizing multi-scale feature extraction from small to large. It can fully consider the key information of the Raman spectrum at different scales, such as local features like peak position and full width at half maximum, providing a more detailed and comprehensive feature representation for subsequent fusion, enhancing the ability to capture subtle spectral changes, and thus improving the accuracy of material composition and structure analysis. Embedding a Gaussian second derivative kernel function in the position-encoding convolution layer, by leveraging the physical information of the Raman spectrum, the network learning process better conforms to the prior knowledge of spectral analysis, accelerating network convergence and improving feature extraction ability; this kernel function can simulate the shape of the Raman spectral feature peaks, guiding the network to prioritize key local features such as peak position and full width at half maximum, making the feature extraction process more in line with the physical characteristics of the Raman spectrum itself, making the obtained Raman position-encoded features more targeted and representative, strongly enhancing the contribution of Raman spectral features in fusion, and further enhancing the sensitivity of the entire fusion feature to changes in material composition and structure, which is beneficial to improving the performance of the detection system.

[0019] Through the near-infrared feature extraction module with a parallel multi-branch structure, different-scale convolution operations are performed on the near-infrared spectrum respectively, and near-infrared local features, group-level features, and global features can be obtained simultaneously (high-frequency convolution kernels are used to locate fine vibration peaks, medium-frequency convolution kernels correspond to group-level feature extraction, and the low-frequency fully connected layer branch is used to capture the global spectral trend), realizing feature extraction of the near-infrared spectrum at multiple levels, which can more completely represent the material information contained in the near-infrared spectrum, providing a richer and more comprehensive feature basis for subsequent interaction and fusion with Raman features, and making the fused features better reflect the comprehensive characteristics of the material.

[0020] After grouping the multi-scale features of Raman and near-infrared by length, a bidirectional cross-attention mechanism is applied to each group to achieve Raman-to-near-infrared attention and near-infrared-to-Raman attention. On the one hand, it fully exploits the complementarity between the two spectral features, enabling them to provide more valuable information to each other. On the other hand, a closer association is established through bidirectional interaction, making the fused bidirectional features contain both the local and fine information of the Raman spectrum and the global and macroscopic information of the near-infrared spectrum. This effectively solves the cross-modal semantic alignment problem, improves the quality and representational ability of the fused features, and helps to enhance the performance of subsequent tasks such as substance classification and recognition based on the fused features. However, through the constructed bidirectional cross-attention mechanism, a bidirectional attention mechanism with Raman and near-infrared as query-key values is adopted to construct a bidirectional evidence chain, which enhances the complementarity of the dual spectra and also contributes to the full fusion of the dual spectral features.

[0021] A dynamic gating fusion module is used to further align the bidirectional features, and a scoring network is adopted to calculate the importance scores of the bidirectional features at different scales, enabling the network to automatically focus on valuable scale features, flexibly allocate the weights between different scale features, and achieve the dynamic fusion of multi-scale bidirectional features. Compared with fixed-weight fusion, this method can automatically adjust the weights of each scale feature according to the detection scenarios of different substances, making the fused features more adaptable and flexible, better highlighting the feature information that is significant for the current task, thereby improving the representational ability of the fused features for various complex samples and further enhancing the generalization performance and detection effect of the entire feature fusion method in practical applications. Brief Description of the Drawings

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0023] Figure 1 It is a schematic flowchart of a feature fusion method for Raman spectroscopy and near-infrared spectroscopy provided by an embodiment of the present invention.

[0024] Figure 2 It is a schematic diagram of the architecture of a position encoding-residual convolution module provided by an embodiment of the present invention.

[0025] Figure 3 It is a schematic diagram of the architecture of a near-infrared feature extraction module provided by an embodiment of the present invention.

[0026] Figure 4 It is a schematic diagram of the architecture of a bidirectional cross-attention module provided by an embodiment of the present invention.

[0027] Figure 5 Schematic diagram of the architecture of the multi-scale gated fusion module provided by an embodiment of the present invention.

[0028] Figure 6 Comparison diagram of different experimental methods provided by an embodiment of the present invention.

[0029] Figure 7 Schematic diagram of the structure of a feature fusion device for Raman spectroscopy and near-infrared spectroscopy provided by an embodiment of the present invention.

[0030] The realization of the invention object, functional features and advantages will be further described with reference to the embodiments and the accompanying drawings. Specific embodiments

[0031] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0032] The content of the present invention will be elaborated in detail below with reference to the embodiments.

[0033] Refer to Figure 1 The flowchart of a feature fusion method for Raman spectroscopy and near-infrared spectroscopy provided by an embodiment of the present invention is shown as follows.

[0034] In this embodiment, the method includes: S11, using a Raman spectrometer and a near-infrared spectrometer to respectively collect the spectral signals of the substance to be detected, and obtaining the corresponding Raman spectrum and near-infrared spectrum.

[0035] S12, inputting the Raman spectrum into the position encoding-residual convolution module for Raman spectrum feature extraction to obtain a plurality of Raman multi-scale features.

[0036] Furthermore, the position encoding-residual convolution module adopts a hierarchical progressive residual convolution architecture. The position encoding-residual convolution module includes a position encoding convolution layer and multiple downsampling layers. In step S12, inputting the Raman spectrum into the position encoding-residual convolution module for Raman spectrum feature extraction to obtain multiple Raman multi-scale features, including: S12-1, inputting the Raman spectrum into the position encoding convolution layer to obtain Raman position encoding features; S12-2, sequentially passing the Raman position encoding features through multiple downsampling layers to obtain Raman medium-scale features and Raman large-scale features. Among them, the downsampling layer includes two residual blocks and a shortcut connection. The residual block includes a first convolution layer, a batch normalization layer, a RELU activation function, and a second convolution layer connected in series in sequence. There is a shortcut connection between the input and output of each residual block; S12-3, using the Raman position encoding features, the Raman medium-scale features, and the Raman large-scale features as multiple Raman multi-scale features.

[0037] Furthermore, in step S12-1, inputting the Raman spectrum into the position encoding convolution layer includes: Embedding a pre-trained Gaussian second derivative kernel function in the position encoding convolution layer, and performing convolution operation on the Raman spectrum through the Gaussian second derivative kernel function to obtain the Raman position encoding features.

[0038] In this embodiment, the method is implemented based on a spectral feature fusion network, which is used as a pre-feature extraction and fusion network for a Raman and near-infrared dual-spectrum material analysis / detection model. Furthermore, the network architecture of the spectral feature fusion network includes a position encoding-residual convolution module, a near-infrared feature extraction module, a bidirectional cross-attention module, and a multi-scale gated fusion module; among them, the position encoding-residual convolution module is used for Raman feature extraction, with the input being the Raman spectrum and the output being Raman multi-scale features (for example, outputting 3 scale features with feature lengths of 256, 128, and 64 respectively); the near-infrared multi-scale feature extraction module is used for near-infrared spectral feature extraction, with the input being the near-infrared (NIR) spectrum and the output being near-infrared multi-scale features (for example, outputting 3 scale features with feature lengths of 256, 128, and 64 respectively); the bidirectional cross-attention module is used for cross-extraction and interaction of spectral features, with the input being the multi-scale features output by the previous 2 feature extraction modules and the output being bidirectional features (for example, outputting 3 bidirectional features corresponding to the feature groups with lengths of 256, 128, and 64 respectively); the dynamic gated fusion module is used for fusing the previous bidirectional features to obtain the final fusion features.

[0039] Refer to Figure 2As shown, the position encoding-residual convolution module adopts a hierarchical progressive residual convolution architecture to achieve multi-scale feature extraction of Raman spectra from small scales to large scales. Specifically, a trainable Gaussian second derivative kernel is embedded in the first-layer convolution , ; Among them, represents the wavenumber coordinate, is the standard deviation parameter of the Gaussian distribution, which controls the width of the kernel function and directly affects the resolution of peak detection. It can take values from 3 to 6.

[0040] That is, is used as the convolution kernel to perform convolution operations on the Raman spectrum. This kernel function simulates the shape of the characteristic peak, forcing the network to give priority to key local features such as peak position and full width at half maximum, extracts the Raman position encoding features of the spectrum, and at the same time suppresses the fluorescence background interference through residual connections. Then, after passing through the downsampling layer composed of two residual blocks and shortcut connections in sequence, Raman medium-scale features and Raman large-scale features are obtained. Among them, the residual block is composed of a first convolutional layer, a batch normalization layer, a RELU activation function, and a second convolutional layer in series, and there is also a shortcut connection between the input and output of each residual block. In a specific implementation, 3 different-scale Raman features are generated through this position encoding-residual convolution module, including Raman position encoding features with a length of 256, Raman medium-scale features with a length of 128, and Raman large-scale features with a length of 64. Specifically, in other implementations, different scales and numbers of layers of features can be changed. For the obtained Raman position encoding features, Raman medium-scale features, and Raman large-scale features, adaptive pooling is also performed to fix the output feature lengths to 256, 128, and 64 respectively for subsequent feature fusion.

[0041] S13. Input the near-infrared spectrum into the near-infrared feature extraction module to extract near-infrared spectrum features, and obtain multiple near-infrared multi-scale features.

[0042] Furthermore, the near-infrared feature extraction module adopts a parallel multi-branch architecture. The near-infrared feature extraction module includes a high-frequency convolutional layer branch, a medium-frequency convolutional layer branch, and a low-frequency convolutional layer branch; in step S13, the input of the near-infrared spectrum into the near-infrared feature extraction module to extract near-infrared spectrum features and obtain multiple Raman multi-scale features includes: S13-1. Perform convolution operations on the near-infrared spectrum through the high-frequency convolutional layer branch to obtain near-infrared local features; S13-2. Perform convolution operations on the near-infrared spectrum through the medium-frequency convolutional layer branch to obtain near-infrared group-level features, where the convolution kernel of the high-frequency convolutional layer branch is smaller than the convolution kernel of the medium-frequency convolutional layer branch; S13-3, perform a convolution operation on the near-infrared spectrum through the low-frequency convolution layer branch to obtain near-infrared global features, where the low-frequency convolution layer branch includes multiple fully connected layers; S13-4, use the near-infrared local features, the near-infrared group-level features, and the near-infrared global features as multiple Raman multi-scale features.

[0043] In this embodiment, the near-infrared feature extraction module extracts complementary features of the near-infrared spectrum through a parallel multi-branch architecture and achieves semantic alignment with the Raman multi-scale features. Refer to Figure 3 As shown, this module uses 3 parallel branches to extract multi-scale features of the near-infrared spectrum respectively. Specifically: the fully connected layer path (low-frequency convolution layer branch) is composed of three fully connected layers in series, reducing the dimension of the near-infrared spectrum from the original length in the order of 1024, 256, and 64, directly mapping the original spectrum of the near-infrared spectrum to a low-dimensional space, and obtaining a near-infrared large-scale feature vector with a length of 64 (that is, for the large-scale feature level, use a fully connected network to directly encode the global spectrum trend, capture macroscopic characteristics such as the baseline slope and overall energy distribution, and obtain near-infrared global features); the intermediate frequency path of sub-band division (intermediate frequency convolution layer branch) uses a wide convolution kernel with a width of 200 to perform a convolution operation on the original spectrum (kernel_size = 200, stride = 100), dividing the near-infrared spectrum into several sub-bands according to a length of 200, independently extracting features for each sub-band and then aligning them to 128 dimensions through adaptive pooling, which can characterize the features of the near-infrared spectrum group region and obtain a near-infrared group-level feature vector (that is, for the sub-band level, use a wide convolution kernel to divide the original spectrum into sub-bands, independently extract features for each sub-band, and retain significant responses through max pooling); the high-frequency convolution path (high-frequency convolution layer branch) uses a convolution kernel with a smaller width to perform a convolution operation on the original spectrum (kernel_size = 20, stride = 10), locates fine vibration peaks, and fixes the output near-infrared local features to a length of 256 through adaptive pooling (that is, use a small convolution kernel to capture details, such as small offsets of C-H stretching peaks).

[0044] S14, input the Raman multi-scale features and the near-infrared multi-scale features into a bidirectional cross-attention module for feature interaction to obtain multiple bidirectional features of different scales.

[0045] Further, in step S14, the input of the Raman multi-scale features and the near-infrared multi-scale features into the bidirectional cross-attention module for feature interaction to obtain multiple bidirectional features of different scales includes: S14-1, respectively select the Raman multi-scale features and the near-infrared multi-scale features with the same length as a group to obtain multiple groups of same-scale features; S14-2. Process each group of the same-scale features through a bidirectional cross-attention mechanism including Raman-to-near-infrared attention and near-infrared-to-Raman attention to obtain Raman-to-near-infrared features and near-infrared-to-Raman features; S14-3. Concatenate the Raman-to-near-infrared features and the near-infrared-to-Raman features of the same group to obtain a plurality of the bidirectional features.

[0046] Further, in step S14-2, the process of processing each group of the same-scale features through a bidirectional cross-attention mechanism including Raman-to-near-infrared attention and near-infrared-to-Raman attention to obtain Raman-to-near-infrared features and near-infrared-to-Raman features includes: S14-2-1. In the Raman-to-near-infrared attention, generate a Raman query vector according to the Raman multi-scale features, and generate a near-infrared key vector and a near-infrared value vector according to the near-infrared multi-scale features; S14-2-2. Calculate the similarity between the Raman query vector and the near-infrared key vector, and obtain the Raman-to-near-infrared attention weight through scaled dot product; S14-2-3. Multiply the Raman-to-near-infrared attention weight by the near-infrared value vector to obtain the Raman-to-near-infrared features; S14-2-4. In the near-infrared-to-Raman attention, generate a near-infrared query vector according to the near-infrared multi-scale features, and generate a Raman key vector and a Raman value vector according to the Raman multi-scale features; S14-2-5. Calculate the similarity between the near-infrared query vector and the Raman key vector, and obtain the near-infrared-to-Raman attention weight through scaled dot product; S14-2-6. Multiply the near-infrared-to-Raman attention weight by the Raman value vector to obtain the near-infrared-to-Raman features.

[0047] In this embodiment, through the bidirectional cross-attention mechanism, the deep interaction between Raman and near-infrared features is realized. The bidirectional cross-attention first generates corresponding query vectors, key vectors, and value vectors through Raman features and near-infrared features. Then, the bidirectional attention retrieval between Raman and near-infrared is performed, including: Raman→near-infrared retrieval: based on the Raman query vector, retrieve the associated pattern with the near-infrared key vector and value vector pair; near-infrared→Raman retrieval: conversely, based on the near-infrared query vector, locate the corresponding evidence chain in the Raman features. Finally, the dual-spectrum features are concatenated to obtain the bidirectional features.

[0048] Specifically, for the Raman multi-scale features and near-infrared multi-scale features (such as the 3 groups in the above embodiments) output by the position encoding-residual convolution module and the near-infrared multi-scale feature extraction module, Raman multi-scale features and near-infrared multi-scale features of the same length are alternately selected as a group (such as a group with a length of 256) for two-way cross-attention mechanism processing. Among them, the two-way cross-attention mechanism includes steps in two directions, namely Raman-to-near-infrared attention and near-infrared-to-Raman attention, and finally feature combination.

[0049] For Raman-to-near-infrared attention, first generate a query vector Q based on the Raman multi-scale features R , and generate a key vector K based on the near-infrared multi-scale features N and a value vector V N ; where ; ; ; In the formula, F Raman and F NIR correspond to the Raman multi-scale features and the near-infrared multi-scale features respectively, , , etc. represent learnable weight vectors; Then calculate the similarity between the Raman query vector Q R and the near-infrared key vector K N through dot product operation, and obtain the Raman-to-near-infrared attention weight Attn R by scaled dot product; that is, ; In the formula, d is the feature length (such as 256, 128, 64), T represents the vector transpose operation, is the transpose form of the vector, and the softmax function is a common activation function used for vector normalization of similarity; Finally, multiply the Raman-to-near-infrared attention weight Attn R by the near-infrared value vector V N to obtain the Raman-to-near-infrared feature F RtoN ; that is, .

[0050] Similarly, for near-infrared-to-Raman attention, first generate a query vector Q based on the near-infrared multi-scale features N , and generate a key vector K based on the Raman multi-scale features R and a value vector V R ; where ; ; ; In the formula, F Raman and F NIR correspond to Raman multi-scale features and near-infrared multi-scale features respectively, , , etc. represent learnable weight vectors; Then calculate the similarity between the near-infrared query vector Q N and the Raman bond vector K R , and use the scaled dot product as the Raman-to-near-infrared attention weight Attn N ; ; where d is the feature length, and the softmax function is a common activation function used for vector normalization of the similarity; Finally, multiply the near-infrared-to-Raman attention weight Attn N by the Raman value vector V R to obtain the near-infrared-to-Raman feature F NtoR ; that is, .

[0051] For a set of Raman-near-infrared multi-scale features, after obtaining the above F RtoN and F NtoR , splice them row by row to obtain the corresponding bidirectional feature F fused ; that is, F fused = [F R ; F N = [F RtoN ; F NtoR .

[0052] The above is the processing process of a set of Raman-near-infrared multi-scale features through the bidirectional cross-attention fusion module. Each set of Raman-near-infrared multi-scale features passes through this module to obtain a corresponding bidirectional feature F fused , and finally several bidirectional features are obtained (such as in the above embodiment, three sets of Raman-near-infrared multi-scale features with lengths of 256, 128, and 64 respectively obtain the corresponding F fused , that is, a total of 3 F fused : F fused1 , F fused2 , F fused3 ).

[0053] S15, fuse the multiple bidirectional features of different scales through the multi-scale gating fusion module to obtain the fusion feature.

[0054] Further, in step S15, the process of fusing the bidirectional features of multiple different scales through the multi-scale gated fusion module to obtain the fused features includes: S15-1, assigning an importance score to the bidirectional feature of each scale through a scoring network, where the scoring network includes multiple levels of fully connected layers; S15-2, using different fully connected layers to project the bidirectional features of multiple different scales to the same length respectively, and performing element-wise weighted summation on the projected bidirectional features according to the corresponding importance scores to obtain the fused features.

[0055] In this embodiment, for the obtained bidirectional features of multiple different scales, a multi-scale gated fusion module is used for processing, and its processing process includes importance score calculation and weighted fusion. For the multiple bidirectional features (F fused1 、F fused2 、F fused3 ) output by the bidirectional cross-attention module in the foregoing embodiment, since the concatenation of the bidirectional features of Raman and near-infrared is performed, the data length is doubled (doubled from 256, 128, 64 to 512, 256, 128 in length). First, an importance score is independently assigned to the bidirectional feature of each scale, then the fully connected layer is used to unify the length of the bidirectional features of different scales, and finally, weighted output is performed to obtain the final fused features.

[0056] Referring to Figure 4 as shown, first, the bidirectional features output by the bidirectional cross-attention module are concatenated along the feature dimension into a joint vector (for this embodiment, the total length after concatenation is 512 + 256 + 128 = 896), and the joint vector is input into a scoring network composed of two levels of fully connected layers (the hierarchical dimensions in the embodiment are 896→256→3). Among them, the first fully connected layer: the input dimension is the same as the concatenated feature dimension (896→256 in the embodiment); the second fully connected layer: the output dimension matches the number of feature groups (256→3 in the embodiment); then weight normalization is performed, and the importance score W k is generated through the Softmax function, that is, W k =Softmax(MLP(F fused k )); where MLP represents inputting the vector into the fully connected layer defined above for calculation, softmax is the activation function, and F fused k represents the kth bidirectional feature.

[0057] Then perform weighted fusion: First, set up independent linear layers for each bi-directional feature to align the lengths of several bi-directional features. That is, use 3 independent linear layers. With the original length of the bi-directional feature as the input length and the target length as the output length, project the 3 bi-directional features to the same length respectively (in the embodiment, use three linear layers of 512→256, 256→256, and 128→256 respectively to unify the 3 bi-directional features of different scales to a length of 256). Finally, perform element-wise weighted summation on the projected bi-directional features according to the importance score w k Perform element-wise weighted summation on the projected bi-directional features to obtain the fused feature F with a unified dimension in the final output final (The output of the embodiment is 256-dimensional); that is,[[]] ; Among them, 、 、 refer to the bi-directional features with the same length after projection, and w k refers to the corresponding importance score.[[]]

[0058] The above spectral feature fusion network is applied to the tea leaf scenario below. A dataset is established using the existing Pu-erh tea Raman-near infrared database. Each origin contains 100 near-infrared spectral data and Raman spectral data, as shown in the following Table 1, the Pu-erh tea origin classification dataset:[[]] Table 1

[0059] A residual network with a similar number of layers to the above embodiment is used for feature extraction of single spectra, and a fully connected layer is used as the classification network. Three schemes of using separate Raman spectra, separate near-infrared spectra, and simple feature fusion are used as the control test groups for the origin classification task respectively. The experimental scheme and results are as Figure 5 and Table 2 below:[[]] Table 2

[0060] Based on the above, the accuracy rate of this scheme (93.8%) is significantly higher than that of the single-spectrum model (Raman 67.2%, near-infrared 75.7%) and the simple feature splicing method (83.5%). Compared with the traditional scheme, the feature extraction and fusion effect of this scheme is better, which can fully fuse the features of Raman and near-infrared spectra and provide a reliable basis for subsequent analysis / detection tasks.[[]]

[0061] Refer to Figure 7 which shows the structural schematic diagram of a feature fusion device for Raman spectra and near-infrared spectra provided by an embodiment of the present invention.[[]]

[0062] In this embodiment, the device 20 includes:[[]] The spectral acquisition unit 21 is configured to respectively acquire spectral signals of a substance to be detected by using a Raman spectrometer and a near-infrared spectrometer, so as to obtain a corresponding Raman spectrum and a near-infrared spectrum; The first extraction unit 22 is configured to input the Raman spectrum into a position encoding-residual convolution module for Raman spectrum feature extraction, so as to obtain a plurality of Raman multi-scale features; The second extraction unit 23 is configured to input the near-infrared spectrum into a near-infrared feature extraction module for near-infrared spectrum feature extraction, so as to obtain a plurality of near-infrared multi-scale features; The feature interaction unit 24 is configured to input the Raman multi-scale features and the near-infrared multi-scale features into a bidirectional cross-attention module for feature interaction, so as to obtain a plurality of bidirectional features with different scales; The feature fusion unit 25 is configured to fuse the plurality of bidirectional features with different scales through a multi-scale gated fusion module to obtain a fusion feature.

[0063] Each unit module of the device 20 can respectively execute the corresponding steps in the above method embodiments, so the unit modules will not be elaborated here. For details, please refer to the descriptions of the corresponding steps above.

[0064] The embodiment of the present invention further provides a device for fusing features of a Raman spectrum and a near-infrared spectrum. The device includes the device for fusing features of a Raman spectrum and a near-infrared spectrum as described above. Among them, the device for fusing features of a Raman spectrum and a near-infrared spectrum can adopt Figure 7 the structure of the embodiment, and correspondingly, it can execute Figure 1 the technical solution of the method embodiment shown. The implementation principle and technical effects are similar. For details, please refer to the relevant records in the above embodiments and will not be elaborated here.

[0065] The device includes: a device with a photographing function such as a mobile phone, a digital camera or a tablet computer, or a device with an image processing function, or a device with an image display function. The device may include components such as a memory, a processor, an input unit, a display unit, and a power supply.

[0066] Among them, the memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as an image playback function, etc.); the data storage area can store data created according to the use of the device, etc. In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory can also include a memory controller to provide access to the memory for the processor and the input unit.

[0067] The input unit can be used to receive input digital or character or image information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls. Specifically, in addition to including a camera, the input unit in this embodiment can also include a touch-sensitive surface (such as a touch display screen) and other input devices.

[0068] The display unit can be used to display information input by the user or information provided to the user and various graphical user interfaces of the device. These graphical user interfaces can be composed of graphics, text, icons, videos, and any combination thereof. The display unit can include a display panel. Optionally, the display panel can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface can cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor to determine the type of touch event. Subsequently, the processor provides a corresponding visual output on the display panel according to the type of touch event.

[0069] The embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium can be the computer-readable storage medium included in the memory in the above embodiment; it can also exist independently and be a computer-readable storage medium not assembled into the device. At least one instruction is stored in the computer-readable storage medium, and the instruction is loaded and executed by the processor to implement Figure 1 the characteristic fusion method of Raman spectroscopy and near-infrared spectroscopy shown. The computer-readable storage medium can be a read-only memory, a magnetic disk, an optical disc, etc.

[0070] The embodiment of the present invention also provides a computer program product, including a computer program / instructions, and the computer program / instructions are loaded and executed by the processor to implement Figure 1 a characteristic fusion method of Raman spectroscopy and near-infrared spectroscopy shown.

[0071] It should be noted that the various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other. For the device embodiments, equipment embodiments, and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, reference can be made to the corresponding parts of the method embodiments.

[0072] Also, in this document, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the said element.

[0073] The above description shows and describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments. Instead, it can be used in various other combinations, modifications and environments, and can be changed within the scope of the inventive concept herein through the above teachings or the techniques or knowledge in the relevant field. Any changes and variations made by those skilled in the art without departing from the spirit and scope of the present invention shall fall within the protection scope of the appended claims of the present invention.

Claims

1. A method for feature fusion of Raman spectroscopy and near-infrared spectroscopy, characterized in that, The method includes: Using a Raman spectrometer and a near-infrared spectrometer to collect the spectral signals of the substance to be detected respectively, obtaining the corresponding Raman spectrum and near-infrared spectrum; Inputting the Raman spectrum into a position encoding-residual convolution module for Raman spectral feature extraction to obtain multiple Raman multi-scale features; Inputting the near-infrared spectrum into a near-infrared feature extraction module for near-infrared spectral feature extraction to obtain multiple near-infrared multi-scale features; Inputting the Raman multi-scale features and the near-infrared multi-scale features into a bidirectional cross-attention module for feature interaction to obtain multiple bidirectional features of different scales; Fusing the multiple bidirectional features of different scales through a multi-scale gated fusion module to obtain a fused feature.

2. The characteristic fusion method of Raman spectroscopy and near-infrared spectroscopy according to claim 1, wherein The position encoding-residual convolution module adopts a hierarchical progressive residual convolution architecture. The position encoding-residual convolution module includes a position encoding convolution layer and multiple downsampling layers. The step of inputting the Raman spectrum into the position encoding-residual convolution module for Raman spectral feature extraction to obtain multiple Raman multi-scale features includes: Inputting the Raman spectrum into the position encoding convolution layer to obtain a Raman position encoding feature; Sequentially passing the Raman position encoding feature through multiple downsampling layers to obtain a Raman medium-scale feature and a Raman large-scale feature. Among them, the downsampling layer includes two residual blocks and a shortcut connection. The residual block includes a first convolution layer, a batch normalization layer, a RELU activation function, and a second convolution layer connected in series in sequence. There is a shortcut connection between the input and output of each residual block; Regarding the Raman position encoding feature, the Raman medium-scale feature, and the Raman large-scale feature as multiple Raman multi-scale features.

3. A method for feature fusion of Raman spectroscopy and near-infrared spectroscopy according to claim 2, characterized in that The step of inputting the Raman spectrum into the position encoding convolution layer includes: Embedding a pre-trained Gaussian second derivative kernel function in the position encoding convolution layer, and performing a convolution operation on the Raman spectrum through the Gaussian second derivative kernel function to obtain the Raman position encoding feature.

4. A method for feature fusion of Raman spectroscopy and near-infrared spectroscopy according to claim 1, characterized in that, The near-infrared feature extraction module adopts a parallel multi-branch architecture. The near-infrared feature extraction module includes a high-frequency convolution layer branch, a medium-frequency convolution layer branch, and a low-frequency convolution layer branch. The step of inputting the near-infrared spectrum into the near-infrared feature extraction module for near-infrared spectral feature extraction to obtain multiple Raman multi-scale features includes: Performing a convolution operation on the near-infrared spectrum through the high-frequency convolution layer branch to obtain a near-infrared local feature; Performing a convolution operation on the near-infrared spectrum through the medium-frequency convolution layer branch to obtain a near-infrared group-level feature, where the convolution kernel of the high-frequency convolution layer branch is smaller than that of the medium-frequency convolution layer branch; Performing a convolution operation on the near-infrared spectrum through the low-frequency convolution layer branch to obtain a near-infrared global feature, where the low-frequency convolution layer branch includes multiple fully connected layers; Regarding the near-infrared local feature, the near-infrared group-level feature, and the near-infrared global feature as multiple Raman multi-scale features.

5. A feature fusion method for Raman spectroscopy and near-infrared spectroscopy according to claim 1, characterized in that, Inputting the Raman multi-scale features and the near-infrared multi-scale features into a bidirectional cross-attention module for feature interaction to obtain multiple bidirectional features of different scales, including: Selecting the Raman multi-scale features and the near-infrared multi-scale features with the same length as a group respectively to obtain multiple groups of same-scale features; Performing bidirectional cross-attention mechanism processing including Raman-to-near-infrared attention and near-infrared-to-Raman attention on each group of the same-scale features to obtain Raman-to-near-infrared features and near-infrared-to-Raman features; Concatenating the Raman-to-near-infrared features and the near-infrared-to-Raman features of the same group to obtain multiple of the bidirectional features.

6. A characteristic fusion method of Raman spectroscopy and near-infrared spectroscopy according to claim 5, characterized in that Performing the bidirectional cross-attention mechanism including Raman-to-near-infrared attention and near-infrared-to-Raman attention on each group of the same-scale features to obtain Raman-to-near-infrared features and near-infrared-to-Raman features, including: In the Raman-to-near-infrared attention, generating a Raman query vector according to the Raman multi-scale features, and generating a near-infrared key vector and a near-infrared value vector according to the near-infrared multi-scale features; Calculating the similarity between the Raman query vector and the near-infrared key vector, and obtaining the Raman-to-near-infrared attention weight through scaled dot product; Multiplying the Raman-to-near-infrared attention weight by the near-infrared value vector to obtain the Raman-to-near-infrared features; In the near-infrared-to-Raman attention, generating a near-infrared query vector according to the near-infrared multi-scale features, and generating a Raman key vector and a Raman value vector according to the Raman multi-scale features; Calculating the similarity between the near-infrared query vector and the Raman key vector, and obtaining the near-infrared-to-Raman attention weight through scaled dot product; Multiplying the near-infrared-to-Raman attention weight by the Raman value vector to obtain the near-infrared-to-Raman features.

7. A method for fusing the characteristics of Raman spectroscopy and near-infrared spectroscopy according to claim 1, characterized in that Fusing the multiple bidirectional features of different scales through a multi-scale gated fusion module to obtain a fused feature, including: Assigning an importance score to each scale of the bidirectional features through a scoring network, where the scoring network includes multiple fully-connected layers; Projecting the multiple bidirectional features of different scales to the same length respectively by using different fully-connected layers, and performing element-wise weighted summation on the projected bidirectional features according to the corresponding importance scores to obtain the fused feature.

8. A feature fusion device for Raman spectroscopy and near-infrared spectroscopy, characterized in that, The device includes: A spectrum acquisition unit, configured to respectively acquire spectrum signals of a substance to be detected by using a Raman spectrometer and a near-infrared spectrometer to obtain corresponding Raman spectra and near-infrared spectra; A first extraction unit, configured to input the Raman spectra into a position encoding-residual convolution module for Raman spectrum feature extraction to obtain multiple Raman multi-scale features; A second extraction unit, configured to input the near-infrared spectra into a near-infrared feature extraction module for near-infrared spectrum feature extraction to obtain multiple near-infrared multi-scale features; A feature interaction unit, configured to input the Raman multi-scale features and the near-infrared multi-scale features into a bidirectional cross-attention module for feature interaction to obtain multiple bidirectional features of different scales; A feature fusion unit, configured to fuse the multiple bidirectional features of different scales through a multi-scale gating fusion module to obtain a fused feature.

9. A characteristic fusion device for Raman spectroscopy and near-infrared spectroscopy, characterized in that, It includes a processor, a memory, and a computer program stored in the memory. The computer program is executed by the processor to implement the steps of a feature fusion method for Raman spectroscopy and near-infrared spectroscopy according to any one of claims 1 to 7.

10. A computer program product, characterized in that, It includes computer programs / instructions. When the computer programs / instructions are executed by a processor, the steps of a feature fusion method for Raman spectroscopy and near-infrared spectroscopy according to any one of claims 1 to 7 are implemented.

Images