A method, device and equipment for fusion of Raman spectroscopy and near-infrared spectroscopy

Abstract

The present invention discloses a method, device, and apparatus for fusion of Raman and near-infrared spectra, comprising: using a Raman spectrometer and a near-infrared spectrometer to respectively collect spectral signals of a substance to be detected, obtaining corresponding Raman and near-infrared spectra; inputting the Raman spectra into a position encoding-residual convolution module for Raman spectral feature extraction to obtain multiple Raman multi-scale features; inputting the near-infrared spectra into a near-infrared feature extraction module for near-infrared spectral feature extraction to obtain multiple near-infrared multi-scale features; inputting the Raman multi-scale features and the near-infrared multi-scale features into a bidirectional cross-attention module for feature interaction to obtain multiple bidirectional features of different scales; and fusing the multiple bidirectional features of different scales through a multi-scale gated fusion module to obtain a fused feature. The method can fully mine and fuse the feature information of the two spectra at different scales, achieving sufficient cross-modal association mining.

Description

Technical Field

[0001] The present invention relates to the technical field of spectral data analysis, and in particular to a method, device and equipment for fusing the features of Raman spectrum and near-infrared spectrum. Background Art

[0002] Infrared spectroscopy uses the vibration of polar chemical bonds within molecules to absorb infrared light of specific wavelengths, forming characteristic absorption peaks. This allows for sensitive detection of the types and contents of functional groups and excels in identifying oxygen- and nitrogen-containing organic compounds and gas molecules. However, it is susceptible to interference from strong absorption by water molecules, making it difficult to analyze wet samples or chemicals in high-humidity environments. Furthermore, it cannot penetrate packaging or dark materials. Raman spectroscopy, based on the frequency shift of inelastic scattered light caused by molecular vibrations, can accurately analyze the vibrations of non-polar bonds and the lattice structure of inorganic materials. It has the advantages of being unaffected by water and able to penetrate transparent packaging to detect samples within. However, its signal strength is low, it is susceptible to interference from fluorescent background, and its detection efficiency is significantly affected by sample morphology.

[0003] Nowadays, the advantages of Raman and near-infrared spectroscopy are increasingly being combined to perform multi-dimensional spectral analysis. Deep learning techniques, particularly convolutional neural networks (CNNs), have demonstrated excellent performance in image recognition and classification tasks, capable of discerning subtle differences in spectral data. Choosing the right network architecture is crucial for classification performance. Related research, such as ResNet-based network architectures and transformer-based network models, has demonstrated the powerful ability of deep learning to handle high-dimensional and complex data.

[0004] However, existing Raman and near-infrared spectral feature fusion technology has problems such as low data fusion efficiency, insufficient semantic alignment due to the lack of cross-modal interaction mechanism, and poor adaptability to complex samples caused by static weight distribution, which restricts its application in complex scenarios such as chemical detection / material analysis. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to propose a method, device and equipment for feature fusion of Raman spectrum and near-infrared spectrum, aiming to solve the problems of insufficient flexibility and insufficient cross-modal correlation mining in existing spectral fusion methods.

[0006] To achieve the above object, the present invention provides a method for fusion of Raman spectroscopy and near-infrared spectroscopy, the method comprising:

[0007] The spectral signals of the substance to be detected are collected by a Raman spectrometer and a near-infrared spectrometer respectively to obtain the corresponding Raman spectrum and near-infrared spectrum;

[0008] Inputting the Raman spectrum into a position encoding-residual convolution module to extract Raman spectrum features to obtain multiple Raman multi-scale features;

[0009] Inputting the near-infrared spectrum into a near-infrared feature extraction module to extract near-infrared spectrum features to obtain multiple near-infrared multi-scale features;

[0010] Inputting the Raman multi-scale features and the near-infrared multi-scale features into a bidirectional cross attention module for feature interaction to obtain a plurality of bidirectional features of different scales;

[0011] The bidirectional features of multiple scales are fused through a multi-scale gated fusion module to obtain a fused feature.

[0012] Preferably, the position encoding-residual convolution module adopts a hierarchical progressive residual convolution architecture, and the position encoding-residual convolution module includes a position encoding convolution layer and multiple downsampling layers; the Raman spectrum is input into the position encoding-residual convolution module for Raman spectrum feature extraction to obtain multiple Raman multi-scale features, including:

[0013] Inputting the Raman spectrum into the position coding convolution layer to obtain Raman position coding features;

[0014] Passing the Raman position coding features sequentially through a plurality of downsampling layers to obtain Raman mid-scale features and Raman large-scale features, wherein the downsampling layer includes two residual blocks and a shortcut connection, and the residual block includes a first convolutional layer, a batch normalization layer, a RELU activation function, and a second convolutional layer connected in series, and a shortcut connection is provided between the input and output of each residual block;

[0015] The Raman position coding feature, the Raman mesoscale feature and the Raman large-scale feature are used as a plurality of the Raman multi-scale features.

[0016] Preferably, inputting the Raman spectrum into the position encoding convolutional layer comprises:

[0017] A pre-trained Gaussian second-order derivative kernel function is embedded in the position coding convolution layer, and a convolution operation is performed on the Raman spectrum through the Gaussian second-order derivative kernel function to obtain the Raman position coding feature.

[0018] Preferably, the near-infrared feature extraction module adopts a parallel multi-branch architecture, and the near-infrared feature extraction module includes a high-frequency convolution layer branch, a medium-frequency convolution layer branch, and a low-frequency convolution layer branch; the near-infrared spectrum is input into the near-infrared feature extraction module to extract near-infrared spectrum features, and multiple Raman multi-scale features are obtained, including:

[0019] Performing a convolution operation on the near-infrared spectrum through the high-frequency convolution layer branch to obtain near-infrared local features;

[0020] Performing a convolution operation on the near-infrared spectrum through the intermediate-frequency convolution layer branch to obtain near-infrared group-level features, wherein the convolution kernel of the high-frequency convolution layer branch is smaller than the convolution kernel of the intermediate-frequency convolution layer branch;

[0021] Performing a convolution operation on the near-infrared spectrum through the low-frequency convolution layer branch to obtain a near-infrared global feature, wherein the low-frequency convolution layer branch includes a plurality of fully connected layers;

[0022] The near-infrared local features, the near-infrared group-level features, and the near-infrared global features are used as a plurality of the Raman multi-scale features.

[0023] Preferably, the Raman multi-scale features and the near-infrared multi-scale features are input into a bidirectional cross attention module for feature interaction to obtain a plurality of bidirectional features of different scales, including:

[0024] Selecting the Raman multi-scale features and the near-infrared multi-scale features with the same length as a group to obtain multiple groups of same-scale features;

[0025] Processing each group of the same-scale features with a bidirectional cross-attention mechanism including Raman-to-near-infrared attention and near-infrared-to-Raman attention to obtain Raman-to-near-infrared features and near-infrared-to-Raman features;

[0026] The Raman to near-infrared features and the near-infrared to Raman features of the same group are spliced together to obtain a plurality of the bidirectional features.

[0027] Preferably, the bidirectional cross attention mechanism of Raman to near infrared attention and near infrared to Raman attention is performed on each group of the same-scale features to obtain Raman to near infrared features and near infrared to Raman features, including:

[0028] In the Raman to NIR attention, a Raman query vector is generated according to the Raman multi-scale feature, and a NIR key vector and a NIR value vector are generated according to the NIR multi-scale feature;

[0029] Calculating the similarity between the Raman query vector and the near-infrared key vector, and obtaining the Raman-to-near-infrared attention weight by scaling the dot product;

[0030] Multiplying the Raman-to-near-infrared attention weight by the near-infrared value vector to obtain the Raman-to-near-infrared feature;

[0031] In the near-infrared Raman attention, a near-infrared query vector is generated according to the near-infrared multi-scale features, and a Raman key vector and a Raman value vector are generated according to the Raman multi-scale features;

[0032] Calculating the similarity between the near-infrared query vector and the Raman key vector, and obtaining the near-infrared Raman attention weight by scaling the dot product;

[0033] The near-infrared Raman attention weight is multiplied by the Raman value vector to obtain the near-infrared Raman feature.

[0034] Preferably, the multi-scale gated fusion module is used to fuse the bidirectional features of multiple different scales to obtain a fused feature, including:

[0035] Assigning an importance score to the bidirectional features at each scale through a scoring network, wherein the scoring network includes multiple levels of fully connected layers;

[0036] Different fully connected layers are used to project the bidirectional features of multiple different scales to the same length respectively, and the projected bidirectional features are weightedly summed element by element according to the corresponding importance scores to obtain the fused features.

[0037] To achieve the above object, the present invention further provides a device for fusion of Raman spectroscopy and near-infrared spectroscopy, the device comprising:

[0038] A spectrum acquisition unit is used to acquire spectral signals of the substance to be detected using a Raman spectrometer and a near-infrared spectrometer, respectively, to obtain corresponding Raman spectra and near-infrared spectra;

[0039] A first extraction unit is configured to input the Raman spectrum into a position encoding-residual convolution module to extract Raman spectrum features to obtain a plurality of Raman multi-scale features;

[0040] A second extraction unit is used to input the near-infrared spectrum into a near-infrared feature extraction module to extract near-infrared spectrum features to obtain multiple near-infrared multi-scale features;

[0041] A feature interaction unit, configured to input the Raman multi-scale features and the near-infrared multi-scale features into a bidirectional cross attention module for feature interaction, thereby obtaining a plurality of bidirectional features at different scales;

[0042] The feature fusion unit is used to fuse the bidirectional features of multiple scales through a multi-scale gated fusion module to obtain a fused feature.

[0043] In order to achieve the above-mentioned objectives, the present invention also proposes a feature fusion device for Raman spectrum and near-infrared spectrum, comprising a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a feature fusion method for Raman spectrum and near-infrared spectrum as described in the above-mentioned embodiment.

[0044] In order to achieve the above objectives, the present invention also proposes a computer-readable storage medium, on which a computer program is stored. The computer program is executed by a processor to implement the steps of a method for fusion of Raman spectroscopy and near-infrared spectroscopy as described in the above embodiment.

[0045] To achieve the above objectives, the present invention further provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of a method for fusion of Raman spectroscopy and near-infrared spectroscopy features as described in the above embodiment.

[0046] Beneficial effects:

[0047] The above solution extracts multi-scale features from Raman and near-infrared spectra, then integrates them through a bidirectional cross-attention module and a multi-scale gated fusion module. This fully exploits and fuses the characteristic information of the two spectra at different scales, combining the strengths of both in a more comprehensive and in-depth manner. Specifically, through a multi-scale pyramid fusion architecture, it retains Raman and near-infrared feature combinations at different scales for feature fusion. This fusion is achieved through multi-level semantic alignment at global, local, and group scales, enabling the full exploitation of cross-modal associations.

[0048] Using a hierarchical residual convolutional architecture, the position-encoding convolutional layer first extracts the position-encoding features of the Raman spectrum. Subsequent downsampling layers, including residual blocks and shortcut connections, sequentially generate medium-scale and large-scale Raman features, achieving multi-scale feature extraction from small to large. This fully considers key information about Raman spectra at different scales, such as peak position and half-width, providing a more detailed and comprehensive feature representation for subsequent fusion, enhancing the ability to capture subtle spectral changes and, consequently, improving the accuracy of composition and structural analysis. A Gaussian second-order derivative kernel function is embedded in the position encoding convolution layer. By utilizing the physical information of the Raman spectrum, the network learning process is made more consistent with the prior knowledge of spectral analysis, the network convergence is accelerated, and the feature extraction capability is improved. The kernel function can simulate the characteristic peak morphology of the Raman spectrum, guide the network to prioritize key local features such as peak position and half-peak width, make the feature extraction process more in line with the physical properties of the Raman spectrum itself, and make the obtained Raman position encoding features more targeted and representative, which effectively improves the contribution of Raman spectral features in fusion, thereby enhancing the sensitivity of the entire fusion feature to changes in material composition and structure, which is beneficial to improving the performance of the detection system.

[0049] By performing convolution operations of different scales on the near-infrared spectrum through the near-infrared feature extraction module with a parallel multi-branch architecture, local near-infrared features, group-level features, and global features can be obtained simultaneously (high-frequency convolution kernels are used to locate fine vibration peaks, medium-frequency convolution kernels correspond to group-level feature extraction, and low-frequency fully connected layer branches are used to capture global spectral trends). This enables feature extraction of near-infrared spectra at multiple levels, more completely characterizing the material information contained in the near-infrared spectrum, and providing a richer and more comprehensive feature basis for subsequent interactive fusion with Raman features, so that the fused features can better reflect the comprehensive characteristics of the substance.

[0050] After grouping the multi-scale features of Raman and near-infrared by length, a bidirectional cross-attention mechanism is applied to each group to achieve Raman-to-near-infrared attention and near-infrared-to-Raman attention. On the one hand, this fully exploits the complementarity between the two spectral features, allowing each to provide more valuable information to the other. On the other hand, a closer relationship is established through bidirectional interaction, so that the fused bidirectional features contain both the local, fine information of the Raman spectrum and the global, macroscopic information of the near-infrared spectrum. This effectively solves the problem of cross-modal semantic alignment, improves the quality and representational capabilities of the fused features, and helps improve the performance of subsequent tasks such as material classification and identification based on the fused features. However, through the constructed bidirectional cross-attention mechanism, a bidirectional attention mechanism using Raman and near-infrared as query-key values is used to build a bidirectional evidence chain, which enhances the complementarity of the two spectra and also helps to fully integrate the two spectra features.

[0051] By employing a dynamic gating fusion module to further align bidirectional features and using a scoring network to calculate the importance scores of bidirectional features at different scales, the network automatically focuses on valuable scale features and flexibly assigns weights between features at different scales, achieving dynamic fusion of multi-scale bidirectional features. Compared to fixed-weight fusion, this approach automatically adjusts the weights of features at each scale based on the detection scenario of different substances, making the fused features more adaptable and flexible, better highlighting feature information that is important to the current task, and thus improving the fused features' ability to represent various complex samples, further enhancing the generalization performance and detection effectiveness of the entire feature fusion method in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1A schematic flow chart of a method for fusion of Raman and near-infrared spectroscopy features provided in one embodiment of the present invention.

[0054] Figure 2 A schematic diagram of the architecture of a position encoding-residual convolution module provided by one embodiment of the present invention.

[0055] Figure 3 A schematic diagram of the architecture of a near-infrared feature extraction module provided in one embodiment of the present invention.

[0056] Figure 4 A schematic diagram of the architecture of a bidirectional cross-attention module provided in one embodiment of the present invention.

[0057] Figure 5 A schematic diagram of the architecture of a multi-scale gated fusion module provided in one embodiment of the present invention.

[0058] Figure 6 A schematic diagram comparing different experimental methods provided in one embodiment of the present invention.

[0059] Figure 7 A schematic structural diagram of a device for fusing Raman and near-infrared spectroscopy features provided in one embodiment of the present invention.

[0060] The realization of the objectives of the invention, the functional features and advantages will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention for which protection is sought, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0062] The present invention is described in detail below with reference to the embodiments.

[0063] Reference Figure 1 FIG2 is a flow chart of a method for fusion of Raman spectroscopy and near-infrared spectroscopy provided by one embodiment of the present invention.

[0064] In this embodiment, the method includes:

[0065] S11, using a Raman spectrometer and a near-infrared spectrometer to respectively collect spectral signals of the substance to be detected, and obtain corresponding Raman spectra and near-infrared spectra.

[0066] S12, inputting the Raman spectrum into a position encoding-residual convolution module to perform Raman spectrum feature extraction to obtain a plurality of Raman multi-scale features.

[0067] Furthermore, the position encoding-residual convolution module adopts a hierarchical progressive residual convolution architecture, and the position encoding-residual convolution module includes a position encoding convolution layer and multiple downsampling layers; in step S12, the Raman spectrum is input into the position encoding-residual convolution module for Raman spectrum feature extraction to obtain multiple Raman multi-scale features, including:

[0068] S12-1, inputting the Raman spectrum into the position coding convolution layer to obtain Raman position coding features;

[0069] S12-2, sequentially passing the Raman position coding features through the plurality of downsampling layers to obtain Raman mid-scale features and Raman large-scale features, wherein the downsampling layer includes two residual blocks and a shortcut connection, and the residual block includes a first convolutional layer, a batch normalization layer, a RELU activation function, and a second convolutional layer connected in series, and a shortcut connection is provided between the input and output of each residual block;

[0070] S12-3, taking the Raman position coding feature, the Raman mesoscale feature and the Raman large-scale feature as a plurality of the Raman multi-scale features.

[0071] Furthermore, in step S12-1, inputting the Raman spectrum into the position encoding convolution layer includes:

[0072] A pre-trained Gaussian second-order derivative kernel function is embedded in the position coding convolution layer, and a convolution operation is performed on the Raman spectrum through the Gaussian second-order derivative kernel function to obtain the Raman position coding feature.

[0073] In this embodiment, the method is implemented based on a spectral feature fusion network, which is used as a pre-feature extraction fusion network for a material analysis / detection model using Raman and near-infrared dual spectroscopy. Furthermore, the network architecture of the spectral feature fusion network includes a position encoding-residual convolution module, a near-infrared feature extraction module, a bidirectional cross-attention module and a multi-scale gated fusion module; wherein the position encoding-residual convolution module is used to extract Raman features, with the input being the Raman spectrum and the output being the Raman multi-scale feature (for example, outputting 3 scale features with feature lengths of 256, 128, and 64, respectively); the near-infrared multi-scale feature extraction module is used to extract near-infrared spectral features, with the input being the near-infrared (NIR) spectrum and the output being near-infrared multi-scale features (for example, outputting 3 scale features with feature lengths of 256, 128, and 64, respectively); the bidirectional cross-attention module is used for cross-extraction and interaction of spectral features, with the input being the multi-scale features output by the aforementioned two feature extraction modules and the output being bidirectional features (for example, outputting 3 bidirectional features corresponding to feature groups of lengths of 256, 128, and 64, respectively); the dynamic gated fusion module is used to fuse the aforementioned bidirectional features to obtain the final fusion feature.

[0074] Reference Figure 2 As shown in Figure 1, the position encoding-residual convolution module adopts a hierarchical residual convolution architecture to achieve multi-scale feature extraction of Raman spectra from small scale to large scale. Specifically, a trainable Gaussian second-order derivative kernel is embedded in the first convolution layer. ,

[0075] ;

[0076] in, represents the wave number coordinate, It is the standard deviation parameter of the Gaussian distribution, controls the width of the kernel function, and directly affects the resolution of peak detection. It can take values of 3-6.

[0077] That is, adopt The convolution kernel is used to perform convolution operations on the Raman spectrum. This kernel function simulates the characteristic peak morphology, forcing the network to prioritize key local features such as peak position and half-width, extracting the Raman position encoding features of the spectrum while suppressing fluorescence background interference through residual connections. Then, after passing through a downsampling layer consisting of two residual blocks and shortcut connections, Raman mid-scale features and Raman large-scale features are obtained. Among them, the residual block is composed of a first convolutional layer, a batch normalization layer, a RELU activation function, and a second convolutional layer in series, and each residual block also has a shortcut connection between the input and output. In a specific implementation, three Raman features of different scales are generated through this position encoding-residual convolution module, including Raman position encoding features of 256 length, Raman mid-scale features of 128 length, and Raman large-scale features of 64 length. In particular, in other implementations, features of different scales and numbers of layers can be used. The obtained Raman position coding features, Raman mid-scale features and Raman large-scale features are also adaptively pooled so that the output feature lengths are fixed to 256, 128, and 64 respectively for subsequent feature fusion.

[0078] S13, inputting the near-infrared spectrum into a near-infrared feature extraction module to extract near-infrared spectrum features to obtain multiple near-infrared multi-scale features.

[0079] Furthermore, the near-infrared feature extraction module adopts a parallel multi-branch architecture, and the near-infrared feature extraction module includes a high-frequency convolution layer branch, a medium-frequency convolution layer branch, and a low-frequency convolution layer branch; in step S13, the near-infrared spectrum is input into the near-infrared feature extraction module for near-infrared spectrum feature extraction to obtain multiple Raman multi-scale features, including:

[0080] S13-1, performing a convolution operation on the near-infrared spectrum through the high-frequency convolution layer branch to obtain near-infrared local features;

[0081] S13-2, performing a convolution operation on the near-infrared spectrum through the intermediate-frequency convolution layer branch to obtain near-infrared group-level features, wherein the convolution kernel of the high-frequency convolution layer branch is smaller than the convolution kernel of the intermediate-frequency convolution layer branch;

[0082] S13-3, performing a convolution operation on the near-infrared spectrum through the low-frequency convolution layer branch to obtain a near-infrared global feature, wherein the low-frequency convolution layer branch includes a plurality of fully connected layers;

[0083] S13-4, using the near-infrared local features, the near-infrared group-level features, and the near-infrared global features as a plurality of the Raman multi-scale features.

[0084] In this embodiment, the near-infrared feature extraction module extracts complementary features of the near-infrared spectrum through a parallel multi-branch architecture and achieves semantic alignment with the Raman multi-scale features. Figure 3 As shown in the figure, this module uses three parallel branches to extract multi-scale features of the near-infrared spectrum. Specifically, the fully connected layer path (low-frequency convolution layer branch) is composed of three fully connected layers in series, which reduces the dimension of the near-infrared spectrum from the original length in the order of 1024, 256, and 64, and directly maps the original spectrum to the low-dimensional space to obtain a near-infrared large-scale feature vector with a length of 64 (that is, for the large-scale feature level, a fully connected network is used to directly encode the global spectral trend, capture macroscopic characteristics such as baseline slope and overall energy distribution, and obtain near-infrared global features); the sub-band divided intermediate frequency path (intermediate frequency convolution layer branch) uses a wide convolution kernel with a width of 200 to perform a convolution operation on the original spectrum (kernel_size=200, stride =100), the near-infrared spectrum is divided into several sub-bands of 200 lengths, and each sub-band is independently extracted and aligned to 128 dimensions through adaptive pooling, which can characterize the characteristics of the near-infrared spectral group region and obtain the near-infrared group-level feature vector (that is, for the sub-band level, a wide convolution kernel is used to divide the original spectrum into sub-bands, each sub-band is independently extracted with features, and significant responses are retained through maximum pooling); the high-frequency convolution path (high-frequency convolution layer branch) uses a smaller width convolution kernel to perform a convolution operation on the original spectrum (kernel_size=20, stride=10), locates fine vibration peaks, and fixes the output near-infrared local features to a length of 256 through adaptive pooling (that is, a small convolution kernel is used to capture details, such as the slight shift of the CH stretching peak).

[0085] S14, inputting the Raman multi-scale features and the near-infrared multi-scale features into a bidirectional cross attention module for feature interaction to obtain a plurality of bidirectional features of different scales.

[0086] Furthermore, in step S14, the Raman multi-scale features and the near-infrared multi-scale features are input into a bidirectional cross attention module for feature interaction to obtain a plurality of bidirectional features of different scales, including:

[0087] S14-1, selecting the Raman multi-scale features and the near-infrared multi-scale features of the same length as a group, to obtain multiple groups of same-scale features;

[0088] S14-2, performing a bidirectional cross attention mechanism on each group of the same-scale features, including Raman-to-near-infrared attention and near-infrared-to-Raman attention, to obtain Raman-to-near-infrared features and near-infrared-to-Raman features;

[0089] S14-3, splicing the Raman to near-infrared features and the near-infrared to Raman features of the same group to obtain a plurality of the bidirectional features.

[0090] Furthermore, in step S14-2, each group of the same-scale features is subjected to a bidirectional cross-attention mechanism including Raman-to-near-infrared attention and near-infrared-to-Raman attention to obtain Raman-to-near-infrared features and near-infrared-to-Raman features, including:

[0091] S14-2-1, in the Raman-to-NIR attention, generating a Raman query vector according to the Raman multi-scale features, and generating a near-infrared key vector and a near-infrared value vector according to the near-infrared multi-scale features;

[0092] S14-2-2, calculating the similarity between the Raman query vector and the near-infrared key vector, and obtaining the Raman-to-near-infrared attention weight by scaling the dot product;

[0093] S14-2-3, multiplying the Raman near-infrared attention weight by the near-infrared value vector to obtain the Raman near-infrared feature;

[0094] S14-2-4, in the near-infrared Raman attention, generating a near-infrared query vector according to the near-infrared multi-scale features, and generating a Raman key vector and a Raman value vector according to the Raman multi-scale features;

[0095] S14-2-5, calculating the similarity between the near-infrared query vector and the Raman key vector, and obtaining the near-infrared Raman attention weight by scaling the dot product;

[0096] S14-2-6, multiplying the near-infrared Raman attention weight by the Raman value vector to obtain the near-infrared Raman feature.

[0097] In this embodiment, a deep interaction between Raman and near-infrared features is achieved through a bidirectional cross-attention mechanism. The bidirectional cross-attention mechanism first generates corresponding query vectors, key vectors, and value vectors using Raman and near-infrared features. A Raman-NIR bidirectional attention search is then performed, including: Raman → NIR search: Based on the Raman query vector, the association pattern between the near-infrared key vector and value vector is retrieved; Near-infrared → Raman search: Based on the near-infrared query vector, the corresponding evidence chain in the Raman feature is located. Finally, the dual-spectral features are spliced to obtain a bidirectional feature.

[0098] Specifically, for the Raman multi-scale features and near-infrared multi-scale features (such as the three groups in the above embodiment) output by the position encoding-residual convolution module and the near-infrared multi-scale feature extraction module, Raman multi-scale features and near-infrared multi-scale features of the same length are alternately selected as a group (such as a group of 256 with the same length) and processed by a bidirectional cross-attention mechanism. The bidirectional cross-attention mechanism includes the steps of attention from Raman to near-infrared and attention from near-infrared to Raman, as well as the final feature combination.

[0099] For Raman to NIR attention, first generate the query vector Q based on the Raman multi-scale features R , and generate a key vector K based on near-infrared multi-scale features N With the value vector V N ;in,

[0100] ;

[0101] ;

[0102] ;

[0103] Where, F Raman and F NIR Corresponding to Raman multi-scale features and near-infrared multi-scale features, 、 、 etc. represent the learnable weight vectors;

[0104] The Raman query vector Q is then calculated by the dot product operation R With the near-infrared bond vector K N The similarity of Raman to near infrared attention weight Attn is obtained by scaling the dot product R ;Right now,

[0105] ;

[0106] Where d is the characteristic length (such as 256, 128, 64), T represents the vector transpose operation, yes The transposed form of the vector. The softmax function is a common activation function used for vector normalization of similarity.

[0107] Finally, the Raman to near-infrared attention weight Attn R and the near infrared value vector V N Multiply them together to get the Raman near-infrared feature F RtoN ;Right now, .

[0108] Similarly, for the near-infrared Raman attention, we first generate the query vector Q based on the near-infrared multi-scale features. N , and generate the key vector K based on the Raman multi-scale features R With the value vector V R ;in,

[0109] ;

[0110] ;

[0111] ;

[0112] Where, F Raman and F NIR Corresponding to Raman multi-scale features and near-infrared multi-scale features, 、 、 etc. represent the learnable weight vectors;

[0113] Then calculate the near infrared query vector Q N and the Raman bond vector K R The similarity is measured by scaling the dot product as the Raman to NIR attention weight Attn N ;

[0114] ;

[0115] Among them, d is the feature length, and the softmax function is a common activation function used for vector normalization of similarity;

[0116] Finally, the near infrared to Raman attention weight Attn N and the Raman value vector V R Multiply them together to get the near-infrared Raman feature F NtoR ;Right now, .

[0117] For a set of Raman-NIR multi-scale features, the above F RtoN and F NtoR Then, the two are spliced row by row to obtain the corresponding bidirectional feature F fused That is, F fused =[F R ; F N ]=[F RtoN ; F NtoR ].

[0118] The above is the processing process of a set of Raman-NIR multi-scale features through the bidirectional cross attention fusion module. Each set of Raman-NIR multi-scale features passes through this module to obtain a corresponding bidirectional feature F fusedFinally, several bidirectional features are obtained (for example, in the above embodiment, three groups of Raman-NIR multi-scale features with lengths of 256, 128, and 64 respectively obtain the corresponding F fused , that is, a total of 3 F fused :F fused1 、F fused2 、F fused3 ).

[0119] S15, fusing the bidirectional features of multiple scales through a multi-scale gated fusion module to obtain a fused feature.

[0120] Furthermore, in step S15, the bidirectional features of multiple scales are fused by a multi-scale gated fusion module to obtain a fused feature, including:

[0121] S15-1, assigning an importance score to the bidirectional feature at each scale through a scoring network, wherein the scoring network includes a multi-level fully connected layer;

[0122] S15-2, using different fully connected layers to project the bidirectional features of multiple different scales to the same length, and performing element-by-element weighted summation of the projected bidirectional features according to the corresponding importance scores to obtain the fused features.

[0123] In this embodiment, a multi-scale gated fusion module is used to process the obtained bidirectional features of different scales, and the processing process includes importance score calculation and weighted fusion. fused1 、F fused2 、F fused3 Due to the concatenation of Raman and near-infrared bidirectional features, the data length is doubled (from 256, 128, and 64 to 512, 256, and 128, respectively). First, an importance score is independently assigned to each scale of the bidirectional features. Then, a fully connected layer is used to unify the lengths of the bidirectional features at different scales. Finally, the weighted output is combined to obtain the final fused feature.

[0124] Reference Figure 4 As shown, the bidirectional features output by the bidirectional cross-attention module are first concatenated along the feature dimension into a joint vector (for this embodiment, the total length after concatenation is 512+256+128=896), and the joint vector is input into a scoring network consisting of two levels of fully connected layers (the hierarchical dimensions in this embodiment are 896→256→3, respectively). In the first fully connected layer, the input dimension is consistent with the concatenated feature dimension (896→256 in this embodiment); the second fully connected layer has an output dimension that matches the number of feature groups (256→3 in this embodiment); then weight normalization is performed, and the importance score W is generated using the Softmax function.k ,Right now,

[0125] W k =Softmax(MLP(F fused k ));

[0126] Among them, MLP means inputting the vector into the fully connected layer defined above for calculation, softmax is the activation function, F fused k represents the kth bidirectional feature.

[0127] Then, weighted fusion is performed: first, an independent linear layer is set for each bidirectional feature to achieve length alignment of several bidirectional features, that is, three independent linear layers are used, with the original length of the bidirectional feature as the input length and the target length as the output length, and the three bidirectional features are projected to the same length (in the embodiment, three linear layers of 512→256, 256→256, and 128→256 are used respectively to unify the three bidirectional features of different scales to a length of 256). Finally, the importance score w is calculated. k Perform element-by-element weighted summation on the projected bidirectional features to obtain the final output fusion feature F with unified dimension final (The embodiment outputs 256 dimensions); that is,

[0128] ;

[0129] in, 、 、 Refers to the bidirectional features with the same length after projection, w k refers to the corresponding importance score.

[0130] The following applies the above-mentioned spectral feature fusion network to the tea scene. A dataset is established using the existing Pu'er tea Raman-NIR database. Each origin contains 100 near-infrared spectral data and Raman spectral data, as shown in Table 1 below:

[0131] Table 1

[0132]

[0133] A residual network with a similar number of layers as in the above embodiment was used for single spectrum feature extraction, and a fully connected layer was used as the classification network. Three schemes, namely, single Raman spectroscopy, single near-infrared spectroscopy, and simple feature fusion, were used as control groups for the origin classification task. The experimental scheme and results are shown in Figure 2. Figure 5 As shown in Table 2 below:

[0134] Table 2

[0135]

[0136] Based on the above, the accuracy of this solution (93.8%) significantly exceeds that of single-spectrum models (67.2% for Raman and 75.7% for near-infrared) and simple feature splicing methods (83.5%). Compared with traditional methods, this solution has better feature extraction and fusion effects, fully integrating the characteristics of Raman and near-infrared spectra, providing a reliable foundation for subsequent analysis and detection tasks.

[0137] Reference Figure 7 FIG2 is a schematic structural diagram of a device for fusing Raman spectroscopy and near-infrared spectroscopy provided by one embodiment of the present invention.

[0138] In this embodiment, the device 20 includes:

[0139] The spectrum acquisition unit 21 is used to respectively acquire the spectrum signals of the substance to be detected using a Raman spectrometer and a near-infrared spectrometer to obtain the corresponding Raman spectrum and near-infrared spectrum;

[0140] A first extraction unit 22 is configured to input the Raman spectrum into a position encoding-residual convolution module to extract Raman spectrum features and obtain a plurality of Raman multi-scale features;

[0141] A second extraction unit 23 is configured to input the near-infrared spectrum into a near-infrared feature extraction module to extract near-infrared spectrum features and obtain a plurality of near-infrared multi-scale features;

[0142] A feature interaction unit 24 is configured to input the Raman multi-scale feature and the near-infrared multi-scale feature into a bidirectional cross attention module for feature interaction to obtain a plurality of bidirectional features at different scales;

[0143] The feature fusion unit 25 is configured to fuse the bidirectional features of multiple scales through a multi-scale gated fusion module to obtain a fused feature.

[0144] Each unit module of the device 20 can respectively execute the corresponding steps in the above method embodiment, so each unit module will not be described in detail here. Please refer to the description of the corresponding steps above for details.

[0145] The embodiment of the present invention further provides a device for fusion of Raman spectrum and near infrared spectrum, the device comprising the above-mentioned device for fusion of Raman spectrum and near infrared spectrum, wherein the device for fusion of Raman spectrum and near infrared spectrum can be used. Figure 7 The structure of the embodiment can be executed accordingly. Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects. For details, please refer to the relevant records in the above embodiments and will not be repeated here.

[0146] The device includes: a mobile phone, digital camera, tablet computer, or other device with a camera function, or a device with an image processing function, or a device with an image display function. The device may include components such as a memory, a processor, an input unit, a display unit, and a power supply.

[0147] Among them, the memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as an image playback function, etc.), etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor and the input unit with access to the memory.

[0148] The input unit can be used to receive input digital, character, or image information, and generate keyboard, mouse, joystick, optical, or trackball signal input related to user settings and function control. Specifically, the input unit of this embodiment can include not only a camera, but also a touch-sensitive surface (such as a touch display) and other input devices.

[0149] The display unit can be used to display information input by the user or information provided to the user and various graphical user interfaces of the device, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit may include a display panel. Optionally, the display panel can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), etc. Furthermore, the touch-sensitive surface can cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor to determine the type of touch event. The processor then provides a corresponding visual output on the display panel based on the type of touch event.

[0150] The embodiment of the present invention further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement Figure 1 The computer readable storage medium may be a read-only memory, a magnetic disk or an optical disk.

[0151] The embodiment of the present invention further provides a computer program product, including a computer program / instruction, which is loaded and executed by a processor to implement Figure 1 A feature fusion method of Raman spectroscopy and near-infrared spectroscopy is shown.

[0152] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. For similar or identical parts between the various embodiments, reference can be made to each other. For the apparatus embodiments, device embodiments, and storage medium embodiments, since they are generally similar to the method embodiments, their descriptions are relatively simple. For relevant parts, reference can be made to the descriptions of the method embodiments.

[0153] Furthermore, in this document, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0154] While the foregoing description shows and describes preferred embodiments of the present invention, it should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments, and can be modified within the scope of the present invention by the teachings herein or by techniques or knowledge in the relevant art. Modifications and variations made by those skilled in the art without departing from the spirit and scope of the present invention are intended to be within the scope of the appended claims.

Claims

1. A method for fusion of Raman spectroscopy and near-infrared spectroscopy, characterized in that: The method comprises: The spectral signals of the substance to be detected are collected by a Raman spectrometer and a near-infrared spectrometer respectively to obtain the corresponding Raman spectrum and near-infrared spectrum; Inputting the Raman spectrum into a position encoding-residual convolution module to extract Raman spectrum features to obtain multiple Raman multi-scale features; Inputting the near-infrared spectrum into a near-infrared feature extraction module to extract near-infrared spectrum features to obtain multiple near-infrared multi-scale features; Inputting the Raman multi-scale features and the near-infrared multi-scale features into a bidirectional cross attention module for feature interaction to obtain a plurality of bidirectional features of different scales; The Raman multi-scale features and the near-infrared multi-scale features are input into a bidirectional cross attention module for feature interaction to obtain a plurality of bidirectional features of different scales, including: Selecting the Raman multi-scale features and the near-infrared multi-scale features with the same length as a group to obtain multiple groups of same-scale features; Processing each group of the same-scale features with a bidirectional cross-attention mechanism including Raman to near-infrared attention and near-infrared to Raman attention to obtain Raman to near-infrared features and near-infrared to Raman features; splicing the Raman-to-near-infrared features and the near-infrared-to-Raman features of the same group to obtain a plurality of bidirectional features; The process of processing each group of the same-scale features by a bidirectional cross-attention mechanism including Raman to near-infrared attention and near-infrared to Raman attention to obtain Raman to near-infrared features and near-infrared to Raman features includes: In the Raman to NIR attention, a Raman query vector is generated according to the Raman multi-scale feature, and a NIR key vector and a NIR value vector are generated according to the NIR multi-scale feature; Calculating the similarity between the Raman query vector and the near-infrared key vector, and obtaining the Raman-to-near-infrared attention weight by scaling the dot product; Multiplying the Raman-to-near-infrared attention weight by the near-infrared value vector to obtain the Raman-to-near-infrared feature; In the near-infrared Raman attention, a near-infrared query vector is generated according to the near-infrared multi-scale features, and a Raman key vector and a Raman value vector are generated according to the Raman multi-scale features; Calculating the similarity between the near-infrared query vector and the Raman key vector, and obtaining the near-infrared Raman attention weight by scaling the dot product; Multiplying the near-infrared Raman attention weight by the Raman value vector to obtain the near-infrared Raman feature; The bidirectional features of multiple scales are fused through a multi-scale gated fusion module to obtain a fused feature.

2. The method for fusion of Raman spectroscopy and near-infrared spectroscopy according to claim 1, characterized in that: The position coding-residual convolution module adopts a hierarchical residual convolution architecture, and the position coding-residual convolution module includes a position coding convolution layer and multiple downsampling layers; the Raman spectrum is input into the position coding-residual convolution module to extract Raman spectrum features, and multiple Raman multi-scale features are obtained, including: Inputting the Raman spectrum into the position coding convolution layer to obtain Raman position coding features; Passing the Raman position coding features sequentially through a plurality of downsampling layers to obtain Raman mid-scale features and Raman large-scale features, wherein the downsampling layer includes two residual blocks and a shortcut connection, and the residual block includes a first convolutional layer, a batch normalization layer, a RELU activation function, and a second convolutional layer connected in series, and a shortcut connection is provided between the input and output of each residual block; The Raman position coding feature, the Raman mesoscale feature and the Raman large-scale feature are used as a plurality of the Raman multi-scale features.

3. The method for fusion of Raman spectroscopy and near-infrared spectroscopy according to claim 2, characterized in that: Inputting the Raman spectrum into the position encoding convolutional layer comprises: A pre-trained Gaussian second-order derivative kernel function is embedded in the position coding convolution layer, and a convolution operation is performed on the Raman spectrum through the Gaussian second-order derivative kernel function to obtain the Raman position coding feature.

4. The method for fusion of Raman spectroscopy and near-infrared spectroscopy according to claim 1, characterized in that: The near-infrared feature extraction module adopts a parallel multi-branch architecture, and the near-infrared feature extraction module includes a high-frequency convolution layer branch, a medium-frequency convolution layer branch, and a low-frequency convolution layer branch; the near-infrared spectrum is input into the near-infrared feature extraction module for near-infrared spectrum feature extraction to obtain multiple Raman multi-scale features, including: Performing a convolution operation on the near-infrared spectrum through the high-frequency convolution layer branch to obtain near-infrared local features; Performing a convolution operation on the near-infrared spectrum through the intermediate-frequency convolution layer branch to obtain near-infrared group-level features, wherein the convolution kernel of the high-frequency convolution layer branch is smaller than the convolution kernel of the intermediate-frequency convolution layer branch; Performing a convolution operation on the near-infrared spectrum through the low-frequency convolution layer branch to obtain a near-infrared global feature, wherein the low-frequency convolution layer branch includes a plurality of fully connected layers; The near-infrared local features, the near-infrared group-level features, and the near-infrared global features are used as a plurality of the Raman multi-scale features.

5. The method for fusion of Raman spectroscopy and near-infrared spectroscopy according to claim 1, characterized in that: The multi-scale gated fusion module is used to fuse the bidirectional features of multiple scales to obtain fused features, including: Assigning an importance score to the bidirectional features at each scale through a scoring network, wherein the scoring network includes multiple levels of fully connected layers; Different fully connected layers are used to project the bidirectional features of multiple different scales to the same length respectively, and the projected bidirectional features are weightedly summed element by element according to the corresponding importance scores to obtain the fused features.

6. A device for fusion of Raman spectroscopy and near-infrared spectroscopy, characterized in that: The device comprises: A spectrum acquisition unit is used to acquire spectral signals of the substance to be detected using a Raman spectrometer and a near-infrared spectrometer, respectively, to obtain corresponding Raman spectra and near-infrared spectra; A first extraction unit is configured to input the Raman spectrum into a position encoding-residual convolution module to extract Raman spectrum features to obtain a plurality of Raman multi-scale features; A second extraction unit is used to input the near-infrared spectrum into a near-infrared feature extraction module to extract near-infrared spectrum features to obtain multiple near-infrared multi-scale features; A feature interaction unit is used to input the Raman multi-scale features and the near-infrared multi-scale features into a bidirectional cross attention module for feature interaction to obtain a plurality of bidirectional features of different scales; wherein the feature interaction unit is further used to: Selecting the Raman multi-scale features and the near-infrared multi-scale features with the same length as a group to obtain multiple groups of same-scale features; Processing each group of the same-scale features with a bidirectional cross-attention mechanism including Raman to near-infrared attention and near-infrared to Raman attention to obtain Raman to near-infrared features and near-infrared to Raman features; splicing the Raman-to-near-infrared features and the near-infrared-to-Raman features of the same group to obtain a plurality of bidirectional features; The process of processing each group of the same-scale features by a bidirectional cross-attention mechanism including Raman to near-infrared attention and near-infrared to Raman attention to obtain Raman to near-infrared features and near-infrared to Raman features includes: In the Raman to NIR attention, a Raman query vector is generated according to the Raman multi-scale feature, and a NIR key vector and a NIR value vector are generated according to the NIR multi-scale feature; Calculating the similarity between the Raman query vector and the near-infrared key vector, and obtaining the Raman-to-near-infrared attention weight by scaling the dot product; Multiplying the Raman-to-near-infrared attention weight by the near-infrared value vector to obtain the Raman-to-near-infrared feature; In the near-infrared Raman attention, a near-infrared query vector is generated according to the near-infrared multi-scale features, and a Raman key vector and a Raman value vector are generated according to the Raman multi-scale features; Calculating the similarity between the near-infrared query vector and the Raman key vector, and obtaining the near-infrared Raman attention weight by scaling the dot product; Multiplying the near-infrared Raman attention weight by the Raman value vector to obtain the near-infrared Raman feature; The feature fusion unit is used to fuse the bidirectional features of multiple scales through a multi-scale gated fusion module to obtain a fused feature.

7. A device for fusion of Raman spectroscopy and near-infrared spectroscopy, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of the method for characteristic fusion of Raman spectrum and near-infrared spectrum according to any one of claims 1 to 5.

8. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of a method for fusion of Raman spectroscopy and near-infrared spectroscopy as claimed in any one of claims 1 to 5.

Images