Methods, systems, equipment, and media for high-dimensional continuous spectral data analysis based on segment fusion Transformer models.

By using a segment fusion Transformer model, combined with a CNN network and a self-attention mechanism, the problem of feature extraction and analysis of high-dimensional continuous spectral data is solved, achieving efficient and accurate spectral classification, which is applicable to a variety of spectral techniques.

CN118013363BActive Publication Date: 2026-11-17XIDIAN UNIV
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202410161629.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-05
Publication Date
2026-11-17
Estimated Expiration
2044-02-05

AI Technical Summary

Technical Problem

Existing hyperspectral image analysis methods and Raman spectral data analysis methods suffer from insufficient feature extraction capabilities, high computational complexity, low signal-to-noise ratio, and weak model interpretability when processing high-dimensional continuous spectral data. Furthermore, the traditional Transformer model has not been effectively applied to spectral data.

Method used

A segment-fusion-based Transformer model is adopted, which extracts intra-segment features through segmented Transformer sub-modules and combines inter-segment information with a pyramid architecture to fuse them layer by layer. A method suitable for high-dimensional continuous spectral data analysis is designed, and spectral features are extracted using CNN network and self-attention mechanism.

Benefits of technology

It reduces the computational complexity of spectral data analysis, improves the accuracy and interpretability of spectral feature information analysis, achieves efficient spectral classification, and is applicable to various spectral techniques such as Raman spectroscopy, surface-enhanced Raman scattering spectroscopy, and infrared spectroscopy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118013363B_ABST
    Figure CN118013363B_ABST
Patent Text Reader

Abstract

A method, system, device, and medium for high-dimensional continuous spectral data analysis based on a segment fusion Transformer model are described. The method involves first establishing a segmented Transformer submodule for intra-segment feature extraction; then, combining the processed spectral features with a pyramid architecture, fusing the segmented information layer by layer, and setting up a segment fusion submodule for segment fusion; finally, inputting the spectral features into a classifier to output spectral categories. The system, device, and medium utilize this method to combine segmented spectral features with a Transformer, fusing convolutional neural networks and two-dimensional attention to construct an efficient segmented Transformer module for accurate intra-segment feature analysis. The segment fusion Transformer model, through a pyramid design architecture, combines inter-segment information layer by layer, combining intrinsic spectral features with the Transformer to solve the problem of discretization in high-dimensional continuous spectral data. This invention reduces the computational complexity of spectral data, improves the accuracy of spectral intrinsic feature information analysis, and is applicable to Raman spectroscopy, surface-enhanced Raman scattering spectroscopy, infrared spectroscopy, and hyperspectroscopy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of spectral analysis technology, specifically to a method, system, device, and medium for high-dimensional continuous spectral data analysis based on a segment fusion Transformer model. Background Technology

[0002] Traditional hyperspectral image analysis primarily relies on machine learning, with the main steps being feature extraction and classifier training. Feature extraction methods include random minimum generation forests and local covariance matrix representation. Classifier training methods include linear discriminant analysis and support vector machines. However, these machine learning-based hyperspectral image classification methods lack the ability to extract spatial information features. In recent years, deep learning technology has demonstrated powerful discriminative and learning capabilities in hyperspectral image analysis tasks. Convolutional Neural Networks (CNNs), in particular, possess characteristics of local perception and parameter sharing, while preserving the original structure and significantly reducing the number of network parameters during feature learning. However, hyperspectral images have numerous bands with high redundancy, and the importance of each band and each spatial feature varies, leading to increased model complexity and training difficulties.

[0003] Currently, Raman spectroscopy is widely used in the biological field to analyze the composition, properties, and interactions of substances due to its non-invasive measurement capabilities and label-free molecular fingerprinting ability. Raman spectroscopy data typically contains rich intrinsic information about the sample, exhibiting high-dimensional continuity and reflecting the scattering characteristics of substances at continuous wavenumbers. Many chemical components with unique biomolecular information exist in various cell phenotypes, enabling the identification and analysis of individual cells at the single-cell level. However, cells share similar chemical compositions and biomolecular structures, particularly low Raman scattering cross-sections, leading to subtle spectral differences, low signal-to-noise ratios, and Raman peak overlap. Therefore, an efficient data analysis model is needed for single-cell Raman spectroscopy data.

[0004] Continuous sequence discretization is an important data analysis method in artificial intelligence. However, previous works have directly used various machine learning and deep learning algorithms for Raman spectroscopy analysis, focusing more on the texture, shape, and peaks of the spectrum while neglecting the discretization process of continuous spectral data. Therefore, some spectral features are discarded, and the structural information of the spectrum is disrupted, resulting in limited ability to extract the intrinsic spectral information contained within the spectrum itself. Hu Qingqing et al. proposed a novel strategy for continuous Raman spectroscopy analysis. Based on the biological characteristics of cells, they divided the single-cell Raman spectrum into five discrete regions: the fingerprint region, the protein I region, the mixing region, the protein II region, and the genetic material region. Based on the important feature contributions of biomolecules in each region, they proposed a method for segmented weighted similarity discretization of the spectrum, demonstrating the advantages of high accuracy and low bias in single-cell Raman spectroscopy. Although this strategy combines continuous spectral sequence discretization with machine learning algorithms to extract spectral features, especially intrinsic features, significant challenges remain in the discretization of continuous spectra.

[0005] Transformer models, based on self-attention mechanisms, have recently demonstrated outstanding performance in a wide range of language and vision tasks, such as text classification, machine translation, image recognition, object detection, text-to-image synthesis, and question answering. Designed for modeling and transforming discrete sequences, Transformers are renowned for their use of attention to model long-range dependencies in data. A revolutionary deep learning architecture, Transformers are famous for their key features, including self-attention, encoder-decoder architecture, and multi-head attention. They have not only revolutionized the way sequence data is processed but have also found widespread application in natural language processing, computer vision, speech processing, and other fields. Pre-trained Transformer models such as BERT and GPT have become standard in NLP tasks, and their modular structure and flexibility have led to significant success in multiple application areas, driving the development of deep learning.

[0006] Text sequence data consists of discrete symbols with clear boundaries and a well-defined structure, such as words forming sentences, sentences forming paragraphs, and paragraphs forming documents. Spectral data represents a series of continuous values, characterized by high dimensionality, which distinguishes it from discrete text sequences in some ways. Furthermore, spectral data consists of continuous numerical values ​​without clear boundaries, lacking structure and a well-defined hierarchical organization. Transformer technology has several drawbacks in practical spectral applications, such as high computational complexity due to the high dimensionality of spectral data sequences, low signal-to-noise ratio due to high noise in the original data, the diverse and complex biochemical substances reflected in spectra, difficulty in classifying categories, and weak model interpretability. The structural differences between spectral and text sequence data make integrating Transformer with continuous spectral sequences and designing models specifically tailored for spectral data a challenging task. However, no Transformer model specifically designed for the characteristics of spectral data has yet emerged in the field of spectral analysis. Therefore, designing a Transformer model specifically for the characteristics of spectral data is urgently needed to fill the gap in Transformer applications in spectral analysis. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, this invention aims to propose a method, system, device, and medium for high-dimensional continuous spectral data analysis based on a segmented Transformer model. The method involves discretizing the continuous spectrum of a substance according to its inherent characteristics, including its composition and structural properties. Through two stages—extracting features within each segment and fusing features between segments—a Transformer model suitable for analyzing continuous high-dimensional spectra is constructed. This model combines the intrinsic spectral feature segmentation with the Transformer, and incorporates a convolutional neural network and a two-dimensional attention module to construct a segmented Transformer module for analyzing features within each segment. This is achieved through a pyramid... The design architecture combines inter-segment information layer by layer to improve the model's perception domain and fully exploit the long-distance dependencies of spectral features. The segment fusion Transformer model combines intrinsic spectral features with the Transformer to solve the problem of discretization of high-dimensional continuous spectral data, becoming a general framework in the field of high-dimensional continuous spectral sequence data analysis. This invention reduces the computational complexity of spectral data analysis and improves the accuracy of analyzing intrinsic spectral feature information. This invention has broad applicability and significant application value, especially in the field of spectral analysis. It can be effectively applied to various advanced spectroscopic techniques, including Raman spectroscopy, surface-enhanced Raman scattering spectroscopy, infrared spectroscopy, and hyperspectroscopy.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A method for analyzing high-dimensional continuous spectral data based on a segment fusion Transformer model, specifically including the following steps:

[0010] Step 1: Establish a segmented Transformer submodule to extract features within segments;

[0011] Step 2: Based on the spectral features processed in Step 1, and combined with the pyramid architecture, information from different segments is fused layer by layer. A segment fusion submodule is set up to perform segment fusion.

[0012] Step 3: Input the spectral features processed in Step 2 into the classifier, and the classifier outputs the spectral category.

[0013] The specific method for establishing the segmented Transformer submodule in step 1 to extract intra-segment features of spectral features is as follows:

[0014] 1.1 Collect the spectrum of the substance and divide the spectrum into tokens, each token consisting of several data points;

[0015] 1.2 Embed the token into features, project it onto an arbitrary dimension, denoted as C, and convert it into a feature vector;

[0016] 1.3 Based on the intrinsic properties of the substance, including its composition and structural characteristics, the spectrum is divided into n discrete regions, and the spectral features are correspondingly divided into n segments, thereby transforming continuous spectral sequence data into discrete vector data to enhance the expressive power of spectral features.

[0017] 1.4 Combining CNN networks with Transformers to form a segmented Transformer module to retain more information within a segment: key features are extracted from each spectral segment divided in step 1.3, and spatial multi-head self-attention is used to obtain the spectral feature map of each segment. Then, the spectral feature maps of each segment are concatenated together and connected to a dimensional multi-head self-attention module to perform interaction of spectral feature dimensional information and extract spectral feature dimensional information. After the discrete vector data undergoes multi-head self-attention operations on spatial and dimensional information, the CNN network module is used to extract the shape and texture information of the spectrum to obtain the spectral feature vector.

[0018] Step 2 involves fusing the spectral features processed in Step 1, combined with a pyramid architecture, layer by layer, by setting up a segment fusion submodule to perform segment fusion. The specific method is as follows:

[0019] Corresponding to the segmentation of spectral features in step 1, a feature pyramid architecture is adopted to fuse information from different segments layer by layer. The feature pyramid architecture is divided into several layers, each of which consists of an information fusion layer and several Transformer modules. The output of the previous layer is used as the input of the next layer.

[0020] The inter-segment fusion layer consists of an information fusion layer and two Transformer modules. The information fusion layer of the inter-segment fusion layer is used to fuse inter-segment information and generate n-1 new segments. The Transformer module is used to extract features of the new segments. The new segments are input into the Transformer module, where layer normalization is performed first, followed by multi-head self-attention operation and forward propagation. Residual connections are connected between the multi-head self-attention operation and forward propagation to extract features.

[0021] The segment fusion layer consists of an information fusion layer and six Transformer modules. The information fusion layer of the segment fusion layer is used to fuse n-1 new segments input from the first layer to generate (n-1) / 2 new segments. The new segments are input into the Transformer module to perform layer normalization, then multi-head self-attention operation and forward propagation. Residual connections are connected between the multi-head self-attention operation and forward propagation to extract features. This layer can be stacked repeatedly until it is fused into 3 or 2 segments.

[0022] The normalization layer consists of an information fusion layer and two Transformer modules. The information fusion layer of the normalization layer merges the spectral information of the three or two segments input from the segment fusion layer into a new segment. The new segment is then input into the Transformer module, where layer normalization is performed first, followed by spatial multi-head self-attention and dimensional multi-head self-attention operations, and forward propagation. Residual connections are connected between the multi-head self-attention operations and forward propagation for feature extraction. Through this layer-by-layer information fusion, information on the entire spectral feature segment is obtained to comprehensively compare and extract the spectral feature information of the five segments. Finally, the obtained feature information is input into the spectral classifier.

[0023] The method for generating new segments by fusing inter-segment information in the information fusion layer described in step 2 is as follows: divide the spectral feature length by 2, concatenate two adjacent feature vectors together, and perform a linear transformation to obtain a new spectral feature vector.

[0024] The Transformer module described in step 2 is divided into a sliding window Transformer module and a bidirectional visual Transformer module. The inter-segment fusion layer and segment fusion layer of the feature pyramid architecture employ the sliding window Transformer module to extract information from spectral feature segments. The normalization layer uses a bidirectional visual Transformer module with dimensional self-attention and spatial self-attention to extract global spectral features: first, spatial multi-head self-attention is performed, then the spectral feature vector is transposed, and then multi-head self-attention is performed again to obtain the spectral feature vector after dimensional self-attention. Then, the spectral feature vector is transposed back to obtain a spectral feature vector with the same dimension as before the dimensional multi-head self-attention operation. This is followed by a residual connection, and finally input into the forward propagation network of the Transformer module to fuse local features within segments with global spectral features.

[0025] The specific formula for the layer normalization operation is as follows:

[0026]

[0027]

[0028]

[0029]

[0030] Where, x i Let x be the i-th feature of x, x has a total of D features, μ1 is the mean of all features of x, σi is the standard deviation of all features of x, ∈ is a small constant added for numerical stability; γ is a scaling parameter used to adjust the result after normalization; β is a translation parameter used to adjust the result after normalization; ⊙ represents element-wise multiplication.

[0031] The specific formula for the multi-head attention calculation is as follows:

[0032] Q i =XW q

[0033] K i =XW K

[0034] V i =XW V

[0035]

[0036] MultiHead(Q,K,V)=Concact(head1,head2...)W O

[0037] When inputting a two-dimensional tensor X∈R T×d Where T is the sequence length and d is the input dimension; the multi-head attention package contains h attention heads, each with its own weight matrix, and the outputs of these heads are then concatenated together. q ∈R d ×dk W k ∈R d×dk W V ∈R d×dv It is a weight matrix that maps the input to the query, key, and value spaces, respectively. k and d v Each head represents the query, key, and value dimensions; QKV are the query, key, and value matrices, respectively; head i W represents the output of the i-th head. O ∈R T×dv×d It is the weight matrix of the final output projection.

[0038] A high-dimensional continuous spectral data analysis system based on a segment fusion Transformer model includes:

[0039] The segmented Transformer submodule includes an embedding module and a segmented Transformer module. The segmented Transformer module comprises a layer normalization module, a multi-head self-attention module, a forward propagation network, and a convolutional module. Specifically, the embedding module is used in the feature embedding stage of step 1 to convert tokens into feature vectors, providing a richer representation. The layer normalization module acts as a regularizer in steps 1 and 2, mitigating the internal covariate drift problem during deep neural network training. The multi-head attention module is used in the segmented Transformer module of step 1 to enhance strong feature representation and weaken weak feature representation. The forward propagation network is used to extract spectral features in steps 1 and 2. The convolutional module is used in step 1 to extract spectral shape and texture information.

[0040] The segment fusion submodule includes an information fusion layer, a sliding window Transformer module, and a bidirectional vision Transformer module. Both the sliding window Transformer module and the bidirectional vision Transformer module contain a layer normalization module, a multi-head self-attention module, and a forward propagation network. The information fusion layer is used in step 2 to fuse information from different segments. The sliding window Transformer module is used in step 2 to extract information from spectral feature segments. The bidirectional vision Transformer module is used in step 2 to extract information from the entire spectral feature segment. The multi-head attention module is used to enhance strong feature representation and weaken weak feature representation. The forward propagation network is used in steps 1 and 2 to extract spectral features.

[0041] An electronic device for high-dimensional continuous spectral data analysis based on a segment fusion Transformer model includes: a memory and a processor, wherein the memory stores a computer program that causes the processor to execute steps 1 to 3 of the high-dimensional continuous spectral data analysis method described in any one of them.

[0042] A user input program storage medium, wherein the stored computer program, when executed by a processor, is capable of comparing and extracting spectral feature information to obtain spectral classification results based on the high-dimensional continuous spectral data analysis method described in any one of steps 1 to 3.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] This invention addresses the discretization requirements of spectral data by segmenting it according to its inherent characteristics. This transforms the spectral data into a discrete sequence of fragmented data. A segment-level feature extraction module then extracts the effective information from each spectral fragment, converting the data into structured data rich in semantic information and containing abundant intra-segment information. Combining a pyramid architecture, information from different segments is fused layer by layer, resulting in a segment fusion Transformer model. This overcomes the issue of differences in the representation of two types of data, making it more suitable for processing high-dimensional spectral sequence data, improving the accuracy of spectral recognition, and enhancing the representation of spectral features.

[0045] The Transformer model, based on the self-attention mechanism, has recently achieved outstanding performance in language and vision tasks, including text classification, machine translation, image recognition, object detection, text-to-image synthesis, and question answering. The Transformer is a deep learning architecture first proposed by Vaswani et al. in 2017 for processing sequential data, achieving significant success, particularly in Natural Language Processing (NLP). Its core innovation is the self-attention mechanism, and unlike traditional recurrent neural networks (RNNs) and convolutional neural networks (CNNs), the Transformer does not have an explicit sequential processing method; instead, it can consider all positions in the input sequence simultaneously. The self-attention mechanism is the core idea of ​​the Transformer. It allows the model to dynamically assign weights to different positions to better capture dependencies between them when processing sequential data. The self-attention mechanism determines the importance of each position to other positions by calculating attention scores and fusing information from all positions. To enhance the model's ability to model different relationships, the Transformer introduces a multi-head attention mechanism. Each attention head can focus on different positions in the input sequence, thereby capturing different types of relationships and features.

[0046] Traditional Transformer models are primarily designed to process discrete sequence data, such as text sequences in natural language processing tasks. Text sequence data consists of discrete symbols with well-defined boundaries. It typically possesses a well-defined structure, such as words forming sentences, sentences forming paragraphs, and paragraphs forming documents. This structure can be used to extract semantic information and context. However, spectral data is a series of continuous values, characterized by high dimensionality and rich information content, differing from discrete text sequences in some aspects. Furthermore, spectral data consists of continuous numerical values ​​without well-defined boundaries, lacking structure and a clearly defined organizational hierarchy. Therefore, this approach combines Transformer with continuous sequences and designs models specifically for spectral data.

[0047] To address this, this invention proposes a segment fusion Transformer model, which combines the intrinsic features of the spectrum with the Transformer. The spectrum is segmented based on these intrinsic features, and the Transformer is used for feature extraction within each segment and inter-segment fusion. This is a deep learning method suitable for extracting spectral features from high-dimensional spectral sequence data. Simultaneously, the segmented feature extraction module extracts features within each spectral segment, reducing the influence of useless features and enhancing the model's inductive bias capability. The inter-segment fusion strategy fuses features from different segments of the spectrum, extracting effective information from each segment, improving the global feature representation capability of the spectrum, and fusing multi-scale spectral features.

[0048] 1. Segment Fusion Transformer Model: A segment fusion Transformer model was designed and constructed. The overall architecture of the model and the module design of the two stages were carefully designed, and the traditional Transformer structure was optimized. For the extraction of spectral information, the function of spectral classification with high accuracy was achieved.

[0049] 2. Segmented Transformer Submodule: Utilizing prior knowledge of the intrinsic features of the spectrum, this module innovatively combines the bidirectional self-attention mechanism of convolutional neural networks and Transformers to extract features from different regions of the spectrum.

[0050] 3. Segment Fusion Transformer Submodule: In the segment fusion process, through the pyramid architecture design, inter-segment information and segment information are fused, and multi-scale feature information is fused. After one layer of intra-segment feature extraction module and three layers of inter-segment fusion process, five segments of information are fused into one segment of information. Multi-scale spectral feature information is fused to form spectral feature vectors for spectrum classification.

[0051] Compared to existing methods, this invention achieves higher spectral analysis classification accuracy by incorporating the Transformer model, avoids complex spectral preprocessing procedures, meets end-to-end requirements, and is simple and easy to use. This invention demonstrates broad applicability and profound application value, particularly in the field of spectral analysis. It has been proven to effectively integrate with various cutting-edge spectroscopic techniques, covering multiple fields such as Raman spectroscopy, surface-enhanced Raman scattering spectroscopy, infrared spectroscopy, and even hyperspectroscopy. Through the introduction of this invention, these spectroscopic techniques achieve significant improvements in data processing and result interpretation, thereby providing more accurate and reliable analytical tools for scientific research and industrial applications. Attached Figure Description

[0052] Figure 1 This invention integrates the architecture and features of the Transformer model.

[0053] Figure 2 This invention relates to an architecture that integrates the Transformer model and its sub-modules.

[0054] Figure 3 This invention relates to the self-attention mechanism and the multi-head self-attention mechanism.

[0055] Figure 4The following is a visualization of the attention values ​​of cell segments in an embodiment of the present invention, wherein (a) is a heatmap of attention visualization for normal lung cells and breast cells; (b) is a heatmap of attention visualization for four types of breast cancer cells; (c) is a heatmap of attention visualization for four types of lung cancer cells; and (d) is a heatmap of self-attention scores for five segments of ten cell lines.

[0056] Figure 5 This is a heatmap of attention in five segments according to an embodiment of the present invention. The colors from blue to red indicate that the attention weight is higher in the red area.

[0057] Figure 6 The inter-segment fusion variation (ad) of the MCF-10A spectrum in this embodiment of the invention shows the changes in inter-segment fusion and attention in each layer of the network; the dashed line represents the segmentation information based on the intrinsic spectral features, while the solid line represents the changes in spectral fusion; the actual receptive field of the model will change accordingly.

[0058] Figure 7 The following is an example of the inter-segment fusion feature changes of the MCF-10A spectrum in this embodiment of the invention, wherein (a) is the relationship between each layer of the model and inter-segment fusion; (b) is the original spectral image; and (d) is the change of spectral features during the inter-segment fusion process.

[0059] Figure 8 The following is an analysis of the attention values ​​of the segmented Transformer submodule in this embodiment of the invention: (a) the average attention weight of the five segments, (b) the standard deviation of the attention weight of the five segments, (c) the distribution of the relative coefficients of the attention weight of the five segments, (d) the comparison of the intra-class distance of feature vectors exceeding the 0.7 threshold with the original feature vector, (e) the comparison of the intra-class distance of feature vectors below the 0.1 threshold with the original feature vector, where each segment has highlighted parts representing features exceeding the 0.7 threshold and unhighlighted parts representing features below the 0.1 threshold, (f) the attention weight ratio of each segment, and (g) the intra-class and inter-class similarity of the transformed feature vectors.

[0060] Figure 9 The changes in attention weights of each segment during the inter-segment fusion process in this embodiment of the invention are shown in (a) and (b) respectively. (a) shows the variance of the average attention value of each intrinsic segment feature in each layer.

[0061] Figure 10The following diagram illustrates the changes in attention weights for each segment during the inter-segment fusion process in this embodiment of the invention: (a) shows the relationship between layer and segment fusion; (b) shows the changes in contour coefficients of 10 cell features during segment fusion; (c) shows the changes in attention weights for 5 segments during segment fusion; (d) shows the changes in important and unimportant regions; and (e) shows a scatter plot of attention visualization for each layer (from left to right: first layer, second layer, third layer, and fourth layer).

[0062] Figure 11 The following is an analysis of the changes in spectral features caused by inter-segment fusion in an embodiment of the present invention: (a) visualization of the feature vector; (b) information entropy of spectral changes; (c) changes in information gain and inter-segment fusion; (d) changes in inter-segment fusion and CH index; and (e) changes in the silhouette coefficient (SC) during inter-segment fusion.

[0063] Figure 12 The following are experimental results of the model in the embodiments of the present invention: (a) is the classification result of the model; (b) is the ROC curve; and (c) is the visualization of the feature vector. Detailed Implementation

[0064] The present invention will now be described in further detail with reference to the accompanying drawings.

[0065] This invention uses single-cell Raman spectroscopy analysis as an example:

[0066] Combination Figures 1 to 3 The high-dimensional continuous spectral data analysis method based on the segment fusion Transformer model specifically includes the following steps:

[0067] Step 1: Establish a segmented Transformer submodule to extract features within segments;

[0068] 1.1 The single-cell Raman spectrum is divided into tokens, and each token consists of several data points;

[0069] 1.2 Embed the token into features, project it onto an arbitrary dimension, denoted as C, and convert it into a feature vector;

[0070] 1.3 Based on the biological characteristics of single-cell Raman, the single-cell Raman spectrum is divided into discrete regions including fingerprint region, protein I region, mixed region, protein II region and genetic material region, and the single-cell Raman spectral features are divided into 5 segments accordingly, thereby transforming continuous spectral sequence data into discrete vector data to enhance the expressive power of single-cell Raman spectral features.

[0071] 1.4 Combining the CNN network with the Transformer to form a segmented Transformer module preserves more information within each segment: key features are extracted from each spectral segment divided in step 1.3, and spatial multi-head self-attention is used to obtain the spectral feature map of each segment. Then, the spectral feature maps of each segment are concatenated together and connected to a dimensional multi-head self-attention module to perform interaction of spectral feature dimensional information and extract spectral feature dimensional information. After the discrete vector data undergoes multi-head self-attention operations on spatial and dimensional information, the CNN network module is used to extract the shape and texture information of the spectrum to obtain the spectral feature vector.

[0072] Extracting features within segments using the segmented Transformer submodule, see [link / reference]. Figure 2 b. The attention heatmap visually illustrates the importance of each feature in the spectral data. Brighter areas in the attention heatmap indicate areas the model focuses on more. The segmented Transformer submodule has different regions of interest for normal lung cells and breast cells. MRC-5 cells show brighter regions in segment I, while MCF-10A cells have regions of interest in segments II, III, IV, and V. Figure 4 a). T47D cells exhibited focused attention in limited regions of segments I and III, while the other three cell lines in the same series concentrated their attention on segment II. Figure 4 b). This may be attributed to the unique spectral characteristics of T47D cells. A series of spectral regions of interest in the breast cancer cell line showed significant consistency, mainly concentrated on segment II ( Figure 4 c). The model assigns similar levels of attention to cells within the same cell series, with slight variations. However, when processing different cell series, the segment fusion Transformer model shows significant changes in the regions of focus.

[0073] Self-attention score heatmaps describe the correlation between features within a segment through color changes, allowing for the assessment of important and unimportant regions within a segment. Figure 4 Figure d shows self-attention fractional heatmaps for five segments (segments I, II, III, IV, and V) of ten cell lines, namely the fingerprint region, protein I region, mixed region, protein II region, and genetic material region. For segments I and III, i.e., the fingerprint region and the mixed region, the model shows different regions of interest for breast cancer cells and lung cancer cells. In the case of segment I, breast cancer cells show interest in the last region of the segment, specifically the 928 cm⁻¹ representing the protein region. -1 Wavenumber bands. Lung cancer cells are concentrated at the beginning of the band, especially the 625-725 cm⁻¹ region representing genetic material (DNA / RNA). -1Wavenumber segments. For segments II, IV, and V, namely protein I, protein II, and genetic material regions, the model's regions of interest, i.e., high-weight regions, are generally consistent across cell lines but vary in size. This indicates that the intrinsic features of these segments contribute differently to the spectral composition of the cells. Biochemical differences between cells may be small. Visualizing attention heatmaps helps to intuitively understand the basis of model decisions and discover relevant patterns in the spectrum; mapping attention scores to spectral features and reflecting the importance of features through color depth and brightness, ( Figure 5 This provides a visual indication of the model's focus on different features within a segment. Regions of interest also differ within different segments of the same cell line. For example, the final region of segment I corresponds to the biochemical peaks of proteins and is a key region of interest for the model, providing a solid foundation for its interpretability.

[0074] Step 2: Based on the spectral features processed in Step 1, and combined with the pyramid architecture, fuse the information of different segments layer by layer, and set up a segment fusion submodule;

[0075] Corresponding to the segmentation of spectral features in step 1, a feature pyramid architecture is adopted to fuse information from different segments layer by layer. The feature pyramid architecture consists of three layers, each of which is composed of an information fusion layer and several Transformer modules. The output of the previous layer is used as the input of the next layer.

[0076] The first layer consists of an information fusion layer and two Transformer modules. The information fusion layer is used to fuse inter-segment information and generate four new segments. The Transformer modules are used to extract features from the new segments. The new segments are input into the Transformer modules to perform layer normalization, then multi-head self-attention and forward propagation. Residual connections are connected between the multi-head self-attention and forward propagation to extract features.

[0077] The second layer consists of an information fusion layer and two Transformer modules. The information fusion layer of the second layer is used to fuse the information of the four segments input from the first layer to generate two new segments. The new segments are input into the Transformer module to perform layer normalization first, and then perform multi-head self-attention operation and forward propagation. Residual connections are connected between the multi-head self-attention operation and forward propagation to perform feature extraction.

[0078] The third layer consists of an information fusion layer and two Transformer modules. The information fusion layer in the third layer merges the spectral information of the two segments input from the second layer into a new segment. The new segment is then input into the Transformer module, where layer normalization is performed first, followed by spatial multi-head self-attention and dimensional multi-head self-attention operations, and forward propagation. Residual connections are connected between the multi-head self-attention operations and the forward propagation to extract features. Through this layer-by-layer information fusion, information of the entire spectral feature segment is obtained to comprehensively compare and extract the spectral feature information of the five segments. Finally, the obtained feature information is input into the spectral classifier.

[0079] Step 3: Input the spectral features processed in Step 2 into the classifier, and the classifier outputs the spectral category.

[0080] The method for generating new segments by fusing inter-segment information in the information fusion layer described in step 2 is as follows: divide the spectral feature length by 2, concatenate two adjacent feature vectors together, and perform a linear transformation to obtain a new spectral feature vector.

[0081] The Transformer module described in step 2 is divided into a sliding window Transformer module and a bidirectional visual Transformer module. The first and second layers of the feature pyramid architecture use the sliding window Transformer module to extract information from the spectral feature segments. The third layer uses a bidirectional visual Transformer module with dimensional self-attention and spatial self-attention to extract global spectral features: first, spatial multi-head self-attention is performed, then the spectral feature vector is transposed, and then multi-head self-attention is performed again to obtain the spectral feature vector after dimensional self-attention. Then, the spectral feature vector is transposed back to obtain a spectral feature vector with the same dimension as before the dimensional multi-head self-attention operation. After that, a residual connection is connected, and finally, the input is fed into the forward propagation network of the Transformer module to fuse the local features within the segment and the global spectral feature information.

[0082] The specific formula for the layer normalization operation is as follows:

[0083]

[0084]

[0085]

[0086]

[0087] Where, x i μ is the i-th feature of x, and x has a total of D features. l σ is the mean of all features of x.i is the standard deviation of all features of x, ∈ is a small constant added for numerical stability; γ is the scaling parameter (learnable) used to adjust the normalized result; β is the translation parameter (learnable) used to adjust the normalized result; ⊙ represents element-wise multiplication.

[0088] The specific formula for the multi-head attention calculation is as follows:

[0089] Q i =XW q

[0090] K i =XW K

[0091] V i =XW V

[0092]

[0093] MultiHead(Q,K,V)=Concact(head1,head2....)W O

[0094] Here, we assume the input is a two-dimensional tensor X∈R T×d Where T is the sequence length and d is the input dimension; the multi-head attention package contains h attention heads, each with its own weight matrix, and the outputs of these heads are concatenated together; W q ∈R d×dk W k ∈R d×dk W V ∈R d×dv It is a weight matrix that maps the input to the query, key, and value spaces, respectively. k and d v Each head represents the query, key, and value dimensions; QKV are the query, key, and value matrices, respectively; headi represents the output of the i-th head, and W... O ∈R T×dv×d It is the weight matrix of the final output projection.

[0095] During segment fusion, the effectiveness of inter-segment fusion was demonstrated by analyzing the attention values ​​and intra-segment features within each segment. See also Figure 6 a, Figure 6 b、 Figure 6 c. Figure 6d. As fusion progresses, the region of interest within each segment gradually changes. In the first layer, attention is primarily focused on segment II. In the second layer, attention shifts to inter-segment fusion, where solid lines represent new segments based on the spectrum, and dashed lines represent segments based on intrinsic spectral features. The regions of interest within each of the five segments become more pronounced. With the arrival of the third layer, the focus shifts to the fusion across the entire segment hierarchy, and the model's region of interest changes significantly. Segment II is no longer the primary focus because it contains background peaks from the silicon substrate. This further validates the correctness of the evolving region of interest in the model during inter-segment fusion. The gradually increasing region of interest in each segment helps extract more informational features. In the fourth layer, the effective information from each segment is fused into a single representation, providing the model with global spectral information. Each of the five segments plays a crucial role in spectral recognition because they all contain key regions.

[0096] The segment fusion Transformer adopts a pyramid-shaped architecture design, increasing data dimensions, reducing data length, and gradually fusing information from multiple segments to fully extract features. Figure 7 a). Figure 7 b shows the original characteristic curves of the MCF-10A spectrum. Figure 7 Figure d illustrates the changes in the spectral features of the MCF-10A after each module. After feature extraction within each segment, the spectral frequency increases, and the peak value increases. The second layer emphasizes the interaction between segments; the solid line represents the actual receptive domain of the model, and the dashed line represents the segmentation of spectral features. Spectral features are significantly reduced, and peak values ​​decrease. In the third layer, fusion occurs across the entire segment for the first time, resulting in a reduction in peak values ​​and irrelevant features, while the peak values ​​become more prominent. In the final layer, the information fusion layer merges the segments into one segment, extracting the overall information from the five segments through the bidirectional vision Transformer module, further reducing the overall spectral features. However, the peak expression becomes stronger, the features become more pronounced, and the overall contour becomes smoother.

[0097] The following is an experimental simulation of the method of the present invention.

[0098] I. Obtaining the dataset used in the experiment:

[0099] The dataset contains 1003 spectra from five mammary cell lines and 1262 spectra from five lung cell lines, totaling 2265 spectra. These include 201 spectra of MCF-10A cells, 207 spectra of MDA-MB-231 cells, 239 spectra of BT-474 cells, 210 spectra of SK-BR3 cells, 237 spectra of T-47D cells, 239 spectra of SK-MES-1 cells, 290 spectra of H460 cells, 219 spectra of A549 cells, 243 spectra of H69 cells, and 190 spectra of MRC-5 cells. The dataset is divided into a training set and a test set, with 1810 spectra used for training and 455 spectra used for testing.

[0100] All cell lines were obtained from the Cell Resource Center of China United Medical University. Polymerase chain reaction (PCR) and culture confirmed that the cell lines were free from mycoplasma contamination. Cells were cultured in Dulbecco's Modified Eagle's Medium (DMEM). DMEM contained 1% penicillin-streptomycin solution and 10% fetal bovine serum (FBS). The cell culture environment was maintained at 37°C, 5% carbon dioxide (CO2), and 95% atmospheric pressure. Before use, cells were washed three times with phosphate-buffered saline (PBS). After each wash, cells were centrifuged at 1000 rpm for 3 minutes to ensure uniform cell distribution.

[0101] Single-cell spectra were obtained by placing a single cell on a silicon chip using a Finder One spectrometer (Zolix, China). This spectrometer includes a laser diode laser with an excitation wavelength of 532 nm and a back-illuminated, deep-dissipative, Peltier-cooled CCD detector. The target used had a numerical aperture (NA) of 0.65 and a working distance of 40x objective. The integration time for each spectrum was set to 5 seconds.

[0102] II. Experimental Data Analysis:

[0103] Figure 8 (ac) shows the distribution of the mean, standard deviation, and coefficient of variation for the five segments. The model distinguishes between good and poor spectral features by assigning attention scores. Based on the observations of the mean, standard deviation, and coefficient of variation, segment IV performs best. It exhibits a relatively concentrated and high mean distribution, indicating a consistent region of interest within samples from the same cell line, less affected by sample variations. A larger standard deviation indicates significant score variation within the region of interest, while a relatively small coefficient of variation suggests data stability. In contrast, segment IV performs relatively poorly, with a dispersed distribution of mean, standard deviation, and coefficient of variation, indicating higher sensitivity to data variations. Overall, the model is highly robust and almost unaffected by sample variations.

[0104] Attention mechanisms can help models automatically select the most prominent and representative features in the spectrum, thereby enabling accurate classification and identification of various cell lines. Figure 8 (d,e) illustrates the impact of attention on feature selection. The mean intra-class feature distances for the five segments are 0.99, 0.49, 5.41, 6.03, and 8.16, respectively. Intra-class distances within high-attention regions (attention > 0.7) are 0.02, 0.47, 0.02, 0.68, and 0.65. The mean feature distances within low-attention regions (attention < 0.1) are 21.10, 3.92, 19.62, 14.20, and 8.01, respectively. It can be seen that in highly attenuated regions, the intra-class distance is less than the mean feature distance, while in low-attention regions, the intra-class distance is greater than the original feature distance. Reducing the intra-class distance helps identify spectra from the same cell line. The ratio of inter-class distance to intra-class distance reflects the clustering of spectra within the same class and the dispersion of spectra between different classes. Figure 8 f shows the attention weights assigned to the five segments in the first stage, with segments I and III receiving very high attention, accounting for 32.3% and 30.3% of the total attention weights, respectively. These two attention areas account for 62.6% of the total attention. Figure 8 g shows the intra-class distances between the extracted spectral features and the original features for the five spectral segments. The intra-class distances of the original features are 0.03, 0.02, 0.06, 0.03, and 0.03, while the inter-class distances are 4.99, 2.72, 7.25, 3.28, and 4.18. Figure 8 The intra-class distance in `g` is magnified 100 times for visualization, showing how the piecewise feature extraction module alters spectral properties. The information extracted by the piecewise Transformer module enhances the feature distance between spectra from different classes while reducing the distance between spectra from the same class. The effectiveness of the attention region selection is further confirmed by the intra-class feature distance. The effectiveness of the piecewise Transformer module is demonstrated in two ways: the attention mechanism's selection of spectral features and the changes in spectral features.

[0105] The variance of intra-segment attention reflects the model's ability to distinguish between good and bad features. Higher variance indicates greater discriminative power. Except for segment II, the variance of intra-segment features gradually increases in the other four segments. Figure 9 a) The variance of segment II gradually increased from 0.10 to 0.20. Despite some fluctuations, segment II exhibited a stable overall upward trend. The attention fluctuations in segment II were due to the significant impact of the overall information provided in segment I and the partial information in segment III on the attention weights, leading to significant changes in the attention weights. As segment fusion progressed, the average attention value of each segment changed differently at different levels. Figure 9(b) The most noticeable change appears in the attention value of segment II, which initially was 0.63 and consistently higher than other segments by about 0.2. However, as segment fusion progresses, the average attention value of features within a segment gradually decreases, eventually placing segment II second in attention with an attention value of 0.44. This indicates that as segments interact and influence each other, the model gradually focuses on more accurate regions, such as segment I (the fingerprint region). Initially, the average attention value of this segment was relatively low at 0.33. However, with segment fusion and comparison with other segments, it gradually increased to 0.56, indicating an increased contribution. The model extracts features more comprehensively, capturing spectral information characteristics, and gradually weakens the expression of weaker features through the interaction and feature fusion between segments.

[0106] The interaction and fusion of segmented features lead to changes in intra-segment attention. What changes does this bring to intra-segment attention? As the hierarchy progresses, inter-segment fusion and intra-segment features change ( Figure 10 a) Feature selection through attention can preserve or enhance good features while eliminating or weakening bad features. The silhouette coefficient (SC) can be used to measure the tightness between samples of the same class and the separation between samples of different classes; a higher SC indicates better performance. Figure 10 b shows the changes in attention distribution (SC) of the ten cell line feature maps during segmented fusion. Generally, the SC increases between most categories as the hierarchy progresses. Although the initial SC of H460 cells is 1.46, it decreases to 1.41 in the second layer and stabilizes at 1.34. However, the SC of H460 cells remains higher than that of other cell lines, with the highest SC among the other cell lines being 1.21. Attention distribution changes across different levels for each segment (…). Figure 10 c). As segment fusion progresses, the ratio between attention-focused regions (attention value ≥ 0.7) and non-attention-focused regions (attention value ≤ 0.1) gradually changes. Figure 10 d). In the first layer, the focus area accounts for over 60% of each segment, with segment I reaching as high as 98%. This indicates a problem in attention selection. However, as the layers progress and inter-segment information interaction increases, attention to focus areas and non-focus areas gradually separates. In the final layer, focus areas account for 52%, non-focus areas for 45%, and intermediate areas for 3%. This suggests that the model strengthens the expression of good features and weakens the expression of bad features when selecting spectral features. Regarding attention distribution, the attention distribution of spectra within the same category should be consistent. During segment fusion, attention becomes relatively concentrated after being initially more dispersed. Figure 10e). This reflects the stability of the model's focus area on spectral features. As segmented fusion evolves, the interactions between segments increase, and the attention mechanism enhances the distinction between good and bad features, thereby improving the model's ability to fully extract spectral features and increase their expressive power.

[0107] After extracting spectral features using the model, a visualization of the changes in the feature map at each level is shown. Figure 11 a). With each subsequent layer, a significant increase in density was observed between samples within the same cell line, and a gradual separation pattern emerged between different cell lines in the spectral data, clearly visible in the changes in spectral features. Information entropy and information gain were used to assess the importance of spectral features and changes in information content. Information entropy gradually increased from 0.25 to a final 2.56, while information gain increased from 0.01 in the first layer to 0.14. Figure 11 (b, c) Changes in spectral characteristics increased the overall information entropy by 9.24 times and the information gain by 13 times. The ratio of intra-cluster distance to inter-cluster distance (CH index) and the silhouette coefficient (SC) are used to comprehensively evaluate intra-cluster and inter-cluster distances. Generally, larger values ​​indicate better feature selection performance. Figure 11 Tables d and e illustrate the changes in the CH index and SC as the hierarchy progresses. The CH index values ​​are 10.21, 11.23, 140.91, and 340.13, respectively, while the SC values ​​are 0.68, 0.73, 0.78, and 1.28, respectively. As the segments are gradually fused, the selected spectral features become increasingly typical and unique, which is beneficial for identifying spectral categories. The information content of the features and the clustering values ​​in the feature visualization demonstrate the high feature extraction capability brought about by segment fusion, supporting the logic of the rationality and effectiveness of the pyramid fusion design method.

[0108] III. Comparative Experiment:

[0109] To demonstrate the effectiveness of combining spectral feature segmentation with Transformer models and pyramid design structures, three additional Transformer models are proposed. The first model is a Transformer with spectral feature segmentation, involving only spectral feature segmentation without inter-segment fusion and interaction (segmented Transformer). The second model is a Transformer with global spectral features, directly inputting global spectral features without segmentation (Transformer without segmentation fusion). The third model is a Transformer model without segmentation, used as a baseline for comparison (Transformer without intrinsic segmentation). The segmented Transformer, the Transformer without segmentation fusion, and the Transformer without intrinsic segmentation are compared with the segmented fusion Transformer model. The performance of the four Transformer models is studied using four metrics: accuracy, confusion matrix, receiver operating characteristic (ROC) curve, and area under the curve (AUC) value. The accuracies of the segmented Transformer model, the piecewise Transformer model, the Transformer model without segmented fusion, and the Transformer model without intrinsic segmentation are 94.5%, 90.3%, 90.5%, and 91.6%, respectively. The Transformer model with intrinsic feature segmentation and inter-segment feature fusion performs best, demonstrating the suitability of combining intrinsic feature segmentation with Transformer and pyramid architecture for extracting features from high-dimensional continuous sequence spectral data. The confusion matrix and ROC of the segmented Transformer model are compared with the results of the other three Transformer models as follows: Figure 12 As shown. For the segment fusion Transformer model, all cell types except MCF-10A and MRC-5 cells showed the highest accuracy. Similarly, the AUC values ​​of MCF-10A and BT-474 cells were slightly lower than the comparison models, and the ROC curves showed the best performance of the segment fusion Transformer model. Figure 12 b). In Figure 12Table c shows the t-SNE projections of the feature vectors output by the four models. The colored point clouds represent different cell lines, illustrating how the models cluster the spectral data. We can see that points of the same category are clustered together and clearly separated from points of other categories. This clearly demonstrates that the segment fusion Transformer model exhibits the best overall performance, providing evidence for the rationality and effectiveness of combining spectral features (segmentation) with model architecture (inter-segment fusion). Other comparative metrics for the four models are shown in Tables 1 to 5, indicating that the segment fusion Transformer model outperforms the other three Transformer models on all four metrics.

[0110] The Segment Fusion Transformer model was compared with several established machine learning models, and their classification capabilities were evaluated using raw (unprocessed) spectral data. KNN, SVM, logistic regression, and CNN (ConNeXt model) were selected as four commonly used spectral models for comparison. This performance test was conducted on the same intrinsic sample dataset. The dataset consisted of 2,275 spectra from ten cell categories, with a training-to-test ratio of 8:2, randomly split, with 1,810 data points in the training set and 455 data points in the test set. The recognition accuracies of common spectral classification models, including KNN, CNN, logistic regression, and SVM, were 50.7%, 90.1%, 78.9%, and 80.2%, respectively. The Segment Fusion Transformer model achieved a classification accuracy 4.4% higher than the CNN model and more than 14% higher than the logistic regression and SVM classifiers. The CNN classifier performed stably in recognizing normal and cancer cells, achieving 100% accuracy for most cell lines and maintaining approximately 90% accuracy for other cell lines. KNN, SVM, and logistic regression classifiers showed inconsistent performance in binary classification (distinguishing between healthy and cancer cells), with varying accuracy across different cell lines. CNN classifiers exhibited relatively stable specificity, precision, and recall, consistently exceeding 90%. The Segment-fusion Transformer outperformed CNN classifiers in accurately distinguishing between healthy and cancer cells (Table 6). Only the MCF-10A, BT-474, and A549 cell lines were not fully identified by the Segment-fusion Transformer model, but their accuracy was higher than that of the CNN classifiers. The Segment-fusion Transformer model also demonstrated higher specificity, precision, and recall compared to CNN classifiers (Tables 6 to 9), highlighting its superior classification capabilities. The Segment-fusion Transformer model provides a new approach and solution for combining single-cell spectroscopy with Transformers.

[0111] IV. Evaluation Indicators:

[0112] Evaluation methods are crucial for measuring model performance. The performance of this model was evaluated using accuracy, sensitivity, specificity, ROC curves, AUC, and eigenvector clustering visualization.

[0113]

[0114]

[0115]

[0116]

[0117] Acc, Sen, Spec, and Pre represent accuracy, sensitivity, specificity, and precision, respectively. TP represents the positive samples correctly predicted by the model, TN represents the negative samples correctly predicted by the model, P represents positive samples, N represents negative samples, FN represents the negative samples incorrectly predicted by the model, and FP represents the positive samples incorrectly predicted by the model. Feature vector clustering visualization visualizes the feature vectors generated by the last layer of the model—that is, the vectors input into the linear network for classification—using TSNE. This allows for a direct observation of the degree of clustering of similar vectors, thus assessing the model's performance.

[0118] Table 1. Classification results of segment fusion Transformer

[0119]

[0120] Table 2. Classification results of segmented Transformers

[0121]

[0122] Table 3. Classification results of seamless fusion Transformers

[0123]

[0124] Table 4. Classification results of Transformers without intrinsic feature segmentation

[0125]

[0126] Table 5. AUC

[0127]

[0128] Table 6. Classification results of CNN

[0129]

[0130] Table 7. KNN classification results

[0131]

[0132] Table 8. Classification results of SVM

[0133]

[0134] Table 9. Classification results of logistic regression

[0135]

Claims

1. A method for analyzing high-dimensional continuous spectral data based on a segment fusion Transformer model, characterized by comprising the following steps: Step 1: Establish a segmented Transformer submodule to extract intra-segment features of spectral features; Step 2: Based on the spectral features processed in Step 1, and combined with the pyramid architecture, information from different segments is fused layer by layer. A segment fusion submodule is set up to perform segment fusion. Step 3: Input the spectral features processed in Step 2 into the classifier, and the classifier outputs the spectral category; The specific method for establishing the segmented Transformer submodule in step 1 to extract intra-segment features of spectral features is as follows: 1.1 Collect the spectrum of the substance and divide the spectrum into tokens, each token consisting of several data points; 1.2 Embed the token into features, project it onto an arbitrary dimension, denoted as C, and convert it into a feature vector; 1.3 Based on the intrinsic properties of the substance, including its composition and structural characteristics, the spectrum is divided into n discrete regions, and the spectral features are correspondingly divided into n segments, thereby transforming continuous spectral sequence data into discrete vector data to enhance the expressive power of spectral features. 1.4 Combining CNN networks with Transformers to form a segmented Transformer module to retain more information within a segment: extracting key features from each spectral segment divided in step 1.3, and obtaining the spectral feature map of each segment by performing spatial multi-head self-attention operation on the extracted features within each segment; then, stitching the spectral feature maps of each segment together, and then connecting to a dimensional multi-head self-attention module to perform interaction of spectral feature dimensional information and extract spectral feature dimensional information; After the discrete vector data undergoes multi-head self-attention operations on spatial and dimensional information, the shape and texture information of the spectrum is extracted by a CNN network module to obtain spectral feature vectors. Step 2 involves fusing the spectral features processed in Step 1, combined with a pyramid architecture, layer by layer, by setting up a segment fusion submodule to perform segment fusion. The specific method is as follows: Corresponding to the segmentation of spectral features in step 1, a feature pyramid architecture is adopted to fuse information from different segments layer by layer. The feature pyramid architecture is divided into several layers, each of which consists of an information fusion layer and several Transformer modules. The output of the previous layer is used as the input of the next layer. The inter-segment fusion layer consists of an information fusion layer and two Transformer modules. The information fusion layer of the inter-segment fusion layer is used to fuse inter-segment information and generate n-1 new segments. The Transformer module is used to extract features of the new segments. The new segments are input into the Transformer module, where layer normalization is performed first, followed by multi-head self-attention operation and forward propagation. Residual connections are connected between the multi-head self-attention operation and forward propagation to extract features. The segment fusion layer consists of an information fusion layer and six Transformer modules. The information fusion layer of the fusion layer is used to fuse the information of n-1 segments input from the inter-segment fusion layer to generate (n-1) / 2 new segments. The new segments are input into the Transformer module to perform layer normalization, and then multi-head self-attention operation and forward propagation are performed. Residual connections are connected between the multi-head self-attention operation and forward propagation to perform feature extraction. This layer can be repeatedly stacked until it merges into 3 or 2 segments; The normalization layer consists of an information fusion layer and two Transformer modules. The information fusion layer of the normalization layer merges the spectral information of the three or two segments input from the segment fusion layer into a new segment. The new segment is then input into the Transformer module, where layer normalization is performed first, followed by spatial multi-head self-attention and dimensional multi-head self-attention operations, and forward propagation. Residual connections are connected between the multi-head self-attention operations and forward propagation for feature extraction. Through this layer-by-layer information fusion, information on the entire spectral feature segment is obtained to comprehensively compare and extract the spectral feature information of the five segments. Finally, the obtained feature information is input into the spectral classifier.

2. The method for analyzing high-dimensional continuous spectral data based on a segment fusion Transformer model according to claim 1, characterized in that, The method for generating new segments by fusing inter-segment information in the information fusion layer described in step 2 is as follows: divide the spectral feature length by 2, concatenate two adjacent feature vectors together, and perform a linear transformation to obtain a new spectral feature vector.

3. The method for analyzing high-dimensional continuous spectral data based on a segment fusion Transformer model according to claim 1, characterized in that, The Transformer module described in step 2 is divided into a sliding window Transformer module and a bidirectional visual Transformer module. The inter-segment fusion layer and segment fusion layer of the feature pyramid architecture employ the sliding window Transformer module to extract information from spectral feature segments. The normalization layer uses a bidirectional visual Transformer module with dimensional self-attention and spatial self-attention to extract global spectral features: first, spatial multi-head self-attention is performed, then the spectral feature vector is transposed, and then multi-head self-attention is performed again to obtain the spectral feature vector after dimensional self-attention. Then, the spectral feature vector is transposed back to obtain a spectral feature vector with the same dimension as before the dimensional multi-head self-attention operation. This is followed by a residual connection, and finally input into the forward propagation network of the Transformer module to fuse local features within segments with global spectral features.

4. The method for analyzing high-dimensional continuous spectral data based on a segment fusion Transformer model according to claim 1, characterized in that, The specific formula for the layer normalization operation is as follows: Where, x i μ is the i-th feature of x, and x has a total of D features. l σ is the mean of all features of x. i It is the standard deviation of all features of x. γ is a small constant added for numerical stability; γ is a scaling parameter used to adjust the normalized result; β is a translation parameter used to adjust the normalized result; ⊙ represents element-wise multiplication.

5. The method for analyzing high-dimensional continuous spectral data based on a segment fusion Transformer model according to claim 1, characterized in that, The specific formula for the multi-head self-attention calculation is as follows: When inputting a two-dimensional tensor X∈R T×d Where T is the sequence length and d is the input dimension; multi-head attention consists of h attention heads, each with its own weight matrix, and the outputs of these heads are then concatenated together; W q ∈R d×dk W k ∈R d×dk W V ∈R d×dv It is a weight matrix that maps the input to the query, key, and value spaces, respectively. k and d v Each head represents the query, key, and value dimensions; QKV are three matrices representing the query, key, and value matrices, respectively; head i W represents the output of the i-th head. O ∈R T×dv×d It is the weight matrix of the final output projection.

6. A system based on the spectral data analysis method according to any one of claims 1 to 5, characterized in that, include: The segmented Transformer submodule includes an embedding module and a segmented Transformer module. The segmented Transformer module comprises a layer normalization module, a multi-head self-attention module, a forward propagation network, and a convolutional module. Specifically, the embedding module is used in the feature embedding stage of step 1 to convert tokens into feature vectors, providing a richer representation. The layer normalization module acts as a regularizer in steps 1 and 2, mitigating the internal covariate drift problem during deep neural network training. The multi-head attention module is used in the segmented Transformer module of step 1 to enhance strong feature representations and weaken weak feature representations. The forward propagation network is used to extract spectral features in steps 1 and 2. The convolutional module is used in step 1 to extract spectral shape and texture information. The segment fusion submodule includes an information fusion layer, a sliding window Transformer module, and a bidirectional vision Transformer module. Both the sliding window Transformer module and the bidirectional vision Transformer module contain a layer normalization module, a multi-head self-attention module, and a forward propagation network. The information fusion layer is used in step 2 to fuse information from different segments. The sliding window Transformer module is used in step 2 to extract information from spectral feature segments. The bidirectional vision Transformer module is used in step 2 to extract information from the entire spectral feature segment. The multi-head attention module is used to enhance strong feature representation and weaken weak feature representation. The forward propagation network is used in steps 1 and 2 to extract spectral features.

7. An electronic device based on the spectral data analysis method according to any one of claims 1 to 5, characterized in that, include: A memory and a processor, wherein the memory stores a computer program that causes the processor to execute any one of steps 1 to 3 of the high-dimensional continuous spectral data analysis method.

8. A storage medium for receiving user input, wherein the stored computer program, when executed by a processor, is capable of comparing and extracting spectral feature information to obtain spectral classification results based on the high-dimensional continuous spectral data analysis method according to any one of claims 1 to 5.

Citation Information

Patent Citations

  • Hyperspectral image classification method based on spectrum-space self-attention and Transform network

    CN117315481A