A medical sample typing method and system based on ViT and hyperspectral imaging

Abstract

The application discloses a medical sample typing method and system based on ViT and hyperspectral imaging, and relates to the technical field of image processing.The method comprises the following steps: obtaining original hyperspectral data of a medical sample, and performing a pretreatment operation on the original hyperspectral data; performing complex feature extraction on the pretreated original hyperspectral data by using a frequency domain perception spectrum block to obtain an embedding sequence suitable for a ViT model; modeling long-distance dependence relationships between each wave band of the embedding sequence by using a ViT model based on a multi-head self-attention mechanism to obtain a global feature vector; and performing typing prediction on the global feature vector by using a classifier.The application solves the problems of high dimension, great redundancy, insufficient long-distance dependence modeling capability of one-dimensional hyperspectral medical data in the prior art and poor adaptability of a traditional Vision Transformer to medical spectral data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a medical sample typing method and system based on ViT and hyperspectral imaging. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] Hyperspectral imaging (HSI) is an advanced method that combines spectral and imaging techniques, which can finely sample the target tissue in continuous wavebands and obtain high-dimensional data containing spatial and spectral information at the same time. In the medical field, HSI has gradually been applied to tumor detection, pathological typing, intraoperative boundary recognition and other tasks due to its ability to distinguish small pathological differences in tissues. However, one-dimensional hyperspectral data generally has problems such as extremely high dimension, large redundancy between channels, and insufficient spatial structure information, and directly inputting deep learning models can easily lead to overfitting, model convergence difficulties and low computational efficiency.

[0004] In existing research, convolutional neural network (CNN) and recurrent neural network (RNN) are commonly used one-dimensional spectral modeling methods. CNN can extract local spectral patterns through convolution kernels, but its receptive field is limited, and it is difficult to model long-distance dependencies across channels and capture complex global spectral features. While RNN and its variants such as LSTM and GRU can introduce sequence modeling mechanisms, but due to problems such as gradient vanishing and low computational efficiency, their performance on high-dimensional spectral sequences is also limited. In addition, some studies attempt to use dimension reduction methods (such as PCA and ICA) or spectral graph conversion methods (such as Gramian matrix and time-frequency transformation) to assist feature extraction, but these methods often introduce information loss or increase computational complexity, reducing the robustness and interpretability of the model.

[0005] Transformer model is initially applied to natural language processing tasks, and its core multi-head self-attention mechanism can effectively capture long-distance dependency and global context information. In recent years, Vision Transformer (ViT) has been introduced into the field of computer vision and has achieved excellent performance in image classification, object detection and other tasks. The principle is to divide the image into a patch sequence, which is input into the Transformer structure as Token after linear mapping, to realize the modeling of global features. However, ViT is originally designed for two-dimensional image tasks, and how to effectively adapt the structure for one-dimensional hyperspectral medical data is still a problem to be solved. Existing methods mostly input ViT by artificially constructing two-dimensional spectral graphs, but this way not only increases unnecessary redundancy, but also may destroy the physical continuity of the spectrum, resulting in a decrease in the reliability of the diagnosis result. SUMMARY

[0006] In view of the defects in the prior art, the purpose of the present application is to provide a medical sample typing method and system based on ViT and hyperspectral imaging, which solves the problems of high dimension, high redundancy, insufficient long-distance dependency modeling capability of one-dimensional hyperspectral medical data and poor adaptability of traditional Vision Transformer to medical spectral data in the prior art.

[0007] In order to achieve the above purpose, the present application is realized by the following technical scheme:

[0008] The first aspect of the present application provides a medical sample typing method based on ViT and hyperspectral imaging, comprising the following steps:

[0009] Obtain the original hyperspectral data of the medical sample and perform preprocessing operation on the original hyperspectral data;

[0010] Use frequency domain perception spectral block to extract composite features from the preprocessed original hyperspectral data to obtain embedding sequence suitable for ViT model;

[0011] Use ViT model based on multi-head self-attention mechanism to model the long-distance dependency relationship between each waveband of the embedding sequence to obtain global feature vector;

[0012] Use a classifier to predict the global feature vector.

[0013] Further, the preprocessing operation on the original hyperspectral data includes normalization and filtering denoising.

[0014] Further, the specific steps of using frequency domain perception spectral block to extract composite features from the preprocessed original hyperspectral data are:

[0015] performing fast Fourier transform on the pretreated original spectral data to extract an amplitude spectrum and a phase spectrum;

[0016] splicing the amplitude spectrum, the phase spectrum and the original spectrum to form a three-channel spectral feature;

[0017] slicing the three-channel spectral feature based on a sliding window mechanism to obtain a plurality of spectral blocks and map the plurality of spectral blocks into a Token sequence.

[0018] Further, the specific steps of slicing the three-channel spectral feature based on the sliding window mechanism to obtain a plurality of spectral blocks and map the plurality of spectral blocks into a Token sequence are as follows:

[0019] setting sliding window hyperparameters, slicing the three-channel spectral feature by using the sliding window, and generating a plurality of spectral blocks;

[0020] the generated spectral blocks form a token vector after being flattened and linearly mapped, and position encoding is added.

[0021] Further, the ViT model based on the multi-head self-attention mechanism is stacked by a plurality of Transformer blocks, each block including a multi-head self-attention layer, a feedforward neural network layer, layer normalization and a residual connection.

[0022] Further, the multi-head self-attention layer enhances the attention to discriminative bands by introducing a frequency domain saliency weighting factor.

[0023] Further, a cross-entropy loss function is designed for training, and cross-validation is used to evaluate the generalization performance of the trained model.

[0024] The second aspect of the present application provides a medical sample typing system based on ViT and hyperspectral imaging, comprising:

[0025] a data acquisition module configured to acquire original hyperspectral data of a medical sample and perform a pretreatment operation on the original hyperspectral data;

[0026] a first data processing module configured to extract a composite feature from the pretreated original hyperspectral data by using a frequency domain-aware spectral block to obtain an embedding sequence adapted to a ViT model;

[0027] a second data processing module configured to model long-range dependencies between bands of the embedding sequence by using a ViT model based on a multi-head self-attention mechanism to obtain a global feature vector;

[0028] a classification module configured to perform typing prediction on the global feature vector by using a classifier.

[0029] A third aspect of the present invention provides a computer-readable storage medium storing a computer program adapted for loading by a processor and executing steps in the medical sample typing method based on ViT and hyperspectral imaging as described in the first aspect of the present invention.

[0030] A fourth aspect of the present invention provides a computer device comprising:

[0031] A processor, adapted to execute computer programs;

[0032] A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the medical sample typing method based on ViT and hyperspectral imaging as described in the first aspect of the present invention.

[0033] The above one or more technical solutions have the following beneficial effects:

[0034] This invention discloses a medical sample typing method and system based on ViT and hyperspectral imaging. Addressing the problems of high dimensionality, large redundancy, and insufficient long-distance dependency modeling capabilities in existing one-dimensional hyperspectral medical data, as well as the poor adaptability of traditional VisionTransformer to medical spectral data, this invention proposes a medical sample typing method combining Vision Transformer and hyperspectral imaging. This method introduces for the first time a frequency-domain-aware spectral patch construction mechanism (Spectral Patch Embedding with Frequency Attention, SPEFA), overcoming the limitation of existing methods that require converting one-dimensional spectra into two-dimensional spectra. It can directly convert one-dimensional spectral signals into Transformer-compatible embedding sequences without relying on two-dimensional spectra construction, achieving a natural mapping from one-dimensional hyperspectral data to the Transformer structure. This effectively solves the mismatch problem between one-dimensional spectra and ViT structures, and significantly improves typing performance and model interpretability while maintaining spectral physical continuity.

[0035] Unlike existing methods that rely on spectral transformations such as PCA, Gramian matrices, or Markov transformation fields, this invention establishes a joint representation between the spectral and frequency domains by concatenating the original spectrum with its amplitude and phase spectra obtained from the Fast Fourier Transform (FFT) in three channels. The original spectrum and its frequency domain features are jointly modeled, leveraging the amplitude and phase spectrum information from the FFT to enhance the model's ability to capture global patterns. This avoids information loss associated with traditional dimensionality reduction methods, resulting in a physically meaningful composite feature representation. This composite feature not only preserves the temporal information of spectral intensity variations across bands but also introduces energy distribution and phase differences in the frequency domain, enabling the model to simultaneously capture both local spectral variations and global structure. Based on this, this invention generates a frequency-aware spectral block sequence on the concatenated three-channel spectrum using a sliding window mechanism. Each spectral block, after linear mapping, serves as a token input to the Vision Transformer, achieving a lossless transition from one-dimensional spectra to high-dimensional semantic embedding.

[0036] In the Transformer encoding stage, this invention further introduces learnable positional encoding, ensuring the identifiability of the band order in the spectral patches within the model and avoiding the loss of physical meaning caused by information rearrangement. Through a multi-head self-attention mechanism, the model can model the dependencies between spectral bands globally, effectively overcoming the limitations of local receptive fields in CNNs and the low modeling efficiency of RNNs. Simultaneously, this invention introduces a frequency-domain weighting factor into the attention mechanism, giving higher attention weights to bands in the spectrum sensitive to category discrimination. The model can adaptively allocate attention resources according to the actual spectral characteristics of medical samples, achieving adaptive spectral feature selection and improving the stability and interpretability of diagnosis.

[0037] In the classification decision-making stage, this invention employs a lightweight multilayer perceptron structure, which outputs the final classification result by fusing global feature vectors extracted by Transformer. This structure not only improves the model's classification accuracy but also provides clinicians with potential biological explanations through the visualization of attention weight matrices and frequency domain saliency maps, thereby enhancing the credibility of medical artificial intelligence systems.

[0038] This invention not only proposes an innovative method for constructing and embedding spectral features, but also significantly improves classification performance in practical medical sample typing tasks, especially suitable for disease diagnosis scenarios with minimal histological differences and high classification difficulty. This method serves as a crucial foundational framework for future intelligent analysis of hyperspectral medical images, providing strong technical support for clinical pathological auxiliary diagnosis.

[0039] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart of the medical sample typing method based on ViT and hyperspectral imaging in Embodiment 1 of the present invention;

[0042] Figure 2 This is a schematic diagram of the SPEFA module in Embodiment 1 of the present invention. Detailed Implementation

[0043] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0044] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0046] Example 1:

[0047] Embodiment 1 of this invention provides a medical sample typing method based on ViT and hyperspectral imaging, applicable to the classification of hyperspectral data of medical tissue samples, such as... Figure 1As shown, this method first normalizes and filters the original hyperspectral data for noise reduction. Then, it proposes a frequency-domain perceptual spectral patch generation mechanism (Spectral Patch Embedding with Frequency Attention, SPEFA) to concatenate one-dimensional hyperspectral data with its frequency domain features to form spectral patches. A sliding window mechanism is then used to generate token sequences that can be input to the Transformer. Subsequently, a Vision Transformer model is introduced to extract features from the token sequences, and a multi-head attention mechanism is used to model their global dependencies. Finally, a multilayer perceptron is used to achieve sample classification prediction. This invention avoids the redundancy and irreversibility of traditional image transformation methods, and improves the robustness and interpretability of the model.

[0048] Specifically, the following steps are included:

[0049] S1: Obtain the raw hyperspectral data of the medical sample and perform preprocessing operations on the raw hyperspectral data.

[0050] In one specific implementation, the preprocessing of the raw hyperspectral data includes normalization and filtering for noise reduction. These two processes are performed sequentially, with normalization primarily used to eliminate scale differences between samples, while filtering is used to reduce spectral noise and preserve spectral shape.

[0051] S1.1: Perform zero-mean normalization and variance standardization on the spectral data.

[0052] Specifically, for the raw hyperspectral data Normalization and noise reduction processes are performed, among which... Represents the set of real numbers. Represented as the number of samples; This is expressed as the number of spectral channels (i.e., the number of bands).

[0053] During the normalization phase, zero-mean normalization and variance standardization are performed on each spectral curve:

[0054]

[0055] in, For the first The sample at the th The original reflectance of each channel, The result after normalization; Indicates sample The mean; Standard deviation. Normalized data. Used for subsequent filtering and smoothing steps.

[0056] S1.2: A Savitzky-Golay filter is used for smoothing to reduce noise while preserving the spectral shape.

[0057] To further remove random noise from the hyperspectral curves, the normalized spectral data... The Savitzky-Golay filter is applied for spectral smoothing. This method achieves smoothing by performing polynomial fitting within a local window to preserve the original shape of the spectral signal as much as possible.

[0058] .

[0059] in, For window length, The order of the polynomial fitting is given by the given number. The spectral signal before smoothing (i.e., after normalization) This is the smoothed spectral signal.

[0060] S2: Use frequency domain sensing spectral blocks to extract composite features from the preprocessed raw hyperspectral data to obtain an embedding sequence adapted to the ViT model.

[0061] In one specific implementation, this embodiment designs a frequency-domain-aware spectral patch construction mechanism (Spectral Patch Embedding with Frequency Attention, SPEFA), which can directly map one-dimensional spectra into two-dimensional patch embeddings suitable for Vision Transformer models without spatial structure reconstruction, thereby improving diagnostic accuracy, stability and interpretability.

[0062] To adapt one-dimensional spectral data to the ViT model structure, a frequency-domain sensing patch is proposed for extracting composite features from the preprocessed raw hyperspectral data, such as... Figure 2 As shown, the specific steps are as follows:

[0063] S2.1: Perform a fast Fourier transform on the preprocessed raw spectral data to extract the amplitude spectrum and phase spectrum.

[0064] Specifically, each smoothed spectrum Perform Fast Fourier Transform (FFT):

[0065]

[0066] In this formula, the left side of the equal sign Indicates the first The nth sample in the frequency domain Complex values ​​of frequency components (complex spectrum); summation sign on the right side of the equals sign. Indicates indexing by time domain Accumulation, the accumulation range is arrive ,in This represents the number of spectral channels (i.e., the total number of bands). The first one obtained after S1 (preprocessing) Each sample in the band Smoothed and normalized spectral values; exponential term For complex rotation factors, The imaginary unit (satisfying) ), Pi is a constant, and its coefficient is... Defined the first The phase increment corresponding to each frequency. The right-hand side of the formula produces... Plural, real part is written as The virtual part is denoted as .

[0067] In obtaining Then, the amplitude spectrum and phase spectrum are constructed as supplementary features. The amplitude spectrum is defined as:

[0068]

[0069] The left side of the formula For the first The amplitude of each frequency (a non-negative real number), on the right. and They represent complex numbers respectively Real and imaginary parts; square root operation The magnitude is given. The phase spectrum is defined as:

[0070]

[0071] in, For the first Phase angle of each frequency (in radians). The arctangent function is used to calculate the angle corresponding to the ratio of the imaginary part to the real part (note the handling of the real part). Numerical stability issues are commonly encountered in engineering implementations. To avoid quadrant ambiguity.

[0072] S2.2: The amplitude spectrum, phase spectrum and original spectrum are spliced ​​together to form a three-channel spectral feature.

[0073] Specifically, in order to fuse spectral and frequency domain information and preserve complementary features, this embodiment constructs a joint three-channel feature matrix based on the time-domain spectrum, amplitude spectrum, and phase spectrum:

[0074]

[0075] The left side here The right side represents the joint feature matrix of the i-th sample; Represents the time-domain spectral vector (length is...) (row or column vector, depending on implementation convention); symbol This indicates a concatenation operation at the channel level, combining three channels of length [missing information]. The vectors are concatenated into one Matrix; matrix dimension Indicates the number of rows is (Corresponding to time domain, amplitude, and phase channels), number of columns is (Corresponding to each band).

[0076] S2.3: The three-channel spectral features are sliced ​​based on the sliding window mechanism to obtain multiple spectral blocks and map them as token sequences.

[0077] Specifically, in this embodiment, the window size and step size of the sliding window are adjustable hyperparameters. The generated spectral blocks are flattened and linearly mapped to form a token vector, and position encoding is added to maintain the physical continuity of the band order.

[0078] S2.3.1: Set the sliding window hyperparameters and use the sliding window to slice the three-channel spectral features to generate multiple spectral blocks.

[0079] To generate a patch sequence suitable for Transformer, in this A sliding window slice is used on the matrix. Let the window length be . Step size is Then the first Each spectral block is defined as:

[0080]

[0081] Among them, the left side of the equation Indicates the first The first sample A spectral block (is a) (submatrix); colon in the symbol This indicates selecting all rows of the matrix (i.e.) (one channel), and Indicates from the first Listed to number A continuous column interval (including endpoints), and The total number of spectral blocks is usually expressed by the formula... Confirmed, among which This indicates rounding down. In engineering practice, it's possible to choose whether to add zero padding at both ends to ensure that boundary bands are also covered; if padding is used, then... The calculation can be adjusted accordingly. .

[0082] S2.3.2: The generated spectral blocks are flattened and linearly mapped to form token vectors, and positional encoding is added.

[0083] The Transformer architecture essentially deals with one-dimensional sequence vectors (of tokens) rather than two-dimensional matrices. Therefore, each spectral block needs to be flattened into a one-dimensional vector before being fed into the Transformer's input layer. The flattened vector has a length of 3M, but M may vary under different experimental conditions. Since the Transformer requires all tokens to have a uniform embedding dimension d, a linear mapping is necessary. The specific steps are as follows:

[0084] Flatten each spectral block and embed it as a Transformer token using a linear mapping, defined as:

[0085]

[0086] Among them, the left side of the equal sign For the first The first sample Each token (a row vector or column vector, with a defined dimension of 1). ); the operator on the right Indicates will The submatrix is ​​serialized by rows or columns to length A vector (here, Flatten's result is a column vector); It is an embedding matrix (linear projection matrix), where each column is a learnable basis vector; It is a bias vector; therefore, the matrix multiplication and addition result on the right belongs to , and the left side Consistent shape. About Initialization can be performed using Xavier / Glorot initialization or orthogonal initialization to improve convergence, and adjustments can be made during training. Apply weight decay regularization to prevent overfitting.

[0087] Combine all tokens in order to get the [number]. The token sequence of each sample:

[0088] .

[0089] Here Each row (or column, depending on the implementation) represents a token. Indicates the number of tokens. This represents the vector dimension of the token (the input dimension of the Transformer). In practical engineering implementations, batch normalization or LayerNorm can be added to the token to stabilize training; the formula and definition of LayerNorm are given in section S3 later.

[0090] It should be noted that the computational complexity of FFT is O(n log n). It can be parallelized on batch data; window length The choice should be a trade-off between preserving local spectroscopic information and reducing the number of tokens; in engineering, the preferred approach is... The step size is 8, 16, 32, etc.; M is usually chosen. Or M to control the degree of token overlap; if The amplitude spectrum is large, so we can first use energy-preserving band selection or bandpass filtering to reduce meaningless high-frequency noise. Furthermore, to improve robustness, we can analyze the amplitude spectrum... Perform logarithmic compression (e.g.) To suppress numerical instability caused by extreme amplitudes and to adjust the phase Use periodic extended mapping to avoid jumps.

[0091] S3: The ViT model based on the multi-head self-attention mechanism is used to model the long-distance dependencies between different bands of the embedded sequence spectrum and obtain the global feature vector.

[0092] In one specific implementation, the ViT model based on the multi-head self-attention mechanism consists of stacked Transformer Blocks. Each Block includes a multi-head self-attention layer, a feedforward neural network layer, layer normalization, and residual connections. The multi-head self-attention layer enhances attention to the discriminative band by introducing a frequency domain saliency weighting factor. Specifically, it includes the following steps:

[0093] token sequence Before inputting ViT, position encoding is added to preserve spectral order information:

[0094] .

[0095] Among them, the left side of the equal sign This is a token with location encoding; the one on the right. The original token obtained from S2; Indicates the first Each token has a positional encoding vector, where the positional encoding can be a learnable parameter matrix. The Alternatively, fixed sine or cosine position encoding can be used, both of which are applicable within the framework of this invention. The purpose of position encoding is to enable the Transformer to distinguish the order of different tokens, thereby preserving the physical band order of the spectrum.

[0096] In ViT's multi-head self-attention mechanism, the standard single-head attention calculation is as follows:

[0097] .

[0098] Among them, the left side of the equal sign This represents the output matrix of the attention layer; in the right-hand side, These represent the query, key, and value matrices, respectively, typically derived from the input. Obtained through linear mapping, i.e.

[0099] .

[0100] in To all A matrix composed in order; It is a learnable linear projection matrix; therefore In the attention formula, matrix multiplication... The result is The similarity (relevance) score matrix, divided by (in The key / query dimension for each head is used for numerical stability, to prevent the softmax gradient from vanishing due to excessively large dot product values. The softmax operation is performed row-wise (i.e., for each query position), and is defined as applying a vector... The One component:

[0101] .

[0102] Here Natural exponential function. The output is a row vector (or a matrix operated on row-wise), whose elements are non-negative and whose row sum is 0. This can be viewed as an attention weight distribution. The attention weight matrix and... The multiplication produces a context convergence output with dimension 1. .

[0103] Multi-head attention is commonly used in Vitality Techniques (VIT), and it takes the form of:

[0104] .

[0105] And each Defined as , Indicates the first One attention point, among which The left side of the equals sign Indicates will The outputs of each size are concatenated and then subjected to a linear transformation. Mapping back to the original dimension In these formulas, the letters Indicates the number of heads of attention, usually .

[0106] To enhance focus on important frequency bands, this embodiment introduces a frequency domain weighting term into the attention score. (or spectral significance matrix), can be used additively or multiplicatively with... Merging, for example, the expanded attention score can be written as:

[0107] .

[0108] in, For the final attention scoring matrix, matrix elements This indicates that the p-th token is paired with the p-th token. The relative importance of each token; the right side of the equals sign The frequency domain significance matrix is ​​designed, and its elements are calculated from frequency domain information (such as amplitude spectrum difference, phase difference, statistical significance index, etc.), with the same dimension. Here, The introduction of this allows the prior allocation of attention to reflect discriminative information in the frequency domain (superimposed with pure dot product similarity), thus achieving "frequency-domain weighted self-attention". Regarding... The construction method can be represented by the following example (for implementation reference):

[0109] .

[0110] Among them, the left side of the equal sign express In the The value of the position, These are hyperparameters used to adjust the strength of frequency domain prior weights. and These represent the summation values ​​of the amplitude spectrum over the frequency bands (or spectrum centers) covered by the corresponding tokens p and q, respectively, with small constants in the denominator to prevent the inclusion of zero values. Therefore, it can be seen that... The design requires establishing a mapping between the token and its corresponding frequency domain location (e.g. Covering the original frequency index range And use domain knowledge to determine appropriate amplitude or phase statistics as significance indicators.

[0111] A standard Transformer block includes a multi-head attention layer, residual connections and layer normalization (LayerNorm), a feed-forward network (FFN), and another residual +LayerNorm. The specific calculation can be expressed as:

[0112] .

[0113] Among them, the left side of the equal sign This indicates that after the first round of residual connection and The processed intermediate representation; This represents the multi-head attention output defined above; The layer normalization operator is represented by the following commonly used formula:

[0114] .

[0115] in and These are the vectors to be normalized. The mean and variance (calculated by feature dimension). Small constants are used for numerical stability. and For learnable scaling and offset vectors, sign This indicates element-wise multiplication. The right side of the equals sign... For feedforward networks, they are typically represented as two linear transformations with nonlinear activation functions (e.g., GELU).

[0116] If remember ,but For learnable matrices, This is the activation function.

[0117] After several (e.g.) The global feature vector is obtained by stacking Transformer Blocks and using global pooling or retrieving the [CLS] token:

[0118] .

[0119] Where, on the left side of the equal sign For the first The global representation vector of each sample (features output by the model), the right side of the equals sign This represents the function mapping of the entire Transformer encoder, with an output dimension of... .

[0120] To prevent overfitting, dropout can be added to the attention output and FFN, denoted as . Pay attention to the number of heads Number of floors token dimension Dimensions per head As an adjustable hyperparameter, it is selected through cross-validation. For stable training, it is recommended to use weight decay or the AdamW optimizer for the linear weights in the Transformer.

[0121] S4: Use a classifier to perform genotyping prediction on the global feature vector.

[0122] In one specific implementation, the global features output by the Transformer are... Input the classification head (MLP) to obtain the predicted probabilities:

[0123] .

[0124] Among them, the left side of the equal sign Indicates the first The predicted probability vector of each sample; the linear transformation on the right side of the equation. From dimension Mapped to ,in This is the classification weight matrix (each row corresponds to a linear discriminant vector for one class). For bias terms; function Will A real number is mapped to a probability distribution that satisfies and The component on the left side of the equals sign. Indicates sample Category The estimated probability. For multi-class tasks, the highest probability index can be used as the final predicted class, or a sensitivity / specificity balance can be achieved based on a threshold.

[0125] Furthermore, to enhance the model's interpretability, this embodiment combines the attention matrix with the frequency domain saliency matrix. The images are jointly projected back to the original wavelength axis to generate a wavelength importance map. Specifically, a wavelength importance vector can be defined. This is a distribution mapping of the sum of attention for each token across the original channel, that is, the attention weights of the token level along its coverage. The bands are averaged or weighted and allocated to the corresponding bands to obtain It can be used to visualize which bands contribute most to classification, facilitating clinical interpretation.

[0126] S5: Design a cross-entropy loss function for training, and use cross-validation to evaluate the generalization performance after training.

[0127] In one specific implementation, a cross-entropy loss function is designed to train the trainable portion of the entire process from S1 to S4. During the training phase, cross-entropy is used as the primary loss function, formally expressed as:

[0128]

[0129] In this formula, This represents the average cross-entropy loss over the entire training set; Number of training samples; first index of the internal double summation. Traverse the samples, second index Traverse categories (from) arrive ); For the first One-hot encoding of the true label of each sample (if the sample belongs to category). ,but (and the rest are 0). This represents the predicted probability value in equation S4; The loss is the natural logarithm. This loss measures the deviation between the predicted probability distribution and the true distribution, and is minimized during training through backpropagation. .

[0130] To further improve generalization ability, in actual training, a regularization term (such as L2 weight decay) and a class imbalance handling strategy can be added to the cross-entropy loss to form the total loss:

[0131] .

[0132] in The total loss with regularization terms; This is the weight decay coefficient (hyperparameter); Represents the set of all parameters of the model that need to be regularized (e.g.) wait) These parameters Norm squared. If the training data is severely imbalanced, class weights or focal loss can be used instead of standard cross-entropy. Can be introduced as .

[0133] For optimizer selection, AdamW (Adam with weight decay) is recommended, and its parameter update equation is (simplified representation):

[0134] .

[0135] In the formula Indicates the first The parameters for the next iteration For learning rate, and These are the first and second moment estimates of the gradient after bias correction, respectively. The first term is the numerically stable term, and the last term is the weight decay term (equivalent to L2 regularization). The learning rate can employ cosine annealing or a warmup strategy to facilitate stable Transformer training.

[0136] In terms of training strategy, it is recommended to use cross-validation (e.g., five-fold cross-validation) to evaluate generalization performance, and record metrics such as C-index, Accuracy, Recall, and F1; in each experiment, adjust the hyperparameters ( To obtain a robust configuration, perform grid or Bayesian optimization (e.g., Optuna). To reduce the risk of overfitting, data augmentation strategies (e.g., spectral noise injection, random band occlusion / band dropout, amplitude perturbation) can be applied to the training set to improve the model's robustness to measurement errors and device differences.

[0137] This embodiment can also export the attention weight matrix after training. (dimension) ), and map it back to the wavelength axis along the token coverage band to obtain the wavelength saliency map. This saliency map can be compared with pathological features to provide a biological interpretation. Furthermore, ablation experiments can be performed on the model (removing...). (Using only the time domain, only the amplitude spectrum, etc.) to demonstrate the necessity and effectiveness of the SPEFA module and frequency domain weighting in improving fractal performance.

[0138] The histological typing of medical samples obtained by the method in this embodiment can improve typing accuracy and interpretability while maintaining spectral physical continuity, making it suitable for deployment in clinical auxiliary diagnostic systems.

[0139] Example 2:

[0140] Embodiment 2 of the present invention provides a medical sample typing system based on ViT and hyperspectral imaging, comprising:

[0141] The data acquisition module is configured to acquire raw hyperspectral data of medical samples and perform preprocessing operations on the raw hyperspectral data.

[0142] The first data processing module is configured to use frequency domain sensing spectral blocks to extract composite features from the preprocessed raw hyperspectral data to obtain an embedding sequence adapted to the ViT model.

[0143] The second data processing module is configured to use the ViT model based on the multi-head self-attention mechanism to model the long-distance dependencies between different bands of the embedded sequence spectrum and obtain the global feature vector.

[0144] The classification module is configured to use a classifier to perform genotyping predictions on the global feature vector.

[0145] Example 3:

[0146] Embodiment 3 of the present invention provides a computer-readable storage medium storing a computer program adapted for loading by a processor and executing the steps in the medical sample typing method based on ViT and hyperspectral imaging as described in Embodiment 1 of the present invention.

[0147] Example 4:

[0148] Embodiment 4 of the present invention provides a computer device, the device comprising:

[0149] A processor, adapted to execute computer programs;

[0150] A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the steps in the medical sample typing method based on ViT and hyperspectral imaging as described in Embodiment 1 of the present invention.

[0151] The steps and methods involved in Examples 2, 3 and 4 above correspond to those in Example 1. For specific implementation details, please refer to the relevant description section of Example 1.

[0152] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0153] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, an optical medium, or a semiconductor medium, etc.

[0154] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A medical sample typing method based on ViT and hyperspectral imaging, characterized in that, Includes the following steps: Acquire raw hyperspectral data of medical samples and perform preprocessing operations on the raw hyperspectral data; Composite features are extracted from the preprocessed raw hyperspectral data using frequency domain sensing spectral blocks to obtain an embedding sequence adapted to the ViT model, specifically: A fast Fourier transform is performed on the preprocessed raw spectral data to extract the amplitude spectrum and phase spectrum; The amplitude spectrum, phase spectrum, and original spectrum are spliced ​​together to form a three-channel spectral feature; The three-channel spectral features are sliced ​​using a sliding window mechanism to obtain multiple spectral blocks, which are then mapped to a token sequence. The specific steps are as follows: Set the sliding window hyperparameters and use the sliding window to slice the three-channel spectral features to generate multiple spectral blocks; The generated spectral blocks are flattened and linearly mapped to form token vectors, and positional encoding is added. The ViT model based on multi-head self-attention mechanism is used to model the long-range dependencies between different spectral bands of the embedded sequence, and the global feature vector is obtained. The multi-head self-attention mechanism enhances attention to the discriminative band by introducing a frequency domain weighting factor; A classifier is used to perform genotyping prediction on the global feature vector.

2. The medical sample typing method based on ViT and hyperspectral imaging as described in claim 1, characterized in that, Preprocessing of raw hyperspectral data includes normalization and filtering for noise reduction.

3. The medical sample typing method based on ViT and hyperspectral imaging as described in claim 1, characterized in that, The ViT model based on the multi-head self-attention mechanism consists of multiple Transformer Blocks stacked together. Each Block includes a multi-head self-attention layer, a feedforward neural network layer, layer normalization, and residual connections.

4. The medical sample typing method based on ViT and hyperspectral imaging as described in claim 1, characterized in that, We designed a cross-entropy loss function for training and used cross-validation to evaluate the generalization performance after training.

5. A medical sample typing system based on ViT and hyperspectral imaging, characterized in that, include: The data acquisition module is configured to acquire raw hyperspectral data of medical samples and perform preprocessing operations on the raw hyperspectral data. The first data processing module is configured to extract composite features from the preprocessed raw hyperspectral data using frequency domain sensing spectral blocks to obtain an embedding sequence adapted to the ViT model, specifically: A fast Fourier transform is performed on the preprocessed raw spectral data to extract the amplitude spectrum and phase spectrum; The amplitude spectrum, phase spectrum, and original spectrum are spliced ​​together to form a three-channel spectral feature; The three-channel spectral features are sliced ​​using a sliding window mechanism to obtain multiple spectral blocks, which are then mapped to a token sequence. The specific steps are as follows: Set the sliding window hyperparameters and use the sliding window to slice the three-channel spectral features to generate multiple spectral blocks; The generated spectral blocks are flattened and linearly mapped to form token vectors, and positional encoding is added. The second data processing module is configured to use the ViT model based on the multi-head self-attention mechanism to model the long-distance dependencies between different bands of the embedded sequence spectrum and obtain the global feature vector. The multi-head self-attention mechanism enhances attention to the discriminative band by introducing a frequency domain weighting factor; The classification module is configured to use a classifier to perform genotyping predictions on the global feature vector.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-4: a medical sample typing method based on ViT and hyperspectral imaging.

7. A computer device, characterized in that, include: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by the processor, implements the medical sample typing method based on ViT and hyperspectral imaging as described in any one of claims 1-4.

Images