Tissue slice and body fluid smear analysis method fusing elastic scattering spectrum and Raman spectrum

Abstract

The invention discloses a tissue slice and body fluid smear analysis method fusing an elastic scattering spectrum and a Raman spectrum. The tissue slice and body fluid smear analysis method comprises the following steps: step 1, Raman spectrum data acquisition; 2, collecting elastic light scattering data; step 3, data processing; step 4, constructing a classification composite model based on the fusion spectrum; step 5, composite model training; and step 6, obtaining a result prediction value according to the model. The composite model provided by the invention can effectively learn multi-modal spectral characteristics of different types of tissues and body fluid smears, and realizes high-accuracy identification of tumor tissues or body fluid smears; in the aspect of classification performance, the decision-making performance of the model trained by the multi-modal spectral data is obviously superior to that of a model trained by single Raman spectral data or elastic light scattering data, and the method has the advantages of being wide in applicability, capable of obtaining while taking, small in damage to samples, short in time consumption, high in timeliness and the like.

Description

Technical Field

[0001] The present invention relates to a method for analyzing tissue sections and body fluid smears by integrating elastic scattering spectroscopy and Raman spectroscopy, which is used for rapid, label-free, and non-invasive detection of pathological tissues and body fluid smears such as blood and bone marrow fluid, and belongs to the field of in vitro detection. Background Art

[0002] As a new method for studying the chemical composition properties of pathological tissues, Raman spectroscopy technology has the characteristic of "molecular fingerprint spectrum" due to its technical principle based on a physical phenomenon called "Raman scattering", which is essentially a secondary radiation phenomenon of light. When light irradiates on the surface of a substance, in addition to Rayleigh scattering (elastic scattering), a small part of the photon energy changes, resulting in inelastic scattering, and at the same time its frequency changes. The magnitude of the changed frequency is called the Raman shift. By analyzing the magnitude of the Raman shift, information about different molecular vibrations and rotations can be obtained. Therefore, the differences in the molecular composition of pathological tissue components will be reflected in the Raman spectrum. Due to the differences in the biological macromolecule components such as proteins, nucleic acids, and lipids between tumor tissues and normal tissues, their Raman spectra will also show certain differences, which makes it possible to use Raman spectroscopy technology for tumor pathological tissue analysis, classification, and research.

[0003] The principle of laser elastic scattering for exploring the properties of substances is that when light contacts the surface of an object, physical phenomena such as diffuse reflection, scattering, and absorption will occur. Due to different physical properties such as the structure and refractive index of substances, the scattering spectra formed by the scattering of different substances also have certain differences, and the analysis and research results of their signals can provide a reliable basis for the study of the physical and chemical properties of substances.

[0004] In the application of light scattering spectroscopy technology in cell detection, the scattering light technology has high sensitivity and specificity for the differences in the morphological structure of cells at the cell and subcellular levels, including the size of the cell nucleus, the distribution of mitochondria, and the morphological changes of other organelles. Tumor cells are different from normal cells in morphology and structure. Therefore, elastic light scattering spectra have inherent advantages in the rapid non-destructive morphological structure analysis of tumor tissue pathology.

[0005] Deep learning is inspired by the way the human brain processes signals, constructing a multi-layer complex computational model composed of countless "artificial neurons" to learn complex features and relationships in data. Deep learning was initially successfully applied in the field of computer vision to extract image features and achieve goals such as image recognition and object classification. Compared with traditional machine learning methods, deep learning can automatically extract features, omitting the complex manual feature extraction and parameter selection steps in the traditional learning process, greatly improving the speed and efficiency of machine learning in training and decision-making. At the same time, through a large number of complex operations, deep learning can fully extract data pattern information in different feature dimensions, providing an effective automated method for feature discovery, feature hierarchy exploration, and feature interaction of real-time data. The concept of deep learning originated from artificial neural networks, which have several hidden layers between the input and output parts. There are several neuron structures in each hidden layer, and each neuron structure can obtain corresponding weights through learning the training data. The input data is output through the feature extraction of each hidden layer in the output layer to complete the learning and classification tasks of complex data.

[0006] Since the M-P model, artificial neural networks have received extensive attention, and the proposed Hebbian learning rule has laid the algorithmic foundation for neural network learning. With the continuous development of computer hardware and software, more powerful computing power has made it possible to build larger-scale deep learning networks. The proposed large-scale convolutional neural networks and recurrent neural networks have greatly promoted the application of machine learning in computer vision and natural language processing. In recent years, the Transformer architecture based on the attention mechanism has shown great adaptability in different tasks, providing a new path for deep learning. Currently, different deep learning networks are widely used in fields such as image recognition, text translation, and audio processing, and have also been successfully applied in image recognition and pathological analysis in the medical field.

[0007] For the current diagnosis of pathological tissues, traditional pathological diagnosis methods rely on tissue staining and microscopic observation, with problems such as complex operation, long time consumption, and low repeatability. Raman spectroscopy technology, as a label-free and non-invasive detection method, can reflect the chemical composition information of tissues through "molecular fingerprint spectra", but lacks the capture of tissue morphological structure features. Light scattering technology can provide tissue morphological structure information, but its application still faces the challenge of insufficient accuracy. In addition, traditional machine learning methods have problems such as low classification accuracy and poor interpretability in spectral analysis.

[0008] Raman spectroscopy is a complex superimposed signal of various one-dimensional information. In the early years, the related research on Raman spectroscopy was restricted by the limitations of analysis methods, making it difficult to extract useful information characterizing the properties of related substances from complex signals. In recent years, due to the development of artificial intelligence technology, deep learning methods, as powerful tools for data processing, have been widely used in the processing and analysis of Raman data. Combined with corresponding artificial intelligence methods, Raman spectroscopy has been successfully applied in many aspects, including intraoperative localization of tumor resection, histopathological research of tumor tissues, early detection of tumor markers, and identification of tumor boundaries, and good clinical experimental results have been achieved. Compared with traditional staining methods, the pathological analysis method of Raman spectroscopy has the advantages of wide applicability, label-free, and little damage to samples. However, at the same time, when analyzing histopathological information, Raman spectroscopy is difficult to cover the morphological information of tissues at the physical level, showing a certain degree of one-sidedness. Currently, with the rapid development of artificial intelligence, the ability of signal analysis has been greatly improved, and people can extract more spectral signals related to cell characteristics from complex superimposed scattering spectra. At present, elastic light scattering technology has been preliminarily applied to a series of biomedical scenarios such as microbial detection, cell analysis, and tumor tissues. Although there are still certain limitations in accuracy, it is still one of the means expected to achieve non-invasive pathological analysis of tissue morphological structure characteristics.

[0009] In the current pathological tissue examination methods, tissue samples are obtained through biopsy or surgical resection for tissue sections. Fixatives such as formalin are used to prevent tissue degradation. The fixed tissue is embedded in paraffin for cutting. The tissue is cut into thin slices (usually 4-6 microns) with a microtome. Common staining methods include hematoxylin-eosin (HE) staining, special staining (such as PAS, Masson trichrome staining), and immunohistochemical staining. Pathologists make diagnoses by observing tissue structure, cell morphology, and staining characteristics through a microscope. This method has disadvantages such as long sample processing time and subjectivity. For tissue smears, body fluids (such as blood, urine, cerebrospinal fluid, pleural effusion, ascites, sputum, etc.) are obtained through puncture, aspiration, or natural discharge. The body fluid is evenly smeared on a glass slide and fixed. Common staining methods include Gram staining, Giemsa staining, Papanicolaou staining, and special staining (such as acid-fast staining). Cell morphology, pathogens (such as bacteria, fungi, parasites), or tumor cells are observed through a microscope. This method has disadvantages such as inability to provide comprehensive information and subjectivity.

[0010] The Raman spectral signals of tissue cells comprehensively reflect their rich chemical characteristics, while the scattering spectra show the morphological and structural differences of biological tissues. There are certain differences between pathological tissues and normal tissues in both biochemical components and morphological structures. Combining the advantages of Raman technology and light scattering technology in their respective fields, the characteristic spectra of tissues in the component and morphological dimensions are obtained respectively by using the dual-spectrum detection method. Through the study of the fused spectral signals, different tissues can be comprehensively analyzed from the physical and chemical perspectives. With the assistance of artificial intelligence deep learning algorithms, in-depth data mining of multi-modal spectra can be carried out to fully extract the characteristic information of pathological tissues in different aspects, realizing the identification and analysis of pathological tissues. This method reflects the global characteristics of the association between the two while paying attention to the local characteristics of the morphological and chemical characteristics of tissue cells, and has higher accuracy and interpretability than single-spectrum analysis techniques. Summary of the Invention

[0011] The object of the present invention is to provide a method for analyzing tissue sections and body fluid smears by fusing elastic scattering spectra and Raman spectra, and to solve the problems of how to analyze them while keeping the tissues and smears intact, and how to fuse the two types of spectra to analyze multi-dimensional information.

[0012] To achieve the above object, the technical solution adopted by the present invention is: a method for analyzing tissue sections and body fluid smears based on the fusion of elastic scattering spectra and Raman spectra, which includes the following steps:

[0013] Sample preparation:

[0014] Section part: Take out the tissue sample;

[0015] Use a microtome or manually section it, with the thickness controlled at 4-6 microns;

[0016] Fix the section on a glass slide;

[0017] Smear part: Take out the body fluid sample;

[0018] Use a machine or manually smear it, and require uniform smearing;

[0019] Fix the smear.

[0020] Then perform the following operations:

[0021] Step 1: Raman spectral data acquisition;

[0022] Step 2: Elastic light scattering data acquisition;

[0023] Step 3: Data processing;

[0024] Step 4: Construction of a classification composite model based on the fused spectrum;

[0025] Step 5: Composite model training;

[0026] Step 6: Obtain the result prediction value according to the model.

[0027] Furthermore, the said Step 1 includes:

[0028] Step 101: First, fix the sample on the Raman spectrometer stage;

[0029] Step 102: Set the instrument laser power and integration time;

[0030] Step 103: Adjust the probe height to ensure an appropriate laser focusing position;

[0031] Step 104: Pre-acquire the dark current of the device;

[0032] Step 105: Scan the sample, check the characteristic peaks of the substance, and select an appropriate scanning section;

[0033] Step 106: Optimize the integration exposure time parameter according to the pre-test result;

[0034] Step 107: Uniformly select the tissue collection positions, and sequentially collect and record the Raman spectra according to the marked points;

[0035] Step 108: Subtract the pre-acquired dark current from the acquisition result;

[0036] Step 109: Export the file;

[0037] Step 110: Convert the format of the collected file.

[0038] Furthermore, the said Step 2 includes:

[0039] Step 201: First, isolate the external light;

[0040] Step 202: Adjust the bias voltage of the signal amplification module to zero the light scattering acquisition device;

[0041] Step 203: Then fix the fabricated sample sheet;

[0042] Step 204: Adjust the output frequency;

[0043] Step 205: Set the stepping motor speed, and select an appropriate sampling rate, recording time, and recording channel;

[0044] Step 206: Uniformly select the acquisition points for sampling to obtain the raw data;

[0045] Step 207: Export the file,

[0046] Step 208: Convert the format of the collected file.

[0047] Further, step 3 includes:

[0048] Step 301: Smooth and denoise the spectral data signal using a filter;

[0049] Step 302: Perform baseline correction;

[0050] Step 303: Intercept in the Raman spectral signal fingerprint region;

[0051] Step 304: Before the Raman signal and the light scattering signal are used as model inputs, normalize the signals;

[0052] Step 305: Define the processed dual-channel light scattering data sets as X s1 、X s2 , and the Raman data set as X r ;

[0053] Step 306: Reduce the dimension of the light scattering data to be equal to the dimension of the Raman spectral data;

[0054] Step 307: Perform data fusion on the dual-channel light scattering data and the Raman data.

[0055] Furthermore, in the smoothing and denoising operation of step 301: For the spectral data y(t) to be processed, use a polynomial to fit the data:

[0056]

[0057] where ai is the constant coefficient of each term of the polynomial. Select the window length and calculate the mean square error:

[0058]

[0059] Furthermore, the baseline correction process of step 302 is as follows:

[0060] Calculate the fitting error F and the data smoothness R, expressed as:

[0061]

[0062] where x is the original spectral signal, z is the fitted spectral signal, and m is the signal length; At the same time, the smoothing parameter λ is introduced into the calculation of the total penalty term Q to balance the contradiction between the fitting error and the smoothness. The total penalty term Q weighted by the smoothness is expressed as:

[0063]

[0064] where λ is the smoothing parameter, used to control the smoothness of the fitting;

[0065] The penalty term function after the t-th iteration calculation is expressed as:

[0066]

[0067] where represents the weight of the t-th iteration, represents the fitting baseline of this time; during the fitting process, the weight is updated according to the difference between the original signal and the fitting baseline of the previous iteration. If the value of a certain data point is greater than the current fitting baseline, it is determined to be possibly part of a peak and will be ignored in the next iteration, that is, the weight is 0. The function for weight update is expressed as:

[0068]

[0069] In the above iteration process, the algorithm will stop when it reaches the maximum number of iterations or meets certain conditions. Its termination condition is shown in the formula:

[0070] |d t |< 0.001×|x|

[0071] where d t is the difference between the original signal and the fitting baseline in the current iteration.

[0072] Furthermore, the normalization operation in step 304 is as follows:

[0073] The Raman data is standardized in the dataset:

[0074]

[0075] where j represents the j-th feature of the spectrum, μ j represents the mean of all features in this feature dimension, and σ j is its standard deviation, and the calculation formula is:

[0076]

[0077] Normalize the data of the light scattering mode:

[0078]

[0079] where yi represents the signal intensity corresponding to the i-th angular position in each light scattering data, and y′ i represents the normalized value, and the normalized light scattering data is scaled to the range of 0 and 1;

[0080] Before the data of the two modes are input into the model after making the fusion dataset, a unified standardization will be carried out again. Furthermore, the algorithm formula of step 306 is:

[0081] X' s = X s ·W

[0082] where X’ s is the light scattering data set after dimensionality reduction, and the original data is projected into a new feature space so that the projected data has the maximum variance. W is the projection matrix obtained by the algorithm, and by multiplying the original data by the projection matrix, the light scattering data set after dimensionality reduction can be obtained. n s is the dimension of the light scattering data, n r is the dimension of the Raman spectrum data, and m is the number of collected data samples;

[0083] In the data fusion of step 307, to ensure the balance of the data dimension after fusion, the Raman spectrum is copied to form a Raman data set that matches the light scattering data channel, and then the dual-modal data sets are spliced to form a fused spectrum data set. The specific process is as follows:

[0084] X r1 = X r2 = X r

[0085] X rs = [X r1 , X r2 , X′ s1 , X′ s2

[0086] where X r1 and X r2 are the Raman spectrum data set matrices after the copying operation, X’ s1 and X’ s2 are the two-channel light scattering data set matrices after dimensionality reduction, and X rs is the final multi-modal spectrum fusion data set for model input.

[0087] Furthermore, the multi-modal spectrum classification composite model in step 4 includes a local feature extraction network and a full-spectrum correlation analysis network. The data performs the following operations in the two networks:

[0088] (1) The local feature extraction network of the model is composed of a convolutional neural network; in this network, the data is first sent to the feature extraction part; after passing through the feature extraction layer, the data is sent to the classifier corresponding to the convolutional neural network module; where the feature map after feature extraction is expanded into a one-dimensional vector, and then passes through the fully connected layer. Activation and Dropout operations are performed in every two fully connected layers, and finally, weighted fusion is performed with the full-spectrum correlation analysis network before the Softmax layer; ​

[0089] (2) The full-spectrum correlation analysis network of the model is based on the Transformer architecture; in this network, the preprocessed input spectrum first undergoes block embedding and position embedding operations to obtain several spectral blocks containing position information, and then is fed into the Transformer encoder. In each encoder block, the input data forms a residual connection, undergoes a Layernorm operation once, and then is fed into the multi-head attention layer. After a Dropout operation, it is added to the previous residual to form a new residual; the data then undergoes a second Layernorm operation, and after passing through the forward propagation layer, Dropout layer, and adding to the second residual, it is input into the next encoder block; the data after passing through the encoder undergoes an overall Layernorm operation once and then is fed into the classifier of the Transformer module. Finally, it is weighted and fused with the local feature extraction network before the Softmax layer;

[0090] (3) After obtaining the data passed by the two networks, the model will weight the results with certain weights;

[0091] (4) Convert the weighted result into a probability vector for output, as shown in the formula:

[0092]

[0093] where z represents the feature array passed by the previous layer, and p represents the output probability distribution; in the output probability vector, the probability values of all classes are between 0 and 1, and the sum of all probabilities is 1.

[0094] Furthermore, in the selection of the model integration weights, it is divided according to the running results of a single model on the spectral data. The calculation formula for the model fusion weights is as follows:

[0095]

[0096] where yi is the prediction result of a single model, is the final prediction result, w is the weighted weight of the corresponding model, and A is the classification accuracy of the corresponding model on the running results of the spectral data.

[0097] Further, the training of the composite model includes:

[0098] Step 501, manage using a deep learning framework,

[0099] Step 502, use a local programming IDE,

[0100] Step 503, perform the training calculation of the model.

[0101] Compared with the prior art, the present invention has the following beneficial effects:

[0102] Combined with the principle of spectroscopy, in order to increase the scientific nature of model classification, a deep learning network structure adapted to the spectral characteristics is designed according to the data characteristics of the spectrum itself. The convolutional neural network structure widely used in current spectral analysis can effectively capture local spectral features, but it lacks sensitivity to the global correlation of spectral data. The attention mechanism of Transformer can perceive the global dependence relationship of spectra within their characteristic ranges in different signal segments through data training. By integrating the advantages of both, the composite model constructed by combining the two can effectively reduce the recognition errors caused by similar local spectra during the process of identifying tissue properties.

[0103] The baseline calibration algorithm of the present invention, as an adaptive baseline fitting method, overcomes the disadvantages of traditional processing methods that require users to manually adjust and collect prior information such as the spectral peak position information. It approximates the baseline position to be fitted through a cyclic iterative method, improves the overall baseline removal effect in the end, and improves the efficiency of baseline calibration.

[0104] Compared with traditional staining methods, the present invention has many advantages:

[0105] (1) It has wide applicability and can be applied to different types of tumor tissues and cells, etc. The Raman interference of water is small, and it can be used in liquid samples such as liquid biopsies.

[0106] (2) It is label-free and does not require complex pretreatment operations such as staining the samples, and can be sampled immediately after collection.

[0107] (3) It has little damage to the samples. By using the laser irradiation method, the integrity of the samples can be guaranteed during the analysis, and the overall repeatability is high.

[0108] (4) There is a large space for automation in the process. Compared with traditional staining analysis methods, it greatly simplifies the operation steps of pathologists, takes less time, has strong timeliness, and is suitable for on-site clinical detection.

[0109] The composite model can effectively learn the multi-modal spectral features of different types of tissues, achieve high-accuracy tumor tissue identification, with the classification accuracy and precision reaching 97.00% and 97.14% respectively, and is significantly better than the results of single-modal input of Raman spectroscopy and light scattering, fully demonstrating that collecting multi-modal spectral data for training can greatly improve the reliability of model decision-making.

[0110] To verify the final performance of the composite model, the present invention selected several classification methods commonly used in spectral analysis to evaluate the performance of the composite model in this paper. Partial least squares-discriminant analysis (PLS-DA), support vector machine (SVM), linear discriminant analysis (LDA), ResNet, Raman-Trans, and the composite model Combined-Net of this paper were selected respectively. The first three methods are classical machine learning methods commonly used in spectral classification, and the last two methods are deep learning networks that have achieved good results in spectral classification in recent years, namely a typical convolutional neural network and a Transformer architecture. The input data for each model was preprocessed in the same way as in this paper.

[0111] Table 1 Comparison of the performance of each model

[0112]

[0113]

[0114] As can be seen from the data in Table 1, in the scenario of colorectal tumor classification, all indicators of the composite model are 97% and above, reaching the optimal level. It shows that the composite model can extract spectral data features more effectively and achieve higher classification performance compared with traditional machine learning methods or single deep learning networks.

[0115] To verify the superiority of multimodal spectroscopy compared with single spectroscopy, the present invention separately used the data of the light scattering modality and the Raman spectroscopy modality to train the composite model. As shown in Table 2, the classification accuracies obtained were 64.30%, 91.25%, and 97.00% respectively. In terms of classification performance, the decision-making performance of the model trained with multimodal spectral data is significantly better than that of the model trained with single Raman spectral data or elastic light scattering data, fully proving that the multimodal spectral data combining the elastic light scattering signal reflecting the morphological characteristics of colorectal tissue and the Raman spectral signal reflecting the biochemical characteristics of colorectal tissue can significantly improve the classification performance of the model.

[0116] Table 2 Comparison of the performance of models with different types of input data

[0117]

[0118] The proposed composite model can effectively learn the multimodal spectral features of different types of tissues, realize the detection of colorectal tumor tissues with high accuracy, reaching 97.00% and 97.14% in classification accuracy and precision respectively, and is significantly better than the results of single-modal input of Raman spectroscopy and light scattering, fully proving that collecting multimodal spectral data for training can greatly improve the reliability of model decision-making. Description of the drawings

[0119] Figure 1 This is the flowchart of the present invention.

[0120] Figure 2 This is the overall algorithm flowchart of baseline calibration airPLS (Adaptive Iterative Reweighted Penalized Least Squares) in the present invention. Specific Embodiments

[0121] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0122] Example 1, Analysis of Colorectal Tumor Tissue Sections

[0123] Step 01, Collection of Raman spectroscopy data.

[0124] When collecting spectral signals, a gold-plated glass slide is used as a carrier to enhance the tissue Raman signal. The collection process is as follows:

[0125] Step 011: First, place the sample on the gold-plated glass slide, fix it on the stage of the Raman spectrometer, set the laser power of the instrument to 25%, set the integration time to 5 s, adjust the probe height to ensure a proper laser focusing position, and pre-collect the dark current of the device.

[0126] Step 012: Scan the sample in the full spectral range (65 - 3350 cm-1), check the characteristic peaks of the substance, select a suitable scanning section, and optimize parameters such as the integration exposure time according to the pre-test results.

[0127] Step 013: Then, uniformly select the tissue collection positions, collect and record the Raman spectra in sequence according to the marked points, collect about 10 times at each point, subtract the pre-collected dark current from the collection result, and export it in the csv file format on the host computer software.

[0128] Step 014: Use the Numpy package of Python to make an original data file with the npy suffix based on the collected csv file for subsequent spectral preprocessing operations.

[0129] Step 02, Collection of elastic light scattering data.

[0130] Step 21: First, isolate the external light, adjust the bias voltage of the IV conversion amplifier module to zero the whole device. Make a frozen section sample of the fresh sample to be collected data using a cryostat microtome.

[0131] Step 022: Then, fix the made sample slice on the glass slide on the electric turntable, use the on-board button of the MCU board to adjust the output PWM wave frequency, set a suitable stepping motor speed, select appropriate sampling rate, recording time, recording channel and other parameters on the host computer software, and uniformly select the collection points to sample and obtain the original data.

[0132] Step 023: Subsequently, use DAQexpress software to export the original data in csv file format, and use the Numpy library in Python to create the original data file with the suffix of npy for subsequent data processing and model training.

[0133] Step 03: Preprocess the spectral data.

[0134] Step 031: Process the Raman spectral data. Before collecting the Raman spectral signal of the tissue, collect the dark current of the acquisition device itself and subtract the dark current value from the final result.

[0135] Step 032: Preprocess the Raman spectrum of colorectal pathological tissue. Smooth filtering is used to reduce noise interference, baseline correction is used to prevent the influence of fluorescence signal on the baseline, Raman shift range truncation is used to select the data range with rich features, and normalization is used to balance the data dimension. In smooth filtering, the common second-order Savitzky-Golay (SG) smooth filtering in signal processing is adopted in the experiment, and the sliding window length parameter is selected as 21. At the same time, baseline correction, range truncation, and normalization based on the adaptive iteratively reweighted penalized least squares method are performed on the data.

[0136] Step 033: Process the light scattering data. Use the method of subtracting the collected data from the baseline noise interference of the pre-experiment collected instrument itself to remove the baseline drift phenomenon of the light scattering signal. Then, perform simple preprocessing on the original light scattering signal by using the normalization method.

[0137] Step 04: Multimodal data fusion.

[0138] Step 041: For the dual-channel light scattering data set after the preprocessing process in the previous text. First, use the PCA algorithm to reduce the dimension of the light scattering data to be equal to the dimension of the Raman spectral data.

[0139] Step 042: Project the original data into the new feature space so that the projected data has the maximum variance. Multiply the original data by the projection matrix obtained through the algorithm to obtain the light scattering data set with reduced dimension.

[0140] Step 043: Next, perform data fusion on the dual-channel light scattering data and Raman data. To ensure the balance of the data dimension after fusion, copy the Raman spectrum to form a Raman data set that matches the light scattering data channel. Then, horizontally splice the dual-modal data sets to form a fused spectral data set.

[0141] Step 05: Construct a classification composite model for the fused spectrum;

[0142] Step 51: First, input the data into the composite model; the multimodal spectral classification composite model includes a local feature extraction network and a full-spectrum correlation analysis network.

[0143] The following operations are performed in the local feature extraction network:

[0144] (1) The local feature extraction network of the model is composed of a convolutional neural network. In this module, the data is first fed into the feature extraction part. This part consists of 5 Blocks. The first two Blocks are composed of two layers of one-dimensional convolutional layers and one layer of max pooling layer; the last three Blocks are composed of three layers of one-dimensional convolutional layers and one layer of max pooling layer.

[0145] (2) After passing through the feature extraction layer, the data is fed into the classifier corresponding to the convolutional neural network module.

[0146] (3) The feature map after feature extraction will be unfolded into a one-dimensional vector through a flattening layer.

[0147] (4) Then it passes through three fully connected layers, and ReLU activation and Dropout operations are performed once between every two fully connected layers.

[0148] (5) Finally, it is weighted and fused with the full-spectrum correlation analysis network before the Softmax layer.

[0149] The following operations are performed in the full-spectrum correlation analysis network:

[0150] (1) The full-spectrum correlation analysis network of the model is based on the Transformer architecture;

[0151] (2) The input spectrum after preprocessing first undergoes block embedding and position embedding operations to obtain a number of spectral blocks containing position information.

[0152] (3) It is fed into the Transformer encoder, which has 8 encoder blocks. In each encoder block, the input data will form a residual connection and be fed into the multi-head attention layer after passing through a Layernorm once.

[0153] (4) After a Dropout operation, it is added to the previous residual to form a new residual.

[0154] (5) Immediately after the data undergoes a second Layernorm, it is added to the second residual through the feed-forward layer and Dropout layer and then input into the next encoder block.

[0155] (6) The data after passing through the encoder undergoes an overall Layernorm once and then is fed into the classifier of the Transformer module.

[0156] (7) Finally, it is weighted and fused with the local feature extraction network before the Softmax layer.

[0157] Step 52. Then, after obtaining the data transmitted from the previous two modules, the model will weight the results with certain weights.

[0158] Step 53. Use the Softmax function to convert the weighted results into a probability vector for output, as shown in the formula:

[0159]

[0160] where z represents the feature array transmitted from the previous layer, and p represents the output probability distribution. In the output probability vector, the probability values of all classes are between 0 and 1, and the sum of all probabilities is 1.

[0161] In the selection of the model integration weights, they are divided according to the running results of a single model on the spectral data. The calculation formula for the model fusion weights is as follows:

[0162]

[0163] where yi is the prediction result of the individual model, is the final prediction result, w is the weighted weight corresponding to the model, and A is the classification accuracy of the corresponding model's running result on the spectral data. After verification through classification experiments, substituting the classification accuracies of the local feature extraction network and the full-spectrum correlation analysis network in data classification into the above weighted weight formula, the weighted weights are approximately 0.512 and 0.488.

[0164] Step 06. Composite model training.

[0165] Step 061. Use Python as the main programming language,

[0166] Step 062. Use the deep learning framework Pytorch, and the third-party libraries of Python are managed by Anaconda;

[0167] Step 063. The local programming IDE uses Pycharm;

[0168] Step 064. The training of the model uses CUDA technology on a Linux server to accelerate the training calculation with GPU.

[0169] Step 07. Obtain the result prediction value according to the model.

[0170] Example 2. Analysis of bone marrow smears of leukemia cells

[0171] Step 01. Collection of Raman spectral data.

[0172] When collecting the spectral signal, the collection process is as follows:

[0173] Step 011: First, apply the sample on a glass slide, fix it on the stage of the Raman spectrometer. Set the laser power of the instrument to 25%, the integration time to 5 s, adjust the probe height to ensure an appropriate laser focusing position, and pre-acquire the dark current of the device.

[0174] Step 012: Scan the sample in the full spectral range (65 - 3350 cm-1), check the characteristic peaks of the substance, select an appropriate scanning segment, and optimize parameters such as the integration exposure time according to the pre-test results.

[0175] Step 013: Then, uniformly select the tissue collection positions, sequentially collect and record the Raman spectra according to the marked points. Each point is collected about 10 times. Subtract the pre-acquired dark current from the collection results and export them in the csv file format on the host computer software.

[0176] Step 014: Use the Numpy package of Python to create an original data file with the npy suffix based on the collected csv file for subsequent spectral preprocessing operations.

[0177] Step 02: Collection of elastic light scattering data.

[0178] Step 021: First, isolate the external light, adjust the bias voltage of the IV conversion amplifier module to zero the overall device. Uniformly apply the fresh sample to be collected on the glass slide.

[0179] Step 022: Then, fix the prepared sample slide on the glass slide on the electric turntable, use the on-board button of the MCU board to adjust the output PWM wave frequency, set an appropriate stepping motor speed, select appropriate sampling rate, recording time, recording channel and other parameters on the host computer software, and uniformly select the collection points for sampling to obtain the original data.

[0180] Step 023: Then, use the DAQexpress software to export the original data csv file, and use the Numpy library of Python to create an original data file with the npy suffix for subsequent data processing and model training.

[0181] Step 03: Spectral data preprocessing.

[0182] Step 031: Raman spectral data processing: Before collecting the Raman spectral signal of the tissue, collect the dark current of the collection device itself and subtract the dark current value from the final result.

[0183] Step 032: Preprocess the Raman spectrum. Perform smoothing filtering to reduce noise interference, baseline correction to prevent the influence of fluorescence signals on the baseline, Raman shift range truncation to select a data range with rich features, and normalization to balance the data dimensions. In the smoothing filtering, the experiment adopted the common second-order Savitzky-Golay (SG) smoothing filtering in signal processing, with the sliding window length parameter selected as 21. At the same time, baseline correction, range truncation, and normalization based on the adaptive iteratively reweighted penalized least squares method were performed on the data.

[0184] Step 033: Process the light scattering data. Remove the baseline drift of the light scattering signal by subtracting the collected data from the baseline noise interference of the pre-experimentally collected instrument itself. Then, preprocess the original light scattering signal using the normalization method.

[0185] Step 04: Multimodal data fusion.

[0186] Step 041: The dual-channel light scattering data set after the preprocessing process in the previous text. First, use the PCA algorithm to reduce the dimension of the light scattering data to be equal to the dimension of the Raman spectrum data.

[0187] Step 042: Project the original data into a new feature space so that the projected data has the maximum variance. Multiply the original data by the projection matrix obtained through the algorithm to obtain the dimension-reduced light scattering data set.

[0188] Step 043: Next, perform data fusion on the dual-channel light scattering data and Raman data. To ensure the balance of the data dimensions after fusion, copy the Raman spectrum to form a Raman data set that matches the light scattering data channel. Then, horizontally splice the dual-modal data sets to form a fused spectrum data set.

[0189] Step 05: Construct a classification composite model for the fused spectrum;.

[0190] Step 51: First, input the data into the composite model; the multimodal spectrum classification composite model includes a local feature extraction network and a full-spectrum correlation analysis network.

[0191] Perform the following operations in the local feature extraction network:

[0192] (1) The local feature extraction network of the model is composed of a convolutional neural network. In this module, the data is first sent to the feature extraction part. This part consists of 5 Blocks. The first two Blocks are composed of two layers of one-dimensional convolutional layers and one layer of max-pooling layer; the last three Blocks are composed of three layers of one-dimensional convolutional layers and one layer of max-pooling layer.

[0193] (2) After passing through the feature extraction layer, the data is sent to the classifier corresponding to the convolutional neural network module.

[0194] (3) The feature map after feature extraction will be unfolded into a one-dimensional vector through a flattening layer.

[0195] (4) Then, it will pass through three fully connected layers, with ReLU activation and Dropout operations performed once between every two fully connected layers.

[0196] (5) Finally, it will be weighted and fused with the full-spectrum correlation analysis network before the Softmax layer.

[0197] The following operations are performed in the full-spectrum correlation analysis network:

[0198] (1) The full-spectrum correlation analysis network of the model is based on the Transformer architecture.

[0199] (2) The input spectrum after preprocessing first undergoes block embedding and positional embedding operations to obtain several spectral blocks containing positional information.

[0200] (3) It is fed into the Transformer encoder, which has 8 encoder blocks. In each encoder block, the input data will form a residual connection and be fed into the multi-head attention layer after one Layernorm.

[0201] (4) After a Dropout operation, it is added to the previous residual to form a new residual.

[0202] (5) Immediately after the data undergoes a second Layernorm, it is added to the second residual through the feed-forward layer and Dropout layer and then input into the next encoder block.

[0203] (6) The data after passing through the encoder undergoes an overall Layernorm and then is fed into the classifier of the Transformer module.

[0204] (7) Finally, it is weighted and fused with the local feature extraction network before the Softmax layer.

[0205] Step 52: Then, after obtaining the data passed from the previous two modules, the model will weight the results with certain weights.

[0206] Step 53: Use the Softmax function to convert the weighted results into a probability vector for output, as shown in the formula:

[0207]

[0208] where z represents the feature array passed from the previous layer, and p represents the output probability distribution. In the output probability vector, the probability values of all classes are between 0 and 1, and the sum of all probabilities is 1.

[0209] In the selection of model integration weights, they are divided based on the running results of a single model on spectral data. The calculation formula for the model fusion weight is as follows:

[0210]

[0211] Where yi is the prediction result of the individual model, is the final prediction result, w is the weighted weight of the corresponding model, and A is the classification accuracy of the corresponding model's running result on spectral data. After verification through classification experiments, substituting the classification accuracies of the local feature extraction network and the full-spectrum correlation analysis network in data classification into the above weighted weight formula, the weighted weights are approximately 0.512 and 0.488.

[0212] Step 06: Composite model training.

[0213] Step 061: Use Python as the main programming language;

[0214] Step 062: Use the deep learning framework Pytorch, and manage the third-party libraries of Python with Anaconda;

[0215] Step 063: Use Pycharm as the local programming IDE;

[0216] Step 064: Train the model on a Linux server using CUDA technology for GPU-accelerated training calculations.

[0217] Step 07: Obtain the result prediction value according to the model.

[0218] The above shows and describes the basic principles, main features, and advantages of the present invention. Those of ordinary skill in the art should understand that the above embodiments do not limit the protection scope of the present invention in any form. Any technical solutions obtained by means of equivalent replacement and the like fall within the protection scope of the present invention. The parts not involved in the present invention are the same as the prior art or can be implemented using the prior art.

Claims

1. A method for analyzing tissue sections and body fluid smears by integrating elastic scattering spectroscopy and Raman spectroscopy, characterized in that, It includes the following steps: Step 1, Raman spectrum data acquisition; Step 2, elastic light scattering data acquisition; Step 3, data processing; Step 4, construction of a classification composite model based on the fused spectrum; Step 5, composite model training; Step 6, obtaining the result prediction value according to the model.

2. The method for analyzing tissue sections and body fluid smears by integrating elastic scattering spectroscopy and Raman spectroscopy according to claim 1, wherein, The said Step 1 includes: Step 101, first fix the sample on the Raman spectrometer stage; Step 102, set the instrument laser power and integration time; Step 103, adjust the probe height to ensure an appropriate laser focusing position; Step 104, pre-acquire the dark current of the device; Step 105, scan the sample, check the characteristic peaks of the substance, and select an appropriate scanning section; Step 106, optimize the integration exposure time parameter according to the pre-test result; Step 107, uniformly select the tissue collection positions, and sequentially collect and record the Raman spectra according to the marked points; Step 108, subtract the pre-acquired dark current from the acquisition result; Step 109, export the file; Step 110, convert the format of the collected file.

3. The method for analyzing tissue sections and body fluid smears by integrating elastic scattering spectroscopy and Raman spectroscopy according to claim 1, characterized in that The said Step 2 includes: Step 201, first isolate the external light; Step 202, adjust the bias voltage of the signal amplification module to zero the light scattering acquisition device; Step 203, then fix the fabricated sample slice; Step 204, adjust the output frequency; Step 205, set the stepping motor speed, select an appropriate sampling rate, recording time, and recording channel; Step 206, uniformly select the collection points for sampling to obtain the original data; Step 207, export the file, Step 208, convert the format of the collected file.

4. A method for analyzing tissue sections and body fluid smears by integrating elastic scattering spectroscopy and Raman spectroscopy according to claim 1, characterized in that, The said Step 3 includes: Step 301, perform smoothing and denoising on the spectral data signal with a filter; Step 302, perform baseline correction; Step 303, intercept in the Raman spectrum signal fingerprint region; Step 304, perform a normalization operation on the signals before the Raman signal and the light scattering signal are used as model inputs; Step 305: Define the processed dual-channel light scattering data sets as X s1 , X s2 , and the Raman data set as X r ; Step 306, reduce the dimension of the light scattering data to be equal to the dimension of the Raman spectrum data; Step 307, perform data fusion on the light scattering data and Raman data of the dual channels.

5. A method for analyzing tissue sections and body fluid smears by integrating elastic scattering spectroscopy and Raman spectroscopy according to claim 4, characterized in that In the smoothing and denoising operation of the said Step 301: for the spectral data y(t) to be processed, use a polynomial to fit the data: where a i is the constant coefficient of each term of the polynomial. Select the window length and calculate the mean square error:

6. The method for analyzing tissue sections and body fluid smears by integrating elastic scattering spectroscopy and Raman spectroscopy according to claim 4, characterized in that The process of the said Step 302 for baseline correction is as follows: The fitting error F and the data smoothness R are calculated as: where x is the original spectral signal, z is the spectral signal after fitting, and m is the signal length; at the same time, the smoothing parameter λ is introduced into the calculation of the total penalty term Q to balance the contradiction between the fitting error and the smoothness, and the total penalty term Q weighted by it for smoothness is expressed as: where λ is the smoothing parameter used to control the smoothness of the fitting; The penalty term function after the t-th iteration calculation is expressed as: Among them represents the weight of the t-th iteration, represents the fitting baseline for this time; during the fitting process, the weight is updated based on the difference between the original signal and the fitting baseline of the previous iteration. If the value of a certain data point is greater than the current fitting baseline, it is determined to be possibly part of a peak and will be ignored in the next iteration, that is, the weight is 0. The function for weight update is expressed as: In the above iteration process, the algorithm will stop when reaching the maximum number of iterations or meeting certain conditions, and its termination condition is shown in the formula: |d t |<0.001×|x| where d t is the difference between the original signal and the fitting baseline in the current iteration.

7. A method for analyzing tissue sections and body fluid smears by integrating elastic scattering spectroscopy and Raman spectroscopy according to claim 4, characterized in that, The normalization operation of the said Step 304 is as follows: The Raman data is standardized in the dataset: where j represents the j-th feature of the spectrum, and μ j represents the mean of all features in this feature dimension, and σ j is its standard deviation, and the calculation formula is: The data of the light scattering mode is normalized: where yi represents the signal intensity corresponding to the i-th angular position in each light scattering data, and y′ i represents the normalized value, and the normalized light scattering data is scaled to the range of 0 and 1; Before the data of the two modes are input into the model after making the fusion dataset, a unified standardization will be performed again.

8. A method for analyzing tissue sections and body fluid smears by integrating elastic scattering spectroscopy and Raman spectroscopy according to claim 4, characterized in that, The algorithm formula of the said Step 306 is: X' s = X s · W Among them X’ s is the light scattering data set after dimensionality reduction and is to project the original data into a new feature space so that the projected data has the maximum variance. W is the projection matrix obtained through the algorithm and by multiplying the original data by the projection matrix, the light scattering data set after dimensionality reduction can be obtained, n s is the dimension of the light scattering data, n r is the dimension of the Raman spectrum data, and m is the number of collected data samples; In the data fusion of step 307, to ensure the balance of the data dimension after fusion, the Raman spectrum is replicated to form a Raman data set matching the light scattering data channel. Then, the dual-modal data sets are spliced to form a fused spectrum data set. The specific process is as follows: X r1 = X r2 = X r X rs = [X r1 , X r2 , X' s1 , X' s2 ​ where X r1 and X r2 are the Raman spectral dataset matrices after the copying operation, and X’ s1 and X’ s2 are the two-channel light scattering dataset matrices after dimensionality reduction, and X rs is the multimodal spectral fusion dataset finally used for model input.

9. The method for analyzing tissue sections and body fluid smears by integrating elastic scattering spectroscopy and Raman spectroscopy according to claim 1, wherein The classification composite model of the fused spectrum in step 4 includes a local feature extraction network and a full-spectrum correlation analysis network. The data performs the following operations in the two networks: (1) The local feature extraction network of the model is composed of a convolutional neural network. In this network, the data is first sent to the feature extraction part. After passing through the feature extraction layer, the data is sent to the classifier corresponding to the convolutional neural network module. The feature map after feature extraction is expanded into a one-dimensional vector, and then passes through the fully connected layer. Activation and Dropout operations are performed in every two fully connected layers. Finally, weighted fusion is performed with the full-spectrum correlation analysis network before the Softmax layer. (2) The full-spectrum correlation analysis network of the model is based on the Transformer architecture. In this network, the input spectrum after preprocessing first undergoes block embedding and position embedding operations to obtain several spectral blocks containing position information, and then is sent into the Transformer encoder. In each encoder block, the input data forms a residual connection, passes through a Layernorm once, and then is sent into the multi-head attention layer. After a Dropout operation, it is added to the previous residual to form a new residual. The data then undergoes a second Layernorm, and after passing through the feed-forward layer and the Dropout layer, it is added to the second residual and input into the next encoder block. The data after passing through the encoder undergoes an overall Layernorm once and then is sent into the classifier of the Transformer module. Finally, weighted fusion is performed with the local feature extraction network before the Softmax layer. (3) After obtaining the data transmitted by the two networks, the model will weight the results by a certain weight. (4) The weighted result is converted into a probability vector for output, as shown in the formula: where z represents the feature array transmitted by the previous layer, and p represents the output probability distribution. In the output probability vector, the probability values of all classes are between 0 and 1, and the sum of all probabilities is 1.

10. A method for analyzing tissue sections and body fluid smears by integrating elastic scattering spectroscopy and Raman spectroscopy according to claim 9, characterized in that, In the selection of the model integration weight, it is divided according to the running results of a single model on the spectral data. The calculation formula for the model fusion weight is as follows: where yi is the prediction result of an individual model, is the final prediction result, w is the weighted weight of the corresponding model, and A is the classification accuracy of the corresponding model's running result on spectral data.

Images