An implementation method of a multimodal learning convolutional neural network model based on molecular spectral feature acquisition and fusion

Abstract

The present invention combines deep learning technology with micro-polarized imaging, micro-fluorescence imaging and micro-Raman spectroscopy technology, and proposes an implementation method of a multimodal learning convolutional neural network (MML-CNN) model based on molecular spectral feature acquisition and fusion. The method includes: (1) using the local binary pattern algorithm (LBP) to extract the features of polarized images at different angles and superimpose them to achieve pixel-level fusion; (2) using the convolution layer to extract the features of polarized images and fluorescence images respectively and superimpose them to achieve feature-level fusion; (3) using a one-dimensional convolutional neural network (1D-CNN) to discriminate the Raman spectrum, and combine it with the image results, and then use support vector machine (SVM), partial least squares (PLS) or Fisher discriminant method for statistical analysis to achieve decision-level fusion. The present invention combines multimodal information acquisition, multimodal learning, and multi-feature fusion methods, thereby providing a new idea for fast, accurate and intelligent diagnosis of cancer pathology.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology and optical detection technology such as microscopic polarization imaging technology, microscopic fluorescence imaging technology, and microscopic Raman spectroscopy technology, and in particular relates to an implementation method of a multimodal learning convolutional neural network model based on molecular spectral image feature acquisition and fusion. Background Art

[0002] With the development of deep learning technology, the application of neural network models in the medical field has become popular. However, the medical field is different from other fields. The human body is a complex biological system, and a single detection method usually cannot produce accurate results. Therefore, clinical diagnosis of diseases often requires a comprehensive analysis of multiple detection technologies, which requires the neural network model to have the ability to process multiple data types at the same time.

[0003] Multimodal fusion is an effective method, which mainly includes four methods: signal-level fusion, pixel-level fusion, feature-level fusion and decision-level fusion. Existing multimodal fusion neural network models used in the medical field often only involve one of them, and only achieve dual-modal fusion. Therefore, the application of existing neural network models in disease diagnosis is still limited, and its potential has not been fully developed. On this basis, it is of great significance to develop a neural network model that includes multiple feature fusion methods and can process multiple data types at the same time.

[0004] On the other hand, traditional or clinical detection instruments for cancer tissues are often expensive, provide limited data information, and have a low signal-to-noise ratio. On this basis, new optical detection technologies such as Raman spectroscopy detection technology, fluorescence detection technology, and polarized light detection technology have outstanding advantages and are moving towards clinical use, and corresponding detection instruments are constantly being developed. In summary, it will be very meaningful to apply a variety of multimodal fusion methods to the above-mentioned optical detection technologies and use them to establish a multimodal learning convolutional neural network (MML-CNN) model to achieve high-precision intelligent diagnosis of cancer tissues. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide an implementation method of a multimodal learning convolutional neural network model based on the acquisition of multiple molecular spectral features and including three feature fusion methods, so as to integrate the three multimodal fusion methods into a single neural network model, so that it can process three types of data at the same time and achieve high-precision intelligent diagnosis of cancer tissues.

[0006] Technical solution: A method for implementing a multimodal learning convolutional neural network model based on molecular spectral feature acquisition and fusion, comprising the following steps:

[0007] Step 1: Collect micro-polarization images, micro-fluorescence images and micro-Raman spectra at different angles from cancer tissue slice samples;

[0008] Step 2: Use data augmentation methods to expand the data to reduce the risk of overfitting;

[0009] Step 3: Use the local binary pattern (LBP) algorithm to extract the texture features of the microscopic polarization images at different polarization angles, and superimpose them to obtain a fused polarization image to achieve pixel-level fusion;

[0010] Step 4: Use several symmetrical convolutional layers to downsample the fused polarization image and fluorescence image respectively, extract features, and superimpose the results to obtain the fused image features to achieve feature-level fusion;

[0011] Step 5: Establish a one-dimensional convolutional neural network 1D-CNN model and apply it to the Raman spectral data. The result is combined with the result of the fusion image and statistically analyzed using support vector machine SVM, partial least squares PLS or Fisher discriminant method to achieve decision-level fusion.

[0012] Step 6: The model is used to learn and predict the corresponding data of cancer samples to achieve high-precision intelligent diagnosis of cancer; the model is trained according to the k-fold cross-validation method with k=10, and then the model performance is tested with the test set data.

[0013] Furthermore, in step 1, the cancer tissue sample and the normal tissue sample are cut into slices of a certain thickness using a cryogenic microtome for collecting multi-mode data such as microscopic polarization images, microscopic fluorescence images, and microscopic Raman spectra.

[0014] Furthermore, step 2 specifically includes the following points: a certain proportion of data is separated from the total data as a test set, and the remaining data is expanded by data enhancement methods to reduce the risk of overfitting;

[0015] For image data, we use horizontal or vertical flipping, rotation at different angles, arbitrary ratio cropping, and adding 2% Gaussian random noise to expand the data volume;

[0016] For spectral data, the data volume was expanded by shifting left and right by 1 to 2 wave numbers, multiplying multiple spectra by different ratios with a total ratio sum of 100%, and then summing them, adding 2% Gaussian random noise, etc.

[0017] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0018] The implementation method of the multimodal learning convolutional neural network model based on molecular spectral feature acquisition and fusion proposed in the present invention has two main effective effects: 1) The present invention provides a specific idea of ​​integrating multiple multimodal fusion methods into a single neural network model, so that the model has the ability to process multiple different types of data at the same time; 2) The above model is applied in clinical practice to achieve fast and accurate intelligent diagnosis of cancer and provide a reference for doctors. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is the specific structure of a multimodal learning convolutional neural network model based on molecular spectral feature acquisition and fusion provided by the present invention. DETAILED DESCRIPTION

[0020] Now, by referring to exemplary embodiments, the objects and functions of the present invention and the methods for achieving these objects and functions will be explained. However, the present invention is not limited to the exemplary embodiments disclosed below; it can be implemented in different forms. The essence of the description is only to help those skilled in the relevant art to comprehensively understand the specific details of the present invention.

[0021] Figure 1 The specific structure of a multimodal learning convolutional neural network model based on molecular spectral feature acquisition and fusion provided by the present invention is shown, and the structure can be subdivided into three parts: pixel-level fusion, feature-level fusion and decision-level fusion.

[0022] For the pixel-level fusion part, the LBP algorithm is used to extract the texture features of microscopic polarization images with different polarization angles, and they are superimposed to obtain a fused polarization image to achieve pixel-level fusion.

[0023] For the feature-level fusion part, the above-mentioned fused polarized image is input into several convolutional layers, the data size is compressed by downsampling, and the corresponding polarized image features are extracted. At the same time, the fluorescence image is input into several convolutional layers, the data size is compressed by downsampling, and the corresponding fluorescence image features are extracted (the convolutional layer used for the fluorescence image has a symmetrical structure with the convolutional layer used for the polarized image). The polarized image features are then superimposed with the fluorescence image features to achieve feature-level fusion. In order to ensure that the MML-CNN model is easy to train, several Resblock structures are used for further feature extraction of the fused features of the image, while maintaining the feature size unchanged. The fused features are then expanded and a mapping relationship is established with the sample label through the fully connected layer.

[0024] For the decision-level fusion part, a 1D-CNN model is established to process Raman spectra. The model has several convolutional layers, extracts spectral features in a downsampling manner, then expands the spectral features and establishes a mapping relationship with the sample label through a fully connected layer. Subsequently, the output results of the feature-level fusion neural network of the image are combined with the output results of the 1D-CNN of the Raman spectrum using methods such as SVM, PLS or Fisher discriminant to make a high-precision diagnosis of cancer samples and achieve decision-level fusion.

[0025] After the MML-CNN model was built, the cancer tissue samples and normal tissue samples were cut into slices of a certain thickness using a cryostat for the acquisition of microscopic polarization images, microscopic fluorescence images, and microscopic Raman spectra. After obtaining enough data, a certain proportion of the data was separated from it for use as a test set, and the remaining data was expanded by data enhancement to reduce the risk of overfitting. For image data, horizontal or vertical flipping, rotation at different angles, arbitrary proportion cropping, and adding 2% Gaussian random noise were used to expand the data volume; for spectral data, left and right translation of 1 to 2 wave numbers, multiplication of multiple spectra by different proportions (the total proportion sum is 100%) and then summing, and adding 2% Gaussian random noise were used to expand the data volume. The MML-CNN model was trained with the expanded data according to the k-fold cross-validation method with k=10, and then the performance of the model was tested with the test set data to achieve high-precision diagnosis of cancer samples.

[0026] Matters not covered by the present invention are known technologies.

[0027] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for implementing a multimodal learning convolutional neural network model based on molecular spectral feature acquisition and fusion, characterized in that: The following steps are involved: Step 1: Collect micro-polarization images, micro-fluorescence images and micro-Raman spectra at different angles from cancer tissue slice samples; Step 2: Use data augmentation methods to expand the data to reduce the risk of overfitting; Step 3: Use the local binary pattern (LBP) algorithm to extract the texture features of the microscopic polarization images at different polarization angles, and superimpose them to obtain a fused polarization image to achieve pixel-level fusion; Step 4: Use several symmetrical convolutional layers to downsample the fused polarization image and fluorescence image respectively, extract features, and superimpose the results to obtain the fused image features to achieve feature-level fusion; Step 5: Establish a one-dimensional convolutional neural network 1D-CNN model and apply it to the Raman spectral data. The result is combined with the result of the fusion image and statistically analyzed using support vector machine SVM, partial least squares PLS or Fisher discriminant method to achieve decision-level fusion. Step 6: The multimodal learning convolutional neural network model is used to learn and predict the corresponding data of cancer samples to achieve high-precision intelligent diagnosis of cancer; the model is trained according to the k-fold cross-validation method with k=10, and then the model performance is tested with test set data.

2. The method according to claim 1, characterized in that The step 2 includes the following points: A certain proportion of data is separated from the total data as a test set, and the remaining data is expanded through data enhancement methods to reduce the risk of overfitting; For spectral data, the data were expanded by shifting left and right by 1 to 2 wave numbers, multiplying multiple spectra by different ratios with a total ratio sum of 100%, and then summing them. 2% Gaussian random noise was added.

Images