An implementation method of a multimodal learning convolutional neural network model based on multi-feature information acquisition and fusion

Abstract

The present invention combines deep learning technology with polarized light microscopy, bright field microscopy, and hyperspectral microscopy technology, and proposes an implementation method for a multimodal learning convolutional neural network (MML-CNN) model based on the collection and fusion of multiple feature information. The method includes: (1) using the local binary pattern algorithm (LBP) to extract polarized image features at different angles and superimpose them to achieve pixel-level fusion; (2) using the convolution layer to extract polarized image and bright field image features respectively and superimpose them to achieve feature-level fusion; (3) using a three-dimensional convolutional neural network (3D-CNN) to discriminate hyperspectral data and combine it with the image results, and then using support vector machine (SVM), partial least squares (PLS), or Fisher discriminant for statistical analysis to achieve decision-level fusion. The present invention combines multimodal information collection, multimodal learning, and multi-feature fusion methods, thereby providing a new approach to rapid, accurate, and intelligent diagnosis of cancer pathology.

Description

Technical Field

[0001] The present invention belongs to the fields of deep learning technology and polarized light microscopy technology, (NIR-Vis) bright field microscopy technology, and (UV-Vis, NIR, IR) hyperspectral microscopy technology, and in particular relates to an implementation method of a multimodal learning convolutional neural network model based on multiple feature acquisition and fusion. Background Art

[0002] With the development of deep learning technology, the application of neural network models in the medical field has become popular. However, unlike other fields, the human body is a complex biological system, and a single detection method often cannot produce accurate results. Therefore, clinical diagnosis of diseases often requires a comprehensive analysis combining multiple detection technologies, which requires neural network models to be able to process multiple data types simultaneously.

[0003] Multimodal fusion is an effective approach that primarily encompasses four methods: signal-level fusion, pixel-level fusion, feature-level fusion, and decision-level fusion. Existing multimodal fusion neural network models used in the medical field often only utilize one of these methods, achieving only bimodal fusion. Consequently, the application of existing neural network models in disease diagnosis remains limited, and their potential remains untapped. Therefore, developing a neural network model that simultaneously incorporates multiple feature fusion methods and can process a wide range of data types is of great significance.

[0004] On the other hand, traditional or clinical detection instruments for cancer tissue are often expensive, provide limited data information, and have a low signal-to-noise ratio. On this basis, new optical detection technologies such as polarized light microscopy, bright-field microscopy, and spectral microscopy have outstanding advantages and are moving towards clinical application, with corresponding detection instruments being continuously developed. These microscopic imaging and spectral technologies each have their own characteristics and advantages, thereby revealing the different physiological, physical, and biochemical characteristics of diseased tissues. Among them, polarized light microscopy can explain the changes in the ordered arrangement, cross-linking, and birefringence of components such as collagen fibers during the progression of the disease; bright-field microscopy can obtain the fine morphology and changes of the target sample tissue; and spectral imaging can not only reveal the morphological changes of tissues during the progression of the disease, but also provide a richer picture of the changes and distribution of biological macromolecules and trace components within the tissue (such as proteins, lipids, nucleic acids, collagen, calcification, etc.). In summary, applying multiple multimodal fusion methods to the above-mentioned optical detection technology can extract the characteristic information of different diseased tissues as a powerful basis for judgment, and based on this, establish a multimodal learning convolutional neural network (MML-CNN) model to achieve high-precision intelligent diagnosis of cancer tissues, which will be very meaningful and highly reliable. Summary of the Invention

[0005] Purpose of the invention: The purpose of the present invention is to provide an implementation method of a multimodal learning convolutional neural network model based on multiple feature information collection and including three feature fusion methods, thereby integrating the three multimodal fusion methods into a single neural network model, enabling it to process three types of data simultaneously and achieve high-precision intelligent diagnosis of cancer tissue.

[0006] Technical solution: A method for implementing a multimodal learning convolutional neural network model based on multi-feature information collection and fusion, including the following steps:

[0007] Step 1: Collect polarized light microscopy images, NIR-Vis bright-field microscopy images, and UV-Vis, NIR, and IR hyperspectral microscopy images at different angles from cancer tissue slice samples.

[0008] Step 2: Use data augmentation methods to expand the data to reduce the risk of overfitting;

[0009] Step 3: Use the local binary pattern (LBP) algorithm to extract the texture features of polarized microscopic images at different polarization angles, and superimpose them to obtain a fused polarized image to achieve pixel-level fusion.

[0010] Step 4: Use several symmetrical two-dimensional convolutional layers to downsample the fused polarized image and bright field image respectively, extract features, and superimpose the results to obtain the fused image features to achieve feature-level fusion;

[0011] Step 5: Automatically extract a large number of microscopic spectra from the hyperspectral microscopic image and arrange and stack the spectral data into a three-dimensional structure. Then, a three-dimensional convolutional neural network (3D-CNN) model is established and applied to the hyperspectral data to extract spectral features. The spectral features are combined with the fused image features and statistically analyzed using support vector machines (SVM), partial least squares (PLS), or Fisher discriminant to achieve decision-level fusion.

[0012] Step 6: Use the model to learn and predict the corresponding data of cancer samples.

[0013] Furthermore, in step 1, the cancer tissue sample and the normal tissue sample are cut into slices of a certain thickness using a cryogenic microtome for collecting multimodal data of polarized light microscopic images, bright field microscopic images, and hyperspectral microscopic images.

[0014] Furthermore, step 2 specifically includes the following points:

[0015] A certain proportion of data is separated from the total data as a test set, and the remaining data is expanded through data augmentation to reduce the risk of overfitting;

[0016] For image data, horizontal or vertical flipping, rotation at different angles, arbitrary ratio cropping, and 2% Gaussian random noise are added to expand the data volume;

[0017] For the stacked three-dimensional spectral data, all spectra were shifted left and right by 1 to 2 wavenumbers, and multiple spectra were multiplied by different proportions with a total proportion sum of 100%. Then the sum was taken and 2% Gaussian random noise was added to expand the data volume.

[0018] Furthermore, in step 6, the model is trained according to the 10-fold cross-validation method, and then the performance of the model is tested using the test set data.

[0019] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0020] The present invention proposes an implementation method for a multimodal learning convolutional neural network model based on the acquisition and fusion of multiple features, which has two main effective effects: 1) The present invention provides a specific idea for integrating multiple multimodal fusion methods into a single neural network model, so that the model has the ability to process multiple different types of data simultaneously; 2) The above model is applied in clinical practice to achieve fast and accurate intelligent diagnosis of cancer, which can provide a reference for doctors. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is the specific structure of a multimodal learning convolutional neural network model based on multiple feature acquisition and fusion provided by the present invention. DETAILED DESCRIPTION

[0022] The objects and functions of the present invention, as well as methods for achieving these objects and functions, will now be explained with reference to exemplary embodiments. However, the present invention is not limited to the exemplary embodiments disclosed below; it can be implemented in various forms. The essence of the description is merely to help those skilled in the relevant art to comprehensively understand the specific details of the present invention.

[0023] Figure 1 The specific structure of a multimodal learning convolutional neural network model based on multiple feature acquisition and fusion provided by the present invention is shown. The structure can be subdivided into three parts: pixel-level fusion, feature-level fusion and decision-level fusion.

[0024] For the pixel-level fusion part, the LBP algorithm is used to extract the texture features of polarized microscopic images with different polarization angles, and they are superimposed to obtain a fused polarized image to achieve pixel-level fusion.

[0025] For the feature-level fusion part, the above-mentioned fused polarized image is input into several convolutional layers, the data size is compressed by downsampling, and the corresponding polarized image features are extracted. At the same time, the (NIR-Vis) bright field image is input into several convolutional layers, the data size is compressed by downsampling, and the corresponding bright field image features are extracted (the convolutional layer used for the bright field image and the convolutional layer used for the polarized image have a symmetrical structure). The polarized image features and the bright field image features are then superimposed to obtain fused image features, thereby realizing feature-level fusion. In order to ensure that the MML-CNN model is easy to train, several Resblock structures are used for further feature extraction of the fused image features while maintaining the feature size unchanged. The fused features are then expanded, and a mapping relationship is established with the sample labels through the fully connected layer.

[0026] For the decision-level fusion, a large number of spectra are automatically extracted from hyperspectral microscopy images (UV-Vis, NIR, and IR). The spectral data are arranged in parallel according to their 2D coordinates in the image and stacked into a 3D structure. A 3D-CNN model is then built to process this 3D hyperspectral data. This model uses several 3D convolutional layers to extract spectral features using downsampling. These features are then expanded and mapped to sample labels using a fully connected layer.

[0027] Finally, the output results of the feature-level fusion neural network of the image and the output results of the hyperspectral 3D-CNN are combined and learned using methods such as SVM, PLS or Fisher discriminant to make a high-precision diagnosis of cancer samples and achieve decision-level fusion.

[0028] After the above-mentioned MML-CNN model is built, a cryogenic microtome is used to cut cancer tissue samples and normal tissue samples into slices of a certain thickness for the collection of polarized microscopic images, bright field microscopic images, and hyperspectral microscopic images. When sufficient data is obtained, a certain proportion of the data is separated from it and used as a test set. The remaining data is expanded by data enhancement methods to reduce the risk of overfitting. For image data, horizontal or vertical flipping, rotation at different angles, arbitrary proportion cropping, and adding 2% Gaussian random noise are used to expand the data volume; for stacked three-dimensional spectral data, all spectra are simultaneously shifted left and right by 1 to 2 wavenumbers, multiple spectra are multiplied by different proportions (the total proportion sum is 100%) and then summed, and 2% Gaussian random noise is added to expand the data volume. The expanded data is trained on the MML-CNN model according to the k-fold cross-validation method with k=10, and then the performance of the model is tested using the test set data to achieve high-precision diagnosis of cancer samples to verify the effectiveness of the present invention.

[0029] Matters not covered by the present invention are known technologies.

[0030] The above embodiments are intended only to illustrate the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent changes or modifications made in accordance with the spirit of the present invention are intended to be covered by the scope of protection of the present invention.

Claims

1. A method for implementing a multimodal learning convolutional neural network model based on multiple feature acquisition and fusion, characterized by: The following steps are involved: Step 1: Collect polarized light microscopy images, NIR-Vis bright-field microscopy images, and UV-Vis, NIR, and IR hyperspectral microscopy images at different angles from cancer tissue slice samples. Step 2: Use data augmentation methods to expand the data to reduce the risk of overfitting; Step 3: Use the local binary pattern (LBP) algorithm to extract the texture features of polarized microscopic images at different polarization angles, and superimpose them to obtain a fused polarized image to achieve pixel-level fusion. Step 4: Use several symmetrical two-dimensional convolutional layers to downsample the fused polarized microscopy image and the bright field microscopy image respectively, extract features, and superimpose the results to obtain the fused image features to achieve feature-level fusion; Step 5: Automatically extract a large number of microscopic spectra from the hyperspectral microscopic image and arrange and stack the spectral data into a three-dimensional structure. Then, a three-dimensional convolutional neural network (3D-CNN) model is established and applied to the hyperspectral data to extract spectral features. The spectral features are combined with the fused image features and statistically analyzed using support vector machines (SVM), partial least squares (PLS), or Fisher discriminant to achieve decision-level fusion. Step 6: Use the model to learn and predict the corresponding data of cancer samples.

2. The method according to claim 1, characterized in that In step 1, the cancer tissue sample and the normal tissue sample are cut into slices of a certain thickness using a cryogenic microtome for collecting multimodal data of polarized light microscopic images, bright field microscopic images, and hyperspectral microscopic images.

3. The method according to claim 1, characterized in that The step 2 includes the following points: A certain proportion of data is separated from the total data as a test set, and the remaining data is expanded through data augmentation to reduce the risk of overfitting; For image data, horizontal or vertical flipping, rotation at different angles, arbitrary ratio cropping, and 2% Gaussian random noise are added to expand the data volume; For the stacked three-dimensional hyperspectral data, all spectra were shifted left and right by 1 to 2 wavenumbers, and multiple spectra were multiplied by different ratios with a total ratio sum of 100%. Then the sum was taken and 2% Gaussian random noise was added to expand the data volume.

4. The method according to claim 1, wherein In step 6, the model is trained using a 10-fold cross-validation method, and then the performance of the model is tested using the test set data.

Images