An intelligent cancer diagnosis method based on multimodal microscopic imaging and deep learning

Abstract

The present invention discloses a cancer intelligent diagnosis method based on multimodal microscopic imaging and deep learning, and relates to the technical field of microscopic optical image acquisition and biomedical image processing. The method comprises: using a fluorescence polarization microscope to collect bright field, fluorescence and polarization microscopic images of biological tissue slices, and performing image preprocessing; inputting the fluorescence and polarization microscopic images into a convolutional neural network for pre-training, and obtaining image features of several convolutional layers; establishing a bright field microscopic image training network, and performing multiple feature fusions in several convolutional layers in a weighted fusion manner; based on the neural network training results, dynamically adjusting the weight coefficients of the fluorescence and polarization microscopic image features, and establishing an optimal cancer diagnosis model. In the cancer intelligent diagnosis method provided by the present invention, the feature information reflected by the three images is different but complementary to each other. After feature fusion, faster, more accurate and more complex cancer intelligent diagnosis can be achieved.

Description

Technical Field

[0001] The present invention belongs to the field of methods for microscopic optical image acquisition and biomedical image processing, and in particular relates to an intelligent cancer diagnosis method based on multimodal microscopic imaging and deep learning. Background Art

[0002] Common diagnostic methods for cancer include imaging examinations and pathological examinations. Imaging examinations include X-rays, X-CT, magnetic resonance imaging, etc. X-rays and X-CT are widely used in clinical diagnosis as a preoperative diagnostic technology, but they are radioactive and insensitive to high-density cancer tissues, making them unsuitable for routine screening. Although magnetic resonance imaging (MRI) examinations are radiation-free, they are expensive and time-consuming. For the accurate judgment of cancer, microscopic observation of pathological sections in pathology is still the "gold standard" for clinical diagnosis. However, the diagnosis of pathology mostly relies on experienced experts, and the diagnostic process is not only time-consuming, but also affected by subjective factors. For the microscopic observation of pathological sections, since optical microscopes can only observe sections that have been stained to determine the difference between cancerous tissue and normal tissue, the lesion needs to be sampled, frozen, stained and made into sections. The staining and production of pathological sections will increase the length of pathological testing to a certain extent. If the test result is positive, it needs to be re-submitted for inspection, which will further extend the detection time.

[0003] Compared with traditional cancer detection methods, the bright field, fluorescence and polarization imaging modes of optical microscopy can obtain more comprehensive information on microscopic biological components and structures, without the need for tedious steps such as wax embedding, dewaxing and staining, which is simple and fast. Bright field microscopic images of biological tissue sections obtain spatial distribution information, including color, texture, morphology, etc., but cannot characterize specific biological components and structural differences. Fluorescence microscopic images and polarization microscopic images are specific to biological components and their structures. Fluorescence microscopic images characterize the content and distribution information of biological components that can emit spontaneous fluorescence, including collagen, etc.; polarization microscopic images obtained by orthogonal polarization imaging methods reflect the distribution of biological components with birefringence effects and their structural information, including collagen fiber distribution and orientation. Multimodal deep learning technology uses the powerful feature learning ability of deep learning neural networks to extract features, based on the feature differences reflected by different modalities and complementary information, to obtain more effective and complementary features of various images for fusion. These three microscopic imaging modes have different intrinsic mechanisms and provide different but complementary feature information. After feature fusion, more convincing, faster and more accurate intelligent cancer diagnosis can be achieved. Summary of the invention

[0004] The present invention provides a cancer intelligent diagnosis method based on multimodal microscopic imaging and deep learning, which solves key problems such as the fact that the sample information reflected by pathological images is relatively single and cannot reflect biological composition and structural information, and realizes accurate and rapid intelligent diagnosis of cancer tissues.

[0005] The technical solution adopted by the present invention to solve the technical problem is: a cancer intelligent diagnosis method based on multimodal microscopic imaging and deep learning, comprising the following steps:

[0006] 1) Use fluorescence polarization microscopy to collect bright field, fluorescence and polarization microscopic images of biological tissue sections and perform image preprocessing;

[0007] 2) Input the fluorescence and polarization microscopy images into the convolutional neural network for pre-training to obtain image features of several convolutional layers;

[0008] 3) Establish a bright field microscopy image training network and perform multiple feature fusions in a weighted fusion manner in several convolutional layers;

[0009] 4) Based on the neural network training results, the weight coefficients of fluorescence and polarization microscopy image features are dynamically adjusted to establish the optimal cancer diagnosis model.

[0010] The advantage of the method provided by the present invention is that it uses bright field, fluorescence and polarization microscopic imaging of biological tissue sections to replace conventional pathological sections as the input of a multimodal microscopic image discrimination model, implements feature extraction and fusion of multimodal deep learning, and can utilize the differences in biological components and microstructures reflected in different imaging modes of tissue sections to extract more differential information between cancerous and normal tissues, thereby achieving more complex cancer diagnosis tasks; constructing a weighted fusion feature fusion method can make full use of the shallow and deep features of each modality to achieve faster and more accurate intelligent cancer diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flow chart of a cancer intelligent diagnosis method based on multimodal microscopic imaging and deep learning provided by the present invention;

[0012] Figure 2 It is an implementation flow chart of a weighted feature fusion method provided in the present invention; DETAILED DESCRIPTION

[0013] The present invention is further described below in conjunction with specific steps and drawings. It is to be understood that the specific implementation steps described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the drawings.

[0014] Figure 1This is a flow chart of a cancer intelligent diagnosis method based on multimodal microscopic imaging and deep learning provided by the present invention. The present invention provides a cancer intelligent diagnosis method based on multimodal microscopic imaging and deep learning, as described below:

[0015] S110, using a fluorescence polarization microscope to collect bright field, fluorescence and polarization microscopic images of biological tissue sections, and performing image preprocessing.

[0016] In the present invention, the three imaging modes of bright field, fluorescence and polarization microscopy have different inherent mechanisms, and provide different characteristic information but complement each other. Bright field microscopic images of biological tissue sections obtain spatial distribution information, including color, texture, morphology, etc., but cannot characterize specific biological components and structural differences. Fluorescence microscopic images and polarization microscopic images are specific to biological components and their structures, among which fluorescence microscopic images characterize the content and distribution information of biological components that can emit spontaneous fluorescence, including collagen, etc.; polarization microscopic images obtained by orthogonal polarization imaging methods reflect the distribution of biological components with birefringence effects and their structural information, including collagen fiber distribution and orientation, etc.

[0017] The training specimens were processed according to clinical practice and divided into cancer tissue and normal tissue samples. The cancer tissue samples were staged according to pathology and were approximately 0.5 to 1 cm in size. 3 After the tissue samples were washed in saline and quickly frozen in liquid nitrogen, 10 μm thick sections were made using a cryo-microtome and picked up on a glass slide.

[0018] The sample images were collected using a fluorescence polarization microscope, with a 10X eyepiece and a 20X objective lens. The images were collected close to the center of the slice. To increase the correlation of the features, the three images were collected from the same area of ​​the sample.

[0019] Due to the problems of impurities and poor image acquisition signals that may exist in bright field, fluorescence and polarization microscopy images, and in order to better extract features, pseudo-color image enhancement is performed on the images. The specific steps are as follows: first, the original image is converted to a grayscale image; second, the histogram equalization algorithm is used to expand the contrast and grayscale range of the grayscale image; third, the median filter algorithm is used on the grayscale image to reduce the noise generated during the histogram equalization process; finally, the density segmentation algorithm is used to convert the grayscale image into a pseudo-color image.

[0020] S120, inputting the fluorescence and polarization microscopic images into a convolutional neural network for pre-training to obtain image features of several convolutional layers.

[0021] In the present invention, the discrimination model of multimodal microscopic images is mainly based on a deep learning convolutional neural network, in which the number of convolutional layers is 16.

[0022] The labeled fluorescence and polarization microscopy images were input into the feature extraction network for separate training, and the image features of the convolutional layers Conv3, Conv6, Conv9, Conv12, and Conv15 were extracted from the two neural networks in sequence for feature fusion.

[0023] S130, establishing a bright field microscopic image training network, and performing multiple feature fusions in a weighted fusion manner in a plurality of convolutional layers.

[0024] In the present invention, in order to make full use of the feature information of each modality image, a feature fusion method is proposed in the present invention, specifically:

[0025] The model training is based on the neural network of bright field microscopy images, and the feature Y corresponding to the convolutional layer in the neural network of fluorescence and polarization microscopy images is converted into i，j and Z i，j According to the form of weighted fusion and the feature X in the neural network of bright field microscopy image i，j The fused feature X′ i，j Replace the original X i，j Continue training the neural network on brightfield microscopy images.

[0026] The final model will give the patient's true labeling based on the bright field, fluorescence and polarized microscopic images of the given tissue slices, thereby achieving the task of cancer tissue staging. The model will also continue to expand the sample size, further improve the data model, and improve the model's applicability to achieve more complex cancer diagnosis tasks.

[0027] S140. Based on the neural network training results, the weight coefficients of fluorescence and polarization microscopy image features are dynamically adjusted to establish an optimal cancer diagnosis model.

[0028] For the image feature weight coefficients when each modality is fused, the feature weights of bright field microscopy images are fixed, while the weight a of fluorescence and polarization microscopy image features changes from 0.01 to 1, and the weight b changes from 1 to 0.01 accordingly. The weight coefficients are determined based on the diagnostic effect to establish the optimal cancer diagnosis model.

[0029] Figure 2 This is a flow chart of the implementation of a weighted feature fusion method provided in the present invention. The formula for the weighted fusion method is expressed as:

[0030] X′ i，j =X i，j +aY i，j +bZ i，j

[0031] Among them, X i，j , Y i，j , Zi，j Denote the image features of the convolutional layer of the bright field, fluorescence and polarization microscopy image training network, respectively, X′ i，j represents the fusion feature, j is the number of convolutional layers, i is the number of convolutional layer channels, and a and b are the weight coefficients of fluorescence and polarization microscopy image features, respectively.

[0032] The specific fusion steps are as follows: establish a bright field microscopy image training network, and use the image features Y extracted from the convolution layer Conv3 in the fluorescence and polarization microscopy image neural network i，3 , Z i，3 ; According to the weighted fusion method and the image feature X in the convolution layer Conv3 of the bright field microscopy image neural network i，3 Fusion forms feature X′ i，3 ; X′ i，3 Replace X i，3 Input into the bright field microscopy image neural network to continue training; follow the above steps to perform multiple fusions in the convolutional layers Conv6, Conv9, Conv12, and Conv15.

[0033] Matters not covered by the present invention are known technologies.

[0034] The above implementation process is only to illustrate the technical concept and features of the present invention, and its purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and it cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A cancer intelligent diagnosis method based on multimodal microscopic imaging and deep learning, characterized in that: The method comprises the following steps: Step 1: Collect bright field, fluorescence and polarization microscopic images of biological tissue sections through fluorescence polarization microscopy and perform image preprocessing; Step 2: Input the fluorescence and polarization microscopy images into the convolutional neural network for pre-training to obtain image features of several convolutional layers; Step 3: Establish a bright field microscopy image training network, and perform multiple feature fusions in a weighted fusion manner in several convolutional layers. The specific steps of the feature fusion are: (1) Input the labeled fluorescence and polarization microscopy images into the feature extraction network for separate training, and extract the image features of the convolutional layers Conv3, Conv6, Conv9, Conv12, and Conv15 from the two neural networks in turn for feature fusion; (2) Establish a bright field microscopy image training network, and input the image feature Y extracted from the convolutional layer Conv3 in the fluorescence and polarization microscopy image neural networks into the feature extraction network. i，3 , Z i,3 , according to the weighted fusion method and the image feature X in the convolution layer Conv3 of the bright field microscopy image neural network i，3 Fusion forms feature X′ i，3 , X′ i，3 Instead of X i,3 Input into the bright field microscopy image neural network to continue training; perform multiple fusions in the convolutional layers Conv6, Conv9, Conv12, and Conv15 in accordance with the above steps (1) and (2); Step 4: Based on the neural network training results, dynamically adjust the weight coefficients of fluorescence and polarization microscopy image features to establish the optimal cancer diagnosis model.

2. The method according to claim 1, characterized in that The three imaging modes of bright field, fluorescence and polarization microscopy have different intrinsic mechanisms and provide different but complementary characteristic information. Bright field microscopic images of biological tissue sections obtain spatial distribution information, and the main features include color, texture, and morphology; fluorescence microscopic images characterize the content and distribution information of biological components that can emit spontaneous fluorescence, and the main fluorescent components include collagen; polarized microscopic images obtained by orthogonal polarization imaging methods reflect the structure of biological components with birefringence effects and their distribution information, and the main features include the distribution and direction of collagen fibers.

3. The method according to claim 1, characterized in that: The image preprocessing in step 1 specifically includes the following steps: Step 1: Convert the original image into a grayscale image and use the histogram equalization algorithm to expand the contrast and grayscale range of this grayscale image; Step 2: Perform median filtering on the grayscale image expanded in the first step to reduce the noise generated during the histogram equalization process; Step 3: Use density segmentation algorithm to convert the grayscale image filtered in the second step into a pseudo-color image.

4. The method according to claim 1, characterized in that The specific formula of the weighted fusion method in step three is: X′ i，j =X i，j +aY i，j +bZ i，j Among them, X i,j , Y i,j , Z i,j Denote the image features of the convolutional layer of the bright field, fluorescence and polarization microscopy image training network, respectively, X′ i，j represents the fusion feature, j is the number of convolutional layers, i is the number of convolutional layer channels, and a and b are the weight coefficients of fluorescence and polarization microscopy image features, respectively.

Images