A computer-aided diagnosis method for colorectal cancer based on few-shot learning

Abstract

The present invention discloses a computer-aided diagnosis method for colorectal cancer based on few-shot learning, comprising the steps of: extracting the features of a preprocessed query image and a support image set respectively through two backbone networks with the same structure; calculating the cosine similarity of the above features, taking the maximum similarity among all support pixels as the response value, normalizing it and converting it into the image size to obtain a similarity matrix corr as the response value feature map; using the response value feature map, the features of the query image and the features of the support image, transmitting them to a subsequent convolutional network to obtain segmentation and diagnosis results, calculating the corresponding loss, and then backpropagating the error to optimize and update the network parameters. The present invention adopts the maximum similarity response value feature map, which is more concise and effective in guiding segmentation, and also uses prior knowledge in medical diagnosis as a constraint to improve the professionalism and accuracy of the model in diagnosis.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a computer-aided diagnosis method for colorectal cancer based on few-shot learning. Background Art

[0002] Pathological section images belong to the field of medical images. Under the light microscope, the morphological manifestations of cell nuclei are still the main basis for current tumor diagnosis. Cancer is a stubborn disease that is difficult for humans to overcome. As a common malignant tumor in digestive tract diseases, colorectal cancer has relevant data indicating that its incidence rate and fatality rate are among the top three cancers in China, seriously affecting people's lives and health. Early detection of colorectal cancer is of great help in improving its cure rate.

[0003] Current detection methods mostly rely on microscopes and the personal experience of pathologists, making the detection results subjective and different. With the development of image processing technology, computer-aided detection systems for pathological images have emerged. Automatic segmentation and diagnosis of microscopic images of colorectal cancer pathological sections are key problems that need to be solved urgently. A good segmentation and diagnosis algorithm can provide objective and accurate "second opinions" for pathologists.

[0004] Since the mid- to late 20th century, researchers have started to study related technologies for medical image segmentation, hoping to reduce the heavy burden on pathologists through computer-aided technologies. However, these algorithms usually have poor robustness, complex processes, and cannot provide semantic information that is easy for humans to understand.

[0005] As is well known, most deep learning network models for visual tasks require a large amount of labeled data for training. Compared with natural images, the data samples of medical images are relatively few, and fine image annotation often requires a large number of experts for manual operations, which is time-consuming and laborious. The high cost of manual annotation also limits the application of deep learning in the field of medical image segmentation. Therefore, realizing the colorectal cancer diagnosis technology based on few-shot learning can not only save the lives of countless patients, but also has great significance for saving medical resource costs and alleviating doctor-patient contradictions and other issues. Summary of the Invention

[0006] The purpose of the present invention is to minimize the annotation work and realize computer-aided diagnosis of colorectal cancer by a physician. Therefore, a computer-aided diagnosis method for colorectal cancer based on few-shot learning is provided, which can simultaneously realize nuclear segmentation, discrimination of benign and malignant, and diagnosis of differentiation degree, so as to achieve the purpose of assisting diagnosis for pathologists.

[0007] The technical solution adopted to achieve the purpose of the present invention is:

[0008] A computer-aided diagnosis method for colorectal cancer based on few-shot learning, comprising the steps:

[0009] Extract the features of the preprocessed query image and the support image set through two backbone networks with the same structure;

[0010] After calculating the cosine similarity of the above features, the maximum similarity among all support pixels will be used as the response value, which is normalized and then converted into the image size to obtain the similarity matrix corr, serving as the response value feature map;

[0011] Use the response value feature map, the features of the query image, and the features of the support image, and pass them to the subsequent convolutional network to obtain the segmentation and diagnosis results. Calculate the corresponding loss through the loss function, and then backpropagate the error to optimize and update the network parameters.

[0012] The computer-aided diagnosis method for colorectal cancer proposed by the present invention, by utilizing the prior medical diagnosis knowledge related to the information of the support image and the support image mask and the diagnosis result, the model can also achieve accurate segmentation and diagnosis on a small-sample data set, reducing the high cost of manual annotation; while segmenting the nuclei in the microscopic image of the colorectal cancer pathological section, it can judge the benign and malignant nature of the colorectal cancer according to the morphological manifestations of the nuclei, and give the degree of cell differentiation. It can automatically mark the nuclei with different morphological manifestations for pathologists and diagnosticians, and give the benign and malignant nature and the degree of differentiation results of the colorectal cancer pathological section, providing objective and referenceable pathological diagnosis opinions for pathologists and diagnosticians, and achieving the purpose of computer-aided diagnosis; not only reducing the burden on pathologists and providing objective and accurate reference opinions for pathologists, but also improving the accuracy and generalization ability of the model based on small-sample data.

[0013] The present invention provides enhanced information for the query image under a small-sample data set through the extraction of the support image and the support image mask features; the maximum similarity feature response map between the support image features and the query image features improves the generalization ability of the model and enhances the performance of the model; the decoupling of the task branches and the constraint of the prior medical knowledge between the classification tasks avoid the possible feature conflicts and widely different diagnosis results between the task branches, realizing the simultaneous completion of the two tasks of segmentation and diagnosis, providing objective and accurate nuclear localization, benign and malignant classification, and cell differentiation degree results for pathologists and diagnosticians, and achieving the purpose of auxiliary diagnosis.

[0014] The present invention adopts the maximum similarity response value feature map, which has a more concise and effective guiding effect on segmentation, and also uses the prior knowledge in medical diagnosis as a constraint to improve the professionalism and accuracy of the model in diagnosis. Description of the Drawings

[0015] Figure 1It is the processing flow chart of the method for automatically segmenting and diagnosing colorectal cancer based on small samples in the embodiments of the present invention.

[0016] Figures 2 - 3 They are respectively Figure 1 the enlarged views of two parts of Specific embodiments

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0018] Since models based on deep learning usually require a large amount of data samples for learning, but large-scale data and data annotation require high labor costs and time costs, it is particularly important to develop deep learning based on small samples for medical image processing. In addition, most current colorectal cancer diagnosis methods rely on the personal experience of microscopes and pathologists, making the diagnosis results subjective and different. The existing algorithm technologies in medical image segmentation are usually less robust and have a single task. For example, they usually only focus on segmentation and ignore the diagnosis of benign and malignant and differentiation results. Therefore, the automatic segmentation and diagnosis of colorectal cancer pathological section microscopic images is a key problem to be solved urgently. While automatically segmenting various types of cell nuclei, it is necessary to provide objective and accurate pathological diagnosis opinions for pathologists as a reference, reduce the burden on pathologists, save medical costs, and standardize medical diagnoses.

[0019] In view of the problems such as the insufficient quantity and difficult annotation of current colorectal cancer pathological section microscopic data, the present invention innovatively proposes a method for automatically segmenting and diagnosing colorectal cancer based on small sample learning. The features of the support image set extracted by the neural network are calculated with the features of the query image set for cosine similarity, and the obtained maximum similarity is normalized to generate a response value feature map, which then guides the segmentation and diagnosis of the query set.

[0020] Since the present invention calculates the similarity between the feature vectors extracted from the query image and the feature vectors extracted from the support image set to generate a response value feature map to guide the segmentation and diagnosis of the query image, the model can widely learn the feature distributions of different types of images and generate corresponding response value feature maps, so that the model obtains better generalization and provides more accurate segmentation results for pathologists. In the diagnosis stage, using the prior knowledge in known medical diagnosis as a constraint, the model is more professional and reliable in classification diagnosis and provides more credible diagnosis results for pathologists.

[0021] The input of the method for automatically segmenting and diagnosing colorectal cancer based on a small sample in the embodiments of the present invention is the microscopic data of the pathological sections of colorectal cancer. The image data respectively extract image features through a backbone network with the same structure; the corresponding features formed after processing the extracted features are input into different task branches to obtain the required results.

[0022] The method for automatically segmenting and diagnosing colorectal cancer based on a small sample proposed by the present invention extracts the features of the query image and the support image set after preprocessing through two backbone networks with the same structure.

[0023] This step can be expressed as Fq = M(Iq pre ), Fs = M(Is pre ),

[0024] where Iq pre and Is pre respectively represent the query image and the support image set after the preprocessing operation, M represents the backbone network, and Fq and Fs respectively represent the features of the query image and the features of the support image set extracted by the backbone network from the preprocessed query image and the support image set;

[0025] Among them, the data preprocessing operation mainly includes random rotation, random Gaussian blur, random scaling, random horizontal flipping and normalization;

[0026] This preprocessing operation step can be expressed as: I pre = F pre (I), where I represents the input image and F pre represents the preprocessing operation.

[0027] After calculating the cosine similarity of the above-extracted features, the maximum similarity among all support pixels obtained is used as the response value, and after normalization processing, it is converted into the image size to obtain the similarity matrix corr;

[0028] This step can be divided into calculating the cosine similarity to obtain the maximum similarity value, normalization operation, and conversion operation, which can be respectively expressed as:

[0029]

[0030] corr = Upsample(sim),

[0031] Among them, sim1 is the maximum similarity among all pixels obtained after cosine similarity calculation, sim represents the normalized maximum similarity value, ||·||2 represents the L2 norm, that is, the square root of the sum of the squares of each element of the vector, max is used to calculate the maximum value, min is used to calculate the minimum value, ∈ is a non-zero constant, and Upsample represents the bilinear interpolation upsampling operation, that is, sim is bilinearly interpolated and upsampled to the required image size.

[0032] The response value feature map formed by the similarity matrix corr, the features of the query image, and the features of the support image are passed to the subsequent convolutional network to obtain the segmentation and diagnosis results, and the corresponding loss is calculated. Then, the error is backpropagated to optimize and update the network parameters.

[0033] The present invention uses the classical cross-entropy loss, and the loss function is as follows:

[0034]

[0035] Among them, g represents the inference output of the deep convolutional neural network for the input image, and g t represents the label annotated by the dataset.

[0036] As an optional embodiment, as Figure 1 shown, the backbone networks of the query image and the support image both adopt resnet50. The query image uses the backbone network to obtain the outputs {q0, q1, q2, q3, q4} of five convolutional blocks {Q0, Q1, Q2, Q3, Q4} respectively. After bilinearly interpolating and upsampling the feature map of q2 to the size of the q3 feature map and then performing channel concatenation with the q3 feature map, Q_feat is obtained. Then, after passing through a convolutional layer Qd, qd is obtained. qd passes through an average pooling layer operation to obtain Q_avg.

[0037] The same operation as the query image, the support image passes through the outputs {s0, s1, s2, s3, s4} of five convolutional blocks {S0, S1, S2, S3, S4} of resnet50. After bilinearly interpolating and upsampling the feature map of s2 to the size of the s3 feature map and then performing channel concatenation with the s3 feature map, the concatenated feature map passes through a convolutional layer Sd to obtain the feature sd.

[0038] Different from the operations on the query image part, here there is a multiplication operation between the mask image y (segmentation mask) of the support image and the s3 feature map. The result of the multiplication operation is used as the input of the convolutional block S4. The output s4 of the convolutional block S4 continues to be multiplied by the mask image of the support image to obtain S_feat. Calculate the cosine similarity between the feature S_feat of the support image and the feature q4 of the query image, and then normalize the maximum value of the obtained cosine similarity and bilinearly interpolate and upsample it to the similarity matrix corr of the size of the Q_avg feature map. This similarity matrix is helpful for guiding the subsequent segmentation and diagnosis of the query image.

[0039] In addition, the feature sd of the support image and the mask y of the support image are subjected to a weighted global average pooling operation (GAP) to obtain Sgwap_feat, which is used to guide the segmentation and diagnosis of the query image feature together with the similarity matrix corr in the subsequent stage. The weighted global average pooling operation is specifically that sd is multiplied by the segmentation mask y of the support image, the result obtained after multiplication is subjected to an average pooling operation and then divided by the mask y of the support image after average pooling to obtain the output result Sgwap_feat of the global average pooling operation.

[0040] The specific formula is expressed as:

[0041]

[0042] Among them, avgpool represents the average pooling operation, h and w represent the height and width of the sd feature map, and ∈ is a non-zero constant. Next, the feature is input into different task branches to obtain the required results:

[0043] 1. Segmentation branch:

[0044] Sgwap_feat, corr, and Q_avg are concatenated in channels to obtain m0. m0 passes through the convolutional layer M to obtain m1. m1 directly skips the convolutional layer A for the first time and then passes through two consecutive convolutional layers {B0, B1} to obtain b1. The feature b1 and m1 are subjected to an addition operation to obtain the feature pre. The pre feature is input into three different branches respectively:

[0045] Branch 1: The feature pre is input into two convolutional layers C0 and C1 for classification, and then the output result is bilinearly interpolated and upsampled to the size of the query image. Thus, the output result aux of this branch is obtained. This branch is only used in the model training stage, and the loss function of this branch is used to assist branch 2 in updating the parameters in the model;

[0046] Branch 2: The feature pre is input into the convolutional layer R0 to obtain the output result r0, which then passes through two convolutional layers R1 and R2 to obtain the output result r1. After the addition operation of the features r0 and r1, the result is input into two convolutional layers C2 and C3 for classification, and is upsampled to the size of the query image by bilinear interpolation to obtain the output result out;

[0047] Branch 3: After the feature pre is upsampled by bilinear interpolation, it is concatenated with the output result m1 of the convolutional layer M in the channel dimension to obtain the output result rec. After rec passes through the convolutional layer A, it is added to m1 to obtain m2. m2 continues the first operation of m1 and passes through two consecutive convolutional layers {B0, B1} to obtain the output result b1. b1 is added to m2 to obtain the output result pre again, and so on in a loop.

[0048] Among them, the results aux of Branch 1 and out of Branch 2 are respectively used to calculate the loss values with the true labels, and then the losses are backpropagated into the network layer to update the parameters;

[0049] Among them, the loss functions of Branch 1 and Branch 2 are both cross-entropy losses;

[0050] The role of Branch 3 is to cyclically extract features for predicting the results of Branch 1 and Branch 2.

[0051] 2. Binary classification branch:

[0052] Using m0 obtained from the segmentation branch part, the middle-level feature q2 and the high-level feature q4 extracted from the query image, the feature m3 is obtained through channel concatenation. m3 is input into a convolutional layer D0 to obtain d0. After d0 passes through the convolutional layer D1, the result is concatenated with d0 in the channel dimension to obtain d1. Finally, d1 is input into a multi-layer perceptron containing three convolutional layers {D2, D3, D4} to output the benign or malignant judgment result of the microscopic image of the pathological section.

[0053] 3. Multi-classification branch:

[0054] The structure is the same as that of the binary classification branch. Using m0 obtained from the segmentation branch part, the middle-level feature q2 and the high-level feature q4 extracted from the query image, the feature m3 is obtained through channel concatenation. m3 is input into a convolutional layer E0 to obtain e0. After e0 passes through the convolutional layer E1, the result is concatenated with e0 in the channel dimension to obtain e1. Finally, e1 is input into a multi-layer perceptron containing three convolutional layers {E2, E3, E4} to output the differentiation result of cancer cells in the microscopic image of the pathological section.

[0055] Since there is a correlation between benign and malignant outcomes and the differentiation results of cancer cells. For example, the differentiation results of benign colorectal cancer cells usually have only two types: healthy and adenomatous, while the differentiation results of malignant colorectal cancer cells usually correspond to three types: moderately differentiated, moderately to poorly differentiated, and poorly differentiated. In order to make full use of this correlation constraint in medical diagnosis, the present invention pairs binary results with multi-branch results and uses prior knowledge in medical diagnosis to impose constraint penalties on benign / malignant classification and differentiation degree classification, ensuring the credibility and accuracy between the benign / malignant classification results and the differentiation degree classification results.

[0056] It should be noted that the classical cross-entropy loss is used for all branches in the present invention.

[0057] The present invention uses the support image and its segmentation mask, enabling the model to get rid of the bondage of requiring large-scale dataset training. The maximum similarity response feature map between the support image features and the query image features provides guidance for segmentation and classification, improving the accuracy and generalization ability of the model. Moreover, the decoupling of the segmentation and classification branches avoids the conflict of feature parameters between tasks to a certain extent, and the correlation constraint between the two classification branches improves the accuracy of diagnosis, providing an objective auxiliary diagnosis function for pathologists and diagnosticians.

[0058] Compared with other medical image segmentation and diagnosis methods, the present invention uses the model to simultaneously achieve multiple tasks, giving all the results required for assisting pathologists and diagnosticians in diagnosis at one time, and overcoming the inherent drawback of large-scale dataset training in deep learning, reducing the cost required to obtain microscopic data and annotation information of colorectal cancer pathological sections.

[0059] The support image set of the present invention has the advantage of rich types. Therefore, the existence of the support image can make up for the lack of generalization ability in training on small sample datasets in deep learning, significantly enhancing the generalization ability of the model.

[0060] The method for segmenting and diagnosing colorectal cancer cells based on small sample learning proposed by the present invention can achieve the segmentation and diagnosis of microscopic images of colorectal cancer pathological sections under the condition of small sample datasets, reducing the cost of manual annotation of medical images while providing objective and accurate reference opinions for pathologists and diagnosticians, achieving the purpose of computer-aided diagnosis.

[0061] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can still be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A computer-aided diagnosis method for colorectal cancer based on few-shot learning, characterized in that The input is the microscopic data of pathological sections of colorectal cancer, including the steps: Extract the features of the preprocessed query image and the support image set respectively through two backbone networks with the same structure; After calculating the cosine similarity of the above features, the maximum similarity between all pixels is used as the response value, which is normalized and then converted into the image size to obtain the similarity matrix corr, which is used as the response value feature map; Use the response value feature map, the features of the query image and the features of the support image, and pass them to the subsequent convolutional network to obtain the segmentation and diagnosis results. Calculate the corresponding loss through the loss function, and then backpropagate the error to optimize and update the network parameters; The query image uses the backbone network to obtain the outputs {q0, q1, q2, q3, q4} of five convolutional blocks {Q0, Q1, Q2, Q3, Q4} respectively. The feature map q2 is bilinearly interpolated and upsampled to the size of the feature map q3, and then concatenated with the feature map q3 in channels to obtain the feature map Q_feat. Then, after passing through a convolutional layer Qd, the feature map qd is obtained. The feature map qd passes through an average pooling layer operation to obtain Q_avg; The support image is processed by five convolutional blocks {S0, S1, S2, S3, S4} of resnet50 to obtain the outputs {s0, s1, s2, s3, s4}. The feature map s2 is bilinearly interpolated and upsampled to the size of the feature map s3, and then concatenated with the feature map s3 in channels. Then, the concatenated feature map passes through a convolutional layer Sd to obtain the feature map sd; The mask image y of the support image is multiplied with the feature map s3. The result of the multiplication operation is used as the input of the convolutional block S4. The feature map s4 is multiplied with the mask image y of the support image to obtain the feature map S_feat. Calculate the cosine similarity between the feature map S_feat and the feature map q4 of the query image, and then normalize the maximum cosine similarity obtained, and bilinearly interpolate and upsample it into a similarity matrix corr of the size of the feature map Q_avg.

2. The computer-aided diagnosis method for colorectal cancer based on few-shot learning according to claim 1, wherein The calculation method of the similarity matrix corr is as follows: corr = Upsample(sim), where sim1 is the maximum similarity between all pixels obtained after calculating the cosine similarity, sim represents the normalized maximum similarity value, ||·||2 represents the L2 norm, that is, the square root of the sum of the squares of each element of the vector, max is to calculate the maximum value, min is to calculate the minimum value, ∈ is a non-zero constant, Upsample represents the bilinear interpolation and upsampling operation, that is, sim is bilinearly interpolated and upsampled to the required image size, Fq and Fs respectively represent the features of the query image and the features of the support image set extracted by the backbone network from the preprocessed query image and the support image set.

3. The computer-aided diagnosis method for colorectal cancer based on few-shot learning according to claim 1, characterized in that, The loss function is as follows: Among them, g represents the inference output of the deep convolutional neural network for the input image, and g t represents the label annotated in the dataset.

4. The computer-aided diagnosis method for colorectal cancer based on few-shot learning according to claim 1, wherein The backbone network uses resnet50.

5. The computer-aided diagnosis method for colorectal cancer based on few-shot learning according to claim 1, wherein, Perform a weighted global average pooling operation on the feature map sd and the mask image y of the support image to obtain Sgwap_feat. The weighted global average pooling operation is specifically to multiply the feature map sd by the mask image y of the support image, perform an average pooling operation on the resulting product, and then divide it by the mask image y of the support image after average pooling to obtain the output result Sgwap_feat of the global average pooling operation. The specific formula is expressed as: where avgpool represents the average pooling operation, h and w represent the height and width of the feature map sd, and ∈ is a non-zero constant.

6. The computer-aided diagnosis method for colorectal cancer based on few-shot learning according to claim 5, characterized in that, Concatenate the obtained Sgwap_feat, corr, and Q_avg along the channels to get m0. Pass m0 through the convolutional layer M to get m1. m1 directly skips the convolutional layer A for the first time and passes through the subsequent two consecutive convolutional layers {B0, B1} to get b1. Perform an addition operation on the feature b1 and m1 to get the feature pre. The feature pre is input into three different branches respectively: Branch 1: Input the feature pre into two convolutional layers C0 and C1 for classification, and then perform bilinear interpolation upsampling on the output result to the size of the query image, thereby obtaining the output result aux of this branch. This branch is only used in the model training stage, and the loss function of this branch is used to assist Branch 2 in updating the parameters in the model; Branch 2: Input the feature pre into the convolutional layer R0 to get the output result r0, and then pass through two convolutional layers R1 and R2 to get the output result r1. Perform an addition operation on the features r0 and r1, and input the result into two convolutional layers C2 and C3 for classification, and perform bilinear interpolation upsampling to the size of the query image to get the output result out; Branch 3: After bilinear interpolation upsampling the feature pre, concatenate it with the output result m1 of the convolutional layer M along the channels to get the output result rec. After rec passes through the convolutional layer A, add it to m1 to get m2. m2 continues the operation of m1 for the first time, passes through two consecutive convolutional layers {B0, B1} to get the output result b1, and add b1 to m2 to get the output result pre again, and so on; Among them, calculate the loss values of the results aux of Branch 1 and out of Branch 2 respectively with the true annotation, and then backpropagate the loss to the network layer to update the parameters; Among them, the loss functions of Branch 1 and Branch 2 are both cross-entropy losses. The role of Branch 3 is to extract cyclic features for predicting the results of Branch 1 and Branch 2.

7. The computer-aided diagnosis method for colorectal cancer based on few-shot learning according to claim 6, wherein Use the m0 obtained from the segmentation branch part, the middle-level feature q2 and the high-level feature q4 extracted from the query image, and concatenate them along the channels to get the feature m3. Input m3 into a convolutional layer D0 to get d0. After d0 passes through the convolutional layer D1, concatenate the result with d0 along the channels to get d1. Finally, input d1 into a multi-layer perceptron containing three convolutional layers {D2, D3, D4} to output the benign and malignant judgment result of this pathological section microscopic image.

8. The computer-aided diagnosis method for colorectal cancer based on few-shot learning according to claim 7, wherein, The m0 obtained by using the segmentation branch part, the middle-level feature q2 and the high-level feature q4 extracted from the query image are concatenated through channels to obtain the feature m3. The m3 is input into a convolutional layer E0 to obtain e0. After passing through the convolutional layer E1, the result is concatenated with e0 through channels to obtain e1. Finally, e1 is input into a multi-layer perceptron containing three convolutional layers {E2, E3, E4} to output the differentiation result of cancer cells in the microscopic image of the pathological section.

9. The computer-aided diagnosis method for colorectal cancer based on few-shot learning according to claim 1, wherein The aforementioned preprocessing includes random rotation, random Gaussian blur, random scaling, random horizontal flipping, and normalization.

Images