Abdominal lymph node metastasis prediction method, system and equipment based on comparative learning

Abstract

The invention discloses an abdominal lymph node metastasis prediction method, system and device based on comparative learning, belongs to lymph node metastasis prediction in the technical field of artificial intelligence, and aims to solve the technical problem of low lymph node metastasis prediction accuracy caused by the fact that a model is not guided to focus on a certain type / kind of tags during model training in the prior art. The method comprises the following steps: firstly, migrating parameters in a feature extraction module subjected to supervised comparative learning pre-training to a feature extractor, inputting features extracted by the feature extractor into a simple transfer prediction network, and outputting a lymph node transfer probability by the simple transfer prediction network; calculating the contribution degree of each sample image according to the lymph node metastasis probability; and finally, during actual training of the lymph node metastasis prediction model, sorting is performed according to the contribution degree of each sample, positive samples with high contribution degrees are learned preferentially, then negative samples with high contribution degrees are learned, and finally the samples which are difficult to classify clearly are processed. Therefore, the lymph node metastasis prediction accuracy can be greatly improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and relates to lymph node metastasis prediction, and in particular to an abdominal lymph node metastasis prediction method, system, and device based on contrast learning. Background Art

[0002] Colorectal cancer is one of the three major malignant tumors in the world. Lymph node metastasis has a crucial impact on the preoperative clinical staging, treatment strategy formulation and prognosis evaluation of colorectal cancer. Computed tomography (CT), as a widely used economical non-invasive diagnostic tool, provides doctors with a preliminary basis for evaluating lymph node metastasis. However, in clinical practice, due to the displacement of abdominal tissue during surgery, it is difficult to correspond the lymph nodes during surgery with the lymph nodes in preoperative images, that is, there is a lack of a gold standard for lymph node grade pathology, which makes it impossible to accurately evaluate the metastatic status of a single lymph node, and only the patient-level yin and yang can be judged. A further challenge is that the anatomical structure of tissues and organs in the pelvis is intricate, and the lymph nodes in this area are small in size and numerous, and are often closely adhered to the surrounding tissues, making the evaluation more difficult, and doctors need to invest a lot of time and energy in judgment. Even so, doctors' annotations are greatly affected by subjectivity, resulting in poor consistency between different doctors, and missed diagnosis may still occur. Therefore, using deep learning methods to improve the accuracy and efficiency of lymph node metastasis prediction for clinical auxiliary diagnosis and provide relatively more objective diagnostic results, thereby formulating more personalized treatment plans for patients, is of great significance for improving the treatment effect and quality of life of colorectal cancer patients.

[0003] With the development of artificial intelligence technology, it has also made great progress and application in the field of intelligent medicine, especially the use of neural networks to predict lymph node metastasis. The invention patent application with application number 202010297585.3 discloses an artificial intelligence prediction model for peritoneal metastasis of colorectal cancer and a method for constructing the model, which includes: screening cases, outlining the primary tumor lesions of the cases, constructing and verifying a convolutional neural network model based on the cases and the primary tumor lesions, and constructing and verifying a support vector machine model. The artificial intelligence model for simultaneous peritoneal metastasis of colorectal cancer based on CT features has high prediction accuracy, high sensitivity and specificity. The convolutional neural network model is constructed based on the Resnet-18 architecture.

[0004] The invention patent application with application number 202310536861.0 also discloses a method for predicting rectal cancer lymph node metastasis based on a neural network combined with traditional imaging. In order to construct a feature extraction model, it uses deep learning technology to construct a convolutional neural network (CNN) and a U-Net framework based on a fully convolutional neural network, and uses the Pyradiomics package to extract radiological features; finally, the extracted features are selected and a binary classification model is constructed, and the F1-Score is used for evaluation. In the prediction model of rectal cancer lymph node metastasis, it is found that the XGBoost ensemble learning algorithm is better than other traditional algorithms.

[0005] Like the above two invention patent applications, the existing lymph node metastasis prediction model mainly uses CNN network, U-Net network and ResNet network. When the model is trained, most of them are directly input into the model after preprocessing the sample data and dividing the data set to train the model. It rarely involves calculating the contribution or probability value of the sample data and using it as the learning order during model training, so that the model will not focus more on labels with high contribution or probability values, and there is a problem of imbalance in the number of lymph nodes, which affects the accuracy of lymph node metastasis prediction. Summary of the invention

[0006] The purpose of the present invention is to provide a method, system and device for predicting abdominal lymph node metastasis based on contrastive learning in order to solve the technical problem in the prior art that the model training does not guide the model to focus on a certain type / type of labels, resulting in low accuracy of lymph node metastasis prediction.

[0007] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions: A method for predicting abdominal lymph node metastasis based on contrastive learning comprises the following steps: Step 1: Obtain sample and label data; Obtain an abdominal CT sample image, and annotate the location information of the lymph nodes in the abdominal CT sample image and the positive and negative level of the corresponding patient to obtain label data; Step 2, constructing a lymph node metastasis prediction model; Constructing a lymph node metastasis prediction model, which includes a feature extraction module, an attention module, and a feature fusion module; Step 3, pre-training lymph node metastasis prediction model; Input the abdominal CT sample image and label data in step 1 into the lymph node metastasis prediction model constructed in step 2, and pre-train the lymph node metastasis prediction model using a supervised contrast learning method; Step 4: Online fine-tuning prediction; Construct a feature extractor with the same structure as the feature extraction module, and migrate the parameters in the pre-trained feature extraction module to the feature extractor; construct and train a simple transfer prediction network, which includes a first fully connected layer, a normalization layer, a deactivation layer, a second fully connected layer, and an activation layer; The abdominal CT sample image is input into the feature extractor, and the lymph node features output by the feature extractor are input into the simple metastasis prediction network. The simple metastasis prediction network generates the lymph node metastasis probability, and the contribution of each lymph node in the CT image is calculated according to the lymph node metastasis probability; Step 5: Training the lymph node metastasis prediction model According to the contribution of each lymph node in the abdominal CT image, the abdominal CT sample images are input into the lymph node metastasis prediction model for training. The order of sample input during training is: first input the positive abdominal lymph node sample images with high contribution for training, then input the negative abdominal lymph node sample images with high contribution, and finally input the remaining abdominal lymph node sample images; Step 6, real-time transfer prediction; Obtain an abdominal CT image to be predicted, and input the lymph node metastasis prediction model trained in step 5, and the lymph node metastasis prediction model outputs a prediction result that the corresponding lymph node is positive.

[0008] Furthermore, the abdominal CT sample image is preprocessed, and the specific method is as follows: First, the HU value reflecting the tissue density of the abdominal CT sample image was converted to the optimal display effect area [110, 190], and then normalized and the voxel size of the image was adjusted so that all images had a uniform spatial resolution of 0.75*0.75*2mm; Then, the CT image of each patient is cut into images of size (LNs*3)*1*32*32 according to the annotation, and the calibrated lymph node information is stored in the form of an npy file for each patient.

[0009] Further, in step 2, the feature extraction module includes a first convolution block, a first 2.5D convolution layer, a second convolution block, a second 2.5D convolution layer, a third convolution block, and a linear layer arranged in sequence; The first convolution block, the second convolution block, and the third convolution block each include a convolution layer, a normalization layer, an activation layer, a loss layer, and a pooling layer that are arranged in sequence; The first 2.5D convolution layer and the second 2.5D convolution layer both include a feature fusion convolution layer, which concatenates the labeled frame image with the previous frame image and the next frame image, and then performs convolution processing, and outputs a feature map as well as the previous frame image and the next frame image.

[0010] Furthermore, in step 2, the attention module includes a first linear layer, an activation layer, a second linear layer, and a softmax layer arranged in sequence, the tanh activation function of the first linear layer and the activation layer learns a nonlinear representation of the input, and the second linear layer and the softmax layer are used to generate a final attention score; the calculation method of the attention score is: ; in, represents the attention score of the i-th lymph node feature, , They represent the i-th lymph node feature and the j-th lymph node feature respectively. represents the number of lymph node features, , Both represent learnable parameters, Indicates transpose.

[0011] Furthermore, in step 2, the feature fusion module combines the lymph node features output by the feature extraction module The attention score output by the attention module Multiply and sum to get a patient-level feature of size 128 ; Patient-level characteristics The calculation method is: ; in, represents the attention score of the i-th lymph node feature, represents the feature of the i-th lymph node, Represents the number of lymph node features.

[0012] Furthermore, in step 3, when supervising the pre-trained lymph node metastasis prediction model, the loss function for: ; in, represents the total number of samples, , Respectively represent Patient-level characteristics of samples, Patient-level characteristics of samples, , Respectively represent The positive and negative labels of the samples, The positive and negative labels of samples, represents feature normalization, Indicates cosine similarity calculation. represents the temperature parameter; represents the indicator function, when The positive and negative labels of the samples are If the positive and negative labels of the samples are the same, the value is 1, otherwise it is 0.

[0013] Furthermore, in step 4, the contribution of each abdominal CT sample image is obtained by mapping according to the function f according to the probability of lymph node metastasis. The mapped function f is expressed as: ; in, represents the probability of lymph node metastasis.

[0014] Furthermore, in step 5, when training the lymph node metastasis prediction network, the loss function for: ; in, represents the number of lymph node features at the patient level, represents the positive or negative label of the i-th patient, represents the transition probability obtained by patient i through the network, Represents the weight factor.

[0015] A system for predicting abdominal lymph node metastasis based on contrastive learning, comprising: The sample and label data acquisition module is used to acquire the abdominal CT sample image, and mark the location information of the lymph nodes in the abdominal CT sample image and the positive and negative level of the corresponding patient to obtain label data; A lymph node metastasis prediction model construction module is used to construct a lymph node metastasis prediction model. The lymph node metastasis prediction model includes a feature extraction module, an attention module, and a feature fusion module; A lymph node metastasis prediction model pre-training module is used to input the abdominal CT sample image and label data in the sample and label data acquisition module into the lymph node metastasis prediction model constructed by the lymph node metastasis prediction model construction module, and pre-train the lymph node metastasis prediction model using a supervised contrast learning method; An online fine-tuning prediction module is used to construct a feature extractor with the same structure as the feature extraction module, and migrate the parameters in the pre-trained feature extraction module to the feature extractor; a simple transfer prediction network is constructed and trained, and the simple transfer prediction network includes a first fully connected layer, a normalization layer, a deactivation layer, a second fully connected layer, and an activation layer; The abdominal CT sample image is input into the feature extractor, and the lymph node features output by the feature extractor are input into the simple metastasis prediction network. The simple metastasis prediction network generates the lymph node metastasis probability, and the contribution of each lymph node in the CT image is calculated according to the lymph node metastasis probability; The lymph node metastasis prediction model training module is used to input the abdominal CT sample images into the lymph node metastasis prediction model for training according to the contribution of each lymph node in the abdominal CT image. The order of sample input during training is: first input the positive abdominal lymph node sample images with high contribution for training, then input the negative abdominal lymph node sample images with high contribution, and finally input the remaining abdominal lymph node sample images; The real-time metastasis prediction module is used to obtain the abdominal CT image to be predicted and input the lymph node metastasis prediction model trained by the lymph node metastasis prediction model training module. The lymph node metastasis prediction model outputs the prediction result that the corresponding lymph node is positive.

[0016] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0017] The beneficial effects of the present invention are as follows: 1. In the present invention, the parameters in the pre-trained feature extraction module are first transferred to the feature extractor, and the features extracted by the feature extractor are input into the simple transfer prediction network, and the simple transfer prediction network outputs the lymph node metastasis probability; then the contribution of each lymph node in each sample image is calculated according to the lymph node metastasis probability; finally, when actually training the lymph node metastasis prediction model, the samples are sorted according to the contribution of each lymph node in the sample, and the positive samples with high contribution are learned first, and then the negative samples with high contribution are learned, and finally the samples that are difficult to clearly classify are processed; this is intended to guide the model to focus more on those positive lymph nodes with high contribution to patient-level prediction, effectively solve the problem of imbalance in the number of lymph nodes between patients, and greatly improve the accuracy of lymph node metastasis prediction.

[0018] 2. In the present invention, the feature extraction module is based on ResNet and innovatively introduces a 2.5D convolutional layer. This innovation enables each frame of the annotated image to obtain complementary contextual information of the lymph nodes, which to a certain extent compensates for the information loss that may occur when using a 2D network to process 3D data. It can effectively improve the ability to capture lymph node contextual information, retain key information in the image, and improve the accuracy of lymph node metastasis prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 is a schematic diagram of the structure of the lymph node metastasis prediction model in the present invention; Figure 3 It is a structural schematic diagram of the feature extraction module in the present invention; Figure 4 is a schematic diagram of the structure of the attention module in the present invention; Figure 5 It is a schematic diagram of the structure of the simplified transfer prediction network in the present invention; Figure 6 It is a schematic diagram of the simplified transfer prediction network in the present invention when performing probability estimation. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0021] Therefore, based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0022] Example 1 This embodiment provides a method for predicting abdominal lymph node metastasis based on contrast learning, which is used to process and identify abdominal CT images to obtain a prediction result that the corresponding lymph node is positive, that is, to predict whether the lymph node is positive. Figure 1 As shown, the method comprises the following steps: Step 1: Obtain sample and label data; Obtain an abdominal CT sample image, and annotate the location information of the lymph nodes in the abdominal CT sample image and the corresponding patient-level positivity (it should be noted that the positivity mentioned in this application refers to the patient-level positivity), and obtain label data.

[0023] Abdominal CT sample images come from major hospitals in Sichuan. The data is imported from the system and the location information of all lymph nodes of the patient is calibrated. The patient-level positive and negative annotations are obtained to generate the corresponding dictionary npy file, which contains the patient's hospitalization number and the corresponding positive and negative label (1 or 0 integer). Secondly, the sample data is preprocessed, including converting the HU value reflecting the tissue density of the CT image in the abdominal CT sample image to the best display effect area [110, 190], normalizing it, adjusting the voxel size of the image, so that all images have a uniform spatial resolution of 0.75*0.75*2mm, finding the maximum contour of the pelvis and removing unnecessary areas in the image (such as boundaries), and filling the values ​​of irrelevant areas with 170. Finally, according to the annotation, the CT image of each patient is cut into images of size (LNs*3)*1*32*32. LNs is the number of mesenteric lymph nodes of this patient. *3 is because the calibrated middle frame and upper and lower frames are taken. 3 consecutive frames are used for subsequent 2.5D convolution processing. 1 is a single channel. 32*32 represents the size of the cut patch. The calibrated lymph node information is stored in the form of an npy file for each patient. There is a two-dimensional array in it. Each row represents the coordinate zyx of a lymph node, the labeled long diameter, and the partition type related to the relative position of the organ.

[0024] Step 2, constructing a lymph node metastasis prediction model; Construct a lymph node metastasis prediction model, such as Figure 2 As shown, the lymph node metastasis prediction model includes a feature extraction module, an attention module, and a feature fusion module.

[0025] This feature extraction module uses ResNet18 as the basic architecture and innovatively introduces 2.5D convolutional layers between traditional convolutional blocks. Figure 3 As shown, the feature extraction module includes a first convolution block, a first 2.5D convolution layer, a second convolution block, a second 2.5D convolution layer, a third convolution block, and a linear layer arranged in sequence. The first convolution block, the second convolution block, and the third convolution block all include a convolution layer, a normalization layer, an activation layer, a loss layer, and a pooling layer arranged in sequence. The first 2.5D convolution layer and the second 2.5D convolution layer both include a feature fusion convolution layer, which concatenates the annotated frame image with the previous frame image and the next frame image, and then performs convolution processing, and outputs a feature map and the previous frame image and the next frame image. By introducing a 2.5D convolution layer, the feature extraction module enables each frame of the annotated image to obtain complementary contextual information of the lymph nodes, thereby compensating to a certain extent for the information loss that may occur when using a 2D network to process 3D data. In this feature extraction module, the input image passes through a series of convolutional layers, feature fusion and pooling layers to retain the key information of the image, and is finally flattened and further processed through a linear layer to output a 128-dimensional feature vector, namely the lymph node feature.

[0026] like Figure 4 As shown, the attention module includes a first linear layer, an activation layer, a second linear layer, and a softmax layer arranged in sequence. The tanh activation function of the first linear layer and the activation layer helps learn the nonlinear representation of the input, and the second linear layer and the softmax layer are used to generate the final attention score. The calculation method of the attention score is: ; in, represents the attention score of the i-th lymph node feature, , They represent the i-th lymph node feature and the j-th lymph node feature respectively. represents the number of lymph node features, , Both represent learnable parameters, Indicates transpose.

[0027] The feature fusion module combines the lymph node features output by the feature extraction module The attention score output by the attention module Multiply them together to get a patient-level feature of size 128 ; Patient-level characteristics The calculation method is: ; in, represents the attention score of the i-th lymph node feature, represents the feature of the i-th lymph node, Represents the number of lymph node features.

[0028] Step 3, pre-training lymph node metastasis prediction model; The abdominal CT sample images and label data (which may be part of the sample images and their corresponding label data) in step 1 are input into the lymph node metastasis prediction model constructed in step 2, and the lymph node metastasis prediction model is pre-trained using a supervised contrastive learning method.

[0029] The lymph node metastasis prediction model is pre-trained by using a supervised contrastive learning method, wherein the supervised contrastive learning method can be an existing method. During pre-training, some abdominal CT sample images are input into the lymph node metastasis prediction model to pre-train the lymph node metastasis prediction model; the loss function for: ; in, represents the total number of samples, , Respectively represent Patient-level characteristics of samples, Patient-level characteristics of samples, , Respectively represent The positive and negative labels of the samples, The positive and negative labels of samples, represents feature normalization, Indicates cosine similarity calculation. represents the temperature parameter; represents the indicator function, when The positive and negative labels of the samples are If the positive and negative labels of the samples are the same, the value is 1, otherwise it is 0.

[0030] Step 4: Online fine-tuning prediction; First, build a feature extractor with the same structure as the feature extraction module, and migrate and apply the parameters of the pre-trained feature extraction module to the feature extractor; this step can significantly reduce the training time of the feature extractor and improve its performance on new tasks, thereby avoiding the gradient explosion problem that may be encountered when training from scratch. Figure 5 The simple transfer prediction network shown in FIG. 1 includes a first fully connected layer, a normalization layer, an inactivation layer, a second fully connected layer, and an activation layer; and some abdominal CT sample images are used to train the simple transfer prediction network. When training the simple transfer prediction network, its loss function for: ; in, represents the number of lymph node features at the patient level, represents the positive or negative label of the i-th patient, represents the transition probability obtained by patient i through the network, Represents the weight factor.

[0031] After updating the parameters of the feature extractor and training the simple metastasis prediction network, (part of) the abdominal CT sample image is input into the feature extractor, and the feature extractor outputs the lymph node features. The lymph node features output by the feature extractor are input into the simple metastasis prediction network. In the simple metastasis prediction network, the first fully connected layer first learns the hidden features of a single lymph node and normalizes them. The inactivation layer randomly "turns off" some neurons (i.e. sets their output to 0) to prevent overfitting. The second fully connected layer is connected to the activation layer to improve the nonlinear representation ability of the network. The hidden representation of the network is transformed through a series of transformations to finally generate a lymph node metastasis probability of [0, 1]. The temperature parameter t is introduced in the activation layer to adjust the confidence of the model output (i.e., the contribution of each lymph node in the CT image).

[0032] The simplified metastasis prediction network adopts the multi-instance principle, that is, if at least one instance in a package (consisting of multiple samples in a package) is positive, then the entire package is marked as positive. Correspondingly, if all instances in a package are negative, then the package is marked as negative. The contribution is obtained by mapping with function f, and then sorted from high to low according to the contribution. Our strategy is to first learn the lymph nodes with high positive probability, then focus on the lymph nodes with high negative probability, and finally deal with the lymph nodes that are difficult for the model to clearly classify (i.e., the predicted probability is close to 0.5). This method is designed to guide the model to focus more on those positive lymph nodes with high contribution to patient-level prediction and effectively solve the problem of unbalanced number of lymph nodes between patients. Among them, the mapped function f is expressed as: ; in, represents the probability of lymph node metastasis.

[0033] like Figure 6 As shown in the figure, the specific steps of the simple transfer prediction network when performing probability estimation are as follows: First, combined with clinical knowledge, given a three-dimensional coordinate point (x, y, z) of a lymph node and the total dimension D of the encoding (here is 128 dimensions), the position encoding PE (x, y, z) can be expressed as: ; Here, the encoding of each dimension d is generated by applying sine and cosine functions at different frequencies, given by the following formula: ; ; ; This encoding method allows the model to capture the position information of each element when processing sequence or point cloud data, so that the position information can be effectively used without using recursive or convolutional structures. The diameter size partition type is then normalized and encoded, and then concatenated with the lymph node features output by the feature extractor to obtain multimodal features, which are sent to the second fully connected layer for regression prediction to obtain the transfer probability of the current lymph node, and the patient-level transfer probability estimate is gradually corrected through momentum update. The momentum update is expressed as: ; in, , , They represent the x-position encoding of the coordinate on the D-1th dimension, the y-position encoding of the coordinate on the D-1th dimension, and the z-position encoding of the coordinate on the D-1th dimension, respectively. , , They represent the x-position encoding of the coordinates on the d-th dimension, the y-position encoding of the coordinates on the d-th dimension, and the z-position encoding of the coordinates on the d-th dimension, respectively. represents the momentum update parameter, represents the transition probability of the jth patient, It indicates that the new metastasis probability of the i-th lymph node in the patient is obtained after fine-tuning with clinical information.

[0034] Step 5, training a lymph node metastasis prediction model; According to the contribution of each lymph node in the abdominal CT image, the abdominal CT sample images are input into the lymph node metastasis prediction model for training. The order of sample input during training is: first input the positive abdominal lymph node sample images with high contribution for training, then input the negative abdominal lymph node sample images with high contribution, and finally input the remaining abdominal lymph node sample images. Among them, the evaluation criteria for the high contribution of different types (i.e., positive and negative) of lymph nodes are different. The positive lymph node contribution is greater than 0.2 of e, which is considered to be high, and the negative lymph node contribution is greater than 0.1, which is considered to be high.

[0035] The training method used when training the lymph node metastasis prediction model is consistent with the training method in step 4, and will not be repeated here.

[0036] Step 6, real-time transfer prediction; Obtain an abdominal CT image to be predicted, and input the lymph node metastasis prediction model trained in step 5, and the lymph node metastasis prediction model outputs a prediction result that the corresponding lymph node is positive.

[0037] Example 2 This embodiment provides an abdominal lymph node metastasis prediction system based on contrast learning, which is used to process and identify abdominal CT images to obtain a prediction result that the corresponding lymph node is positive, that is, to predict whether the lymph node is positive. It includes: The sample and label data acquisition module is used to acquire the abdominal CT sample image, and mark the location information of the lymph nodes in the abdominal CT sample image and the positive and negative level of the corresponding patient to obtain label data.

[0038] Abdominal CT sample images come from major hospitals in Sichuan. The data is imported from the system and the location information of all lymph nodes of the patient is calibrated. The patient-level positive and negative annotations are obtained to generate the corresponding dictionary npy file, which contains the patient's hospitalization number and the corresponding positive and negative label (1 or 0 integer). Secondly, the sample data is preprocessed, including converting the HU value reflecting the tissue density of the CT image in the abdominal CT sample image to the best display effect area [110, 190], normalizing it, adjusting the voxel size of the image, so that all images have a uniform spatial resolution of 0.75*0.75*2mm, finding the maximum contour of the pelvis and removing unnecessary areas in the image (such as boundaries), and filling the values ​​of irrelevant areas with 170. Finally, according to the annotation, the CT image of each patient is cut into images of size (LNs*3)*1*32*32. LNs is the number of mesenteric lymph nodes of this patient. *3 is because the calibrated middle frame and upper and lower frames are taken. 3 consecutive frames are used for subsequent 2.5D convolution processing. 1 is a single channel. 32*32 represents the size of the cut patch. The calibrated lymph node information is stored in the form of an npy file for each patient. There is a two-dimensional array in it. Each row represents the coordinate zyx of a lymph node, the labeled long diameter, and the partition type related to the relative position of the organ.

[0039] The lymph node metastasis prediction model construction module is used to construct the lymph node metastasis prediction model. The lymph node metastasis prediction model includes a feature extraction module, an attention module, and a feature fusion module.

[0040] The feature extraction module uses ResNet18 as the basic architecture and innovatively introduces 2.5D convolutional layers between traditional convolutional blocks. The feature extraction module includes a first convolutional block, a first 2.5D convolutional layer, a second convolutional block, a second 2.5D convolutional layer, a third convolutional block, and a linear layer, which are arranged in sequence. The first convolutional block, the second convolutional block, and the third convolutional block all include convolutional layers, normalization layers, activation layers, loss layers, and pooling layers, which are arranged in sequence. The first 2.5D convolutional layer and the second 2.5D convolutional layer both include feature fusion convolutional layers, which concatenate the annotated frame image with the previous frame image and the next frame image, and perform convolution processing, and output feature maps as well as the previous frame image and the next frame image. By introducing a 2.5D convolutional layer, the feature extraction module enables each annotated image frame to obtain complementary contextual information of the lymph nodes, thereby compensating to a certain extent for the information loss that may occur when using a 2D network to process 3D data. In this feature extraction module, the input image passes through a series of convolutional layers, feature fusion and pooling layers to retain the key information of the image, and is finally flattened and further processed through a linear layer to output a 128-dimensional feature vector, namely the lymph node feature.

[0041] The attention module includes the first linear layer, activation layer, second linear layer and softmax layer, which are arranged in sequence. The tanh activation function of the first linear layer and activation layer helps to learn the nonlinear representation of the input, and the second linear layer and softmax layer are used to generate the final attention score. The calculation method of the attention score is: ; in, represents the attention score of the i-th lymph node feature, , They represent the i-th lymph node feature and the j-th lymph node feature respectively. represents the number of lymph node features, , Both represent learnable parameters, Indicates transpose.

[0042] The feature fusion module combines the lymph node features output by the feature extraction module The attention score output by the attention module Multiply and sum to get a patient-level feature of size 128 ; Patient-level characteristics The calculation method is: ; in, represents the attention score of the i-th lymph node feature, represents the feature of the i-th lymph node, Represents the number of lymph node features.

[0043] The lymph node metastasis prediction model pre-training module is used to input the abdominal CT sample images and label data (which may be partial sample images and their corresponding label data) in the sample and label data acquisition module into the lymph node metastasis prediction model constructed by the lymph node metastasis prediction model construction module, and pre-train the lymph node metastasis prediction model using the supervised contrast learning method.

[0044] The lymph node metastasis prediction model is pre-trained by using a supervised contrastive learning method, wherein the supervised contrastive learning method can be an existing method. During pre-training, some abdominal CT sample images are input into the lymph node metastasis prediction model to pre-train the lymph node metastasis prediction model; when pre-training the lymph node metastasis prediction model, the loss function for: ; in, represents the total number of samples, , Respectively represent Patient-level characteristics of samples, Patient-level characteristics of samples, , Respectively represent The positive and negative labels of the samples, The positive and negative labels of samples, represents feature normalization, Indicates cosine similarity calculation. represents the temperature parameter; represents the indicator function, when The positive and negative labels of the samples are If the positive and negative labels of the samples are the same, the value is 1, otherwise it is 0.

[0045] An online fine-tuning prediction module is used to construct a feature extractor with the same structure as the feature extraction module, and migrate the parameters in the pre-trained feature extraction module to the feature extractor; a simple transfer prediction network is constructed and trained, and the simple transfer prediction network includes a first fully connected layer, a normalization layer, an inactivation layer, a second fully connected layer and an activation layer; the abdominal CT sample image is input into the feature extractor, and the lymph node features output by the feature extractor are input into the simple transfer prediction network, and the simple transfer prediction network generates a lymph node metastasis probability, and calculates the contribution of each lymph node in the CT image according to the lymph node metastasis probability.

[0046] First, build a feature extractor with the same structure as the feature extraction module, and migrate and apply the parameters of the pre-trained feature extraction module to the feature extractor; this step can significantly reduce the training time of the feature extractor and improve its performance on new tasks, thereby avoiding the gradient explosion problem that may be encountered when training from scratch. Figure 5 The simple transfer prediction network shown in FIG. 1 includes a first fully connected layer, a normalization layer, an inactivation layer, a second fully connected layer, and an activation layer; and some abdominal CT sample images are used to train the simple transfer prediction network. When training the simple transfer prediction network, its loss function for: ; in, represents the number of lymph node features at the patient level, represents the positive or negative label of the i-th patient, represents the transition probability obtained by patient i through the network, Represents the weight factor.

[0047] After updating the parameters of the feature extractor and training the simple metastasis prediction network, (part of) the abdominal CT sample image is input into the feature extractor, and the feature extractor outputs the lymph node features. The lymph node features output by the feature extractor are input into the simple metastasis prediction network. In the simple metastasis prediction network, the first fully connected layer first learns the hidden features of a single lymph node and normalizes them. The inactivation layer randomly "turns off" some neurons (i.e. sets their output to 0) to prevent overfitting. The second fully connected layer is connected to the activation layer to improve the nonlinear representation ability of the network. The hidden representation of the network is transformed through a series of transformations to finally generate a lymph node metastasis probability of [0, 1]. The temperature parameter t is introduced in the activation layer to adjust the confidence of the model output (i.e., the contribution of each lymph node in the CT image).

[0048] The simplified metastasis prediction network adopts the multi-instance principle, that is, if at least one instance in a package (consisting of multiple samples in a package) is positive, then the entire package is marked as positive. Correspondingly, if all instances in a package are negative, then the package is marked as negative. The contribution is obtained by mapping with function f, and then sorted from high to low according to the contribution. Our strategy is to first learn the lymph nodes with high positive probability, then focus on the lymph nodes with high negative probability, and finally deal with the lymph nodes that are difficult for the model to clearly classify (i.e., the predicted probability is close to 0.5). This method is designed to guide the model to focus more on those positive lymph nodes with high contribution to patient-level prediction and effectively solve the problem of unbalanced number of lymph nodes between patients. Among them, the mapped function f is expressed as: ; in, represents the probability of lymph node metastasis.

[0049] The specific steps of the simple transfer prediction network when performing probability estimation are as follows: First, combined with clinical knowledge, given a three-dimensional coordinate point (x, y, z) of a lymph node and the total dimension D of the encoding (here is 128 dimensions), the position encoding PE (x, y, z) can be expressed as: ; Here, the encoding of each dimension d is generated by applying sine and cosine functions at different frequencies, given by the following formula: ; ; ; This encoding method allows the model to capture the position information of each element when processing sequence or point cloud data, so that the position information can be effectively used without using recursive or convolutional structures. The diameter size partition type is then normalized and encoded, and then concatenated with the lymph node features output by the feature extractor to obtain multimodal features, which are sent to the second fully connected layer for regression prediction to obtain the transfer probability of the current lymph node, and the patient-level transfer probability estimate is gradually corrected through momentum update. The momentum update is expressed as: ; in, , , They represent the x-position encoding of the coordinate on the D-1th dimension, the y-position encoding of the coordinate on the D-1th dimension, and the z-position encoding of the coordinate on the D-1th dimension, respectively. , , They represent the x-position encoding of the coordinates on the d-th dimension, the y-position encoding of the coordinates on the d-th dimension, and the z-position encoding of the coordinates on the d-th dimension, respectively. represents the momentum update parameter, represents the transition probability of the jth patient, It indicates that the new metastasis probability of the i-th lymph node in the patient is obtained after fine-tuning with clinical information.

[0050] The lymph node metastasis prediction model training module is used to input the abdominal CT sample images into the lymph node metastasis prediction model for training according to the contribution of each lymph node in the abdominal CT image. The order of sample input during training is: first input the positive abdominal lymph node sample images with high contribution for training, then input the negative abdominal lymph node sample images with high contribution, and finally input the remaining abdominal lymph node sample images. Among them, the evaluation criteria for the high contribution of different types (i.e., positive and negative) of lymph nodes are different. The positive lymph node contribution is greater than 0.2 of e, which is considered to have a high contribution, and the negative lymph node contribution is greater than 0.1, which is considered to have a high contribution. The training method used in training the lymph node metastasis prediction model is consistent with the training method in step 4, and will not be repeated here.

[0051] The real-time metastasis prediction module is used to obtain the abdominal CT image to be predicted and input the lymph node metastasis prediction model trained by the lymph node metastasis prediction model training module. The lymph node metastasis prediction model outputs the prediction result that the corresponding lymph node is positive.

[0052] Example 3 A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of an abdominal lymph node metastasis prediction method based on contrast learning.

[0053] The computer device may be a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with the user through a keyboard, a mouse, a remote control, a touch pad, or a voice control device.

[0054] The memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory can be an internal storage unit of the computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is often used to store the operating system and various application software installed on the computer device, such as the program code of the abdominal lymph node metastasis prediction method based on contrast learning, etc. In addition, the memory can also be used to temporarily store various types of data that have been output or are to be output.

[0055] The processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip in some embodiments. The processor is generally used to control the overall operation of the computer device. In the present embodiment, the processor is used to run the program code or process data stored in the memory, such as running the program code of the abdominal lymph node metastasis prediction method based on contrast learning.

[0056] Example 4 A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor executes the steps of a method for predicting abdominal lymph node metastasis based on contrast learning.

[0057] The computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned abdominal lymph node metastasis prediction method based on contrast learning.

[0058] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including several instructions for a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the abdominal lymph node metastasis prediction method based on contrast learning described in the embodiment of the present application.

Claims

1. A method for predicting abdominal lymph node metastasis based on contrastive learning, characterized in that: The steps include: Step 1: Obtain sample and label data; Obtain an abdominal CT sample image, and annotate the location information of the lymph nodes in the abdominal CT sample image and the positive and negative level of the corresponding patient to obtain label data; Step 2, constructing a lymph node metastasis prediction model; Constructing a lymph node metastasis prediction model, which includes a feature extraction module, an attention module, and a feature fusion module; Step 3, pre-training lymph node metastasis prediction model; Input the abdominal CT sample image and label data in step 1 into the lymph node metastasis prediction model constructed in step 2, and pre-train the lymph node metastasis prediction model using a supervised contrast learning method; Step 4: Online fine-tuning prediction; Construct a feature extractor with the same structure as the feature extraction module, and migrate the parameters in the pre-trained feature extraction module to the feature extractor; construct and train a simple transfer prediction network, which includes a first fully connected layer, a normalization layer, a deactivation layer, a second fully connected layer, and an activation layer; The abdominal CT sample image is input into the feature extractor, and the lymph node features output by the feature extractor are input into the simple metastasis prediction network. The simple metastasis prediction network generates the lymph node metastasis probability, and the contribution of each lymph node in the CT image is calculated according to the lymph node metastasis probability; Step 5, training a lymph node metastasis prediction model; According to the contribution of each lymph node in the abdominal CT image, the abdominal CT sample images are input into the lymph node metastasis prediction model for training. The order of sample input during training is: first input the positive abdominal lymph node sample images with high contribution for training, then input the negative abdominal lymph node sample images with high contribution, and finally input the remaining abdominal lymph node sample images; Step 6, real-time transfer prediction; Obtain an abdominal CT image to be predicted, and input the lymph node metastasis prediction model trained in step 5, and the lymph node metastasis prediction model outputs a prediction result that the corresponding lymph node is positive.

2. The method for predicting abdominal lymph node metastasis based on contrastive learning according to claim 1, characterized in that: The abdominal CT sample images are preprocessed as follows: First, the HU value reflecting the tissue density of the abdominal CT sample image was converted to the optimal display effect area [110, 190], and then normalized and the voxel size of the image was adjusted so that all images had a uniform spatial resolution of 0.75*0.75*2mm; Then, the CT image of each patient is cut into images of size (LNs*3)*1*32*32 according to the annotation, and the calibrated lymph node information is stored in the form of an npy file for each patient.

3. The method for predicting abdominal lymph node metastasis based on contrastive learning according to claim 1, characterized in that: In step 2, the feature extraction module includes a first convolution block, a first 2.5D convolution layer, a second convolution block, a second 2.5D convolution layer, a third convolution block, and a linear layer, which are arranged in sequence; The first convolution block, the second convolution block, and the third convolution block each include a convolution layer, a normalization layer, an activation layer, a loss layer, and a pooling layer that are arranged in sequence; The first 2.5D convolution layer and the second 2.5D convolution layer both include a feature fusion convolution layer, which concatenates the labeled frame image with the previous frame image and the next frame image, and then performs convolution processing, and outputs a feature map as well as the previous frame image and the next frame image.

4. The method for predicting abdominal lymph node metastasis based on contrastive learning according to claim 1, characterized in that: In step 2, the attention module includes a first linear layer, an activation layer, a second linear layer, and a softmax layer arranged in sequence, the tanh activation function of the first linear layer and the activation layer learns a nonlinear representation of the input, and the second linear layer and the softmax layer are used to generate the final attention score; The attention score is calculated as: ； in, represents the attention score of the i-th lymph node feature, , They represent the i-th lymph node feature and the j-th lymph node feature respectively. represents the number of lymph node features, , Both represent learnable parameters, Indicates transpose.

5. The method for predicting abdominal lymph node metastasis based on contrastive learning according to claim 1, characterized in that: In step 2, the feature fusion module combines the lymph node features output by the feature extraction module The attention score output by the attention module Multiply and sum to get a patient-level feature of size 128 ; Patient-level characteristics The calculation method is: ； in, represents the attention score of the i-th lymph node feature, represents the feature of the i-th lymph node, Represents the number of lymph node features.

6. The method for predicting abdominal lymph node metastasis based on contrastive learning according to claim 1, characterized in that: In step 3, when supervising the pre-trained lymph node metastasis prediction model, the loss function for: ； in, represents the total number of samples, , Respectively represent Patient-level characteristics of samples, Patient-level characteristics of samples, , Respectively represent The positive and negative labels of the samples, The positive and negative labels of samples, represents feature normalization, Indicates cosine similarity calculation. represents the temperature parameter; represents the indicator function, when The positive and negative labels of the samples are If the positive and negative labels of the samples are the same, the value is 1, otherwise it is 0.

7. The method for predicting abdominal lymph node metastasis based on contrastive learning according to claim 1, characterized in that: In step 4, the contribution of each abdominal CT sample image is obtained by mapping the lymph node metastasis probability according to function f. The mapped function f is expressed as: ； in, represents the probability of lymph node metastasis.

8. The method for predicting abdominal lymph node metastasis based on contrastive learning according to claim 1, characterized in that: In step 5, when training the lymph node metastasis prediction network, the loss function for: ； in, represents the number of lymph node features at the patient level, represents the positive or negative label of the i-th patient, represents the transition probability obtained by patient i through the network, Represents the weight factor.

9. A system for predicting abdominal lymph node metastasis based on contrastive learning, characterized in that: include: The sample and label data acquisition module is used to acquire the abdominal CT sample image, and mark the location information of the lymph nodes in the abdominal CT sample image and the positive and negative level of the corresponding patient to obtain label data; A lymph node metastasis prediction model construction module is used to construct a lymph node metastasis prediction model. The lymph node metastasis prediction model includes a feature extraction module, an attention module, and a feature fusion module; A lymph node metastasis prediction model pre-training module is used to input the abdominal CT sample image and label data in the sample and label data acquisition module into the lymph node metastasis prediction model constructed by the lymph node metastasis prediction model construction module, and pre-train the lymph node metastasis prediction model using a supervised contrast learning method; An online fine-tuning prediction module is used to construct a feature extractor with the same structure as the feature extraction module, and migrate the parameters in the pre-trained feature extraction module to the feature extractor; a simple transfer prediction network is constructed and trained, and the simple transfer prediction network includes a first fully connected layer, a normalization layer, a deactivation layer, a second fully connected layer, and an activation layer; The abdominal CT sample image is input into the feature extractor, and the lymph node features output by the feature extractor are input into the simple metastasis prediction network. The simple metastasis prediction network generates the lymph node metastasis probability, and the contribution of each lymph node in the CT image is calculated according to the lymph node metastasis probability; The lymph node metastasis prediction model training module is used to input the abdominal CT sample images into the lymph node metastasis prediction model for training according to the contribution of each lymph node in the abdominal CT image. The order of sample input during training is: first input the positive abdominal lymph node sample images with high contribution for training, then input the negative abdominal lymph node sample images with high contribution, and finally input the remaining abdominal lymph node sample images; The real-time metastasis prediction module is used to obtain the abdominal CT image to be predicted and input the lymph node metastasis prediction model trained by the lymph node metastasis prediction model training module. The lymph node metastasis prediction model outputs the prediction result that the corresponding lymph node is positive.

10. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 8.

Images