Abdominal lymph node metastasis prediction method, system and device based on contrastive learning

Abstract

The present invention discloses a method, system and device for predicting abdominal lymph node metastasis based on contrastive learning, which belongs to the field of lymph node metastasis prediction in the field of artificial intelligence technology. The purpose is to solve the technical problem that the model training in the prior art does not guide the model to focus on a certain type / type of label, resulting in low accuracy of lymph node metastasis prediction. The method first migrates the parameters in the feature extraction module after supervised contrastive learning pre-training to the feature extractor, and inputs the features extracted by the feature extractor into a simple transfer prediction network, and the simple transfer prediction network outputs the probability of lymph node metastasis; then the contribution of each sample image is calculated according to the probability of lymph node metastasis; finally, when actually training the lymph node metastasis prediction model, the samples are sorted according to their contribution, and positive samples with high contribution are learned first, and then negative samples with high contribution are learned, and finally samples that are difficult to clearly classify are processed; this can greatly improve the accuracy of lymph node metastasis prediction.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and relates to lymph node metastasis prediction, and in particular to a method, system and device for abdominal lymph node metastasis prediction based on contrastive learning. Background Art

[0002] Colorectal cancer is one of the three major malignancies worldwide. Lymph node metastasis plays a crucial role in preoperative clinical staging, treatment strategy development, and prognostic assessment. Computed tomography (CT), a widely used, economical, and non-invasive diagnostic tool, provides physicians with a preliminary basis for assessing lymph node metastasis. However, in clinical practice, due to the displacement of abdominal tissue during surgery, it is difficult to precisely match intraoperative lymph nodes with those seen on preoperative imaging. This lack of a gold standard for lymph node grade pathology makes it difficult to accurately assess the metastatic status of individual lymph nodes, and only allows for patient-level positivity assessment. Further challenges arise from the complex anatomy of pelvic tissues and organs, as well as the small, numerous, and often tightly adherent lymph nodes in this region, making assessment significantly more challenging and requiring physicians to devote considerable time and effort to assessment. Despite this, physician annotation is subject to significant subjectivity, resulting in poor inter-doctoral consistency and the potential for missed diagnoses. Therefore, using deep learning methods to improve the accuracy and efficiency of lymph node metastasis prediction for clinical auxiliary diagnosis and provide relatively more objective diagnostic results, thereby formulating more personalized treatment plans for patients, is of great significance for improving the treatment effect and quality of life of colorectal cancer patients.

[0003] With the development of artificial intelligence technology, it has also made great progress and application in the field of intelligent medicine, especially the use of neural networks to predict lymph node metastasis. The invention patent application with application number 202010297585.3 discloses an artificial intelligence prediction model for peritoneal metastasis of colorectal cancer and a method for constructing the model, which includes: screening cases, outlining the primary tumor lesions of the cases, constructing and verifying a convolutional neural network model based on the cases and the primary tumor lesions, and constructing and verifying a support vector machine model. The artificial intelligence model for simultaneous peritoneal metastasis of colorectal cancer based on CT features has high prediction accuracy, high sensitivity and specificity. The convolutional neural network model is constructed based on the Resnet-18 architecture.

[0004] The invention patent application with application number 202310536861.0 also discloses a method for predicting rectal cancer lymph node metastasis based on a neural network combined with traditional imaging. In order to construct a feature extraction model, it uses deep learning technology to construct a convolutional neural network (CNN) and a U-Net framework based on a fully convolutional neural network, and uses the Pyradiomics package to extract radiological features; finally, feature selection is performed on the extracted features and a binary classification model is constructed and evaluated using F1-Score. In the prediction model of rectal cancer lymph node metastasis, it was found that the XGBoost ensemble learning algorithm is more effective than other traditional algorithms.

[0005] As in the two aforementioned invention patent applications, existing lymph node metastasis prediction models primarily utilize CNN, U-Net, and ResNet networks. During model training, most preprocess and partition sample data into datasets before directly inputting them into the model for training. This rarely involves calculating the contribution or probability of the sample data and prioritizing this in model training. This results in the model focusing less on labels with high contribution or probability values, leading to an imbalance in the number of lymph nodes and affecting the accuracy of lymph node metastasis prediction. Summary of the Invention

[0006] The purpose of the present invention is to provide a method, system and device for predicting abdominal lymph node metastasis based on contrastive learning in order to solve the technical problem in the prior art that the model training does not guide the model to focus on a certain type / type of labels, resulting in low accuracy of lymph node metastasis prediction.

[0007] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0008] A method for predicting abdominal lymph node metastasis based on contrastive learning comprises the following steps:

[0009] Step 1: Obtain sample and label data;

[0010] Obtain abdominal CT sample images, and annotate the location information of the lymph nodes in the abdominal CT sample images and the positive and negative levels of the corresponding patient levels to obtain label data;

[0011] Step 2: construct a lymph node metastasis prediction model;

[0012] Construct a lymph node metastasis prediction model, which includes a feature extraction module, an attention module, and a feature fusion module;

[0013] Step 3: pre-training the lymph node metastasis prediction model;

[0014] Input the abdominal CT sample images and label data in step 1 into the lymph node metastasis prediction model constructed in step 2, and pre-train the lymph node metastasis prediction model using a supervised contrastive learning method;

[0015] Step 4: Online fine-tuning prediction;

[0016] Construct a feature extractor with the same structure as the feature extraction module and migrate the parameters of the pre-trained feature extraction module to the feature extractor; construct and train a simple transfer prediction network, which includes a first fully connected layer, a normalization layer, a deactivation layer, a second fully connected layer, and an activation layer;

[0017] The abdominal CT sample image is input into the feature extractor. The lymph node features output by the feature extractor are input into the simple metastasis prediction network. The simple metastasis prediction network generates the lymph node metastasis probability and calculates the contribution of each lymph node in the CT image based on the lymph node metastasis probability.

[0018] Step 5: Train the lymph node metastasis prediction model

[0019] According to the contribution of each lymph node in the abdominal CT image, the abdominal CT sample images are input into the lymph node metastasis prediction model for training. The order of sample input during training is: first input the positive abdominal lymph node sample images with high contribution for training, then input the negative abdominal lymph node sample images with high contribution, and finally input the remaining abdominal lymph node sample images;

[0020] Step 6, real-time transfer prediction;

[0021] Obtain the abdominal CT image to be predicted and input the lymph node metastasis prediction model trained in step 5. The lymph node metastasis prediction model outputs a prediction result that the corresponding lymph node is positive.

[0022] Furthermore, the abdominal CT sample image is preprocessed as follows:

[0023] First, the HU values ​​reflecting the tissue density of the abdominal CT sample images were converted to the optimal display effect area [110, 190], and then normalized and the voxel size of the images was adjusted to make all images have a uniform spatial resolution of 0.75*0.75*2mm;

[0024] Then, according to the annotation, the CT image of each patient is cut into an image of size (LNs*3)*1*32*32, and the calibrated lymph node information is stored in the form of an npy file for each patient.

[0025] Furthermore, in step 2, the feature extraction module includes a first convolution block, a first 2.5D convolution layer, a second convolution block, a second 2.5D convolution layer, a third convolution block, and a linear layer, which are arranged in sequence;

[0026] The first convolution block, the second convolution block, and the third convolution block each include a convolution layer, a normalization layer, an activation layer, a dropout layer, and a pooling layer arranged in sequence;

[0027] The first 2.5D convolution layer and the second 2.5D convolution layer both include a feature fusion convolution layer. The feature fusion convolution layer concatenates the labeled frame image with the previous frame image and the next frame image, performs convolution processing, and outputs a feature map as well as the previous frame image and the next frame image.

[0028] Furthermore, in step 2, the attention module includes a first linear layer, an activation layer, a second linear layer, and a softmax layer arranged in sequence. The tanh activation function of the first linear layer and the activation layer learns the nonlinear representation of the input, and the second linear layer and the softmax layer are used to generate the final attention score. The calculation method of the attention score is:

[0029] ;

[0030] in, represents the attention score of the i-th lymph node feature, 、 Respectively represent the characteristics of the i-th lymph node and the j-th lymph node, represents the number of lymph node features, 、 Both represent learnable parameters, Indicates transpose.

[0031] Furthermore, in step 2, the feature fusion module combines the lymph node features output by the feature extraction module The attention score output by the attention module Multiply and sum to get patient-level features of size 128 Patient-level characteristics The calculation method is:

[0032] ;

[0033] in, represents the attention score of the i-th lymph node feature, represents the characteristics of the i-th lymph node, Represents the number of lymph node features.

[0034] Furthermore, in step 3, when supervising the pre-trained lymph node metastasis prediction model, the loss function for:

[0035] ;

[0036] in, represents the total number of samples, 、 Respectively represent Patient-level characteristics of samples, Patient-level characteristics of samples, 、 Respectively represent The positive and negative labels of the samples, The positive and negative labels of the samples, represents feature normalization, Indicates cosine similarity calculation, represents the temperature parameter; Represents the indicator function, when The positive and negative labels of the first sample and the If the positive and negative labels of the samples are the same, the value is 1, otherwise it is 0.

[0037] Furthermore, in step 4, the contribution of each abdominal CT sample image is obtained by mapping the lymph node metastasis probability according to the function f. The mapped function f is expressed as:

[0038] ;

[0039] in, represents the probability of lymph node metastasis.

[0040] Furthermore, in step 5, when training the lymph node metastasis prediction network, the loss function for:

[0041] ;

[0042] in, represents the number of lymph node features at the patient level, represents the positive or negative label of the i-th patient, represents the transfer probability obtained by patient i through the network, Represents the weight factor.

[0043] A system for predicting abdominal lymph node metastasis based on contrastive learning, comprising:

[0044] The sample and label data acquisition module is used to obtain abdominal CT sample images, mark the location information of the lymph nodes in the abdominal CT sample images and the positive and negative levels of the corresponding patients, and obtain label data;

[0045] Lymph node metastasis prediction model construction module, used to construct a lymph node metastasis prediction model, which includes a feature extraction module, an attention module, and a feature fusion module;

[0046] A lymph node metastasis prediction model pre-training module is used to input the abdominal CT sample images and label data in the sample and label data acquisition module into the lymph node metastasis prediction model constructed by the lymph node metastasis prediction model construction module, and pre-train the lymph node metastasis prediction model using a supervised contrast learning method;

[0047] The online fine-tuning prediction module is used to build a feature extractor with the same structure as the feature extraction module and migrate the parameters of the pre-trained feature extraction module to the feature extractor; the simple transfer prediction network is built and trained. The simple transfer prediction network includes a first fully connected layer, a normalization layer, a deactivation layer, a second fully connected layer, and an activation layer;

[0048] The abdominal CT sample image is input into the feature extractor. The lymph node features output by the feature extractor are input into the simple metastasis prediction network. The simple metastasis prediction network generates the lymph node metastasis probability and calculates the contribution of each lymph node in the CT image based on the lymph node metastasis probability.

[0049] The lymph node metastasis prediction model training module is used to input abdominal CT sample images into the lymph node metastasis prediction model for training based on the contribution of each lymph node in the abdominal CT image. The order of sample input during training is: first input positive abdominal lymph node sample images with high contribution for training, then input negative abdominal lymph node sample images with high contribution, and finally input the remaining abdominal lymph node sample images;

[0050] The real-time metastasis prediction module is used to obtain the abdominal CT image to be predicted and input the lymph node metastasis prediction model trained by the lymph node metastasis prediction model training module. The lymph node metastasis prediction model outputs the prediction result that the corresponding lymph node is positive.

[0051] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0052] The beneficial effects of the present invention are as follows:

[0053] 1. In the present invention, the parameters in the pre-trained feature extraction module are first transferred to the feature extractor, and the features extracted by the feature extractor are input into the simple transfer prediction network, and the simple transfer prediction network outputs the lymph node metastasis probability; then, the contribution of each lymph node in each sample image is calculated based on the lymph node metastasis probability; finally, when actually training the lymph node metastasis prediction model, the samples are sorted according to the contribution of each lymph node, and positive samples with high contribution are learned first, then negative samples with high contribution are learned, and finally samples that are difficult to clearly classify are processed; this is intended to guide the model to focus more on those positive lymph nodes that have a high contribution to patient-level prediction, effectively solve the problem of imbalance in the number of lymph nodes between patients, and greatly improve the accuracy of lymph node metastasis prediction.

[0054] 2. In the present invention, the feature extraction module is based on ResNet and innovatively introduces a 2.5D convolutional layer. This innovation enables each frame of the annotated image to obtain complementary contextual information of the lymph nodes, which to a certain extent compensates for the information loss that may occur when using a 2D network to process 3D data. It can effectively improve the ability to capture lymph node contextual information, retain key information in the image, and improve the accuracy of lymph node metastasis prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic flow diagram of the present invention;

[0056] Figure 2 Schematic diagram of the structure of the lymph node metastasis prediction model of the present invention;

[0057] Figure 3 It is a structural diagram of the feature extraction module in the present invention;

[0058] Figure 4 Schematic diagram of the structure of the attention module in the present invention;

[0059] Figure 5 It is a schematic diagram of the structure of the simplified transfer prediction network in the present invention;

[0060] Figure 6 It is a schematic diagram of the simplified transfer prediction network in the present invention when performing probability estimation. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0062] Therefore, based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0063] Example 1

[0064] This embodiment provides a method for predicting abdominal lymph node metastasis based on contrast learning, which is used to process and identify abdominal CT images to obtain a prediction result of whether the corresponding lymph node is positive, that is, to predict whether the lymph node is positive. Figure 1 As shown, the method includes the following steps:

[0065] Step 1: Obtain sample and label data;

[0066] Obtain an abdominal CT sample image, and annotate the location information of the lymph nodes in the abdominal CT sample image and the corresponding patient-level positivity (it should be noted that the positivity mentioned in this application refers to the patient-level positivity) to obtain label data.

[0067] Abdominal CT sample images were collected from major hospitals in Sichuan. The data was imported from the system and the location information of all patient lymph nodes was calibrated. Patient-level positive / negative labels were obtained and a corresponding dictionary npy file was generated. This file contains the patient's hospitalization number and the corresponding positive / negative label (1 or 0 integer). Next, the sample data was preprocessed. This included converting the HU values ​​(Human Units of Scale, or HU) in the abdominal CT sample images, which reflect tissue density, to the optimal display range [110, 190]. These values ​​were then normalized and the voxel size was adjusted to a uniform spatial resolution of 0.75*0.75*2mm for all images. The maximum pelvic contour was then found and unnecessary areas (such as the border) were removed from the image. The values ​​of irrelevant areas were filled with 170. Finally, based on the annotation, the CT image of each patient is cut into an image of size (LNs*3)*1*32*32. LNs is the number of mesenteric lymph nodes in this patient. *3 is because the calibrated middle frame and upper and lower frames are taken. Three consecutive frames are used for subsequent 2.5D convolution processing. 1 is a single channel. 32*32 represents the size of the cut patch. The calibrated lymph node information is stored in the form of an npy file for each patient. It contains a two-dimensional array. Each row represents the coordinate zyx of a lymph node, the labeled long diameter, and the partition type related to the relative position of the organ.

[0068] Step 2: construct a lymph node metastasis prediction model;

[0069] Construct a lymph node metastasis prediction model, such as Figure 2 As shown in the figure, the lymph node metastasis prediction model includes a feature extraction module, an attention module, and a feature fusion module.

[0070] This feature extraction module uses ResNet18 as its basic architecture and innovatively introduces 2.5D convolutional layers between traditional convolutional blocks. Figure 3 As shown, the feature extraction module includes a first convolution block, a first 2.5D convolution layer, a second convolution block, a second 2.5D convolution layer, a third convolution block, and a linear layer arranged in sequence. The first convolution block, the second convolution block, and the third convolution block all include a convolution layer, a normalization layer, an activation layer, a loss layer, and a pooling layer arranged in sequence. The first 2.5D convolution layer and the second 2.5D convolution layer both include a feature fusion convolution layer, which concatenates the annotated frame image with the previous frame image and the next frame image, and then performs convolution processing, and outputs a feature map as well as the previous frame image and the next frame image. By introducing a 2.5D convolution layer, the feature extraction module enables each annotated image frame to obtain complementary contextual information of the lymph nodes, thereby compensating to a certain extent for the information loss that may occur when using a 2D network to process 3D data. In this feature extraction module, the input image passes through a series of convolutional layers, feature fusion and pooling layers to retain the key information of the image. It is finally flattened and further processed through a linear layer to output a 128-dimensional feature vector, namely the lymph node feature.

[0071] like Figure 4 As shown in the figure, the attention module includes a first linear layer, an activation layer, a second linear layer, and a softmax layer. The tanh activation function of the first linear layer and the activation layer helps learn the nonlinear representation of the input. The second linear layer and the softmax layer are used to generate the final attention score. The calculation method of the attention score is:

[0072] ;

[0073] in, represents the attention score of the i-th lymph node feature, 、 Respectively represent the characteristics of the i-th lymph node and the j-th lymph node, represents the number of lymph node features, 、 Both represent learnable parameters, Indicates transpose.

[0074] The feature fusion module combines the lymph node features output by the feature extraction module The attention score output by the attention module Multiply together to obtain a patient-level feature with a size of 128 dimensions ; Patient-level characteristics The calculation method is:

[0075] ;

[0076] in, represents the attention score of the i-th lymph node feature, represents the characteristics of the i-th lymph node, Represents the number of lymph node features.

[0077] Step 3: pre-training the lymph node metastasis prediction model;

[0078] The abdominal CT sample images and label data (which may be partial sample images and their corresponding label data) in step 1 are input into the lymph node metastasis prediction model constructed in step 2, and the lymph node metastasis prediction model is pre-trained using a supervised contrastive learning method.

[0079] The lymph node metastasis prediction model is pre-trained using a supervised contrastive learning method, wherein the supervised contrastive learning method can be an existing method. During pre-training, some abdominal CT sample images are input into the lymph node metastasis prediction model to pre-train the lymph node metastasis prediction model; the loss function for:

[0080] ;

[0081] in, represents the total number of samples, 、 Respectively represent Patient-level characteristics of samples, Patient-level characteristics of samples, 、 Respectively represent The positive and negative labels of the samples, The positive and negative labels of the samples, represents feature normalization, Indicates cosine similarity calculation, represents the temperature parameter; Represents the indicator function, when The positive and negative labels of the first sample and the If the positive and negative labels of the samples are the same, the value is 1, otherwise it is 0.

[0082] Step 4: Online fine-tuning prediction;

[0083] First, build a feature extractor with the same structure as the feature extraction module, and migrate and apply the parameters of the pre-trained feature extraction module to the feature extractor; this step can significantly reduce the training time of the feature extractor and improve its performance on new tasks, thus avoiding the gradient explosion problem that may be encountered when training from scratch. Figure 5The simple transfer prediction network shown in the figure includes the first fully connected layer, the normalization layer, the inactivation layer, the second fully connected layer and the activation layer; and some abdominal CT sample images are used to train the simple transfer prediction network. When training the simple transfer prediction network, its loss function is for:

[0084] ;

[0085] in, represents the number of patient-level lymph node features, represents the positive or negative label of the i-th patient, represents the transfer probability obtained by patient i through the network, Represents the weight factor.

[0086] After updating the parameters of the feature extractor and training the simple metastasis prediction network, (part of) the abdominal CT sample image is input into the feature extractor, which outputs the lymph node features. The lymph node features output by the feature extractor are then input into the simple metastasis prediction network. In the simple metastasis prediction network, the first fully connected layer first learns the hidden features of a single lymph node and normalizes them. The inactivation layer randomly "turns off" some neurons (i.e., sets their output to 0) to prevent overfitting. The second fully connected layer is connected to the activation layer to improve the nonlinear representation ability of the network. The hidden representation of the network is transformed through a series of transformations to finally generate a lymph node metastasis probability of [0, 1]. The temperature parameter t is introduced in the activation layer to adjust the confidence of the model output (i.e., the contribution of each lymph node in the CT image).

[0087] The simple metastasis prediction network adopts the multi-instance principle, that is, if at least one instance in a package (consisting of multiple samples) is positive, then the entire package is marked as positive. Correspondingly, if all instances in a package are negative, then the package is marked as negative. The contribution is mapped according to the function f and then sorted from high to low by contribution. Our strategy is to prioritize lymph nodes with a high positive probability, then focus on lymph nodes with a high negative probability, and finally process lymph nodes that the model cannot clearly classify (i.e., those with a predicted probability close to 0.5). This method is designed to guide the model to focus more on positive lymph nodes that contribute highly to patient-level predictions and effectively address the problem of imbalanced lymph node numbers between patients. The mapping function f is expressed as:

[0088] ;

[0089] in, represents the probability of lymph node metastasis.

[0090] like Figure 6 As shown in Figure 2, the specific steps of the simple transfer prediction network when performing probability estimation are as follows:

[0091] First, combined with clinical knowledge, given a three-dimensional coordinate point (x, y, z) of a lymph node and the total dimension D of the encoding (here is 128 dimensions), the position encoding PE(x, y, z) can be expressed as:

[0092] ;

[0093] Here, the encoding of each dimension d is generated by applying sine and cosine functions at different frequencies, given by the following formula:

[0094] ;

[0095] ;

[0096] ;

[0097] This encoding method allows the model to capture the position information of each element when processing sequence or point cloud data, thereby effectively utilizing position information without using recursive or convolutional structures. The diameter size partition type is then normalized and encoded, and then concatenated with the lymph node features output by the feature extractor to obtain multimodal features. This is then fed into the second fully connected layer for regression prediction to obtain the current lymph node metastasis probability, and the patient-level metastasis probability estimate is gradually corrected through momentum update. The momentum update is expressed as:

[0098] ;

[0099] in, 、 、 They represent the x-position encoding of the coordinate on the D-1th dimension, the y-position encoding of the coordinate on the D-1th dimension, and the z-position encoding of the coordinate on the D-1th dimension, respectively. 、 、 Respectively represent the x-position encoding of the coordinate on the d-th dimension, the y-position encoding of the coordinate on the d-th dimension, and the z-position encoding of the coordinate on the d-th dimension, represents the momentum update parameter, represents the transition probability of the jth patient, It indicates the new metastasis probability of the i-th lymph node in the patient after fine-tuning with clinical information.

[0100] Step 5: training a lymph node metastasis prediction model;

[0101] Based on the contribution of each lymph node in the abdominal CT images, sample abdominal CT images are fed into the lymph node metastasis prediction model for training. The training sequence is as follows: first, sample images of positive abdominal lymph nodes with high contribution are fed, followed by sample images of negative abdominal lymph nodes with high contribution, and finally, the remaining sample images are fed. The contribution of different lymph nodes (i.e., positive or negative) is evaluated using different criteria: a positive lymph node is considered high if its contribution is greater than 0.2, while a negative lymph node is considered high if its contribution is greater than 0.1.

[0102] The training method used to train the lymph node metastasis prediction model is the same as the training method in step 4 and will not be repeated here.

[0103] Step 6, real-time transfer prediction;

[0104] Obtain the abdominal CT image to be predicted and input the lymph node metastasis prediction model trained in step 5. The lymph node metastasis prediction model outputs a prediction result that the corresponding lymph node is positive.

[0105] Example 2

[0106] This embodiment provides an abdominal lymph node metastasis prediction system based on contrastive learning, which is used to process and identify abdominal CT images to obtain a prediction result of whether the corresponding lymph node is positive, that is, to predict whether the lymph node is positive. It includes:

[0107] The sample and label data acquisition module is used to obtain abdominal CT sample images, and mark the location information of the lymph nodes in the abdominal CT sample images and the positive and negative levels of the corresponding patients to obtain label data.

[0108] Abdominal CT sample images were collected from major hospitals in Sichuan. The data was imported from the system and the location information of all patient lymph nodes was calibrated. Patient-level positive / negative labels were obtained and a corresponding dictionary npy file was generated. This file contains the patient's hospitalization number and the corresponding positive / negative label (1 or 0 integer). Next, the sample data was preprocessed. This included converting the HU values ​​(Human Units of Scale, or HU) in the abdominal CT sample images, which reflect tissue density, to the optimal display range [110, 190]. These values ​​were then normalized and the voxel size was adjusted to a uniform spatial resolution of 0.75*0.75*2mm for all images. The maximum pelvic contour was then found and unnecessary areas (such as the border) were removed from the image. The values ​​of irrelevant areas were filled with 170. Finally, based on the annotation, the CT image of each patient is cut into an image of size (LNs*3)*1*32*32. LNs is the number of mesenteric lymph nodes in this patient. *3 is because the calibrated middle frame and upper and lower frames are taken. Three consecutive frames are used for subsequent 2.5D convolution processing. 1 is a single channel. 32*32 represents the size of the cut patch. The calibrated lymph node information is stored in the form of an npy file for each patient. It contains a two-dimensional array. Each row represents the coordinate zyx of a lymph node, the labeled long diameter, and the partition type related to the relative position of the organ.

[0109] The lymph node metastasis prediction model construction module is used to construct a lymph node metastasis prediction model. The lymph node metastasis prediction model includes a feature extraction module, an attention module, and a feature fusion module.

[0110] The feature extraction module uses ResNet18 as its basic architecture and innovatively introduces 2.5D convolution layers between traditional convolution blocks. The feature extraction module includes a first convolution block, a first 2.5D convolution layer, a second convolution block, a second 2.5D convolution layer, a third convolution block, and a linear layer, which are arranged in sequence. The first convolution block, the second convolution block, and the third convolution block all include a convolution layer, a normalization layer, an activation layer, a dropout layer, and a pooling layer, which are arranged in sequence. The first 2.5D convolution layer and the second 2.5D convolution layer both include a feature fusion convolution layer, which concatenates the annotated frame image with the previous and next frames, and then performs convolution processing, and outputs a feature map as well as the previous and next frames. By introducing 2.5D convolution layers, the feature extraction module enables each annotated image frame to obtain complementary contextual information of the lymph nodes, thereby compensating to a certain extent for the information loss that may occur when using a 2D network to process 3D data. In this feature extraction module, the input image passes through a series of convolutional layers, feature fusion and pooling layers to retain the key information of the image. It is finally flattened and further processed through a linear layer to output a 128-dimensional feature vector, namely the lymph node feature.

[0111] The attention module consists of a first linear layer, an activation layer, a second linear layer, and a softmax layer. The tanh activation function of the first linear layer and the activation layer helps learn the nonlinear representation of the input. The second linear layer and the softmax layer are used to generate the final attention score. The calculation method of the attention score is:

[0112] ;

[0113] in, represents the attention score of the i-th lymph node feature, 、 Respectively represent the characteristics of the i-th lymph node and the j-th lymph node, represents the number of lymph node features, 、 Both represent learnable parameters, Indicates transpose.

[0114] The feature fusion module combines the lymph node features output by the feature extraction module The attention score output by the attention module Multiply and sum to get patient-level features of size 128 Patient-level characteristics The calculation method is:

[0115] ;

[0116] in, represents the attention score of the i-th lymph node feature, represents the characteristics of the i-th lymph node, Represents the number of lymph node features.

[0117] The lymph node metastasis prediction model pre-training module is used to input the abdominal CT sample images and label data (which can be partial sample images and their corresponding label data) in the sample and label data acquisition module into the lymph node metastasis prediction model constructed by the lymph node metastasis prediction model construction module, and pre-train the lymph node metastasis prediction model using the supervised contrast learning method.

[0118] The supervised contrastive learning method is used to pre-train the lymph node metastasis prediction model, wherein the supervised contrastive learning method can adopt the existing method. During pre-training, some abdominal CT sample images are input into the lymph node metastasis prediction model to pre-train the lymph node metastasis prediction model; when pre-training the lymph node metastasis prediction model, the loss function for:

[0119] ;

[0120] in, represents the total number of samples, 、 Respectively represent Patient-level characteristics of samples, Patient-level characteristics of samples, 、 Respectively represent The positive and negative labels of the samples, The positive and negative labels of the samples, represents feature normalization, Indicates cosine similarity calculation, represents the temperature parameter; Represents the indicator function, when The positive and negative labels of the first sample and the If the positive and negative labels of the samples are the same, the value is 1, otherwise it is 0.

[0121] An online fine-tuning prediction module is used to construct a feature extractor with the same structure as the feature extraction module and migrate the parameters of the pre-trained feature extraction module to the feature extractor. A simple transfer prediction network is constructed and trained. The simple transfer prediction network includes a first fully connected layer, a normalization layer, an inactivation layer, a second fully connected layer, and an activation layer. The abdominal CT sample image is input into the feature extractor, and the lymph node features output by the feature extractor are input into the simple transfer prediction network. The simple transfer prediction network generates lymph node metastasis probabilities and calculates the contribution of each lymph node in the CT image based on the lymph node metastasis probabilities.

[0122] First, build a feature extractor with the same structure as the feature extraction module, and migrate and apply the parameters of the pre-trained feature extraction module to the feature extractor; this step can significantly reduce the training time of the feature extractor and improve its performance on new tasks, thus avoiding the gradient explosion problem that may be encountered when training from scratch. Figure 5 The simple transfer prediction network shown in the figure includes the first fully connected layer, the normalization layer, the inactivation layer, the second fully connected layer and the activation layer; and some abdominal CT sample images are used to train the simple transfer prediction network. When training the simple transfer prediction network, its loss function is for:

[0123] ;

[0124] in, represents the number of patient-level lymph node features, represents the positive or negative label of the i-th patient, represents the transfer probability obtained by patient i through the network, Represents the weight factor.

[0125] After updating the parameters of the feature extractor and training the simple metastasis prediction network, (part of) the abdominal CT sample image is input into the feature extractor, which outputs the lymph node features. The lymph node features output by the feature extractor are then input into the simple metastasis prediction network. In the simple metastasis prediction network, the first fully connected layer first learns the hidden features of a single lymph node and normalizes them. The inactivation layer randomly "turns off" some neurons (i.e., sets their output to 0) to prevent overfitting. The second fully connected layer is connected to the activation layer to improve the nonlinear representation ability of the network. The hidden representation of the network is transformed through a series of transformations to finally generate a lymph node metastasis probability of [0, 1]. The temperature parameter t is introduced in the activation layer to adjust the confidence of the model output (i.e., the contribution of each lymph node in the CT image).

[0126] The simple metastasis prediction network adopts the multi-instance principle, that is, if at least one instance in a package (consisting of multiple samples) is positive, then the entire package is marked as positive. Correspondingly, if all instances in a package are negative, then the package is marked as negative. The contribution is mapped according to the function f and then sorted from high to low by contribution. Our strategy is to prioritize lymph nodes with a high positive probability, then focus on lymph nodes with a high negative probability, and finally process lymph nodes that the model cannot clearly classify (i.e., those with a predicted probability close to 0.5). This method is designed to guide the model to focus more on positive lymph nodes that contribute highly to patient-level predictions and effectively address the problem of imbalanced lymph node numbers between patients. The mapping function f is expressed as:

[0127] ;

[0128] in, represents the probability of lymph node metastasis.

[0129] The specific steps of the simple transfer prediction network when performing probability estimation are as follows:

[0130] First, combined with clinical knowledge, given a three-dimensional coordinate point (x, y, z) of a lymph node and the total dimension D of the encoding (here is 128 dimensions), the position encoding PE(x, y, z) can be expressed as:

[0131] ;

[0132] Here, the encoding of each dimension d is generated by applying sine and cosine functions at different frequencies, given by the following formula:

[0133] ;

[0134] ;

[0135] ;

[0136] This encoding method allows the model to capture the position information of each element when processing sequence or point cloud data, thereby effectively utilizing position information without using recursive or convolutional structures. The diameter size partition type is then normalized and encoded, and then concatenated with the lymph node features output by the feature extractor to obtain multimodal features. This is then fed into the second fully connected layer for regression prediction to obtain the current lymph node metastasis probability, and the patient-level metastasis probability estimate is gradually corrected through momentum update. The momentum update is expressed as:

[0137] ;

[0138] in, 、 、 They represent the x-position encoding of the coordinate on the D-1th dimension, the y-position encoding of the coordinate on the D-1th dimension, and the z-position encoding of the coordinate on the D-1th dimension, respectively. 、 、 Respectively represent the x-position encoding of the coordinate on the d-th dimension, the y-position encoding of the coordinate on the d-th dimension, and the z-position encoding of the coordinate on the d-th dimension, represents the momentum update parameter, represents the transition probability of the jth patient, It indicates the new metastasis probability of the i-th lymph node in the patient after fine-tuning with clinical information.

[0139] The lymph node metastasis prediction model training module is used to train the model by inputting sample abdominal CT images based on the contribution of each lymph node in the abdominal CT images. The training sequence is as follows: first, sample images of positive abdominal lymph nodes with high contribution are input, followed by sample images of negative abdominal lymph nodes with high contribution, and finally, the remaining sample images are input. The contribution evaluation criteria for different lymph node types (i.e., positive and negative) differ: a positive lymph node is considered high if its contribution is greater than 0.2 of e, while a negative lymph node is considered high if its contribution is greater than 0.1. The training method used for training the lymph node metastasis prediction model is the same as that used in step 4 and will not be further described here.

[0140] The real-time metastasis prediction module is used to obtain the abdominal CT image to be predicted and input the lymph node metastasis prediction model trained by the lymph node metastasis prediction model training module. The lymph node metastasis prediction model outputs the prediction result that the corresponding lymph node is positive.

[0141] Example 3

[0142] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of a method for predicting abdominal lymph node metastasis based on contrastive learning.

[0143] The computer device may be a desktop computer, a notebook computer, a PDA, a cloud server, etc. The computer device may interact with the user via a keyboard, a mouse, a remote control, a touchpad, or a voice control device.

[0144] The memory includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or D interface display memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory can be an internal storage unit of the computer device, such as the hard disk or memory of the computer device. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk equipped with the computer device, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. Of course, the memory can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is often used to store the operating system and various application software installed on the computer device, such as the program code of the abdominal lymph node metastasis prediction method based on contrastive learning. In addition, the memory can also be used to temporarily store various types of data that have been output or are about to be output.

[0145] The processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor is generally used to control the overall operation of the computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data, such as running the program code of the abdominal lymph node metastasis prediction method based on contrastive learning.

[0146] Example 4

[0147] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor is caused to perform the steps of a method for predicting abdominal lymph node metastasis based on contrastive learning.

[0148] The computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned abdominal lymph node metastasis prediction method based on contrast learning.

[0149] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment method can be implemented by means of software plus the necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including several instructions for enabling a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the abdominal lymph node metastasis prediction method based on contrast learning described in the embodiment of the present application.

Claims

1. A method for predicting abdominal lymph node metastasis based on contrastive learning, characterized in that: The steps include: Step 1: Obtain sample and label data; Obtain an abdominal CT sample image, and annotate the location information of the lymph nodes in the abdominal CT sample image and the positive and negative level of the corresponding patient to obtain label data; Step 2, constructing a lymph node metastasis prediction model; Constructing a lymph node metastasis prediction model, which includes a feature extraction module, an attention module, and a feature fusion module; The feature extraction module includes a first convolution block, a first 2.5D convolution layer, a second convolution block, a second 2.5D convolution layer, a third convolution block, and a linear layer, which are arranged in sequence; The first convolution block, the second convolution block, and the third convolution block each include a convolution layer, a normalization layer, an activation layer, a loss layer, and a pooling layer that are arranged in sequence; The first 2.5D convolution layer and the second 2.5D convolution layer both include a feature fusion convolution layer, which concatenates the labeled frame image with the previous frame image and the next frame image, and then performs convolution processing, and outputs a feature map and the previous frame image and the next frame image; Step 3, pre-training lymph node metastasis prediction model; Input the abdominal CT sample image and label data in step 1 into the lymph node metastasis prediction model constructed in step 2, and pre-train the lymph node metastasis prediction model using a supervised contrast learning method; Step 4: Online fine-tuning prediction; Construct a feature extractor with the same structure as the feature extraction module, and migrate the parameters in the pre-trained feature extraction module to the feature extractor; construct and train a simple transfer prediction network, which includes a first fully connected layer, a normalization layer, a deactivation layer, a second fully connected layer, and an activation layer; The abdominal CT sample image is input into the feature extractor, and the lymph node features output by the feature extractor are input into the simple metastasis prediction network. The simple metastasis prediction network generates the lymph node metastasis probability, and the contribution of each lymph node in the CT image is calculated according to the lymph node metastasis probability; Step 5, training a lymph node metastasis prediction model; According to the contribution of each lymph node in the abdominal CT image, the abdominal CT sample images are input into the lymph node metastasis prediction model for training. The order of sample input during training is: first input the positive abdominal lymph node sample images with high contribution for training, then input the negative abdominal lymph node sample images with high contribution, and finally input the remaining abdominal lymph node sample images; Step 6, real-time transfer prediction; Obtain an abdominal CT image to be predicted, and input the lymph node metastasis prediction model trained in step 5, and the lymph node metastasis prediction model outputs a prediction result that the corresponding lymph node is positive.

2. The method for predicting abdominal lymph node metastasis based on contrastive learning according to claim 1, characterized in that: The abdominal CT sample images are preprocessed as follows: First, the HU value reflecting the tissue density of the abdominal CT sample image was converted to the optimal display effect area [110, 190], and then normalized and the voxel size of the image was adjusted so that all images had a uniform spatial resolution of 0.75*0.75*2mm; Then, the CT image of each patient is cut into images of size (LNs*3)*1*32*32 according to the annotation, and the calibrated lymph node information is stored in the form of an npy file for each patient.

3. The method for predicting abdominal lymph node metastasis based on contrastive learning according to claim 1, characterized in that: In step 2, the attention module includes a first linear layer, an activation layer, a second linear layer, and a softmax layer arranged in sequence, the tanh activation function of the first linear layer and the activation layer learns a nonlinear representation of the input, and the second linear layer and the softmax layer are used to generate the final attention score; The attention score is calculated as: ； in, represents the attention score of the i-th lymph node feature, , They represent the i-th lymph node feature and the j-th lymph node feature respectively. represents the number of lymph node features, , Both represent learnable parameters, Indicates transpose.

4. The method for predicting abdominal lymph node metastasis based on contrastive learning according to claim 1, characterized in that: In step 2, the feature fusion module combines the lymph node features output by the feature extraction module The attention score output by the attention module Multiply and sum to get a patient-level feature of size 128 ; Patient-level characteristics The calculation method is: ； in, represents the attention score of the i-th lymph node feature, represents the feature of the i-th lymph node, Represents the number of lymph node features.

5. The method for predicting abdominal lymph node metastasis based on contrastive learning according to claim 1, characterized in that: In step 3, when supervising the pre-trained lymph node metastasis prediction model, the loss function for: ； in, represents the total number of samples, , Respectively represent Patient-level characteristics of samples, Patient-level characteristics of samples, , Respectively represent The positive and negative labels of the samples, The positive and negative labels of samples, represents feature normalization, Indicates cosine similarity calculation. represents the temperature parameter; represents the indicator function, when The positive and negative labels of the samples are If the positive and negative labels of the samples are the same, the value is 1, otherwise it is 0.

6. The method for predicting abdominal lymph node metastasis based on contrastive learning according to claim 1, characterized in that: In step 4, the contribution of each abdominal CT sample image is obtained by mapping the lymph node metastasis probability according to function f. The mapped function f is expressed as: ； in, represents the probability of lymph node metastasis.

7. The method for predicting abdominal lymph node metastasis based on contrastive learning according to claim 1, characterized in that: In step 5, when training the lymph node metastasis prediction network, the loss function for: ； in, represents the number of lymph node features at the patient level, represents the positive or negative label of the i-th patient, represents the transition probability obtained by patient i through the network, Represents the weight factor.

8. A system for predicting abdominal lymph node metastasis based on contrastive learning, characterized in that: include: The sample and label data acquisition module is used to acquire the abdominal CT sample image, and mark the location information of the lymph nodes in the abdominal CT sample image and the positive and negative level of the corresponding patient to obtain label data; A lymph node metastasis prediction model construction module is used to construct a lymph node metastasis prediction model, which includes a feature extraction module, an attention module, and a feature fusion module; the feature extraction module includes a first convolution block, a first 2.5D convolution layer, a second convolution block, a second 2.5D convolution layer, a third convolution block, and a linear layer arranged in sequence; the first convolution block, the second convolution block, and the third convolution block all include a convolution layer, a normalization layer, an activation layer, a loss layer, and a pooling layer arranged in sequence; the first 2.5D convolution layer and the second 2.5D convolution layer both include a feature fusion convolution layer, which concatenates the labeled frame image with the previous frame image and the next frame image, and then performs convolution processing, and outputs a feature map and the previous frame image and the next frame image; A lymph node metastasis prediction model pre-training module is used to input the abdominal CT sample image and label data in the sample and label data acquisition module into the lymph node metastasis prediction model constructed by the lymph node metastasis prediction model construction module, and pre-train the lymph node metastasis prediction model using a supervised contrast learning method; An online fine-tuning prediction module is used to construct a feature extractor with the same structure as the feature extraction module, and migrate the parameters in the pre-trained feature extraction module to the feature extractor; a simple transfer prediction network is constructed and trained, and the simple transfer prediction network includes a first fully connected layer, a normalization layer, a deactivation layer, a second fully connected layer, and an activation layer; The abdominal CT sample image is input into the feature extractor, and the lymph node features output by the feature extractor are input into the simple metastasis prediction network. The simple metastasis prediction network generates the lymph node metastasis probability, and the contribution of each lymph node in the CT image is calculated according to the lymph node metastasis probability; The lymph node metastasis prediction model training module is used to input the abdominal CT sample images into the lymph node metastasis prediction model for training according to the contribution of each lymph node in the abdominal CT image. The order of sample input during training is: first input the positive abdominal lymph node sample images with high contribution for training, then input the negative abdominal lymph node sample images with high contribution, and finally input the remaining abdominal lymph node sample images; The real-time metastasis prediction module is used to obtain the abdominal CT image to be predicted and input the lymph node metastasis prediction model trained by the lymph node metastasis prediction model training module. The lymph node metastasis prediction model outputs the prediction result that the corresponding lymph node is positive.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 7.

Images