Primary drug resistance prediction system and method

Abstract

The invention belongs to the field of medical data processing, particularly relates to a primary drug resistance prediction system and method, and aims to solve the problems that in the prior art, PTF prediction is low in accuracy and seriously depends on manual annotation. The method comprises the following steps: acquiring a PET image, a CT image and clinical information of a patient, and preprocessing the PET image and the CT image; inputting the preprocessed PET image and the preprocessed CT image into a PET branch and a CT branch respectively; generating a space attention weight map, executing a space weighting operation by using the space attention weight map, and performing splicing in a channel dimension; inputting the PET fusion features and the CT fusion features into a PET classifier and a CT classifier, fusing the output single-mode features, and inputting the fused single-mode features into a preset dual-mode classifier for prediction; and in combination with clinical information of the patient, determining a primary drug resistance prediction score of the patient. According to the method, iconography information and clinical information are fused, so that the accuracy of primary drug resistance prediction is effectively improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of medical data processing, and particularly relates to a primary drug resistance prediction system and method. BACKGROUND

[0002] Diffuse large B-cell lymphoma (DLBCL) is the most common histological subtype of lymphoma, with high heterogeneity, accounting for about 60% of non-Hodgkin's lymphoma, especially in Asia. Clinically, the first-line standard treatment (R-CHOP) of DLBCL mainly uses rituximab combined with cyclophosphamide, doxorubicin, vincristine and prednisone, which significantly improves the prognosis of patients. However, as many as 15%-25% of patients receiving R-CHOP treatment show primary treatment failure (PTF), with a median survival time of less than one year. In recent years, the emergence of intensive treatment regimens such as R-CHOP combined with targeted drugs or chimeric antigen receptor T cell therapy provides more treatment options for high-risk patients under standard R-CHOP regimen.

[0003] The international prognostic index (IPI) based on clinical factors has been widely used to screen high-risk patients receiving intensive treatment. However, IPI has fundamental limitations in predicting PTF, as it is based only on clinical parameters and cannot capture the biological heterogeneity of tumors, resulting in misclassification of patients who respond well to standard R-CHOP therapy as high-risk. 18F-FDG-PET / CT is another important strategy for evaluating DLBCL tumor burden and treatment response. PET-derived biomarkers such as metabolic tumor volume (MTV), total lesion glycolysis (TLG), interlesion maximum distance (Dmax), and radiomic features related to tumor heterogeneity have been shown to have independent prognostic value from IPI. However, most related technologies still focus only on the largest lesions, making it difficult to reflect the most aggressive features, resulting in inaccurate prediction results. In addition, these analyses rely heavily on manual annotation, consuming a lot of manpower and time, and at the same time, annotation is easily affected by differences between different observers, with poor reproducibility. SUMMARY

[0004] In order to solve the above problems in the prior art, i.e., the prior art has low accuracy in PTF prediction and relies heavily on manual annotation, in a first aspect, the application provides a primary drug resistance prediction system, comprising:

[0005] a data acquisition module configured to acquire PET images, CT images and clinical information of a patient, and pre-process the PET images and CT images;

[0006] The feature extraction module is configured to input the preprocessed PET image and CT image into a PET branch and a CT branch of a preset feature extraction network respectively, the PET branch and the CT branch comprising a plurality of weight-shared three-dimensional residual convolution blocks for extracting initial PET features and initial CT features respectively;

[0007] The feature integration module is configured to generate a spatial attention weight map based on feature maps at the highest level in the initial PET features and the initial CT features, perform spatial weighting operation on the feature maps at the middle and low levels using the spatial attention weight map, and splice the weighted multi-level feature maps in the channel dimension to obtain PET fusion features and CT fusion features.

[0008] The image prediction module is configured to input the PET fusion features and the CT fusion features into a preset PET classifier and a preset CT classifier respectively, fuse single-modal features output by the PET classifier and the CT classifier, and input the fused single-modal features into a preset dual-modal classifier for prediction to obtain an image prediction score, wherein the PET classifier and the CT classifier take the difference between their respective prediction results as an endogenous supervision signal, and are trained through a contrast learning strategy to align the single-modal features output by the PET classifier and the CT classifier.

[0009] The comprehensive prediction module is configured to determine a primary drug resistance prediction score of the patient based on the image prediction score and in combination with clinical information of the patient.

[0010] In some preferred embodiments, the PET image and the CT image are preprocessed by the following method:

[0011] The PET image and the CT image are rigidly registered to align the PET image to the corresponding CT image space.

[0012] The registered PET image and CT image are subjected to intensity standardization processing and resampling by cubic spline interpolation to unify the PET image and the CT image to a preset size.

[0013] In some preferred embodiments, the PET branch and the CT branch are weight-shared three-dimensional convolutional neural network structures.

[0014] The initial features output by the feature extraction module are spliced in the channel, comprising:

[0015] The output features of the PET branch and the CT branch at each residual stage are spliced in the channel to form the PET fusion features and the CT fusion features.

[0016] In some preferred embodiments, the PET classifier, the CT classifier and the dual-modality classifier have the same structure, each of which is composed of three fully connected layers, two ReLU activation functions and one Dropout layer, the dimensions of the fully connected layers are 480, 128 and 2 respectively, and the parameter of the Dropout layer is 0.5.

[0017] In some preferred embodiments, the dual-modality classifier takes the element-wise addition result of the aligned single-modality features as input, and the classification optimization process of the PET classifier, the CT classifier and the dual-modality classifier adopts a binary cross-entropy loss function to update the global parameters, which satisfies:

[0018] ;

[0019] wherein, represents the number of samples, represents the true label (0 or 1) of the i-th sample, represents the probability of the i-th sample being predicted as PTF.

[0020] In some preferred embodiments, the training process of the PET classifier and the CT classifier comprises:

[0021] In at least one training batch, the output results of the PET classifier and the CT classifier are determined respectively, and the output results are single-modality features of the PET modality and the CT modality respectively;

[0022] When the output results of the PET classifier and the CT classifier are both correct, the cosine similarity between the two single-modality features is minimized;

[0023] When the output results of the PET classifier and the CT classifier are both incorrect, the cosine similarity between the two single-modality features is maximized;

[0024] When only one of the output results of the PET classifier and the CT classifier is correct, the feature representation of the correct single-modality feature is fixed, and the cosine similarity between the incorrect single-modality feature and the correct single-modality feature is minimized to promote the incorrect single-modality feature to converge to the correct single-modality feature.

[0025] In some preferred embodiments, in the training process of the PET classifier and the CT classifier, when the output results of the PET classifier and the CT classifier are both correct, the loss function thereof satisfies:

[0026] ;

[0027] When the output results of the PET classifier and the CT classifier are both wrong, the loss function satisfies:

[0028] ;

[0029] When only one single modality feature in the output results of the PET classifier and the CT classifier is correct, the loss function satisfies:

[0030] ;

[0031] wherein, represents a set of samples in which both PET and CT modalities are correctly predicted in a batch; represents a set of samples in which both PET and CT modalities are incorrectly predicted in a training batch; represents a set of samples in which the prediction results of PET and CT modalities are inconsistent in a batch, B represents the batch size, represents the PET and CT modality feature representation of sample ; represents the cosine similarity between the single modality feature correctly predicted for sample and the single modality feature incorrectly predicted for sample ; represents a temperature coefficient.

[0032] In some preferred embodiments, based on the image prediction score, in combination with the clinical information of the patient, the primary drug resistance prediction score of the patient is determined, and the method is:

[0033] Based on a pre-constructed logistic regression model, independent predictors related to primary drug resistance are screened from the clinical information to determine the clinical prediction score of the patient;

[0034] The image prediction score and the clinical prediction score are weighted and fused through a preset gating synergistic fusion mechanism to obtain the primary drug resistance prediction score of the patient.

[0035] In some preferred embodiments, the comprehensive prediction module is further configured to:

[0036] According to the primary drug resistance prediction score of the patient, the patient is classified into a risk group;

[0037] If the patient belongs to a high-risk group, a preset first treatment plan is matched for the patient;

[0038] If the patient belongs to a low-risk group, a preset second treatment plan is matched for the patient.

[0039] The application provides a primary drug resistance prediction method.

[0040] PET images and CT images of a patient are acquired, and the PET images and the CT images are preprocessed;

[0041] The preprocessed PET images and the preprocessed CT images are respectively input into a PET branch and a CT branch of a preset feature extraction network, the PET branch and the CT branch comprise a plurality of weight-shared three-dimensional residual convolution blocks and are respectively used for extracting initial PET features and initial CT features;

[0042] Based on feature maps in the highest level of the initial PET features and the initial CT features, a spatial attention weight map is generated, and the spatial attention weight map is used to perform a spatial weighting operation on the feature maps in the middle and low levels, the weighted multi-level feature maps are spliced in a channel dimension, and PET fusion features and CT fusion features are obtained;

[0043] The PET fusion features and the CT fusion features are respectively input into a preset PET classifier and a preset CT classifier, single-modality features output by the PET classifier and the CT classifier are fused, and the fused single-modality features are input into a preset dual-modality classifier for prediction, and an image prediction score is obtained, wherein the PET classifier and the CT classifier take the difference between respective prediction results as an endogenous supervision signal, and are trained through a contrast learning strategy, so that the PET classifier and the CT classifier output aligned single-modality features;

[0044] Based on the image prediction score, the primary drug resistance prediction score of the patient is determined in combination with clinical information of the patient.

[0045] The application has the following beneficial effects:

[0046] The primary drug resistance prediction method provided by the application realizes accurate, comprehensive and interpretable comprehensive prediction of the primary drug resistance risk of a patient before treatment by deeply fusing multi-modality imaging information and clinical information.

[0047] In one aspect, the feature extraction network in the application adopts a unique dual-path feature extraction architecture of a PET branch and a CT branch, can capture active features of functional metabolic images and morphological texture features of anatomical structure images respectively, simultaneously performs channel-level splicing and interaction through a feature fusion module, effectively overcomes the problem of the limitation of single-modality imaging information, makes the imaging features mined by the model more comprehensive and profound, and further ensures the maximization of information utilization through multi-level prediction (a PET classifier, a CT classifier and a dual-modality classifier), so that the image prediction score output finally has higher accuracy and robustness.

[0048] In another aspect, the present application adopts contrastive learning to realize difference-driven modal alignment optimization to reduce multi-modal prediction bias, specifically by using the prediction difference between the PET classifier and the CT classifier as an endogenous supervision signal to dynamically constrain the feature relationship between the two modalities (PET and CT), thereby prompting the model to align cross-modal features in the shared latent space, so that features from different modalities but representing similar pathological information can be effectively fused, thereby improving the prediction accuracy and robustness of the final dual-modality classifier. BRIEF DESCRIPTION OF DRAWINGS

[0049] Other features, objects, and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments, when read in conjunction with the accompanying drawings:

[0050] Figure 1 is a framework schematic diagram of a primary drug resistance prediction system provided by an embodiment of the present application;

[0051] Figure 2 is a flowchart schematic diagram of a primary drug resistance prediction method proposed by an embodiment of the present application;

[0052] Figure 3 is an algorithm schematic diagram for PTF prediction of DLBCL patients proposed by an embodiment of the present application;

[0053] Figure 4 is a flowchart schematic diagram of an image prediction module of a primary drug resistance prediction system proposed by an embodiment of the present application;

[0054] Figure 5 is a patient primary drug resistance lesion identification atlas example proposed by an embodiment of the present application;

[0055] Figure 6 is a survival curve graph about different treatment methods proposed by an embodiment of the present application;

[0056] Figure 7 is a structural schematic diagram of a computer system proposed by an embodiment of the present application. DETAILED DESCRIPTION

[0057] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that only the parts related to the application are shown in the drawings for ease of description.

[0058] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and embodiments.

[0059] Please refer to Figure 1 The first embodiment of the present application provides a primary drug resistance prediction system, comprising:

[0060] A data acquisition module 10 is configured to acquire PET images, CT images, and clinical information of a patient, and to pre-process the PET images and CT images;

[0061] A feature extraction module 20 is configured to input the pre-processed PET images and CT images into trained PET branches and CT branches, respectively, the PET branches and CT branches comprising a plurality of weight-shared three-dimensional residual convolution blocks, for extracting initial PET features and initial CT features, respectively;

[0062] A feature integration module 30 is configured to generate a spatial attention weight map based on feature maps of the highest level in the initial PET features and the initial CT features, and to perform a spatial weighting operation on the feature maps of the middle and low levels using the spatial attention weight map, and to concatenate the weighted multi-level feature maps in the channel dimension to obtain PET fusion features and CT fusion features;

[0063] An image prediction module 40 is configured to input the PET fusion features and the CT fusion features into trained PET classifiers and CT classifiers, respectively, and to input single-modal features output by the PET classifiers and the CT classifiers after fusion into a pre-set dual-modal classifier for prediction to obtain an image prediction score, wherein the PET classifiers and the CT classifiers take the difference between their respective prediction results as an endogenous supervision signal, and are trained through a contrast learning strategy to align the single-modal features output by the PET classifiers and the CT classifiers;

[0064] A comprehensive prediction module 50 is configured to determine a primary drug resistance prediction score of the patient based on the image prediction score and in combination with the clinical information of the patient.

[0065] In this embodiment, first, PET images, CT images, and clinical information of a patient are collected and pre-processed.

[0066] Specifically, original three-dimensional positron emission tomography (PET) images and computed tomography (CT) images of a patient before receiving first-line treatment can be collected from a picture archiving and communication system (PACS). At the same time, corresponding clinical information is extracted from an electronic medical record system, including the patient's gender, age, international prognosis index (IPI) score, and efficacy evaluation record after treatment, which will serve as the true label for subsequent related model training.

[0067] As a preferred implementation, the pre-processing of the PET images and the CT images comprises:

[0068] The PET image and the CT image are rigidly registered to align the PET image to the corresponding CT image space; the registered PET image and CT image are intensity normalized and resampled by cubic spline interpolation to unify the PET image and the CT image to a preset size.

[0069] In this embodiment, the PET and CT images are first rigidly registered to eliminate the body position difference of the patient during the scanning process, so as to ensure that the images of the two modalities are accurately aligned in the spatial coordinate system;

[0070] Secondly, the registered images are intensity normalized, such as using the Z-score normalization method, that is, subtracting the mean value and dividing by the standard deviation, to eliminate the intensity distribution difference caused by different scanning devices or parameter settings;

[0071] Thirdly, in order to ensure the consistency of the data dimension of the input network, the cubic spline interpolation algorithm is used to resample the voxels of all images in this embodiment, and the spatial size of the image volume is uniformly adjusted to a fixed value (such as 256x128x128 voxels).

[0072] In this embodiment, a double-branch three-dimensional feature extraction network is designed and trained in advance as the feature extraction module, which includes a PET branch and a CT branch, respectively used to receive the preprocessed three-dimensional PET image and three-dimensional CT image. The core structure of each branch is based on an improved three-dimensional residual network (such as 3D ResNet) to effectively capture the local details and global context features of the three-dimensional image.

[0073] The PET branch in this embodiment focuses on the features of the metabolically active region, while the CT branch focuses on the anatomical morphology and texture features, which enables the network to learn the most representative features in the two different modalities images respectively.

[0074] In this embodiment, the output channel number of each residual stage in the original ResNet structure can be adjusted from [64, 128, 256, 512] to [32, 64, 128, 256] to adapt to the feature density of medical images and reduce the model parameters.

[0075] Among them, the weight sharing mechanism is adopted between the PET branch and the CT branch. That is, the convolution kernel weights of the corresponding layers in the two branches remain consistent and are updated synchronously during the training process. This mechanism promotes the network to learn the common spatial structure features between the two modalities and realizes the collaborative learning of the feature space. After this module, the network outputs the initial three-dimensional feature spectrum representing the PET modality and the initial three-dimensional feature spectrum representing the CT modality respectively.

[0076] On this basis, the initial PET features and the initial CT features are spliced and fused in this embodiment.

[0077] The feature splicing and fusion involved in this embodiment is specifically splicing the above two feature vectors in the channel dimension to form a more hierarchical multi-modal feature vector.

[0078] The feature integration module receives multi-level feature maps output from each three-dimensional residual convolution block in the double-branch feature extraction network. Taking the PET branch as an example, the highest level feature map (with 256 channels) with the richest semantic information is used to generate a spatial attention weight map through a small convolutional network. The weight map reflects the importance distribution of the tumor region in space.

[0079] Subsequently, the spatial attention weight map is adjusted to the same spatial size as the middle and low level feature maps through upsampling, and is multiplied element by element with them to achieve weighted optimization of the middle and low level features, that is, to enhance the features related to the tumor core region and suppress background noise. Specifically, the middle and low level feature maps refer to the feature maps of the remaining channels other than the highest level feature maps in the feature extraction network. Taking the highest level channel as 256, the channel numbers of the middle and low level features are 128, 64 and 32 respectively.

[0080] Finally, the multi-level feature maps optimized by weighting are concatenated (Concatenation) with the original highest level feature maps in the channel dimension to obtain a PET fusion feature containing rich multi-scale information. In this embodiment, the channel number is 32+64+128+256=480. At the same time, the same operation is performed on the feature maps of the CT branch to obtain a CT fusion feature with a channel number of 480.

[0081] On this basis, the PET and CT multi-level fusion features obtained in the above process are respectively input into independent single-modal classifiers (e.g., composed of global average pooling layers and fully connected layers).

[0082] In this embodiment, to realize effective alignment between modalities, a difference-driven contrast learning mechanism is constructed based on the outputs of the PET classifier and the CT classifier. This mechanism takes the difference between the prediction results of the two single-modal classifiers as an endogenous supervision signal, and dynamically constrains the feature relationship between modalities according to the following three strategies:

[0083] In at least one training batch, the output results of the PET classifier and the CT classifier are determined, and the output results are single-modal features of the PET modality and the CT modality;

[0084] When the output results of the PET classifier and the CT classifier are both correct, the global parameters are updated by back propagation using a preset first loss function to minimize the cosine similarity between the single-modality features.

[0085] When the output results of the PET classifier and the CT classifier are both incorrect, the global parameters are updated by back propagation using a preset second loss function to maximize the cosine similarity between the single-modality features.

[0086] When only one of the output results of the PET classifier and the CT classifier is correct, the global parameters are updated by back propagation using a preset third loss function to fix the feature representation of the correct modality feature and minimize the cosine similarity between the incorrect single-modality feature and the correct single-modality feature, so as to promote the incorrect single-modality feature to converge to the correct single-modality feature.

[0087] The global parameters refer to all configurable parameters and weight values in the feature extraction module, the feature integration module, and the image prediction module in the above embodiments.

[0088] Specifically, when the prediction results of the PET classifier and the CT classifier are both correct, specific loss function optimization is performed to promote the feature representations of the two modalities to move away from each other in the latent space, i.e., to reduce the cosine similarity between them, so as to encourage the two modalities to learn complementary and different discriminative information.

[0089] When the prediction results of the two classifiers are both incorrect, the cosine similarity between their feature representations is increased to promote the model to strengthen the common information leading to common errors in the shared latent space, so as to facilitate subsequent correction.

[0090] Meanwhile, when only the prediction result of one modality is correct, the feature representation of the correct modality is fixed in the training, and the feature representation of the incorrect modality is made to move closer to the feature representation of the correct modality through optimization, thereby achieving knowledge transfer and effective correction of the incorrect modality.

[0091] More specifically, in the training process of the PET classifier and the CT classifier, when the output results of the PET classifier and the CT classifier are both correct, the loss function satisfies:

[0092] .

[0093] When the output results of the PET classifier and the CT classifier are both incorrect, the loss function satisfies:

[0094] .

[0095] When only one single modality feature is correct in the output results of the PET classifier and the CT classifier, the loss function thereof satisfies:

[0096]

[0097] wherein, represents a set of samples in which both PET and CT modalities are correctly predicted in a batch; represents a set of samples in which both PET and CT modalities are incorrectly predicted in a batch; represents a set of samples in which the prediction results of PET and CT modalities are inconsistent in a batch, and B represents the batch size, represents the cosine similarity between the PET and CT modality feature representations of sample represents the cosine similarity between the correctly predicted single modality feature of sample and the incorrectly predicted single modality feature of sample represents a temperature coefficient.

[0098] After the above training process, the trained PET classifier and CT classifier are obtained, which are respectively used to output the aligned and optimized PET features and CT features of a single modality. The aligned and optimized PET features and CT features are input into the dual-modality classifier after being element-wise weighted and fused, and finally the prediction score based on PET / CT images is generated.

[0099] As a feasible implementation, the classification optimization process of the entire network adopts a binary cross-entropy loss function for global parameter updating, satisfying:

[0100]

[0101] wherein, represents the number of samples, represents the true label (0 or 1) of the i-th sample, represents the probability of the i-th sample being predicted as PTF.

[0102] More specifically, please refer to Figure 4 , Figure 4 which shows the specific workflow of the primary drug resistance prediction system in this embodiment, wherein the starting point of the workflow is the original three-dimensional medical image of the patient of the sample, which is divided into two parallel channels, the upper channel is the CT image, and the lower channel is the PET image. The CT and PET images are respectively sent into the respective feature extractors (i.e., three-dimensional residual networks based on weight sharing mechanism).

[0103] As Figure 4 ​​​​The image data of the patient is converted into corresponding feature vectors, i.e., CT features and PET features (i.e., the initial CT features and the initial PET features involved in the above embodiment), after being subjected to operations such as multi-layer convolution and pooling, as indicated by the solid line in FIG. 1.

[0104] The CT features are input into the CT classifier to generate a preliminary CT prediction score, and correspondingly, the PET features are also input into the independent PET classifier to generate a PET prediction score.

[0105] Figure 4 The multimodal contrast loss indicated by the dashed line in FIG. 1 is used to determine the accuracy of the respective predictions by comparing the prediction scores of the single modalities with the true labels.

[0106] According to the determination result (for example: both correct, both incorrect, CT incorrect and PET correct, or PET incorrect and CT correct), a loss value (i.e., the loss function of each item) is calculated, which is back-propagated through the path indicated by the dashed line and finally acts on the CT features and the PET features. Specifically:

[0107] If both are predicted correctly, the CT features and the PET features will be away from each other, maintaining their respective uniqueness; if both are predicted incorrectly, the CT features and the PET features will be close to each other, learning the common points of each other to determine more essential and correct discrimination information; if one is correct and the other is incorrect, the incorrect features will converge to the correct features.

[0108] After the above optimization process, the CT features and the PET features optimized through contrast learning are subjected to an element-by-element addition fusion process and then input into the final dual-modality classifier, which outputs a final comprehensive prediction score. This score is used as the final quantitative evaluation of the model on the risk of primary drug resistance of the patient and is further used to calculate the binary cross-entropy loss to drive the parameter update of the entire model.

[0109] Further, in this embodiment, while determining the image prediction score of the patient, the most relevant independent predictors of primary drug resistance are screened from the numerous clinical information of the patient.

[0110] Specifically, a plurality of clinical information including gender, age, IPI score, and gene typing can be collected, and univariate and multivariate logistic regression analysis methods can be used to evaluate the correlation between each clinical variable and the primary treatment outcome.

[0111] For example, based on the clinical information such as gender, age, IPI, and gene typing, the correlation between each clinical variable and the primary treatment outcome is evaluated by univariate and multivariate logistic regression analysis. For example, a P value less than 0.05 can be used as a significance criterion to screen out independent predictors highly correlated with treatment failure.

[0112] The embodiment uses a large number of cases for testing, and finally sets a statistical significance standard of P value less than 0.05, and the analysis result shows that the IPI score is an independent prediction factor highly correlated with the failure of primary treatment, so the embodiment finally selects the IPI score as a representative clinical index for subsequent fusion analysis.

[0113] As a feasible implementation manner, the image prediction score and the clinical prediction score are fused through a pre-constructed logistic regression model, and the embodiment does not limit the specific form of the logistic regression model, and any feasible implementation manner can be selected by a person skilled in the art.

[0114] In addition, the image prediction score and the clinical prediction score can also be fused through a method such as weighted average, and the specific weight proportion of the image prediction score and the clinical prediction score can be set according to different treatment plans received by the patient.

[0115] On the basis of the above-mentioned implementation manner, the embodiment combines the image prediction score and the IPI score of the patient to generate a final comprehensive prediction score.

[0116] As a preferred implementation manner, the embodiment adopts a gated collaborative fusion mechanism to weight fuse the image prediction score and the IPI score. The mechanism includes a gating unit which can adaptively learn and assign the weights of the image score and the IPI score according to the input data, so as to realize the dynamic collaboration and complementation of the two kinds of information. Finally, the fusion module outputs a comprehensive primary drug resistance prediction score (comprehensive PTF score), which is used to comprehensively evaluate the treatment failure risk of the patient.

[0117] Further, in the embodiment, the PET branch and the CT branch are three-dimensional convolutional neural network structures sharing weights.

[0118] The above-mentioned initial features output by the feature extraction module are spliced according to channels, including channel splicing of the output features of the PET branch and the CT branch at each residual stage to form the PET fusion features and the CT fusion features.

[0119] More specifically, the PET classifier, the CT classifier and the dual-mode classifier constructed in the embodiment have the same structure, all of which are composed of three fully connected layers, two ReLU activation functions and one Dropout layer. The dimensions of the fully connected layers are 480, 128 and 2 respectively, and the parameter of the Dropout layer is 0.5. The input of the PET classifier is , the input of the CT classifier is , and the input of the dual-mode classifier is and Element-wise addition obtains PET / CT multimodal fusion features , the dimension is 480.

[0120] Referring to Figures 2-3 , based on the primary drug resistance prediction system shown above Figure 1 The second embodiment of the application provides a primary drug resistance prediction method, which comprises the following steps:

[0121] Step S10, acquiring PET images, CT images and clinical information of a patient, and pre-processing the PET images and CT images;

[0122] Step S20, inputting the pre-processed PET images and CT images into a PET branch and a CT branch of a preset feature extraction network, respectively, wherein the PET branch and the CT branch comprise a plurality of weight-shared three-dimensional residual convolution blocks, and are used for extracting initial PET features and initial CT features, respectively;

[0123] Step S30, generating a spatial attention weight map based on feature maps at the highest level in the initial PET features and the initial CT features, performing a spatial weighting operation on the feature maps at the middle and low levels by using the spatial attention weight map, splicing the weighted multi-level feature maps in the channel dimension, and obtaining PET fusion features and CT fusion features;

[0124] Step S40, inputting the PET fusion features and the CT fusion features into a preset PET classifier and a preset CT classifier, respectively, fusing single-modality features output by the PET classifier and the CT classifier, and inputting the fused single-modality features into a preset dual-modality classifier for prediction to obtain an image prediction score, wherein the PET classifier and the CT classifier take the difference between the respective output results as an endogenous supervision signal, and adopt a contrast learning strategy for training to dynamically constrain the feature relationship between the PET modality and the CT modality;

[0125] Step S50, determining a primary drug resistance prediction score of the patient based on the image prediction score and in combination with the clinical information of the patient.

[0126] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and related description of the above-described method can refer to the corresponding process in the foregoing system embodiments, which will not be described herein.

[0127] In addition, the third embodiment of the application provides a construction method of a primary drug resistance lesion identification atlas for high-risk DLBCL patients, which comprises the following steps:

[0128] Collect the mid-term PET / CT images of DLBCL patients who have PTF after receiving first-line treatment, compare them with the initial PET / CT images based on clinical experience, and find the lesions that still exist after the patients receive 3 to 4 cycles of initial treatment (i.e. primary drug-resistant lesions);

[0129] All positions and number of lesions are marked and counted by experienced doctors;

[0130] Based on the method described in the first embodiment, a visual heat map is generated and matched with the doctor's marked positions.

[0131] In this embodiment, the related model trained in the above embodiments can be used to process the pre-treatment PET / CT images of high-risk DLBCL patients to be analyzed. Specifically, a visual heat map can be further generated based on the foregoing embodiments by using a visual algorithm such as gradient-weighted class activation mapping (Grad-CAM). The heat map can highlight the areas that the deep learning model pays most attention to in the three-dimensional image space when making the primary drug resistance prediction.

[0132] Then, the generated heat map is post-processed, for example, a threshold (such as 0.5) is set, and the areas in the heat map with activation values higher than the threshold are identified as potential drug-resistant lesion areas. These identified areas constitute the identification atlas of primary drug-resistant lesions. The clinician can superimpose this atlas with the original image of the patient, if there is a drug-resistant lesion, it is considered that the primary drug-resistant lesion is identified, if there is no drug-resistant lesion, it is considered that the identification is wrong. In addition, if the doctor has marked a primary drug-resistant lesion in a certain part, but the heat map has not identified it, it is considered that the identification is wrong.

[0133] Based on this embodiment, the malignant lesions with higher malignancy can be quickly located among many lesions, providing important visual assistance for formulating precise puncture biopsy plans or local radiotherapy plans.

[0134] Please refer to Figure 5 , Figure 5 As a patient primary drug-resistant lesion identification atlas example, the primary drug-resistant lesion identification atlas obtained in this embodiment for high-risk DLBCL patients can be used to screen lesions with higher malignancy from multiple lesions to assist in puncture lesion selection and precision treatment.

[0135] The fourth embodiment of the present application provides a new treatment decision-making method for targeted drugs, which comprises:

[0136] Collect the images and clinical information of patients receiving R-CHOP-X treatment (genetic subtype guided targeted drug plus R-CHOP), match the same number of patients receiving R-CHOP treatment through genotyping and IPI scoring, and collect the corresponding images and clinical information.

[0137] Based on the comprehensive prediction score obtained by the primary drug resistance prediction method provided in the first embodiment, the patients are divided into risk groups, and if the patient belongs to the high-risk group, the patient is recommended to receive R-CHOP-X treatment, and if the patient belongs to the low-risk group, the patient is recommended to receive R-CHOP treatment, which can ensure the treatment effect and prognosis of the patient while avoiding resource waste.

[0138] The embodiment provides a new targeted drug treatment decision assistance scheme based on the method described in the first embodiment. Specifically, first, the comprehensive primary drug resistance prediction score of each newly diagnosed DLBCL patient is calculated using the method described in the first embodiment.

[0139] Second, according to the risk threshold determined in advance or through clinical cohort study, the patients are risk stratified. For example, patients with a comprehensive score higher than a certain threshold are divided into a high-risk group, and patients with a score lower than the threshold are divided into a low-risk group.

[0140] Finally, based on the risk stratification result, individualized treatment recommendations are provided to clinicians. For patients identified as high-risk, it is recommended that they use targeted drugs for their specific genetic subtypes (i.e., R-CHOP-X regimen) in combination with the standard R-CHOP regimen or consider other intensive treatment. For low-risk patients, the standard R-CHOP treatment regimen is recommended. This decision-making process can ensure the treatment effect and improve the prognosis of patients while effectively avoiding over-treatment of low-risk patients, thereby saving medical resources.

[0141] Please refer to Figure 6 , Figure 6 is a survival curve diagram of different treatment methods. The embodiment constructs a new targeted drug treatment decision based on the risk stratification of the above score, thereby recommending different treatment regimens for patients of different risks to avoid resource waste and over-treatment.

[0142] Reference will now be made to Figure 7 , which shows a structural schematic diagram of a computer system of a server suitable for implementing the method, system, and device embodiments of the present application. Figure 7 The server shown is merely an example and should not impose any limitations on the functions and use range of the embodiments of the present application.

[0143] As Figure 7As shown, the computer system includes a central processing unit (CPU) 301 which can perform various appropriate actions and processes in accordance with a program stored in a read only memory (ROM) 302 or a program loaded from a storage section 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the system are also stored. The central processing unit 301, the read only memory 302, and the random access memory are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0144] Connected to the input / output interface 305 are an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the input / output interface 305 as necessary. A removable media 311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 310 as necessary, so that a computer program read therefrom is installed in the storage section 308 as necessary.

[0145] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present application. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 309, and / or installed from the removable media 311. When the computer program is executed by the central processing unit 301, the above-described functions defined in the methods of the present application are performed. It should be noted that the computer readable medium described above in the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, be - but is not limited to - an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above.

[0146] More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0147] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0148] The computer program product of the present application can be a computer program embodied on a non-transitory computer readable medium. Such non-transitory computer readable medium can include, but is not limited to, floppy diskettes, CD-ROMs, DVDs, flash memories, memory sticks, and hard drives.

[0149] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.

[0150] The term "comprising" or any other similar term is intended to encompass the inclusion of one or more stated elements, steps, integers, or components, but not to the exclusion of any other elements, steps, integers, or components. It is further noted that the claims can be drafted to exclude any elements, steps, integers, or components.

[0151] The technical solutions of the present application have been described above in conjunction with the preferred embodiments shown in the drawings.

[0152] The above description is merely illustrative of the application, and not in limitation of the application. Changes and modifications can be made by those skilled in the art. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.

Claims

1. A de novo resistance prediction system, characterized in that, The method comprises the following steps: a data acquisition module configured to acquire PET images, CT images, and clinical information of a patient, and to preprocess the PET images and the CT images; a feature extraction module configured to input the preprocessed PET images and CT images into trained PET branches and CT branches respectively, wherein the PET branches and the CT branches comprise a plurality of weight-shared three-dimensional residual convolution blocks, and are used to extract initial PET features and initial CT features respectively; a feature integration module configured to generate a spatial attention weight map based on feature maps at the highest level among the initial PET features and the initial CT features, to perform a spatial weighting operation on the feature maps at the middle and low levels using the spatial attention weight map, to splice the weighted multi-level feature maps in the channel dimension, and to obtain PET fusion features and CT fusion features; an image prediction module configured to input the PET fusion features and the CT fusion features into trained PET classifiers and CT classifiers respectively, to fuse single-modal features output by the PET classifiers and the CT classifiers, and to input the fused single-modal features into a preset dual-modal classifier for prediction to obtain an image prediction score, wherein the PET classifiers and the CT classifiers use the difference between their respective prediction results as an endogenous supervision signal, and are trained through a contrast learning strategy to align the single-modal features output by the PET classifiers and the CT classifiers; a comprehensive prediction module configured to determine a primary drug resistance prediction score of the patient based on the image prediction score and the clinical information of the patient.

2. The de novo resistance prediction system of claim 1, wherein, The method for preprocessing the PET images and the CT images is as follows: perform rigid registration on the PET images and the CT images to align the PET images to the corresponding CT image space; perform intensity standardization processing on the registered PET images and CT images, and resample the PET images and the CT images through cubic spline interpolation to unify the PET images and the CT images to a preset size.

3. The de novo resistance prediction system of claim 1, wherein, The PET branches and the CT branches are weight-shared three-dimensional convolutional neural network structures; The method for splicing the initial features output by the feature extraction module according to channels is as follows: channel splice the output features of the PET branches and the CT branches at each residual stage to form the PET fusion features and the CT fusion features.

4. The de novo resistance prediction system of claim 3, wherein, The image prediction module is a triple classifier architecture, the PET classifiers, the CT classifiers, and the dual-modal classifier have the same structure, and each of the PET classifiers, the CT classifiers, and the dual-modal classifier is composed of three fully connected layers, two ReLU activation functions, and one Dropout layer, the dimensions of the fully connected layers are 480, 128, and 2 respectively, and the parameter of the Dropout layer is 0.

5.

5. The de novo resistance prediction system of claim 1, wherein, The dual-modal classifier takes the element-wise addition result of the single-modal features as input, the output of the dual-modal classifier is used as a parameter of a binary cross-entropy loss function, the binary cross-entropy loss function is used for global parameter updating, and the formula of the binary cross-entropy loss function is as follows: ； In the formula, denotes the number of samples, denotes the true label of the i-th sample, denotes the probability that the i-th sample is predicted as primary drug-resistant.

6. The de novo resistance prediction system of claim 1, wherein, The training process of the feature extraction module, the feature integration module, and the image prediction module includes: In at least one training batch, the output results of the PET classifier and the CT classifier are determined, and the output results are single-modality features of the PET modality and the CT modality; When the output results of the PET classifier and the CT classifier are both correct, a preset first loss function is used to perform back propagation update on the global parameters to minimize the cosine similarity between the single-modality features; When the output results of the PET classifier and the CT classifier are both incorrect, a preset second loss function is used to perform back propagation update on the global parameters to maximize the cosine similarity between the single-modality features; When only one single-modality feature in the output results of the PET classifier and the CT classifier is correct, a preset third loss function is used to perform back propagation update on the global parameters to fix the feature representation of the correct modality feature and minimize the cosine similarity between the incorrect single-modality feature and the correct single-modality feature, so as to promote the incorrect single-modality feature to converge to the correct single-modality feature.

7. The de novo resistance prediction system of claim 6, wherein, The formula of the first loss function is: ； The formula of the second loss function is: ； The formula of the third loss function is: ； In the formula, , represents the set of samples in which both PET and CT modalities are correctly predicted in a batch, , represents the set of samples in which both PET and CT modalities are incorrectly predicted in a training batch, , represents the set of samples in which the prediction results of PET and CT modalities are inconsistent in a batch, B represents the size of a training batch, , represents the cosine similarity between the PET and CT modal features of sample , , represents the cosine similarity between the single-modality feature correctly predicted for sample and the single-modality feature incorrectly predicted for sample , , represents the temperature coefficient.

8. The de novo resistance prediction system of claim 1, wherein, Based on the image prediction score, the clinical information of the patient is combined to determine the primary drug resistance prediction score of the patient, and the method is: Based on the pre-constructed logistic regression model, the independent prediction factors related to primary drug resistance in the clinical information are screened to determine the clinical prediction score of the patient; The image prediction score and the clinical prediction score are weighted and fused through a preset gating synergistic fusion mechanism to obtain the primary drug resistance prediction score of the patient.

9. The de novo resistance prediction system of claim 1, wherein, The comprehensive prediction module is further configured to: According to the primary drug resistance prediction score of the patient, the patient is divided into a risk group; If the patient belongs to a high-risk group, a preset first treatment plan is matched for the patient; If the patient belongs to a low-risk group, a preset second treatment plan is matched for the patient.

10. A method of de novo resistance prediction, characterized in that, The method includes: Obtaining the PET image, the CT image, and the clinical information of the patient, and preprocessing the PET image and the CT image; The preprocessed PET image and CT image are respectively input into the PET branch and the CT branch of the preset feature extraction network, and the PET branch and the CT branch include a plurality of weight-shared three-dimensional residual convolution blocks, which are respectively used to extract initial PET features and initial CT features; Based on the feature maps of the highest level in the initial PET features and the initial CT features, a spatial attention weight map is generated, and the spatial attention weight map is used to perform spatial weighting operation on the feature maps of the middle and low levels, and the weighted multi-level feature maps are spliced in the channel dimension to obtain PET fusion features and CT fusion features; The PET fusion feature and the CT fusion feature are respectively input into a preset PET classifier and a CT classifier, single-modality features output by the PET classifier and the CT classifier are fused, and are input into a preset double-modality classifier for prediction to obtain an image prediction score, wherein the PET classifier and the CT classifier take the difference between respective prediction results as an endogenous supervision signal, and are trained through a contrast learning strategy to make the single-modality features output by the PET classifier and the CT classifier aligned; Based on the image prediction score, the primary drug resistance prediction score of the patient is determined in combination with the clinical information of the patient.

Images