Prediction method and device for immunotherapy curative effect and medical imaging equipment

Abstract

The invention relates to the technical field of medical imaging equipment, and discloses a prediction method for the curative effect of immunotherapy, and the method comprises the steps: obtaining a radiomics feature vector and a deep learning feature vector of an original tumor sequence; wherein the feature space of the radiomics feature vector and the feature space of the deep learning feature vector are aligned; constructing a positive sample pair and a negative sample pair by using the radiomics feature vector and the deep learning feature vector; and obtaining an immunotherapy curative effect prediction model by maximizing the similarity of the positive sample pair and minimizing the similarity of the negative sample pair based on the initial prediction model so as to realize immunotherapy curative effect prediction by using the immunotherapy curative effect prediction model. According to the scheme, priori knowledge of radiomics is migrated to the deep learning model, and the accuracy of the immunotherapy curative effect prediction model in tumor therapy curative effect prediction can be improved. The invention further discloses a prediction device for the immunotherapy curative effect and medical imaging equipment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical imaging technology, for example to a method and device for predicting the efficacy of immunotherapy, and a medical imaging device. BACKGROUND

[0002] Immunotherapy is a new tumor treatment method in recent years, which is different from traditional surgery, chemotherapy, radiotherapy and targeted therapy. Immunotherapy activates T cells by releasing immune checkpoint inhibitors to enhance the body's immune system attack on tumor cells. Immunotherapy has been proven to have significant efficacy in clinical trials, and has shown better treatment effect than conventional chemotherapy in the treatment of various malignant tumors including HCC, especially for malignant and chemically resistant cancers. Although immunotherapy shows excellent efficacy in clinical practice, especially for chemotherapy-resistant malignant tumors, the efficacy of immunotherapy varies from person to person, and the current efficacy prediction method still has limitations. According to research, only 20% to 50% of patients have a significant response to immunotherapy. In addition, the cost of immunotherapy is relatively high, and the cost of using an immunotherapy drug is tens of thousands of yuan. Therefore, if HCC patients sensitive to immunotherapy can be quickly and accurately screened before surgery, the treatment effect can be greatly improved, the survival rate of patients can be improved, and unnecessary medical expenses can be saved.

[0003] Previous studies have shown that micro-gene, protein and molecular changes are closely related to macro-tumor imaging features, and changes in macro-imaging features may reflect different expressions of gene or protein patterns at the micro level. In related technologies, pathological examination images or postoperative long-term follow-up medical images are used as data labels to obtain tumor micro information, so as to realize the efficacy prediction of tumor immunotherapy through medical imaging.

[0004] In the process of implementing the embodiments of the present disclosure, it is found that at least the following problems exist in the related art:

[0005] The data labels of pathological examination images or postoperative long-term follow-up medical images are extremely few, which greatly affects the accuracy of the immunotherapy efficacy prediction model in predicting the efficacy of tumor treatment.

[0006] It should be noted that the information disclosed in the above BACKGROUND section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0007] In order to have a basic understanding of some aspects of the disclosed embodiments, the following is a simple summary. The summary is not a general review, nor is it intended to determine key / important components or delineate the scope of protection of these embodiments, but as a prelude to the detailed description that follows.

[0008] The embodiments of the present disclosure provide a method and device for predicting the efficacy of immunotherapy, and a medical imaging device, to improve the accuracy of an immunotherapy efficacy prediction model in predicting the efficacy of tumor treatment.

[0009] In some embodiments, the method for predicting the efficacy of immunotherapy comprises: obtaining an imageomics feature vector and a deep learning feature vector of a primary tumor sequence; wherein the feature spaces of the imageomics feature vector and the deep learning feature vector are aligned; constructing a positive sample pair by using a primary sequence imageomics feature in the imageomics feature vector and a same sequence enhanced deep learning feature in the deep learning feature vector; constructing a negative sample pair by using the primary sequence imageomics feature and a different label sequence deep learning feature in the deep learning feature vector; obtaining an immunotherapy efficacy prediction model based on an initial prediction model by maximizing the similarity of the positive sample pair and minimizing the similarity of the negative sample pair, to realize the prediction of the efficacy of immunotherapy by using the immunotherapy efficacy prediction model.

[0010] In some embodiments, the method for predicting the efficacy of immunotherapy comprises: obtaining a tumor image sequence to be detected; inputting the tumor image sequence to be detected into the aforementioned immunotherapy efficacy prediction model to obtain an immunotherapy efficacy prediction model output; and determining an immunotherapy efficacy prediction result by using the immunotherapy efficacy prediction model output.

[0011] In some embodiments, the device for predicting the efficacy of immunotherapy comprises a processor and a memory storing program instructions, wherein the processor is configured to execute the aforementioned method for predicting the efficacy of immunotherapy when executing the program instructions.

[0012] In some embodiments, the medical imaging device comprises: a medical imaging device body; and the aforementioned device for predicting the efficacy of immunotherapy, which is installed on the medical imaging device body.

[0013] The method and device for predicting the efficacy of immunotherapy, and the medical imaging device provided by the embodiments of the present disclosure can achieve the following technical effects:

[0014] In the technical solutions of the present application, the imageomics feature vector and the deep learning feature vector of the original tumor sequence are obtained, then the imageomics features and the enhanced deep learning features of the same sequence are used to form a positive sample pair, the imageomics features and the deep learning features of different labeled sequences are used to form a negative sample pair, and based on an initial prediction model, the immunotherapy efficacy prediction model is obtained by maximizing the similarity of the positive sample pair and minimizing the similarity of the negative sample pair, so as to realize the immunotherapy efficacy prediction by using the immunotherapy efficacy prediction model. The prior knowledge of imageomics is transferred to the deep learning model, which can effectively improve the performance of the small sample deep learning model and the accuracy of the tumor immunotherapy efficacy prediction, and further improve the accuracy of the data-driven method represented by deep learning in the tumor treatment efficacy prediction.

[0015] The foregoing general description and the following description are merely exemplary and explanatory, and are not intended to limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0016] One or more embodiments are exemplarily illustrated by corresponding drawings, which do not constitute limitations on the embodiments, elements with the same reference numerals in the drawings are shown as similar elements, the drawings do not constitute proportional limits, and wherein:

[0017] Figure 1 is a flowchart of a prediction method for immunotherapy efficacy provided by an embodiment of the present disclosure;

[0018] Figure 2 is a flowchart of another prediction method for immunotherapy efficacy provided by an embodiment of the present disclosure;

[0019] Figure 3 is a flowchart of another prediction method for immunotherapy efficacy provided by an embodiment of the present disclosure;

[0020] Figure 4 is a flowchart of another prediction method for immunotherapy efficacy provided by an embodiment of the present disclosure;

[0021] Figure 5 is a framework diagram of an immunotherapy efficacy prediction model provided by an embodiment of the present disclosure;

[0022] Figure 6 is a schematic diagram of constructing a positive sample pair and a negative sample pair provided by an embodiment of the present disclosure;

[0023] Figure 7 is a structural diagram of a prediction device for immunotherapy efficacy provided by an embodiment of the present disclosure;

[0024] Figure 8is a schematic diagram of a medical image device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to enable a more detailed understanding of the features and technical content of the embodiments of the present disclosure, the implementation of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, which are only used for reference and do not limit the embodiments of the present disclosure. In the following technical description, in order to facilitate explanation, a plurality of details are provided to provide a full understanding of the disclosed embodiments. However, one or more embodiments can still be implemented without these details. In other cases, well-known structures and devices can be simplified to facilitate the drawings.

[0026] The terms "first", "second", and the like in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion.

[0027] Unless otherwise specified, the term "a plurality of" means two or more. In the embodiments of the present disclosure, the character " / " represents that the objects before and after are in an "or" relationship. For example, A / B means: A or B. The term "and / or" is a description of the association between objects, which means that there can be three relationships. For example, A and / or B means: A or B, or, A and B, the three relationships. The term "corresponding to" can refer to an association or binding relationship, A corresponding to B means that there is an association or binding relationship between A and B.

[0028] In the process of immunotherapy of tumors, some microscopic biomarkers are related to the effect of immunotherapy. At present, in clinical practice, the recognized effective efficacy prediction biomarkers include the following: PD-L1, CD-3, and CD-8, etc. Studies have shown that the higher the expression content of these markers, the higher the sensitivity of patients to immunotherapy, and the two have a positive correlation. For example, usually when the expression level of PD-L1 is more than 50%, the cancer patient presents a high sensitivity state to immunotherapy.

[0029] With the development of artificial intelligence technology, advanced image analysis techniques have been applied to predict tumor biomarkers, and imaging features have gradually become an important reference for analyzing the prognosis of tumors (e.g., liver cancer). In the process of predicting the efficacy of tumor treatment, the results of pathological examination or long-term follow-up after surgery are generally used as the gold standard for data labeling, and the cost of data acquisition is extremely high, so the prediction of tumor efficacy is basically based on a small sample. Deep metric meta-learning methods perform well in small sample classification tasks, and they rely on a large amount of similar historical task data. However, in the field of medical data, it is often difficult to obtain large amounts of similar historical task data. When similar historical task data is insufficient, model training is prone to overfitting, resulting in poor feature learning. Moreover, the introduction of multi-task learning increases the computational load and reduces learning efficiency. Therefore, deep metric meta-learning methods still face great limitations in practical applications. In the task of predicting the efficacy of tumor immunotherapy, there is very limited similar historical task data, so it is not suitable to directly apply existing deep metric meta-learning models.

[0030] As a technology for high-throughput extraction of image features from standard medical images, radiomics integrates rich tumor analysis medical knowledge and has advantages such as fast feature extraction and low training data requirements, making it suitable for small sample model construction. In the technical solutions of the present disclosure, radiomics features are used as a kind of prior auxiliary information to guide a small sample deep convolutional neural network to learn an embedding vector mapping function (such as an Embedding mapping function), and a contrast learning strategy is used in the guidance process to make the embedding vector mapping of the sample feature vector and the enhanced radiomics feature vector of the sample in the embedding space close to each other, while making different class samples far away from each other. Through the contrast learning process, the output of the embedding vector mapping function is closer to the distribution of the prior features, and the knowledge transfer is completed when the loss function approaches zero. The learned embedding vector mapping function is transferred to the downstream tumor treatment efficacy prediction task, fine-tuned with a small number of labeled samples, and finally the immunotherapy efficacy prediction model is trained. The immunotherapy efficacy prediction model has prior knowledge of related tasks, and through a supervised contrast learning idea, the prior feature distribution is transferred to the immunotherapy efficacy prediction model, without relying on a large amount of similar tasks to obtain better prediction performance.

[0031] In combination Figure 1 As shown in the drawings, the prediction method for immunotherapy efficacy provided by the embodiments of the present disclosure includes the following steps:

[0032] S101, obtaining a radiomics feature vector and a deep learning feature vector of a primary tumor sequence; wherein the feature spaces of the radiomics feature vector and the deep learning feature vector are aligned.

[0033] The original tumor sequence includes an MRI (Magnetic Resonance Imaging) tumor sequence, a CT (Computed Tomography) tumor sequence, a DSA (Digital Subtraction Angiography) tumor sequence, or an Ultrasound tumor sequence.

[0034] Optionally, the feature space alignment of the radiomics feature vector and the deep learning feature vector is controlled in the following manner: the extracted radiomics features of the original tumor sequence are encoded into a radiomics feature vector of a set dimension through a fully connected layer; and the extracted deep learning features of the original tumor sequence are encoded into a deep learning feature vector of a set dimension through a fully connected layer.

[0035] Here, the set dimension can be 512 dimensions or 1024 dimensions, which can be set by a person skilled in the art according to actual conditions.

[0036] The following takes the MRI sequence as an example to describe the extraction manner of the radiomics feature vector and the deep learning feature vector of the original tumor sequence.

[0037] The radiomics features of the MRI sequence are extracted by a two-dimensional radiomics encoder (for example, PyRadiomics), and the extracted features are divided into texture features, intensity features, morphological features, and filter features, wherein: the texture features include 16 gray level zone matrix (GLSZM) features, 24 gray level co-occurrence matrix (GLCM) features, 5 neighborhood gray tone difference matrix (NGTDM) features, 16 gray level run-length matrix (GLRLM) features, and 14 gray level dependence matrix (GLDM) features; the intensity features include first-order statistical features; and the morphological features include two-dimensional morphological features. In addition, different image filters are applied to the original MRI tumor image, and the above features except for the morphological features are also extracted in sequence for the obtained filtered MRI tumor image. The image filters include gradient, wavelet, square, square root, logarithm, exponential, Gaussian Laplacian operator (LoG), and local binary 2D pattern. The extracted radiomics features are output as a 512-dimensional feature vector through a one-layer fully connected Embedding.

[0038] The deep learning features of the MRI sequence are extracted by a deep learning feature extractor (for example, Resnet-50). The network structure of the deep learning feature extractor mainly includes seven parts: the first part is to perform convolution, regularization, activation function and maximum pooling operation on the input image; the second to fifth parts use residual modules, each residual block includes three convolution layers, plus a convolution layer before each residual block that changes the dimension of the residual block, and the network has a total of 1+3x(3+4+6+3)=49 convolution layers; the output of the last convolution layer is subjected to a global average pooling operation, and then a fully connected layer is connected to obtain a deep learning feature vector with the same output dimension as the radiomics feature vector, and the feature spaces of the two are aligned.

[0039] In S102, a positive sample pair is constructed by using the original sequence radiomics features in the radiomics feature vector and the same sequence enhanced deep learning features in the deep learning feature vector.

[0040] In S103, a negative sample pair is constructed by using the original sequence radiomics features and the different label sequence deep learning features in the deep learning feature vector.

[0041] In S104, based on the initial prediction model, the immunotherapy efficacy prediction model is obtained by maximizing the similarity of the positive sample pair and minimizing the similarity of the negative sample pair, so as to realize the immunotherapy efficacy prediction by using the immunotherapy efficacy prediction model.

[0042] Optionally, based on the initial prediction model, the immunotherapy efficacy prediction model is obtained by maximizing the similarity of the positive sample pair and minimizing the similarity of the negative sample pair, which includes training the initial prediction model by using a first loss function, so that the positive sample pair is close in the feature space and the negative sample pair is far away in the feature space, and the immunotherapy efficacy prediction model is obtained.

[0043] The initial prediction model can be a basic feature extraction and fusion network architecture that is not optimized by contrast learning, and its core function is to provide feature generation capability for subsequent contrast learning and dynamically adjust the feature space distribution through a loss function. In some possible implementation schemes, the initial prediction model includes a radiomics encoder, a deep learning encoder and a feature space alignment layer.

[0044] In some specific application scenarios, it is assumed that the size of a training batch is N, that is, it contains N sequences, and a batch includes one positive sample sequence S p (multiple positive sample sequences can also be included, and this embodiment is described by taking one positive sample sequence S p as an example), and the remaining N-1 are negative sample sequences. The radiomics features of this batch are {R p ,R n(1) ,R n(k) ,…,Rn(N-1)}, the deep learning feature is {D p_aug ,D n(1) ,D n(k) ,…,D n(N-1)}, where R p Indicates S p The radiomics feature vector set of the sequence, S p_aug For S p The enhanced sequence, D p_aug is the enhanced sequence S p_aug A set of deep learning feature vectors. Each feature vector set is a set of multiple slice features, for example, R p ={r p1 , r p2 ,…,r pm}, D p_aug ={d p_aug1 , d p_aug2 ,…,d p_augm}, m is the total number of slices. R from the positive sample sequence p With D p_aug Form a positive sample pair, D n(k) All come from negative sample sequences, R p With D n(k) To make the slice features from the same label similar and the slice features from different labels dissimilar, the first loss function can be expressed as follows:

[0045]

[0046] Where N is the total number of samples in the batch, N0 is the number of positive samples in the batch, k is the index of the number of negative sample sequences in a batch, m is the number of slices in each sequence, and i is the index of the sequence slice. Before calculating the first loss function, the feature vector is L2 regularized.

[0047] When the number of positive samples N0 contained in the batch is 1, the first loss function can be expressed as follows:

[0048]

[0049] In other specific application scenarios, in order to make the slice features from the same label similar and the slice features from different labels dissimilar, the first loss function can be expressed as follows:

[0050]

[0051] wherein τ is a similarity scaling parameter, and the value range is 0.05-0.5, for example, 0.05, 0.1, 0.2, 0.3, 0.35, 0.4, 0.45, 0.5. τ is used to balance the gradient contribution of positive and negative sample pairs, and control the discrimination of the feature space. When τ is small, the immunotherapy efficacy prediction model pays more attention to difficult samples (negative samples with similar similarity), and enhances the class boundary; when τ is large, the immunotherapy efficacy prediction model treats all samples equally, which may lead to slow convergence.

[0052] By using the prediction method for immunotherapy efficacy provided in the embodiments of the present disclosure, the radiomics feature vector and the deep learning feature vector of the original tumor sequence are obtained, then the radiomics features and the enhanced deep learning features of the same sequence are used to form a positive sample pair, the radiomics features and the deep learning features of different label sequences are used to form a negative sample pair, and based on the initial prediction model, the immunotherapy efficacy prediction model is obtained by maximizing the similarity of the positive sample pair and minimizing the similarity of the negative sample pair, so as to realize the prediction of the immunotherapy efficacy by using the immunotherapy efficacy prediction model. The prior knowledge of radiomics is transferred to the deep learning model, which can effectively improve the performance of the small sample deep learning model and the accuracy of the tumor immunotherapy efficacy prediction, and further improve the accuracy of the data-driven method represented by deep learning in the tumor treatment efficacy prediction.

[0053] In some embodiments, the positive sample pair is constructed by using the original sequence radiomics features in the radiomics feature vector and the same sequence enhanced deep learning features in the deep learning feature vector, including: extracting the original sequence radiomics features in the radiomics feature vector of the positive label according to the number of slices contained in the radiomics feature vector of the positive label, to obtain a radiomics feature vector set; extracting the deep learning features in the radiomics enhanced feature vector corresponding to the radiomics feature vector of the positive label according to the number of slices contained in the radiomics feature vector of the positive label, to obtain a first deep learning feature vector set; matching the elements in the radiomics feature vector set with the elements in the first deep learning feature vector set one by one, to obtain the positive sample pair.

[0054] In some specific application scenarios, combined with Figure 6 As shown in the figure, the positive label sequence Positive Sp with segmented tumor area (Tumor area) contains m slices {Slice1, Slice2, …, Slice m}. After extracting the 2D radiomics features of each slice by using the radiomics encoder (Radiomics Encoder), a set of radiomics feature vector set is obtained, denoted as R p = {r p1 , r p2 , …, r pmAfter data augmentation (random horizontal and vertical flipping, translation, rotation and shearing) of Sp, Sp_aug is obtained. After extracting 2D deep learning features from all slices of Sp_aug using the convolutional neural network encoder (CNNEncoder), a set of deep learning feature vectors is obtained, namely the first deep learning feature vector set D p_aug ={d p_aug1 , d p_aug2 ,…,d p_augm}. The radiomics feature vector set R of the original sequence Sp slice p ={r p1 , r p2 ,…,r pm} and the deep learning feature vector set D of Sp_aug slice p_aug ={d p_aug1 , d p_aug2 ,…,d p_augm} are matched one by one to form a positive sample pair (Positive Sample Pair). p , D p_aug ) can be {(r p1 , d p_aug1 ), (r p2 , d p_aug2 ),…,(r pm , d p_augm )}.

[0055] In some embodiments, negative sample pairs are constructed using the original sequence imaging omics features and the deep learning features of different label sequences in the deep learning feature vector, including: extracting the deep learning features in the imaging omics feature vector of the negative label according to the number of slices contained in the imaging omics feature vector of the negative label to obtain a second deep learning feature vector set; matching the elements in the imaging omics feature vector set with the elements in the second deep learning feature vector set one by one to obtain a negative sample pair.

[0056] In some specific application scenarios, continue to combine Figure 6 As shown in the figure, the negative label sequence Negative Sn of the segmented tumor area contains m slices {Slice1, Slice2, ..., Slice m}. For all slices of the negative label sequence Sn, the deep learning features are extracted by the 2D convolutional neural network encoder (CNN Encoder) to obtain a set, namely the second deep learning feature vector set D n ={d n1 , d n2 ,…,d nm}, set D nelements in R p p1 p2 pm elements in R p n p1 n1 p2 n2 pm nm

[0057] Optionally, the similarity of the positive sample pair is determined in the following manner: a first cosine similarity of the original sequence radiomics features and the same sequence enhanced deep learning features in the positive sample pair is calculated; and the first cosine similarity is taken as the similarity of the positive sample pair.

[0058] The similarity of the positive sample pair is calculated according to the following formula:

[0059]

[0060] wherein Similarity(Rp, Dp_aug) is the similarity of the positive sample pair, Rp i is a component of the radiomics feature vector set Rp, and Dp_aug i is a component of the first deep learning feature vector set D p_aug .

[0061] Optionally, the similarity of the negative sample pair is determined in the following manner: a second cosine similarity of the original sequence radiomics features and the different label sequence deep learning features in the negative sample pair is calculated; and the second cosine similarity is taken as the similarity of the negative sample pair.

[0062] The similarity of the negative sample pair is calculated according to the following formula:

[0063]

[0064] wherein Similarity(Rp, Dn) is the similarity of the positive sample pair, Rp i is a component of the radiomics feature vector set Rp, and Dn i is a component of the second deep learning feature vector set D n .

[0065] The embodiments of the present disclosure provide a prediction method for immunotherapy efficacy, which combines the deep learning technology and the radiomics technology. Figure 2 As shown in FIG. 1, the method comprises the following steps: ​​​​​​​​​​​

[0066] S201, obtain an imageomics feature vector and a deep learning feature vector of the original tumor sequence; wherein the feature spaces of the imageomics feature vector and the deep learning feature vector are aligned.

[0067] S202, according to the number of slices contained in the imageomics feature vector of the positive label, extract the original sequence imageomics features in the imageomics feature vector of the positive label, and obtain an imageomics feature vector set.

[0068] S203, according to the number of slices contained in the imageomics feature vector of the positive label, extract the deep learning features in the imageomics enhanced feature vector corresponding to the imageomics feature vector of the positive label, and obtain a first deep learning feature vector set.

[0069] S204, match the elements in the imageomics feature vector set with the elements in the first deep learning feature vector set one by one, and obtain a positive sample pair.

[0070] S205, according to the number of slices contained in the imageomics feature vector of the negative label, extract the deep learning features in the imageomics feature vector of the negative label, and obtain a second deep learning feature vector set.

[0071] S206, match the elements in the imageomics feature vector set with the elements in the second deep learning feature vector set one by one, and obtain a negative sample pair.

[0072] S207, based on the initial prediction model, by maximizing the similarity of the positive sample pair and minimizing the similarity of the negative sample pair, obtain an immunotherapy efficacy prediction model, so as to realize immunotherapy efficacy prediction by using the immunotherapy efficacy prediction model.

[0073] In the embodiments of the present disclosure, by integrating the imageomics feature vector and the deep learning feature vector and ensuring the alignment of the feature spaces of the two, the multi-dimensional information in the tumor sequence can be comprehensively and deeply mined. The imageomics features capture the key information such as morphology and texture in medical images, while the deep learning features extract higher-level abstract features through a neural network model. The combination of the two significantly improves the accuracy and robustness of the immunotherapy efficacy prediction model. At the same time, the positive sample pair is composed of the original sequence imageomics features and the corresponding deep learning features of the same patient, ensuring the consistency and correlation between samples, which helps the model to learn the feature combination that truly reflects the efficacy of immunotherapy. The negative sample pair is constructed by matching the features of different labels or different patients, which increases the generalization ability of the model, enabling it to more accurately distinguish between good and poor samples. Through the training strategy of maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs, not only the classification performance of the immunotherapy efficacy prediction model is optimized, but also its applicability in the immunotherapy efficacy prediction task is improved. The prediction method significantly improves the accuracy and practicality of the immunotherapy efficacy prediction model, bringing new technical breakthroughs and clinical application value to the field of cancer immunotherapy.

[0074] In some embodiments, the prediction method for immunotherapy efficacy further comprises: introducing a self-attention mechanism and a multiple-instance joint voting strategy into the immunotherapy efficacy prediction model to obtain an optimized immunotherapy efficacy prediction model, so as to realize immunotherapy efficacy prediction by using the optimized immunotherapy efficacy prediction model.

[0075] Optionally, introducing a self-attention mechanism into the immunotherapy efficacy prediction model comprises: calculating the attention weight between slices in the original tumor sequence; adjusting the weight of the slices in the original tumor sequence according to the attention weight, and fusing the original sequence imageomics features in the original tumor sequence according to the weight to obtain a new feature vector; and predicting the slice probability again through the new feature vector.

[0076] After the immunotherapy efficacy prediction model is trained, the self-attention mechanism is used to learn the spatial feature information. The self-attention mechanism can analyze the correlation (i.e. attention weight) between slices, and then perform weighted summation according to the correlation. After encoding by the self-attention mechanism, a new feature vector that integrates context-related information can be obtained. The new feature vector then outputs the prediction probability of each slice through Softmax.

[0077] Optionally, introducing a multiple-instance joint voting strategy (JP-MIL) into the immunotherapy efficacy prediction model comprises: training the immunotherapy efficacy prediction model using a second loss function, and calculating the prediction probability of the original tumor sequence by combining the slice probabilities in all original tumor sequences.

[0078] The second loss function can be expressed as follows:

[0079]

[0080] where P(y = 1|S) is the predicted probability of a positive label sequence, P(y = 0|S) is the predicted probability of a negative label sequence, J is the number of positive sample sequences in the batch training data, K is the number of negative sample sequences in the batch training data, N is the total number of batch data, i.e. N = J + K, W P = K / N, W N = J / N.

[0081] In practical applications, those skilled in the art can set the second loss function according to the actual situation, for example, other class balanced cross-entropy loss functions.

[0082] The predicted probability of the original tumor sequence can be calculated by combining the probabilities of all slices in the original tumor sequence according to the following formula:

[0083]

[0084] where P sequence is the predicted probability of the original tumor sequence with a positive label, p is the probability of each slice containing a tumor, i is the index value of the sequence slice, i.e. the i-th slice of the sequence, and n is the total number of slices contained in the sequence.

[0085] In the multi-instance joint voting strategy, the instances in the positive sample must have at least one positive instance, so the predicted probability of the original tumor sequence with a positive label can be considered as the negation of the probability that all slices do not contain tumors at the same time, i.e. it can be expressed as 1 minus the product of the probabilities of all slices with negative labels in the sequence.

[0086] The following specific examples are given to illustrate the principle of the immune therapy efficacy prediction model optimized by the combination of the self-attention mechanism and the multi-instance joint voting strategy:

[0087] Suppose the MRI sequence of a liver cancer patient contains 5 tumor slices, and the immune therapy efficacy prediction model predicts the probability of each slice being positive (significant efficacy) as follows: p = [0.8, 0.7, 0.3, 0.2, 0.1].

[0088] The role of the self-attention mechanism:

[0089] The self-attention calculates the correlation between slices by querying (Query), key (Key), and value (Value):

[0090]

[0091] The attention weights are calculated as: w = [0.4, 0.3, 0.1, 0.1, 0.1]. Thus, slices 1 and 2 are given higher weights because they contain tumor core regions.

[0092] The original features F = [f1, f2, f3, f4, f5] are fused by weights:

[0093]

[0094] Thus, the fused features Fnew focus more on high-weight slices (e.g., slices 1 and 2) and suppress low-weight noise.

[0095] The slice probabilities are re-predicted by the weighted features, which can be adjusted to: p' = [0.85, 0.75, 0.25, 0.15, 0.05].

[0096] Thus, the key slice probabilities are increased, and the noise slice probabilities are decreased.

[0097] The role of the JP-MIL strategy:

[0098] According to the formula i.e., P JP-MIL = 1 - (1 - 0.8)(1 - 0.7)(1 - 0.3)(1 - 0.2)(1 - 0.1) ≈ 0.97.

[0099] Thus, even if there are only two high-probability slices, the sequence-level prediction is still significantly biased towards positive.

[0100] Combining the above self-attention mechanism and JP-MIL strategy, the self-attention mechanism and multiple-instance joint voting strategy are jointly optimized, and the optimized slice probability p' = [0.85, 0.75, 0.25, 0.15, 0.05], based on the optimized probability to calculate the sequence-level result: P JP-MIL = 1 - (1 - 0.85)(1 - 0.75)(1 - 0.25)(1 - 0.15)(1 - 0.05) ≈ 0.99. After the self-attention improves the key slice probabilities, JP-MIL further amplifies the sequence-level confidence, thereby significantly improving the prediction robustness of the optimized immunotherapy efficacy prediction model under small samples.

[0101] Combining, for example, Figure 5The framework of the immunotherapy efficacy prediction model shown uses a 2D Radiomics feature extractor (2D Radiomics Encoder) to respectively encode the radiomics features of each slice of the blue primary tumor sequence, and then uses a 2D CNN feature extractor (2D Deep learning Encoder) to respectively encode the deep learning features of each slice of the orange enhanced image (or different label sequence image) of the primary tumor sequence. After that, the standardized Embedding feature vectors of the two sequence slices are paired, and the encoded feature vectors are made as close as possible in the embedding space through a measurement function. If the sequences are from different labels, the encoded feature vectors are made as far away as possible. Then the good encoder of the contrast learning is migrated to the multi-instance learning framework, and the output features of the encoding are learned through the self-attention mechanism (Attention), and then the slice probability is output through Softmax. The final sequence probability is jointly contributed by all slice probabilities.

[0102] In combination Figure 3 As shown, the prediction method for immunotherapy efficacy provided by the embodiments of the present disclosure includes the following steps:

[0103] S301, obtaining a radiomics feature vector and a deep learning feature vector of a primary tumor sequence; wherein the feature spaces of the radiomics feature vector and the deep learning feature vector are aligned.

[0104] S302, constructing a positive sample pair using a primary sequence radiomics feature in the radiomics feature vector and a same sequence enhanced deep learning feature in the deep learning feature vector.

[0105] S303, constructing a negative sample pair using a primary sequence radiomics feature and a different label sequence deep learning feature in the deep learning feature vector.

[0106] S304, based on an initial prediction model, obtaining an immunotherapy efficacy prediction model by maximizing the similarity of the positive sample pair and minimizing the similarity of the negative sample pair.

[0107] S305, introducing a self-attention mechanism and a multi-instance joint voting strategy into the immunotherapy efficacy prediction model to obtain an optimized immunotherapy efficacy prediction model, so as to realize immunotherapy efficacy prediction by using the optimized immunotherapy efficacy prediction model.

[0108] The prediction method for immunotherapy efficacy provided by the embodiments of the present disclosure introduces a self-attention mechanism and a multi-instance joint voting strategy into the immunotherapy efficacy prediction model. The self-attention mechanism can dynamically adjust the weights between different features, automatically identify and emphasize those features that are most critical to the prediction result, and capture the complex relationships between features, especially the nonlinear and long-distance dependent relationships, thereby improving the prediction accuracy of the model. In addition, the multi-instance joint voting strategy is introduced into the prediction of tumor immunotherapy efficacy. Since each patient may include image data or detection results at multiple time points, these data can be regarded as multiple instances. By integrating the prediction results of these instances, the information of multiple data sources can be comprehensively considered, thereby obtaining more stable and reliable prediction conclusions.

[0109] In combination Figure 4 The prediction method for immunotherapy efficacy provided by the embodiments of the present disclosure includes the following steps:

[0110] S401, obtaining a tumor image sequence to be detected.

[0111] Here, the tumor image sequence to be detected is derived from a medical imaging device, such as CT, MRI, or PET, etc. The tumor image sequence to be detected includes key information such as the morphology, size, and location of the tumor in the patient's body.

[0112] S402, inputting the tumor image sequence to be detected into an immunotherapy efficacy prediction model to obtain an immunotherapy efficacy prediction model output.

[0113] The tumor image sequence is input into the trained immunotherapy efficacy prediction model. By inputting the image data of the patient to be detected into the immunotherapy efficacy prediction model, the calculation process inside the immunotherapy efficacy prediction model can be triggered, and then the immunotherapy efficacy prediction output for the patient is generated.

[0114] S403, determining the immunotherapy efficacy prediction result by using the immunotherapy efficacy prediction model output.

[0115] The immunotherapy efficacy prediction model output can be a probability value or a score value. For example, when the immunotherapy efficacy prediction model output is 0.75, it means that the patient has a 75% chance of good efficacy after receiving immunotherapy, and a 25% chance of poor or ineffective efficacy.

[0116] The method for predicting the efficacy of immunotherapy provided by the embodiments of the present disclosure provides a new and efficient method for predicting the efficacy of immunotherapy. Through the automatic and intelligent prediction process, doctors can more quickly and accurately understand the treatment prospects of patients, thereby formulating more reasonable and effective treatment plans, which is of great significance for improving the efficacy of immunotherapy, reducing patient risks, and optimizing the allocation of medical resources.

[0117] In some possible implementation manners, the method for predicting the efficacy of immunotherapy can be extended to support the collaborative processing of tasks such as the prediction of the efficacy of immunotherapy, tumor staging, and type classification. Through multi-task collaboration, the medical imaging device can not only predict the efficacy, but also provide accurate diagnostic information and automatically generate medical records, thereby comprehensively improving the decision support capability.

[0118] In combination with Figure 7 The embodiments of the present disclosure provide a device 700 for predicting the efficacy of immunotherapy, which includes a processor 70 and a memory 71, and can further include a communication interface 72 and a bus 73. The processor 70, the communication interface 72, and the memory 71 can communicate with each other through the bus 73. The communication interface 72 can be used for information transmission. The processor 70 can invoke the logical instructions in the memory 71 to execute the method for predicting the efficacy of immunotherapy of the above-mentioned embodiments.

[0119] In addition, the logical instructions in the memory 71 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer-readable storage medium.

[0120] The memory 71 is a computer-readable storage medium, which can be used to store software programs, computer-executable programs, and the like, such as program instructions / modules corresponding to the method in the embodiments of the present disclosure. The processor 70 executes the function application and data processing by running the program instructions / modules stored in the memory 71, that is, implements the method for predicting the efficacy of immunotherapy in the above-mentioned method embodiments.

[0121] The memory 71 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the terminal device, and the like. In addition, the memory 71 can include a high-speed random access memory, and can further include a non-volatile memory.

[0122] The device for predicting the efficacy of immunotherapy provided by the embodiment of the present disclosure is used to obtain the imaging genomics feature vector and deep learning feature vector of the original tumor sequence, and then use the imaging genomics features and enhanced deep learning features of the same sequence to form a positive sample pair, and use the imaging genomics features and deep learning features of different label sequences to form a negative sample pair. Based on the initial prediction model, by maximizing the similarity of the positive sample pairs and minimizing the similarity of the negative sample pairs, an immunotherapy efficacy prediction model is obtained, thereby realizing immunotherapy efficacy prediction using the immunotherapy efficacy prediction model. Migrating the prior knowledge of imaging genomics to the deep learning model can effectively improve the performance of the small sample deep learning model and the accuracy in predicting the efficacy of tumor immunotherapy, thereby improving the accuracy of data-driven methods represented by deep learning in predicting the efficacy of tumor treatment.

[0123] Combine Figure 8 As shown, an embodiment of the present disclosure provides a medical imaging device 80 , comprising a medical imaging device body 800 , and the aforementioned device for predicting the efficacy of immunotherapy 700 , which is installed in the medical imaging device body 800 .

[0124] Embedding the immunotherapy efficacy prediction function into medical imaging equipment can realize real-time processing and prediction of medical imaging data, greatly enhancing the clinical application value of medical imaging equipment, and transforming medical imaging equipment from a simple diagnostic tool to an intelligent medical device with decision-making support functions. It has profound and broad significance in the field of medical imaging.

[0125] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned method for predicting the efficacy of immunotherapy.

[0126] An embodiment of the present disclosure provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the above-mentioned method for predicting the efficacy of immunotherapy.

[0127] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0128] The technical solutions of the embodiments of the present disclosure can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes one or more instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present disclosure. The aforementioned storage medium can be a non-transitory storage medium, including a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc. various media that can store program codes, or can be a transitory storage medium.

[0129] The above description and accompanying drawings sufficiently illustrate the embodiments of the present disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, process, and other changes. The embodiments represent only possible variations. Unless expressly required, individual components and functions are optional, and the order of operation may vary. Portions and features of some embodiments may be included in or substituted for portions and features of other embodiments. The scope of the embodiments of the present disclosure includes the entire scope of the claims, including all available equivalents thereof. When used in this application, although the terms "first," "second," etc. may be used in this application to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element can be called a second element, and similarly, a second element can be called a first element, without changing the meaning of the description, as long as all occurrences of "first element" are consistently renamed and all occurrences of "second element" are consistently renamed. The first element and the second element are both elements, but they may not be the same element. Furthermore, the terms used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of one or more of the associated listings. In addition, when used in this application, the term "comprise" and its variations "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups of these. Without further limitation, an element defined by the sentence "comprising a..." does not exclude the presence of other identical elements in the process, method or apparatus comprising the element. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can be referred to the description of the method part.

[0130] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods for each specific application to realize the described functions, but such implementation should not be considered beyond the scope of the embodiments of the present disclosure. The skilled person can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0131] In the embodiments disclosed herein, the disclosed methods, products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units can only be a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms. The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to implement the embodiments. In addition, each functional unit in the embodiments of the present disclosure can be integrated in one processing unit, or each unit can be a physically independent unit, or two or more units can be integrated in one unit.

[0132] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

Claims

1. A method for predicting the efficacy of immunotherapy, characterized in that: include: Obtaining radiomics feature vectors and deep learning feature vectors of the original tumor sequence; wherein the feature spaces of the radiomics feature vectors and the deep learning feature vectors are aligned; The positive sample pairs are constructed using the original sequence radiomics features in the radiomics feature vector and the same sequence enhanced deep learning features in the deep learning feature vector; Negative sample pairs are constructed using the original sequence radiomics features and the deep learning features of different label sequences in the deep learning feature vector; Based on the initial prediction model, an immunotherapy efficacy prediction model is obtained by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs, so as to realize immunotherapy efficacy prediction using the immunotherapy efficacy prediction model.

2. The prediction method according to claim 1, characterized in that The feature space alignment of the radiomics feature vector and the deep learning feature vector is controlled as follows: The extracted radiomic features of the original tumor sequence are encoded into radiomic feature vectors of set dimensions through a fully connected layer; The deep learning features of the extracted original tumor sequence are encoded into a deep learning feature vector of a set dimension through a fully connected layer.

3. The prediction method according to claim 1, wherein: The positive sample pairs are constructed using the original sequence radiomics features in the radiomics feature vector and the same sequence enhanced deep learning features in the deep learning feature vector, including: According to the number of slices contained in the radiomics feature vector of the positive label, the original sequence radiomics features in the radiomics feature vector of the positive label are extracted to obtain a set of radiomics feature vectors; Extracting deep learning features from the radiomics enhancement feature vector corresponding to the radiomics feature vector of the positive label according to the number of slices contained in the radiomics feature vector of the positive label to obtain a first deep learning feature vector set; Match the elements in the radiomics feature vector set with the elements in the first deep learning feature vector set one by one to obtain positive sample pairs; and / or, Negative sample pairs are constructed using the original sequence radiomics features and the different label sequence deep learning features in the deep learning feature vector, including: Extracting deep learning features from the radiomics feature vector of the negative label according to the number of slices contained in the radiomics feature vector of the negative label to obtain a second deep learning feature vector set; The elements in the radiomics feature vector set are matched one by one with the elements in the second deep learning feature vector set to obtain negative sample pairs.

4. The prediction method according to claim 1, wherein: The similarity of the positive sample pair is determined as follows: Calculate the first cosine similarity between the original sequence radiomics features and the enhanced deep learning features of the same sequence in the positive sample pair; The first cosine similarity is used as the similarity of the positive sample pair; and / or, The similarity of the negative sample pair is determined as follows: Calculate the second cosine similarity between the original sequence radiomics features and the deep learning features of different label sequences in the negative sample pairs; The second cosine similarity is used as the similarity of the negative sample pair.

5. The prediction method according to claim 1, wherein: Based on the initial prediction model, the immunotherapy efficacy prediction model is obtained by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs, including: The first loss function is used to train the initial prediction model so that the positive sample pairs are close in the feature space and the negative sample pairs are far away in the feature space, thereby obtaining an immunotherapy efficacy prediction model.

6. The prediction method according to any one of claims 1 to 5, characterized in that: Also includes: The self-attention mechanism and multi-instance joint voting strategy are introduced into the immunotherapy efficacy prediction model to obtain an optimized immunotherapy efficacy prediction model, so as to realize immunotherapy efficacy prediction using the optimized immunotherapy efficacy prediction model.

7. The prediction method according to claim 6, characterized in that The self-attention mechanism is introduced into the immunotherapy efficacy prediction model, including: Calculate the attention weights between slices in the original tumor sequence; The weights of the slices in the original tumor sequence are adjusted according to the attention weights, and the original sequence imaging features in the original tumor sequence are fused according to the weights to obtain a new feature vector; Re-predict the slice probability using the new feature vector; and / or, A multi-instance joint voting strategy is introduced into the immunotherapy efficacy prediction model, including: The second loss function is used to train the immunotherapy efficacy prediction model, and the slice probabilities in all original tumor sequences are combined to calculate the predicted probability of the original tumor sequence.

8. A method for predicting the efficacy of immunotherapy, characterized in that: include: Obtaining an imaging sequence of the tumor to be detected; Inputting the tumor image sequence to be detected into the immunotherapy efficacy prediction model according to any one of claims 1 to 7 to obtain an immunotherapy efficacy prediction model output; The immunotherapy efficacy prediction model output is used to determine the immunotherapy efficacy prediction results.

9. A device for predicting the efficacy of immunotherapy, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to perform the method for predicting the efficacy of immunotherapy according to any one of claims 1 to 7 when executing the program instructions.

10. A medical imaging device, characterized in that: include; Medical imaging equipment itself; The device for predicting the efficacy of immunotherapy as claimed in claim 9 is installed on a medical imaging device.

Images