Method and device for prognosis prediction and treatment scheme processing of hypopharyngeal cancer induction chemotherapy and medium

Abstract

The application discloses a processing method and device of hypopharyngeal carcinoma induction chemotherapy prognosis prediction and treatment scheme and a medium, which can be applied to the technical field of big data processing. The application obtains a plurality of historical multi-modal information of a plurality of historical primary treatment hypopharyngeal carcinoma patients before induction chemotherapy, including a first CT image, first clinical text information, a first induction chemotherapy scheme selected by the historical primary treatment hypopharyngeal carcinoma patient and total survival time. Then, the preset model is trained through the historical multi-modal information, and then the score cutoff value corresponding to the training completion of the preset model is determined. Then, the current multi-modal information of the current primary treatment hypopharyngeal carcinoma patient before induction chemotherapy is input into the trained preset model to predict a current risk value. Then, the target recommended scheme of induction chemotherapy is determined based on the current risk value and the score cutoff value, so that the determined target recommended scheme is closer to the actual situation, and the matching degree of the recommended chemotherapy scheme and the actual situation is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data processing, and in particular to a hypopharyngeal cancer induction chemotherapy prognosis prediction and treatment scheme processing method and device and medium. BACKGROUND

[0002] In the related art, head and neck squamous cell carcinoma (HNSCC) is the sixth most common malignant tumor worldwide. Among them, although the incidence of hypopharyngeal squamous cell carcinoma (HPSCC, hereinafter referred to as hypopharyngeal cancer) is low, because of its hidden location and strong invasiveness, it is often diagnosed in the late stage. The traditional surgical treatment method for locally advanced HPSCC is total laryngectomy, but this surgery will damage the patient's voice function and may cause postoperative complications such as pharyngeal fistula. Induction chemotherapy (IC) as a laryngeal preservation strategy has been increasingly used as the initial treatment for locally advanced HPSCC to improve the quality of life of patients. However, different patients have different responses to induction chemotherapy. Patients who do not respond to induction chemotherapy not only have to bear the economic pressure of drugs and suffer from the toxic and side effects of chemotherapy, but may even delay the disease due to tumor progression, so it is very important to identify hypopharyngeal cancer patients who respond to induction chemotherapy as soon as possible. At present, the method of constructing a model based on the image of a hypopharyngeal cancer patient before induction chemotherapy to predict the efficacy of hypopharyngeal cancer induction chemotherapy can only assist in determining whether to perform induction chemotherapy, but cannot recommend a specific induction chemotherapy scheme according to the actual situation.

[0003] In summary, the technical problems existing in the related art need to be improved. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a hypopharyngeal cancer induction chemotherapy prognosis prediction and treatment scheme processing method and device and medium, which can improve the matching degree of the recommended chemotherapy scheme and the actual situation.

[0005] To achieve the above-mentioned purpose, one aspect of the embodiments of the present application provides a hypopharyngeal cancer induction chemotherapy prognosis prediction and treatment scheme processing method, which comprises the following steps:

[0006] Obtaining historical multi-modal information of a plurality of historically treated hypopharyngeal cancer patients before induction chemotherapy, wherein the historical multi-modal information comprises a first CT image, first clinical text information, a first induction chemotherapy scheme selected by the historically treated hypopharyngeal cancer patient, and a total survival period corresponding to the first induction chemotherapy scheme;

[0007] training a preset model by the historical multi-modal information, the preset model comprising a risk assessment model for prognosis prediction and treatment scheme of hypopharyngeal cancer induced chemotherapy;

[0008] determining a score cutoff value corresponding to the preset model after the training is completed;

[0009] obtaining current multi-modal information of a hypopharyngeal cancer patient currently treated before induction chemotherapy, the current multi-modal information comprising a second CT image and second clinical text information;

[0010] inputting the current multi-modal information into the trained preset model to predict a current risk value;

[0011] determining a target recommended scheme of induction chemotherapy according to the current risk value and the score cutoff value.

[0012] In some embodiments, the preset model comprises a CT image preprocessing module, a CT image feature extraction module, a multi-modal feature screening module, and a multi-modal feature fusion and prediction module.

[0013] In some embodiments, the training of the preset model by the historical multi-modal information comprises:

[0014] preprocessing the first CT image by the CT image preprocessing module to obtain a region of interest mask file corresponding to the first CT image and a third CT image;

[0015] performing first feature extraction on the region of interest mask file and the third CT image by the CT image feature extraction module to obtain first CT radiomics features;

[0016] performing second feature extraction on the region of interest mask file and the third CT image by the CT image feature extraction module using a preset deep learning network model to obtain deep learning features;

[0017] screening the first CT radiomics features by the multi-modal feature screening module to obtain second CT radiomics features, and screening the first clinical text information according to the overall survival to obtain third clinical text information;

[0018] inputting the second CT radiomics features, the deep learning features, and the third clinical text information into the multi-modal feature fusion and prediction module for overall risk prediction of induction chemotherapy prognosis and independent risk prediction of induction chemotherapy prognosis corresponding to each of the first induction chemotherapy schemes to obtain an overall risk value of first induction chemotherapy prognosis and a first induction chemotherapy prognosis independent risk value corresponding to each of the first induction chemotherapy schemes; and calculating an overall loss function of the multi-modal feature fusion and prediction module.

[0019] adjusting model parameters of the multi-modal feature fusion and prediction module according to the overall loss function.

[0020] In some embodiments, the preset deep learning network model comprises an input layer, an initial convolutional layer, a first stage processing layer, a second stage processing layer, a third stage processing layer, a fourth stage processing layer, a global average pooling layer, a full connection layer, and an output layer.

[0021] The input layer receives the region of interest mask file and the third CT image.

[0022] The first stage processing layer comprises three residual attention blocks, each of which is obtained by connecting a residual block and a channel attention mechanism module in series.

[0023] The second stage processing layer comprises four residual attention blocks.

[0024] The third stage processing layer comprises six residual attention blocks.

[0025] The fourth stage processing layer comprises three residual attention blocks.

[0026] The input layer, the initial convolutional layer, the first stage processing layer, the second stage processing layer, the third stage processing layer, the fourth stage processing layer, the global average pooling layer, the full connection layer, and the output layer are connected in series.

[0027] The output layer is used to output deep learning features.

[0028] In some embodiments, the multi-modal feature fusion and prediction module comprises a full connection layer, a group-aware attention layer, a gated cross-modal fusion layer, an overall prediction head, a first group prediction head, a second group prediction head, a third group prediction head, and a fourth group prediction head.

[0029] The full connection layer is used to receive the second CT imageomics features, the deep learning features, and the third clinical text information.

[0030] The full connection layer, the group-aware attention layer, and the gated cross-modal fusion layer are connected in series.

[0031] The input ends of the overall prediction head, the first group prediction head, the second group prediction head, the third group prediction head, and the fourth group prediction head are connected to the output end of the gated cross-modal fusion layer.

[0032] In some embodiments, the determining of the score cutoff value corresponding to the preset model training completion comprises:

[0033] generating an ROC curve according to the first induction chemotherapy overall prognosis risk value after determining that the multimodal feature fusion and prediction module in the preset model is trained;

[0034] calculating the Youden index corresponding to each point in the ROC curve;

[0035] selecting the first induction chemotherapy overall prognosis risk value corresponding to the point with the maximum Youden index in all Youden indexes as the score cutoff value.

[0036] In some embodiments, the determining the target recommended induction chemotherapy scheme according to the current risk value and the score cutoff value comprises:

[0037] when the current induction chemotherapy overall prognosis risk value in the current risk value is less than the score cutoff value, obtaining all second induction chemotherapy prognosis independent risk values in the current risk value, each second induction chemotherapy prognosis independent risk value corresponding to a second induction chemotherapy scheme;

[0038] sorting all second induction chemotherapy prognosis independent risk values;

[0039] generating the target recommended induction chemotherapy scheme corresponding to the current primary treated hypopharyngeal carcinoma patient based on the sorting result and the second induction chemotherapy prognosis independent risk value.

[0040] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application proposes a processing device for hypopharyngeal carcinoma induction chemotherapy prognosis prediction and treatment scheme, the device comprises:

[0041] The first module is configured to obtain historical multimodal information of a plurality of historical primary treated hypopharyngeal carcinoma patients before induction chemotherapy, wherein the historical multimodal information comprises a first CT image, first clinical text information, a first induction chemotherapy scheme selected by the historical primary treated hypopharyngeal carcinoma patient, and a total survival period corresponding to the first induction chemotherapy scheme.

[0042] The second module is configured to train a preset model by using the historical multimodal information, wherein the preset model comprises a risk assessment model for hypopharyngeal carcinoma induction chemotherapy prognosis prediction and treatment scheme.

[0043] The third module is configured to determine a score cutoff value corresponding to the preset model after the training of the preset model is completed.

[0044] The fourth module is configured to obtain current multimodal information of a current primary treated hypopharyngeal carcinoma patient before induction chemotherapy, wherein the current multimodal information comprises a second CT image and second clinical text information.

[0045] a fifth module configured to input the current multi-modal information into the trained preset model to predict a current risk value;

[0046] a sixth module configured to determine a target recommended regimen of induction chemotherapy according to the current risk value and the score cutoff value.

[0047] To achieve the above object, another aspect of the embodiments of the present application provides a computer device, comprising:

[0048] at least one processor;

[0049] at least one memory configured to store at least one program;

[0050] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0051] To achieve the above object, another aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method.

[0052] The embodiments of the present application at least have the following beneficial effects: the present application provides a processing method and device of prognosis prediction and treatment regimen of hypopharyngeal cancer induction chemotherapy and a medium, the regimen obtains the historical multi-modal information of a plurality of historical primary treatment hypopharyngeal cancer patients before induction chemotherapy after including the first CT image, the first clinical text information, the first induction chemotherapy regimen selected by the historical primary treatment hypopharyngeal cancer patient and the total survival period corresponding to the first induction chemotherapy regimen, trains the preset model through the historical multi-modal information, then determines the score cutoff value corresponding to the preset model after the training is completed, inputs the current multi-modal information of the current primary treatment hypopharyngeal cancer patient before induction chemotherapy into the trained preset model to predict the current risk value, and then determines the target recommended regimen of induction chemotherapy based on the current risk value and the score cutoff value, so that the determined target recommended regimen is closer to the actual situation, and the matching degree of the recommended chemotherapy regimen and the actual situation is improved. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is a flowchart of the processing method of prognosis prediction and treatment regimen of hypopharyngeal cancer induction chemotherapy provided by the embodiments of the present application;

[0054] Figure 2 is a module schematic diagram of the preset model provided by the embodiments of the present application;

[0055] Figure 3 is a module schematic diagram of the 3D SE-Resnet-50 deep learning model provided by the embodiments of the present application;

[0056] Figure 4 is a module schematic diagram of a multi-modal feature fusion and prediction module provided by an embodiment of the present application;

[0057] Figure 5 is a flowchart of a scheme recommendation by a preset model provided by an embodiment of the present application;

[0058] Figure 6 is a structural schematic diagram of a processing device for hypopharyngeal cancer induction chemotherapy prognosis prediction and treatment scheme provided by an embodiment of the present application;

[0059] Figure 7 is a hardware structural schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description relates to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation described in the following exemplary embodiments does not represent all the implementations consistent with the embodiments of the present application, but is only an example of devices and methods consistent with some aspects of the embodiments of the present application.

[0061] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0062] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by those skilled in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0064] Before the embodiments of the present application are described in detail, first, some of the terms and terminology involved in the embodiments of the present application are explained, and the terms and terminology involved in the embodiments of the present application are applicable to the following explanations:

[0065] Induction chemotherapy (also known as neoadjuvant chemotherapy) is a systemic chemotherapy performed before surgery or radiotherapy, aiming to reduce tumor volume, alleviate symptoms, and create more favorable conditions for subsequent local treatment.

[0066] Induction chemotherapy prognosis refers to the comprehensive evaluation of disease progression, treatment effect and long-term survival of patients after completing induction chemotherapy (neoadjuvant chemotherapy).

[0067] In the related art, head and neck squamous cell carcinoma (HNSCC) is the sixth most common malignant tumor worldwide. Among them, although the incidence of hypopharyngeal squamous cell carcinoma (HPSCC) is relatively low, due to its hidden location and strong invasiveness, it is often diagnosed in the late stage. The traditional surgical treatment method for locally advanced HPSCC is total laryngectomy, but this surgery will damage the patient's voice function and may cause postoperative complications such as pharyngeal fistula. Induction chemotherapy (IC) as a laryngeal preservation strategy has been increasingly used as the initial treatment for locally advanced HPSCC to improve the quality of life of patients. However, different patients have different responses to induction chemotherapy. Patients who do not respond to induction chemotherapy not only have to bear the economic pressure of drugs and suffer from the side effects of chemotherapy, but may also delay the disease due to tumor progression, so it is very important to identify HPSCC patients who respond to induction chemotherapy as soon as possible. The main drugs currently used in the induction chemotherapy regimen for HPSCC include traditional chemotherapy drugs (taxanes, platinum, 5-FU), immunotherapy drugs (PD-1 inhibitors), and targeted therapy drugs (EGFR inhibitors). The commonly used induction chemotherapy regimens derived from them include TP regimen (taxanes + platinum), TPF regimen (taxanes + platinum + 5-FU), TPK regimen (taxanes + platinum + PD-1 inhibitor), and TPR regimen (taxanes + platinum + EGFR inhibitor). The laryngeal function preservation rate, pathological remission rate, and survival benefit of different regimens also differ. However, at present, the method of constructing a model based on the image of the hypopharyngeal carcinoma patient before induction chemotherapy to predict the efficacy of induction chemotherapy for hypopharyngeal carcinoma can only assist in determining whether to perform induction chemotherapy, but cannot recommend a specific induction chemotherapy regimen based on the actual situation.

[0068] Therefore, the application provides a hypopharyngeal cancer induction chemotherapy prognosis prediction and treatment scheme processing method and device and medium, which can effectively improve the matching degree of the recommended chemotherapy scheme and the actual situation.

[0069] The hypopharyngeal cancer induction chemotherapy prognosis prediction and treatment scheme processing method provided by the application is related to the technical field of big data processing. The hypopharyngeal cancer induction chemotherapy prognosis prediction and treatment scheme processing method provided by the application can be applied to a terminal, can be applied to a server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto. The server end can be configured as a stand-alone physical server, can be configured as a server cluster composed of multiple physical servers or a distributed system, can be configured as a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms, and the server can also be a node server in a blockchain network. The software can be an application that implements the hypopharyngeal cancer induction chemotherapy prognosis prediction and treatment scheme processing method, and the like, but is not limited to the above forms.

[0070] It should be noted that in each specific embodiment of the application, when relevant processing needs to be performed according to user information, user behavior data, user historical data, and user location information and other data related to the identity or characteristics of the user, the user's permission or consent will be obtained first, and the collection, use, and processing of the data will comply with relevant laws, regulations, and standards. In addition, when the application embodiments need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the application embodiments to normally operate will be obtained.

[0071] The application embodiments will be specifically described below with reference to the accompanying drawings:

[0072] Figure 1 is an optional flowchart of the hypopharyngeal cancer induction chemotherapy prognosis prediction and treatment scheme processing method provided by the application, Figure 1 The method in can include but is not limited to steps S110 to S160:

[0073] Step S110, obtaining historical multi-modal information of a plurality of historically initial treatment hypopharyngeal cancer patients before induction chemotherapy, wherein the historical multi-modal information comprises a first CT image, first clinical text information, a first induction chemotherapy scheme selected by the historically initial treatment hypopharyngeal cancer patient, and an overall survival period corresponding to the first induction chemotherapy scheme;

[0074] Step S120, training a preset model through the historical multi-modal information, wherein the preset model comprises a hypopharyngeal cancer induction chemotherapy prognosis prediction and treatment scheme risk assessment model;

[0075] Step S130, determining a score cutoff value corresponding to the preset model after training is completed;

[0076] Step S140, obtaining current multi-modal information of a current initial treatment hypopharyngeal cancer patient before induction chemotherapy, wherein the current multi-modal information comprises a second CT image and second clinical text information;

[0077] Step S150, inputting the current multi-modal information into the trained preset model to predict a current risk value;

[0078] Step S160, determining a target recommended scheme of induction chemotherapy according to the current risk value and the score cutoff value.

[0079] It can be understood that the first CT image and the second CT image involved in the embodiment can both be portal phase head and neck enhancement 3D CT images, and these CT images can be stored in DICOM format. The first clinical text information and the second clinical text information can include but are not limited to age, gender, tumor differentiation grade, T stage, N stage, M stage, clinical stage (according to the AJCC 8th edition staging system), white blood cell count, red blood cell count, hemoglobin, platelet count, neutrophil count, lymphocyte count, monocyte count, eosinophil count, basophil count, alanine aminotransferase, aspartate aminotransferase, serum albumin, C-reactive protein, procalcitonin, and calculation of inflammatory indicators including systemic immune inflammatory index (= platelet count x neutrophil count / lymphocyte count), neutrophil / lymphocyte ratio, platelet / lymphocyte ratio, and lymphocyte / monocyte ratio. The first induction chemotherapy scheme and the second induction chemotherapy scheme can be one or more of the TP scheme, the TPF scheme, the TPK scheme, and the TPR scheme. The overall survival period (OS, overall survival) can be recorded as a death event and a death date if the patient dies within the follow-up period, or as a censored event and a follow-up date if the patient survives.

[0080] When the multiple historical primary treatment of hypopharyngeal cancer patients is completed, the historical multi-modal information collection before induction chemotherapy is collected, and all the collected information is divided into a training set and a test set according to a ratio of 7:3. Among them, the training set is used to train the preset model including the prognosis prediction of pharyngeal cancer induction chemotherapy and the risk assessment model of the treatment plan, and the test set is used to test the preset model in the training process.

[0081] It can be understood that, as shown in Figure 2 , the preset model of the embodiment can include but is not limited to including a CT image preprocessing module, a CT image feature extraction module, a multi-modal feature screening module, and a multi-modal feature fusion and prediction module. Based on Figure 2 the preset model structure as shown, the preset model is trained by the training set in the historical multi-modal information, and specifically, the training process of the embodiment includes but is not limited to the following steps:

[0082] The first CT image is preprocessed by the CT image preprocessing module to obtain a region of interest mask file corresponding to the first CT image and a third CT image;

[0083] The region of interest mask file and the third CT image are subjected to first feature extraction by the CT image feature extraction module to obtain first CT radiomics features;

[0084] The region of interest mask file and the third CT image are subjected to second feature extraction by the CT image feature extraction module using a preset deep learning network model to obtain deep learning features;

[0085] The first CT radiomics features are screened by the multi-modal feature screening module to obtain second CT radiomics features, and the first clinical text information is screened according to the overall survival to obtain third clinical text information;

[0086] The second CT radiomics features, the deep learning features, and the third clinical text information are input into the multi-modal feature fusion and prediction module for overall risk prediction of induction chemotherapy prognosis and independent risk prediction of induction chemotherapy prognosis corresponding to each first induction chemotherapy scheme, to obtain a first induction chemotherapy prognosis overall risk value and a first induction chemotherapy prognosis independent risk value corresponding to each first induction chemotherapy scheme; and calculate the overall loss function of the multi-modal feature fusion and prediction module;

[0087] The model parameters of the multi-modal feature fusion and prediction module are adjusted according to the overall loss function.

[0088] It can be understood that the process of the CT image preprocessing module preprocessing the first CT image can be manually delineating a hypopharyngeal tumor region of interest (ROI) in the continuous layers of the first CT image by using the 3D Slicer software to obtain a region of interest mask file; at the same time, performing image resampling, image intensity normalization and image cropping operations on the first CT image to obtain a third CT image. Among them, image resampling refers to uniformly resampling the voxel size of the first CT image to 1*1*1nm; image intensity normalization refers to limiting the HU value (Hounsfield unit) range of the first CT image to a certain range by using the maximum and minimum normalization method; the image cropping operation refers to cropping the CT image layers that do not contain the manually delineated ROI.

[0089] It can be understood that the process of the CT image feature extraction module performing first feature extraction on the region of interest mask file and the third CT image can be extracting low-order feature morphological features, intensity histogram features, texture features from the ROI region in the region of interest mask file by using the PyRadiomics Python package, and then combining the above low-order feature morphological features after wavelet transform and Laplacian of Gaussian (LoG) conversion to obtain high-order features to form the first CT image feature. Among them, CT image feature refers to the quantifiable structure and function information extracted from medical images by high-throughput technology, which is used to describe the disease state and assist clinical decision-making. The essence lies in converting visual images into a computable feature space, and mining potential diagnostic or predictive value through statistical analysis and machine learning models. The main feature types of CT image feature include but are not limited to shape features, intensity features, texture features, topological features, etc.

[0090] In the process of the CT image feature extraction module using the preset deep learning network model to perform second feature extraction on the region of interest mask file and the third CT image, the preset deep learning network model of the present embodiment can adopt a 3D SE-Resnet-50 deep learning model. Taking the 3D SE-Resnet-50 deep learning model for feature extraction as an example, the present embodiment inputs the region of interest mask file and the third CT image into the 3D SE-Resnet-50 deep learning model to perform feature extraction on the region of interest mask file and the third CT image to obtain deep learning features.

[0091] Specifically, the 3D SE-Resnet-50 deep learning model can be a model obtained after introducing a three-dimensional convolution operation and an SE (Squeeze-and-Excitation) attention module into a pre-trained Resnet-50 model framework, which can be suitable for processing three-dimensional CT image data, capturing image feature information and improving the attention ability of the model to key features. As shown in Figure 3 As shown in the figure, the preset deep learning network model includes an input layer, an initial convolution layer, a first stage processing layer, a second stage processing layer, a third stage processing layer, a fourth stage processing layer, a global average pooling layer, a full connection layer and an output layer; the input layer receives the region of interest mask file and the third CT image; the first stage processing layer includes 3 residual attention blocks, each residual attention block is obtained by connecting a residual block and a channel attention (SE) mechanism module in series; the second stage processing layer includes 4 residual attention blocks; the third stage processing layer includes 6 residual attention blocks; the fourth stage processing layer includes 3 residual attention blocks; the input layer, the initial convolution layer, the first stage processing layer, the second stage processing layer, the third stage processing layer, the fourth stage processing layer, the global average pooling layer, the full connection layer and the output layer are connected in series; the output layer is used for outputting deep learning features. In this embodiment, the plurality of residual attention blocks in each stage processing layer can be in series or in parallel, and can be adaptively adjusted according to actual conditions. The residual attention block can be an SE attention module introduced after the residual block.

[0092] It can be understood that the process of the multi-modal feature screening module processing the first CT imageomic features and the first clinical text can be to sequentially perform Z-score normalization, Spearman correlation analysis and LASSO-Cox regression processing on all the first CT imageomic features. Among them, the Spearman correlation analysis can be used to exclude second CT imageomic features with a correlation coefficient greater than 0.95. LASSO-Cox regression means introducing the L1 regularization term in LASSO (Least Absolute Shrinkage and Selection Operator) into the Cox regression model, λ is a regularization parameter, the partial likelihood deviance under different λ is calculated by 10-fold cross-validation, and the λ value that makes the cross-validation partial likelihood deviance minimum is selected as the best value, and the imageomic features with non-zero coefficients at this time are selected as the fourth CT image. At the same time, the first clinical text information is sequentially subjected to univariate and multivariate Cox regression analysis on overall survival (OS), and the text feature information that shows significant prognostic prediction significance (P<0.05) in statistics is selected as the third clinical text information.

[0093] In the embodiments of the present application, as shown in Figure 4 The multi-modal feature fusion and prediction module includes a full connection layer, a group-aware attention layer, a gated cross-modal fusion layer, an overall prediction head, a first group prediction head, a second group prediction head, a third group prediction head, and a fourth group prediction head. The full connection layer is configured to receive the fourth CT image, the deep learning feature, and the third clinical text information. The full connection layer, the group-aware attention layer, and the gated cross-modal fusion layer are connected in series. The input ends of the overall prediction head, the first group prediction head, the second group prediction head, the third group prediction head, and the fourth group prediction head are connected to the output end of the gated cross-modal fusion layer. It can be understood that the full connection layer is configured to fuse the second CT imageomics feature, the deep learning feature, and the third clinical text information after receiving the second CT imageomics feature, the deep learning feature, and the third clinical text information. The group-aware attention layer is configured to assign an independent attention head subset to each induction chemotherapy group to dynamically calculate the interaction between the intra-group and cross-group features. The group-aware attention layer can be represented by the following formula:

[0094]

[0095] wherein Q g is a query matrix of the induction chemotherapy group g, K g is a key matrix of the induction chemotherapy group g, K overall is a global key matrix, V g is a value matrix of the induction chemotherapy group g, d k is the dimension of the query matrix and the key matrix, is a matrix splicing operation.

[0096] The gated cross-modal fusion layer is configured to dynamically adjust the weight of each modality in each induction chemotherapy group. The adjustment formula is as follows:

[0097]

[0098] wherein W m is a gated weight matrix of the modality m, is an embedding vector of the modality m, h group is an embedding vector of the induction chemotherapy group, and sigma is a sigmoid activation function.

[0099] It can be understood that the embodiments further include an output layer composed of the overall prediction head, the first group prediction head, the second group prediction head, the third group prediction head, and the fourth group prediction head. The overall prediction head is configured to output an overall induction chemotherapy risk value S overall . The first group prediction head is configured to output an induction chemotherapy prognosis independent risk value S TPThe third group prediction head is configured to output an independent risk value S of induction chemotherapy prognosis of an induction chemotherapy regimen TPK TPF The third group prediction head is configured to output an independent risk value S of induction chemotherapy prognosis of an induction chemotherapy regimen TPK TPK The fourth group prediction head is configured to output an independent risk value S of induction chemotherapy prognosis of an induction chemotherapy regimen TPR TPR .

[0100] In the embodiment, when the multi-modal feature fusion and prediction module in the preset model is trained, a Cox partial log-likelihood function can also be constructed as a loss function of each prediction head

[0101]

[0102] In the formula, δ i is an event indicator (1 represents a death event, and 0 represents a censored event); Y i is the survival time of the ith individual; X i is the multi-modal vector of the ith individual; and β is a regression coefficient vector.

[0103] Then, a stratified Cox loss function is constructed as a loss function of the entire model:

[0104]

[0105] In the formula, is a total prognosis prediction loss function, is a prognosis prediction loss function of the kth induction chemotherapy regimen.

[0106] In the training process of the preset model, the embodiment can also output the total risk value S overall of induction chemotherapy of the training set, draw a receiver operating characteristic (ROC) curve, calculate a maximum Youden's Index, and select a cutoff value S cutoff to divide the high-risk group and the low-risk group of induction chemotherapy prognosis. Moreover, the preset model is tested by using the test set, and the prediction effect of the preset model is evaluated by drawing an ROC curve, calculating an AUC value, drawing a calibration curve, performing Decision Curve Analysis (DCA), and drawing a KM (Kaplan-Meier) curve during the testing process. Based on the evaluation of the prediction effect, the model parameters are adjusted to improve the model precision of the preset model.

[0107] It can be understood that the embodiment determines the score cutoff value S cutoff after determining that the multi-modal feature fusion and prediction module in the preset model is trained. The ROC curve is generated according to the overall risk value of the first induction chemotherapy. Then, the Youden index corresponding to each point in the ROC curve is calculated. Then, the first induction chemotherapy overall risk value corresponding to the point with the maximum Youden index in all Youden indexes is selected as the score cutoff value.

[0108] In the embodiment of the present application, after the training of the preset model and the determination of the cutoff value are completed, the embodiment determines the target recommendation scheme of the induction chemotherapy according to the current risk value and the score cutoff value. Specifically, when the current overall risk value of the induction chemotherapy in the current risk value is less than the score cutoff value, the embodiment can obtain all second overall risk values of the induction chemotherapy in the current risk value, sort all second overall risk values of the induction chemotherapy, and generate the target recommendation scheme of the induction chemotherapy corresponding to the current newly diagnosed hypopharyngeal cancer patient based on the sorting result and the second overall risk value of the induction chemotherapy. Each second overall risk value of the induction chemotherapy corresponds to a second induction chemotherapy scheme. For example, as shown in Figure 5 For a newly diagnosed hypopharyngeal cancer patient, after the current multi-modal information is input into the preset model, the preset model outputs the overall risk value S overall of the induction chemotherapy. Then, the overall risk value S overall of the induction chemotherapy is compared with the score cutoff value S cutoff to determine whether the current patient belongs to the high-risk group or the low-risk group. For the high-risk group (S overall >S cutoff ), the induction chemotherapy is not recommended. For the low-risk group (S overall <S cutoff ), the induction chemotherapy is recommended. According to the S TP , S TPF , S TPK , and S TPR output by the model, the four treatment risks are sorted from low to high, and the target recommendation scheme of the induction chemotherapy composed of the recommendation order of the treatment scheme and the corresponding risk score is output to assist the clinical doctors in decision-making.

[0109] From the above content, it can be known that the method of the embodiment of the present application introduces the attention mechanism, thereby promoting the cross-group and cross-modal interaction of the integrated enhanced CT and clinical text multi-modal feature information, and further making the recommendation scheme output by the preset model trained based on the multi-modal information closer to the actual situation, which can provide clinical decision support for the precision treatment of hypopharyngeal cancer patients.

[0110] Referring to Figure 6The embodiment of the application provides a processing device for hypopharyngeal cancer induction chemotherapy prognosis prediction and treatment scheme, and the device comprises:

[0111] The first module 610 is configured to acquire historical multi-modal information of a plurality of historical primary treated hypopharyngeal cancer patients before induction chemotherapy, wherein the historical multi-modal information comprises a first CT image, first clinical text information, a first induction chemotherapy scheme selected by the historical primary treated hypopharyngeal cancer patient, and a total survival period corresponding to the first induction chemotherapy scheme.

[0112] The second module 620 is configured to train a preset model by using the historical multi-modal information, wherein the preset model comprises a risk assessment model for hypopharyngeal cancer induction chemotherapy prognosis prediction and treatment scheme.

[0113] The third module 630 is configured to determine a score cutoff value corresponding to the preset model after training.

[0114] The fourth module 640 is configured to acquire current multi-modal information of a current primary treated hypopharyngeal cancer patient before induction chemotherapy, wherein the current multi-modal information comprises a second CT image and second clinical text information.

[0115] The fifth module 650 is configured to input the current multi-modal information into the trained preset model to predict a current risk value.

[0116] The sixth module 660 is configured to determine a target recommended scheme for induction chemotherapy according to the current risk value and the score cutoff value.

[0117] It can be understood that the contents in the above method embodiments are applicable to the device embodiments, the device embodiments specifically realize the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0118] The embodiment of the application further provides a computer device, which comprises a memory and a processor, the memory stores a computer program, and the processor realizes the above method when executing the computer program. The computer device can be any intelligent terminal such as a tablet computer, a vehicle-mounted computer and the like.

[0119] It can be understood that the contents in the above method embodiments are applicable to the device embodiments, the device embodiments specifically realize the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0120] Please refer to Figure 7 , Figure 7 The hardware structure of the computer device of another embodiment is illustrated, and the computer device comprises:

[0121] The processor 710 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0122] The memory 720 can be implemented by a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), and the like. The memory 720 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 720 and are called and executed by the processor 710 to implement the above-mentioned method of the embodiments of the present application.

[0123] The input / output interface 730 is configured to implement information input and output.

[0124] The communication interface 740 is configured to implement the communication interaction between the computer device and other devices, and can realize the communication through a wired manner (for example, a USB, a network cable, and the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, and the like).

[0125] The bus 750 is configured to transmit information between the components (for example, the processor 710, the memory 720, the input / output interface 730, and the communication interface 740) of the computer device.

[0126] The processor 710, the memory 720, the input / output interface 730, and the communication interface 740 are connected to each other through the bus 750 to realize the communication connection between the devices.

[0127] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by the processor to implement the above-mentioned method.

[0128] It can be understood that the contents in the above-mentioned method embodiments are all applicable to the present storage medium embodiments, the functions implemented by the present storage medium embodiments are the same as those of the above-mentioned method embodiments, and the beneficial effects achieved by the present storage medium embodiments are also the same as those achieved by the above-mentioned method embodiments.

[0129] The embodiments described in the specification are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0130] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0131] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0132] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0133] The terms "first", "second", "third", "fourth" and the like (if any) in the specification and above-described drawings of the present application are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0134] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b and c can be single or multiple.

[0135] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed mutual units can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0136] The units described above as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0137] In addition, each functional unit in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0138] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0139] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for predicting the prognosis and managing the treatment plan for hypopharyngeal carcinoma undergoing induction chemotherapy, characterized in that, The method includes the following steps: The historical multimodal information of multiple hypopharyngeal cancer patients who had previously received initial treatment before induction chemotherapy was obtained. The historical multimodal information included a first CT image, a first clinical text information, the first induction chemotherapy regimen selected by the patients who had previously received initial treatment, and the overall survival corresponding to the first induction chemotherapy regimen. The preset model is trained using the historical multimodal information, and the preset model includes a risk assessment model for predicting the prognosis of hypopharyngeal carcinoma induction chemotherapy and treatment plans. Determine the score cutoff value corresponding to the completion of the training of the preset model; Obtain current multimodal information of a hypopharyngeal cancer patient undergoing initial treatment before induction chemotherapy, wherein the current multimodal information includes a second CT image and second clinical text information; The current multimodal information is input into the trained preset model to predict the current risk value; The target recommended regimen for induction chemotherapy is determined based on the current risk value and the score cutoff value; The preset model includes a CT image preprocessing module, a CT image feature extraction module, a multimodal feature filtering module, and a multimodal feature fusion and prediction module. The multimodal feature fusion and prediction module includes a fully connected layer, a group-aware attention layer, a gated cross-modal fusion layer, and an output layer. The output layer includes an overall prediction head, a first group prediction head, a second group prediction head, a third group prediction head, and a fourth group prediction head. The fully connected layer is used to receive the fourth CT image, deep learning features, and third clinical text information. The fully connected layer, the group-aware attention layer, and the gated cross-modal fusion layer are connected in series. The inputs of the overall prediction head, the first group prediction head, the second group prediction head, the third group prediction head, and the fourth group prediction head are all connected to the output of the gated cross-modal fusion layer. The step of determining the target recommended regimen for induction chemotherapy based on the current risk value and the score cutoff value includes: When the overall prognostic risk value of the current induction chemotherapy in the current risk value is less than the score cutoff value, all independent prognostic risk values ​​of the second induction chemotherapy in the current risk value are obtained, and each independent prognostic risk value of the second induction chemotherapy corresponds to one second induction chemotherapy regimen; all independent prognostic risk values ​​of the second induction chemotherapy include the independent prognostic risk value of the induction chemotherapy regimen TP output by the first group prediction head. Independent risk values ​​for induction chemotherapy prognosis of TPF in the second group of predicted head output induction chemotherapy regimens Independent risk values ​​for prognostic outcomes of TPK-based induction chemotherapy regimens in the third group predicting head output. Independent risk values ​​for prognostic outcomes of TPR induction chemotherapy regimens in the fourth group predicting head output. ; The independent prognostic risk values ​​of all second induction chemotherapy were ranked; Based on the ranking results and the second induction chemotherapy prognostic independent risk value, a target recommendation for the induction chemotherapy is generated for the currently initially treated hypopharyngeal cancer patient.

2. The method according to claim 1, characterized in that, The step of training the preset model using the historical multimodal information includes: The first CT image is preprocessed by the CT image preprocessing module to obtain the region of interest mask file and the third CT image corresponding to the first CT image; The CT image feature extraction module performs first feature extraction on the region of interest mask file and the third CT image to obtain first CT image omics features. The CT image feature extraction module uses a preset deep learning network model to perform second feature extraction on the region of interest mask file and the third CT image to obtain deep learning features. The first CT radiomics feature is filtered by the multimodal feature filtering module to obtain the second CT radiomics feature, and the first clinical text information is filtered according to the overall survival to obtain the third clinical text information. The second CT image omics features, the deep learning features, and the third clinical text information are input into the multimodal feature fusion and prediction module to perform overall prognostic risk prediction of induction chemotherapy and independent prognostic risk prediction of each first induction chemotherapy regimen, thereby obtaining the overall prognostic risk value of the first induction chemotherapy and the independent prognostic risk value of each first induction chemotherapy regimen; and the overall loss function of the multimodal feature fusion and prediction module is calculated. The model parameters of the multimodal feature fusion and prediction module are adjusted based on the overall loss function.

3. The method according to claim 2, characterized in that, The preset deep learning network model includes an input layer, an initial convolutional layer, a first-stage processing layer, a second-stage processing layer, a third-stage processing layer, a fourth-stage processing layer, a global average pooling layer, a fully connected layer, and an output layer. The input layer receives the region of interest mask file and the third CT image; The first stage processing layer includes three residual attention blocks, each of which is obtained by concatenating one residual block and one channel attention mechanism module; The second-stage processing layer includes four residual attention blocks; The third-stage processing layer includes 6 residual attention blocks; The fourth-stage processing layer includes three residual attention blocks; The input layer, the initial convolutional layer, the first stage processing layer, the second stage processing layer, the third stage processing layer, the fourth stage processing layer, the global average pooling layer, the fully connected layer, and the output layer are connected in series. The output layer is used to output deep learning features.

4. The method according to claim 2, characterized in that, Determining the score cutoff value corresponding to the completion of the preset model training includes: After the multimodal feature fusion and prediction module in the preset model has completed training, an ROC curve is generated based on the overall prognostic risk value of the first induction chemotherapy. Calculate the Youden index corresponding to each point in the ROC curve; The overall prognostic risk value of the first induction chemotherapy is selected as the score cutoff value at the point corresponding to the largest Youden index among all the Youden indices.

5. A device for predicting the prognosis and processing treatment plans for hypopharyngeal cancer induction chemotherapy, characterized in that, The device includes: The first module is used to acquire historical multimodal information of multiple hypopharyngeal cancer patients who were previously treated before induction chemotherapy. The historical multimodal information includes a first CT image, a first clinical text information, the first induction chemotherapy regimen selected by the previously treated hypopharyngeal cancer patients, and the overall survival corresponding to the first induction chemotherapy regimen. The second module is used to train a preset model using the historical multimodal information. The preset model includes a risk assessment model for predicting the prognosis of hypopharyngeal carcinoma induction chemotherapy and treatment plans. The third module is used to determine the score cutoff value corresponding to the completion of the training of the preset model; The fourth module is used to obtain the current multimodal information of the hypopharyngeal cancer patient before induction chemotherapy, the current multimodal information including the second CT image and the second clinical text information; The fifth module is used to input the current multimodal information into the trained preset model to predict the current risk value; The sixth module is used to determine the target recommended regimen for induction chemotherapy based on the current risk value and the score cutoff value; The preset model includes a CT image preprocessing module, a CT image feature extraction module, a multimodal feature filtering module, and a multimodal feature fusion and prediction module. The multimodal feature fusion and prediction module includes a fully connected layer, a group-aware attention layer, a gated cross-modal fusion layer, and an output layer. The output layer includes an overall prediction head, a first group prediction head, a second group prediction head, a third group prediction head, and a fourth group prediction head. The fully connected layer is used to receive the fourth CT image, deep learning features, and third clinical text information. The fully connected layer, the group-aware attention layer, and the gated cross-modal fusion layer are connected in series. The inputs of the overall prediction head, the first group prediction head, the second group prediction head, the third group prediction head, and the fourth group prediction head are all connected to the output of the gated cross-modal fusion layer. The step of determining the target recommended regimen for induction chemotherapy based on the current risk value and the score cutoff value includes: When the overall prognostic risk value of the current induction chemotherapy in the current risk value is less than the score cutoff value, all independent prognostic risk values ​​of the second induction chemotherapy in the current risk value are obtained, and each independent prognostic risk value of the second induction chemotherapy corresponds to one second induction chemotherapy regimen; all independent prognostic risk values ​​of the second induction chemotherapy include the independent prognostic risk value of the induction chemotherapy regimen TP output by the first group prediction head. Independent risk values ​​for induction chemotherapy prognosis of TPF in the second group of predicted head output induction chemotherapy regimens Independent risk values ​​for prognostic outcomes of TPK-based induction chemotherapy regimens in the third group predicting head output. Independent risk values ​​for prognostic outcomes of TPR induction chemotherapy regimens in the fourth group predicting head output. ; The independent prognostic risk values ​​of all second induction chemotherapy were ranked; Based on the ranking results and the second induction chemotherapy prognostic independent risk value, a target recommendation for the induction chemotherapy is generated for the currently initially treated hypopharyngeal cancer patient.

6. A computer device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4.

Images