Postoperative concurrent risk prediction method, electronic equipment and storage medium

Abstract

The invention provides a postoperative concurrent risk prediction method, electronic equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: constructing an initial model based on a Transform architecture, and training the initial model based on a preset training data set to obtain a multi-modal deep learning radiology model; acquiring preoperative medical image data and clinical data of the to-be-detected object; based on the preoperative medical image data, radiomics features are extracted; carrying out feature fusion on the radiomics features and clinical data to obtain multi-modal input features; and inputting the multi-modal input features into a multi-modal deep learning radiology model, and sequentially processing the multi-modal input features by a multi-head attention module, a feedforward neural network module and a classification head module in the multi-modal deep learning radiology model to obtain the risk probability of the target postoperative complications of the to-be-detected object. The automatic prediction from the original image to the postoperative risk probability is realized, the human intervention is reduced, and the clinical availability is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method for predicting postoperative complications, an electronic device, and a storage medium. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, AI models have been applied to more and more business areas to assist in completing various tasks in various business areas.

[0003] In the medical field, AI models have been used for medical image analysis, auxiliary analysis of medical test results, and auxiliary prediction of postoperative complications. However, existing AI models still have certain limitations in terms of predictive accuracy. Summary of the Invention

[0004] The purpose of this application is to provide a method, electronic device, and storage medium for predicting postoperative complications, in order to address the shortcomings of the prior art and solve the technical problems existing in the prior art.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide a method for predicting postoperative complications, the method comprising: An initial model is built based on the Transformer architecture, and the initial model is trained based on a preset training dataset to obtain a multimodal deep learning radiometric model. Acquire preoperative medical imaging and clinical data of the subjects to be tested; Based on the preoperative medical imaging data, radiomics features were extracted; The radiomics features are fused with the clinical data to obtain multimodal input features; The multimodal input features are input into the multimodal deep learning radiology model, and are processed sequentially by the multi-head attention module, feedforward neural network module and classification head module in the multimodal deep learning radiology model to obtain the risk probability of the subject developing the target postoperative complications.

[0006] Optionally, the step of inputting the multimodal input features into a pre-trained multimodal deep learning radial model, where the multi-head attention module, feedforward neural network module, and classification head module of the multimodal deep learning radial model are processed sequentially, includes: The multimodal input feature set is embedded to convert each feature into a high-dimensional vector representation, resulting in an embedded feature sequence. A learnable [CLS] tag is added to the beginning position of the embedded feature sequence to form a feature embedding sequence containing the [CLS] tag. The feature embedding sequence containing the [CLS] label is input into the multi-head attention module, which captures the dependencies of different subspaces and generates an intermediate representation containing global dependency information. The intermediate representation containing global dependency information is fed to the feedforward neural network module, which performs a nonlinear transformation on the intermediate representation containing global dependency information to obtain a joint representation vector. The joint representation vector integrates the collaborative decision-making information of all modalities. The joint representation vector is input into the classification head module to obtain the risk probability of the target postoperative complications of the subject output by the classification head.

[0007] Optionally, the preoperative medical imaging data includes: preoperative abdominal CT images; The extraction of radiomics features based on the preoperative medical imaging data includes: Using a pre-trained tumor segmentation model, the primary gastric tumor in the preoperative abdominal CT image is segmented to obtain the tumor region; Body composition analysis was performed on the preoperative abdominal CT images to obtain the skeletal muscle region, subcutaneous fat region, and visceral fat region. The radiomics features are obtained based on the tumor region, the skeletal muscle region, the subcutaneous fat region, and the visceral fat region.

[0008] Optionally, obtaining the radiomics features based on the tumor region, the skeletal muscle region, the subcutaneous fat region, and the visceral fat region includes: Using the pre-built PyRadiomics library, radiomics features were extracted from the tumor region, the skeletal muscle region, the subcutaneous fat region, and the visceral fat region, respectively. The radiomics features included: first-order statistics, shape features, and various texture matrix features.

[0009] Optionally, before fusing the radiomics features with the clinical data to obtain a multimodal input feature set, the method further includes: The radiomics features were effect-corrected using a pre-defined ComBat algorithm to obtain the corrected radiomics features. Principal component analysis was used to verify the corrected radiomics features, and the verified corrected radiomics features were then standardized to obtain standardized radiomics features.

[0010] Optionally, the step of fusing the radiomics features with the clinical data to obtain a multimodal input feature set includes: The SelectKBest method based on mutual information was used to filter the standardized radiomics features to obtain the k most discriminative radiomics features. The k radiomics features are concatenated with the clinical data to obtain the multimodal input feature set.

[0011] Optionally, the initial model is trained based on preset training samples to obtain a multimodal deep learning radiometric model, including: Obtain the preset training dataset, which comes from multiple institutions and contains multiple training samples and label information for each training sample. The label information is used to indicate whether the target postoperative complication has occurred. Feature extraction is performed on each training sample in the training dataset to obtain a multimodal sample feature set; The multimodal sample feature set is input into the initial model, and the initial model outputs the prediction result. Based on the prediction results and the label information of each training sample, the loss result is calculated based on the target loss function, and the initial model is iteratively corrected according to the loss result until the prediction result of the initial model meets the preset conditions or the number of iterations reaches the preset number. The initial model at the end of the iteration is used as the multimodal deep learning radiometric model.

[0012] Optionally, the method further includes: The prediction results of the initial model are interpreted using a pre-built Kernel SHAP algorithm to obtain the Shapley value of each training sample, thereby quantifying the contribution of each training sample to the prediction results output by the initial model.

[0013] Secondly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform any of the method steps provided in the first aspect embodiment.

[0014] Secondly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs any of the methods described in the first aspect of the embodiments.

[0015] The beneficial effects of this application are: This application provides a method, electronic device, and storage medium for predicting postoperative complication risks. In this application, an initial model is constructed based on the Transformer architecture, and trained on a pre-set training dataset to obtain a multimodal deep learning radiology model. Then, features are extracted from the preoperative medical imaging data of the subject to obtain radiomics features. The clinical data and radiomics features of the subject are fused to obtain multimodal input features, which are then input into the multimodal deep learning radiology model. The model is processed sequentially by a multi-head attention module, a feedforward neural network module, and a classification head module to predict the probability of the subject developing the target postoperative complication. This achieves automated prediction from raw images to postoperative risk probability, reducing human intervention and improving clinical usability. It is also widely applicable in clinical practice and can reduce postoperative complication risk stratification. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a method for predicting postoperative complications provided in this application embodiment; Figure 2 A schematic diagram of the architecture of a multimodal deep learning radiographic model provided in an embodiment of this application; Figure 3 A flowchart illustrating another method for predicting postoperative complications provided in this application embodiment; Figure 4 This is a schematic diagram of the overall process of a postoperative complication risk prediction method provided in an embodiment of this application; Figure 5 This is a schematic diagram of the overall process of a postoperative complication risk prediction method provided in an embodiment of this application; Figure 6 A flowchart illustrating another method for predicting postoperative complications provided in this application embodiment; Figure 7A schematic diagram of the overall process of another postoperative complication risk prediction method provided in this application embodiment; Figure 8 A flowchart illustrating another method for predicting postoperative complications provided in this application embodiment; Figure 9 A flowchart illustrating another method for predicting postoperative complications provided in this application embodiment; Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0019] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0020] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0021] Optionally, refer to Figure 1 The diagram shown is a flowchart illustrating a postoperative complication risk prediction method provided in an embodiment of this application. The method is executed by an electronic device with data processing capabilities, such as a computer or server. Figure 1 As shown, the method includes: S101. Construct an initial model based on the Transformer architecture and train the initial model based on a preset training dataset to obtain a multimodal deep learning radiometric model.

[0022] Optionally, to improve the accuracy of the prediction results, it is proposed that an initial model can be built based on the Transformer architecture, and a large number of training samples, including preoperative imaging data, clinical data and their corresponding postoperative results (such as whether complications occurred), can be used to train the initial model iteratively multiple times, and finally train a "multimodal deep learning radiology model" that can complete classification or regression tasks.

[0023] S102. Obtain the preoperative medical imaging data and clinical data of the subject to be tested.

[0024] For example, the subject of the test can be a gastric cancer patient, so that the risk probability of the gastric cancer patient developing the target postoperative complication can be predicted based on a "multimodal deep learning radiology model".

[0025] Preoperative medical imaging data may include: two-dimensional CT images and three-dimensional MRI images of the abdomen of gastric cancer patients taken during surgery.

[0026] Clinical data can include structured variables such as age, sex, BMI, medical history, laboratory test results (e.g., white blood cell count, liver and kidney function), ASA score, and type of surgery for gastric cancer patients.

[0027] In one feasible approach, for example, the patient's case data, namely the patient's preoperative medical imaging data and clinical data, can be read from the patient database of a medical institution.

[0028] S103. Extract radiomics features based on preoperative medical imaging data.

[0029] Preoperative medical imaging data may include: two-dimensional CT images and three-dimensional MRI images of the abdomen of gastric cancer patients taken during surgery.

[0030] In one feasible approach, preoperative medical imaging data can be processed to extract radiomics features, such as morphological features, first-order statistical features, texture features, and wavelet features.

[0031] The morphological features include tumor volume, surface area, sphericity, etc.; first-order statistical features include gray mean, variance, skewness, kurtosis, etc.; texture features include GLCM (gray-level co-occurrence matrix), GLRLM (gray-level run length matrix), Gabor filter response, etc.; wavelet features include multi-scale features extracted after wavelet transform.

[0032] S104. Integrate radiomics features with clinical data to obtain multimodal input features.

[0033] Optionally, in order to improve the accuracy of the prediction results, it is also necessary to fuse radiomics features with clinical data. For example, a feature splicing method can be used to combine radiomics features with clinical data into a multimodal input feature.

[0034] S105. Input the multimodal input features into the multimodal deep learning radiology model. The multi-head attention module, feedforward neural network module and classification head module in the multimodal deep learning radiology model process the features sequentially to obtain the risk probability of the subject developing the target postoperative complications.

[0035] The risk probability of postoperative complications is a scalar value within the interval [0,1].

[0036] Optionally, such as Figure 2 As shown, the multimodal input features are fed into the multimodal deep learning radiology model trained above. The model is then processed sequentially by two multi-head attention modules, a feedforward neural network module, and a classification head module. Specifically, the multi-head attention modules automatically identify potential correlation patterns between radiomics features and clinical data, improving the efficiency of multimodal data fusion. The feedforward neural network module performs nonlinear mapping on the attention-weighted features, enhancing the model's ability to fit complex pathological mechanisms and significantly improving prediction accuracy. Finally, the classification head module outputs the risk probability of the target postoperative complications, allowing clinicians to set risk grading thresholds according to actual needs, enhancing the system's practicality and flexibility. Therefore, in this embodiment, the sequential collaborative work of these modules forms an end-to-end risk prediction pathway, achieving automated inference from raw images to postoperative risk probabilities, reducing human intervention while improving clinical usability.

[0037] Therefore, the postoperative complication risk prediction method proposed in this application can accurately predict the probability of the target postoperative complication occurring in the subject, which is widely applicable in clinical practice and can reduce the risk stratification of postoperative complications.

[0038] In summary, this application provides a method for predicting postoperative complication risks. In this application, an initial model is constructed based on the Transformer architecture, and the initial model is trained using a pre-set training dataset to obtain a multimodal deep learning radiology model. Then, features are extracted from the preoperative medical imaging data of the subject to obtain radiomics features. The clinical data and radiomics features of the subject are fused to obtain multimodal input features, which are then input into the multimodal deep learning radiology model. The model is processed sequentially by a multi-head attention module, a feedforward neural network module, and a classification head module to predict the probability of the subject developing the target postoperative complication. This achieves automated prediction from raw images to postoperative risk probability, reducing human intervention and improving clinical usability. It is also widely applicable in clinical practice and can reduce the risk stratification of postoperative complications.

[0039] Optionally, refer to Figure 3 As shown, in step S105 above, the multimodal input features are input into a pre-trained multimodal deep learning radial model, and processed sequentially by the multi-head attention module, feedforward neural network module, and classification head module in the multimodal deep learning radial model, including: S201. Perform feature embedding on the multimodal input features to convert each feature into a high-dimensional vector representation, obtain the embedded feature sequence, and add a learnable [CLS] label at the beginning position of the embedded feature sequence to form a feature embedding sequence containing the [CLS] label.

[0040] In one feasible approach, as referenced Figure 4 As shown, feature embedding is performed on the multimodal input features, assuming the standardized input is... To convert each feature into a d-dimensional dimension The vector representation is used, and a learnable [CLS] tag is added to the beginning position of the embedded feature sequence to form a feature embedding sequence containing the [CLS] tag. .

[0041] S202. Input the feature embedding sequence containing the [CLS] label into the multi-head attention module, which captures the dependencies of different subspaces and generates an intermediate representation containing global dependency information.

[0042] S203. Input the intermediate representation containing global dependency information into the feedforward neural network module. The feedforward neural network module performs a nonlinear transformation on the intermediate representation containing global dependency information to obtain the joint representation vector.

[0043] Among them, the joint representation vector integrates collaborative decision-making information from all modalities.

[0044] S204. Input the joint representation vector into the classification head module to obtain the risk probability of the target postoperative complications of the test object output by the classification head.

[0045] Optionally, continue to refer to Figure 4 As shown, the feature embedding sequence containing the [CLS] label is input into the L-layer encoder, which includes two multi-head self-attention modules and a feed-forward neural network module. The feed-forward neural network module has residual connections and normalization layers. The multi-head attention modules calculate the correlation score between the [CLS] label and all feature labels to capture the dependencies between different subspaces and generate an intermediate representation containing global dependency information.

[0046] Then, the intermediate representation containing global dependency information is input into the feedforward neural network module, which performs nonlinear transformation and feature extraction on the intermediate representation containing global dependency information to obtain the joint representation vector. ; and the joint representation vector (i.e., the vector containing the final [CLS] tag) The input is fed into the classification head module to obtain the risk probability of the target postoperative complications of the subject output by the classification head, as shown in the following formula (1): (1) in, and These are the learnable parameters of the output layer, while the original feature vector x is not used in the execution of the learning model.

[0047] Optionally, preoperative medical imaging data includes: preoperative abdominal CT images; refer to Figure 5 As shown, step S103 above includes: S301. Using a pre-trained tumor segmentation model, the primary gastric tumor in the preoperative abdominal CT image is segmented to obtain the tumor region.

[0048] For example, the tumor segmentation model can be trained using a deep learning-based nnU-Net model. The deep learning-based nnU-Net model is trained using 200 contrast-enhanced CT scan images and their corresponding manually annotated tumor masks, including a training set (n=160) and a test set (n=40). The specific training process includes: (1) Resample the CT scan image and adjust its size to 1.25mm x 0.93mm x 0.93mm. 3 ; (2) Normalize the truncated intensity using the 0.5-99.5 percentile range, and then perform Z-standardization; (3) The training configuration is a six-level 3D nnU-Net framework with channel depths ranging from 32 to 320, and is achieved by combining Dice and cross-entropy loss functions (Adam optimizer, learning rate = 3 × 10). -4 The batch size was optimized to 2, and the training process took more than 1000 epochs.

[0049] To improve the model's generalization ability, data augmentation strategies and five-fold cross-validation were introduced. The model was implemented using PyTorch on the Windows operating system, achieving a Dice coefficient of 0.890. The trained nnU-Net model was subsequently applied to segment primary tumors.

[0050] Optionally, after obtaining the preoperative abdominal CT image, it is necessary to segment the preoperative abdominal CT image. For example, the nnU-Net model based on deep learning can be used to segment the primary gastric tumor in the preoperative abdominal CT image to obtain the tumor region, so that radiological features can be extracted from the tumor region in the future.

[0051] S302. Perform body composition analysis on preoperative abdominal CT images to obtain skeletal muscle region, subcutaneous fat region and visceral fat region.

[0052] S303. Radiomic characteristics are obtained based on tumor region, skeletal muscle region, subcutaneous fat region, and visceral fat region.

[0053] Optionally, in body composition analysis, 3D Slicer software (version 5.6.1) can be used to segment skeletal muscle and subcutaneous fat in preoperative abdominal CT images. On axial non-enhanced CT images at the level of the third lumbar vertebra (L3), skeletal muscle, subcutaneous fat, and visceral fat regions can be sliced ​​to obtain skeletal muscle region, subcutaneous fat region, and visceral fat region.

[0054] The muscle area (SMA), subcutaneous fat area (SFA), and visceral fat area (VFA) were semi-automatically segmented using standard soft tissue window settings (window width of 350 HU and window layer of 40-HU). Skeletal muscles included the rectus abdominis, abdominal wall muscles, psoas major, quadratus lumborum, and erector spinae, using a predefined HU range from -29 to 150. Adipose tissue segmentation used a HU value range from -190 to -30.

[0055] Then, radiomics features were obtained based on the tumor region, skeletal muscle region, subcutaneous fat region, and visceral fat region, thus realizing the extraction of radiomics features.

[0056] Optionally, step S303 above includes: Using the pre-built PyRadiomics library, radiomics features were extracted from the tumor region, the skeletal muscle region, the subcutaneous fat region, and the visceral fat region, respectively. The radiomics features included: first-order statistics, shape features, and various texture matrix features.

[0057] In one feasible approach, radiomics features were extracted from segmented CT regions corresponding to primary tumors, SMA, SFA, and VFA. This process utilized the PyRadiomics library (version 3.1.0) within the built-in 3D Slicer platform. A standardized configuration was applied for the CT radiomics features, with key parameters including: resampling voxel size of [1.0, 1.0, 1.0] mm, bin width of 25, interpolator set to sitkBSpline, and symmetric GLCM enabled. 1130 radiomics features were extracted from each region of interest (ROI), including: first-order statistics, shape features, and various texture matrix features, including: gray-level co-occurrence matrix (GLCM), gray-level run-length matrix (GLRLM), gray-level size region matrix (GLSZM), gray-level dependency matrix (GLDM), and neighboring gray-level difference matrix (NGTDM).

[0058] Optionally, refer to Figure 6 As shown, before step S104 above, the procedure also includes: S401. Using the pre-defined ComBat algorithm, effect correction is performed on the radiomics features to obtain the corrected radiomics features.

[0059] S402. Principal component analysis was used to verify the corrected radiomics features, and the verified corrected radiomics features were standardized to obtain standardized radiomics features.

[0060] Optionally, before feature fusion of radiomics features with the clinical data, a pre-defined ComBat coordination algorithm can be used to perform effect correction on the radiomics features to obtain corrected radiomics features. To evaluate the effectiveness of the ComBat coordination algorithm, principal component analysis is also needed to validate the corrected radiomics features; see reference... Figure 7 As shown, the validated corrected radiomics features are standardized, such as by using Z-standardization, to obtain the standardized radiomics features.

[0061] The Z-normalization process is as follows: To ensure comparability between features and enhance the stability of the model during prediction, the validated corrected radiomics features are first subjected to Z-score standardization. Clinical data and radiomics features may have different units and scales, which could introduce bias. Z-standardization can be used to eliminate this bias and make the features comparable in subsequent stages. Specifically, for the j-th feature in the training set, the Z-score is calculated as shown in formula (2): (2) in, The original values ​​of the features, The mean, The standard deviation is denoted as .

[0062] After standardization, all features will have zero mean and unit mean, making them suitable for training deep learning models.

[0063] Optionally, refer to Figure 8 As shown, step S104 above includes: S501. The SelectKBest method based on mutual information is used to screen the standardized radiomics features and obtain the k most discriminative radiomics features.

[0064] S502. Concatenate the k radiomics features with the clinical data to obtain a multimodal input feature set.

[0065] In one feasible approach, after data standardization, the next step is feature selection. Given that the radiomics dataset contains 4520 features, many of which may be redundant or irrelevant, it is necessary to reduce dimensionality and improve model efficiency. Therefore, the SelectKBest method based on mutual information is used to select the k most discriminative radiomics features.

[0066] The mutual information I(X,Y) between two variables X and Y is used to measure their dependence, and the calculation method is shown in the following formula (3): (3) in, and They represent and entropy, yes and The joint entropy.

[0067] The top k features with the highest mutual information are selected and then combined with clinical variables (such as CCI and surgical scores) to form the final 2+k dimensional feature input set. This step not only reduces the dimensionality of the data but also enhances the model's discriminative ability.

[0068] Optionally, refer to Figure 9 As shown, step S101 above includes: S601. Obtain the preset training dataset.

[0069] The training dataset comes from multiple institutions and contains multiple training samples and label information for each training sample. The label information is used to indicate whether the target postoperative complication has occurred.

[0070] S602. Extract features from each training sample in the training dataset to obtain a multimodal sample feature set.

[0071] S603. Input the multimodal sample feature set into the initial model, and the initial model outputs the prediction result; S604. Based on the prediction results and the label information of each training sample, calculate the loss result based on the target loss function, and iteratively correct the initial model according to the loss result until the prediction result of the initial model meets the preset conditions or the number of iterations reaches the preset number. The initial model at the end of the iteration is used as the multimodal deep learning radiometric model.

[0072] Optionally, considering the excellent performance of the Transformer architecture in handling high loads, the Transformer architecture is chosen to build the initial model, and five-fold cross-validation combined with random search is used to optimize key hyperparameters, including the number of selected features k, the Transformer hidden dimension dim, the learning rate Ir, and the focusing parameter v.

[0073] The multimodal sample feature set is input into the initial model. The Transformer's self-attention mechanism captures the complex interactions between features by processing each continuous feature as a "label" and projecting it into a d-dimensional space. Specifically, the feature labels and the feature sequence with [CLS] labels are input into the Transformer model as shown in the following formula (4): (4) To address the class imbalance problem, a weighted random sampler is used during training to ensure that each class is sufficiently representative, and the output prediction results are processed by the sigmoid function.

[0074] The model is optimized using the Focal Loss function, which focuses on difficult-to-classify models by reducing the loss weights on well-classified models. The loss function is calculated as shown in formula (5): (5) in, This represents the predicted probability of the true class. γ is the balancing factor, and it is the focusing parameter that controls the weighting of easily classified samples. The optimal hyperparameter is then selected through 20 randomized trials. .

[0075] Prediction and Evaluation: Using Optimal Parameters After the model is trained, the probability of each sample is generated by the FT-Transformer through [CLS] representation, rather than using a linear model directly on the original feature data.

[0076] Let the standardized input be Each continuous feature is labeled and transformed into a dimensional label. And by adding the [CLS] flag to the prefix, we get a final flag: .

[0077] Then, based on the prediction results and the label information of each training sample, the loss result is calculated based on the target loss function, and the initial model is iteratively corrected according to the loss result until the prediction result of the initial model meets the preset conditions or the number of iterations reaches the preset number. The initial model at the end of the iteration is used as the multimodal deep learning radiometric model.

[0078] Optionally, the method further includes: A pre-built Kernel SHAP algorithm is used to interpret the prediction results of the initial model and obtain the Shapley value of each training sample to quantify the contribution of each training sample to the prediction results of the initial model output.

[0079] Optionally, to enhance the interpretability of the model, it is also necessary to interpret the prediction results of the initial model using a pre-built Kernel SHAP algorithm, obtaining the Shapley value of each training sample to quantify the contribution of each training sample to the prediction results output by the initial model, as follows: The Kernel SHAP algorithm is used to approximate the Shapley value of the black-box model. , representing the predicted probability after processing by the sigmoid function.

[0080] In instance x, when At that time, define binary alliances. Indicates which features are retained. Sampling is performed by keeping the current feature unchanged while replacing other features. Additionally, samples are drawn from the background set B (using 100 random values ​​to construct an inference sample). The Shapley value of feature i is expressed as shown in the following formula (6): (6) By solving the weighted linear equation, the Kernel SHAP algorithm can effectively approximate these values. Based on regression analysis using sampling consortium (sample size nsamples=200): when nsamples=200, the calculated result of the formula is as follows: ; in, As a baseline, the solution satisfies the local accuracy condition. It is additive and consistent, making it suitable for explaining complex models such as those described above.

[0081] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device can be integrated into a terminal device or a chip of a terminal device, and the electronic device can be a computing device with data processing functions.

[0082] The electronic device includes: processor 1001 and memory 1002.

[0083] The memory 1002 is used to store programs, and the processor 1001 calls the programs stored in the memory 1002 to execute the above method embodiments. The specific implementation and technical effects are similar, and will not be described again here.

[0084] Optionally, this application also provides a program product, such as a computer-readable storage medium, including a program that, when executed by a processor, performs the above-described method embodiments.

[0085] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0086] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0087] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0088] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for predicting postoperative complications, characterized in that, The method includes: An initial model is built based on the Transformer architecture, and the initial model is trained based on a preset training dataset to obtain a multimodal deep learning radiometric model. Acquire preoperative medical imaging and clinical data of the subjects to be tested; Based on the preoperative medical imaging data, radiomics features were extracted; The radiomics features are fused with the clinical data to obtain multimodal input features; The multimodal input features are input into the multimodal deep learning radiology model, and are processed sequentially by the multi-head attention module, feedforward neural network module and classification head module in the multimodal deep learning radiology model to obtain the risk probability of the subject developing the target postoperative complications.

2. The method according to claim 1, characterized in that, The process of inputting the multimodal input features into a pre-trained multimodal deep learning radiometric model, where they are processed sequentially by the multi-head attention module, feedforward neural network module, and classification head module within the multimodal deep learning radiometric model, includes: The multimodal input feature row is embedded to convert each feature into a high-dimensional vector representation, resulting in an embedded feature sequence. A learnable [CLS] tag is added to the beginning position of the embedded feature sequence to form a feature embedding sequence containing the [CLS] tag. The feature embedding sequence containing the [CLS] label is input into the multi-head attention module, which captures the dependencies of different subspaces and generates an intermediate representation containing global dependency information. The intermediate representation containing global dependency information is fed to the feedforward neural network module, which performs a nonlinear transformation on the intermediate representation containing global dependency information to obtain a joint representation vector. The joint representation vector integrates the collaborative decision-making information of all modalities. The joint representation vector is input into the classification head module to obtain the risk probability of the target postoperative complications of the subject output by the classification head.

3. The method according to claim 1, characterized in that, The preoperative medical imaging data includes: preoperative abdominal CT images; The extraction of radiomics features based on the preoperative medical imaging data includes: Using a pre-trained tumor segmentation model, the primary gastric tumor in the preoperative abdominal CT image was segmented to obtain the tumor region. Body composition analysis was performed on the preoperative abdominal CT images to obtain the skeletal muscle region, subcutaneous fat region, and visceral fat region. The radiomics features are obtained based on the tumor region, the skeletal muscle region, the subcutaneous fat region, and the visceral fat region.

4. The method according to claim 3, characterized in that, The radiomics features obtained based on the tumor region, the skeletal muscle region, the subcutaneous fat region, and the visceral fat region include: Using the pre-built PyRadiomics library, radiomics features were extracted from the tumor region, the skeletal muscle region, the subcutaneous fat region, and the visceral fat region, respectively. The radiomics features included: first-order statistics, shape features, and various texture matrix features.

5. The method according to claim 1, characterized in that, Before fusing the radiomics features with the clinical data to obtain a multimodal input feature set, the method further includes: The radiomics features were effect-corrected using a pre-defined ComBat algorithm to obtain the corrected radiomics features. Principal component analysis was used to verify the corrected radiomics features, and the verified corrected radiomics features were then standardized to obtain standardized radiomics features.

6. The method according to claim 5, characterized in that, The step of fusing the radiomics features with the clinical data to obtain a multimodal input feature set includes: The SelectKBest method based on mutual information was used to filter the standardized radiomics features to obtain the k most discriminative radiomics features. The k radiomics features are concatenated with the clinical data to obtain the multimodal input feature set.

7. The method according to claim 1, characterized in that, The initial model is trained based on preset training samples to obtain a multimodal deep learning radiometric model, including: Obtain the preset training dataset, which comes from multiple institutions and contains multiple training samples and label information for each training sample. The label information is used to indicate whether the target postoperative complication has occurred. Feature extraction is performed on each training sample in the training dataset to obtain a multimodal sample feature set; The multimodal sample feature set is input into the initial model, and the initial model outputs the prediction result. Based on the prediction results and the label information of each training sample, the loss result is calculated based on the target loss function, and the initial model is iteratively corrected according to the loss result until the prediction result of the initial model meets the preset conditions or the number of iterations reaches the preset number. The initial model at the end of the iteration is used as the multimodal deep learning radiometric model.

8. The method according to claim 7, characterized in that, The method further includes: The prediction results of the initial model are interpreted using a pre-built Kernel SHAP algorithm to obtain the Shapley value of each training sample, thereby quantifying the contribution of each training sample to the prediction results output by the initial model.

9. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is in operation, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which is executed by a processor to perform the method as described in any one of claims 1-8.

Images