Method for predicting recurrence of ovarian cancer based on multi-modal fusion deep learning model

Abstract

The invention discloses a method for predicting ovarian cancer recurrence based on a multi-modal fusion deep learning model, and relates to the technical field of deep learning, and the method comprises the following steps: obtaining multi-modal data of a to-be-predicted object from data tables of a plurality of databases; the multi-modal data is medical information related to ovarian cancer detection; carrying out preprocessing on the multi-modal data; performing local feature extraction on the preprocessed multi-modal data to obtain multi-modal local features; performing global feature extraction based on the multi-modal local features to obtain multi-modal global features; carrying out fusion processing on the multi-modal local features and the multi-modal global features to obtain multi-modal fusion features; and inputting the multi-modal fusion features into a trained ovarian cancer recurrence prediction model, and outputting the ovarian cancer recurrence probability of the to-be-predicted object. According to the method, the local texture features and the global semantic features are subjected to combined modeling processing, and the prediction accuracy of the ovarian cancer recurrence risk is improved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and more specifically to a method for predicting ovarian cancer recurrence based on a multimodal fusion deep learning model. Background Art

[0002] Ovarian cancer is a common gynecological malignancy. Although its incidence is relatively low, its mortality rate ranks first among gynecological malignancies. The recurrence rate of ovarian cancer is also high. Once recurring, the patient's prognosis is poor. Therefore, how to assess the risk of recurrence of ovarian cancer has become the focus of clinical research. Traditional methods for assessing the risk of ovarian cancer recurrence mainly rely on a single type of biomarker, such as serum CA125 levels, or imaging examinations to assess the size and location of the tumor. However, these methods often do not provide enough information to accurately assess the risk of recurrence of ovarian cancer. In addition, due to the large heterogeneity of ovarian cancer, it is difficult for a single indicator to fully reflect the overall condition of the patient, which limits the accuracy of the assessment, and a single indicator cannot achieve personalized prediction of the risk of ovarian cancer recurrence.

[0003] In recent years, with the development of big data technology and artificial intelligence, multimodal data fusion technology has gradually come into people's attention. Multimodal data fusion refers to the integration of data from different sources and in different forms to obtain a more comprehensive situation assessment by integrating this information. However, multimodal data fusion has a certain complexity, which makes the feature extraction task extremely difficult and challenging. Traditional target detection models are not good at feature refinement processing, and cannot achieve joint modeling processing of local texture features and global semantic features. The feature extraction ability is weak, which affects the final prediction results.

[0004] Therefore, how to improve the prediction accuracy of ovarian cancer recurrence risk and achieve personalized prediction is an urgent problem that technicians in this field need to solve. Summary of the invention

[0005] In view of the above problems, the present invention provides a method for predicting ovarian cancer recurrence based on a multimodal fusion deep learning model to at least solve some of the technical problems mentioned in the above background technology.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] In one aspect, the present invention provides a method for predicting ovarian cancer recurrence based on a multimodal fusion deep learning model, comprising the following steps:

[0008] S1. Acquire multimodal data of an object to be predicted from data tables of multiple databases; the multimodal data is relevant medical information for detecting ovarian cancer;

[0009] S2. preprocessing the multimodal data;

[0010] S3, extracting local features from the preprocessed multimodal data to obtain multimodal local features;

[0011] S4, extracting global features based on the multimodal local features to obtain multimodal global features;

[0012] S5, fusing the multimodal local features with the multimodal global features to obtain multimodal fusion features;

[0013] S6. Input the multimodal fusion features into a trained ovarian cancer recurrence prediction model, and output the ovarian cancer recurrence probability of the object to be predicted.

[0014] Furthermore, the multimodal data includes imaging data, genomic data, pathological data and clinical information.

[0015] Furthermore, the step S2 specifically includes:

[0016] (1) Preprocessing the imaging data:

[0017] Performing normalization, resizing, data enhancement and filtering denoising on the imaging data; and segmenting the region of interest in the denoised imaging data;

[0018] (2) Preprocessing the genomics data:

[0019] Removing sequencing reads whose quality is lower than a preset value in the genomics data;

[0020] (3) Preprocessing the pathological data:

[0021] Remove duplicate data and outliers from the pathology data, and fill in missing data;

[0022] Convert categorical variables in pathology data into numerical representation;

[0023] (4) Preprocessing the clinical information:

[0024] Remove duplicate data and outliers in the clinical information and fill in missing data;

[0025] performing keyword extraction on text in the clinical information;

[0026] The identity information in the clinical information is desensitized.

[0027] Furthermore, the step S3 specifically includes:

[0028] (1) Extract local features of the preprocessed imaging data through a convolutional neural network to obtain local features of the image;

[0029] (2) Extract local features from the preprocessed genomics data using a long short-term memory network to obtain local gene features;

[0030] (3) extracting local features of the preprocessed pathological data through a residual network to obtain pathological local features;

[0031] (4) Extract local features from the preprocessed clinical information using a multi-layer perceptron to obtain clinical local features;

[0032] (5) The image local features, the gene local features, the pathological local features and the clinical local features are fused to obtain multimodal local features.

[0033] Furthermore, step S4 is implemented by using a maximum pooling operation.

[0034] Furthermore, in step S5,

[0035] Before fusing the multimodal local features with the multimodal global features, the multimodal local features and the multimodal global features are firstly standardized.

[0036] Furthermore, the training process of the ovarian cancer recurrence prediction model includes:

[0037] P1. Obtain multimodal training data of ovarian cancer patients and corresponding recurrence result labels;

[0038] P2. Obtain corresponding multimodal training fusion features based on the multimodal training data of ovarian cancer patients;

[0039] P3. Determine the corresponding initial weight value for the multimodal training fusion feature;

[0040] P4. Perform weighted summation on the multimodal training fusion features and corresponding weight values;

[0041] P5. During the forward propagation process, the weighted summation result is converted into the probability value of recurrence of ovarian cancer patients through the sigmoid function;

[0042] P6. Calculate the loss value according to the probability value and the recurrence result label;

[0043] P7. During the back propagation process, the weight value is updated by the loss value;

[0044] P8. Repeat the above steps P4-P8 based on the updated weight values ​​until the regression model converges, which serves as the trained ovarian cancer recurrence prediction model.

[0045] Furthermore, it also includes:

[0046] S7. Provide guidance and suggestions based on the probability of recurrence of ovarian cancer.

[0047] On the other hand, the present invention provides a prediction system for ovarian cancer recurrence based on a multimodal fusion deep learning model, applying the above method, the prediction system includes an acquisition module, a preprocessing module, a local feature extraction module, a global feature extraction module, a fusion module and a prediction module;

[0048] The acquisition module is used to acquire multimodal data of the object to be predicted from data tables of multiple databases; the multimodal data is relevant medical information for detecting ovarian cancer;

[0049] The preprocessing module is used to preprocess the multimodal data;

[0050] The local feature extraction module is used to extract local features from the preprocessed multimodal data to obtain multimodal local features;

[0051] The global feature extraction module is used to extract global features based on the multimodal local features to obtain multimodal global features;

[0052] The fusion module is used to fuse the multimodal local features with the multimodal global features to obtain multimodal fusion features;

[0053] The prediction module is used to input the multimodal fusion features into the trained ovarian cancer recurrence prediction model, and output the ovarian cancer recurrence probability of the object to be predicted.

[0054] Further, the preprocessing module includes an imaging data preprocessing submodule, a genomics data preprocessing submodule, a pathology data preprocessing submodule and a clinical information preprocessing submodule;

[0055] The imaging data preprocessing submodule is used to perform normalization, size adjustment, data enhancement and filtering denoising on the imaging data; and segment the region of interest in the denoised imaging data;

[0056] The genomics data preprocessing submodule is used to remove sequencing reads with quality lower than a preset value in the genomics data;

[0057] The pathology data preprocessing submodule is used to remove duplicate data and abnormal values ​​in the pathology data and fill in missing data; convert categorical variables in the pathology data into numerical representation;

[0058] The clinical information preprocessing submodule is used to remove duplicate data and abnormal values ​​in the clinical information and fill in missing data; extract keywords from the text in the clinical information; and desensitize the identity information in the clinical information.

[0059] Furthermore, the local feature extraction module includes an image local feature extraction submodule, a gene local feature extraction submodule, a pathology local feature extraction submodule, a clinical local feature extraction submodule and a local feature fusion submodule;

[0060] The image local feature extraction submodule is used to extract local features from the preprocessed imaging data through a convolutional neural network to obtain image local features;

[0061] The gene local feature extraction submodule is used to extract local features from the preprocessed genomics data through a long short-term memory network to obtain gene local features;

[0062] The pathological local feature extraction submodule is used to extract local features from the preprocessed pathological data through a residual network to obtain pathological local features;

[0063] The clinical local feature extraction submodule is used to extract local features from the preprocessed clinical information through a multi-layer perceptron to obtain clinical local features;

[0064] The local feature fusion submodule is used to fuse the image local features, the gene local features, the pathological local features and the clinical local features to obtain multimodal local features.

[0065] Further, a guidance module is included;

[0066] The guidance module is used to provide guidance suggestions based on the probability of recurrence of ovarian cancer.

[0067] Furthermore, the ovarian cancer recurrence prediction model is trained by a model training module; the model training module includes a training data acquisition submodule, a training feature acquisition submodule, a weight value determination submodule, a weighted summation submodule, a probability value calculation submodule, a loss value calculation submodule and an update submodule;

[0068] The training data acquisition submodule is used to acquire multimodal training data of ovarian cancer patients and corresponding recurrence result labels;

[0069] The training feature acquisition submodule is used to acquire corresponding multimodal training fusion features according to the multimodal training data of ovarian cancer patients;

[0070] The weight value determination submodule is used to determine the corresponding initial weight value for the multimodal training fusion feature;

[0071] The weighted summation submodule is used to perform weighted summation on the multimodal training fusion features and the corresponding weight values;

[0072] The probability value calculation submodule is used to convert the weighted sum result into the probability value of recurrence of ovarian cancer patients through the sigmoid function during the forward propagation process;

[0073] The loss value calculation submodule is used to calculate the loss value according to the probability value and the recurrence result label;

[0074] The updating submodule is used to update the weight value according to the loss value during the backward propagation process.

[0075] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method for predicting ovarian cancer recurrence based on a multimodal fusion deep learning model, which has the following beneficial effects:

[0076] The present invention combines local texture features and global semantic features for modeling, thereby improving the prediction accuracy of ovarian cancer recurrence risk.

[0077] The present invention predicts the recurrence probability based on each patient's imaging data, genomic data, pathological data and clinical information multimodal data, which can achieve personalized prediction and further improve the prediction accuracy.

[0078] The present invention uses convolutional neural networks, long short-term memory networks, residual networks and multi-layer perceptrons to extract local features of different data respectively, which can improve the feature extraction accuracy and help improve the subsequent prediction accuracy.

[0079] Other features and advantages of the present invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the present invention.

[0080] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0082] Figure 1 A schematic flow chart of a method for predicting ovarian cancer recurrence based on a multimodal fusion deep learning model provided in an embodiment of the present invention.

[0083] Figure 2 A schematic diagram of the framework of a system for predicting ovarian cancer recurrence based on a multimodal fusion deep learning model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0084] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0085] Embodiment 1:

[0086] The embodiment of the present invention discloses a method for predicting ovarian cancer recurrence based on a multimodal fusion deep learning model, see Figure 1 As shown, the following steps are included:

[0087] S1. Acquire multimodal data of an object to be predicted from data tables of multiple databases; the multimodal data is relevant medical information for detecting ovarian cancer;

[0088] S2, preprocessing multimodal data;

[0089] S3, extracting local features from the preprocessed multimodal data to obtain multimodal local features;

[0090] S4, extracting global features based on the multimodal local features to obtain multimodal global features;

[0091] S5, fusing the multimodal local features with the multimodal global features to obtain multimodal fusion features;

[0092] S6. Input the multimodal fusion features into a trained ovarian cancer recurrence prediction model, and output the ovarian cancer recurrence probability of the object to be predicted.

[0093] S7. Provide guidance and recommendations based on the probability of ovarian cancer recurrence.

[0094] This method can effectively improve the prediction accuracy of ovarian cancer recurrence risk, which helps patients to receive timely treatment; and the ovarian cancer recurrence prediction model in the present invention can obtain the ovarian cancer recurrence probability, and the specific recurrence probability value can provide a strong reference for the patient's subsequent diagnosis.

[0095] Next, each of the above steps will be described in detail.

[0096] In the above step S1, multimodal data of the object to be predicted is obtained from data tables of multiple databases; multimodal data fusion refers to the integration of data from different sources and in different forms, which specifically includes the following in the embodiment of the present invention:

[0097] (1) Imaging data: such as CT scans, MRI images, and ultrasound images, which help doctors observe the location, size, and shape of the tumor;

[0098] (2) Genomic data: such as gene expression profiles and mutation information, used to understand gene mutations or changes in expression levels related to cancer;

[0099] (3) Pathological data, such as cell structure and morphological features in tissue sections, used to determine the type and grade of the tumor;

[0100] (4) Clinical information: personal information and treatment history of the patient, such as age, gender, surgical history, chemotherapy records, etc.;

[0101] Integrating these data helps capture patient characteristics from multiple perspectives and ensures improved predictive accuracy of the ovarian cancer recurrence prediction model.

[0102] In the above step S2, the multimodal data obtained above is preprocessed; specifically, it includes:

[0103] (1) Preprocessing of imaging data:

[0104] Normalize, resize, enhance and filter the imaging data to reduce image noise; and segment the region of interest in the denoised imaging data;

[0105] Among them, normalizing the grayscale value can eliminate the impact of differences in imaging equipment; adjusting the image size to be consistent is helpful for subsequent feature extraction; enhancing the imaging data by rotating, flipping and scaling can increase sample diversity and improve model robustness; image noise can be reduced by line filtering denoising; and by segmenting the region of interest in the denoised imaging data, background interference can be reduced and the focus can be placed on key areas.

[0106] (2) Preprocessing of genomics data:

[0107] Remove sequencing reads with quality lower than the preset value in genomic data to ensure data accuracy; map sequencing reads to the reference genome to determine the location of sequencing reads in the genome;

[0108] (3) Preprocessing of pathological data:

[0109] ① Remove duplicate data and outliers in pathology data, and fill in missing data;

[0110] ②Convert categorical variables in pathology data into numerical representations through label encoding, such as gender, tumor grade, etc.;

[0111] (4) Preprocessing of clinical information:

[0112] ① Remove duplicate data and outliers in clinical information and fill in missing data;

[0113] ② Extract keywords from the text in clinical information to facilitate subsequent feature extraction;

[0114] ③ Desensitize the identity information in clinical information to protect patient privacy.

[0115] In the above step S3, local feature extraction is performed on the preprocessed multimodal data to obtain multimodal local features; specifically, the process includes:

[0116] (1) The local features of the preprocessed imaging data are extracted through a convolutional neural network (CNN) to obtain the local features of the image; it is expressed as:

[0117] f img =CNN(I)

[0118] Among them, f img Represents the local features of the image; CNN represents convolutional neural network; I represents the input image;

[0119] (2) Extract local features of the preprocessed genomic data through the long short-term memory network to obtain local gene features; expressed as:

[0120] f gene =RNN(G)

[0121] Among them, f gene Represents the local features of genes; RNN represents the long short-term memory network; G represents the input gene sequence;

[0122] (3) Extract local features of the preprocessed pathological data through the residual network to obtain pathological local features; expressed as:

[0123] fpath =ResNet(P)

[0124] Among them, f path Represents pathological local features; ResNet represents residual network; P represents input pathological image;

[0125] (4) Extract local features of the preprocessed clinical information through a multi-layer perceptron to obtain clinical local features; expressed as:

[0126] f clin =MLP(C)

[0127] Among them, f clin represents clinical local features; MLP represents multi-layer perceptron; C represents clinical data;

[0128] (5) The image local features, gene local features, pathological local features and clinical local features are fused to obtain multimodal local features; expressed as:

[0129] f multi =[f img ;f gene ;f path ;f clin ]

[0130] Among them, f multi represents the fused multi-modal local features; [;] represents the concatenation of feature vectors;

[0131] The above local feature extraction constitutes the basic part of the entire multimodal fusion deep learning model, in which each step aims to capture the key information of ovarian cancer recurrence from different perspectives. In this way, multi-source data can be effectively used to improve the accuracy and robustness of prediction.

[0132] In the above step S4, in order to better understand the overall situation of ovarian cancer recurrence, global feature extraction is performed based on the multimodal local features obtained above to obtain multimodal global features; this step can be implemented by using a maximum pooling operation, which is expressed as:

[0133] f global =MaxPool(f multi )

[0134] Among them, f global Represents multimodal global features; MaxPool represents the maximum pooling operation.

[0135] In the above step S5, the multimodal local features are fused with the multimodal global features to obtain multimodal fused features. Since the multimodal local features and the multimodal global features may have different scales and distributions, they need to be standardized before fusion. The fusion process is expressed as:

[0136] f fusion =[f multi ;f global ]

[0137] Among them, f fusion represents multimodal fusion features; f multi represents multimodal local features; f global Represents multimodal global features; [;] represents the concatenation of feature vectors;

[0138] In the above step S6, the multimodal fusion features are input into the trained ovarian cancer recurrence prediction model to output the ovarian cancer recurrence probability of the target ovarian cancer patient;

[0139] Among them, the training process of the ovarian cancer recurrence prediction model includes:

[0140] P1. Obtain multimodal training data of ovarian cancer patients and corresponding recurrence result labels; the multimodal training data includes imaging data, genomic data, pathological data and clinical information; the recurrence result label represents whether the disease recurred within one year (0 = no recurrence, 1 = recurrence)

[0141] P2. Obtain corresponding multimodal training fusion features based on the multimodal training data of ovarian cancer patients; the specific acquisition steps can refer to the above steps S3-S5;

[0142] P3. Determine the corresponding initial weight value for the multimodal training fusion features;

[0143] P4. Perform weighted summation on the multimodal training fusion features and the corresponding weight values; expressed as:

[0144] Z=w0+w1·x1+w2·x2+w3·x3+…+w n ·x n

[0145] Among them, Z represents the weighted sum structure; w0 represents the bias term; x1, x2, x3 and x n are all eigenvalues ​​in the multimodal training fusion feature; n means there are n eigenvalues ​​in the multimodal training fusion feature; w1, w2, w3 and w n x1, x2, x3 and x n The weight value of

[0146] P5. During the forward propagation process, the weighted summation result is converted into the probability value of recurrence of ovarian cancer patients through the sigmoid function; it is expressed as:

[0147]

[0148] Among them, p represents the probability of recurrence in patients with ovarian cancer;

[0149] P6. Calculate the loss value according to the probability value and the recurrence result label; the loss function used in the embodiment of the present invention is the cross entropy loss function; it is expressed as:

[0150] L(p i ,y i )=-y i ·log(p i )-(1-y i )log(1-p i )

[0151] Among them, p i represents the probability of recurrence of the i-th ovarian cancer patient; y i Represents the recurrence result label corresponding to the i-th ovarian cancer patient;

[0152] P7. During the back propagation process, the weight value is updated by the loss value;

[0153] P8. Repeat the above steps P4-P8 based on the updated weight values ​​until the regression model converges, which serves as the trained ovarian cancer recurrence prediction model.

[0154] In the above step S7, guidance suggestions are provided according to the probability of ovarian cancer recurrence; for example:

[0155] (1) When the probability value is 0%-20%, it is considered low risk. It is recommended to continue to follow the current treatment plan and conduct regular review and monitoring; imaging examinations and tumor marker tests are performed every 3-6 months;

[0156] (2) When the probability value is 21%-50%, it is considered to be a moderate risk. It is recommended to discuss adjuvant therapy or other treatment methods with the doctor. Perform imaging examinations and tumor marker tests every 2-3 months, and consider adding biomarker monitoring.

[0157] (3) When the probability value is 51%-80%, it is considered high risk; it is recommended to communicate with the doctor to discuss treatment options and possible side effect management; perform imaging examinations and tumor marker tests once a month, and consider adding more frequent clinical evaluations.

[0158] (4) When the probability value is 81%-100%, it is considered to be extremely high risk and chemotherapy, targeted therapy or immunotherapy is recommended; imaging examinations and tumor marker tests should be performed once a week or as recommended by the doctor, and any new symptoms or signs should be closely monitored.

[0159] Embodiment 2:

[0160] The embodiment of the present invention also provides a prediction system for ovarian cancer recurrence based on a multimodal fusion deep learning model, which applies the above method; see Figure 2 As shown, the prediction system includes an acquisition module, a preprocessing module, a local feature extraction module, a global feature extraction module, a fusion module and a prediction module; wherein the acquisition module is used to acquire multimodal data of an object to be predicted from data tables of multiple databases; the multimodal data is relevant medical information for detecting ovarian cancer; the preprocessing module is used to preprocess the multimodal data; the local feature extraction module is used to extract local features from the preprocessed multimodal data to obtain multimodal local features; the global feature extraction module is used to extract global features based on the multimodal local features to obtain multimodal global features; the fusion module is used to fuse the multimodal local features with the multimodal global features to obtain multimodal fusion features; the prediction module is used to input the multimodal fusion features into a trained ovarian cancer recurrence prediction model, and output the ovarian cancer recurrence probability of the object to be predicted.

[0161] The above-mentioned preprocessing module includes an imaging data preprocessing submodule, a genomic data preprocessing submodule, a pathology data preprocessing submodule and a clinical information preprocessing submodule; wherein the imaging data preprocessing submodule is used to perform normalization, resizing, data enhancement and filtering denoising on the imaging data; and to segment the region of interest in the denoised imaging data; the genomic data preprocessing submodule is used to remove sequencing reads in the genomic data whose quality is lower than a preset value; the pathology data preprocessing submodule is used to remove duplicate data and outliers in the pathology data, and fill in missing data; convert categorical variables in the pathology data into numerical representations; the clinical information preprocessing submodule is used to remove duplicate data and outliers in the clinical information, and fill in missing data; extract keywords from the text in the clinical information; and desensitize the identity information in the clinical information.

[0162] The above-mentioned local feature extraction module includes an image local feature extraction submodule, a gene local feature extraction submodule, a pathology local feature extraction submodule, a clinical local feature extraction submodule and a local feature fusion submodule; wherein the image local feature extraction submodule is used to perform local feature extraction on the preprocessed imaging data through a convolutional neural network to obtain image local features; the gene local feature extraction submodule is used to perform local feature extraction on the preprocessed genomic data through a long short-term memory network to obtain gene local features; the pathology local feature extraction submodule is used to perform local feature extraction on the preprocessed pathology data through a residual network to obtain pathology local features; the clinical local feature extraction submodule is used to perform local feature extraction on the preprocessed clinical information through a multi-layer perceptron to obtain clinical local features; the local feature fusion submodule is used to fuse image local features, gene local features, pathology local features and clinical local features to obtain multimodal local features.

[0163] The prediction system also includes a guidance module for providing guidance suggestions based on the probability of ovarian cancer recurrence; for example:

[0164] (1) When the probability value is 0%-20%, it is considered low risk. It is recommended to continue to follow the current treatment plan and conduct regular review and monitoring; imaging examinations and tumor marker tests are performed every 3-6 months;

[0165] (2) When the probability value is 21%-50%, it is considered to be a moderate risk. It is recommended to discuss adjuvant therapy or other treatment methods with the doctor. Perform imaging examinations and tumor marker tests every 2-3 months, and consider adding biomarker monitoring.

[0166] (3) When the probability value is 51%-80%, it is considered high risk; it is recommended to communicate with the doctor to discuss treatment options and possible side effect management; perform imaging examinations and tumor marker tests once a month, and consider adding more frequent clinical evaluations.

[0167] (4) When the probability value is 81%-100%, it is considered to be extremely high risk and chemotherapy, targeted therapy or immunotherapy is recommended; imaging examinations and tumor marker tests should be performed once a week or as recommended by the doctor, and any new symptoms or signs should be closely monitored.

[0168] The above-mentioned ovarian cancer recurrence prediction model is trained through a model training module; the model training module includes a training data acquisition submodule, a training feature acquisition submodule, a weight value determination submodule, a weighted summation submodule, a probability value calculation submodule, a loss value calculation submodule and an update submodule; wherein the training data acquisition submodule is used to obtain multimodal training data of ovarian cancer patients and corresponding recurrence result labels; the training feature acquisition submodule is used to obtain corresponding multimodal training fusion features according to the multimodal training data of ovarian cancer patients; the weight value determination submodule is used to determine the corresponding initial weight value for the multimodal training fusion features; the weighted summation submodule is used to perform weighted summation of the multimodal training fusion features and the corresponding weight values; the probability value calculation submodule is used to convert the weighted summation result into the probability value of recurrence of ovarian cancer patients through the sigmoid function during the forward propagation process; the loss value calculation submodule is used to calculate the loss value according to the probability value and the recurrence result label; the update submodule is used to update the weight value through the loss value during the backward propagation process.

[0169] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0170] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for predicting ovarian cancer recurrence based on a multimodal fusion deep learning model, characterized in that: The following steps are involved: S1. Acquire multimodal data of an object to be predicted from data tables of multiple databases; the multimodal data is relevant medical information for detecting ovarian cancer; S2. preprocessing the multimodal data; S3, extracting local features from the preprocessed multimodal data to obtain multimodal local features; S4, extracting global features based on the multimodal local features to obtain multimodal global features; S5, fusing the multimodal local features with the multimodal global features to obtain multimodal fusion features; S6. Input the multimodal fusion features into a trained ovarian cancer recurrence prediction model, and output the ovarian cancer recurrence probability of the object to be predicted.

2. The method for predicting ovarian cancer recurrence based on a multimodal fusion deep learning model according to claim 1, characterized in that: The multimodal data includes imaging data, genomic data, pathological data and clinical information.

3. The method for predicting ovarian cancer recurrence based on a multimodal fusion deep learning model according to claim 2, characterized in that: The step S2 specifically includes: (1) Preprocessing the imaging data: Performing normalization, resizing, data enhancement and filtering denoising on the imaging data; and segmenting the region of interest in the denoised imaging data; (2) Preprocessing the genomics data: Removing sequencing reads whose quality is lower than a preset value in the genomics data; (3) Preprocessing the pathological data: Remove duplicate data and outliers from the pathology data, and fill in missing data; Convert categorical variables in pathology data into numerical representation; (4) Preprocessing the clinical information: Remove duplicate data and outliers in the clinical information and fill in missing data; performing keyword extraction on text in the clinical information; The identity information in the clinical information is desensitized.

4. The method for predicting ovarian cancer recurrence based on a multimodal fusion deep learning model according to claim 2, characterized in that: The step S3 specifically includes: (1) Extract local features of the preprocessed imaging data through a convolutional neural network to obtain local features of the image; (2) Extract local features from the preprocessed genomics data using a long short-term memory network to obtain local gene features; (3) extracting local features of the preprocessed pathological data through a residual network to obtain pathological local features; (4) Extract local features from the preprocessed clinical information using a multi-layer perceptron to obtain clinical local features; (5) The image local features, the gene local features, the pathological local features and the clinical local features are fused to obtain multimodal local features.

5. The method for predicting ovarian cancer recurrence based on a multimodal fusion deep learning model according to claim 1, characterized in that: The step S4 is implemented by using a maximum pooling operation.

6. The method for predicting ovarian cancer recurrence based on a multimodal fusion deep learning model according to claim 1, characterized in that: In the step S5, before fusing the multimodal local features with the multimodal global features, the multimodal local features and the multimodal global features are first standardized.

7. The method for predicting ovarian cancer recurrence based on a multimodal fusion deep learning model according to claim 1, characterized in that: The training process of the ovarian cancer recurrence prediction model includes: P1. Obtain multimodal training data of ovarian cancer patients and corresponding recurrence result labels; P2. Obtain corresponding multimodal training fusion features based on the multimodal training data of ovarian cancer patients; P3. Determine the corresponding initial weight value for the multimodal training fusion feature; P4. Perform weighted summation on the multimodal training fusion features and corresponding weight values; P5. During the forward propagation process, the weighted summation result is converted into the probability value of recurrence of ovarian cancer patients through the sigmoid function; P6. Calculate the loss value according to the probability value and the recurrence result label; P7. During the back propagation process, the weight value is updated by the loss value; P8. Repeat the above steps P4-P8 based on the updated weight values ​​until the regression model converges, which serves as the trained ovarian cancer recurrence prediction model.

8. A prediction system for ovarian cancer recurrence based on a multimodal fusion deep learning model, characterized in that: The prediction system includes an acquisition module, a preprocessing module, a local feature extraction module, a global feature extraction module, a fusion module and a prediction module; The acquisition module is used to acquire multimodal data of the object to be predicted from data tables of multiple databases; the multimodal data is relevant medical information for detecting ovarian cancer; The preprocessing module is used to preprocess the multimodal data; The local feature extraction module is used to extract local features from the preprocessed multimodal data to obtain multimodal local features; The global feature extraction module is used to extract global features based on the multimodal local features to obtain multimodal global features; The fusion module is used to fuse the multimodal local features with the multimodal global features to obtain multimodal fusion features; The prediction module is used to input the multimodal fusion features into the trained ovarian cancer recurrence prediction model, and output the ovarian cancer recurrence probability of the object to be predicted.

9. The prediction system for ovarian cancer recurrence based on a multimodal fusion deep learning model according to claim 8, characterized in that: The preprocessing module includes an imaging data preprocessing submodule, a genomics data preprocessing submodule, a pathology data preprocessing submodule and a clinical information preprocessing submodule; The imaging data preprocessing submodule is used to perform normalization, size adjustment, data enhancement and filtering and denoising on the imaging data; And segment the region of interest in the denoised imaging data; The genomics data preprocessing submodule is used to remove sequencing reads with quality lower than a preset value in the genomics data; The pathology data preprocessing submodule is used to remove duplicate data and abnormal values ​​in the pathology data and fill in missing data; convert categorical variables in the pathology data into numerical representation; The clinical information preprocessing submodule is used to remove duplicate data and abnormal values ​​in the clinical information and fill in missing data; extract keywords from the text in the clinical information; and desensitize the identity information in the clinical information.

10. The prediction system for ovarian cancer recurrence based on a multimodal fusion deep learning model according to claim 8, characterized in that: The local feature extraction module includes an image local feature extraction submodule, a gene local feature extraction submodule, a pathology local feature extraction submodule, a clinical local feature extraction submodule and a local feature fusion submodule; The image local feature extraction submodule is used to extract local features from the preprocessed imaging data through a convolutional neural network to obtain image local features; The gene local feature extraction submodule is used to extract local features from the preprocessed genomics data through a long short-term memory network to obtain gene local features; The pathological local feature extraction submodule is used to extract local features from the preprocessed pathological data through a residual network to obtain pathological local features; The clinical local feature extraction submodule is used to extract local features from the preprocessed clinical information through a multi-layer perceptron to obtain clinical local features; The local feature fusion submodule is used to fuse the image local features, the gene local features, the pathological local features and the clinical local features to obtain multimodal local features.

Images