A modeling method and system based on multimodal digital twin technology

Abstract

The present invention discloses a modeling method and system based on multimodal digital twin technology, which belongs to the field of medical data processing. The processing method includes collecting patient pathological data and constructing a multimodal digital twin database; integrating the data in the multimodal digital twin database and constructing a digital twin model; constructing a multimodal inversion model according to the data in the multimodal digital twin database; sorting out the drug efficacy of different treatment types according to the type of brain tumor treatment, forming a treatment database, integrating the drug characteristics in the treatment database and the data in the multimodal digital twin database; inputting the treatment scheme effect prediction results generated by the inversion model into the digital twin model, judging the treatment characteristics of different drugs for patients and predicting the penetration effect of different drugs on the blood-brain barrier. The present invention improves the treatment accuracy of brain tumors by integrating the multimodal data of brain tumors, constructing twin models and inversion models.

Description

Technical Field

[0001] The present invention relates to the field of medical data processing, and in particular to a modeling method and system based on multimodal digital twin technology. Background Art

[0002] Brain tumors are one of the most challenging primary intracranial tumors, with high incidence, high recurrence, high mortality and low cure rates. Brain tumors are usually located in functionally critical brain areas, such as the motor cortex and language center. The surgical risk is extremely high, and the slightest mistake will affect the patient's important functions, such as language, movement and memory. Although surgery is one of the most commonly used treatments, it is often very difficult to completely remove the tumor, especially for malignant tumors such as gliomas, which have blurred boundaries and easily invade healthy brain tissue. In addition, some brain tumors (such as glioblastomas) are highly resistant to conventional treatments (such as radiotherapy and chemotherapy) and have a high recurrence rate. Even if the initial treatment is effective, the tumor may recur rapidly. Each brain tumor patient has different tumor biological characteristics, such as individual differences in genetic variation, treatment responsiveness and drug resistance, so it is difficult to formulate a unified treatment plan. At the same time, the blood-brain barrier, as a natural protective barrier of the brain, limits the effective penetration of most drugs, and many drugs cannot effectively reach the tumor site, thereby weakening the effectiveness of chemotherapy and targeted therapy. Radiotherapy, chemotherapy, and surgery itself can also cause serious side effects, such as nerve damage, memory loss, and epileptic seizures, which pose a significant threat to patients' quality of life.

[0003] With the development of digital twin technology, precision treatment of brain tumors has become more feasible. As a technology that virtualizes and simulates physical entities, digital twin technology provides predictive analysis, monitoring and optimization by accurately mapping the state and behavior of physical entities and synchronizing and updating their growth and evolution in real time. This technology was originally used in the industrial field, but with the advancement of artificial intelligence, cloud computing and big data technology, digital twin technology has gradually penetrated into the medical field, especially in precision medicine, and has shown great application potential. In brain tumor treatment, digital twin technology can establish a personalized tumor digital twin model for each patient. By combining multimodal twin data, such as MRI images, CT scans, genomic mutation data and clinical treatment data, accurate tumor digital twin models for different patients are constructed.

[0004] Although digital twin technology has shown great promise in brain tumor treatment, it also faces some challenges. For example, how to efficiently fit multimodal twin data, ensure the accuracy and real-time performance of the twin model, how to automatically generate accurate treatment plans based on the different disease characteristics of patients through the constructed digital twin model and system, and inversely predict their growth direction and evolution speed; how to predict the penetration effect of drugs on the blood-brain barrier based on the characteristics of various drugs and multimodal twin data, so as to facilitate medical staff to understand and predict the effects of chemoradiotherapy and targeted therapy in advance. These problems need to be further resolved. Summary of the invention

[0005] One of the purposes of the present invention is to provide a modeling method based on multimodal digital twin technology and to solve the shortcomings of the prior art, such as the difficulty in predicting and judging pathological processes such as the direction and evolution speed of tumor growth, which leads to the inability to simulate the evolution and development of tumor characteristics and behaviors, the inability to predict tumor resistance in the body and early drug selection, and the inability to accurately guide the formulation of tumor radiotherapy targets and other clinical treatment problems.

[0006] The present invention is implemented through the following technical scheme: a modeling method based on multimodal digital twin technology, including: S100, collecting patient pathological data and constructing a multimodal digital twin database, wherein the pathological data includes imaging data, genomic data, clinical data and patient physiological data; S200, integrating the data in the multimodal digital twin database to construct a digital twin model, integrating the data in the multimodal digital twin database to ensure that all modal data are identified with the same patient ID, and using imaging data and genetic data as the basis of twin information, using a unified patient ID as a unique identifier, and mapping the patient's clinical data and patient physiological data to the twin information to construct and obtain a personalized digital twin model of the patient, through which the digital twin model reflects the growth direction and evolution rate of the patient's tumor at different stages. degree, and update the digital twin model in real time according to the patient's pathological data, clinical data, physiological data and treatment status to reflect the latest tumor status; S300, build a multimodal inversion model according to the data of the multimodal digital twin database in step S100; S400, according to the type of brain tumor treatment, organize the drug efficacy of different treatment types to form a treatment database, and integrate the drug properties in the treatment database with the data in the multimodal digital twin database. The treatment types include chemotherapy, targeted therapy, immunotherapy and hormone therapy; S500, input the treatment plan effect generated by the inversion model into the digital twin model for prediction, record the prediction results on the digital twin model, and judge the treatment characteristics of different drugs for patients according to the prediction results, and predict the penetration effect of different drugs on the blood-brain barrier and the evolution trend of brain tumors.

[0007] Furthermore, the imaging data includes magnetic resonance imaging data, X-ray imaging data and CT imaging data; the genomic data includes the patient's gene expression data, gene data and proteomics data; the clinical data includes medical history, treatment records, signs and symptoms data; the physiological data includes heart rate, blood pressure, height, weight sign data for dynamic monitoring.

[0008] Furthermore, step S100 also includes a data preprocessing step. After collecting the patient's pathological data, the collected data is first preprocessed, and then a multimodal digital twin database is constructed based on the preprocessed data. Specifically, the following steps are included: S110, removing noise and invalid information in the collected imaging data, genomic data, clinical data, and patient physiological data through data cleaning; S120, formatting data from different sources uniformly through data standardization. Different standardizations can be used for different data so that the same data has the same scale. For imaging data, mean and standard deviation standardization can be used, wherein the mean is subtracted from each image and divided by the standard deviation to ensure that the images input into the model are on the same scale. For genomic data, clinical data, and patient physiological data, the Z-score algorithm can be used for standardization. The value of the growth rate can be scaled to a smaller numerical range by logarithmic transformation and then standardized by Z-score algorithm; S130, feature extraction is used to extract key features from imaging, genetic, clinical and patient physiological data for model input, and high-order features in imaging data are automatically extracted by pre-trained CNN model. For genetic data, Lasso regression algorithm can be used to screen out genes with significant differential expression between tumor tissue and normal tissue, identify gene pathway models related to tumor growth direction and speed, extract key genes related to targeted therapy, and identify gene features closely related to tumor prognosis, and input the gene features into gene pathway models to predict tumor growth direction and speed; for clinical data and patient physiological data, time series analysis can be performed to extract trend features and fluctuation features of patients' vital signs and blood tests.

[0009] Furthermore, mapping the patient's clinical data and patient physiological data to the twin information base includes, first, spatially aligning the processed image data, and using automatic segmentation technology to extract the tumor area from the image to ensure the spatial consistency of different image data; second, connecting the gene expression and mutation information in the genomic data with the known gene annotation database to ensure the standardization and unification of the gene data, identifying the gene pathways related to brain tumors, and then combining them with the extracted tumor area to form the twin information base; finally, using the unified patient ID as the unique identifier, mapping the clinical data and patient physiological data to the twin information base.

[0010] Furthermore, constructing a multimodal inversion model includes the following sub-steps: S310, determining the target data range for inversion based on the data in the constructed multimodal digital twin database, wherein the target data range is generated by calibrating the positions of all tumor cells in the lesion area on the image data and the gene data, and combining the twin information basis in step S200 to generate verification conditions, and then determining the constraint conditions based on the clinical data and the patient's physiological data, generating the tumor cell spatial coordinates corresponding to the lesion area, and the set of tumor cell spatial coordinates is the target data range; S320, extracting the principal components of each data in the multimodal digital twin database through the PCA algorithm, and using the extracted principal components as the characteristic basis functions of the gene expression pattern, and constructing a multimodal data dictionary based on the extracted characteristic basis functions; S330, according to the principle of matching pursuit method, matching the data to be inverted point by point with the characteristic basis functions in the multimodal data dictionary, and extracting the data pairs with the highest matching degree to form an inverted multimodal data body.

[0011] Furthermore, constructing a multimodal data dictionary includes the following steps: S321, using the PCA algorithm to extract principal components from imaging data, genomic data, clinical data and patient physiological data; S322, then using the K-means algorithm to divide the principal component data extracted from each type of data into several clusters, and the cluster center is the characteristic basis function; S323, combining the obtained characteristic basis functions to form a multimodal data dictionary containing multiple types of data.

[0012] Furthermore, determining the data pair with the greatest matching degree includes the following steps: S331, normalizing the data in the multimodal data dictionary using the L2 norm normalization method to obtain a first multimodal vector S332, taking the data point to be inverted as the center, obtaining a data segment of a predetermined time length as the second multimodal vector , and the first vector is normalized by L2 norm normalization to obtain the third multimodal vector ; S333, calculate the dot product of the first multimodal vector and the third multimodal vector, determine the data pair with the largest dot product calculation result as the data pair with the highest matching degree, extract the multimodal data pair with the highest matching degree, and thus form an inverted multimodal data body.

[0013] Furthermore, the integration in step S400 includes, S410, extracting features related to drug efficacy and side effects from the drug database, the features including drug action mechanism, metabolic pathways and side effects; S420, extracting the patient's physiological characteristics, genetic characteristics, medical history and treatment records from the multimodal digital twin database; S430, then using cosine similarity to calculate the similarity between drug characteristics and patient characteristics, and using the similarity data as an inversion parameter, inputting it into the inversion model to generate a treatment effect prediction result.

[0014] Furthermore, chemotherapy drugs may include temozolomide, carmustine, lomustine and vincristine, targeted therapy drugs may include bevacizumab, dabrafenib and trametinib, immunotherapy drugs may include pembrolizumab, nivolumab and pembrolizumab, and hormone therapy drugs may include bromocriptine and carbenicillin.

[0015] On the other hand, the present invention provides a modeling system based on multimodal digital twin technology. The multimodal data processing system includes a multimodal digital twin database construction unit, a digital twin model construction unit and a multimodal inversion model construction unit. The multimodal digital twin database construction unit is configured to collect patient pathological data and preprocess the collected data, and construct a multimodal digital twin database with the preprocessed data; the digital twin model construction unit is connected to the multimodal digital twin database construction unit, and is configured to integrate the data in the multimodal digital twin database to construct a digital twin model, using imaging data and gene data as the twin information basis, using a unified patient ID as a unique identifier, mapping the patient's clinical data and patient physiological data to the twin information basis, and constructing a personalized digital twin model for the patient. A twin model is used to reflect the growth and evolution of a patient's tumor at different stages, and the model is updated in real time according to the patient's physiological data and treatment status to reflect the latest tumor status; a multimodal inversion model construction unit is connected to a multimodal digital twin database construction unit and a digital twin model construction unit, and is configured to construct a multimodal inversion model according to the data in the multimodal digital twin database; a treatment database construction unit is connected to a multimodal digital twin database construction unit and a multimodal inversion model construction unit, and is configured to organize the efficacy of drugs of different treatment types according to the type of brain tumor treatment to form a treatment database, integrate the drug characteristics in the treatment database with the data in the multimodal digital twin database to obtain inversion parameters, and input the inversion parameters into the inversion model to generate a treatment plan effect prediction result.

[0016] Furthermore, the multimodal inversion model building unit also includes a multimodal data dictionary subunit and a data pair matching subunit, wherein:

[0017] The multimodal data dictionary subunit is configured to determine the target data range for inversion based on the data in the constructed multimodal digital twin database, wherein the target data range is generated by calibrating the positions of all tumor cells in the lesion area on the image data and gene data, and combining the twin information basis to generate verification conditions, and then determine the constraint conditions based on the clinical data and the patient's physiological data to generate the tumor cell spatial coordinates corresponding to the lesion area, and the set of tumor cell spatial coordinates is the target data range; extract the principal components of each data in the multimodal digital twin database through the PCA algorithm, and use the extracted principal components as the characteristic basis functions of the gene expression pattern, and construct a multimodal data dictionary based on the extracted characteristic basis functions; according to the principle of matching pursuit method, the data to be inverted is matched point by point with the characteristic basis functions in the multimodal data dictionary, and the data pairs with the largest matching degree are extracted to form the inverted multimodal data body. The data pair matching subunit is configured to unitize the data in the multimodal data dictionary using the L2 norm unitization method, and obtain the dot product of the first multimodal vector and the third multimodal vector, and determine the data pair with the largest dot product calculation result as the data pair with the largest matching degree. The multimodal data pairs with the greatest matching degree are extracted to form an inverted multimodal data volume.

[0018] On the other hand, the present invention provides a computer device, comprising: a processor; and a memory storing a computer program, wherein when the computer program is executed by the processor, the modeling method based on the multimodal digital twin technology as described above is implemented.

[0019] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0020] 1. The present invention integrates the multimodal data of brain tumors to form a complete multimodal digital twin database, and constructs a digital twin model and inversion model through the data in this database, thereby improving the accuracy of brain tumor treatment.

[0021] 2. The present invention constructs a multimodal digital twin database and a digital twin model, which makes it possible to directly use the processed data in the multimodal digital twin database to establish inversion constraint information when constructing the inversion model, without considering the inversion wavelet problem and the initial model construction problem. At the same time, under the premise of accurately calibrating the spatial coordinates of tumor cells, the therapeutic drug data is combined to involve the drug data in the inversion, thereby improving the consistency of the inversion results and the treatment effects.

[0022] 3. The present invention establishes a digital twin model to monitor and optimize the tumor treatment status and prediction results in real time. It can promptly discover problems in the treatment process and take measures to make adjustments, thereby improving the accuracy of brain tumor treatment. At the same time, based on the inversion model, it realizes the prediction and judgment of the drug's penetration effect on the blood-brain barrier according to the characteristics of various drugs and multimodal data, which is convenient for medical staff to analyze the advantages and disadvantages of different treatment plans in advance and predict the patient's prognosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0024] Figure 1 A method flow chart provided for exemplary embodiment 1 of the present invention.

[0025] Figure 2 A system structure diagram provided for exemplary embodiment 2 of the present invention.

[0026] Figure 3 A computer device structure diagram provided for exemplary embodiment 3 of the present invention.

[0027] Description of reference numerals:

[0028] 100 - modeling system based on multimodal digital twin technology, 110 - multimodal digital twin database construction unit, 120 - digital twin model construction unit, 130 - multimodal inversion model construction unit, 140 - treatment database construction unit, 200 - computer equipment, 210 - memory, 220 - processor. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0030] Embodiment 1:

[0031] In this embodiment, a specific implementation step of a modeling method based on multimodal digital twin technology is disclosed. The flowchart of the modeling method is as follows: Figure 1 As shown, the specific steps include:

[0032] Step 1: Collect patient pathological data and build a multimodal digital twin database.

[0033] The pathological data includes imaging data, genomic data, clinical data, and patient physiological data.

[0034] The imaging data includes magnetic resonance imaging (MRI) data, X-ray imaging data, and CT imaging data. The genomic data includes the patient's gene expression data, gene data, and proteomics data. The clinical data includes medical history, treatment records, signs and symptoms data. The physiological data includes heart rate, blood pressure, height, weight and other signs data for dynamic monitoring.

[0035] It also includes a data preprocessing step, which includes the following sub-steps:

[0036] 1) First, remove noise and invalid information from the data through data cleaning.

[0037] It should be noted that the image data is the most likely to generate noise in pathological data, and image data has an extremely important weight in the diagnosis and treatment of brain tumors, so the most important thing in the data preprocessing step is to process the image data.

[0038] Specifically, convolution filtering can be used to remove noise from MRI, CT and other imaging data to maintain image clarity.

[0039] 2) Then, data from different sources are formatted in a unified manner through data standardization to facilitate subsequent processing.

[0040] Since there will be many duplicate data points for the same patient, deduplication is performed before standardization. In particular, duplicate records in clinical data can be identified by unique identifiers (such as patient ID), thereby achieving data deduplication.

[0041] Then different standardizations can be used to process different data, so that the same data has the same scale, which is convenient for subsequent analysis operations.

[0042] For image data, mean subtraction can be used, that is, subtracting the mean and dividing by the standard deviation for each image to ensure that the images input to the model are at the same scale.

[0043] Genomic data, clinical data and patient physiological data can be standardized using the Z-score algorithm.

[0044] Among them, for values ​​that show exponential growth in genomic data, a logarithmic transformation can be used to scale them to a smaller numerical range and then standardized using the Z-score algorithm.

[0045] 3) Use feature extraction to extract key features from imaging, genetic, clinical, and patient physiological data for model input.

[0046] Specifically, the pre-trained CNN model can be used to automatically extract high-order features from MRI, CT and other imaging data, such as the edge, shape and size of the tumor.

[0047] For genetic data, the Lasso regression algorithm can be used to screen out genes that are significantly differentially expressed between tumor tissues and normal tissues, identify tumor-related gene pathways, extract key genes related to targeted therapy, and identify gene features that are closely related to tumor prognosis.

[0048] For clinical data and patient physiological data, time series analysis can be performed to extract the trend characteristics and fluctuation characteristics of patients' vital signs and blood tests.

[0049] By constructing a multimodal digital twin database, effective integration of brain tumor-related data is achieved, and the construction of subsequent twin models and inversion models is facilitated by preprocessing different data.

[0050] Step 2: Integrate the data in the multimodal digital twin database to build a digital twin model.

[0051] Specifically, after processing the data in the multimodal digital twin database in step 1, the data in the multimodal digital twin database is integrated. The data in the multimodal digital twin database is integrated to ensure that the data of all modalities are identified with the same patient ID, and the imaging data and genetic data are used as the basis of the twin information, and the unified patient ID is used as the unique identifier. On the basis of mapping the patient's clinical data and the patient's physiological data to the twin information, a personalized digital twin model of the patient is constructed and obtained. The digital twin model reflects the growth and evolution of the patient's tumor at different stages, and updates the digital twin model in real time according to the patient's physiological data and treatment status to reflect the latest tumor status.

[0052] The processed image data and gene data are used as the twin information basis, and the unified patient ID is used as the unique identifier. The patient's clinical data and patient physiological data are mapped to the twin information basis to build a personalized digital twin model for the patient. The digital twin model reflects the growth and evolution of the patient's tumor at different stages. The model is updated in real time based on the patient's physiological data and treatment status to reflect the latest tumor status.

[0053] Specifically, data mapping includes, first, spatially registering the processed image data to ensure the spatial consistency of different image data. Automatic segmentation technology is used to extract the tumor area from the image. Then the gene expression and mutation information in the genomic data are connected to the known gene annotation database (such as Ensembl, NCBI Gene) to ensure the standardization and unification of the gene data. And identify the gene pathways related to brain tumors, and then combine them with the extracted tumor area to form a twin information foundation. Finally, the unified patient ID is used as a unique identifier to map the clinical data and patient physiological data to the twin information foundation.

[0054] Step 3: Construct a multimodal inversion model based on the data in the multimodal digital twin database constructed in step 1.

[0055] Specifically, the construction of the multimodal inversion model includes:

[0056] 1) Determine the target data range for inversion based on the constructed multimodal digital twin database;

[0057] Specifically, by calibrating the positions of all tumor cells in the lesion area on the imaging data and genetic data, verification conditions are generated based on the twin information, and then the constraints are determined based on the clinical data and the patient's physiological data to generate the spatial coordinates of the tumor cells corresponding to the lesion area.

[0058] 2) The principal components of each data in the multimodal digital twin database are extracted through the PCA algorithm, and the extracted principal components are used as the characteristic basis functions of the gene expression pattern. A multimodal data dictionary is constructed based on the extracted characteristic basis functions.

[0059] Specifically, the multimodal data dictionary includes the following steps:

[0060] The PCA algorithm is used to extract principal components from imaging data, genomic data, clinical data, and patient physiological data. The K-means algorithm is then used to divide the principal component data extracted from each type of data into several clusters, with the cluster center being the characteristic basis function. The obtained characteristic basis functions are combined to form a multimodal data dictionary containing multiple types of data.

[0061] 3) According to the principle of matching pursuit method, the data to be inverted are matched point by point with the characteristic basis functions in the multimodal data dictionary, and the data pairs with the highest matching degree are extracted to form the inverted multimodal data body.

[0062] Specifically, the method for determining the data pair with the greatest matching degree is as follows:

[0063] The data in the multimodal data dictionary is normalized using the L2 norm normalization method to obtain a first multimodal vector .

[0064] Centered on the data point to be inverted, a data segment of a predetermined time length is obtained as the second multimodal vector , and the first vector is normalized by L2 norm normalization to obtain the third multimodal vector .

[0065] The dot product of the first multimodal vector and the third multimodal vector is calculated, and the data pair with the largest dot product calculation result is determined as the data pair with the largest matching degree. The multimodal data pair with the largest matching degree is extracted to form an inverted multimodal data volume.

[0066] Through the above steps, an accurate multimodal inversion model can be constructed, and the synergy of multiple data sources can be used to improve the accuracy and reliability of the inversion. The multimodal inversion model constructed can be used to further analyze the characteristics of the tumor, such as location, size, gene expression pattern, etc. And by combining genetic data, the genetic characteristics related to the tumor can be analyzed to provide a basis for personalized treatment.

[0067] At the same time, after the inversion model is built, the model parameters can be adjusted to predict the specific effects of drugs on tumors, such as drug penetration, target inhibition effect, etc. Medical staff can intuitively predict the treatment effect based on the inversion model and digital twin model, and optimize personalized treatment plans based on the prediction results to improve treatment effects and reduce side effects.

[0068] Step 4: According to the type of brain tumor treatment, organize the drug efficacy of different treatment types to form a treatment database.

[0069] The types of treatment include chemotherapy, targeted therapy, immunotherapy, and hormone therapy.

[0070] Among them, chemotherapy drugs may include temozolomide, carmustine, lomustine and vincristine.

[0071] Targeted therapy drugs may include bevacizumab, dabrafenib, and trametinib.

[0072] Immunotherapy drugs may include pembrolizumab, nivolumab, and pembrolizumab.

[0073] Hormone therapy drugs may include bromocriptine and carbetocide.

[0074] The drug characteristics in the treatment database and the data in the multimodal digital twin database can be integrated first to obtain the drug side effects or compatibility with the patient's specific genotype, making the evaluation of treatment effects more accurate.

[0075] Specifically, the integration steps include:

[0076] Features related to drug efficacy and side effects are extracted from a drug database, including drug action mechanisms, metabolic pathways, and side effects.

[0077] Extract patients’ physiological characteristics, genetic features, medical history, and treatment records from the multimodal digital twin database;

[0078] Then, the cosine similarity is used to calculate the similarity between drug characteristics and patient characteristics, and the similarity data is used as the inversion parameter and input into the inversion model to generate the prediction results of the treatment effect.

[0079] It should be noted that the inversion model can be used to formulate treatment plans based on the different disease characteristics of patients. However, predicting the specific effects of the formulated treatment plans is a difficult problem. By constructing the data of the treatment database and combining it with the data in the multimodal digital twin database, the inversion model can be used to predict the effects of different treatment plans. At the same time, through the combination with the digital twin model, the penetration effect of drugs on the blood-brain barrier can be predicted, which makes it easier for medical staff to understand the treatment effects of different drug combinations and achieve more accurate treatment guidance for each patient.

[0080] Step 5: The treatment effect prediction results generated by the inversion model are input into the digital twin model, the prediction results are recorded on the digital twin model, and the therapeutic characteristics of different drugs for patients are judged based on the results and the penetration effects of different drugs on the blood-brain barrier are predicted.

[0081] It should be noted that the digital twin model is used to construct a virtual tumor model, while the inversion model is used to predict the subsequent development of the tumor. By processing the patient's multimodal data, the digital twin model and the inversion model together form an organic system to help medical staff better predict the biological behavior of the patient's brain tumor, thereby enabling medical staff to predict the condition and biological behavior of brain tumors in advance, improve treatment plans, and judge the prognosis in advance, providing accurate digital twin-assisted treatment and prediction.

[0082] Embodiment 2:

[0083] like Figure 2 As shown, the modeling system 100 based on the multimodal digital twin technology includes: a multimodal digital twin database construction unit 110, a digital twin model construction unit 120, a multimodal inversion model construction unit 130, and a treatment database construction unit 140.

[0084] The multimodal digital twin database construction unit 110 is configured to collect patient pathological data and preprocess the collected data, and construct a multimodal digital twin database with the preprocessed data.

[0085] The digital twin model construction unit 120 is configured to integrate the data in the multimodal digital twin database, construct a digital twin model, use imaging data and gene data as the twin information basis, use a unified patient ID as a unique identifier, map the patient's clinical data and patient physiological data to the twin information basis, and construct a personalized digital twin model of the patient. The digital twin model reflects the growth and evolution of the patient's tumor at different stages, and updates the model in real time according to the patient's physiological data and treatment status to reflect the latest tumor condition.

[0086] The multimodal inversion model construction unit 130 is configured to construct a multimodal inversion model according to the data in the multimodal digital twin database.

[0087] Specifically, the multimodal inversion model building unit may further include a multimodal data dictionary subunit and a data pair matching subunit.

[0088] Among them, the multimodal data dictionary subunit is configured to determine the target data range for inversion based on the data in the constructed multimodal digital twin database. The target data range is generated by calibrating the positions of all tumor cells in the lesion area on the image data and gene data, and combining the twin information basis to generate verification conditions. Then, the constraints are determined based on the clinical data and the patient's physiological data to generate the tumor cell spatial coordinates corresponding to the lesion area. The set of tumor cell spatial coordinates is the target data range; the principal components of each data in the multimodal digital twin database are extracted by the PCA algorithm, and the extracted principal components are used as the characteristic basis functions of the gene expression pattern, and a multimodal data dictionary is constructed based on the extracted characteristic basis functions; according to the principle of matching pursuit method, the data to be inverted are matched point by point with the characteristic basis functions in the multimodal data dictionary, and the data pairs with the highest matching degree are extracted to form an inverted multimodal data body.

[0089] The data pair matching subunit is configured to unitize the data in the multimodal data dictionary using an L2 norm unitization method to obtain a first multimodal vector ; Taking the data point to be inverted as the center, obtain a data segment of a predetermined time length as the second multimodal vector , and the first vector is normalized by L2 norm normalization to obtain the third multimodal vector ; Calculate the dot product of the first multimodal vector and the third multimodal vector, and determine the data pair with the largest dot product calculation result as the data pair with the largest matching degree. Extract the multimodal data pair with the largest matching degree, thereby forming an inverted multimodal data body.

[0090] The treatment database construction unit 140 is configured to organize the drug efficacy of different treatment types according to the type of brain tumor treatment, form a treatment database, integrate the drug characteristics in the treatment database with the data in the multimodal digital twin database, obtain inversion parameters, input the inversion parameters into the inversion model, and generate a treatment plan effect prediction result.

[0091] Embodiment 3:

[0092] like Figure 3 As shown, a computer device 200 includes a memory 210 and a processor 220. A computer program is stored in the memory 210. When the computer program is executed by the processor, the modeling method based on the multimodal digital twin technology as described in Example 1 is implemented.

[0093] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A modeling method based on multimodal digital twin technology, characterized in that: The modeling method comprises: S100, collect patient pathological data and build a multimodal digital twin database, The pathological data includes: imaging data, genomic data, clinical data and patient physiological data; S200, integrating the data in the multimodal digital twin database to construct a digital twin model, Integrate the data in the multimodal digital twin database to ensure that all modal data are identified with the same patient ID, and use imaging data and genetic data as the twin information basis. Use a unified patient ID as a unique identifier, map the patient's clinical data and patient physiological data to the twin information basis, and build and obtain the patient's personalized digital twin model. The digital twin model reflects the patient's tumor growth direction and evolution speed at different stages. The digital twin model is updated in real time based on the patient's pathological data, clinical data, physiological data, and treatment status to reflect the latest tumor status. S300, constructing a multimodal inversion model according to the data of the multimodal digital twin database in step S100; S400, according to the type of brain tumor treatment, sort out the efficacy of drugs of different treatment types to form a treatment database, and integrate the drug characteristics in the treatment database with the data in the multimodal digital twin database, wherein the treatment types include: chemotherapy, targeted therapy, immunotherapy, and hormone therapy; S500, inputting the treatment plan effect generated by the inversion model into the digital twin model for prediction, recording the prediction results on the digital twin model, and judging the therapeutic characteristics of different drugs for patients based on the prediction results, and predicting the penetration effect of different drugs on the blood-brain barrier and the evolution trend of brain tumors; The step S200 of mapping the patient's clinical data and the patient's physiological data onto the twin information includes: First, the processed image data are spatially registered, and the tumor area is extracted from the image using automatic segmentation technology to ensure the spatial consistency of different image data; Secondly, the gene expression and mutation information in the genomic data are connected with the known gene annotation database to ensure the standardization and unification of the gene data, identify the gene pathways related to brain tumors, and then combine them with the extracted tumor areas to form the twin information basis; Finally, the unified patient ID is used as a unique identifier to map clinical data and patient physiological data onto the twin information base; The step S300 of constructing a multi-modal inversion model includes the following sub-steps: S310, determining the target data range for inversion based on the data in the constructed multi-modal digital twin database, The target data range generates verification conditions by calibrating the positions of all tumor cells in the lesion area on the image data and gene data, and combining the twin information basis in step S200, and then determines the constraint conditions according to the clinical data and the patient's physiological data to generate the tumor cell spatial coordinates corresponding to the lesion area. The set of tumor cell spatial coordinates is the target data range; S320, extracting the principal components of each data in the multimodal digital twin database by using the PCA algorithm, using the extracted principal components as characteristic basis functions of the gene expression pattern, and constructing a multimodal data dictionary according to the extracted characteristic basis functions; S330. According to the principle of matching pursuit method, the data to be inverted are matched point by point with the characteristic basis functions in the multimodal data dictionary, and the data pairs with the greatest matching degree are extracted to form an inverted multimodal data body.

2. The modeling method based on multimodal digital twin technology according to claim 1 is characterized in that: The imaging data include: magnetic resonance imaging data, X-ray imaging data and CT imaging data; the genomic data include: the patient's gene expression data, gene data and proteomics data; the clinical data include: medical history, treatment records, signs and symptoms data; the physiological data include: heart rate, blood pressure, height, weight sign data, for dynamic monitoring.

3. The modeling method based on multimodal digital twin technology according to claim 1 is characterized in that: The step S100 also includes a data preprocessing step. After collecting the patient's pathological data, the collected data is first preprocessed, and then a multimodal digital twin database is constructed based on the preprocessed data. Specifically, the steps include: S110, removing noise and invalid information from the collected imaging data, genomic data, clinical data, and patient physiological data through data cleaning; S120. Through data standardization, data from different sources are uniformly formatted, and different data are processed using different standards, so that the same data has the same scale. For image data, the mean and standard deviation are used to standardize the image data. By subtracting the mean from the pixel value of the image data and dividing it by the standard deviation, the images input to the model are ensured to be on the same scale. The genomic data, clinical data and patient physiological data are standardized using the Z-score algorithm. Among them, for the exponentially growing values ​​in the genomic data, logarithmic transformation is used to scale them to a smaller numerical range and then normalized by the Z-score algorithm; S130, use feature extraction to extract key features from imaging, genetic, clinical and patient physiological data for model input, Automatically extract high-level features from image data through pre-trained CNN models. The Lasso regression algorithm is used to screen the genes with significant differential expression between tumor tissue and normal tissue for the gene data, identify the gene pathway model related to the direction and speed of tumor growth, extract the key genes related to targeted therapy, and identify the gene features closely related to tumor prognosis. The gene features are input into the gene pathway model to predict the direction and speed of tumor growth; Time series analysis is performed on clinical data and patient physiological data to extract trend characteristics and fluctuation characteristics of patients' vital signs and blood tests.

4. The modeling method based on multimodal digital twin technology according to claim 1 is characterized in that: The construction of the multimodal data dictionary comprises the following steps: S321, extract principal components from imaging data, genomic data, clinical data and patient physiological data using PCA algorithm; S322, then use the K-means algorithm to divide the principal component data extracted from each type of data into several clusters, and the cluster center is the characteristic basis function; S323. Combining the obtained characteristic basis functions to form a multimodal data dictionary containing multiple types of data.

5. The modeling method based on multimodal digital twin technology according to claim 1 is characterized in that: The determination of the data pair with the greatest matching degree comprises the following steps: S331, normalize the data in the multimodal data dictionary using the L2 norm normalization method to obtain a first multimodal vector ; S332, taking the data point to be inverted as the center, obtaining a data segment of a predetermined time length as a second multimodal vector , and the first vector is normalized by L2 norm normalization to obtain the third multimodal vector ; S333, obtaining the dot product of the first multimodal vector and the third multimodal vector, determining the data pair with the largest dot product calculation result as the data pair with the greatest matching degree, extracting the multimodal data pair with the greatest matching degree, thereby forming an inverted multimodal data body.

6. The modeling method based on multimodal digital twin technology according to claim 1 is characterized in that: The integration in step S400 includes: S410, extracting features related to drug efficacy and side effects from a drug database, wherein the features include drug action mechanisms, metabolic pathways, and side effects; S420, extracting the patient's physiological characteristics, genetic characteristics, medical history and treatment records from the multimodal digital twin database; S430, then use cosine similarity to calculate the similarity between drug characteristics and patient characteristics, and use the similarity data as inversion parameters, input into the inversion model, and generate a prediction result of the treatment effect.

7. A modeling system based on multimodal digital twin technology, the multimodal data processing system comprising: A multimodal digital twin database construction unit, a digital twin model construction unit, a multimodal inversion model construction unit, and a treatment database construction unit, wherein: The multimodal digital twin database construction unit is configured to collect patient pathological data and preprocess the collected data, and construct a multimodal digital twin database with the preprocessed data; The digital twin model construction unit is connected to the multimodal digital twin database construction unit, and is configured to integrate the data in the multimodal digital twin database to construct a digital twin model. The imaging data and gene data are used as the twin information basis, and the unified patient ID is used as the unique identifier. The patient's clinical data and patient physiological data are mapped to the twin information basis to construct a personalized digital twin model of the patient. The digital twin model reflects the growth and evolution of the patient's tumor at different stages, and updates the model in real time according to the patient's physiological data and treatment status to reflect the latest tumor status. The multimodal inversion model construction unit is connected to the multimodal digital twin database construction unit and the digital twin model construction unit, and is configured to construct a multimodal inversion model according to the data in the multimodal digital twin database; The treatment database construction unit is connected to the multimodal digital twin database construction unit and the multimodal inversion model construction unit, and is configured to sort out the drug efficacy of different treatment types according to the type of brain tumor treatment to form a treatment database, integrate the drug characteristics in the treatment database with the data in the multimodal digital twin database to obtain inversion parameters, input the inversion parameters into the inversion model, and generate a prediction result of the treatment effect; The digital twin model construction unit maps the patient's clinical data and the patient's physiological data onto the twin information basis, including: First, the processed image data are spatially registered, and the tumor area is extracted from the image using automatic segmentation technology to ensure the spatial consistency of different image data; Secondly, the gene expression and mutation information in the genomic data are connected with the known gene annotation database to ensure the standardization and unification of the gene data, identify the gene pathways related to brain tumors, and then combine them with the extracted tumor areas to form the twin information basis; Finally, the unified patient ID is used as a unique identifier to map clinical data and patient physiological data onto the twin information base; The multimodal inversion model construction unit includes the following submodules: The data range determination submodule is configured to determine the target data range of inversion based on the data in the constructed multimodal digital twin database, wherein the target data range generates verification conditions by calibrating the positions of all tumor cells in the lesion area on the image data and gene data, and combining the twin information basis in the digital twin model construction unit, and then determines the constraint conditions based on the clinical data and the patient's physiological data, and generates the tumor cell spatial coordinates corresponding to the lesion area, wherein the set of tumor cell spatial coordinates is the target data range; The principal component extraction submodule is configured to extract the principal components of each data in the multimodal digital twin database through the PCA algorithm, and use the extracted principal components as the characteristic basis functions of the gene expression pattern, and construct a multimodal data dictionary based on the extracted characteristic basis functions; The data matching subunit is configured to match the data to be inverted with the characteristic basis functions in the multimodal data dictionary point by point according to the principle of matching pursuit method, and extract the data pairs with the greatest matching degree to form the inverted multimodal data body.

8. A computer device, characterized in that: The device comprises: processor; A memory stores a computer program, which, when executed by a processor, implements the modeling method based on multimodal digital twin technology as described in any one of claims 1 to 6.

Images