Vascular risk assessment model system based on multi-modal fusion strategy and deep learning

Abstract

The invention discloses a vascular risk assessment model system based on a multi-modal fusion strategy and deep learning. An acquisition module acquires examination and assay data, image data, gene detection data and clinical data of a plurality of historical patients, and marks various vascular disease conditions; the preprocessing module is used for standardizing numerical data, carrying out one-hot coding or entity embedding on category data, converting time series data into a fixed-length vector, extracting iconography features from image data and / or learning image deep features from the image data, extracting features from text data, carrying out dimensionality reduction on SNPs data, and carrying out normalization and feature selection on gene expression profile data; the feature sample construction module constructs a training set and a verification set by using the features of each historical patient after multi-modal preprocessing; the model construction module constructs a multi-task deep learning model; and the multi-task learning training module trains and verifies the model, optimizes hyper-parameters of the model by using an optimization algorithm, and adjusts the hyper-parameters to be optimal to obtain a target multi-task deep learning model.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, multimodal data processing, and model building, specifically a vascular risk assessment model system based on multimodal fusion strategies and deep learning. This invention is based on a Beijing Natural Science Foundation project, "Exploration of the Immunological Characteristics and Potential Intervention Targets for Aortoiliac Artery Occlusive Disease," with a grant number of 7252061. Background Art

[0002] Vascular diseases primarily include venous diseases (such as deep vein thrombosis and chronic venous insufficiency) and arterial diseases (such as lower limb arteriosclerosis obliterans, coronary artery disease, and cerebrovascular disease). Currently, clinical risk assessments for venous and arterial diseases are often performed independently, with one technology only capable of assessing the risk of venous or arterial disease. There is a lack of comprehensive technology that can simultaneously assess the risk of both venous and arterial diseases in the same patient.

[0003] Existing diagnostic methods typically rely on a single or limited combination of examination methods. For example, ultrasound examinations are primarily used for venous diseases, while angiography or CT angiography (CTA) are more commonly used for arterial diseases. Doctors must comprehensively analyze examination results from various sources (such as laboratory reports, imaging reports, and genetic testing reports), as well as the patient's clinical manifestations and medical history, to make a diagnosis. This analysis process is not only time-consuming but also requires a high level of physician experience and expertise, and is prone to subjectivity and diagnostic discrepancies.

[0004] In recent years, artificial intelligence (AI), particularly machine learning, has demonstrated tremendous potential in medical image analysis and disease diagnosis. Studies have applied AI to the diagnosis of specific vascular diseases, such as using AI to analyze ultrasound images for deep vein thrombosis and assessing arterial stenosis by analyzing CTA images. However, most of these models target a single disease or data modality, making it difficult to comprehensively assess a patient's overall vascular health.

[0005] Therefore, there is an urgent need to develop an intelligent assessment system that can integrate multimodal patient data (including laboratory tests, CT, MRI, genetic testing, etc.) and perform risk assessments on a variety of venous and arterial vascular diseases, so as to assist doctors in conducting more comprehensive, accurate and efficient vascular disease diagnosis and risk stratification. Summary of the Invention

[0006] In response to the limitations and shortcomings of existing technologies in the diagnosis and assessment of vascular diseases, the present invention provides a vascular risk assessment model system based on multimodal fusion strategy and deep learning, which can perform unified, comprehensive and refined risk assessment of multiple venous and arterial diseases in the same patient.

[0007] The present invention solves the above technical problems through the following technical solutions:

[0008] The present invention provides a vascular risk assessment model system based on multimodal fusion strategy and deep learning, comprising:

[0009] An acquisition module is used to acquire the examination and laboratory data, imaging data, genetic testing data, and clinical data of multiple historical patients and to annotate various vascular disease conditions. The examination and laboratory data include numerical, categorical, and / or time-series data; the clinical data include text, numerical, and / or categorical data; and the genetic testing data include SNPs data and / or gene expression profile data.

[0010] Preprocessing module, used to standardize numerical data, perform one-hot encoding or entity embedding on categorical data, convert time series data into fixed-length vectors using time series embedding methods, segment image data using image segmentation techniques to segment regions of interest and extract imaging features from them, and / or use deep learning-based feature extractors to learn deep image features from image data, extract features from text data using word embedding and NLP techniques to obtain text features, perform dimensionality reduction on SNPs data to obtain gene features, and perform normalization and feature selection on gene expression data to obtain gene features;

[0011] Feature sample construction module, used to construct training sets and validation sets using multimodal pre-processed features of each historical patient;

[0012] The model building module is used to build a multi-task deep learning model, including an input layer, a multi-layer convolutional layer with a residual structure, a fusion layer for weighted feature fusion based on a multi-head attention mechanism, a shared parameter layer based on the attention mechanism, and task-specific layers for venous and arterial diseases. It also defines the multi-task loss function, initializes the model hyperparameters, and sets the value range of each hyperparameter.

[0013] The multi-task learning training module is used to train and verify the model using the training set and validation set, optimize the model hyperparameters using the optimization algorithm, adjust the hyperparameters to the optimal value based on the multi-task loss function, and obtain the target multi-task deep learning model.

[0014] The positive progress effect of the present invention is:

[0015] 1) Comprehensiveness and uniformity: This invention constructs a unified multi-task deep learning model that can simultaneously assess the risk of multiple venous diseases and multiple arterial diseases (especially important arterial diseases such as lower extremity arteriosclerosis obliterans) in the same patient within a single model. This helps doctors obtain a complete picture of the patient's overall vascular health status and avoids the information fragmentation and one-sided evaluation that may result from traditional discrete diagnosis.

[0016] 2) Improved diagnostic accuracy and refinement: By integrating multimodal data (laboratory, imaging, genetic, and clinical data) and leveraging an advanced multi-task learning framework, the present invention can more deeply explore the complex relationships between data, thereby improving diagnostic accuracy. It also outputs multiple clear risk levels for various venous and arterial diseases, providing more refined and actionable information for clinical decision-making.

[0017] 3) Early risk identification and personalized intervention: Through comprehensive risk assessment, this invention helps to identify potential risks of venous and arterial vascular diseases earlier, especially those venous and arterial problems that may exist simultaneously or affect each other (comorbidities), which provides an important basis for formulating personalized prevention and treatment strategies.

[0018] 4) Assist clinical decision-making and improve diagnosis and treatment efficiency: The present invention can quickly process large amounts of complex patient data and provide clear risk assessment results. It can serve as a powerful assistant to doctors, reduce their workload, and improve diagnosis and treatment efficiency and consistency.

[0019] 5) The present invention adopts an improved gold panning algorithm to optimize the hyperparameters of the multi-task deep learning model. It has high optimization efficiency, is not prone to falling into local optimality, and has good convergence. Self-adaptive weights are added to the migration strategy, mining strategy, and collaboration strategy, which is more conducive to convergence.

[0020] 6) In the present invention, when using the improved gold panning algorithm for multi-task learning training, the gold panning algorithm is not directly initialized for population as in the prior art, and the optimal hyperparameters of the model are obtained through continuous iteration. The existing optimization process requires a large amount of data to be processed, and the optimization efficiency is not high. Instead, the multi-task deep learning model is first preliminarily trained and verified using a training set and a validation set, so as to obtain a preliminary multi-task loss function value Loss as the fitness. When Loss≤L1, it indicates that the fitness is relatively close to the optimal fitness. At this time, a smaller number of gold panners N1 can be set. When the fitness Loss>L1, it indicates that the fitness is relatively far from the optimal fitness. At this time, a larger number of gold panners N2 can be set. Thereafter, the population of the improved gold panning algorithm is initialized through the preliminarily trained hyperparameters and their variations, which can reduce the amount of data required to be processed in the optimization process of the improved gold panning algorithm and improve the optimization efficiency.

[0021] 7) In the present invention, when initializing the gold-mining group, the group is not initialized arbitrarily within the upper and lower limits as in the prior art, which is too random. Instead, the multi-task deep learning model is preliminarily trained, and the model hyperparameters after the preliminary training are used as the position of a gold digger in the gold-mining group. Then, the model hyperparameters after the preliminary training are used as the benchmark and randomly mutated to obtain the positions of the remaining N-1 gold diggers in the gold-mining group. This gold-mining group initialization method is more targeted, and the randomness achieved on a certain benchmark can more quickly find the optimal hyperparameters.

[0022] 8) In the present invention, when the improved gold panning algorithm is used for position update, unlike the existing gold panning algorithm, when the random value r is 0<r<1 / 3, the migration strategy is used for position update; when the random value r is 1 / 3≤r<2 / 3, the mining strategy is used for position update; when the random value r is 2 / 3≤r<1, the collaborative strategy is used for position update. Instead, this scheme sets an iteration coefficient τ related to the number of iterations. In the early stage of the iteration, the migration strategy or the mining strategy is used for position update. Moreover, when the optimal fitness of the group is not updated for a set number of iterations, the migration and mining combined strategy is used for position update to prevent the improved gold panning algorithm from falling into a local optimum. In the late stage of the iteration, the mining strategy or the collaborative strategy is used for position update. Moreover, when the optimal fitness of the group is not updated for a set number of iterations, the mining and collaborative combined strategy is used for position update to prevent the improved gold panning algorithm from falling into a local optimum. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 FIG. 4 is a structural block diagram of a vascular risk assessment model system according to a preferred embodiment of the present invention.

[0024] Figure 2 This is a diagram of a multi-task deep learning model for a preferred embodiment of the present invention.

[0025] Figure 3 This is a multi-layer convolutional layer diagram with a residual structure in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0027] For ease of description, only the parts related to the present invention are shown in the accompanying drawings. The first, second, etc. involved in the present invention are only for the convenience of describing the technical solution of the present invention and do not have a specific limiting effect. They are all general references and do not constitute a limiting effect on the technical solution of the present invention.

[0028] like Figure 1 As shown, this embodiment provides a vascular risk assessment model system based on multimodal fusion strategy and deep learning, including an acquisition module 1, a preprocessing module 2, a feature sample construction module 3, a model construction module 4, a multi-task learning training module 5 and a prediction module 6.

[0029] The acquisition module 1 is used to obtain the examination and test data, imaging data, genetic test data and clinical data of multiple historical patients, and to mark various vascular disease conditions. Among these historical patients, there are both those without vascular diseases and those with different types of venous and arterial vascular diseases.

[0030] 1) Inspection test data include:

[0031] Complete biochemistry set: covers liver function, kidney function, electrolytes, blood sugar, blood lipids (total cholesterol, triglycerides, high-density lipoprotein cholesterol HDL-C, low-density lipoprotein cholesterol LDL-C), etc.

[0032] Arterial disease-specific indicators: homocysteine ​​(Hcy), glycosylated hemoglobin (HbA1c), high-sensitivity C-reactive protein (hs-CRP), D-dimer, fibrinogen, etc.

[0033] Indicators related to venous diseases: D-dimer, coagulation function indicators (PT, APTT, TT, Fbg), etc.

[0034] Among the above data, some are numerical data, some are categorical data, and some are time series data (such as Hcy values ​​measured multiple times).

[0035] 2) Image data includes:

[0036] CT (computed tomography) data: such as lower limb artery CTA, pulmonary artery CTA (CTPA), abdominal and pelvic CT venous phase imaging, etc.

[0037] MRI (magnetic resonance imaging) data: such as magnetic resonance angiography (MRA), magnetic resonance venography (MRV), and MRI sequences for specific tissues (such as the brain and heart).

[0038] Ultrasound data: such as vascular Doppler ultrasound, cardiac ultrasound, etc.

[0039] 3) Genetic testing data includes:

[0040] Single nucleotide polymorphism (SNP) data: such as gene loci associated with venous thromboembolism (VTE) risk (such as F5 Leiden, F2 G20210A), gene loci associated with atherosclerosis or LEAD (such as SNPs in the 9p21 region, IPO5, HDAC9, etc.), etc.

[0041] Gene expression profile data.

[0042] 4) Clinical data include:

[0043] Patient basic information: age, gender, and body mass index (BMI).

[0044] Lifestyle factors: smoking history, drinking history, and exercise habits.

[0045] Complications: hypertension, diabetes, hyperlipidemia, malignant tumors, etc.

[0046] Medication history.

[0047] The doctor's initial diagnosis or clinical impression notes of the patient.

[0048] Among the above data, some are numerical data, some are categorical data, and some are text data.

[0049] The preprocessing module 2 is used to process the acquired raw data to make it suitable for subsequent models.

[0050] 1) Check the test data processing and obtain the test characteristics:

[0051] Numerical data: standardization (such as z-score standardization).

[0052] Categorical data: one-hot encoding or entity embedding.

[0053] Time series data: Use time series embedding methods (such as RNN / LSTM-based embedding) to convert it into a fixed-length vector.

[0054] 2) Image data processing to obtain image features:

[0055] Imaging features: Image segmentation technology is used to segment the image data into regions of interest (ROIs) (such as blood vessels, plaques, and thrombi), and high-throughput quantitative features describing the texture, shape, intensity, etc. of the lesion area are extracted from them to form imaging features.

[0056] Deep image features: A deep learning-based feature extractor (using pre-trained CNN or 3D CNN as a feature extractor) is used to learn deep image features from image data.

[0057] 3) Genetic testing data processing to obtain genetic characteristics:

[0058] SNPs data: Usually encoded as 0, 1, 2 (representing homozygous wild type, heterozygous type, and homozygous variant type). Due to the high dimensionality of SNPs data, dimensionality reduction (such as singular value decomposition (SVD)) or a learnable embedding layer is required to map it to a low-dimensional space.

[0059] Gene expression profile data: normalization and feature selection.

[0060] 4) Clinical data processing to obtain clinical characteristics:

[0061] Numerical data: standardization (such as z-score standardization).

[0062] Categorical data: one-hot encoding or entity embedding.

[0063] Text data: Use word embedding (such as Word2Vec) and NLP technology to extract features to obtain text features.

[0064] The feature sample construction module 3 is used to construct a training set and a validation set using the multimodal pre-processed features (laboratory features, imaging features, gene features, and clinical features) of each historical patient.

[0065] The model building module 4 is used to build a multi-task deep learning model, including an input layer, a multi-layer convolutional layer with a residual structure, a fusion layer for feature weighted fusion based on a multi-head attention mechanism, a shared parameter layer based on an attention mechanism, and a task-specific layer for venous and arterial diseases (see Figure 2 ), define the multi-task loss function, initialize the model hyperparameters and set the value range of each hyperparameter.

[0066] Among them, the multi-layer convolutional layer with residual structure includes: a first convolutional layer composed of a convolutional layer and a pooling layer, a second convolutional layer composed of two convolutional layers, a third convolutional layer composed of two convolutional layers, a fourth convolutional layer composed of two convolutional layers, and a fully connected layer (see Figure 3 ).

[0067] The two convolutional layers in the second convolutional layer form a first residual structure through a jump connection line, the two convolutional layers in the third convolutional layer form a second residual structure through a jump connection line, and the two convolutional layers in the fourth convolutional layer form a third residual structure through a jump connection line.

[0068] The number of convolution kernels in the two convolution layers of each of the second, third, and fourth convolution layers is the same. The number of convolution kernels in the first, second, third, and fourth convolution layers decreases successively. The number of convolution kernels in the first and second convolution layers is in a multiple relationship, the number of convolution kernels in the second and third convolution layers is in a multiple relationship, and the number of convolution kernels in the third and fourth convolution layers is in a multiple relationship.

[0069] The fusion layer for feature weighted fusion based on the multi-head attention mechanism includes a multi-head attention layer and a fully connected layer.

[0070] A shared parameter layer based on the attention mechanism is used to learn common pathophysiological features and risk factors among different vascular diseases.

[0071] Task-specific layers for venous and arterial diseases: These layers are used to predict the risk of various venous diseases (such as deep vein thrombosis, pulmonary embolism, and varicose veins) and arterial diseases (such as lower extremity arteriosclerosis obliterans, carotid atherosclerosis, and coronary atherosclerosis). After the shared parameter layer, the network branches into multiple parallel sub-networks or output heads, each of which corresponds to a specific prediction task.

[0072] Venous disease tasks: such as risk prediction heads for deep vein thrombosis (DVT), risk prediction heads for pulmonary embolism (PE), risk prediction heads for lower limb venous insufficiency, etc.

[0073] Arterial disease tasks: such as risk prediction heads for lower limb arteriosclerosis obliterans (LEAD), prediction heads for carotid atherosclerotic plaque stability, and prediction heads for future event risks of coronary artery disease (CAD).

[0074] The multi-task learning training module 5 is used to train and verify the model using the training set and the validation set, optimize the model hyperparameters using the optimization algorithm, adjust the hyperparameters to the optimal value based on the multi-task loss function, and obtain the target multi-task deep learning model.

[0075] Among them, the optimization algorithm: adds adaptive weights to the migration strategy, mining strategy and collaboration strategy in the gold mining algorithm, creates a migration and mining combination strategy that combines the migration strategy and the mining strategy, and creates a mining and collaboration combination strategy that combines the mining strategy and the collaboration strategy, and constructs an improved gold mining algorithm.

[0076] Migration mining combined strategy:

[0077]

[0078] Mining collaborative integration strategy:

[0079]

[0080] In combination with strategy,

[0081] In the above formula, represents the new position of the gold digger i, Indicates the current location of the gold digger i, represents the migration vector, A1 and A2 represent the variable coefficients, represents the current position of the randomly selected gold digger g, represents the mining vector, represents the collaboration vector, r1, r2, and r3 are all random numbers between 0 and 1, W(t) represents the adaptive weight, and W min represents the minimum weight, W max represents the maximum weight, t represents the current number of iterations, and T represents the maximum number of iterations.

[0082] This solution uses an improved gold panning algorithm to optimize the hyperparameters of multi-task deep learning models. This algorithm boasts high optimization efficiency, is less susceptible to local optima, and exhibits excellent convergence. The inclusion of adaptive weights W(t) in the migration, mining, and collaboration strategies further facilitates convergence. The adaptive weight W(t) decreases as the number of iterations t increases.

[0083] The multi-task learning training module 5 is used to input the initialization model hyperparameters, use the training set and the validation set to perform preliminary training and verification on the multi-task deep learning model, obtain the preliminary multi-task loss function L = the weighted sum of the loss functions of each single task, where the weight is adjusted according to the importance of the task or the convergence speed of the model, analyze the multi-task loss function Loss and the preset threshold L1, and when Loss ≤ L1, set the number of gold diggers to N1, and when Loss>L1, set the number of gold diggers to N2, N1<N2.

[0084] The multi-task learning training module 5 is also used to initialize each gold digger in the improved gold panning algorithm using the values ​​of the model hyperparameters (hyperparameters after preliminary training and verification) and their randomly mutated hyperparameter values. The improved gold panning algorithm is used to continuously optimize the hyperparameters to obtain the optimal hyperparameters. The optimal model constructed with the optimal hyperparameters is used as the target multi-task deep learning model. Specifically:

[0085] 1) At this point, after initial training, the model hyperparameter value is H1. Each gold miner in the gold mining group represents a set of hyperparameters. H1 is used to initialize the position of a gold miner. H1 is randomly mutated within the range of hyperparameter values ​​to initialize the positions of the other N-1 gold miners. N is the number of gold miners in the gold mining group, N is N1 or N2, and T is the maximum number of iterations.

[0086] 2) Substitute the hyperparameters corresponding to each gold miner in 1) into the multi-task deep learning model and train the model using the training set. Obtain the multi-task loss function L for each gold miner. The minimum value is the optimal fitness of the group, and the current position of the gold miner corresponding to the optimal group fitness is the optimal gold mine location. Each gold miner's multi-task loss function L is its current fitness, and its current fitness is its individual optimal fitness.

[0087] 3) When 0.5<τ<1, the migration strategy or mining strategy is used to update the position, and when the optimal fitness of the group corresponding to the preset number of iterations (such as two times) is not updated, the migration and mining combined strategy is used to update the position; when τ≤0.5, the mining strategy or collaboration strategy is used to update the position, and when the optimal fitness of the group corresponding to the preset number of iterations (such as two times) is not updated, the mining and collaboration combined strategy is used to update the position; τ is the iteration coefficient that gradually decreases with the increase of the number of iterations,

[0088] The hyperparameters corresponding to the updated positions of each gold digger are substituted into the corresponding multi-task deep learning model and trained separately using the training set to obtain the multi-task loss function L corresponding to each gold digger as the current fitness. The current fitness of each gold digger is compared with its individual best fitness. When the current fitness of the gold digger is better than its individual best fitness, the individual best fitness of the gold digger is updated to be the current fitness of the gold digger, and the position of the gold digger is updated, otherwise it is not updated; the minimum value of the individual best fitness of each gold digger is taken as the current group best fitness. If the current group best fitness is better than the group best fitness, the group best fitness is updated to be the current group best fitness, and the current position of the gold digger corresponding to the current group best fitness is taken as the best gold mine position, otherwise it is not updated.

[0089] 4) Determine whether the optimal fitness of the group reaches the preset threshold L2 or the number of iterations reaches the maximum number of iterations T. If so, the optimal gold mine location is used as the optimal hyperparameter, and the optimal model constructed by the optimal hyperparameter is used as the target multi-task deep learning model. L2 < L1, otherwise enter 3) for the next iteration.

[0090] When initializing the gold-mining group, this solution does not randomly initialize the group within upper and lower limits like the existing technology. Instead, it first conducts preliminary training on the multi-task deep learning model and uses the model hyperparameters after preliminary training as the position of a gold digger in the gold-mining group. Then, using the model hyperparameters after preliminary training as the benchmark, it is randomly mutated to obtain the positions of the remaining N-1 gold diggers in the gold-mining group. This gold-mining group initialization method is more targeted and can quickly find the optimal hyperparameters.

[0091] When using the improved gold-mining algorithm for position updates, this solution does not use the migration strategy for position updates when the random value r is 0<r<1 / 3; the mining strategy for position updates when the random value r is 1 / 3≤r<2 / 3; or the collaborative strategy for position updates when the random value r is 2 / 3≤r<1, as in the existing gold-mining algorithm. Instead, this solution sets an iteration coefficient τ related to the number of iterations. In the early iteration phase, the migration strategy or the mining strategy is used for position updates. If the optimal fitness of the group has not been updated for a set number of iterations, the migration and mining strategy is used for position updates to prevent the improved gold-mining algorithm from falling into a local optimum. In the late iteration phase, the mining strategy or the collaborative strategy is used for position updates. If the optimal fitness of the group has not been updated for a set number of iterations, the mining and collaborative strategy is used to prevent the improved gold-mining algorithm from falling into a local optimum.

[0092] After receiving the prediction instruction, the system calls the acquisition module 1, the preprocessing module 2 and the prediction module 6 in sequence; the acquisition module 1 is used to obtain the examination and test data, imaging data, genetic test data and clinical data of the person to be predicted; the preprocessing module 2 is used to preprocess these data to obtain multimodal features, and the multimodal features include laboratory features, imaging features, genetic features and clinical features; the prediction module 6 is used to input the multimodal features into the target multi-task deep learning model, extract features through multiple convolutional layers, fuse the multimodal features through the fusion layer to generate a unified state vector, share the state vector through the shared parameter layer, and predict the risks of multiple venous and arterial diseases through the task-specific layer, and output the risk level of at least one venous vascular disease and at least one arterial vascular disease of the person to be predicted.

[0093] In this solution, for each target disease, multiple different and clear risk levels (such as low risk, medium risk, high risk, or specific probability values) can be output.

[0094] Although specific embodiments of the present invention have been described above, those skilled in the art will appreciate that these are merely illustrative and that the scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, and such changes and modifications are intended to fall within the scope of the present invention.

Claims

1. A vascular risk assessment model system based on multimodal fusion strategy and deep learning, characterized by: include: An acquisition module is used to acquire the examination and laboratory data, imaging data, genetic testing data, and clinical data of multiple historical patients and to annotate various vascular disease conditions. The examination and laboratory data include numerical, categorical, and / or time-series data; the clinical data include text, numerical, and / or categorical data; and the genetic testing data include SNPs data and / or gene expression profile data. Preprocessing module, used to standardize numerical data, perform one-hot encoding or entity embedding on categorical data, convert time series data into fixed-length vectors using time series embedding methods, segment image data using image segmentation techniques to segment regions of interest and extract imaging features from them, and / or use deep learning-based feature extractors to learn deep image features from image data, extract features from text data using word embedding and NLP techniques to obtain text features, perform dimensionality reduction on SNPs data to obtain gene features, and perform normalization and feature selection on gene expression data to obtain gene features; Feature sample construction module, used to construct training sets and validation sets using multimodal pre-processed features of each historical patient; The model building module is used to build a multi-task deep learning model, including an input layer, a multi-layer convolutional layer with a residual structure, a fusion layer for weighted feature fusion based on a multi-head attention mechanism, a shared parameter layer based on the attention mechanism, and task-specific layers for venous and arterial diseases. It also defines the multi-task loss function, initializes the model hyperparameters, and sets the value range of each hyperparameter. The multi-task learning training module is used to train and verify the model using the training set and validation set, optimize the model hyperparameters using the optimization algorithm, adjust the hyperparameters to the optimal value based on the multi-task loss function, and obtain the target multi-task deep learning model.

2. The vascular risk assessment model system based on multimodal fusion strategy and deep learning as claimed in claim 1, characterized in that: The system further comprises a prediction module, which, upon receiving a prediction instruction, sequentially calls an acquisition module, a preprocessing module and a prediction module; The acquisition module is used to obtain the examination and laboratory data, imaging data, genetic testing data and clinical data of the person to be predicted; The preprocessing module is used to preprocess the data to obtain multimodal features, which include laboratory features, imaging features, genetic features and clinical features; The prediction module is used to input multimodal features into a target multi-task deep learning model, perform feature extraction through multiple convolutional layers, fuse the multimodal features through a fusion layer to generate a unified state vector, perform shared learning on the state vector through a shared parameter layer, perform risk prediction of multiple venous and arterial diseases through a task-specific layer, and output the risk level of at least one venous disease and at least one arterial disease of the person to be predicted.

3. The vascular risk assessment model system based on multimodal fusion strategy and deep learning as claimed in claim 1, characterized in that: Optimization algorithm: Adaptive weights are added to the migration strategy, mining strategy, and collaboration strategy in the gold panning algorithm to create a migration-mining strategy that combines the migration strategy with the mining strategy, and a mining-collaboration strategy that combines the mining strategy with the collaboration strategy, thus constructing an improved gold panning algorithm. The multi-task learning training module is used to input the initialization model hyperparameters, use the training set and the validation set to perform preliminary training and verification on the multi-task deep learning model, obtain the preliminary multi-task loss function L=the weighted sum of the loss functions of each single task, where the weight is adjusted according to the task importance or the model convergence speed, analyze the multi-task loss function Loss and the preset threshold L1, when Loss≤L1, set the number of gold diggers to N1, when Loss>L1, set the number of gold diggers to N2, N1<N2, use the values ​​of the model hyperparameters at this time and the values ​​of the randomly mutated hyperparameters to initialize each gold digger in the improved gold panning algorithm, use the improved gold panning algorithm to continuously optimize the hyperparameters for training to obtain the optimal hyperparameters, and the optimal model constructed by the optimal hyperparameters is used as the target multi-task deep learning model.

4. The vascular risk assessment model system based on multimodal fusion strategy and deep learning as claimed in claim 3, characterized in that: The multi-task learning training module is used to initialize each gold digger in the improved gold panning algorithm using the values ​​of the model hyperparameters at this time and the values ​​of the randomly mutated hyperparameters. The improved gold panning algorithm is used to continuously optimize the hyperparameters to obtain the optimal hyperparameters. The optimal model constructed with the optimal hyperparameters is used as the target multi-task deep learning model: 1) After initial training, the model hyperparameter value is H1. Each digger in the digger group represents a set of hyperparameters. H1 is used to initialize the position of a digger. H1 is randomly mutated within the range of hyperparameter values ​​to initialize the positions of the other N-1 diggers. N is the number of groups, N is N1 or N2, and T is the maximum number of iterations. 2) Substitute the hyperparameters corresponding to each gold prospector in 1) into the multi-task deep learning model and train the model using the training set to obtain the multi-task loss function L corresponding to each gold prospector. The minimum value is used as the optimal fitness of the group, and the current position of the gold prospector corresponding to the optimal fitness of the group is used as the optimal gold mine location; 3) When 0.5<τ<1, the migration strategy or mining strategy is used to update the position. When τ≤0.5, the mining strategy or collaborative strategy is used to update the position. τ is the iteration coefficient that gradually decreases with the increase of the number of iterations. Substitute the hyperparameters corresponding to the position of each gold digger to be updated into the corresponding multi-task deep learning model and train them separately using the training set to obtain the multi-task loss function L corresponding to each gold digger as the current fitness. Compare the current fitness of each gold digger with its individual best fitness. When the current fitness of the gold digger is better than its individual best fitness, update the individual best fitness of the gold digger to = the current fitness of the gold digger, and update the position of the gold digger. Otherwise, do not update. Then, the minimum value of the individual best fitness of each gold digger is used as the current group best fitness. If the current group best fitness is better than the group best fitness, update the group best fitness to = the current group best fitness. The current position of the gold digger corresponding to the current group best fitness is used as the best gold mine position. Otherwise, do not update. 4) Determine whether the optimal fitness of the group reaches the preset threshold L2 or the number of iterations reaches the maximum number of iterations T. If so, the optimal gold mine location is used as the optimal hyperparameter, and the optimal model constructed by the optimal hyperparameter is used as the target multi-task deep learning model. L2 < L1, otherwise enter 3) for the next iteration.

5. The vascular risk assessment model system based on multimodal fusion strategy and deep learning according to claim 4, characterized in that: In 3), when 0.5<τ<1, the migration strategy or mining strategy is used to update the position, and when the optimal fitness of the group corresponding to the preset number of consecutive iterations is not updated, the migration and mining combined strategy is used to update the position; When τ≤0.5, the mining strategy or the collaborative strategy is used to update the position, and when the optimal fitness of the group corresponding to the preset number of consecutive iterations is not updated, the mining and collaborative strategy is used to update the position.

6. The vascular risk assessment model system based on multimodal fusion strategy and deep learning according to claim 4, characterized in that: Iteration coefficient 7. The vascular risk assessment model system based on multimodal fusion strategy and deep learning as claimed in claim 1, characterized in that: The multi-layer convolutional layers with residual structure include: the first convolutional layer consisting of a convolutional layer and a pooling layer, the second convolutional layer consisting of two convolutional layers, the third convolutional layer consisting of two convolutional layers, the fourth convolutional layer consisting of two convolutional layers, and the fully connected layer; The two convolutional layers in the second convolutional layer form a first residual structure through a jump connection line, the two convolutional layers in the third convolutional layer form a second residual structure through a jump connection line, and the two convolutional layers in the fourth convolutional layer form a third residual structure through a jump connection line; Among them, the number of convolution kernels in the two convolution layers of each of the second convolution layer, the third convolution layer and the fourth convolution layer is the same, and the number of convolution kernels in the first convolution layer, the second convolution layer, the third convolution layer and the fourth convolution layer decreases successively. The number of convolution kernels in the first convolution layer and the second convolution layer is in a multiple relationship, the number of convolution kernels in the second convolution layer and the third convolution layer is in a multiple relationship, and the number of convolution kernels in the third convolution layer and the fourth convolution layer is in a multiple relationship.

8. The vascular risk assessment model system based on multimodal fusion strategy and deep learning as claimed in claim 1, characterized in that: The fusion layer for feature weighted fusion based on the multi-head attention mechanism includes a multi-head attention layer and a fully connected layer.

9. The vascular risk assessment model system based on multimodal fusion strategy and deep learning as claimed in claim 1, characterized in that: The imaging data includes CT data, MRI data and / or ultrasound data; Clinical data included basic patient information, lifestyle factors, comorbidities, medication history, and preliminary diagnosis information.

10. The vascular risk assessment model system based on multimodal fusion strategy and deep learning as claimed in claim 1, characterized in that: Use pre-trained CNN or 3D CNN as feature extractor; Use singular value decomposition (SVD) to reduce the dimensionality of SNPs data, or use a learnable embedding layer to map SNPs data to a low-dimensional space.

Images