Vascular risk assessment model system based on multi-modal fusion strategy and deep learning

Abstract

The application is based on a blood vessel risk assessment model system based on a multi-modal fusion strategy and deep learning: an acquisition module acquires examination and test data, image data, gene detection data and clinical data of a plurality of historical patients, and labels various blood vessel disease conditions; a preprocessing module standardizes numerical data, one-hot encodes or entity embedding of category data, converts time series data into fixed length vectors, extracts imaging features from image data and / or learns deep image features from the image data, extracts features from text data, reduces SNPs data, normalizes gene expression profile data and selects features; a feature sample construction module constructs a training set and a validation set using the multi-modal preprocessed features of each historical patient; a model construction module constructs a multi-task deep learning model; a multi-task learning training module trains and validates the model, optimizes the hyperparameters of the model using an optimization algorithm, and adjusts the hyperparameters to the optimal to obtain a target multi-task deep learning model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, multi-modal data processing and model construction, in particular to a blood vessel risk assessment model system based on multi-modal fusion strategy and deep learning. The present application is derived from the project: Beijing Municipal Natural Science Foundation: "Exploration of immunological characteristics and potential intervention targets of aorto-iliac arteriosclerosis obliterans", number: 7252061. BACKGROUND

[0002] Vascular diseases mainly include venous vascular diseases (such as deep vein thrombosis, chronic venous insufficiency, etc.) and arterial vascular diseases (such as lower extremity arteriosclerosis obliterans, coronary artery disease, cerebrovascular disease, etc.). Currently, the risk assessment of venous vascular diseases and arterial vascular diseases is often carried out independently in clinical practice. One technology can only assess the risk of venous vascular diseases or arterial vascular diseases, and there is a lack of technology that can comprehensively assess the risk of venous vascular diseases and arterial vascular diseases for the same patient.

[0003] Existing diagnostic methods usually rely on single or limited combination of examination means, for example, ultrasonography is mainly used for venous diseases, and angiography or CT angiography (CTA) is more commonly used for arterial diseases. Doctors need to analyze the results from different sources (such as laboratory reports, imaging reports, genetic testing reports, etc.) and the clinical manifestations and medical history of patients to make a diagnosis. This analysis process not only consumes time, but also requires high experience and professional knowledge of doctors, which is prone to subjectivity and diagnostic differences.

[0004] In recent years, artificial intelligence (AI) technology, especially machine learning, has shown great potential in medical image analysis and disease diagnosis. Some studies have applied AI to the diagnosis of specific vascular diseases, such as using AI to analyze ultrasonography images to diagnose deep vein thrombosis, or using AI to analyze CTA images to assess the degree of arterial stenosis. However, these models are mostly aimed at a single disease or a single data modality, making it difficult to comprehensively assess the overall vascular health of a patient.

[0005] Therefore, there is an urgent need to develop an intelligent assessment system that can integrate multi-modal patient data (including examination laboratory, CT, MRI, genetic testing, etc.) and assess the risk of multiple venous and arterial vascular diseases to assist doctors in making more comprehensive, accurate and efficient vascular disease diagnosis and risk stratification. SUMMARY

[0006] The present application provides a blood vessel risk assessment model system based on multi-modal fusion strategy and deep learning, which can comprehensively and finely assess the risk of multiple venous and arterial vascular diseases for the same patient, to solve the problems and deficiencies of the limitations of existing vascular disease diagnosis and assessment technology.

[0007] The present application solves the above technical problems by the following technical solutions:

[0008] The present application provides a blood vessel risk assessment model system based on a multi-modal fusion strategy and deep learning, comprising:

[0009] An acquisition module is configured to acquire examination and laboratory data, image data, gene detection data and clinical data of a plurality of historical patients, and perform various blood vessel disease condition annotations, wherein the examination and laboratory data comprises numerical, categorical and / or time series data, the clinical data comprises text, numerical and / or categorical data, and the gene detection data comprises SNPs data and / or gene expression profile data;

[0010] A preprocessing module is configured to normalize the numerical data, perform one-hot encoding or entity embedding on the categorical data, convert the time series data into a fixed length vector using a time series embedding method, segment the region of interest from the image data using an image segmentation technique and extract image features therefrom, and / or learn image deep features from the image data using a deep learning-based feature extractor, extract text features from the text data using word embedding and NLP techniques, obtain gene features by reducing the dimension of the SNPs data, and obtain gene features by normalizing and selecting features from the gene expression profile data;

[0011] A feature sample construction module is configured to construct a training set and a validation set using the multi-modal preprocessed features of the historical patients;

[0012] A model construction module is configured to construct a multi-task deep learning model comprising an input layer, a multi-layer convolutional layer with a residual structure, a fusion layer for feature weighted fusion based on a multi-head attention mechanism, a shared parameter layer based on an attention mechanism, and a task-specific layer for venous and arterial blood vessel diseases, define a multi-task loss function, initialize model hyperparameters and set the value range of each hyperparameter;

[0013] A multi-task learning training module is configured to train and validate the model using the training set and the validation set, optimize the model hyperparameters using an optimization algorithm, adjust the hyperparameters to the optimal based on the multi-task loss function, and obtain a target multi-task deep learning model.

[0014] The positive progress effect of the present application is that:

[0015] 1) Comprehensive and unified: The present application constructs a unified multi-task deep learning model, which can simultaneously assess the risk of multiple venous vascular diseases and multiple arterial vascular diseases (especially important arterial diseases such as lower extremity arteriosclerosis obliterans (LEAD)) in a single model, helping doctors to obtain the overall vascular health status of patients and avoiding the information fragmentation and one-sided evaluation that may be caused by traditional separate diagnosis.

[0016] 2) Improved diagnostic accuracy and refinement: The present application can improve the accuracy of diagnosis by integrating multi-modal data (laboratory data, image data, genetic data, and clinical data) and using an advanced multi-task learning framework to more deeply mine the complex relationships between data. At the same time, it outputs multiple clear risk levels for multiple venous and arterial diseases, providing more refined and operational information for clinical decision-making.

[0017] 3) Early risk identification and personalized intervention: The present application helps to identify potential venous and arterial vascular disease risks earlier through comprehensive risk assessment, especially those that may coexist or interact with each other (comorbidity), which provides an important basis for developing personalized prevention and treatment strategies.

[0018] 4) Assist clinical decision-making and improve diagnosis and treatment efficiency: The present application can quickly process large amounts of complex patient data and provide clear risk assessment results, which can serve as a powerful assistant for doctors, reducing their workload and improving diagnosis and treatment efficiency and consistency.

[0019] 5) The present application uses an improved gold panning algorithm to optimize the hyperparameters of the multi-task deep learning model, which has high optimization efficiency, is not easy to fall into local optimum, and has good convergence. In the migration strategy, mining strategy and cooperation strategy, adaptive weights are added to facilitate convergence.

[0020] 6) In the present application, when using the improved gold panning algorithm for multi-task learning training, unlike the existing technology that directly initializes the population of the gold panning algorithm, the optimal hyperparameters of the model are obtained through continuous iteration. The amount of data processed in the existing optimization process is large, and the optimization efficiency is not high. Instead, the training set and the validation set are used to preliminarily train and verify the multi-task deep learning model, so as to obtain a preliminary multi-task loss function value Loss as the fitness. When Loss≤L1, it indicates that the fitness is relatively close to the best fitness. At this time, a smaller number of gold diggers N1 can be set. When the fitness Loss>L1, it indicates that the fitness is relatively far from the best fitness. At this time, a larger number of gold diggers N2 can be set. Thereafter, the improved gold panning algorithm is initialized by the hyperparameters obtained by preliminary training and their mutations, which can reduce the amount of data processed in the optimization process of the improved gold panning algorithm and improve the optimization efficiency.

[0021] 7) In the present application, when initializing the gold panning group, unlike the prior art, the group is initialized in the upper and lower limit range very randomly, too random, but first the multi-task deep learning model is preliminarily trained, the model hyperparameters after preliminary training are used as the position of one gold panner in the gold panning group, and then the model hyperparameters after preliminary training are used as the benchmark to randomly mutate to obtain the positions of the remaining N-1 gold panners in the gold panning group. This kind of gold panning group initialization method is more targeted, realizes randomness on a certain benchmark, and can find the optimal hyperparameters more quickly.

[0022] 8) In the present application, when updating the position by using the improved gold panning algorithm, unlike the existing gold panning algorithm, when the random value r takes a value of 0 BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The structure block diagram of the blood vessel risk assessment model system of the preferred embodiment of the present application.

[0024] Figure 2 The multi-task deep learning model diagram of the preferred embodiment of the present application.

[0025] Figure 3 The multi-layer convolutional layer diagram with residual structure of the preferred embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0027] For the convenience of description, only the parts related to the present application are shown in the drawings. The first, second, etc. involved in the present application are only for the convenience of describing the technical solutions of the present application and do not have a specific limiting effect, all of which are generic and do not constitute a limiting effect on the technical solutions of the present application.

[0028] As shown in Figure 1 The embodiment provides a blood vessel risk assessment model system based on a multi-modal fusion strategy and deep learning, which comprises an acquisition module 1, a preprocessing module 2, a feature sample construction module 3, a model construction module 4, a multi-task learning training module 5 and a prediction module 6.

[0029] The acquisition module 1 is used for acquiring examination and laboratory data, image data, gene detection data and clinical data of a plurality of historical patients, and performing various blood vessel disease condition labeling, wherein the historical patients include both personnel without blood vessel diseases and personnel with different kinds of arterial and venous blood vessel diseases.

[0030] 1) The examination and laboratory data comprises:

[0031] Complete biochemical set: covering liver function, kidney function, electrolyte, blood sugar, blood lipid (total cholesterol, triglyceride, high-density lipoprotein cholesterol HDL-C, low-density lipoprotein cholesterol LDL-C) and the like.

[0032] Arterial disease specific indicators: homocysteine (Hcy), glycosylated hemoglobin (HbA1c), high-sensitivity C-reactive protein (hs-CRP), D-dimer, fibrinogen and the like.

[0033] Venous disease related indicators: D-dimer, coagulation function indicators (PT, APTT, TT, Fbg) and the like.

[0034] Among the above data, some are numerical data, some are categorical data, and some are time series data (such as multiple measurements of Hcy value).

[0035] 2) The image data comprises:

[0036] CT (computed tomography) data: such as lower extremity arterial CTA, pulmonary arterial CTA (CTPA), abdominal and pelvic CT venous phase imaging and the like.

[0037] MRI (magnetic resonance imaging) data: such as magnetic resonance angiography (MRA), magnetic resonance venography (MRV), MRI sequences for specific tissues (such as brain, heart) and the like.

[0038] Ultrasound data: such as vascular Doppler ultrasound, cardiac ultrasound and the like.

[0039] 3) The gene detection data comprises:

[0040] Single nucleotide polymorphism (SNPs) data: genetic loci associated with venous thromboembolism (VTE) risk (e.g. F5 Leiden, F2 G20210A), genetic loci associated with atherosclerosis or LEAD (e.g. SNPs in the 9p21 region, IPO5, HDAC9, etc.), and the like.

[0041] Gene expression profile data.

[0042] 4) Clinical data includes:

[0043] Patient basic information: age, gender, body mass index (BMI).

[0044] Lifestyle factors: smoking history, drinking history, exercise habits.

[0045] Comorbidities: hypertension, diabetes, hyperlipidemia, malignancy, and the like.

[0046] Medication history.

[0047] Doctor's preliminary diagnosis or clinical impression notes for the patient.

[0048] Among the above data, some are numerical data, some are categorical data, and some are text data.

[0049] The preprocessing module 2 is used to process the obtained raw data to make it suitable for subsequent models.

[0050] 1) Check the laboratory data processing to obtain the laboratory features:

[0051] Numerical data: standardization processing (such as z-score standardization).

[0052] Categorical data: one-hot encoding or entity embedding.

[0053] Time series data: convert it to a fixed length vector using time series embedding methods (such as RNN / LSTM-based embedding).

[0054] 2) Image data processing to obtain image features:

[0055] Imaging features: image segmentation techniques are used to segment the image data to obtain the region of interest (ROI) (such as blood vessels, plaques, thrombus), and high-throughput quantitative features describing the texture, shape, intensity, etc. of the lesion area are extracted from it to form imaging features.

[0056] Deep image features: Deep image features are learned from image data using a deep learning based feature extractor (using a pre-trained CNN or 3D CNN as the feature extractor).

[0057] 3) Gene detection data processing, obtain gene features:

[0058] SNPs data: Usually encoded as 0, 1, 2 (representing homozygous wild type, heterozygous, homozygous mutant), due to the high dimensionality of SNPs data, dimensionality reduction (such as singular value decomposition SVD) or using a learnable embedding layer to map it to a low-dimensional space is needed.

[0059] Gene expression profile data: Normalization and feature selection are performed.

[0060] 4) Clinical data processing, obtain clinical features:

[0061] Numerical data: Standardization (such as z-score standardization).

[0062] Categorical data: One-hot encoding or entity embedding.

[0063] Text data: Use word embedding (such as Word2Vec) and NLP techniques to extract features to obtain text features.

[0064] The feature sample construction module 3 is used to construct training set and validation set using the multi-modal pre-processed features (laboratory features, image features, gene features, clinical features) of each historical patient.

[0065] The model construction module 4 is used to construct a multi-task deep learning model, including an input layer, a multi-layer convolutional layer with a residual structure, a fusion layer based on a multi-head attention mechanism for feature weighted fusion, a shared parameter layer based on an attention mechanism, and a task-specific layer for venous and arterial vascular diseases (see Figure 2 ), define the multi-task loss function, initialize the model hyperparameters and set the value range of each hyperparameter.

[0066] Among them, the multi-layer convolutional layer with a residual structure includes: a first convolutional layer composed of a convolutional layer and a pooling layer, a second convolutional layer composed of two convolutional layers, a third convolutional layer composed of two convolutional layers, a fourth convolutional layer composed of two convolutional layers, and a fully connected layer (see Figure 3 ).

[0067] The two convolutional layers in the second convolutional layer form a first residual structure through a jump connection line, the two convolutional layers in the third convolutional layer form a second residual structure through a jump connection line, and the two convolutional layers in the fourth convolutional layer form a third residual structure through a jump connection line.

[0068] The number of convolution kernels of each of the two convolution layers in the second convolution layer, the third convolution layer and the fourth convolution layer is the same, the number of convolution kernels of the first convolution layer, the second convolution layer, the third convolution layer and the fourth convolution layer decreases in turn, the number of convolution kernels of the first convolution layer and the second convolution layer is in a multiple relationship, the number of convolution kernels of the second convolution layer and the third convolution layer is in a multiple relationship, and the number of convolution kernels of the third convolution layer and the fourth convolution layer is in a multiple relationship.

[0069] The fusion layer for feature weighting fusion based on the multi-head attention mechanism includes a multi-head attention layer and a full connection layer.

[0070] The shared parameter layer based on the attention mechanism is used to learn the common pathophysiological characteristics and risk factors among different vascular diseases.

[0071] Task-specific layers for venous and arterial vascular diseases: used for risk prediction for multiple venous vascular diseases (such as deep vein thrombosis, pulmonary embolism, lower extremity varicose veins, etc.) and multiple arterial vascular diseases (such as lower extremity arteriosclerosis obliterans, carotid atherosclerosis, coronary atherosclerosis, etc.) respectively. After the shared parameter layer, multiple parallel sub-networks or output heads are branched out, and each sub-network / output head corresponds to a specific prediction task.

[0072] Venous disease tasks: such as a risk prediction head for deep vein thrombosis (DVT), a risk prediction head for pulmonary embolism (PE), a risk prediction head for lower extremity venous insufficiency, etc.

[0073] Arterial disease tasks: such as a risk prediction head for lower extremity arteriosclerosis obliterans (LEAD), a prediction head for carotid atherosclerotic plaque stability, a prediction head for future event risk of coronary artery disease (CAD), etc.

[0074] The multi-task learning training module 5 is used to train and verify the model using the training set and the verification set, to perform hyperparameter optimization on the model using an optimization algorithm, to adjust the hyperparameters to be optimal based on a multi-task loss function, and to obtain a target multi-task deep learning model.

[0075] Among them, the optimization algorithm: the migration strategy, the mining strategy and the cooperation strategy in the gold panning algorithm are all added with adaptive weights, a migration and mining combined strategy combining the migration strategy and the mining strategy is created, a mining and cooperation combined strategy combining the mining strategy and the cooperation strategy is created, and an improved gold panning algorithm is constructed.

[0076] Migration and mining combined strategy:

[0077]

[0078] Mining and cooperation combined strategy:

[0079]

[0080] In the strategy,

[0081] In the above formula, represents the new position of the gold digger i, represents the current position of the gold digger i, represents the migration vector, A1, A2 represents the variable coefficient, represents the current position of the randomly selected gold digger g, represents the exploitation vector, represents the cooperation vector, r1, r2, r3 are random numbers between 0 and 1, W(t) represents the adaptive weight, W min represents the minimum weight, W max represents the maximum weight, t represents the current iteration number, T represents the maximum iteration number.

[0082] In this scheme, the improved gold digger algorithm is used to optimize the hyperparameters of the multi-task deep learning model, which has high optimization efficiency, is not easy to fall into local optimum, and has good convergence. Among them, the adaptive weight W(t) is added in the migration strategy, exploitation strategy and cooperation strategy, which is more conducive to convergence. The value of the adaptive weight W(t) gradually decreases with the increase of the iteration number t.

[0083] The multi-task learning training module 5 is used to input the initialized model hyperparameters, and the training set and the validation set are used to preliminarily train and verify the multi-task deep learning model, and obtain a preliminary multi-task loss function L = the weighted sum of each single task loss function, wherein the weight is adjusted according to the task importance or the model convergence speed, and the multi-task loss function Loss is analyzed and compared with the preset threshold L1. When Loss≤L1, the number of gold diggers is set to N1, and when Loss>L1, the number of gold diggers is set to N2, N1<N2.

[0084] The multi-task learning training module 5 is also used to initialize each gold digger in the improved gold digger algorithm using the value of the model hyperparameters (the hyperparameters after preliminary training and verification) and the hyperparameter values randomly varied therefrom, and to continuously optimize and train the hyperparameters using the improved gold digger algorithm to obtain the optimal hyperparameters. The optimal model constructed by the optimal hyperparameters is used as the target multi-task deep learning model. Specifically:

[0085] 1) The value of the model hyperparameters after preliminary training at this time is H1, each gold digger in the gold digger population represents a group of hyperparameters, H1 is used to initialize the position of a gold digger, and H1 is randomly varied within the value range of each hyperparameter to initialize the positions of the other N-1 gold diggers, N is the number of gold digger population, N is N1 or N2, and T is the maximum iteration number.

[0086] 2) Put the corresponding hyperparameters of each gold digger in the multi-task deep learning model in 1) respectively, and train the model using the training set to obtain the multi-task loss function L corresponding to each gold digger, wherein the minimum value is the group optimal fitness, and the current position of the gold digger corresponding to the group optimal fitness is the best gold mine position. The multi-task loss function L of each gold digger is used as its current fitness, and the current fitness of each gold digger is its individual optimal fitness.

[0087] 3) When 0.5 < τ < 1, the migration strategy or the mining strategy is used to update the position, and when the group optimal fitness corresponding to the continuous preset number (such as twice) of iterations is not updated, the migration and mining combined strategy is used to update the position; when τ ≤ 0.5, the mining strategy or the cooperation strategy is used to update the position, and when the group optimal fitness corresponding to the continuous preset number (such as twice) of iterations is not updated, the mining and cooperation combined strategy is used to update the position; τ is an iteration coefficient that gradually decreases with the increase of the number of iterations,

[0088] Put the corresponding hyperparameters of each gold digger to be updated into the corresponding multi-task deep learning model and train it using the training set to obtain the multi-task loss function L corresponding to each gold digger as the current fitness. Compare the current fitness of each gold digger with its individual optimal fitness. If the current fitness of the gold digger is better than its individual optimal fitness, update the individual optimal fitness of the gold digger = the current fitness of the gold digger, and update the position of the gold digger, otherwise do not update. Then the minimum value of the individual optimal fitness of each gold digger is the current group optimal fitness. If the current group optimal fitness is better than the group optimal fitness, update the group optimal fitness = the current group optimal fitness, and the current position of the gold digger corresponding to the current group optimal fitness is the best gold mine position, otherwise do not update.

[0089] 4) Determine whether the group optimal fitness reaches the preset threshold L2 or the number of iterations reaches the maximum iteration number T. If yes, the best gold mine position is used as the optimal hyperparameter, and the optimal model constructed by the optimal hyperparameter is used as the target multi-task deep learning model, L2 < L1, otherwise go to 3) for the next iteration.

[0090] The present scheme does not initialize the gold digging group in the upper and lower limit range as randomly as the prior art, but first trains the multi-task deep learning model, uses the model hyperparameters after preliminary training as the position of one gold digger in the gold digging group, and then randomly mutates the model hyperparameters after preliminary training to obtain the positions of the remaining N-1 gold diggers in the gold digging group. This way of initializing the gold digging group is more targeted and can find the optimal hyperparameter more quickly.

[0091] The scheme does not use the migration strategy for position updating when the random value r is 0 < r < 1 / 3, the mining strategy for position updating when the random value r is 1 / 3 ≤ r < 2 / 3, and the cooperation strategy for position updating when the random value r is 2 / 3 ≤ r < 1, unlike the existing gold panning algorithm. Instead, the scheme sets an iteration coefficient τ related to the number of iterations. In the early iteration stage, the migration strategy or the mining strategy is used for position updating, and when the group optimal fitness of the continuous set number of iterations is not updated, the migration-mining combined strategy is used for position updating to prevent the improved gold panning algorithm from falling into local optimum. In the later iteration stage, the mining strategy or the cooperation strategy is used for position updating, and when the group optimal fitness of the continuous set number of iterations is not updated, the mining-cooperation combined strategy is used for position updating to prevent the improved gold panning algorithm from falling into local optimum.

[0092] After the system receives the prediction instruction, the acquisition module 1, the preprocessing module 2 and the prediction module 6 are called in turn; the acquisition module 1 is used to acquire the inspection and test data, image data, gene detection data and clinical data of the to-be-predicted person; the preprocessing module 2 is used to preprocess the data to obtain multi-modal features, including test features, image features, gene features and clinical features; the prediction module 6 is used to input the multi-modal features into the target multi-task deep learning model, perform feature extraction through a plurality of convolution layers, fuse the multi-modal features through a fusion layer to generate a unified state vector, perform shared learning on the state vector through a shared parameter layer, perform a plurality of venous and arterial blood vessel disease risk predictions through a task-specific layer, and output the risk level of at least one venous blood vessel disease and the risk level of at least one arterial blood vessel disease of the to-be-predicted person.

[0093] In the scheme, for each target disease, a plurality of different and explicit risk levels (such as low risk, medium risk, high risk, or specific probability values) can be output.

[0094] Although the specific embodiments of the present application are described above, those skilled in the art should understand that these are only illustrative, and the protection scope of the present application is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present application, and these changes and modifications all fall within the protection scope of the present application.

Claims

1. A blood vessel risk assessment model system based on a multi-modal fusion strategy and deep learning, characterized in that, The system comprises: an acquisition module, configured to acquire examination and test data, image data, gene detection data and clinical data of a plurality of historical patients, and perform various blood vessel disease condition labeling, wherein the examination and test data comprises numerical, categorical and / or time-series data, the clinical data comprises text, numerical and / or categorical data, and the gene detection data comprises SNPs data and / or gene expression profile data; a preprocessing module, configured to perform standardization on the numerical data, one-hot encoding or entity embedding on the categorical data, time series embedding on the time-series data to convert the time-series data into a fixed-length vector, image segmentation on the image data to segment out a region of interest and extract image features therefrom, or use a deep learning-based feature extractor to learn deep image features from the image data, use word embedding and NLP technology to extract features from the text data to obtain text features, perform dimension reduction on the SNPs data to obtain gene features, and perform normalization and feature selection on the gene expression profile data to obtain gene features; a feature sample construction module, configured to construct a training set and a verification set by using the preprocessed features of the historical patients in multiple modalities; a model construction module, configured to construct a multi-task deep learning model, comprising an input layer, a multi-layer convolutional layer with a residual structure, a fusion layer for feature weighted fusion based on a multi-head attention mechanism, a shared parameter layer based on an attention mechanism, and a task-specific layer for venous and arterial blood vessel diseases, define a multi-task loss function, initialize model hyperparameters and set the value range of each hyperparameter; a multi-task learning training module, configured to train and verify the model by using the training set and the verification set, optimize the model by using an optimization algorithm, adjust the hyperparameters to the optimal based on the multi-task loss function, and obtain a target multi-task deep learning model; the optimization algorithm: adaptive weights are added to the migration strategy, mining strategy and collaboration strategy in the gold panning algorithm, a migration-mining combination strategy combining the migration strategy and the mining strategy is created, a mining-collaboration combination strategy combining the mining strategy and the collaboration strategy is created, and an improved gold panning algorithm is constructed; the multi-task learning training module is configured to input the initialized model hyperparameters, perform preliminary training and verification on the multi-task deep learning model by using the training set and the verification set, obtain a preliminary multi-task loss function L=the weighted sum of each single-task loss function, wherein the weights are adjusted according to the task importance or the model convergence speed, analyze the multi-task loss function Loss and a preset threshold L1, set the number of gold panners as N1 when Loss≤L1, and set the number of gold panners as N2 when Loss>L1, N1<N2, initialize each gold panner in the improved gold panning algorithm by using the numerical value of the model hyperparameters and the numerical value of the randomly varied hyperparameters, continuously optimize the hyperparameters by using the improved gold panning algorithm to obtain optimal hyperparameters, and use the optimal model constructed by the optimal hyperparameters as the target multi-task deep learning model.

2. The blood vessel risk assessment model system based on multi-modal fusion strategy and deep learning according to claim 1, characterized in that, The system further comprises a prediction module, and the system sequentially calls the acquisition module, the preprocessing module and the prediction module after receiving a prediction instruction. The acquisition module is configured to acquire examination and test data, image data, gene test data, and clinical data of a to-be-predicted person; The preprocessing module is configured to preprocess the data to obtain multi-modal features, including test features, image features, gene features, and clinical features; The prediction module is configured to input the multi-modal features into a target multi-task deep learning model, perform feature extraction through a plurality of convolutional layers, perform fusion on the multi-modal features through a fusion layer to generate a unified state vector, perform shared learning on the state vector through a shared parameter layer, perform a plurality of arterial and venous vascular disease risk predictions through a task-specific layer, and output a risk level of at least one venous vascular disease and a risk level of at least one arterial vascular disease of the to-be-predicted person.

3. The blood vessel risk assessment model system based on multi-modal fusion strategy and deep learning according to claim 1, characterized in that, The multi-task learning training module is configured to initialize each gold digger in the improved gold digger algorithm using the numerical value of the model hyperparameter at this time and the hyperparameter numerical value of the random variation thereof, continuously optimize and train the hyperparameter using the improved gold digger algorithm to obtain an optimal hyperparameter, and use the optimal model constructed by the optimal hyperparameter as the target multi-task deep learning model: 1) The numerical value of the model hyperparameter after the preliminary training at this time is H1, each gold digger in the gold digger group represents a group of hyperparameters, H1 is used to initialize the position of a certain gold digger, and the random variation of H1 in the value range of each hyperparameter is used to initialize the positions of the other N-1 gold diggers, N is the number of groups, N is N1 or N2, and T is the maximum number of iterations; 2) The hyperparameters corresponding to each gold digger in 1) are substituted into the multi-task deep learning model respectively, and the training set is used for model training respectively, to obtain the multi-task loss function L corresponding to each gold digger, wherein the minimum value is used as the best fitness of the group, and the current position of the gold digger corresponding to the best fitness of the group is used as the best gold mine position; 3) When 0.5 < x < 1, update the position using the migration strategy or the exploitation strategy, < 1, update the position using the migration strategy or the exploitation strategy, ≤ 0.5, update the position using the exploitation strategy or the cooperation strategy, is an iteration coefficient that gradually decreases with the increase of the iteration number; The hyperparameters corresponding to the to-be-updated positions of each gold digger are substituted into the corresponding multi-task deep learning model respectively, and the training set is used for training respectively, to obtain the multi-task loss function L corresponding to each gold digger as the current fitness. The current fitness of each gold digger is compared with the individual best fitness thereof. When the current fitness of the gold digger is better than the individual best fitness thereof, the individual best fitness of the gold digger is updated to be equal to the current fitness of the gold digger, and the position of the gold digger is updated. Otherwise, no update is performed. The minimum value in the individual best fitness of each gold digger is used as the current group best fitness. When the current group best fitness is better than the group best fitness, the group best fitness is updated to be equal to the current group best fitness, and the current position of the gold digger corresponding to the current group best fitness is used as the best gold mine position. Otherwise, no update is performed. 4) It is judged whether the group best fitness reaches a preset threshold L2 or the number of iterations reaches the maximum number of iterations T. If yes, the best gold mine position is used as the optimal hyperparameter, and the optimal model constructed by the optimal hyperparameter is used as the target multi-task deep learning model. L2 < L1. Otherwise, the next iteration is performed in 3).

4. The blood vessel risk assessment model system based on multi-modal fusion strategy and deep learning according to claim 3, wherein, In 3), when 0.5 < x < 1, When 0 < x < 1, the position is updated by using the migration strategy or the exploitation strategy, and when the optimal fitness of the corresponding population is not updated for a preset number of continuous iterations, the position is updated by using the migration-exploitation combination strategy. When When ≤0.5, the position is updated by using the exploitation strategy or the cooperation strategy, and when the corresponding group optimal fitness is not updated for a preset number of continuous iterations, the position is updated by using the exploitation and cooperation combined strategy.

5. The blood vessel risk assessment model system based on multi-modal fusion strategy and deep learning according to claim 3, wherein, Iteration coefficient .

6. The multi-modal fusion strategy and deep learning based blood vessel risk assessment model system of claim 1, wherein, The multi-layer convolutional layer added with the residual structure comprises: a first convolutional layer composed of one convolutional layer and one pooling layer, a second convolutional layer composed of two convolutional layers, a third convolutional layer composed of two convolutional layers, a fourth convolutional layer composed of two convolutional layers, and a fully connected layer; The two convolutional layers in the second convolutional layer form a first residual structure through a jump connection line, the two convolutional layers in the third convolutional layer form a second residual structure through a jump connection line, and the two convolutional layers in the fourth convolutional layer form a third residual structure through a jump connection line; The number of convolution kernels of the two convolutional layers in each layer of the second convolutional layer, the third convolutional layer and the fourth convolutional layer is the same, the number of convolution kernels of the first convolutional layer, the second convolutional layer, the third convolutional layer and the fourth convolutional layer decreases in turn, the number of convolution kernels of the first convolutional layer and the second convolutional layer is in a multiple relationship, the number of convolution kernels of the second convolutional layer and the third convolutional layer is in a multiple relationship, and the number of convolution kernels of the third convolutional layer and the fourth convolutional layer is in a multiple relationship.

7. The multi-modal fusion strategy and deep learning based vascular risk assessment model system of claim 1, wherein, The fusion layer for feature weighting fusion based on the multi-head attention mechanism comprises a multi-head attention layer and a fully connected layer.

8. The multi-modal fusion strategy and deep learning based blood vessel risk assessment model system of claim 1, wherein, The image data comprises CT data, MRI data and / or ultrasound data; The clinical data comprises patient basic information, lifestyle factors, comorbidities, medication history and preliminary diagnosis information.

9. The multi-modal fusion strategy and deep learning based blood vessel risk assessment model system of claim 1, wherein, A pre-trained CNN or 3D CNN is used as a feature extractor; Singular value decomposition (SVD) is used to reduce the dimensionality of the SNPs data, or a learnable embedding layer is used to map the SNPs data to a low-dimensional space.

Images