Deep learning method and device for joint optimization based on blood vessel center line guidance

Abstract

The invention provides a deep learning method and device for task joint optimization based on blood vessel center line guidance, and solves the problem that the specific recognition of blood vessel morphology and the blood flow function cannot be cooperatively promoted. The method comprises the following steps: in the process of global image feature extraction by an encoder, performing data enhancement on a local image with morphological specificity according to the guidance of a blood vessel center line, extracting local image features, and forming fused image features by fusing the local image features and global image features; and a feature bidirectional correction process is formed between decoders of different decoding branches. The use of vascular centerline guidance provides gated feature optimization and computational efficiency adjustments to the coding process. And forming a control mechanism for adjusting the local feature extraction efficiency by using the specific form data guided by the blood vessel center line. The scarcity problem of hemodynamic labeling is relieved, and the generalization ability of the model is improved. And cooperative improvement of anatomical reconstruction precision and hemodynamic prediction is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning in medical imaging, and particularly relates to a deep learning method and device for task joint optimization guided by a vascular centerline. Background Art

[0002] To accurately evaluate the myocardial hemodynamic abnormalities caused by coronary artery stenosis, invasive methods are usually adopted to obtain gold standard hemodynamic function parameters such as fractional flow reserve (FFR) and instantaneous wave-free ratio (iFR). The angiographic images using CTA (Computed Tomography Angiography) technology can be used to construct a vascular geometric model in a non-invasive manner, but there are defects in analyzing the degree of functional ischemia only from the anatomical perspective of vascular characteristics to form a quantification of hemodynamic function indicators. Through computational fluid dynamics techniques such as CT-FFR (Computed Tomography-Fractional Flow Reserve) and QFR (Quantitative Flow Ratio), on the basis of the vascular geometric model, by setting physiological boundary conditions and state parameters, the blood flow state of the real blood vessel can be simulated to quantify the characteristic parameters of the blood flow function at the target position. Existing deep learning techniques can form a segmentation detection model for vascular morphology and a functional prediction model for hemodynamics, and quickly make a targeted assessment of the vascular morphology or blood flow function of the target blood vessel in combination with the patient's specific baseline data.

[0003] In deep learning techniques, the segmentation task for vascular morphology and the regression task for blood flow function usually both include an encoder structure for feature extraction and a decoder structure for restoring the spatial dimension or forming a regression trend. Task training mostly adopts a separate learning paradigm, that is, for a single task requirement, a segmentation model or a prediction model for vascular morphology segmentation is constructed through independent training data, objective functions, feature extraction modules, segmentation detection modules or regression prediction modules.

[0004] From the prior knowledge of fluid mechanics, it can be known that there is a relatively complex and reliable association between blood flow function changes and vascular morphology changes. On the one hand, the positions where the vascular morphology geometric parameters mutate often cause changes in the blood flow function parameters. On the other hand, the drastic changes in the blood flow function parameters can be used to infer the changes in the vascular morphology geometric parameters. In the existing technology, a cascade mode is usually adopted to connect the detection model and the regression model in series step by step, and the vascular morphology and blood flow function are processed separately and evaluated step by step.

[0005] It can be seen that the mode of treating the vascular anatomical segmentation detection and the blood flow function prediction as independent tasks will lead to the following key defects:

[0006] First, the pathological correlation between vascular anatomical morphology and blood flow function parameters has not been effectively modeled. While weakening the model correlation, the lack of details in training data amplifies the cascading errors between models, easily leading to inconsistent diagnostic conclusions. For example, although the anatomical stenosis is significant, the hemodynamic parameters do not reach the ischemic threshold.

[0007] Second, during the training process, the labeled data for each task has not been collaboratively optimized. The scarcity of hemodynamic labels will limit the generalization ability of the model.

[0008] Third, during the training process, the specific anatomical morphology of blood vessels has not been focused on, which may affect the model efficiency and accuracy.

[0009] Although there are also joint training modes for multi-tasks, more attention has been paid to the detection of functional abnormalities and the estimation of physiological conditions, resulting in the neglect of the physical consistency between anatomical reconstruction and functional prediction. Summary of the Invention

[0010] In view of the above problems, the embodiments of the present invention provide a deep learning method and device for task joint optimization guided by the vascular centerline, which solve the technical problem that in the existing abnormal analysis model of myocardial hemodynamics, the specific recognition of vascular morphology and the physical consistency of using blood flow function cannot be synergistically promoted to improve the overall analysis accuracy.

[0011] The deep learning method for task joint optimization guided by the vascular centerline according to the embodiments of the present invention includes:

[0012] During the process of the encoder extracting global image features, local image features are extracted after data augmentation of local images with morphological specificity guided by the vascular centerline, and fused image features are formed by fusing local image features and global image features;

[0013] A feature two-way correction process is formed between the decoders of different decoding branches.

[0014] In one embodiment of the present invention, the forming of the feature two-way correction process between the decoders of different decoding branches includes:

[0015] According to the task type, the decoder configures the decoding of the fused image features to form at least one decoding branch for vascular morphology tasks and at least one decoding branch for blood flow function tasks;

[0016] In a pair of decoding branches of different task types, a feature two-way correction process is formed between the decoders of the two task types to correct the output features of one decoder with the output features of the other decoder.

[0017] In one embodiment of the present invention, the forming of the fused image features includes:

[0018] Perform global sampling on the input image containing blood vessels under the guidance of the blood vessel centerline to form a global ROI;

[0019] Extract global image features of the global ROI through an encoder;

[0020] Determine position attention information in the global ROI according to blood vessel specificity judgment rules, and perform local sampling in the global ROI according to the position attention information to form a local ROI dataset;

[0021] Upsample and perform data augmentation on the local ROI, then extract local image features using an additional encoder, and perform position encoding of the local image features using the blood vessel centerline;

[0022] Use the attention mechanism to fuse the local image features and the global image features to form fused image features.

[0023] In one embodiment of the present invention, the blood vessel specificity judgment rules include the judgment of blood vessel stenosis rate, blood vessel bifurcation, and blood vessel curvature.

[0024] In one embodiment of the present invention, the step of determining position attention information in the global ROI according to blood vessel specificity judgment rules and performing local sampling in the global ROI according to the position attention information to form a local ROI dataset is replaced by:

[0025] Adjust the local ROI in the global ROI for local sampling to form a local ROI dataset, and determine the position attention information of the local ROI including the specific morphology of the blood vessel according to the blood vessel specificity judgment rules

[0026] In one embodiment of the present invention, the upsampling is set according to the resolution level of the fusion of local image features and global image features.

[0027] In one embodiment of the present invention, the additional encoder extracts local features in a cascaded 3D convolution manner and fuses them with the position encoding of the blood vessel centerline points to obtain local image features.

[0028] In one embodiment of the present invention, the size of the local ROI is dynamically adjusted according to the blood vessel specificity judgment rules.

[0029] In one embodiment of the present invention, the blood vessel centerline contains stenosis information and is obtained by one of the following methods:

[0030] When the centerline is known, use cross-sectional 2D-Unet segmentation to obtain the cross-sectional diameter and calculate the stenosis degree along the centerline. When the centerline is unknown, use lightweight 3D-Unet to segment the whole image and extract the centerline coordinates, and calculate the blood vessel stenosis on the centerline.

[0031] In one embodiment of the present invention, the step of configuring a decoder according to the task type to decode the fused image features to form at least one decoding branch for vascular morphology tasks and at least one decoding branch for blood flow functional tasks includes:

[0032] Set a segmentation decoding branch reflecting vascular morphology tasks, and set a segmentation decoder corresponding to the same-level encoder in the segmentation decoding branch; the vascular morphology tasks include at least one of contour, inner diameter, inner wall mask, or topological shape;

[0033] Set a prediction decoding branch reflecting blood flow functional tasks, and set a prediction decoder corresponding to the same-level encoder in the prediction decoding branch; the blood flow functional tasks include at least one of blood pressure parameters, blood flow parameters, and CTFFR.

[0034] In one embodiment of the present invention, the input features of the feature bidirectional correction process include the feature output of the same-level decoder as the feature to be corrected, and also include the feature output of the same-level decoder as the control feature, the feature output of the previous-level decoder, or the downsampled feature output of the task final output result, and the interaction judgment is formed by using the control feature to control the feature bidirectional correction process.

[0035] In one embodiment of the present invention, the feature bidirectional correction process includes:

[0036] Establish a forward correction process for correcting functional features according to morphological features between two decoders corresponding to the task type of the decoding branch;

[0037] Meanwhile, establish a reverse correction process for correcting morphological features according to functional features between the two decoders.

[0038] In one embodiment of the present invention, the forward correction process includes:

[0039] Generate spatial attention weights according to the segmentation confidence map of morphological features, dynamically modulate the functional feature map, and obtain the corrected functional features.

[0040] In one embodiment of the present invention, the reverse correction process includes:

[0041] Correct the morphological features according to the change gradient direction of the functional features to obtain the corrected morphological features.

[0042] In one embodiment of the present invention, the formation of the feature bidirectional correction process includes:

[0043] Establish a forward correction process for correcting functional features according to morphological features and the distance heat map of the vascular centerline between two decoders corresponding to the task type of the decoding branch;

[0044] Meanwhile, a reverse correction process for correcting morphological features according to functional features and the blood flow direction guided by the vascular centerline is established between the two decoders.

[0045] In one embodiment of the present invention, the two-way correction process of forming features includes:

[0046] Set a measurement threshold for the average confidence of morphological features in the gating judgment process, and determine whether to activate the two-way feature correction process according to the measurement result.

[0047] In one embodiment of the present invention, the setting of the measurement threshold for the average confidence of morphological features in the gating judgment process and determining whether to activate the two-way feature correction process according to the measurement result includes:

[0048] Obtain the morphological features to be corrected, the functional features to be corrected, the control segmentation features and the control functional features, and form a confidence map and an average confidence according to the control functional features;

[0049] When the average confidence is greater than the measurement threshold, directly output the morphological features to be corrected and the functional features to be corrected from the segmentation decoder and the prediction decoder respectively;

[0050] When the average confidence is less than the measurement threshold, activate the two-way feature correction process.

[0051] In one embodiment of the present invention, the two-way correction process of forming features includes:

[0052] Output a confidence map (ConfMap) for determining channels and a heat map of the distance from the vascular centerline through 3x3x3 convolution of the control functional feature (F_control) to form functional correction data;

[0053] Output the functional gradient (FFRGrad) of the same channel through 3x3x3 convolution of the control segmentation feature (S_control) and superimpose the blood flow direction guided by the vascular centerline to form morphological correction data;

[0054] Correct the morphological features to be corrected (S_feat) according to the morphological correction data to form the corrected morphological features (S_feat_cor);

[0055] Correct the functional features to be corrected (F_feat) according to the functional correction data to form the corrected functional features (F_feat_cor).

[0056] In one embodiment of the present invention, it further includes:

[0057] The vascular morphology parameters and blood flow function parameters obtained from the decoding branches of two task types are used as independent variables of physical constraint conditions. According to the residuals between the predicted values and the true values of the dependent variables of the physical constraint conditions, a physical constraint loss function is determined as a component of the total loss function. The decoder and the feature bidirectional correction process are optimized through the gradient backpropagation of the total loss function.

[0058] In an embodiment of the present invention, the determining the physical constraint loss function as a component of the total loss function and optimizing the decoder and the feature bidirectional correction process through the gradient backpropagation of the total loss function includes:

[0059] Determining a segmentation loss function according to the residuals between the prediction result and the ground truth of the segmentation task;

[0060] Determining a prediction loss function according to the residuals between the prediction result and the ground truth of the prediction task;

[0061] Selecting a segmentation task and a CTFFR prediction task, obtaining the vessel radius parameter from the result of the segmentation task and the pressure difference parameter from the result of the CTFFR prediction task. The vessel radius parameter and the pressure difference parameter are used to obtain the predicted blood flow value through the Poiseuille flow equation, and the vessel radius parameter and the pressure difference parameter are used to form the true value of the vessel blood flow through the fluid continuity equation. A physical constraint loss function is determined according to the residuals between the predicted blood flow value and the true value of the vessel blood flow;

[0062] Forming a total loss according to the segmentation loss, the prediction loss, and the physical constraint loss, and optimizing the network parameters of the decoder and the feature bidirectional correction process through the gradient backpropagation of the total loss function.

[0063] The deep learning device for joint optimization based on vessel centerline guidance according to an embodiment of the present invention includes:

[0064] An encoder, which is used to perform data enhancement on local images with morphological specificity according to the vessel centerline guidance during the process of global image feature extraction by the encoder, and then extract local image features, and form fused image features by fusing the local image features and the global image features;

[0065] A decoding correction module, which is used to form a feature bidirectional correction process among the decoders of different decoding branches.

[0066] In an embodiment of the present invention, the decoding correction module includes:

[0067] A decoder, which is used to configure the decoder according to the task type to decode the fused image features, and form a decoding branch for at least one vascular morphology task and a decoding branch for at least one blood flow function task;

[0068] A bidirectional correction decoder is used to form a feature bidirectional correction process between two decoders in a pair of decoding branches of different task types, where the output features of one decoder are used to correct the output features of the other decoder.

[0069] In an embodiment of the present invention, it further includes:

[0070] A physical constraint formation module is used to use the vascular morphology parameters and blood flow function parameters obtained from the decoding branches of two task types as independent variables of physical constraint conditions, determine the physical constraint loss function as a component of the total loss function according to the residual between the predicted value and the true value of the dependent variable of the physical constraint conditions, and optimize the decoder and the feature bidirectional correction process through the gradient backpropagation of the total loss function.

[0071] The electronic device according to an embodiment of the present invention includes:

[0072] A processor, a memory, and an interface for communicating with a gateway;

[0073] The memory is used to store programs and data, and the processor calls the programs stored in the memory to execute the above-mentioned method.

[0074] The computer-readable storage medium according to an embodiment of the present invention includes a program, and the program is used to execute the above-mentioned method when executed by a processor.

[0075] The deep learning method and device for task joint optimization based on vascular centerline guidance according to an embodiment of the present invention achieve a synergistic improvement in anatomical reconstruction accuracy and hemodynamic prediction through end-to-end joint optimization. Using vascular centerline guidance provides gated feature optimization and computational efficiency adjustment for the encoding process. Using the specific morphological data of the vascular centerline guidance forms a control mechanism for adjusting the local feature extraction efficiency. The model uses two types of labels of hemodynamic function and vascular morphology, alleviates the scarcity problem of hemodynamic annotation, and improves the generalization ability of the model. At the same time, using prior knowledge to form physical constraints on the output results ensures the rationality of the blood flow parameter prediction results and improves the segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 The flowchart of the deep learning method for task joint optimization based on vascular centerline guidance according to an embodiment of the present invention is shown.

[0077] Figure 1-1 The flowchart of forming the encoding-decoding process in the deep learning method for task joint optimization based on vascular centerline guidance according to an embodiment of the present invention is shown.

[0078] Figure 2The figure shows a schematic flow chart of multi-task optimization using a U-Net backbone network in the deep learning method for task joint optimization based on blood vessel centerline guidance according to an embodiment of the present invention.

[0079] Figure 2-1 The figure shows a schematic flow chart of forming local image features in the deep learning method for task joint optimization based on blood vessel centerline guidance according to an embodiment of the present invention.

[0080] Figure 3 The figure shows a schematic structural diagram of an optimization model formed by using a U-Net backbone network in the deep learning method for task joint optimization based on blood vessel centerline guidance according to an embodiment of the present invention.

[0081] Figure 4 The figure shows a schematic structural diagram of a gated bidirectional correction module in the optimization model formed by the deep learning method for task joint optimization based on blood vessel centerline guidance according to an embodiment of the present invention.

[0082] Figure 5 The figure shows a schematic diagram of the feature acquisition path of the feature bidirectional correction process in the optimization model formed by the deep learning method for task joint optimization based on blood vessel centerline guidance according to an embodiment of the present invention.

[0083] Figure 6 The figure shows a schematic structural diagram of the loss function in the optimization model formed by the deep learning method for task joint optimization based on blood vessel centerline guidance according to an embodiment of the present invention.

[0084] Figure 7 The figure shows a schematic structural diagram of the deep learning device for task joint optimization based on blood vessel centerline guidance according to an embodiment of the present invention.

[0085] Figure 8 The figure is a schematic network structure diagram of the deep learning device for task joint optimization based on blood vessel centerline guidance according to an embodiment of the present invention.

[0086] Figure 9 The figure shows a schematic structural diagram of an electronic device according to an embodiment of the present invention. Detailed implementation manners

[0087] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described below in conjunction with the accompanying drawings and specific implementation manners. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0088] A deep learning method for task joint optimization based on blood vessel centerline guidance in an embodiment of the present invention is as follows Figure 1 as shown. In Figure 1 this embodiment, it includes:

[0089] Step 100: During the process of global image feature extraction by the encoder, local image features are extracted after data augmentation of local images with morphological specificity guided by the blood vessel centerline, and fused image features are formed by fusing local image features and global image features.

[0090] Those skilled in the art can understand that in neural network architectures such as CNN and Transformers, processing solutions using an encoder-decoder structure are included for task types such as classification segmentation, object detection, prediction regression, and semantic segmentation. For example, in the U-Net image segmentation model under the CNN architecture, one of the U-Net type networks such as Classic U-Net, Swin-UNETR, Attention U-Net, ResU-Net, and UNet++ is used as the backbone network. The encoder downsamples to extract image features and expand the receptive field, and the decoder upsamples to restore the spatial resolution and perform feature fusion, thereby performing high-precision pixel-level image segmentation. In practical applications, the image segmentation model with the same task objective can be replaced with, for example, the combination of the encoder and decoder under the Transformers architecture.

[0091] Those skilled in the art can understand that through prior knowledge, it can be verified that there are mutual associations and influences between tasks related to blood vessel morphology and blood flow function. For different task decoding requirements, a targeted encoder is needed to extract graphic features. During the process of extracting image features, the encoder can be constructed according to task requirements, and can be uniformly constructed for rich feature types, or can be separately constructed and mixedly utilized according to different feature types. Using a hybrid encoder can establish rich types of annotation labels for image features at the same coding level, and can alleviate the scarcity of feature annotation caused by the limitations of a single encoder.

[0092] Those skilled in the art can understand that the blood vessel centerline, as a form of organization of rich blood vessel-related information, forms a position encoding by combining the spatial orientation path positioning with blood vessel information through the point coordinates of the blood vessel centerline. The information types carried by the blood vessel centerline include, but are not limited to, geometric morphological information such as position, orientation, length, and curvature, branch and confluence, topological structure information such as connectivity and hierarchical structure, and hemodynamic information such as stenosis position and stenosis degree.

[0093] The extraction processes of global image features and local image features are for the input image containing blood vessels. Medical images containing blood vessels include, but are not limited to, medical images such as 3D CTA, X-ray angiography, magnetic resonance angiography (MRA - Magnetic Resonance Angiography), and vascular ultrasound imaging.

[0094] The global ROI image or local ROI image is obtained from the image containing blood vessels by setting the ROI area size. The global ROI image directly uses the encoder for global image feature extraction. The local ROI image and the blood vessel information related to the spatial position features of the local ROI image are obtained by using the guiding effect of the blood vessel centerline. The size of the global ROI or local ROI can be dynamically adjusted according to the task characteristics. The dynamic adjustment ensures the dynamic focusing accuracy of the local ROI and can effectively balance the feature signal sampling efficiency. For example, dynamic adjustment is performed according to the blood vessel morphology or according to the specific judgment of the blood vessel morphology to determine the local ROI image of the blood vessel. The rules of specific judgment include, but are not limited to, the parameter thresholds set for morphological and functional parameters such as local stenosis rate, centerline curvature, and blood vessel bifurcation. According to the rules of specific judgment, it is judged whether there is specificity in the point position of the blood vessel centerline, and the specific position of the blood vessel is determined by using the spatial guidance of the blood vessel centerline according to the point position.

[0095] The local ROI image of the blood vessel obtained through the blood vessel centerline information is upsampled to increase the resolution. After increasing the sample diversity and resolution accuracy through data augmentation methods such as geometric transformation, brightness contrast adjustment, or noise addition, while performing local image feature extraction of the local ROI image using an additional encoder for the local micro-morphology of the blood vessel, position encoding is performed according to the corresponding point coordinate information guided by the blood vessel centerline. Furthermore, the attention mechanism is used to perform cross-modal fusion of the local image features and the global image features (at one resolution) output by the encoder to form a fused image feature output for the decoder to perform subsequent processing.

[0096] Step 100a: A feature two-way correction process is formed between the decoders of different decoding branches.

[0097] Decoding branches are formed for tasks oriented to blood vessel morphology and blood vessel blood flow characteristics. The two-way correction of the output features of the two decoders is formed by using the differences between the decoders in the decoding branches and the mutual influence of the tasks. So that the decoders of two different task types can promote each other in the output feature accuracy.

[0098] The deep learning method for task joint optimization based on vascular centerline guidance in the embodiments of the present invention improves the efficiency and accuracy of encoding and decoding by using the information guided by the vascular centerline to construct high-precision annotation data suitable for multiple tasks, forms collaborative optimization for the overall model, alleviates the problem of scarce annotation for the analysis of functional states other than vascular morphology, and ensures the generalization ability of the model. At the same time, the association between vascular morphological reconstruction and other blood flow function predictions is realized, increasing the interpretability and rationality of the prediction results.

[0099] A deep learning method for task joint optimization based on vascular centerline guidance in an embodiment of the present invention is as Figure 1-1 shown. In Figure 1-1 , step 100a of this embodiment may specifically include:

[0100] Step 200: Configure a decoder according to the task type to decode the fused image features, and form a decoding branch for at least one vascular morphology task and a decoding branch for at least one blood flow function task.

[0101] Those skilled in the art can understand that the decoder is configured according to the decoding task requirements, and the encoded graphic features are spliced, connected, and decoded by the decoder to form a task decoding branch. By using different task combinations, the overall expression performance of the model is improved. The task types mainly include two types, namely those for vascular morphology and those for blood flow function. The related tasks of vascular morphology include, but are not limited to, tasks such as the segmentation detection of vascular boundaries, vascular diameters, or intima related to vascular morphology. The related tasks of blood flow function include, but are not limited to, tasks such as the functional prediction of flow rate, blood pressure, or CTFFR related to blood flow function. There are relevant prior knowledge correlations between the two task types. During the decoding process, a decoder corresponding to the relevant tasks of the task type needs to be formed, and the relevant tasks in one task type are formed by using the decoder. Each task type includes at least one relevant task, and one relevant task forms a decoding branch. Specifically, there can be a many-to-many or one-to-many relationship between the number of relevant tasks of the two task types.

[0102] In each decoding branch, a decoder is set according to the task requirements and the number of decoders is determined. Generally, in each decoding branch, the decoder type corresponds to the decoding task requirements, and the number of decoders corresponds to the number of encoders, and they are at the same level. In practical applications, different decoder architectures can be adopted, including but not limited to CNN-based decoders, RNN-based decoders, Transformers-based decoders, etc. For example, for the encoder-decoder backbone network formed by U-Net, the number of decoders in each decoding branch is basically corresponding to the number of encoders in the encoding process. The encoder and decoder features at the same level with different resolutions are spliced and connected through skip connections, and upsampled step by step until the original resolution layer of the input image, and then the task results are output.

[0103] Step 300: In a pair of decoding branches of different task types, a feature two-way correction process is formed between the decoders of the two task types to correct the output features of one decoder by using the output features of the other decoder.

[0104] Those skilled in the art can understand that different decoding branches include decoders corresponding to task types. The decoders participating in the feature two-way correction are at the same level in different decoding branches, forming a pair of decoders that can correct each other's features through the feature two-way correction process. The feature two-way correction process acts bidirectionally on a pair of decoders at the same level, forms a correction to the features received from another decoder according to the features received from one decoder, and changes the feature outputs of the two decoders through feature correction. In the feature two-way correction process, the feature input (related to blood vessel morphology or blood flow function) includes the features to be corrected, for example, it can include the feature outputs of the decoders at the same level (as the features to be corrected). The feature input also includes control features, such as the feature outputs of the decoders at the same level (the features to be corrected as control features) or the feature outputs of the previous-level decoders (as control features), the downsampled feature outputs including the final output results of the tasks (as control features), etc., or a combination of the above several outputs. There is at least one feature two-way correction process between a pair of decoding branches for blood vessel morphology and blood flow function, and there can also be a feature two-way correction process during each decoding at the same level.

[0105] The output of the feature two-way correction process is the corrected features output after the feature correction of the decoders at the same level. The decoders at the same level use the corrected features to replace the original features, and then re-decode after splicing and skip connection to form the output features sent to the next level.

[0106] In the feature two-way correction process, interaction judgments can be formed by using control features to control the correction process. The interaction judgment modes include but are not limited to interaction based on the attention mechanism, interaction based on the gating mechanism, interaction based on skip connection, or interaction of adversarial training, etc. In an embodiment of the present invention, the two-way correction process can constitute an (interactive) decoder structure.

[0107] In an embodiment of the present invention, when the number of decoding branches for each task type is greater than one, at least one feature two-way correction process can be formed between two decoding branches of different task types.

[0108] The deep learning method for task joint optimization guided by the vascular centerline in the embodiments of the present invention uses the blood flow functional state that has a prior correlation with the vascular morphology to establish a joint analysis means for the vascular morphology, forming a joint promotion for the analysis of vascular morphology and function. By using the information guided by the vascular centerline to guide the construction of high-precision annotation data suitable for multiple tasks, the efficiency and accuracy of encoding and decoding are improved, the overall model is synergistically optimized, the problem of scarce annotation for the analysis of functional states other than vascular morphology is alleviated, and the generalization ability of the model is ensured. By constructing a bidirectional correction process between the decoding branches of multiple task types, synergistic optimization between different state analyses is formed, and the accuracy of each state analysis is improved.

[0109] A deep learning method for task joint optimization guided by the vascular centerline in an embodiment of the present invention uses a U-Net backbone network for task joint optimization as Figure 2 shown. The optimized model structure formed by the deep learning method for task joint optimization guided by the vascular centerline in an embodiment of the present invention is as Figure 3 shown. The local image features formed in the deep learning method for task joint optimization guided by the vascular centerline in an embodiment of the present invention are as Figure 2-1 shown. Combining Figure 2 、 2-1 and Figure 3 shown, in this embodiment, on the basis of the optimization method in the above embodiment, the Swin UNetR network is used as the backbone network of the encoding and decoding structure, forming parallel, layer-by-layer upsampling segmentation decoding branches and prediction decoding branches. In step 100, it includes:

[0110] Step 110: Perform global sampling on the input image containing blood vessels under the guidance of the vascular centerline to form a global ROI.

[0111] The point coordinate information of the vascular centerline provides a path positioning for the spatial orientation of the blood vessel. According to the point coordinate information, the basic spatial position of the blood vessel in the image containing the blood vessel can be determined, such as the distribution of the blood vessel in the image coordinate space. The global ROI is formed according to the basic spatial position. The global ROI can be a fixed-size slice image formed according to the image containing the blood vessel, or a spliced image such as a straightened image or a probe image formed according to the vascular centerline. The size of the global ROI can be preset according to the image containing the blood vessel or adjusted according to the content in the global ROI.

[0112] Step 120: Extract global image features from the global ROI through the encoder.

[0113] In an embodiment of the present invention, a swin-transformer encoder is selected as a hybrid encoder, which can simultaneously adopt a hierarchical structure and a window-based self-attention mechanism to obtain associated spatial feature representations and semantic information. The hybrid encoder can capture the features of an image at different scales step by step, gradually converting the input medical image into a low-resolution spatial feature representation while retaining important semantic information.

[0114] In an embodiment of the present invention, the hybrid encoder performs 4-stage downsampling on an image containing blood vessels to obtain four levels of 1 / 2 resolution feature maps, 1 / 4 resolution feature maps, 1 / 8 resolution feature maps, 1 / 16 resolution feature maps, and a 1 / 32 feature map at the bottleneck.

[0115] Step 130: Determine position attention information according to the blood vessel specificity judgment rule in the global ROI, and perform local sampling in the global ROI according to the position attention information to form a local ROI dataset.

[0116] The blood vessel specificity judgment rule is directed at the information representing the specific morphology of the blood vessel carried by the blood vessel center line. The position attention information set of the local specific morphology of the blood vessel relative to the point coordinates on the blood vessel center line can be determined. Local sampling is performed in the global ROI according to the position attention information to form local ROIs of all specific morphology blood vessels. The local ROI is centered on the point coordinates on the blood vessel center line.

[0117] Step 140: Upsample and perform data augmentation on the local ROI, then extract local image features using an additional encoder, and perform position encoding on the local image features using the blood vessel center line.

[0118] In an embodiment of the present invention, upsampling is used to ensure the resolution compatibility of the local ROI with that of a feature map formed by the encoder. Data augmentation is used to increase sample diversity. Position encoding is used to make the feature map of the local ROI obtain spatial position compatibility with the feature map of the global ROI.

[0119] Step 150: Use the attention mechanism to fuse the local image features and the global image features to form fused image features.

[0120] The position and resolution of the feature map of the local ROI are adapted to the fusion requirements. The fused image features formed by feature fusion of self-attention or multi-scale attention according to the position encoding and the determined resolution carry rich features of the specific morphology of the local blood vessel.

[0121] Such as Figure 2-1As shown, in an embodiment of the present invention, the vascular specificity judgment rules include, but are not limited to, judgment rules such as the state of vascular stenosis rate (threshold), the state of vascular bifurcation (switch value), and the state of vascular curvature (threshold). The judgment rules are based on prior laws and professional practices, can serve as a verifiable gold standard in the diagnosis and treatment process, can be gradually updated and optimized, and can form a morphological focus related to coordinates to fully schedule processing efficiency and balance processing computing power. For example, in prior knowledge, the location where the vascular geometric topology undergoes a mutation often causes changes in blood flow parameters: when a blood vessel has a lesion and the lumen of the blood vessel becomes smaller, the true lumen of the blood vessel is smaller than the lumens of the blood vessels upstream and downstream of the blood vessel, and the FFR value drops suddenly along the center line direction of the blood vessel; when a blood vessel bifurcates, even the diameter of a non-lesioned sub-vessel may be slightly smaller than that of the parent vessel. According to the laws of fluid mechanics, the blood flow in the parent vessel is equal to the sum of the blood flows in the sub-vessels. At this time, the pressure value (CTFFR can be understood as a derived value of the pressure value) between the parent vessel and the sub-vessels is continuously distributed without a sudden drop, while the flow value shows a sudden drop; for a position with a large vascular curvature, due to the possible generation of complex blood flow patterns locally, local blood pressure instability and uneven flow velocity may occur. The drastic changes in blood flow parameters can be used to infer the changes in geometric parameters: when the FFR value drops suddenly, it can be determined that there is a mutation in the local vascular topology; when the FFR value does not change or changes insignificantly while there are obvious lesions (such as high calcification, artifacts, etc.) in the image, it can be determined that the lesions are mainly due to errors caused by the imaging method rather than the true morphology of the blood vessel.

[0122] The deep learning method based on vascular centerline guidance for joint optimization in the embodiment of the present invention forms a focusing mechanism for dynamic local ROI guided by the vascular centerline. The specific features of the vascular morphology reflected by the guidance of the vascular centerline are increased while being effectively recognized, making the image feature extraction more accurate and the feature types more abundant. At the same time, the attention mechanism is used in combination with coordinate encoding to form an effective fusion of relevant image features, ensuring the synchronous improvement of the annotation of vascular morphology and blood flow function data in terms of type and data volume, and avoiding the problem of scarce annotation in the deep learning training process.

[0123] In an embodiment of the present invention, the replaced

[0124] Step 130: Adjust the local ROI in the global ROI to perform local sampling to form a local ROI dataset, and determine the location attention information of the local ROI including the specific morphology of the blood vessel according to the vascular specificity judgment rules.

[0125] While performing global sampling to form the global ROI, local sampling can be performed according to a fixed or dynamic sampling window size to form the local ROI, providing a concurrent adjustment mechanism for the acquisition and encoding processes of global image features and local image features, and effectively using computing power resources to control the encoding efficiency.

[0126] As Figure 3 shown, in an embodiment of the present invention, the upsampling and data augmentation in step 140 are set according to the resolution levels of the fusion of local image features and global image features. This enables feature fusion to be achieved at any encoding level according to the requirements of the self-attention mechanism, ensuring the flexibility of adapting the local sampling data volume to the feature fusion level.

[0127] As Figure 2-1 shown, in an embodiment of the present invention, the additional encoder extracts local features by cascading 3D convolutions and fuses them with the position encoding of the vascular centerline points to obtain local image features. When using high-precision local sampling signals to obtain more abstract and high-level image features, the position information of the vascular centerline points is incorporated into the image features, enabling the local image features and global image features to obtain the spatial basis of the attention mechanism.

[0128] As Figure 2-1 shown, in an embodiment of the present invention, interpolation or a super-resolution network such as ESRGAN is used for upsampling to obtain the high resolution of the local image, which is beneficial for extracting the morphological features of small windows at high resolution (such as detailed features like plaques and vessel walls).

[0129] The vascular centerline contains stenosis information. It is usually obtained by one of the following methods:

[0130] When the centerline is known, cross-sectional 2D-Unet segmentation is used to obtain the cross-sectional diameter, and the stenosis degree along the centerline is calculated. When the centerline is unknown, lightweight 3D-Unet is used to segment the whole image and extract the centerline coordinates, and the vascular stenosis on the centerline is calculated. A targeted stenosis information extraction process is formed according to the complexity of directly using the spatial guidance information of the centerline, improving the efficiency and accuracy of vascular recognition.

[0131] The deep learning method for task joint optimization based on vascular centerline guidance in the embodiments of the present invention uses the vascular centerline information guidance to provide gated feature optimization and calculation efficiency adjustment for the encoding process. A control mechanism for adjusting the local feature extraction efficiency is formed using the specific morphological data guided by the vascular centerline. An effective fusion of local super-resolution upsampling features and global image features is formed using the coordinate space guidance of the vascular centerline, improving the accuracy of image features at key positions during the encoding process and ensuring that more accurate detailed features can be obtained during the decoding process. While the vascular centerline guides the local image features to improve the spatial feature accuracy of the vascular morphology, it simultaneously improves the semantic features of the related blood flow function, alleviating the problem of scarce annotation of blood flow states other than the vascular morphology and ensuring the generalization ability of the model.

[0132] Combined with Figure 2 and Figure 3As shown in the figure, in this embodiment, based on the optimization method of the above embodiment, step 200 includes:

[0133] Step 210: Set a segmentation decoding branch reflecting the vascular morphology task, and set a segmentation decoder corresponding to the encoder of the same level in the segmentation decoding branch.

[0134] The morphological parameters corresponding to the vascular morphology task include but are not limited to contour, inner diameter, inner wall mask, or topological shape, etc.

[0135] Each level of the segmentation decoder in the segmentation decoding branch, also known as the decoder unit (S-Decoder-Block), performs a skip connection on the decoded output of the segmentation decoder of the previous level and the encoding result of the same level, then decodes and sends it to the next level, and upsamples step by step until reaching the original resolution layer of the input image.

[0136] Specifically, the segmentation decoding branch performs deconvolution and normalized upsampling through the segmentation decoder, combines the skip connection decoding of the output features of the encoder with the same resolution in the same level, and finally connects a 1x1x1 convolution after the features at the original resolution layer, and uses the sigmoid activation function to generate a segmentation probability map.

[0137] Step 220: Set a prediction decoding branch reflecting the blood flow function task, and set a prediction decoder corresponding to the encoder of the same level in the prediction decoding branch.

[0138] The functional parameters corresponding to the blood flow function task include but are not limited to blood pressure parameters, blood flow parameters, CTFFR, etc.

[0139] Each level of the prediction decoder in the prediction decoding branch, also known as the decoder unit (F-Decoder-Block), splices and performs a skip connection on the decoded output of the previous level and the encoding result of the same level, then decodes and sends it to the next level, and upsamples step by step until reaching the original resolution layer of the input image.

[0140] Specifically, the prediction upsampling branch performs deconvolution and normalized upsampling through the prediction decoder, combines the skip connection decoding of the output features of the encoder with the same resolution in the same level, and finally connects a 1x1x1 convolution or uses a fully connected layer after the features at the original resolution layer to achieve parameter prediction.

[0141] In an embodiment of the present invention, there are at least two types of decoding branches for tasks, one corresponding to the segmentation task of vascular morphology and the other corresponding to the prediction task of blood flow function. The two tasks can include one segmentation decoding branch and several prediction decoding branches, include several segmentation decoding branches and one prediction decoding branch, or include several segmentation decoding branches and several prediction decoding branches. According to the task requirements, a multi-task joint optimization training mode of vascular morphology and blood flow function can be formed.

[0142] The deep learning method for task joint optimization guided by the vascular centerline in the embodiments of the present invention realizes the association between vascular morphological reconstruction and the prediction of other functional state parameters through multiple decoding branches, forms the information complementarity between vascular morphology and blood flow function, and increases the interpretability and rationality of the prediction results. An association task pair can be established between any vascular morphology task and any blood flow function task according to the association established based on prior knowledge. For example, the cooperation scenario between the coronary artery vascular morphology task and the CTFFR function task, etc.

[0143] Combined with Figure 2 and Figure 3 As shown, in an embodiment of the present invention, based on the optimization method in the above embodiment, step 300 includes:

[0144] Step 310: Establish a forward correction process for correcting the functional features according to the morphological features between two decoders of the task type corresponding to the decoding branch.

[0145] One of the two decoding branches is a prediction decoding branch, and the other is a segmentation decoding branch. In the forward correction process, receive the morphological features (i.e., to be corrected) from the segmentation decoder and the functional features (i.e., to be corrected) from the prediction decoder, generate spatial attention weights according to the segmentation confidence map of the morphological features, dynamically modulate the functional feature map, and obtain the corrected functional features. Improve the prediction accuracy of the functional features. Output the corrected functional features and feedback them back to the prediction decoder to replace the original functional features, and then perform the subsequent upsampling and hierarchical decoding process after jumping connection with the output of the same-level encoder and decoding in the prediction decoder.

[0146] Step 320: At the same time, establish a reverse correction process for correcting the morphological features according to the functional features between the two decoders.

[0147] In the reverse correction process, for the received morphological features (i.e., to be corrected) and functional features (i.e., to be corrected), correct the morphological features according to the change gradient direction of the functional features to obtain the corrected morphological features. Improve the segmentation boundary accuracy of the morphological features. Output the corrected morphological features and feedback them back to the segmentation decoder to replace the original morphological features, and then perform the subsequent upsampling and hierarchical decoding process after jumping connection with the output of the same-level encoder and decoding in the segmentation decoder.

[0148] The deep learning method for task joint optimization guided by the vascular centerline in the embodiments of the present invention utilizes the physical function state that has a prior correlation with the vascular morphology, and trains and forms the mutual promotion and collaborative optimization of vascular morphology analysis and blood flow function state analysis by using the fusion image features containing local high-precision vascular-specific morphological features. It overcomes the technical problems of low model generalization ability caused by the lack of labeled type data and the lack of joint optimization of tasks, which cannot accurately express the consistency of physical features.

[0149] Combined with Figure 2 and Figure 3 As shown, in one embodiment of the present invention, based on the optimization method in the above embodiment, step 300 includes:

[0150] Step 315: Establish a forward correction process for correcting the functional features according to the morphological features and the vascular centerline distance heat map between two decoders corresponding to the task types of the decoding branches.

[0151] Generate spatial attention weights according to the segmentation confidence map of the morphological features, dynamically modulate the functional feature map, and obtain the corrected functional features. When calculating the vascular segmentation confidence in the morphological features, combining the vascular centerline distance heat map can reduce the calculation amount and improve the segmentation focusing accuracy around the blood vessels.

[0152] Step 325: At the same time, establish a reverse correction process for correcting the morphological features according to the functional features and the vascular centerline guiding the blood flow direction between the two decoders.

[0153] Correct the morphological features according to the change gradient direction of the functional features to obtain the corrected morphological features. In the physical real situation, the guiding information of the vascular centerline for the blood flow direction combined with the decrease of the FFR value along the blood flow direction can reflect the presence of lesions. Therefore, using the centerline guiding direction information and calculating the functional gradient along the blood flow direction can capture the real vascular morphological changes.

[0154] The deep learning method for task joint optimization guided by the vascular centerline in the embodiments of the present invention utilizes the distance guidance and flow direction guidance of the vascular centerline to enhance the mutual promotion and collaborative optimization of vascular morphology analysis and blood flow function state analysis.

[0155] In the deep learning method for task joint optimization guided by the vascular centerline in one embodiment of the present invention, the gated feature bidirectional correction module in the optimization model formed by using a U-Net backbone network is as Figure 4 shown. Combined with Figure 2 and Figure 4 shown, in one embodiment of the present invention, step 300 includes:

[0156] Step 330: Set the measurement threshold for the average confidence of morphological features in the Gate Decision, and determine whether to activate the Correction Layer according to the measurement result.

[0157] The Gate Decision is used to determine the transfer direction of the morphological features to be corrected (S_feat) and the functional features to be corrected (F_feat) (usually at the previous level) according to the confidence of the morphological features output by the segmentation decoder (usually at the previous level). Use the measurement threshold for the average confidence in the Gate Decision as the activation signal for switching the Correction Layer.

[0158] Combined with Figure 2 and Figure 4 As shown, in an embodiment of the present invention, based on the above embodiment, determining to activate the Correction Layer includes:

[0159] Step 340: Obtain the morphological features to be corrected (S_feat), the functional features to be corrected (F_feat), the control segmentation features (S_control) and the control functional features (F_control), and form a confidence map and an average confidence according to the control functional features.

[0160] As Figure 4 shown, in an embodiment of the present invention, the control functional features (F_control) are subjected to 1x1x1 convolution and combined with the Sigmoid activation function operation through the Conf Head to obtain the confidence map and the average confidence (conf) of the morphological features at the previous level.

[0161] Step 350: When the average confidence is greater than the measurement threshold, directly output the morphological features to be corrected (S_feat) and the functional features to be corrected (F_feat) from the segmentation decoder and the prediction decoder respectively.

[0162] When the average confidence is greater than the measurement threshold, the Gate Decision does not activate the Correction Layer, and directly outputs the morphological features and the functional features from the morphological decoder and the functional decoder.

[0163] Step 360: When the average confidence is less than the measurement threshold, activate the Correction Layer.

[0164] As Figure 4 shown, in an embodiment of the present invention, the Correction Layer includes:

[0165] Determine the confidence map (Conf Map) of the channel and form the functional correction data of the vascular centerline distance heat map by performing 3x3x3 convolution output on the control functional feature (F_control) (formed by the segmentation feature);

[0166] Perform 3x3x3 convolution output on the control separation feature (S_control) (formed by the functional feature) to obtain the functional gradient (FFR Grad) of the same channel, and superimpose the blood flow direction guided by the vascular centerline to form the morphological correction data;

[0167] Correct the morphological feature to be corrected (S_feat) according to the morphological correction data to form the corrected morphological feature (S_feat_cor);

[0168] Correct the functional feature to be corrected (F_feat) according to the functional correction data to form the corrected functional feature (F_feat_cor).

[0169] In the subsequent steps, in the original decoder, splice the corrected features with the encoder output of the same level, perform skip connection, then decode and output in the decoder. In the corresponding upsampling branch, perform upsampling through convolution + normalization until the original resolution layer, and output the final result by connecting different task heads.

[0170] In an embodiment of the present invention, the input of the feature bidirectional correction process is the decoded feature of the previous layer, and the output is the feature that needs to be read into and spliced with the skip connection in the current layer network. At the input end, use the morphological feature of the previous layer or the previous layer as the control functional feature to calculate the segmentation confidence, and use the functional feature of the previous layer as the control morphological feature to calculate the functional gradient, so as to realize the correction of the input segmentation feature (morphological feature to be corrected) and functional feature (functional feature to be corrected). In actual use, the feature to be corrected in the feature bidirectional correction process comes from the decoder of the previous layer, and the control feature is not limited to using the output feature of the decoder of the previous layer. It can be the output feature of a certain layer decoder in the previous layer, the downsampled feature of the output result of the final prediction head, or a combination of several methods, etc. The output of the feature bidirectional correction process can be the corrected morphological feature (S_feat_cor) and the corrected functional feature (F_feat_cor), or the result after decoding after splicing the skip link features, etc.

[0171] In an embodiment of the present invention, the inputs of the two control features in the feature bidirectional correction process can also come from different levels, or a combination of different methods.

[0172] In an embodiment of the present invention, the feature to be corrected in the feature bidirectional correction process comes from the decoder of the previous layer, and the control feature in the feature bidirectional correction process can be the same as the feature to be corrected.

[0173] The deep learning method based on the guidance of the vascular centerline for task joint optimization in the embodiments of the present invention can form the mutual promotion and collaborative optimization of the accuracy of vascular morphology analysis and the accuracy of blood flow function analysis, and at the same time, can form a gating mechanism to flexibly configure the correction level and correction timing for the two decoders of the task types in the corresponding decoding branches during the two-way feature correction process. When the segmentation confidence is low, the two-way feature correction process is activated, otherwise the original features are used for fast calculation. This can balance the computational efficiency and computational accuracy of the model.

[0174] In an embodiment of the present invention, a two-way feature correction process can be set between the decoders of the same level of the task types in the corresponding decoding branches, and the activation timing of the two-way feature correction process is adjusted by a gating switch.

[0175] Such as Figure 3 As shown, in an embodiment of the present invention, according to the above deep learning method, an optimized model is formed, taking the U-Net network as the backbone network, including an encoding branch and two decoding branches. In the encoding branch, it includes:

[0176] A dynamic sampling window DSW for collecting global ROI images and local ROI images;

[0177] A dynamic focusing module DMM for upsampling and data augmentation of the local ROI image and then using an additional encoder for local image feature extraction;

[0178] An input layer Input for forming the normalization and data augmentation of the input image;

[0179] A window partitioning layer Patch-Partition for window partitioning and position encoding of the input image;

[0180] An explicit embedding layer Linear-Embed for linearly transforming the partitioned image patches to form a definite feature representation and / or a feature map of feature vectors;

[0181] An encoder module Blocks for gradually reducing the size of the feature map, increasing the abstraction degree of the features, and extracting feature information of different scales; in this embodiment, the encoder module uses a swin-transformer encoder;

[0182] A merging module Merging for reducing the size of the feature map and increasing the number of channels while performing downsampling operations to extract higher-level semantic information;

[0183] Bottleneck features Bottleneck-Features for retaining the feature map output by the last encoder module in the encoding process.

[0184] In one decoding branch, it includes:

[0185] Segmentation-Head, which is used to restore the spatial resolution of the feature map and fuse the feature information at different levels in the encoder to form the final segmentation result of the blood vessel morphology;

[0186] S-Decoder-Block, which is used for upsampling, feature splicing and convolution processing to gradually restore the size of the feature map.

[0187] In another decoding branch, it includes:

[0188] CTFFR-Head, which is used to restore the semantic information of the feature map and form the final prediction result of the CTFFR value;

[0189] F-Decoder-Block, which is used to convert the features extracted by the encoder into the predicted values of CTFFR.

[0190] Between the two decoding branches, it includes:

[0191] Two Corr-Blocks, which are used to be set between the two decoders at the same level, act on the two decoders bidirectionally, form the correction of the features received from another decoder according to the features received from one decoder, and change the feature outputs of the two decoders through feature correction;

[0192] Between Segmentation-Head and CTFFR-Head, it includes:

[0193] Loss-Block, which is used to calculate the objective loss of the prediction results of the Segmentation-Head and CTFFR-Head according to the segmentation ground truth S_gt and the CTFFR ground truth F_gt.

[0194] As Figure 4 shown, in an embodiment of the present invention, in the optimized model for blood vessel segmentation formed according to the above optimization method, the gated bidirectional correction module of the feature bidirectional correction decoder includes:

[0195] Conf-head, which is used to calculate the average confidence using convolution conv and sigmoid function;

[0196] Gate-Decision, which is used to judge the average confidence conf according to the measurement threshold threshold and form the interactive control of whether to activate the bidirectional correction process;

[0197] Correction-layer, which is used to form the bidirectional correction process.

[0198] As Figure 4 shown, in an embodiment of the present invention, the correction layer Correction-layer includes:

[0199] A CTFFR gradient calculation module FFR-Grad for calculating the CTFFR gradient direction;

[0200] A segmentation feature correction module Seg-correction for correcting the segmentation features through convolution conv processing according to the CTFFR gradient direction;

[0201] A confidence map calculation module Conf-Map for calculating the confidence map;

[0202] A prediction feature correction module FFR-Correction for correcting the FFR prediction value through additive-fusion according to the confidence weight.

[0203] In an embodiment of the present invention, in the deep learning method for task joint optimization based on blood vessel centerline guidance, the feature acquisition paths in the feature two-way correction process are as Figure 5 shown. In Figure 5 , the input feature sources of the feature two-way correction process are diverse. The input end uses the segmentation features of the previous layer to calculate the segmentation confidence (which can be understood as controlling the functional features), and uses the previous layer's CTFFR features (which can be understood as controlling the segmentation features) to calculate the CTFFR gradient, so as to realize the correction of the input segmentation features (the features to be corrected) and the CTFFR features (the features to be corrected). In actual use, the features to be corrected in the feature two-way correction process come from the previous layer's decoder. The control features are not limited to using the previous layer's decoded features, and can be the features of a certain previous layer, the downsampled or interpolated features of the output result of the final prediction head, or a combination of several methods, etc. The output of the two-way correction module can be the output of the corrected features, or the result after splicing the skip connection features, etc. Moreover, the inputs of the two control features in the two-way correction process can also come from different layers, or a combination of different methods.

[0204] In Figure 5 , the dotted part is the function control features that can be selected, including:

[0205] a) Skip-layer, the decoded features obtained from the previous previous layer, obtaining features from a larger receptive field and lower resolution, suitable for global correction from high-level semantic features (such as the overall shape of blood vessels, etc.), with a small amount of calculation, and for the current calculated feature results being known, without affecting the network calculation order;

[0206] b) Pre-layer, the decoded features obtained from the previous layer, with consistent feature resolution, without calculations such as interpolation, upsampling, and downsampling, having high computational efficiency, a simple model, and for the current calculated feature results being known, without affecting the network calculation order;

[0207] c) Down-sample, the confidence features obtained by downsampling from the output result, the features being closer to the output result, capable of providing low-level details, enhancing the correction effect, without information loss, but the prediction result of the current layer being unknown. Therefore, the two correction modules can only correct alternately, and the model needs to be trained step by step, with a more complex model design.

[0208] When the pre-layer connection method is selected and the control features come from the previous layer, the features to be corrected are used as control features at the same time (i.e., S_control = S_feat).

[0209] Combined Figure 4 and Figure 5 As shown, the gating mechanism formed by the gating module is also an optional module, which can balance the computational efficiency and its corresponding model effect. When computational efficiency is not considered, the gating module can be discarded, so as to realize the correction of the features of all input correction modules.

[0210] In an embodiment of the present invention, in the deep learning method for task joint optimization based on blood vessel centerline guidance, the loss function in the optimization model formed by a U-Net backbone network is as Figure 6 shown. Combined Figure 1-1 and Figure 6 shown, this embodiment further includes:

[0211] Step 400: Using the blood vessel morphology parameters and blood flow function parameters obtained by the decoding branches of the two task types as the independent variables of the physical constraint conditions, determining the physical constraint loss function as a component of the total loss function according to the residual between the predicted value and the true value of the dependent variable of the physical constraint conditions, and optimizing the decoder and the feature two-way correction process through the gradient backpropagation of the total loss function.

[0212] Those skilled in the art can understand that the segmentation result is formed by dividing the task header, and the prediction result is formed by predicting the task header. The segmentation result and the prediction result are for the corresponding vascular morphological parameters and blood flow function parameters. By determining a specific physical constraint condition (physical equation) with the vascular morphological parameters and blood flow function parameters as independent variables, the predicted value of the dependent variable parameter can be obtained based on the predicted values formed by the vascular morphological parameters and blood flow function parameters. The physical constraint loss function is determined by the residual between the predicted value of the dependent variable parameter and the true value of the dependent variable parameter obtained by quantitative calculation. Taking the physical constraint loss function as part of the total loss function of the model, the parameters of the network are iteratively backpropagated using the total loss function to optimize the model accuracy. The specific physical constraint conditions include, but are not limited to, the control equation of Poiseuille flow in fluid mechanics, the continuity equation of incompressible fluids, the momentum equation, etc. The specific physical constraint conditions can be global or local. For example, the blood flow in all blood vessels needs to satisfy the Poiseuille flow related to the segmented blood vessel diameter; the blood vessels at the bifurcation should satisfy the continuity equation, that is, the total blood flow into each branch is equal to the blood flow in the parent blood vessel, etc. The physical constraint methods and formulas used are optional, and other similar physical equations that can achieve the constraint between geometric information and functional parameter information can be used as appropriate.

[0213] The optimization effect of the deep learning method based on the guidance of the vascular centerline for task joint optimization in the embodiment of the present invention is achieved by adding the loss of composite physical conditions. Using the parameter equation that conforms to the physical laws of fluid mechanics to constrain the functional parameters predicted by the model, calculating the residual after substituting the predicted value into the equation, making the prediction result more conform to the physical laws, and improving the rationality of blood flow function prediction. The physical constraint indirectly acts on the morphological channels such as blood vessel segmentation through the bidirectional interactive correction structure designed in the model, and synchronously improves the accuracy of the segmentation result.

[0214] Combined Figure 2 and Figure 6 As shown in, in an embodiment of the present invention, for the segmentation task and the CTFFR prediction task of the model, step 400 includes:

[0215] Step 410: Determine the segmentation loss function according to the residual between the prediction result and the segmentation ground truth of the segmentation task.

[0216] Specifically, the segmentation loss Seg_loss is determined according to the residual between the segmentation prediction result S_pred and the segmentation ground truth S_gt of the segmentation task. The segmentation loss function L_s adopts the Dice (Dice Coefficient) loss function combined with the BCE (BinaryCross Entropy) loss function.

[0217] Step 420: Determine the prediction loss function according to the residual between the prediction result and the prediction ground truth of the prediction task.

[0218] Specifically, the CTFFR loss FFRLoss is determined according to the residual between the predicted result F_pred and the true value F_gt of the CTFFR prediction. The loss function L_F of the CTFFR loss adopts the MSE (Mean Squared Error) loss function.

[0219] Step 430: Select a segmentation task and a CTFFR prediction task; obtain the blood vessel radius parameter using the result of the segmentation task and obtain the pressure difference parameter from the result of the CTFFR prediction task. The blood flow prediction value is obtained through the Poiseuille flow equation using the blood vessel radius parameter and the pressure difference parameter, and the true value of the blood vessel blood flow is formed through the fluid continuity equation using the blood vessel radius parameter and the pressure difference parameter. The physical constraint loss function is determined according to the residual between the blood flow prediction value and the true value of the blood vessel blood flow.

[0220] Specifically, the predicted value Ves-R of the blood vessel radius is calculated from the segmentation map of the segmentation task, and the pressure difference FFR between the blood vessel distal pressure and the coronary ostium pressure is calculated using the pressure prediction value of the CTFFR prediction task. The blood flow prediction value Q is obtained by substituting the pressure difference formed by the predicted values and the blood vessel radius into the Poiseuille flow formula. At the same time, according to the fluid continuity equation (|Q _in -Q _out |), the physical law constraint on the blood vessel inflow blood volume Q _in , outflow blood volume Q _out is used to judge the residual between the blood flow prediction value and the true value, and the physical constraint phisics-constraints is established, and the physical constraint loss is determined. The physical constraint loss function L_p adopts the MSE (Mean Squared Error) loss function.

[0221] Step 440: Form the total loss according to the segmentation loss, prediction loss, and physical constraint loss, and use the total loss function to optimize the network parameters of the decoder and the feature two-way correction process through gradient backpropagation.

[0222] Total loss function = segmentation loss function L_s + CTFFR loss loss function L_F + physical constraint loss function L_p

[0223] The network parameters of the decoder and the feature two-way correction process are iteratively optimized through the total loss function until the accuracy requirement of the model is met.

[0224] An embodiment of the present invention is a deep learning device for task joint optimization based on blood vessel centerline guidance as Figure 7 shown. In Figure 7 , this embodiment includes:

[0225] An encoder 10, which is used to extract local image features after data augmentation of local images with morphological specificity according to the guidance of the vascular centerline during the process of global image feature extraction by the encoder, and form fused image features by fusing the local image features and the global image features;

[0226] A decoding correction module 10a, which is used to form a feature two-way correction process between decoders of different decoding branches.

[0227] As Figure 7 shown, in an embodiment of the present invention, the decoding correction module 10a includes: a decoder 20, which is used to configure the decoder to decode the fused image features according to the task type, and form at least one decoding branch for the vascular morphology task and at least one decoding branch for the blood flow function task;

[0228] A two-way correction decoder 30, which is used to form a feature two-way correction process of using the output features of one decoder to correct the output features of another decoder between the decoders of two task types in a pair of decoding branches of different task types.

[0229] As Figure 7 shown, in an embodiment of the present invention, the encoding setting module 10 includes:

[0230] A global image sampling unit 11, which is used to perform global sampling on the input image data containing blood vessels according to the guidance of the vascular centerline to form a global ROI;

[0231] A global feature extraction unit 12, which is used to perform global image feature extraction on the global ROI through the encoder;

[0232] A local image selection unit 13, which is used to determine position attention information according to the vascular specificity judgment rule in the global ROI, and perform local sampling in the global ROI according to the position attention information to form a local ROI dataset;

[0233] A local feature extraction unit 14, which is used to perform upsampling and data augmentation on the local ROI, extract local image features using an additional encoder, and perform position encoding of the local image features using the vascular centerline;

[0234] A fused feature formation unit 15, which is used to fuse the local image features and the global image features using the attention mechanism to form fused image features.

[0235] As Figure 7 shown, in an embodiment of the present invention, the decoding setting module 20 includes:

[0236] A segmentation task setting unit 21, which is used to set a segmentation decoding branch reflecting the vascular morphology task, and set a segmentation decoder corresponding to the same-level encoder in the segmentation decoding branch;

[0237] A prediction task setting unit 22, configured to set a prediction decoding branch reflecting a hemodynamic function task, and set a prediction decoder corresponding to the encoder at the same level in the prediction decoding branch.

[0238] As Figure 7 shown, in an embodiment of the present invention, the bidirectional correction decoder 30 includes:

[0239] A forward correction unit 31, configured to establish a forward correction process for correcting the functional features according to the morphological features between two decoders of the task type corresponding to the decoding branch;

[0240] A reverse correction unit 32, configured to simultaneously establish a reverse correction process for correcting the morphological features according to the functional features between the two decoders.

[0241] As Figure 7 shown, in an embodiment of the present invention, the bidirectional correction decoder 30 further includes:

[0242] A forward correction enhancement unit 31a, configured to establish a forward correction process for correcting the functional features according to the morphological features and the vascular centerline distance heat map between two decoders of the task type corresponding to the decoding branch;

[0243] A reverse correction enhancement unit 32a, configured to simultaneously establish a reverse correction process for correcting the morphological features according to the functional features and the blood flow direction guided by the vascular centerline between the two decoders.

[0244] As Figure 7 shown, in an embodiment of the present invention, the bidirectional correction decoder 30 further includes:

[0245] A gating switch unit 33, configured to set a measurement threshold for the average confidence of the morphological features during the gating judgment process, and determine whether to activate the feature bidirectional correction process according to the measurement result.

[0246] As Figure 7 shown, in an embodiment of the present invention, the bidirectional correction decoder 30 further includes:

[0247] A confidence acquisition unit 34, configured to acquire the morphological features to be corrected, the functional features to be corrected, the control segmentation features and the control functional features, and form a confidence map and an average confidence according to the control functional features;

[0248] A feature direct connection unit 35, configured to directly output the morphological features to be corrected and the functional features to be corrected from the segmentation decoder and the prediction decoder respectively when the average confidence is greater than the measurement threshold;

[0249] The bidirectional correction execution unit 36 is used to activate the feature bidirectional correction process when the average confidence is less than the measurement threshold.

[0250] As Figure 7 shown, in an embodiment of the present invention, it further includes:

[0251] The physical constraint formation module 40 is used to use the vascular morphology parameters and blood flow function parameters obtained by the decoding branches of the two task types as the independent variables of the physical constraint conditions, determine the physical constraint loss function as a component of the total loss function according to the residual between the predicted value and the true value of the dependent variable of the physical constraint conditions, and optimize the decoder and the feature bidirectional correction process through the gradient backpropagation of the total loss function.

[0252] As Figure 7 shown, in an embodiment of the present invention, the physical constraint formation module 40 includes:

[0253] The segmentation loss formation unit 41 is used to determine the segmentation loss function according to the residual between the prediction result and the segmentation true value of the segmentation task;

[0254] The prediction loss formation unit 42 is used to determine the prediction loss function according to the residual between the prediction result and the prediction true value of the prediction task;

[0255] The physical constraint loss formation unit 43 is used to select a segmentation task and a CTFFR prediction task; obtain the vascular radius parameter from the segmentation task result and the pressure difference parameter from the CTFFR prediction task result, obtain the blood flow prediction value through the Poiseuille flow equation with the vascular radius parameter and the pressure difference parameter, form the true value of the vascular blood flow through the fluid continuity equation with the vascular radius parameter and the pressure difference parameter, and determine the physical constraint loss function according to the residual between the blood flow prediction value and the true value of the vascular blood flow;

[0256] The iterative optimization unit 44 is used to form the total loss according to the segmentation loss, the prediction loss and the physical constraint loss, and optimize the network parameters of the decoder and the feature bidirectional correction process through the gradient backpropagation of the total loss function.

[0257] The network structure of the deep learning device for task joint optimization based on vascular centerline guidance in an embodiment of the present invention is as Figure 8 shown. In Figure 8 this embodiment includes:

[0258] The dynamic sampling module (DSW) is used to perform data sampling of local ROI and global ROI for vascular-related images using the vascular centerline;

[0259] A Dynamic Focusing Module (DMM) is used to upsample (High_res) the local ROI filtered by the judgment rule (Rule), perform data augmentation to extract local image features, and fuse (Feature Fusion) them with global image features;

[0260] An Encoder is used to extract global image features from the global ROI (downsampling);

[0261] A Decoder is used to form different decoding branches for different task types and decode the fused image features;

[0262] A Bidirectional Correction Decoder (BDC Decoder) is used to form a feature bidirectional correction process between two decoders in a pair of decoding branches of different task types, where the output features of one decoder are used to correct the output features of the other decoder.

[0263] In an embodiment of the present invention, the decoder includes a morphological decoder and a functional decoder. The decoder is configured according to the task type to decode the fused image features, forming at least one decoding branch for the vascular morphological task and at least one decoding branch for the blood flow functional task.

[0264] The deep learning device for task joint optimization based on vascular centerline guidance in the embodiments of the present invention forms a network structure based on the encoder and decoder. Through the association extraction structure of global and local features formed by using the vascular centerline during the encoding process, an equilibrium between the accuracy and efficiency of detailed vascular feature extraction is achieved. Through the bidirectional correction decoder between the decoding branch structures formed by the decoder, a bidirectional correction process of the features of the same-level decoders is formed. By using the blood flow functional state that has a prior correlation with the vascular morphology, a means of joint analysis of the vascular morphology is established, forming a joint promotion of the analysis of vascular morphology and function. This ensures the physical consistency of anatomical reconstruction and functional prediction.

[0265] The embodiments of the present application also provide a specific implementation manner of an electronic device that can implement all the steps in the methods in the above embodiments. Refer to Figure 9 , the electronic device 600 specifically includes the following:

[0266] A processor 610, a memory 620, a communication unit 630, and a bus 640;

[0267] Among them, the processor 610, the memory 620, and the communication unit 630 communicate with each other through the bus 640; the communication unit 630 is used to implement information transmission between related devices such as server-side devices and terminal devices.

[0268] The processor 610 is used to call the computer program in the memory 620. When the processor executes the computer program, all steps in the deep learning method for task joint optimization based on blood vessel centerline guidance in the above embodiments are implemented.

[0269] Those of ordinary skill in the art should understand that the memory can be, but is not limited to, random access memory (Random Access Memory, abbreviated as RAM), read only memory (Read Only Memory, abbreviated as ROM), programmable read only memory (Programmable Read-Only Memory, abbreviated as PROM), erasable programmable read only memory (Erasable Programmable Read-Only Memory, abbreviated as EPROM), electrically erasable programmable read only memory (Electric Erasable Programmable Read-Only Memory, abbreviated as EEPROM), etc. Among them, the memory is used to store programs, and the processor executes the programs after receiving execution instructions. Further, the software programs and modules in the above memory may also include an operating system, which may include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide a running environment for other software components.

[0270] The processor can be an integrated circuit chip with signal processing capabilities. The above-mentioned processor can be a general-purpose processor, including a central processing unit (Central Processing Unit, abbreviated as CPU), a network processor (Network Processor, abbreviated as NP), etc. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0271] The present application also provides a computer-readable storage medium, and the computer-readable storage medium includes a program, and the program is used to execute the deep learning method for task joint optimization based on blood vessel centerline guidance provided in any one of the foregoing method embodiments when executed by a processor.

[0272] Those of ordinary skill in the art should understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments; and the foregoing storage medium includes: ROM, RAM, magnetic disk or optical disk and other various media that can store program codes, and the specific type of the medium is not limited in the present application.

[0273] As described above, it is only the preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A deep learning method for joint optimization guided by the vascular centerline, characterized in that, Including: During the process of the encoder extracting global image features, local image features with morphological specificity are enhanced according to the guidance of the blood vessel centerline and then the local image features are extracted. The fused image features are formed by fusing the local image features and the global image features; A feature two-way correction process is formed between the decoders of different decoding branches.

2. The deep learning method according to claim 1, wherein The forming of the feature two-way correction process between the decoders of different decoding branches includes: Configuring the decoder according to the task type to decode the fused image features, forming at least one decoding branch for the blood vessel morphology task and at least one decoding branch for the blood flow function task; In a pair of decoding branches of different task types, a feature two-way correction process is formed between the decoders of the two task types to correct the output features of one decoder with the output features of the other decoder.

3. The deep learning method according to claim 1, wherein The forming of the fused image features includes: Performing global sampling on the input image containing blood vessels according to the guidance of the blood vessel centerline to form a global ROI; Extracting global image features from the global ROI through the encoder; Determining position attention information in the global ROI according to the blood vessel specificity judgment rule, and performing local sampling in the global ROI according to the position attention information to form a local ROI dataset; Upsampling and data enhancing the local ROI, then using an additional encoder to extract local image features, and performing position encoding of the local image features using the blood vessel centerline; Using the attention mechanism to fuse the local image features and the global image features to form the fused image features.

4. The deep learning method according to claim 3, wherein The blood vessel specificity judgment rule includes the judgment of blood vessel stenosis rate, blood vessel bifurcation, and blood vessel curvature.

5. The deep learning method according to claim 3, characterized in that The determining position attention information in the global ROI according to the blood vessel specificity judgment rule and performing local sampling in the global ROI according to the position attention information to form a local ROI dataset is replaced by: Adjusting the local ROI in the global ROI for local sampling to form a local ROI dataset, and determining the position attention information of the local ROI including the specific morphology of the blood vessel according to the blood vessel specificity judgment rule.

6. The deep learning method according to claim 3, wherein The upsampling is set according to the resolution level of the fusion of the local image features and the global image features.

7. The deep learning method according to claim 3, characterized in that The additional encoder extracts local features in a cascaded 3D convolution manner and fuses them with the blood vessel centerline point position encoding to obtain local image features.

8. The deep learning method according to claim 3, wherein The size of the local ROI is dynamically adjusted according to the blood vessel specificity judgment rule.

9. The deep learning method according to claim 3, wherein The blood vessel centerline contains stenosis information and is obtained by one of the following methods: When the centerline is known, the cross-section diameter is obtained by 2D-Unet segmentation of the cross-section, and the stenosis degree along the centerline is calculated. When the centerline is unknown, lightweight 3D-Unet is used to segment the whole image and extract the centerline coordinates, and the blood vessel stenosis on the centerline is calculated.

10. The deep learning method according to claim 2, wherein The configuring the decoder according to the task type to decode the fused image features, forming at least one decoding branch for the blood vessel morphology task and at least one decoding branch for the blood flow function task includes: Setting a segmentation decoding branch reflecting the blood vessel morphology task, and setting a segmentation decoder corresponding to the same-level encoder in the segmentation decoding branch; The blood vessel morphology task includes at least one of contour, inner diameter, inner wall mask, or topological shape; A prediction decoding branch reflecting a hemodynamic function task is set, and a prediction decoder corresponding to the encoder at the same level is set in the prediction decoding branch; the hemodynamic function task includes at least one of blood pressure parameters, blood flow parameters, and CTFFR.

11. The deep learning method according to claim 2, wherein The input features of the feature bidirectional correction process include the feature output of the decoder at the same level as the feature to be corrected, and also include the feature output of the decoder at the same level as the control feature, the feature output of the previous-level decoder, or the downsampled feature output of the final output result of the task. The control feature is used to form an interactive judgment to control the feature bidirectional correction process.

12. The deep learning method according to claim 2, wherein The feature bidirectional correction process includes: Establish a forward correction process for correcting functional features based on morphological features between two decoders of the task type corresponding to the decoding branch; At the same time, establish a reverse correction process for correcting morphological features based on functional features between the two decoders.

13. The deep learning method according to claim 12, wherein The forward correction process includes: Generate a spatial attention weight according to the segmentation confidence map of the morphological features, and dynamically modulate the functional feature map to obtain the corrected functional features.

14. The deep learning method according to claim 12, characterized in that, The reverse correction process includes: Correct the morphological features according to the change gradient direction of the functional features to obtain the corrected morphological features.

15. The deep learning method according to claim 2, wherein The forming of the feature bidirectional correction process includes: Establish a forward correction process for correcting functional features based on morphological features and the vascular centerline distance heat map between two decoders of the task type corresponding to the decoding branch; At the same time, establish a reverse correction process for correcting morphological features based on functional features and the blood flow direction guided by the vascular centerline between the two decoders.

16. The deep learning method according to claim 2, wherein The forming of the feature bidirectional correction process includes: Set a measurement threshold for the average confidence of the morphological features in the gating judgment process, and determine whether to activate the feature bidirectional correction process according to the measurement result.

17. The deep learning method according to claim 16, wherein The setting of the measurement threshold for the average confidence of the morphological features in the gating judgment process and determining whether to activate the feature bidirectional correction process according to the measurement result includes: Obtain the morphological features to be corrected, the functional features to be corrected, the control segmentation features, and the control functional features, and form a confidence map and an average confidence according to the control functional features; When the average confidence is greater than the measurement threshold, directly output the morphological features to be corrected and the functional features to be corrected from the segmentation decoder and the prediction decoder respectively; When the average confidence is less than the measurement threshold, activate the feature bidirectional correction process.

18. The deep learning method according to claim 16, characterized in that, The forming of the feature bidirectional correction process includes: Output a confidence map (ConfMap) for determining channels and a vascular centerline distance heat map through 3x3x3 convolution of the control functional feature (F_control) to form functional correction data; Output the functional gradient (FFRGrad) of the same channel through 3x3x3 convolution of the control segmentation feature (S_control) and superimpose the blood flow direction guided by the vascular centerline to form morphological correction data; Correct the morphological features to be corrected (S_feat) according to the morphological correction data to form the corrected morphological features (S_feat_cor); The to-be-corrected functional feature (F_feat) is corrected according to the function-corrected data to form the corrected functional feature (F_feat_cor).

19. The deep learning method according to claim 2, wherein It further includes: Using the vascular morphology parameters and blood flow function parameters obtained by the decoding branches of the two task types as the independent variables of the physical constraint conditions, determining the physical constraint loss function as a component of the total loss function according to the residual between the predicted value and the true value of the dependent variable of the physical constraint conditions, and optimizing the decoder and the feature two-way correction process through the gradient backpropagation of the total loss function.

20. The deep learning method according to claim 19, wherein The determination of the physical constraint loss function as a component of the total loss function and the optimization of the decoder and the feature two-way correction process through the gradient backpropagation of the total loss function include: Determining the segmentation loss function according to the residual between the prediction result and the segmentation true value of the segmentation task; Determining the prediction loss function according to the residual between the prediction result and the prediction true value of the prediction task; Selecting a segmentation task and a CTFFR prediction task, obtaining the vessel radius parameter from the segmentation task result and the pressure difference parameter from the CTFFR prediction task result. The blood flow prediction value is obtained through the Poiseuille flow equation with the vessel radius parameter and the pressure difference parameter, and the true value of the vessel blood flow is formed through the fluid continuity equation with the vessel radius parameter and the pressure difference parameter. The physical constraint loss function is determined according to the residual between the blood flow prediction value and the true value of the vessel blood flow; Forming the total loss according to the segmentation loss, the prediction loss, and the physical constraint loss, and optimizing the network parameters of the decoder and the feature two-way correction process through the gradient backpropagation of the total loss function.

21. A deep learning device for joint optimization guided by the blood vessel centerline, characterized in that It includes: An encoder, which is used to enhance the data of the local image with morphological specificity according to the guidance of the vessel centerline during the global image feature extraction process of the encoder, extract the local image features, and form the fused image features by fusing the local image features and the global image features; A decoding correction module, which is used to form a feature two-way correction process between the decoders of different decoding branches.

22. The deep learning device according to claim 20, wherein The decoding correction module includes: A decoder, which is used to configure the decoder according to the task type to decode the fused image features, and form at least one decoding branch for the vascular morphology task and at least one decoding branch for the blood flow function task; A two-way correction decoder, which is used to form a feature two-way correction process of using the output features of one decoder to correct the output features of another decoder between the decoders of two task types in a pair of decoding branches of different task types.

23. The deep learning device according to claim 22, characterized in that, It further includes: A physical constraint formation module, which is used to use the vascular morphology parameters and blood flow function parameters obtained by the decoding branches of the two task types as the independent variables of the physical constraint conditions, determine the physical constraint loss function as a component of the total loss function according to the residual between the predicted value and the true value of the dependent variable of the physical constraint conditions, and optimize the decoder and the feature two-way correction process through the gradient backpropagation of the total loss function.

24. An electronic device, characterized in that, It includes: A processor, a memory, and an interface for communicating with the gateway; The memory is used to store programs and data, and the processor calls the programs stored in the memory to execute the method according to any one of claims 1 to 20.

25. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a program which, when executed by a processor, is used to perform the method according to any one of claims 1 to 20.

Images