Task joint optimization method and device for blood vessel morphology and blood vessel blood flow characteristics

Abstract

The invention provides a task joint optimization method and device for blood vessel morphology and blood vessel blood flow characteristics, and solves the technical problem that blood vessel segmentation cannot utilize the physical consistency of blood flow functions. The method comprises the following steps: in an encoding process, carrying out image feature extraction by utilizing a hybrid encoder at least suitable for two task types; according to the task type, configuring a decoder to decode the image features, and forming a decoding branch of at least one blood vessel morphology task type and a decoding branch of at least one blood flow functional task type; in a pair of decoding branches of different task types, a feature bidirectional correction process for correcting the output feature of one decoder by using the output feature of the other decoder is formed between the decoders of the two task types. And collaborative improvement of blood vessel anatomical reconstruction and hemodynamic precision is realized. The problem of scarcity of hemodynamic labeling is relieved, and the generalization ability and segmentation precision of the model are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning in medical imaging, and particularly relates to a method and device for joint optimization of tasks oriented to vascular morphology and vascular blood flow characteristics. Background Art

[0002] To accurately evaluate the myocardial hemodynamic abnormalities caused by coronary artery stenosis, invasive methods are usually used to obtain gold-standard blood flow function parameters such as fractional flow reserve (FFR) and instantaneous wave-free ratio (iFR). The angiographic images using CTA (Computed Tomography Angiography) technology can non-invasively construct a vascular geometric model, but there are defects in analyzing the degree of functional ischemia from only the anatomical perspective of vascular characteristics to form a quantitative blood flow function index. Through computational fluid dynamics techniques such as CT-FFR (Computed Tomography-Fractional Flow Reserve) and QFR (Quantitative Flow Ratio), on the basis of the vascular geometric model, by setting physiological boundary conditions and state parameters, the blood flow state of the real blood vessel can be simulated to quantify the characteristic parameters of the blood flow function at the target position. Existing deep learning techniques can form a segmentation detection model for vascular morphology and a functional prediction model for hemodynamics, and quickly make a targeted assessment of the target blood vessel in terms of vascular morphology or blood flow function in combination with the patient's specific baseline data.

[0003] In deep learning techniques, the segmentation task for vascular morphology and the regression task for blood flow function usually both include an encoder structure for feature extraction and a decoder structure for restoring the spatial dimension or forming a regression trend. Task training mostly adopts a separate learning paradigm, that is, for a single task requirement, a segmentation model or a prediction model for vascular morphology segmentation is constructed through independent training data, objective functions, feature extraction modules, segmentation detection modules or regression prediction modules. A cascade mode is used to cascade step by step between the detection model and the prediction model, and the vascular morphology and blood flow function are processed separately and evaluated step by step.

[0004] From the prior knowledge of fluid mechanics, it can be known that there is a credible association between the change in blood flow function and the change in vascular morphology, and the change in vascular morphology can be deduced from the change in blood flow functional parameters. However, due to the one-way cascade between models, the model correlation is weakened, and only the vascular segmentation result can affect the prediction of blood flow function parameters, so that the vascular morphology segmentation error accompanying the detection model will be transmitted to the prediction model, resulting in a cascade error in the prediction of blood flow function parameters.

[0005] It can be seen that the mode of treating vascular anatomical segmentation detection and blood flow function prediction as independent tasks will lead to the following key defects:

[0006] First, the pathological correlation between vascular anatomical morphology and blood flow function parameters has not been effectively modeled, which easily leads to inconsistent diagnostic conclusions. For example, although the anatomical stenosis is significant, the hemodynamic parameters do not reach the ischemia threshold.

[0007] Second, the labeled data for each task during the training process has not been co-optimized. The scarcity of hemodynamic labels will limit the generalization ability of the model.

[0008] Third, each model mostly relies on fully supervised training and fails to effectively integrate weakly labeled data or cross-modal prior knowledge, resulting in a lack of consistency between the prediction results and the true physical laws, which restricts clinical deployment.

[0009] In the prior art, although there is a training mode that adopts multi-task joint, it pays more attention to the detection of functional abnormalities and the corresponding physiological structure estimation in the same diagnostic purpose, and essentially ignores the physical consistency between morphological anatomical reconstruction and blood flow function prediction. Summary of the Invention

[0010] In view of the above problems, the embodiments of the present invention provide a task joint optimization method and device for vascular morphology and vascular blood flow characteristics, which solve the technical problems that the existing vascular segmentation detection cannot utilize the physical consistency of blood flow function and cannot form a synergistic promotion to improve the overall analysis accuracy.

[0011] The task joint optimization method for vascular morphology and vascular blood flow characteristics according to the embodiments of the present invention includes:

[0012] During the encoding process, a hybrid encoder that adapts to at least two task types is used to extract image features;

[0013] According to the task type, a decoder is configured to decode the image features, forming at least one decoding branch of the vascular morphology task type and at least one decoding branch of the blood flow function task type;

[0014] In a pair of decoding branches of different task types, a feature bidirectional correction process is formed between the decoders of the two task types, using the output features of one decoder to correct the output features of the other decoder.

[0015] In an embodiment of the present invention, the using a hybrid encoder that adapts to at least two task types to extract image features includes:

[0016] Obtain an image containing blood vessels for preprocessing;

[0017] During the encoding process, for the classification task and the regression task, a hybrid encoder is used to hierarchically encode the image containing blood vessels to obtain image features of several resolutions.

[0018] In one embodiment of the present invention, the preprocessing of the acquired image containing blood vessels includes:

[0019] Using the point coordinate information of the blood vessel centerline to encode the direction and position of the image containing blood vessels divided by the window, and forming feature extraction for the local details of the image containing blood vessels.

[0020] In one embodiment of the present invention, the configuration of the decoder according to the task type to decode the fused image features to form at least one decoding branch for the blood vessel morphology task and at least one decoding branch for the blood flow function task includes:

[0021] Setting a segmentation decoding branch reflecting the blood vessel morphology task, and setting a segmentation decoder corresponding to the same-level encoder in the segmentation decoding branch; the blood vessel morphology task includes at least one of contour, inner diameter, inner wall mask or topological shape;

[0022] Setting a prediction decoding branch reflecting the blood flow function task, and setting a prediction decoder corresponding to the same-level encoder in the prediction decoding branch; the blood flow function task includes at least one of blood pressure parameters, blood flow parameters, and CTFFR.

[0023] In one embodiment of the present invention, the input features of the feature bidirectional correction process include the feature output of the same-level decoder as the feature to be corrected, and also include the feature output of the same-level decoder as the control feature, the feature output of the previous-level decoder, or the downsampled feature output of the task final output result, and an interaction judgment is formed using the control feature to control the feature bidirectional correction process.

[0024] In one embodiment of the present invention, the feature bidirectional correction process includes:

[0025] Establish a forward correction process for correcting the functional features according to the morphological features between two decoders corresponding to the task types of the decoding branches;

[0026] At the same time, establish a reverse correction process for correcting the morphological features according to the functional features between the two decoders.

[0027] In one embodiment of the present invention, the forward correction process includes:

[0028] Generating spatial attention weights according to the segmentation confidence map of the morphological features, dynamically modulating the functional feature map, and obtaining the corrected functional features.

[0029] In one embodiment of the present invention, the reverse correction process includes:

[0030] Correcting the morphological features according to the change gradient direction of the functional features to obtain the corrected morphological features.

[0031] In one embodiment of the present invention, the feature two-way correction process includes:

[0032] Set a measurement threshold for the average confidence of morphological features in the gate decision process (Gate Decision), and determine whether to activate the feature two-way correction process (Correction Layer) according to the measurement result.

[0033] In one embodiment of the present invention, the setting of the measurement threshold for the average confidence of morphological features in the gate decision process and determining whether to activate the feature two-way correction process includes:

[0034] Obtain the morphological feature to be corrected (S_feat), the functional feature to be corrected (F_feat), the control segmentation feature (S_control), and the control functional feature (F_control), and form a confidence map and an average confidence according to the control functional feature;

[0035] When the average confidence is greater than the measurement threshold, directly output the morphological feature to be corrected (S_feat) and the functional feature to be corrected (F_feat) from the segmentation decoder and the prediction decoder respectively;

[0036] When the average confidence is less than the measurement threshold, activate the feature two-way correction process.

[0037] In one embodiment of the present invention, the feature two-way correction process includes:

[0038] Output functional correction data of the confidence map (ConfMap) for determining channels by performing 3x3x3 convolution on the control functional feature (F_control);

[0039] Output the functional gradient (FFRGrad) of the same channel by performing 3x3x3 convolution on the control segmentation feature (S_control) to form morphological correction data;

[0040] Correct the morphological feature to be corrected (S_feat) according to the morphological correction data to form the corrected morphological feature (S_feat_cor);

[0041] Correct the functional feature to be corrected (F_feat) according to the functional correction data to form the corrected functional feature (F_feat_cor).

[0042] In one embodiment of the present invention, it further includes:

[0043] Using the vascular morphological parameters and blood flow function parameters obtained from two types of tasks as the independent variables of the physical constraint conditions, determining the physical constraint loss function as a component of the total loss function according to the residual between the predicted value and the true value of the dependent variable of the physical constraint conditions, and optimizing the decoder and the feature two-way correction process through the gradient backpropagation of the total loss function.

[0044] In one embodiment of the present invention, the determining the physical constraint loss function as a component of the total loss function and optimizing the decoder and the feature two-way correction process through the gradient backpropagation of the total loss function includes:

[0045] Determining a segmentation loss function according to the residual between the prediction result and the segmentation true value of the segmentation task;

[0046] Determining a prediction loss function according to the residual between the prediction result and the prediction true value of the prediction task;

[0047] Selecting a segmentation task and a CTFFR prediction task of the model, obtaining the vascular radius parameter from the result of the segmentation task, obtaining the pressure difference parameter from the result of the CTFFR prediction task, obtaining the predicted blood flow value through the Poiseuille flow equation with the vascular radius parameter and the pressure difference parameter, forming the true value of the vascular blood flow through the fluid continuity equation with the vascular radius parameter and the pressure difference parameter, and determining the physical constraint loss function according to the residual between the predicted blood flow value and the true value of the vascular blood flow;

[0048] Forming a total loss according to the segmentation loss, the prediction loss and the physical constraint loss, and optimizing the network parameters of the decoder and the feature two-way correction process through the gradient backpropagation of the total loss function.

[0049] The task joint optimization device for vascular morphology and vascular blood flow characteristics in the embodiment of the present invention includes:

[0050] An encoder, configured to use a hybrid encoder adapted to at least two task types to extract image features during the encoding process of the vascular segmentation model;

[0051] A decoder, configured to decode the image features according to the task type, forming at least one decoding branch of the vascular morphology task type and at least one decoding branch of the blood flow function task type;

[0052] A two-way correction decoder, configured to form a feature two-way correction process of using the output features of one decoder to correct the output features of another decoder between the two decoders in a pair of decoding branches of different task types.

[0053] In one embodiment of the present invention, it further includes:

[0054] A physical constraint formation module is configured to use the vascular morphological parameters and blood flow function parameters obtained from two task types as independent variables of physical constraint conditions, determine a physical constraint loss function as a component of the total loss function based on the residual between the predicted value and the true value of the dependent variable of the physical constraint conditions, and optimize the decoder and the feature two-way correction process through backpropagation of the total loss function gradient.

[0055] The electronic device according to an embodiment of the present invention includes:

[0056] A processor, a memory, and an interface for communicating with a gateway;

[0057] The memory is used to store programs and data, and the processor calls the programs stored in the memory to execute the above method.

[0058] The computer-readable storage medium according to an embodiment of the present invention includes a program that, when executed by a processor, is used to execute the above method.

[0059] The electronic device according to an embodiment of the present invention includes:

[0060] A processor, a memory, and an interface for communicating with a gateway;

[0061] The memory is used to store programs and data, and the processor calls the programs stored in the memory to execute the above method.

[0062] The computer-readable storage medium according to an embodiment of the present invention is characterized in that the computer-readable storage medium includes a program that, when executed by a processor, is used to execute the above method.

[0063] The task joint optimization method and device for vascular morphology and vascular blood flow characteristics according to an embodiment of the present invention achieve a collaborative improvement in anatomical reconstruction accuracy and hemodynamic prediction through end-to-end joint optimization. The model uses two types of labels for blood flow function and vascular morphology, alleviates the scarcity problem of hemodynamic annotation, and improves the generalization ability of the model. At the same time, prior knowledge is used to form physical constraints on the output results, ensuring the rationality of the blood flow parameter prediction results and improving the segmentation accuracy. Description of the Drawings

[0064] Figure 1 The figure shows a schematic flow chart of a task joint optimization method for vascular morphology and vascular blood flow characteristics according to an embodiment of the present invention.

[0065] Figure 2 The figure shows a schematic flow chart of multi-task optimization using a U-Net backbone network in a task joint optimization method for vascular morphology and vascular blood flow characteristics according to an embodiment of the present invention.

[0066] Figure 3 The figure shows a schematic structural diagram of an optimization model formed by using a U-Net backbone network for the task joint optimization method for vascular morphology and vascular blood flow characteristics according to an embodiment of the present invention.

[0067] Figure 4 The figure shows a schematic structural diagram of a gated bidirectional correction module in the optimization model formed by the task joint optimization method for vascular morphology and vascular blood flow characteristics according to an embodiment of the present invention.

[0068] Figure 5 The figure shows a schematic diagram of the feature acquisition path of the feature bidirectional correction process in the optimization model formed by the task joint optimization method for vascular morphology and vascular blood flow characteristics according to an embodiment of the present invention.

[0069] Figure 6 The figure shows a schematic structural diagram of the loss function in the optimization model formed by the task joint optimization method for vascular morphology and vascular blood flow characteristics according to an embodiment of the present invention.

[0070] Figure 7 The figure shows a schematic structural diagram of the task joint optimization device for vascular morphology and vascular blood flow characteristics according to an embodiment of the present invention.

[0071] Figure 8 The figure shows a schematic network structure diagram of the task joint optimization device for vascular morphology and vascular blood flow characteristics according to an embodiment of the present invention.

[0072] Figure 9 The figure shows a schematic structural diagram of an electronic device according to an embodiment of the present invention. Detailed implementation manners

[0073] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described below with reference to the accompanying drawings and specific implementation manners. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0074] A task joint optimization method for vascular morphology and vascular blood flow characteristics according to an embodiment of the present invention is as Figure 1 shown. In Figure 1 , this embodiment includes:

[0075] Step 100: In the encoding process, use a hybrid encoder that adapts to at least two task types to extract image features.

[0076] Those skilled in the art can understand that in neural network architectures such as CNN and Transformers, processing solutions using an encoder-decoder structure are included for task types such as classification segmentation, object detection, prediction regression, and semantic segmentation. For example, in the U-Net image segmentation model under the CNN architecture, one of the U-Net type networks such as Classic U-Net, Swin-UNETR, Attention U-Net, ResU-Net, UNet++ is used as the backbone network. The encoder downsamples to extract image features and expand the receptive field, and the decoder upsamples to restore the spatial resolution and perform feature fusion, thereby performing high-precision pixel-level image segmentation. In practical applications, the image segmentation model with the same task purpose can be replaced with, for example, the encoder-decoder combination method under the Transformers architecture. By using multiple encoders to downsample the input image, image features at different low resolutions can be obtained.

[0077] Among the two types of tasks, there is a task related to blood vessel morphology, such as the segmentation and detection tasks of blood vessel boundaries, blood vessel diameters, or intima related to blood vessel morphology, and there is also a task related to blood flow function, such as the function prediction tasks of flow rate, blood pressure, or CTFFR related to blood flow function. For different task requirements, during the downsampling process, targeted encoders are needed to extract graphic features. The hybrid encoder can establish rich types of annotation labels for image features at the same encoding level, which can alleviate the scarcity of feature annotation caused by the limitations of a single encoder.

[0078] Step 200: Configure the decoder to decode the image features according to the task type, forming at least one decoding branch of the blood vessel morphology task type and at least one decoding branch of the blood flow function task type.

[0079] Those skilled in the art can understand that the decoder is configured according to the decoding task requirements, and the decoder is used to splice, connect, and decode the encoded graphic features to form task decoding branches. By using different task combinations, the overall expression performance of the model is improved. The task types mainly include two types, one is related to blood vessel morphology and the other is related to blood function. The tasks related to blood vessel morphology include, but are not limited to, the segmentation and detection tasks of blood vessel boundaries, blood vessel diameters, or intima related to blood vessel morphology. The tasks related to blood flow function include, but are not limited to, the function prediction tasks of flow rate, blood pressure, or CTFFR related to blood flow function. There are relevant prior knowledge correlations between the two task types. During the decoding process, decoders corresponding to the relevant tasks of the task types are needed. The decoders are used to form the relevant tasks in one task type, and each task type includes at least one relevant task, and one relevant task forms a decoding branch. Specifically, there can be a many-to-many or one-to-many relationship between the number of relevant tasks of the two task types.

[0080] In each decoding branch, set the decoder and determine the number of decoders according to the task requirements. Generally, in each decoding branch, the type of decoder corresponds to the decoding task requirements, and the number of decoders corresponds to the number of encoders and is at the same level. In practical applications, different decoder architectures can be adopted, including but not limited to CNN-based decoders, RNN-based decoders, Transformers-based decoders, etc. For example, for the encoder-decoder backbone network formed by U-Net, the number of decoders in each decoding branch basically corresponds to the number of encoders in the encoding process. Through skip connections, the features of the encoder and decoder at the same level under different resolutions are spliced and connected, and upsampled step by step until the original resolution layer of the input image, and then the task result is output.

[0081] Step 300: In a pair of decoding branches of different task types, form a feature two-way correction process between the decoders of the two task types, where the output features of one decoder are used to correct the output features of the other decoder.

[0082] Those skilled in the art can understand that different decoding branches include decoders corresponding to task types. The decoders participating in the feature two-way correction are at the same level in different decoding branches, forming a pair of decoders that can correct each other's features through the feature two-way correction process. The feature two-way correction process acts bidirectionally on a pair of decoders at the same level, forming a correction to the features received from the other decoder based on the features received from one decoder, and changing the feature outputs of the two decoders through feature correction. In the feature two-way correction process, the feature input (related to blood vessel morphology or blood flow function) includes the features to be corrected, for example, it can include the feature outputs of the decoders at the same level (as the features to be corrected). The feature input also includes control features, such as the feature outputs of the decoders at the same level (the features to be corrected as control features), the feature outputs of the decoders at the previous level (as control features), the downsampled feature outputs of the final task output (as control features), etc., or a combination of the above several outputs. There is at least one feature two-way correction process between a pair of decoding branches for blood vessel morphology and blood function, and there can also be a feature two-way correction process during each decoding at the same level.

[0083] The output of the feature two-way correction process is the corrected features output after correcting the features of the decoders at the same level. The decoders at the same level use the corrected features to replace the original features, and then re-decode after splicing and skip connections to form the output features sent to the next level.

[0084] During the feature bidirectional correction process, control features can be used to form interactive judgments to control the correction process. The interactive judgment modes include, but are not limited to, interaction based on the attention mechanism, interaction based on the gating mechanism, interaction based on skip connections, or interaction based on adversarial training, etc. In an embodiment of the present invention, the bidirectional correction process can constitute an (interactive) decoder structure.

[0085] In an embodiment of the present invention, when there are more than one decoding branches for each task type, at least one feature bidirectional correction process can be formed between two decoding branches of different task types.

[0086] The task joint optimization method for vascular morphology and vascular blood flow characteristics according to the embodiment of the present invention utilizes other physical function states (hydrodynamics or hemodynamics) that have a prior correlation with the vascular morphology to form an analysis promotion for the vascular state. Through the construction of multi-task decoding branches, the association between vascular morphological reconstruction and the prediction of other functional state parameters is realized, increasing the interpretability and rationality of the prediction results. By constructing annotation data adapted to multi-tasks to jointly optimize the overall model, the problem of scarce annotation for the analysis of functional states other than vascular morphology is alleviated, ensuring the generalization ability of the model. Through the bidirectional correction process, the joint optimization between different state analyses is formed, improving the accuracy of each state analysis.

[0087] In an embodiment of the present invention, in the task joint optimization method for vascular morphology and vascular blood flow characteristics, a U-Net backbone network is used for multi-task optimization as Figure 2 shown. According to the task joint optimization method for vascular morphology and vascular blood flow characteristics of an embodiment of the present invention, the optimized model structure formed by using a U-Net backbone network is as Figure 3 shown.

[0088] Combined with Figure 2 and Figure 3 shown, in this embodiment, based on the optimization method of the above embodiment, the Swin-UNETR network is used as the backbone network in the encoding process of the vascular segmentation model. In step 100, it includes:

[0089] Step 110: Obtain an image containing blood vessels for preprocessing.

[0090] Medical images containing blood vessels include, but are not limited to, medical images such as CTA, X-ray angiography, magnetic resonance angiography (MRA - Magnetic Resonance Angiography), and vascular ultrasound.

[0091] Necessary preprocessing is performed on the image containing blood vessels, such as normalization, image window division, data augmentation, and label processing, etc.

[0092] In an embodiment of the present invention, the image containing blood vessels divided by the window is directionally position-encoded using the point coordinate information of the blood vessel centerline. As Figure 3 shown, the image containing blood vessels is a processed 3D CTA image with an original resolution of 512x512x256.

[0093] In an embodiment of the present invention, the information content carried by the blood vessel centerline can also be used to form feature extraction for the local details of the CTA image. The information guided by the blood vessel centerline is used to guide the construction of high-precision annotation data suitable for multi-tasks to improve the efficiency and accuracy of encoding and decoding, form collaborative optimization for the overall model, alleviate the problem of scarce annotation for the analysis of functional states other than blood vessel morphology, and ensure the generalization ability of the model.

[0094] Step 120: During the encoding process, for the classification task and the regression task, a hybrid encoder is used to hierarchically encode the image containing blood vessels to obtain image features at several resolutions.

[0095] In an embodiment of the present invention, using the swin-transformer encoder as the hybrid encoder can simultaneously adopt a hierarchical structure and a window-based self-attention mechanism to obtain associated spatial feature representations and semantic information. It can capture the features of the image at different scales, gradually convert the input medical image into a low-resolution spatial feature representation, and at the same time retain important semantic information. In an embodiment of the present invention, the CTA image undergoes 4-stage swin-transformer downsampling to obtain four levels of feature maps, namely, the 1 / 2 resolution feature map (128x128x64x48), the 1 / 4 resolution feature map (64x64x32x96), the 1 / 8 resolution feature map (32x32x16x192), the 1 / 16 resolution feature map (16x16x8x384), and the 1 / 32 feature map (8x8x4x768) at the bottleneck.

[0096] The task joint optimization method for blood vessel morphology and blood flow characteristics in the embodiment of the present invention uses a hybrid encoder to collect image features, performs data annotation based on the image features, and optimizes the annotation data of the overall model. The attention mechanism and windowed convolution operation of the hybrid encoder can better process large-size images and capture global context information. While ensuring the extraction of spatial features of blood vessel morphology, the hybrid encoder simultaneously extracts semantic features related to blood functions, alleviates the problem of scarce annotation for blood flow states other than blood vessel morphology, and ensures the generalization ability of the model.

[0097] Combined with Figure 2and Figure 3 As shown in Figure 3 , in this embodiment, based on the optimization method of the above embodiment, step 200 includes:

[0098] Step 210: Set a segmentation decoding branch reflecting the vascular morphology task, and set a segmentation decoder corresponding to the same-level encoder in the segmentation decoding branch.

[0099] The morphological parameters corresponding to the vascular morphology task include but are not limited to contour, inner diameter, inner wall mask, or topological shape, etc.

[0100] The segmentation decoder of each level in the segmentation decoding branch, also called the decoder unit (S-Decoder-Block), performs a skip connection on the decoded output of the previous-level segmentation decoder and the same-level encoding result, then decodes and sends it to the next level, and upsamples level by level until reaching the original resolution layer of the input image.

[0101] Specifically, the segmentation decoding branch performs deconvolution and normalized upsampling through the segmentation decoder, combines the skip connection decoding of the output features of the same-resolution encoder in the same level, and finally connects a 1x1x1 convolution after the features at the original resolution layer, and uses the sigmoid activation function to generate a segmentation probability map.

[0102] Step 220: Set a prediction decoding branch reflecting the blood flow functional task, and set a prediction decoder corresponding to the same-level encoder in the prediction decoding branch.

[0103] The functional parameters corresponding to the blood flow functional task include but are not limited to blood pressure parameters, blood flow parameters, CTFFR, etc.

[0104] The prediction decoder of each level in the prediction decoding branch, also called the decoder unit (F-Decoder-Block), splices and performs a skip connection on the previous-level decoded output and the same-level encoding result, then decodes and sends it to the next level, and upsamples level by level until reaching the original resolution layer of the input image.

[0105] Specifically, the prediction upsampling branch performs deconvolution and normalized upsampling through the prediction decoder, combines the skip connection decoding of the output features of the same-resolution encoder in the same level, and finally connects a 1x1x1 convolution or uses a fully connected layer after the features at the original resolution layer to implement parameter prediction.

[0106] In an embodiment of the present invention, the decoding branch includes at least two types of tasks, one corresponding to the segmentation task of blood vessel morphology, and the other corresponding to the prediction task of blood flow function. The two tasks may include one segmentation decoding branch and several prediction decoding branches, several segmentation decoding branches and one prediction decoding branch, or several segmentation decoding branches and several prediction decoding branches. According to the task requirements, a training mode of multi-task joint optimization of blood vessel morphology and blood flow function can be formed.

[0107] The task joint optimization method for blood vessel morphology and blood flow characteristics in the embodiment of the present invention realizes the association between blood vessel morphology reconstruction and the prediction of other functional state parameters, forms information complementarity between blood vessel morphology and blood flow function, and increases the interpretability and rationality of the prediction results. An association task can be established between any blood vessel morphology task and any blood flow function task according to the association established based on prior knowledge. For example, the cooperation scenario between the coronary artery blood vessel morphology task and the CTFFR function task, etc.

[0108] Combined with Figure 2 and Figure 3 As shown, in an embodiment of the present invention, based on the optimization method of the above embodiment, step 300 includes:

[0109] Step 310: Establish a forward correction process for correcting functional features according to morphological features between two decoders of the task types corresponding to the decoding branch.

[0110] One of the two decoding branches is a prediction decoding branch, and the other is a segmentation decoding branch. In the forward correction process, receive the morphological features from the segmentation decoder (i.e., to be corrected) and the functional features from the prediction decoder (i.e., to be corrected), generate spatial attention weights according to the segmentation confidence map of the morphological features, dynamically modulate the functional feature map, and obtain the corrected functional features. Improve the prediction accuracy of the functional features. Output the corrected functional features and feedback them back to the prediction decoder to replace the original functional features, and then decode them after jumping connection with the output of the same-level encoder in the prediction decoder, and then perform the subsequent upsampling and hierarchical decoding process.

[0111] Step 320: At the same time, establish a reverse correction process for correcting morphological features according to functional features between the two decoders.

[0112] During the reverse correction process, for the received (i.e., to be corrected) morphological features and (i.e., to be corrected) functional features, the morphological features are corrected according to the change gradient direction of the functional features to obtain the corrected morphological features. The segmentation boundary accuracy of the morphological features is improved. The corrected morphological features are output and fed back into the segmentation decoder to replace the original morphological features. After jumping connection with the output of the encoder at the same level, they are decoded and output in the segmentation decoder, and then the subsequent upsampling and decoding processes are carried out step by step.

[0113] The task joint optimization method for vascular morphology and vascular blood flow characteristics in the embodiments of the present invention uses the physical function state that has a prior correlation with the vascular morphology to form mutual promotion and collaborative optimization between vascular morphology analysis and blood flow function state analysis. It overcomes the technical problems of low model generalization ability caused by the lack of labeled type data and the inability of tasks to accurately express the consistency of physical features due to the lack of collaborative optimization.

[0114] In the task joint optimization method for vascular morphology and vascular blood flow characteristics in an embodiment of the present invention, a gated feature bidirectional correction module in an optimization model formed by a U-Net backbone network is as Figure 4 shown. Combining Figure 2 and Figure 4 shown, in an embodiment of the present invention, step 300 includes:

[0115] Step 330: Set the measurement threshold of the average confidence of the morphological features in the Gate Decision, and determine whether to activate the Correction Layer according to the measurement result.

[0116] The Gate Decision is used to judge the transfer direction of the to-be-corrected morphological features (S_feat) and the to-be-corrected functional features (F_feat) (usually at the upper level) according to the confidence of the morphological features output by the segmentation decoder (usually at the upper level). The measurement threshold of the average confidence in the Gate Decision is used as the activation signal for switching the Correction Layer.

[0117] Combining Figure 2 and Figure 4 shown, in an embodiment of the present invention, on the basis of the above embodiment, judging to activate the Correction Layer includes:

[0118] Step 340: Obtain the to-be-corrected morphological features (S_feat), the to-be-corrected functional features (F_feat), the control segmentation features (S_control) and the control functional features (F_control), and form a confidence map and an average confidence according to the control functional features.

[0119] AsFigure 4 As shown, in an embodiment of the present invention, a 1x1x1 convolution is performed on the control function feature (F_control) (formed by the segmentation feature) through the confidence task head (Conf Head), combined with the Sigmoid activation function operation, to obtain the confidence map and average confidence (conf) of the morphological features of the previous level.

[0120] Step 350: When the average confidence is greater than the measurement threshold, directly output the morphological feature to be corrected (S_feat) and the functional feature to be corrected (F_feat) from the segmentation decoder and the prediction decoder respectively.

[0121] When the average confidence is greater than the measurement threshold, the gating judgment process does not activate the feature bidirectional correction process, and directly outputs the morphological feature and the functional feature from the morphological decoder and the functional decoder.

[0122] Step 360: When the average confidence is less than the measurement threshold, activate the feature bidirectional correction process.

[0123] As Figure 4 shown, in an embodiment of the present invention, the feature bidirectional correction process includes:

[0124] Output the confidence map (Conf Map) function correction data for determining channels by performing 3x3x3 convolution on the control function feature (F_control) (formed by the segmentation feature);

[0125] Output the functional gradient (FFR Grad) of the same channel to form morphological correction data by performing 3x3x3 convolution on the control segmentation feature (S_control) (formed by the functional feature);

[0126] Correct the morphological feature to be corrected (S_feat) according to the morphological correction data to form the corrected morphological feature (S_feat_cor);

[0127] Correct the functional feature to be corrected (F_feat) according to the functional correction data to form the corrected functional feature (F_feat_cor).

[0128] In subsequent steps, in the original decoder, the corrected features are concatenated with the outputs of the encoder of the same level, and after skip connection, they are decoded and output in the decoder. In the corresponding upsampling branch, upsampling is performed through convolution + normalization until the original resolution layer, and the final result is output by connecting different task heads.

[0129] In one embodiment of the present invention, the input of the feature bidirectional correction process is the decoded feature of the upper layer, and the output is the feature that needs to be read into and concatenated with the skip connection in this layer of the network. At the input end, the morphological feature of the upper layer or the previous layer is used as the control function feature to calculate the segmentation confidence, and the function feature of the upper layer is used as the control morphological feature to calculate the function gradient, so as to realize the correction of the input segmentation feature (morphological feature to be corrected) and the function feature (functional feature to be corrected). In actual use, the feature to be corrected in the feature bidirectional correction process comes from the upper layer decoder, and the control feature is not limited to using the output feature of the upper layer decoder. It can be the output feature of a certain layer decoder in the previous layer, the downsampled feature of the output result of the final prediction head, or a combination of several methods, etc. The output of the feature bidirectional correction process can be the corrected morphological feature (S_feat_cor) and the corrected functional feature (F_feat_cor), or the result after decoding after concatenating the skip link features, etc.

[0130] In one embodiment of the present invention, the inputs of the two control features in the feature bidirectional correction process can also come from different levels, or a combination of different methods.

[0131] In one embodiment of the present invention, the feature to be corrected in the feature bidirectional correction process comes from the upper layer decoder, and the control feature of the feature bidirectional correction process can be the same as the feature to be corrected.

[0132] The task joint optimization method for vascular morphology and vascular blood flow characteristics in the embodiments of the present invention can form a mutual promotion and collaborative optimization between the accuracy of vascular morphology analysis and the accuracy of blood flow functional analysis, and at the same time, can form a gating mechanism to flexibly configure the correction level and correction timing of the feature bidirectional correction process among relevant upsampling branches. When the segmentation confidence is low, the feature bidirectional correction process is activated, otherwise the original feature is used for fast calculation. This can balance the computational efficiency and computational accuracy of the model.

[0133] In one embodiment of the present invention, the feature bidirectional correction process can be set between the decoders at the same level of the task type corresponding to the decoding branch, and the activation timing of the feature bidirectional correction process is adjusted through the gating judgment process.

[0134] As Figure 3 shown, in one embodiment of the present invention, for the optimized model of vascular segmentation formed according to the above optimization method, with the U-Net network as the backbone network, it includes an encoding branch and two decoding branches. In the encoding branch, it includes:

[0135] An input layer Input, used to form the normalization and data augmentation of the input image;

[0136] A window partitioning layer Patch-Partition, used to perform window partitioning and position encoding on the input image;

[0137] The explicit embedding layer Linear-Embed is used to perform a linear transformation on the divided image patches to form a feature map of a determined feature representation and / or feature vector;

[0138] The encoder module Blocks is used to gradually reduce the size of the feature map, increase the degree of feature abstraction, and extract feature information at different scales; in this embodiment, the encoder module uses a swin-transformer encoder;

[0139] The merging module Merging is used to increase the number of channels while reducing the size of the feature map during the downsampling operation to extract higher-level semantic information;

[0140] The bottleneck feature Bottleneck-Features is used to retain the feature map output by the last encoder module in the encoding process.

[0141] In one decoding branch, it includes:

[0142] The segmentation task head Segmentation-Head is used to restore the spatial resolution of the feature map and fuse the feature information at different levels in the encoder to form the final segmentation result of the blood vessel morphology;

[0143] The segmentation decoder S-Decoder-Block is used to gradually restore the size of the feature map through upsampling, feature splicing, and convolution processing.

[0144] In another decoding branch, it includes:

[0145] The prediction task head CTFFR-Head is used to restore the semantic information of the feature map to form the final prediction result of the CTFFR value;

[0146] The prediction decoder F-Decoder-Block is used to convert the features extracted by the encoder into the predicted value of CTFFR.

[0147] Between the two decoding branches, it includes:

[0148] Two feature bidirectional correction decoders Corr-Block are used to be set between the two decoders at the same level, acting on the two decoders bidirectionally, forming a correction to the features received from the other decoder according to the features received from one decoder, and changing the feature output of the two decoders through feature correction;

[0149] Between the segmentation task head Segmentation-Head and the prediction task head CTFFR-Head, it includes:

[0150] The loss calculation module Loss-Block is used to calculate the target loss of the prediction results of the segmentation task head and the prediction task head according to the segmentation ground truth S_gt and the CTFFR ground truth F_gt.

[0151] As Figure 4 shown, in an embodiment of the present invention, in the optimized model of blood vessel segmentation formed according to the above optimization method, the gated bidirectional correction module of the seat feature bidirectional correction decoder includes:

[0152] The confidence head Conf-head is used to calculate the average confidence using convolution conv and the sigmoid function;

[0153] The gating module Gate-Decision is used to judge the average confidence conf according to the measurement threshold threshold, and form an interactive control for activating the bidirectional correction process;

[0154] The correction layer Correction-layer is used to form the bidirectional correction process.

[0155] As Figure 4 shown, in an embodiment of the present invention, the correction layer Correction-layer includes:

[0156] The CTFFR gradient calculation module FFR-Grad is used to calculate the CTFFR gradient direction;

[0157] The segmentation feature correction module Seg-correction is used to correct the segmentation features through convolution conv processing according to the CTFFR gradient direction;

[0158] The confidence map calculation module Conf-Map is used to calculate the confidence map;

[0159] The prediction feature correction module FFR-Correction is used to correct the FFR prediction value through additive-fusion according to the confidence weight.

[0160] In an embodiment of the present invention, the feature acquisition path in the feature bidirectional correction process of the deep learning method for task joint optimization based on blood vessel centerline guidance is as Figure 5 shown. In Figure 5In it, the input features of the feature bidirectional correction process are diverse. At the input end, the segmentation confidence (which can be understood as the control function feature) is calculated using the segmentation features of the previous layer, and the CTFFR gradient is calculated using the CTFFR features of the previous layer (which can be understood as the control segmentation features) to achieve the correction of the input segmentation features (features to be corrected) and the CTFFR features (features to be corrected). In actual use, the features to be corrected in the feature bidirectional correction process come from the previous layer decoder. The control features are not limited to using the decoding features of the previous layer, and can be the features of a certain previous layer, the downsampled or interpolated features of the output result of the final prediction head, or a combination of several methods, etc. The output of the bidirectional correction module can be the output of the corrected features, or the result after splicing the skip connection features, etc. Moreover, the inputs of the two control features in the bidirectional correction process can also come from different layers, or a combination of different methods.

[0161] In Figure 5 it, the dotted part is the function control features that can be selected, including:

[0162] a) Skip-layer, the decoding features obtained from the previous previous layer, obtaining features from a larger receptive field and lower resolution, suitable for global correction from high-level semantic features (such as the overall shape of blood vessels), with small computational cost, known for the current calculated feature results, and does not affect the network calculation order;

[0163] b) Pre-layer, the decoding features obtained from the previous layer, with the same feature resolution, no need for calculations such as interpolation, upsampling, and downsampling, high computational efficiency, simple model, known for the current calculated feature results, and does not affect the network calculation order;

[0164] c) Down-sample, the confidence features obtained by downsampling from the output result, the features are closer to the output result, can provide low-level details, strengthen the correction effect, without information loss, but the prediction results of the current layer are unknown. Therefore, the two correction modules can only correct alternately, the model needs to be trained step by step, and the model design is more complex.

[0165] When the pre-layer connection method is selected and the control feature comes from the previous layer, the feature to be corrected is used as the control feature at the same time (i.e., S_control = S_feat).

[0166] Combined with Figure 4 and Figure 5 as shown, the gating mechanism formed by the gating module is also an optional module, which can balance the computational efficiency and its corresponding model effect. When the computational efficiency is not considered, the gating module can be discarded, so as to achieve the correction of all the features input to the correction module.

[0167] In an embodiment of the present invention, in the task joint optimization method for blood vessel morphology and blood flow characteristics, the loss function in the optimization model formed by a U-Net backbone network is as Figure 6 shown. Combining Figure 1 and Figure 6 shown, this embodiment further includes:

[0168] Step 400: Using the blood vessel morphology parameters and blood flow function parameters obtained from two task types as the independent variables of the physical constraint conditions, determine the physical constraint loss function as a component of the total loss function according to the residual between the predicted value and the true value of the dependent variable of the physical constraint conditions, and optimize the decoder and the feature bidirectional correction process through the backpropagation of the total loss function gradient.

[0169] Those skilled in the art can understand that the segmentation result is formed through the segmentation task head, and the prediction result is formed through the prediction task head. The segmentation result and the prediction result are for the corresponding blood vessel morphology parameters and blood flow function parameters. By determining specific physical constraint conditions (physical equations) with blood vessel morphology parameters and blood flow function parameters as independent variables, the predicted value of the dependent variable parameter can be obtained based on the predicted values formed by the blood vessel morphology parameters and blood flow function parameters. The physical constraint loss function is determined according to the residual between the predicted value of the dependent variable parameter and the true value of the dependent variable parameter obtained through quantitative calculation. Taking the physical constraint loss function as a part of the total loss function of the model, the network is iteratively backpropagated with parameters using the total loss function to optimize the model accuracy. Specific physical constraint conditions include, but are not limited to, the control equations of Poiseuille flow in fluid mechanics, the continuity equation of incompressible fluids, the momentum equation, etc. The specific physical constraint conditions can be global or local. For example: the blood flow in all blood vessels needs to satisfy the Poiseuille flow related to the segmented blood vessel diameter; the blood vessels at the bifurcation should satisfy the continuity equation, that is, the total blood flow into each branch is equal to the blood flow in the parent blood vessel, etc. The physical constraint methods and formulas used are optional, and other similar physical equations that can achieve the constraint between geometric information and functional parameter information can be used as appropriate.

[0170] The optimization effect of the task joint optimization method for blood vessel morphology and blood flow characteristics in the embodiment of the present invention on the model is achieved by adding the loss of composite physical conditions. Using the parametric equation that conforms to the physical laws of fluid mechanics to constrain the functional parameters predicted by the model, calculating the residual after substituting the predicted value into the equation, making the prediction result more in line with the physical laws, and improving the rationality of blood flow function prediction. The physical constraint indirectly acts on the morphological channels such as blood vessel segmentation through the bidirectional interactive correction structure designed in the model, and synchronously improves the accuracy of the segmentation result.

[0171] Combining Figure 1 and Figure 6As shown in the figure, in an embodiment of the present invention, for the segmentation task and CTFFR prediction task of the model, step 400 includes:

[0172] Step 410: Determine the segmentation loss function according to the residual between the prediction result and the ground truth of the segmentation task.

[0173] Specifically, determine the segmentation loss Seg_loss according to the residual between the segmentation prediction result S_pred and the segmentation ground truth S_gt of the segmentation task. The segmentation loss function L_s adopts the Dice (Dice Coefficient) loss function combined with the BCE (Binary Cross Entropy) loss function.

[0174] Step 420: Determine the prediction loss function according to the residual between the prediction result and the prediction ground truth of the prediction task.

[0175] Specifically, determine the CTFFR loss FFRLoss according to the residual between the prediction result F_pred and the CTFFR prediction ground truth F_gt. The CTFFR loss function L_F adopts the MSE (Mean Squared Error) loss function.

[0176] Step 430: Select one segmentation task and one CTFFR prediction task; obtain the blood vessel radius parameter from the result of the segmentation task and the pressure difference parameter from the result of the CTFFR prediction task. The blood vessel radius parameter and the pressure difference parameter are used to obtain the blood flow prediction value through the Poiseuille flow equation. The blood vessel radius parameter and the pressure difference parameter form the blood vessel blood flow ground truth through the fluid continuity equation. According to the residual between the blood flow prediction value and the blood vessel blood flow ground truth, determine the physical constraint loss function.

[0177] Specifically, calculate the predicted value Ves-R of the blood vessel radius from the segmentation map of the segmentation task, calculate the pressure difference FFR between the distal blood vessel pressure and the coronary ostium pressure according to the pressure prediction value of the CTFFR prediction task, and obtain the blood flow prediction value Q by substituting the pressure difference formed by the predicted values and the blood vessel radius into the Poiseuille flow formula. At the same time, according to the fluid continuity equation (|Q _in -Q _out |) for the physical law constraint of the blood inflow volume Q _in and the blood outflow volume Q _out to judge the residual between the blood flow prediction value and the true value, establish the physical constraint phisics-constraints, determine the physical constraint loss, and the physical constraint loss function L_p adopts the MSE (Mean Squared Error) loss function.

[0178] Step 440: Form a total loss based on the segmentation loss, prediction loss, and physical constraint loss, and use the total loss function to optimize the network parameters of the decoder and the feature bidirectional correction process through gradient backpropagation.

[0179] Total loss function = segmentation loss function L_s + CTFFR loss function L_F + physical constraint loss function L_p

[0180] Iteratively optimize the network parameters of the decoder and the feature bidirectional correction process through the total loss function until the accuracy requirements of the model are met.

[0181] An apparatus for jointly optimizing tasks related to vascular morphology and vascular blood flow characteristics according to an embodiment of the present invention is as Figure 7 shown. In Figure 7 , this embodiment includes:

[0182] An encoder 10, configured to extract image features using a hybrid encoder adapted to at least two task types during the encoding process;

[0183] A decoder 20, configured to decode the image features according to the task type, and form at least one decoding branch for the vascular morphology task type and at least one decoding branch for the blood flow function task type;

[0184] A bidirectional correction decoder 30, configured to form a feature bidirectional correction process for correcting the output features of one decoder using the output features of another decoder between the decoders of two task types in a pair of decoding branches of different task types.

[0185] As Figure 7 shown, in an embodiment of the present invention, the downsampling encoding setting module 10 includes:

[0186] An image receiving unit 11, configured to obtain an image containing blood vessels for preprocessing;

[0187] A feature extraction unit 12, configured to perform hierarchical encoding on the image containing blood vessels using a hybrid encoder for the classification task and the regression task during the encoding process to obtain image features of several resolutions.

[0188] As Figure 7 shown, in an embodiment of the present invention, the upsampling task setting module 20 includes:

[0189] A segmentation task setting unit 21, configured to set a segmentation decoding branch reflecting the vascular morphology task, and set a segmentation decoder corresponding to the same-level encoder in the segmentation decoding branch;

[0190] A prediction task setting unit 22 is configured to set a prediction decoding branch reflecting a blood flow functional task, and a prediction decoder corresponding to the same-level encoder is set in the prediction decoding branch.

[0191] As Figure 7 shown, in an embodiment of the present invention, the feature bidirectional correction module 30 includes:

[0192] A forward correction unit 31 is configured to establish a forward correction process for correcting functional features according to morphological features between two decoders of the task type corresponding to the decoding branch.

[0193] A reverse correction unit 32 is configured to simultaneously establish a reverse correction process for correcting morphological features according to functional features between the two decoders.

[0194] As Figure 7 shown, in an embodiment of the present invention, the feature bidirectional correction module 30 further includes:

[0195] A gating switch unit 33 is configured to set a measurement threshold for the average confidence of morphological features in the gating judgment process, and determine whether to activate the feature bidirectional correction process according to the measurement result.

[0196] As Figure 7 shown, in an embodiment of the present invention, the feature bidirectional correction module 30 further includes:

[0197] A confidence acquisition unit 34 is configured to acquire morphological features to be corrected, functional features to be corrected, control segmentation features and control functional features, and form a confidence map and an average confidence according to the control functional features;

[0198] A feature direct connection unit 35 is configured to directly output the morphological features to be corrected and the functional features to be corrected from the segmentation decoder and the prediction decoder respectively when the average confidence is greater than the measurement threshold;

[0199] A bidirectional correction execution unit 36 is configured to activate the feature bidirectional correction process when the average confidence is less than the measurement threshold.

[0200] As Figure 7 shown, in an embodiment of the present invention, it further includes:

[0201] A physical constraint formation module 40 is configured to use the vascular morphological parameters and blood flow functional parameters obtained by two task types as independent variables of physical constraint conditions, determine a physical constraint loss function as a component of the total loss function according to the residual between the predicted value and the true value of the dependent variable of the physical constraint conditions, and optimize the decoder and the feature bidirectional correction process through the backpropagation of the total loss function gradient.

[0202] As Figure 7As shown, in an embodiment of the present invention, the physical constraint formation module 40 includes:

[0203] A segmentation loss formation unit 41, configured to determine a segmentation loss function according to the residual between the prediction result and the segmentation ground truth of the segmentation task;

[0204] A prediction loss formation unit 42, configured to determine a prediction loss function according to the residual between the prediction result and the prediction ground truth of the prediction task;

[0205] A physical constraint loss formation unit 43, for a segmentation task and a CTFFR prediction task; obtaining a blood vessel radius parameter by using the segmentation task result, obtaining a pressure difference parameter from the CTFFR prediction task result, obtaining a blood flow prediction value through the Poiseuille flow equation with the blood vessel radius parameter and the pressure difference parameter, forming a blood vessel blood flow ground truth through the fluid continuity equation with the blood vessel radius parameter and the pressure difference parameter, and determining a physical constraint loss function according to the residual between the blood flow prediction value and the blood vessel blood flow ground truth;

[0206] An iterative optimization unit 44, configured to form a total loss according to the segmentation loss, the prediction loss, and the physical constraint loss, and optimize the network parameters of the decoder and the feature two-way correction process by using the total loss function through gradient backpropagation.

[0207] The network structure of the task joint optimization device for blood vessel morphology and blood vessel blood flow characteristics in an embodiment of the present invention is as Figure 8 shown. In Figure 8 this embodiment includes:

[0208] An encoder, configured to extract image features by using a hybrid encoder adapted to at least two task types during the encoding process;

[0209] A decoder, configured to decode the image features according to the task type configuration of the decoder to form at least one decoding branch of the blood vessel morphology task type and at least one decoding branch of the blood flow functional task type;

[0210] A two-way correction decoder, configured to form a feature two-way correction process of using the output features of one decoder to correct the output features of another decoder between two decoders in a pair of decoding branches of different task types.

[0211] In an embodiment of the present invention, the decoder includes a morphological decoder and a functional decoder, and decodes the fused image features according to the task type configuration of the decoder to form at least one decoding branch of the blood vessel morphology task and at least one decoding branch of the blood flow functional task.

[0212] The task joint optimization device for vascular morphology and vascular blood flow characteristics in the embodiments of the present invention forms a network structure based on an encoder-decoder. By performing a two-way correction on the decoder features at the same level through a two-way correction decoder among the decoding branch structures formed by the decoder, a joint analysis method for vascular morphology is established using the blood flow functional state that has a prior correlation with vascular morphology, forming a joint promotion for the analysis of vascular morphology and function. This ensures the physical consistency of anatomical reconstruction and functional prediction.

[0213] An embodiment of the present application also provides a specific implementation manner of an electronic device that can implement all the steps in the method in the above embodiments. Refer to Figure 9 , the electronic device 600 specifically includes the following:

[0214] A processor 610, a memory 620, a communication unit 630, and a bus 640;

[0215] Among them, the processor 610, the memory 620, and the communication unit 630 complete mutual communication through the bus 640; the communication unit 630 is used to implement information transmission between related devices such as server-side devices and terminal devices.

[0216] The processor 610 is used to call the computer program in the memory 620, and when the processor executes the computer program, it implements all the steps in the task joint optimization method for vascular morphology and vascular blood flow characteristics in the above embodiments.

[0217] Those of ordinary skill in the art should understand that the memory can be, but is not limited to, a random access memory (Random Access Memory, abbreviated as RAM), a read-only memory (Read Only Memory, abbreviated as ROM), a programmable read-only memory (Programmable Read-Only Memory, abbreviated as PROM), an erasable programmable read-only memory (Erasable Programmable Read-Only Memory, abbreviated as EPROM), an electrically erasable programmable read-only memory (Electric Erasable Programmable Read-Only Memory, abbreviated as EEPROM), etc. Among them, the memory is used to store the program, and the processor executes the program after receiving the execution instruction. Further, the software programs and modules in the above memory may also include an operating system, which may include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide a running environment for other software components.

[0218] The processor can be an integrated circuit chip with the ability to process signals. The aforementioned processor can be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc. It can implement or execute the various methods, steps, and logical block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0219] The present application also provides a computer-readable storage medium. The computer-readable storage medium includes a program that, when executed by a processor, is used to execute the task joint optimization method for vascular morphology and vascular blood flow characteristics provided in any of the foregoing method embodiments.

[0220] Those of ordinary skill in the art should understand that all or part of the steps to implement the foregoing method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the foregoing method embodiments; and the foregoing storage medium includes: various media that can store program codes such as ROM, RAM, magnetic disks, or optical discs. The specific type of the medium is not limited in the present application.

[0221] As described above, the above are only the preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A task joint optimization method for vascular morphology and vascular blood flow characteristics, characterized in that Including: During the encoding process, a hybrid encoder that can adapt to at least two task types is used to extract image features; According to the task type, a decoder is configured to decode the image features, forming at least one decoding branch for the vascular morphology task type and at least one decoding branch for the blood flow functional task type; In a pair of decoding branches of different task types, a feature two-way correction process is formed between the decoders of the two task types, where the output features of one decoder are used to correct the output features of the other decoder.

2. The optimization method according to claim 1, characterized in that, The use of a hybrid encoder that can adapt to at least two task types to extract image features includes: Obtaining an image containing blood vessels for preprocessing; During the encoding process, for the classification task and the regression task, the hybrid encoder is used to hierarchically encode the image containing blood vessels to obtain image features at several resolutions.

3. The optimization method according to claim 2, wherein The obtaining of an image containing blood vessels for preprocessing includes: Using the point coordinate information of the blood vessel centerline to perform direction and position encoding on the image containing blood vessels divided by the window, forming feature extraction for the local details of the image containing blood vessels.

4. The optimization method according to claim 1, characterized in that The configuring of the decoder according to the task type to decode the fused image features, forming at least one decoding branch for the vascular morphology task and at least one decoding branch for the blood flow functional task includes: Setting a segmentation decoding branch reflecting the vascular morphology task, and setting a segmentation decoder corresponding to the same-level encoder in the segmentation decoding branch; the vascular morphology task includes at least one of contour, inner diameter, inner wall mask, or topological shape; Setting a prediction decoding branch reflecting the blood flow functional task, and setting a prediction decoder corresponding to the same-level encoder in the prediction decoding branch; the blood flow functional task includes at least one of blood pressure parameters, blood flow parameters, CTFFR.

5. The optimization method according to claim 1, wherein The input features of the feature two-way correction process include the feature output of the decoder at the same level as the feature to be corrected, as well as the feature output of the decoder at the same level as the control feature, the feature output of the previous-level decoder, or the downsampled feature output of the task final output result. The interaction judgment is formed using the control feature to control the feature two-way correction process.

6. The optimization method according to claim 1, wherein, The feature two-way correction process includes: Establishing a forward correction process for correcting the functional features based on the morphological features between the two decoders of the task types corresponding to the decoding branches; At the same time, establishing a reverse correction process for correcting the morphological features based on the functional features between the two decoders.

7. The optimization method according to claim 6, characterized in that, The forward correction process includes: Generating spatial attention weights according to the segmentation confidence map of the morphological features, dynamically modulating the functional feature map, and obtaining the corrected functional features.

8. The optimization method according to claim 6, wherein The reverse correction process includes: Correcting the morphological features according to the change gradient direction of the functional features to obtain the corrected morphological features.

9. The optimization method according to claim 1, wherein The feature two-way correction process includes: Setting a measurement threshold for the average confidence of the morphological features in the Gate Decision process, and determining whether to activate the Correction Layer according to the measurement result.

10. The optimization method according to claim 9, characterized in that, The process of setting the measurement threshold for the average confidence level of morphological features in the gating judgment, and determining whether to activate the feature bidirectional correction process according to the measurement result includes: Obtain the morphological feature to be corrected (S_feat), the functional feature to be corrected (F_feat), the control segmentation feature (S_control), and the control functional feature (F_control), and form a confidence map and an average confidence level according to the control functional feature; When the average confidence level is greater than the measurement threshold, directly output the morphological feature to be corrected (S_feat) and the functional feature to be corrected (F_feat) from the segmentation decoder and the prediction decoder respectively; When the average confidence level is less than the measurement threshold, activate the feature bidirectional correction process.

11. The optimization method according to claim 9, characterized in that, The feature bidirectional correction process includes: Output the confidence map (ConfMap) function correction data for determining channels by performing 3x3x3 convolution on the control functional feature (F_control); Output the functional gradient (FFRGrad) of the same channel by performing 3x3x3 convolution on the control segmentation feature (S_control) to form the morphological correction data; Correct the morphological feature to be corrected (S_feat) according to the morphological correction data to form the corrected morphological feature (S_feat_cor); Correct the functional feature to be corrected (F_feat) according to the function correction data to form the corrected functional feature (F_feat_cor).

12. The optimization method according to claim 1, wherein It also includes: Using the vascular morphological parameters and blood flow function parameters obtained from two types of tasks as the independent variables of the physical constraint conditions, determining the physical constraint loss function as a component of the total loss function according to the residual between the predicted value and the true value of the dependent variable of the physical constraint conditions, and optimizing the decoder and the feature bidirectional correction process through the gradient backpropagation of the total loss function.

13. The optimization method according to claim 11, wherein, The process of determining the physical constraint loss function as a component of the total loss function and optimizing the decoder and the feature bidirectional correction process through the gradient backpropagation of the total loss function includes: Determine the segmentation loss function according to the residual between the prediction result and the segmentation true value of the segmentation task; Determine the prediction loss function according to the residual between the prediction result and the prediction true value of the prediction task; Select a segmentation task and a CTFFR prediction task of the model, obtain the vascular radius parameter from the result of the segmentation task, obtain the pressure difference parameter from the result of the CTFFR prediction task, obtain the predicted blood flow value through the Poiseuille flow equation with the vascular radius parameter and the pressure difference parameter, form the true value of the vascular blood flow through the fluid continuity equation with the vascular radius parameter and the pressure difference parameter, and determine the physical constraint loss function according to the residual between the predicted blood flow value and the true value of the vascular blood flow; Form the total loss according to the segmentation loss, the prediction loss, and the physical constraint loss, and optimize the network parameters of the decoder and the feature bidirectional correction process through the gradient backpropagation of the total loss function.

14. A task joint optimization device for vascular morphology and vascular blood flow characteristics, characterized in that, It includes: An encoder, which is used to extract image features using a hybrid encoder that adapts to at least two types of task types during the encoding process of the vascular segmentation model; A decoder, configured to decode image features according to a task type to form at least one decoding branch of a vascular morphology task type and at least one decoding branch of a blood flow functional task type; A bidirectional correction decoder, configured to form a feature bidirectional correction process of using the output features of one decoder to correct the output features of another decoder between two decoders in a pair of decoding branches of different task types.

15. The optimization device according to claim 14, wherein It further includes: A physical constraint formation module, configured to use the vascular morphology parameters and blood flow function parameters obtained from two task types as independent variables of physical constraint conditions, determine a physical constraint loss function as a component of the total loss function according to the residual between the predicted value and the true value of the dependent variable of the physical constraint conditions, and optimize the decoder and the feature bidirectional correction process through the gradient backpropagation of the total loss function.

16. An electronic device, characterized in that, It includes: A processor, a memory, and an interface for communicating with a gateway; The memory is used to store programs and data, and the processor calls the programs stored in the memory to execute the method according to any one of claims 1 to 13.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a program, and the program is used to execute the method according to any one of claims 1 to 13 when executed by the processor.

Images