Blood vessel OCT image training method, generation method and device based on DCN cross-modal fusion network and structure-blood flow coupling modeling

Abstract

The invention discloses a blood vessel OCT image training method, a blood vessel OCT image generating method and a blood vessel OCT image training device based on a DCN cross-modal fusion network and structure-blood flow coupling modeling. The method comprises the following steps: acquiring image data marked with a blood vessel segmentation result and a plaque position; acquiring the preprocessed OCT image data; registering the image data marked with the blood vessel segmentation result and the plaque position and the OCT image data so as to obtain registered image data; acquiring hemodynamic parameters; inputting the coronary artery CTA image, the image marked with the blood vessel segmentation result and the plaque position and hemodynamic parameters into a cross-modal fusion network based on deformable convolution so as to obtain CTA features of the cross-modal fusion network; and inputting the CTA features of the cross-modal fusion network into a trained OCT generation network so as to obtain a cross-section OCT image containing blood vessel wall and plaque information. The method can replace part of invasive OCT examination, assist doctors in assessing plaque vulnerability and guide interventional therapy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a blood vessel OCT image training method based on a DCN cross-modal fusion network and structure-blood flow coupling modeling, a blood vessel OCT image generation method based on a DCN cross-modal fusion network and structure-blood flow coupling modeling, and a blood vessel OCT image training device based on a DCN cross-modal fusion network and structure-blood flow coupling modeling. BACKGROUND

[0002] In the diagnosis and analysis of coronary stenosis and vulnerable plaques, OCT imaging is the "gold standard" for evaluating the microscopic structure of coronary and cerebral vessels, but it has obvious limitations. OCT imaging requires invasive catheter operation, which has the risk of vascular injury, thrombosis and infection, and poor patient tolerance, especially for elderly or comorbid patients. Moreover, the cost of a single OCT detection is high, and it requires the cooperation of a special interventional team. Computed tomography angiography (CTA) is a recognized non-invasive imaging method for diagnosing coronary arteries, and is currently an important method for early screening and definitive diagnosis of coronary heart disease. It can be used to identify coronary plaques and assess the degree of vascular stenosis. Although CTA is non-invasive, it lacks spatial resolution and cannot clearly show key plaque characteristics such as fibrous cap thickness and microcalcification.

[0003] Therefore, it is desirable to have a technical solution to overcome or at least alleviate at least one of the aforementioned deficiencies of the prior art.

[0004] SUMMARY

[0005] The present application aims to provide a blood vessel OCT image training method based on a DCN cross-modal fusion network and structure-blood flow coupling modeling to overcome or at least alleviate at least one of the aforementioned deficiencies of the prior art.

[0006] To achieve the above-mentioned purpose, the present application provides a blood vessel OCT image training method based on a DCN cross-modal fusion network and structure-blood flow coupling modeling, which comprises:

[0007] Obtaining a plurality of sets of coronary CTA images and OCT images of patients, the OCT images being preliminarily aligned in space with the coronary CTA images;

[0008] Pretreating each of the CTA images to obtain image data marked with vessel segmentation results and plaque positions;

[0009] Pretreating each of the OCT images to obtain pretreated OCT image data;

[0010] register the image data marked with the blood vessel segmentation result and the plaque position and the preprocessed OCT image data of the same patient, so as to obtain each registered image data;

[0011] obtain the hemodynamic parameters of the same patient, obtain the deformable convolution-based cross-modal fusion network and the OCT generation network;

[0012] train the deformable convolution-based cross-modal fusion network and the OCT generation network through each registered image data and the hemodynamic parameters, so as to obtain the trained deformable convolution-based cross-modal fusion network and the trained OCT generation network.

[0013] Optionally, the obtaining of the coronary CTA image and the OCT image of each group of patients, and the preliminary alignment of the spatial positions of the OCT image and the coronary CTA image comprises:

[0014] collecting N groups of paired coronary CTA and OCT images, wherein the CTA data is high-resolution CT angiography, covering the target blood vessels, containing enhanced blood vessel cavities, plaques and vessel wall information, and the OCT data is intravascular OCT image of the same patient as the CTA, ensuring the preliminary alignment of the spatial positions.

[0015] Optionally, the pre-processing of each CTA image to obtain image data marked with the blood vessel segmentation result and the plaque position comprises:

[0016] obtaining a pre-trained segmentation model;

[0017] inputting each CTA image into the pre-trained segmentation model to obtain mask image data after segmenting blood vessels and plaques;

[0018] removing noise from the mask image data combined with morphological operations to retain the branch blood vessel topology;

[0019] taking the blood vessel centerline as the reference system, unifying the coordinate system origin and direction matrix of the mask image data, so as to obtain image data marked with the blood vessel segmentation result and the plaque position.

[0020] Optionally, the registration of the image data marked with the blood vessel segmentation result and the plaque position and the preprocessed OCT image data of the same patient to obtain registered image data comprises:

[0021] performing polar to Cartesian coordinate conversion on the OCT image based on the catheter marker point, and then performing motion artifact compensation and coordinate system transformation to correct the image, so as to obtain the corrected OCT image.

[0022] Optionally, registering the image data marked with the blood vessel segmentation result and the plaque position and the pre-processed OCT image data to obtain the registered image data includes:

[0023] The corrected OCT images were matched with the corresponding positions of CTA images by vascular branch points and calcified plaque locations, and the mutual information maximization algorithm was used to optimize the registration.

[0024] The present application also provides a method for generating vascular OCT images based on a DCN cross-modal fusion network and structure-blood flow coupling modeling. The method for generating vascular OCT images based on a DCN cross-modal fusion network and structure-blood flow coupling modeling includes:

[0025] Obtaining a deformable convolution-based cross-modal fusion network and a trained OCT generation network trained by the vascular OCT image training method based on the DCN cross-modal fusion network and structure-blood flow coupling modeling as described above;

[0026] Obtaining coronary artery CTA images and hemodynamic parameters of the patient to be generated;

[0027] Inputting the coronary artery CTA image of the patient to be generated and the hemodynamic parameters into the trained deformable convolution-based cross-modal fusion network, thereby obtaining CTA features of the cross-modal fusion network;

[0028] The CTA features of the cross-modal fusion network are input into the trained OCT generation network to obtain a cross-sectional OCT image containing vascular wall and plaque information.

[0029] Optionally, inputting the coronary artery CTA image of the patient to be generated and the hemodynamic parameters into the trained deformable convolution-based cross-modal fusion network to obtain CTA features of the cross-modal fusion network includes:

[0030] encoding the coronary artery CTA image of the patient to be generated into a CTA feature map through a 3D deformable convolutional network;

[0031] Preprocessing the coronary artery CTA image of the patient to be generated, thereby obtaining image data marked with a blood vessel segmentation result and a plaque location;

[0032] The image data marked with the blood vessel segmentation results and plaque locations is encoded into a heat map, and multiplied channel by channel with the CTA feature map to obtain the first fusion feature;

[0033] Extract dynamic characteristics based on hemodynamic parameters;

[0034] The dynamic characteristics are fused with the first fusion characteristics using a multi-head attention mechanism to obtain CTA characteristics of the cross-modal fusion network.

[0035] Optionally, the trained OCT generation network comprises a structural similarity loss, and the structural similarity loss function is as follows:

[0036] L smi = alpha * || y gen - y real ||1 + (1-alpha) * (1-SSIM(y gen , y real ))

[0037] Wherein, alpha is a weight coefficient, SSIM is a structural similarity index, L smi is a structural similarity loss function, y gen is a generated OCT image, and y real is a real OCT image.

[0038] The trained OCT generation network further comprises a structure-blood flow coupling loss, and the structure-blood flow coupling loss is as follows:

[0039]

[0040] Wherein, theta lipid is a lipid core arc in the OC image, M loss-wss is a low WSS mask, L align is a structure-blood flow loss term.

[0041] Optionally, the final loss function of the trained OCT generation network is as follows:

[0042] L all = L smi + gamma * L align

[0043] Wherein, L all is a final loss function, L smi is a structural similarity loss function, gamma = 0.5, and L align is a structure-blood flow coupling loss.

[0044] The present application also provides a blood vessel OCT image training device based on a DCN cross-modal fusion network and structure-blood flow coupling modeling, comprising:

[0045] An image acquisition module is configured to acquire a plurality of groups of coronary CTA images and OCT images based on the same patient, and the OCT images are preliminarily aligned with the coronary CTA images in spatial position;

[0046] A CTA image preprocessing module is configured to preprocess each of the CTA images, thereby acquiring image data marked with a blood vessel segmentation result and a plaque position;

[0047] An OCT image preprocessing module is configured to preprocess each of the OCT images, thereby acquiring preprocessed OCT image data;

[0048] A registration module is configured to register the image data marked with the blood vessel segmentation result and the plaque position and the preprocessed OCT image data of the same patient, thereby acquiring registered image data;

[0049] A hemodynamic parameter acquisition module is configured to acquire hemodynamic parameters of the same patient;

[0050] A model acquisition module is configured to acquire a deformable convolution-based cross-modal fusion network and an OCT generation network;

[0051] A training module is configured to train the deformable convolution-based cross-modal fusion network and the OCT generation network respectively by using each of the registered image data and the hemodynamic parameters, thereby acquiring a trained deformable convolution-based cross-modal fusion network and a trained OCT generation network.

[0052] The present application generates high-resolution OCT images by fusing the anatomical structure features of CTA imaging and the CFD (fluid mechanics) hemodynamic parameters of blood vessels, and evaluates plaque vulnerability. The method can replace part of the invasive OCT examination, and assist doctors in evaluating plaque vulnerability and guiding interventional treatment. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 FIG. 1 is a flowchart of a deformable convolution-based cross-modal fusion network and structure-blood coupling modeling-based vascular OCT image training method according to an embodiment of the present application.

[0054] Figure 2 FIG. 6 is a schematic diagram of deformable convolution according to the present application.

[0055] Figure 3 FIG. 7 is a schematic diagram of a deformable convolution-based cross-modal fusion network according to the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the implementation of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below in conjunction with the drawings in the embodiments of this application. In the drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described embodiments are part of the embodiments of this application, not all of the embodiments. The embodiments described below with reference to the drawings are exemplary and are intended to be used to explain this application, and should not be understood as limitations on this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The embodiments of this application are described in detail below in conjunction with the drawings.

[0057] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be understood as limiting the scope of protection of this application.

[0058] like Figure 1 The vascular OCT image training method based on the DCN cross-modal fusion network and structure-blood flow coupling modeling shown includes:

[0059] Step 1: Acquire coronary artery CTA images and OCT images of multiple groups of patients, and preliminarily align the spatial positions of the OCT images with the coronary artery CTA images;

[0060] Step 2: Preprocessing each of the CTA images to obtain image data marked with blood vessel segmentation results and plaque locations;

[0061] Step 3: Preprocessing each of the OCT images to obtain preprocessed OCT image data;

[0062] Step 4: Registering the image data marked with the blood vessel segmentation results and plaque location and the pre-processed OCT image data belonging to the same patient to obtain registered image data;

[0063] Step 5: Obtain hemodynamic parameters of the same patient;

[0064] Step 6: Obtain a cross-modal fusion network based on deformable convolution and an OCT generation network;

[0065] Step 7: training the deformable convolution-based cross-modality fusion network and the OCT generation network respectively by the registered image data and the blood flow dynamics parameters, so as to obtain the trained deformable convolution-based cross-modality fusion network and the trained OCT generation network.

[0066] In the embodiment, the multiple groups of coronary CTA images and OCT images of the same patient are obtained, and the OCT images are preliminarily aligned in space with the coronary CTA images, including:

[0067] N groups of paired coronary CTA and OCT images are collected, wherein the CTA data is high-resolution CT angiography, covering the target blood vessels, containing enhanced blood vessel cavities, plaques and vessel wall information, and the OCT data is intravascular OCT image of the same patient as the CTA, ensuring that the spatial position is preliminarily aligned with the CTA.

[0068] For example, there can be 100 groups of patient data, each group of patient data including at least one coronary CTA image (when there are two or more, the two coronary CTA images are different in time sequence) and at least one OCT image (when there are two or more, the two OCT images are different in time sequence), and the OCT image of each group of patient data is preliminarily aligned in space with the coronary CTA image. It can be understood that if there are two or more images in time sequence, the alignment can also be performed in time sequence.

[0069] In the embodiment, the multiple groups of coronary CTA images and OCT images of the same patient are obtained, and the OCT images are preliminarily aligned in space with the coronary CTA images, including:

[0070] A pre-trained segmentation model is obtained;

[0071] Each of the CTA images is input into the pre-trained segmentation model, so as to obtain mask image data after segmentation of blood vessels and plaques;

[0072] The mask image data is combined with a morphological operation to remove noise and retain the branch blood vessel topology;

[0073] The mask image data is taken as a reference system with a blood vessel centerline, and the coordinate system origin and direction matrix are unified, so as to obtain image data marked with blood vessel segmentation results and plaque positions.

[0074] In the embodiment, the OCT image data is preprocessed, and the preprocessed OCT image data is obtained, including:

[0075] The OCT image is converted from polar coordinates to Cartesian coordinates based on the catheter marker point (such as the radio frequency signal of the OCT imaging catheter), and then motion artifact compensation and coordinate system transformation are used to correct the image, so as to obtain the corrected OCT image.

[0076] In the embodiment, the image data marked with the blood vessel segmentation result and the plaque position and the preprocessed OCT image data of the same patient are registered, so as to obtain the registered image data, which comprises:

[0077] The corrected OCT image is matched with the corresponding position of the CTA through the bifurcation and the calcified plaque position, and the mutual information maximization algorithm is used to optimize the registration.

[0078] In the embodiment, the CTA image is segmented to extract the centerline and the lumen-wall boundary by using a model including but not limited to 3D U-Net++, and then morphological operations such as erosion and expansion are combined to remove noise and retain the branch vessel topology. Finally, the origin and direction matrix of the coordinate system are unified with the vessel centerline as the reference system. The OCT image is converted from polar coordinates to Cartesian coordinates based on the catheter marker point (such as the radio frequency signal of the OCT imaging catheter), and then motion artifact compensation and coordinate system transformation are used to correct the image. Finally, the OCT cross section is matched with the corresponding position of the CTA through the bifurcation and the calcified plaque position, and the mutual information maximization algorithm is used to optimize the registration.

[0079] The application also provides a blood vessel OCT image generation method based on a DCN cross-modal fusion network and structure-blood flow coupling modeling, which comprises:

[0080] The deformable convolution-based cross-modal fusion network trained by the DCN cross-modal fusion network and structure-blood flow coupling modeling-based blood vessel OCT image training method and the trained OCT generation network are obtained;

[0081] The coronary CTA image and the blood flow dynamics parameters of the patient to be generated are obtained;

[0082] The coronary CTA image and the blood flow dynamics parameters of the patient to be generated are input into the trained deformable convolution-based cross-modal fusion network, so as to obtain the CTA features of the cross-modal fusion network;

[0083] The CTA feature of the cross-modal fusion network is input into the trained OCT generation network, so as to obtain a cross-sectional OCT image containing blood vessel wall and plaque information.

[0084] In the embodiment, the input of the coronary CTA image to be generated of the patient and the hemodynamic parameter into the trained cross-modal fusion network based on deformable convolution includes:

[0085] The coronary CTA image to be generated of the patient is encoded into a CTA feature map by a 3D deformable convolution network;

[0086] The coronary CTA image to be generated of the patient is preprocessed, so as to obtain image data marked with a blood vessel segmentation result and a plaque position;

[0087] The image data marked with the blood vessel segmentation result and the plaque position is encoded into a heat map, which is multiplied with the CTA feature map channel by channel to obtain a first fusion feature;

[0088] The hemodynamic parameter is extracted to obtain a dynamic feature;

[0089] The dynamic feature and the first fusion feature are fused by using a multi-head attention mechanism to obtain the CTA feature of the cross-modal fusion network.

[0090] Specifically, the input of the cross-modal fusion network based on deformable convolution in the application includes a CTA image, a blood vessel segmentation result, a plaque position and a hemodynamic distribution calculated by CFD.

[0091] In order to adapt to the bending of blood vessels and the diversity of plaque morphology, and improve the feature extraction capability of complex anatomical structures, a 3D deformable convolution network is used to encode a CTA image block (64x64x64) into a CTA feature map. Let the input CTA image be I CTA ∈R H ×W×D After passing through a 3D deformable convolution layer:

[0092] F CTA =DCN(I CTA ;θ DCN )

[0093] Wherein, θ DCN is a deformable convolution parameter, and F CTA is an output feature map. In the deformable convolution formula:

[0094]

[0095] where p represents a position coordinate (e.g., (x, y, z) in a three-dimensional image) on the output feature map, corresponding to the center position of the region to be sampled on the input feature map, F out (p) represents the output value of the position. p k is the standard deviation of the k-th sampling point in the convolution kernel, Δp k is a learnable offset, used to dynamically adjust the position of each sampling point, w k is a weight parameter.

[0096] Δp k is the core of deformable convolution, allowing the model to adaptively adjust the sampling position to adapt to complex geometric structures (such as curved blood vessels), generated by the input feature map and a single-layer convolution, as shown in Figure 2

[0097] The CTA segmentation results (blood vessel mask, plaque position) are encoded into a heat map, multiplied with the feature map channel by channel to enhance the response of the blood vessel wall and plaque region, and suppress the background noise.

[0098] F CTA = F CTA ⊙(Sigmoid(M vessel + H plaque ))

[0099] where M vessel is the blood vessel mask, and H plaque represents the center position of the plaque.

[0100] To combine the hemodynamic parameters calculated by CFD, first, the wall shear stress and blood flow velocity distribution simulated by CFD are extracted by a 3D convolution network (two layers of 3D convolution, channel number: 2→32→64, activation function: LeakyReLU). Since the CFD features are already aligned with the CTA images, the two are fused through a multi-head attention mechanism to improve the ability to capture different modal relationships, resulting in further enhanced CTA features. Finally, the enhanced CTA features are input into the OCT generation network to generate reasonable OCT images.

[0101] Multi-head attention is used to enhance the model's ability to capture different modal relationships:

[0102] Attention(Q, K, V) = Concat(head1, …, head h )W o

[0103] The calculation of each attention head is:

[0104]

[0105] ​where Q is a linear mapping of CTA features and the number of channels of Q is set as C q , K, V are representations of CFD features. h is the number of attention heads, which is set to 8 or 16 for optimal results. d k = C q / h represents the dimension of each head. W i Q , W i V represents a learnable projection matrix; W o represents an output projection matrix.

[0106] To enhance the response to local hemodynamic features, a spatial modulation factor a e R B×1×H×W×D :

[0107] a = sigma(Conv3D([Q; K]))

[0108] where sigma is a Sigmoid function, and the final fused feature is:

[0109] F fusion = Q + a o Attention(Q, K, V)

[0110] In this embodiment, the trained OCT generation network is based on an OCT generation network including but not limited to a conditional diffusion model (or other diffusion model), and the input is the CTA feature of the cross-modal fusion network, and the final cross-sectional OCT image containing blood vessel wall and plaque information is generated.

[0111] To stabilize the OCT image generation process and improve the generation quality, the present application adopts two loss terms, image similarity loss and structure-blood flow coupling loss, to jointly constrain. The image similarity loss consists of image structure similarity and pixel-level L1 norm. The L1 norm is to constrain the absolute difference of pixel values between the generated image and the real image, avoiding the generated result to be excessively blurred or deviate from the basic gray distribution; the structure similarity loss constrains the continuity of the generated image and the real OCT in the key structures such as blood vessel wall layering and plaque edge.

[0112] The trained OCT generation network includes a structure similarity loss, and the structure similarity loss function is as follows:

[0113] L smi = a · ||y gen -y real ||1 + (1-a) · (1-SSIM(y gen , y real ))

[0114] Wherein a is a weight coefficient, SSIM is a structural similarity index, multiple experiments show that when the value of a is small (less than 0.3), the continuity of the OCT image is better, and the image quality is better, but there are problems such as position loss and content distortion. When the value of a is large (greater than 0.5), the content is more complete, but the continuity is poor; the present application sets 0.35 as the final parameter; y gen is the generated OCT image, y real is the real OCT image.

[0115] To strengthen the pathological guidance of hemodynamic parameters to the generation of OCT images, a structure-blood coupling loss is designed to define the correlation between hemodynamic parameters (such as low WSS area) and plaque vulnerability, and the loss function is used to constrain the better generation of OCT lipid core curvature and fiber cap thickness.

[0116] Generate low WSS mask:

[0117] M loss-WSS =Π(WSS<1.5Pa)∈{0,1}

[0118] In this embodiment, if the CFD simulation shows local low WSS, the corresponding generated OCT image has a lipid core curvature (θ lipid ≥180°), otherwise an L2 norm penalty is applied.

[0119]

[0120] Wherein, θ lipid is the lipid core curvature in the OCT image, M loss-wss is the low WSS mask, L align is the structure-blood loss term.

[0121] The loss term of the final network is as follows, γ=0.5:

[0122] L all =L smi +γL align

[0123] In this embodiment, two-stage training is adopted, the first stage adopts 1 / 3 of the data to train the cross-modal fusion network based on deformable convolution and the OCT generation network, and only optimizes the pixel-level L1 loss and SSIM loss. In the second stage (fine-tuning stage), the structure-blood coupling loss is introduced, and all the data is used for training, and the gradient clipping strategy is used to prevent the generation mode from collapsing during training.

[0124] Two post-processing operations are performed on the generated OCT image, one is to remove the generated noise by using non-local mean denoising based on the confidence map of the generated image, and the other is to perform morphological thinning on the intima boundary of the blood vessel to improve the visualization accuracy of the fiber cap thickness.

[0125] The application proposes a blood vessel OCT image generation technology based on a deformable convolution (DCN: Deformable Convolution Network) cross-modal fusion network and structure-blood flow coupling modeling, which realizes high-fidelity image generation from CTA to OCT through cross-modal feature fusion and physical constraint modeling.

[0126] The overall flow of the application is shown in Figure 1 The paired CTA (3D image) and OCT cross-sectional image (intravascular OCT image of the same patient as CTA) are collected. First, 3D U-Net++ is used to segment the blood vessels and plaques of the CTA image, and the blood vessel centerline, lumen-wall and plaque structure are extracted; at the same time, the motion artifact compensation and coordinate system transformation are performed on the paired OCT image to realize the correction of the image. Then, the OCT cross-section is matched with the corresponding position of the CTA through the bifurcation and calcified plaque position, and the multi-modal registration result is optimized using the mutual information maximization algorithm. The next step is to calculate the CFD hemodynamic parameters of the blood vessels, including the wall shear stress distribution and blood flow velocity distribution.

[0127] The input of the cross-modal fusion network based on deformable convolution includes: CTA image block, blood vessel segmentation result, plaque position and CFD calculated hemodynamic distribution. The specific process includes using a 3D deformable convolution network to encode the CTA image block into a CTA feature map, then encoding the CTA segmentation result (blood vessel mask, plaque position) into a heat map, and multiplying it with the CTA feature map channel by channel. In order to combine the CFD calculated hemodynamic parameters, the wall shear stress and blood flow velocity distribution simulated by CFD are input into the cross-modal fusion network as additional channels. Finally, the OCT cross-sectional image is generated by the OCT generation network, which contains the three-layer structure of the blood vessel wall (intima, media, and adventitia) and the plaque details. Two loss functions are used to control the model during the training stage. In addition to the image similarity loss between the OCT real image and the OCT generated image, the correlation rule between the hemodynamic parameters (such as low WSS area) and the plaque vulnerability is defined, and the structure-blood loss function is used to constrain the lipid core curvature and fiber cap thickness of the generated OCT.

[0128] The blood vessel OCT image generation method based on the DCN cross-modal fusion network and structure-blood flow coupling modeling of the application comprises cross-modal fusion based on DCN and blood flow-structure joint modeling, wherein a deformable convolution is embedded in the fusion network, the receptive field is dynamically adjusted according to the CTA blood vessel curvature, multi-scale features are generated, and the deformation problem of curved blood vessels is solved; in addition, the low wall shear stress area simulated by CFD is associated with the lipid core distribution of OCT, and the pathological rationality of the generated image is improved through a physical constraint loss function.

[0129] The application also provides a blood vessel OCT image training device based on a DCN cross-modal fusion network and structure-blood flow coupling modeling, which comprises an image acquisition module, a CTA image preprocessing module, an OCT image preprocessing module, a registration module, a kinetic parameter acquisition module, a model acquisition module and a training module.

[0130] The image acquisition module is used to acquire a plurality of groups of patient coronary CTA images and OCT images, and the OCT images are preliminarily aligned with the coronary CTA images in spatial position.

[0131] The CTA image preprocessing module is used to preprocess each CTA image, thereby acquiring image data marked with blood vessel segmentation results and plaque positions.

[0132] The OCT image preprocessing module is used to preprocess each OCT image, thereby acquiring preprocessed OCT image data.

[0133] The registration module is used to register the image data marked with blood vessel segmentation results and plaque positions and the preprocessed OCT image data belonging to the same patient, thereby acquiring registered image data.

[0134] The kinetic parameter acquisition module is used to acquire the hemodynamic parameters of the same patient.

[0135] The model acquisition module is used to acquire a cross-modal fusion network based on deformable convolution and an OCT generation network.

[0136] The training module is used to train the cross-modal fusion network based on deformable convolution and the OCT generation network respectively through each registered image data and hemodynamic parameter, thereby acquiring a trained cross-modal fusion network based on deformable convolution and a trained OCT generation network.

[0137] Finally, it should be pointed out that: the above examples are only to illustrate the technical solutions of the present application, rather than limit them. Although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A vascular OCT image training method based on DCN cross-modal fusion network and structure-blood flow coupling modeling, characterized by: The vascular OCT image training method based on the DCN cross-modal fusion network and structure-blood flow coupling modeling includes: Acquiring coronary artery CTA images and OCT images of multiple groups of patients, wherein the spatial positions of the OCT images are preliminarily aligned with the coronary artery CTA images; Preprocessing each of the CTA images to obtain image data marked with blood vessel segmentation results and plaque locations; Preprocessing each of the OCT images to obtain preprocessed OCT image data; Registering image data marked with blood vessel segmentation results and plaque locations and pre-processed OCT image data belonging to the same patient, thereby obtaining registered image data; Obtain hemodynamic parameters of the same patient; Obtain a cross-modal fusion network based on deformable convolution and an OCT generation network; The deformable convolution-based cross-modal fusion network and the OCT generation network are trained respectively through each registered image data and hemodynamic parameters, thereby obtaining a trained deformable convolution-based cross-modal fusion network and a trained OCT generation network.

2. The vascular OCT image training method based on DCN cross-modal fusion network and structure-blood flow coupling modeling according to claim 1 is characterized in that: The acquiring of coronary artery CTA images and OCT images of multiple groups of patients, and the preliminary alignment of the spatial positions of the OCT images with the coronary artery CTA images include: N sets of paired coronary artery CTA and OCT images were collected. The CTA data were high-resolution CT vascular imaging, covering the target vessels and including enhanced information on the vascular lumen, plaques, and vessel walls. The OCT data should be the intravascular OCT images of the same patient as the CTA to ensure preliminary alignment of the spatial position with the CTA.

3. The vascular OCT image training method based on DCN cross-modal fusion network and structure-blood flow coupling modeling according to claim 2 is characterized in that: The preprocessing of each of the CTA images to obtain image data marked with blood vessel segmentation results and plaque locations includes: Get a pre-trained segmentation model; Inputting each of the CTA images into a pre-trained segmentation model to obtain mask image data after segmenting blood vessels and plaques; The mask image data is combined with morphological operations to remove noise and retain the branched vascular topology; The mask image data is taken as a reference system with the blood vessel centerline, and the coordinate system origin and direction matrix are unified to obtain image data marked with the blood vessel segmentation results and plaque positions.

4. The vascular OCT image training method based on DCN cross-modal fusion network and structure-blood flow coupling modeling as claimed in claim 3 is characterized in that: Preprocessing each of the OCT images to obtain preprocessed OCT image data includes: The OCT image is converted from polar coordinates to Cartesian coordinates based on the catheter marker points, and then motion artifact compensation and coordinate system transformation are used to achieve image correction, thereby obtaining a corrected OCT image.

5. The vascular OCT image training method based on DCN cross-modal fusion network and structure-blood flow coupling modeling as claimed in claim 4, characterized in that: The registering the image data marked with the blood vessel segmentation result and the plaque position and the pre-processed OCT image data belonging to the same patient to obtain the registered image data includes: The corrected OCT images were matched with the corresponding positions of CTA images by vascular branch points and calcified plaque locations, and the mutual information maximization algorithm was used to optimize the registration.

6. A method for generating vascular OCT images based on DCN cross-modal fusion network and structure-blood flow coupling modeling, characterized in that: The method for generating vascular OCT images based on the DCN cross-modal fusion network and structure-blood flow coupling modeling includes: Obtaining a deformable convolution-based cross-modal fusion network and a trained OCT generation network trained by the vascular OCT image training method based on the DCN cross-modal fusion network and structure-blood flow coupling modeling according to any one of claims 1 to 5; Obtaining coronary artery CTA images and hemodynamic parameters of the patient to be generated; Inputting the coronary artery CTA image of the patient to be generated and the hemodynamic parameters into the trained deformable convolution-based cross-modal fusion network, thereby obtaining CTA features of the cross-modal fusion network; The CTA features of the cross-modal fusion network are input into the trained OCT generation network to obtain a cross-sectional OCT image containing vascular wall and plaque information.

7. The method for generating vascular OCT images based on a DCN cross-modal fusion network and structure-blood flow coupling modeling according to claim 6, characterized in that: Inputting the coronary artery CTA image of the patient to be generated and the hemodynamic parameters into the trained deformable convolution-based cross-modal fusion network to obtain CTA features of the cross-modal fusion network includes: encoding the coronary artery CTA image of the patient to be generated into a CTA feature map through a 3D deformable convolutional network; Preprocessing the coronary artery CTA image of the patient to be generated, thereby obtaining image data marked with a blood vessel segmentation result and a plaque location; The image data marked with the blood vessel segmentation results and plaque locations is encoded into a heat map, and multiplied channel by channel with the CTA feature map to obtain the first fusion feature; Extract dynamic characteristics based on hemodynamic parameters; The multi-head attention mechanism is used to fuse the dynamic features with the first fusion features to obtain the CTA features of the cross-modal fusion network.

8. The method for generating vascular OCT images based on a DCN cross-modal fusion network and structure-blood flow coupling modeling according to claim 7, characterized in that: The trained OCT generation network includes a structural similarity loss, and the structural similarity loss function is as follows: L smi =α·‖y gen -y real ‖1+(1-α)·(1-SSIM(y gen ,y real )); among them, α is the weight coefficient, SSIM is the structural similarity index, L smi is the structural similarity loss function; y gen is the generated OCT image, y real It is a real OCT image; The trained OCT generation network further includes a structure-blood flow coupling loss, which is as follows: Among them, θ lipid is the arc of the lipid core in the OC image, M loss-wss For low WSS mask, L align is the structure-blood flow loss term.

9. The method for generating vascular OCT images based on a DCN cross-modal fusion network and structure-blood flow coupling modeling according to claim 8, characterized in that: The final loss function of the trained OCT generation network is as follows: L all =L smi +γL align Among them, L all is the final loss function, L smi is the structural similarity loss function; γ = 0.5; L align It is the loss of structure-blood flow coupling.

10. A vascular OCT image training device based on DCN cross-modal fusion network and structure-blood flow coupling modeling, characterized in that: The vascular OCT image training device based on the DCN cross-modal fusion network and structure-blood flow coupling modeling includes: An image acquisition module, wherein the image acquisition module is used to acquire coronary artery CTA images and OCT images of multiple groups of patients, wherein the spatial positions of the OCT images are preliminarily aligned with the coronary artery CTA images; A CTA image preprocessing module, configured to preprocess each of the CTA images to obtain image data marked with a blood vessel segmentation result and a plaque location; An OCT image preprocessing module, wherein the OCT image preprocessing module is used to preprocess each of the OCT images and obtain preprocessed OCT image data; a registration module, the registration module being used to register image data marked with blood vessel segmentation results and plaque locations and pre-processed OCT image data belonging to the same patient, thereby obtaining registered image data; A dynamic parameter acquisition module, wherein the dynamic parameter acquisition module is used to obtain hemodynamic parameters of the same patient; A model acquisition module, which is used to acquire a cross-modal fusion network based on deformable convolution and an OCT generation network; A training module is used to train the deformable convolution-based cross-modal fusion network and the OCT generation network respectively through various registered image data and hemodynamic parameters, thereby obtaining a trained deformable convolution-based cross-modal fusion network and a trained OCT generation network.

Images