Method for processing a cardiovascular image and electronic device

Abstract

The application introduces a kind of cardiovascular image processing method and electronic equipment.Belongs to the technical field of cardiovascular image processing.The method comprises: obtaining the first mask corresponding to the cardiovascular structure according to the first image obtained under TOF mode;Obtain the second mask corresponding to the tissue structure according to the second image obtained under T1 mode;Based on cross attention network, obtain the spatial correlation of tissue structure and cardiovascular structure in first image and second image;Based on spatial correlation, align cardiovascular structure and tissue in space;Determine the anomaly between continuous frames based on continuous alignment result.The application can eliminate the geometric difference between modalities, such as movement or resolution difference during scanning, ensure the accurate position of cardiovascular structure on tissue structure, and finally obtain the composite image of precise registration of cardiovascular structure and tissue structure, which can find out artifacts and assist clinical diagnosis.

Description

Technical Field

[0001] This invention relates to the field of cardiovascular image processing, and more particularly to a method and electronic device for processing cardiovascular images. Background Technology

[0002] With the continuous development of interventional surgery, the requirements for visualization of surgical pathways are becoming increasingly stringent. Although current methods for extracting tissue structures in the T1 mode of magnetic resonance imaging and cardiovascular vessels in the TOF mode can achieve good results, accurately fusing tissue structures and cardiovascular vessels plays a crucial role in the success rate of the surgery, and the fusion accuracy of existing technologies is relatively low. In addition, existing technologies also have certain limitations in engineering visualization, namely, poor visualization effects and incomplete visualization range. Summary of the Invention

[0003] At least one aspect and advantage of the invention will be set forth in part in the description which follows, or may be apparent from the description, or may be obtained by practicing the subject matter of this disclosure.

[0004] According to a first aspect of the present invention, a method for processing cardiovascular images is provided, comprising: The first mask corresponding to the cardiovascular structure is obtained from the first image acquired in TOF mode; The second mask corresponding to the tissue structure is obtained from the second image acquired in T1 mode; Spatial correlations between tissue structures and cardiovascular structures in the first and second images are obtained using a cross-attention network. Align cardiovascular structures and tissue structures in space based on spatial correlation; Anomalies between consecutive frames are determined based on the continuous alignment results.

[0005] Optionally, the first mask is extracted based on VisionTransformer or a pre-trained model.

[0006] Optionally, the process of obtaining the spatial correlation between the tissue structure and the cardiovascular structure includes: The first and second masks are encoded respectively to generate corresponding multi-channel tissue feature maps; Use cross-attention networks to obtain the degree of attention that tissue structures pay to blood vessel distribution and spatial attention weight maps; The spatial attention weighting map is used to identify the spatial correlation between tissue structures and cardiovascular structures.

[0007] Optionally, the cross-attention network employs a multi-head attention mechanism.

[0008] Optionally, the step of aligning cardiovascular structures and tissue structures in space includes: The spatial correlation group of the first image, the second image, the tissue structure and the cardiovascular structure is input into the third model to obtain a three-dimensional differentiable deformable field. The first image is deformed using a three-dimensional differentiable deformable field to obtain the deformed third image. Align the third image with the second image.

[0009] Optionally, the process of obtaining the second mask includes: The second image is preprocessed to obtain the preprocessed image; Perform tissue segmentation on the preprocessed image to obtain the corresponding label map of the tissue; The second mask is obtained based on the label map corresponding to the organization.

[0010] Optionally, the spatial correlation extraction process of the tissue structure and cardiovascular structure includes: The first shallow features corresponding to the first image and the second shallow features corresponding to the second image are obtained by using the first encoder and the second encoder respectively. Use convolutional kernels of different sizes to extract semantic information at different levels; A cross-attention fusion network is used to construct the cross-modal interaction between the first shallow feature and the second shallow feature, generating a fused cross-modal feature representation; The fused cross-modal feature representation is modeled using the global context integration module to generate high-level semantic features; The high-level semantic features are progressively upsampled to the original image resolution using a decoder module to generate a segmentation feature map with the original image resolution. The output layer is used to classify and map the segmented feature map to obtain a joint label map of tissue structure and cardiovascular system. The tissue structure and cardiovascular joint label map is used to identify the spatial relationship between tissue structures and cardiovascular structures.

[0011] Optionally, the cross-attention fusion network is a bidirectional attention mode.

[0012] Optionally, the feature spectrum of the image can be obtained by performing a Fourier transform on the spatially aligned image; Differences are determined based on the feature spectra of multiple images obtained in succession; The probability of pulsation artifacts is determined based on abnormal bright spots.

[0013] According to a second aspect of the present invention, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein the memory stores a computer program executable by said at least one processor, said computer program being executed by said at least one processor to cause said at least one processor to perform a cardiovascular image processing method as described in the first aspect of the present invention.

[0014] This invention eliminates geometric differences between modalities through multimodal fusion, such as head movement or resolution differences during scanning, ensuring the accurate positioning of cardiovascular vessels in tissue structures and ultimately obtaining a composite image with precise registration of cardiovascular vessels and tissue structures, which can assist in clinical diagnosis. Attached Figure Description

[0015] Figure 1 This is a flowchart of a cardiovascular image processing method according to an embodiment of the present invention. Detailed Implementation

[0016] The present disclosure will now be discussed with reference to several exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and thus implement the present disclosure, and are not intended to imply any limitation on the scope of the disclosure.

[0017] According to one embodiment of the present invention, a method for processing cardiovascular images includes steps 1100-1500.

[0018] Step 1100: Obtain the first mask corresponding to the cardiovascular structure based on the first image obtained in TOF mode.

[0019] The first image is the original image acquired in TOF (Time-of-Flight) mode, which includes cardiovascular structures and other surrounding tissue noise.

[0020] A mask image is a special type of image data, typically a two-dimensional array with the same dimensions as the original image. Each pixel value on the mask indicates the processing method corresponding to a pixel in the original image. The first mask is a binary mask of the cardiovascular structure extracted from the first image using various methods, such as thresholding. The cardiovascular region is represented by 1, and other regions by 0.

[0021] Step 1200: Obtain the second mask corresponding to the tissue structure based on the second image obtained in T1 mode.

[0022] The T1-weighted mode is used to display tissue structures (such as fat, anatomical tissues, etc.) and has high spatial resolution. The second image is acquired in T1 mode, with high contrast of tissue structures but weak vascular signals. The second mask is the mask image corresponding to the tissue structures extracted from the second image.

[0023] Step 1300: Obtain the spatial correlation between tissue structures and cardiovascular structures in the first and second images based on a cross-attention network.

[0024] Cross-attention networks are used to model the spatial relationships between cardiovascular systems and tissue structures. They can be used to label the spatial coexistence or repulsion between cardiovascular systems and tissues, such as whether a blood vessel passes through a specific tissue region.

[0025] Step 1400: Align cardiovascular structures and tissue structures in space based on spatial correlation.

[0026] During spatial alignment, precise matching of cardiovascular structures in TOF with tissue structures in T1 can eliminate geometric differences between modalities, such as movement or resolution differences during scanning, ensuring the accurate positioning of cardiovascular structures on tissue structures. Ultimately, a composite image with precise registration of cardiovascular and tissue structures can be obtained, which can assist in clinical diagnosis.

[0027] Step 1500: Determine anomalies between consecutive frames based on the consecutive alignment results.

[0028] Continuous alignment results represent the alignment results between multiple consecutive frames and can be used to identify artifacts.

[0029] In this embodiment, the first mask is extracted based on VisionTransformer or a pre-trained model.

[0030] VisionTransformer is a model that successfully transfers the Transformer architecture from natural language processing to the field of computer vision. It processes image data through a pure attention mechanism without the need for traditional convolution operations.

[0031] In image segmentation tasks, the architecture combining VIT (VisionTransformer) with a 3D U-Net decoder is an advanced deep learning method, especially suitable for vascular segmentation of three-dimensional medical images such as TOF-MRA (Time-of-Flight Magnetic Resonance Angiography).

[0032] VIT acts as an encoder, processing the input data and adding learnable 3D positional encodings to preserve spatial information. It also includes a Transformer layer that extracts global contextual features through multi-head self-attention, making it suitable for capturing long-range dependencies in blood vessels.

[0033] The 3D U-Net decoder gradually restores spatial resolution through transposed convolution or interpolation, and fuses features from different layers in the VIT encoder with the corresponding layers in the decoder to compensate for the loss of local details.

[0034] When exporting intermediate files, since the output is a 3D array with the same size as the original image, indicating whether each voxel belongs to the cardiovascular system (e.g., a binarized mask), it can be exported as a VTK file using the VTK library; or exported as an NLFTI format using Nibabel. If the data exists, it can be converted and exported to the appropriate format as needed.

[0035] In this embodiment, step 1300 includes steps 1310-1330.

[0036] Step 1310: Encode the first mask and the second mask respectively to generate corresponding multi-channel tissue feature maps.

[0037] Step 1320: Use a cross-attention network to obtain the degree of attention of tissue structures to blood vessel distribution and spatial attention weight map.

[0038] Step 1330: The spatial attention weight map is used to identify the spatial correlation between tissue structures and cardiovascular structures.

[0039] Multi-channel tissue feature mapping is a technique that enhances tissue characterization by fusing feature information from different modalities or levels. In this embodiment, the first mask is obtained in Time-of-Flight (TOF) mode, and the second mask is obtained in T1 mode; these two belong to different modalities.

[0040] The cross-attention network here employs a multi-head attention mechanism. The spatial attention weight map is used to optimize model performance, indicating which regions deserve more attention.

[0041] This approach employs a guided cross-modal attention mechanism to help the model understand "what blood vessels are near the tissue." During the calculation of cross-modal attention weights, the model generates an attention weight matrix representing the "attention level" of each tissue location to the blood vessel location. Higher weights indicate a stronger spatial correlation between the tissue region and the corresponding blood vessel.

[0042] For spatial attention weights, the output is a two-dimensional or three-dimensional attention weight map. Each pixel / voxel value represents the degree of attention (i.e., the strength of correlation) that the tissue at that location has towards blood vessels. Visualization reveals which regions are closely connected to which blood vessels.

[0043] In this embodiment, step 1400 includes steps 1410-1430.

[0044] Step 1410: Input the spatial correlation group of the first image, the second image, the tissue structure and the cardiovascular structure into the third model to obtain a three-dimensional differentiable deformable field.

[0045] Step 1420: Use a three-dimensional differentiable deformable field to deform the first image to obtain the deformed third image.

[0046] Step 1430: Align the third image with the second image.

[0047] You can use something like VoxelMorph: 1) Data preparation: Load TOF and T1 images and their corresponding vascular masks, tissue masks, and joint label maps; 2) Preprocessing: Standardization, cropping, and registration to a standard space (such as MNI space); 3) Input Construction: The image and its corresponding mask are concatenated to form a multi-channel input, which is not considered as a loss term. The original image and its corresponding mask (e.g., blood vessel mask, tissue mask, etc.) are concatenated together to form a multi-channel input tensor. The purpose of this is to allow the network to directly access this additional information, thereby better learning the spatial relationships between different modalities; 4) Model building: Using the VoxelMorph architecture (UNet+STN); 5) Apply deformation field: deform the TOF image and mask; 6) Evaluation results: The consistency of the deformed structure was evaluated using the joint label diagram.

[0048] When building a model, it can be evaluated using image similarity loss, which measures the similarity between the deformed source image and the target image.

[0049] In this embodiment, the process of obtaining the second mask includes: preprocessing the second image to obtain a preprocessed image; performing tissue segmentation on the preprocessed image to obtain a label map corresponding to the tissue; and obtaining the second mask based on the label map corresponding to the tissue.

[0050] The preprocessed image retains only the tissue structure, with reduced noise and artifacts. The preprocessing procedure includes denoising, bias field correction, and normalization. During denoising, anisotropic filtering or deep learning methods can be used to reduce image noise. Bias field correction eliminates brightness differences in MRI images caused by magnetic field inhomogeneities. Normalization adjusts the image intensity range.

[0051] By performing tissue segmentation on the preprocessed image, it is divided into different tissues, such as fat and muscle. Various methods can be used, including map-based registration algorithms, clustering algorithms, and deep learning. In the label map, each pixel / voxel is labeled with its corresponding tissue category.

[0052] Determine the category of target tissue to be included in the mask based on task requirements, such as including only fat or muscle, or including all tissues. Set the label of the target tissue in the label map to 1, and the rest to 0 to obtain the second mask.

[0053] In this embodiment, step 1300 includes: The first shallow features corresponding to the first image and the second shallow features corresponding to the second image are obtained by using the first encoder and the second encoder respectively. Use convolutional kernels of different sizes to extract semantic information at different levels; A cross-attention fusion network is used to construct the cross-modal interaction between the first shallow feature and the second shallow feature, generating a fused cross-modal feature representation; The fused cross-modal feature representation is modeled using the global context integration module to generate high-level semantic features; The high-level semantic features are progressively upsampled to the original image resolution using a decoder module to generate a segmentation feature map with the original image resolution. The output layer is used to classify and map the segmented feature map to obtain a joint label map of tissue structure and cardiovascular system. The tissue structure and cardiovascular joint label map is used to identify the spatial relationship between tissue structures and cardiovascular structures.

[0054] The combined labeling of tissue structures and cardiovascular system is to better present the anatomical structures in the image.

[0055] Shallow features are those extracted from the input layer in a deep learning model. Multi-scale convolutional kernels are used to extract local semantic information from different levels to enhance the model's ability to perceive structures at different scales.

[0056] Cross-attention fusion networks can employ a unidirectional attention mechanism, where T1 features serve as the query and TOF features as the key and value. They can also be bidirectional, meaning they can use each other as key and value for queries. Here, the cross-attention fusion network uses a bidirectional attention pattern.

[0057] The global context integration module uses standard Transformer Blocks, each of which contains a self-attention layer and a feedforward neural network.

[0058] The decoder module incorporates shallow features extracted from the encoder during upsampling to recover spatial details. The joint label map of tissue structure and cardiovascular system generated by the output layer has the format (B, K, D, H, W), where B represents the batch size, K represents the number of classes, D represents the number of voxels in the depth direction, H represents the height, and W represents the width.

[0059] In this embodiment, the method further includes: obtaining the feature spectrum of the image based on Fourier transform of the spatially aligned image; determining the differences based on the feature spectra of multiple consecutively obtained images; and determining the probability of pulsation artifacts based on abnormal bright spots.

[0060] First, a two-dimensional Fast Fourier Transform is performed on the image, and the amplitude of the spectrum is calculated. During this process, phase information is removed. In this process, the Fourier transform converts the spatial domain image into a frequency domain representation. Periodic afterimages become bright spots in the frequency domain, while normal anatomical structures are concentrated in low-frequency regions.

[0061] At this point, if a vertical afterimage exists, a bright line will be detected in the horizontal or near-horizontal direction. If a horizontal afterimage exists, an afterimage will be generated in the vertical or near-vertical direction. The above afterimages may correspond to afterimages caused by heartbeats or blood flow fluctuations; this invention does not distinguish between them, i.e., only the corresponding regions are detected.

[0062] Next, create a zero-mask of the same size as the image. Then, select a strip area in the horizontal or vertical direction, for example, create a center strip with a height or width of 4 or 5 pixels (it can be in the horizontal row direction or the vertical column direction), and set its mask value to 1. Correspondingly, the width or height of the strip can be 1 / 2, 1 / 3 or other preset values ​​of the image width or height.

[0063] The mask then extracts the spectrum of the specified region and calculates the maximum value and global average energy of the region. The ghosting index is defined as the ratio of the regional maximum value to the global maximum value. Processing continuously spatially aligned images yields a continuous alignment sequence. When a continuous sequence is obtained, anomaly detection algorithms can be used to identify outliers within this sequence. Since continuous artifacts persist while occasional artifacts disappear occasionally, the former, due to its persistent existence, will not be detected; however, occasional artifacts can be detected using this method. For the former, the ghosting index can be tracked to determine if it exceeds a preset value, such as 2.5 or 5.

[0064] This embodiment introduces an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the at least one processor to perform a cardiovascular image processing method as described in any embodiment of the present invention.

[0065] This embodiment introduces a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the cardiovascular image processing method as described in any embodiment of the present invention.

[0066] While specific embodiments of the present invention have been described in detail by way of examples, those skilled in the art should understand that the above examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of the invention.

[0067] Those skilled in the art will recognize that the modules and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0068] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and equipment can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0069] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0070] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0071] In addition, the functional modules in the embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0072] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0073] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

[0074] It should be understood that the sequence numbers of the steps in the invention's content and embodiments do not absolutely imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention. The foregoing description of embodiments of this disclosure has been provided for illustrative and descriptive purposes. The foregoing description is not exhaustive and is not intended to limit this disclosure to the exact form disclosed. Various modifications and variations may exist based on the foregoing teachings, or various modifications and variations may be derived from the practice of this disclosure. These embodiments were chosen and described to illustrate the principles of this disclosure and its practical application, so that those skilled in the art can utilize this disclosure in various implementations and modifications suitable for the specific purpose of the concept.

Claims

1. A method of processing a cardiovascular image, characterized by, include: The first mask corresponding to the cardiovascular structure is obtained from the first image acquired in TOF mode; The second mask corresponding to the tissue structure is obtained from the second image acquired in T1 mode; Spatial correlations between tissue structures and cardiovascular structures in the first and second images are obtained using a cross-attention network. Align cardiovascular structures and tissue structures in space based on spatial correlation; Anomalies between consecutive frames are determined based on the continuous alignment results; The process of obtaining the spatial correlation between the tissue structure and cardiovascular structure includes: The first and second masks are encoded respectively to generate corresponding multi-channel tissue feature maps; Use cross-attention networks to obtain the degree of attention that tissue structures pay to blood vessel distribution and spatial attention weight maps; The spatial attention weighting map is used to identify the spatial correlation between tissue structures and cardiovascular structures.

2. The cardiovascular image processing method as described in claim 1, characterized in that, The first mask is extracted based on VisionTransformer or a pre-trained model.

3. The cardiovascular image processing method as described in claim 1, characterized in that, The cross-attention network employs a multi-head attention mechanism.

4. The cardiovascular image processing method as described in claim 1, characterized in that, The steps involved in spatially aligning cardiovascular and tissue structures include: The spatial correlation group of the first image, the second image, the tissue structure and the cardiovascular structure is input into the third model to obtain a three-dimensional differentiable deformable field. The first image is deformed using a three-dimensional differentiable deformable field to obtain the deformed third image. Align the third image with the second image.

5. The cardiovascular image processing method as described in claim 1, characterized in that, The process of obtaining the second mask includes: The second image is preprocessed to obtain the preprocessed image; Perform tissue segmentation on the preprocessed image to obtain the corresponding label map of the tissue; The second mask is obtained based on the label map corresponding to the organization.

6. The method for processing cardiovascular images as described in claim 1, characterized in that, The process of extracting the spatial correlation between tissue structure and cardiovascular structure includes: The first shallow features corresponding to the first image and the second shallow features corresponding to the second image are obtained by using the first encoder and the second encoder respectively. Use convolutional kernels of different sizes to extract semantic information at different levels; A cross-attention fusion network is used to construct the cross-modal interaction between the first shallow feature and the second shallow feature, generating a fused cross-modal feature representation; The fused cross-modal feature representation is modeled using the global context integration module to generate high-level semantic features; The high-level semantic features are progressively upsampled to the original image resolution using a decoder module to generate a segmentation feature map with the original image resolution. The output layer is used to classify and map the segmented feature map to obtain a joint label map of tissue structure and cardiovascular system. The tissue structure and cardiovascular joint label map is used to identify the spatial relationship between tissue structures and cardiovascular structures.

7. The cardiovascular image processing method as described in claim 6, characterized in that, The cross-attention fusion network is a bidirectional attention mode.

8. The method for processing cardiovascular images as described in claim 1, characterized in that, The feature spectrum of the image is obtained by performing a Fourier transform on the spatially aligned image. Differences are determined based on the feature spectra of multiple images obtained in succession; The probability of pulsation artifacts is determined based on abnormal bright spots.

9. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the at least one processor to perform the cardiovascular image processing method as described in any one of claims 1-8.

Images