A multi-modal fusion method for arrhythmia radioablation

Abstract

This invention discloses a multimodal fusion method for radiotherapy ablation of arrhythmias, comprising the following steps: Step S1, extracting features of ETV and STV by using a feature detection algorithm to extract ETV and STV features from electrophysiological images and scar images; Step S2, matching features by using a feature matching algorithm to find the correspondence between ETV and STV features; Step S3, fusing images by fusing the two images according to the matched feature pairs, applying a transformation model to calculate the transformation parameters based on the correspondence between the feature pairs, and then mapping one image onto the other. This invention combines feature extraction, feature matching, and image fusion techniques to effectively combine different types of medical images, thereby obtaining more comprehensive and detailed cardiac information. Compared with existing technologies, this invention can not only improve the accuracy of ablation surgery but also reduce the risk caused by visual misjudgment.

Description

Technical Field

[0001] This invention discloses a multimodal fusion method for radioablation of arrhythmias, relating to the field of medical image fusion technology. Background Technology

[0002] The greatest advantages of STAR (Symptom-Assisted Radiotherapy) for arrhythmias lie in its non-invasiveness, precision, safety, and effectiveness. The applicant's preliminary research has shown that patients have a good perioperative experience during STAR treatment of arrhythmias, and no patients experienced clear short-term or long-term adverse reactions during postoperative follow-up (1 year). One of the core challenges in STAR treatment of arrhythmias is precise target localization, which is also a research difficulty in current technologies.

[0003] Conventional techniques for target localization involve combining surface electrocardiography, echocardiography, cardiac MRI, and cardiac CT with electrophysiological mapping signals. These signals are comprehensively analyzed by an electrophysiologist, and the target segment is determined according to the 17-segment classification method defined by the American Society of Echocardiography. Finally, a preliminary target volume (GTV) is delineated using 4D-CT images in a specialized treatment planning system (TPS). After correcting for the effects of respiratory motion and cardiac pulsation, the final target volume (PTV) is formed. This localization method is susceptible to the surgeon's experience and clinical skills, and may result in mislocalization, missed localization, or off-target effects, leading to treatment failure.

[0004] Content of this invention

[0005] The purpose of this invention is to provide a multimodal fusion method for radioablation of arrhythmias, so as to achieve more accurate and objective target localization.

[0006] To achieve the above-mentioned technical objectives and effects, the invention is implemented through the following technical solution:

[0007] A multimodal fusion method for radiotherapy ablation of arrhythmias includes the following specific steps:

[0008] S1. Extract features from ETV and STV.

[0009] Features of ETV and STV were extracted from electrophysiological images and scar images using a feature detection algorithm.

[0010] S2, Matching features,

[0011] The correspondence between ETV and STV features is found using a feature matching algorithm;

[0012] S3, Image fusion

[0013] Based on the matched feature pairs, the two images are fused using a transformation model. The transformation parameters are calculated based on the correspondence of the feature pairs, and then one image is mapped onto the other.

[0014] Furthermore, the feature detection algorithm is the SIFT (Scale-Invariant Feature Transform) feature extraction algorithm, specifically including:

[0015] S1.1 scale spatial extremum detection,

[0016] Gaussian blur is applied to each layer of the image to generate a scale space. Potential key points are found by searching for extreme points in the scale space.

[0017] S1.2 Key Point Location,

[0018] At each candidate keypoint location, the keypoint position is accurately determined by fitting neighboring data;

[0019] S1.3 Direction Allocation,

[0020] Assign one or more orientations to each keypoint, and all subsequent operations on the image data are transformed relative to the orientation, scale, and position of the keypoints, thus providing invariance to these transformations;

[0021] S1.4 Keypoint Descriptor,

[0022] Within the neighborhood of the keypoint, the image gradient is measured at a selected scale to generate a descriptor for the keypoint.

[0023] Furthermore, the feature detection algorithm is the SURF (Speeded Up Robust Features) feature extraction algorithm, specifically including:

[0024] S1.1 Constructs a scale space,

[0025] The scale space of the input image is constructed at different scales using Gaussian blur and downsampling methods;

[0026] S1.2 Key Point Detection

[0027] In scale space, keypoints are detected using the determinant and trace of the Hessian matrix;

[0028] S1.3 Direction Allocation,

[0029] Calculate the Haar wavelet response around each keypoint and assign a principal direction to the keypoint using a sliding direction window;

[0030] S1.4 Descriptor generation,

[0031] The Haar wavelet response is calculated at the scale and orientation of the keypoints, and the descriptor of the keypoints is generated.

[0032] Furthermore, the feature matching algorithm is a nearest neighbor matching method based on Euclidean distance, specifically including:

[0033] S2.1 Calculate the eigenvectors,

[0034] For each keypoint, calculate its feature vector;

[0035] S2.2 Calculate the distance.

[0036] For a given keypoint, calculate its Euclidean distance to the feature vectors of all other keypoints;

[0037] S2.3 Nearest Neighbor Matching

[0038] Find the key point corresponding to the nearest feature vector and use it as the match for the given key point.

[0039] Furthermore, the transformation model is an affine transformation, specifically including:

[0040] 3.1 Select key points,

[0041] Select three key points that are not on a straight line;

[0042] 3.2 Calculate the transformation matrix.

[0043] By solving the system of equations, we can find the affine transformation matrix that maps the key points in the ETV image to the corresponding key points in the STV image.

[0044] 3.3 Applying transformations,

[0045] The calculated affine transformation matrix is ​​used to transform the ETV image to obtain an image that matches the STV image.

[0046] Furthermore, the transformation model is an affine transformation, specifically including:

[0047] 3.1 Select key points,

[0048] Select four key points that are not on a straight line;

[0049] 3.2 Calculate the transformation matrix.

[0050] By solving the system of equations, we can find the perspective transformation matrix that can map the key points in the ETV image to the corresponding key points in the STV image;

[0051] 3.3 Applying transformations,

[0052] The calculated perspective transformation matrix is ​​used to transform the ETV image to obtain an image that matches the STV image.

[0053] Furthermore, step S3 is followed by step S4, which adds a "widening the edge" step, specifically including:

[0054] S4.1 requires determining the area where the edge expands.

[0055] Create a two-dimensional array of masks with the same dimensions as the image. In this array, set the values ​​within the ITV area to 0 and the values ​​outside to 1.

[0056] S4.2 uses the morphological expansion operation to "enlarge" the edge.

[0057] Morphological dilation operations can be implemented using sliding windows (such as convolution kernels). If any element within the window is 1, then the corresponding output of that window is 1.

[0058] This step expands the edge area;

[0059] S4.3 applies the enlarged edges to the original image.

[0060] This can be achieved by multiplying the original image by the enlarged edge to obtain the enlarged ITV area.

[0061] Furthermore, after step S4, the method also includes steps for identifying and labeling OARs, namely step S5, which involves identifying and labeling OARs, specifically including:

[0062] S5.1 uses existing image recognition algorithms to identify and label OARs in images;

[0063] S5.2 The OAR regions marked in the image can be labeled using different colors or image values;

[0064] S5.3 incorporates these marked areas into the calculation process during image fusion to ensure that these areas are properly processed when performing operations such as edge enlargement.

[0065] Furthermore, step S5 also includes:

[0066] In the process of expanding the edge, S5.4 needs to judge each pixel that may be expanded. If the pixel is located in the already marked OAR area, it will not be expanded.

[0067] S5.5 If, during the expansion process, it is found that the edge of the expansion enters the OAR area, then the expansion needs to be stopped, or the direction of expansion needs to be changed to avoid the OAR area;

[0068] After all expansion operations are completed, S5.6 requires recalculating the edge of the ITV to ensure that the new edge is completely contained within the ITV and does not include any OAR area.

[0069] Furthermore, in step S5.1, the image recognition algorithm is a convolutional neural network, and the specific implementation steps are as follows:

[0070] S5.11 runs a trained CNN model to identify and label OARs in images;

[0071] During the edge expansion process, S5.12 judges each pixel that may be expanded. If the pixel is located within the already marked OAR area, it will not be expanded.

[0072] S5.13 If, during the expansion process, it is found that the edge of the expansion enters the OAR area, then the expansion shall be stopped, or the direction of expansion shall be changed to avoid the OAR area;

[0073] S5.14 After all expansion operations are completed, recalculate the edge of the ITV to ensure that the new edge is completely contained within the ITV and does not include any OAR area.

[0074] Beneficial effects:

[0075] This invention combines feature extraction, feature matching, and image fusion techniques to effectively integrate different types of medical images, thereby obtaining more comprehensive and detailed cardiac information. Compared to existing technologies, this invention not only improves the precision of ablation surgery but also reduces the risks caused by visual misinterpretation. Furthermore, the implementation of this invention provides an effective tool for conducting more complex research and treatment of cardiac diseases.

[0076] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0077] Figure 1 This is a flowchart of the multimodal fusion method for radiotherapy ablation of arrhythmias described in an embodiment of the present invention;

[0078] Figure 2 The figure shows the actual implementation result of the multimodal fusion method for radiotherapy ablation of arrhythmias described in this embodiment of the invention. Detailed Implementation

[0079] To more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings.

[0080] This invention discloses a multimodal fusion method for radiotherapy ablation of arrhythmias, comprising the following specific steps:

[0081] S1. Extract the features of ETV and STV. The features of ETV and STV are extracted from electrophysiological images and scar images using a feature detection algorithm.

[0082] This step extracts useful information from different types of images. The features of ETV and STV can reflect the electrophysiological activity and structural characteristics of the myocardium, which is crucial for precise radioablation.

[0083] S2. Matching features: Find the correspondence between ETV and STV features through feature matching algorithms.

[0084] This step determines the correspondence between identical regions in the two types of images to ensure that the same anatomical structures can be correctly aligned during image fusion. This alignment helps us understand and observe the target region from different perspectives and scales, thereby more accurately locating the ablation target.

[0085] S3. Image fusion: Based on the matched feature pairs, the two images are fused. This is achieved by applying a transformation model, calculating the transformation parameters based on the correspondence of the feature pairs, and then mapping one image onto the other.

[0086] This step involves fusing two types of images into a single image to allow for observation and analysis of the target region within a unified visual framework. This fusion not only increases the amount of information we can glean from the image but also provides a more comprehensive and detailed perspective, helping doctors develop more precise treatment plans.

[0087] Example 1

[0088] In this embodiment, based on the aforementioned scheme, the feature detection algorithm in step 1 is the SIFT (Scale-Invariant Feature Transform) feature extraction algorithm, specifically including:

[0089] S1.1 scale spatial extremum detection,

[0090] Gaussian blur is applied to each layer of the image to generate a scale space. Potential key points are found by searching for extrema in the scale space.

[0091] This step in multimodal fusion can handle images from different sources (such as MRI and CT), which may have different resolutions and scales. By generating a scale space on each image layer and finding extrema, we can find stable keypoints at different scales, providing crucial information for subsequent image alignment and fusion.

[0092] The scale space can be constructed in the following way:

[0093] L(x,y,σ)=G(x,y,z)*I(x,y);

[0094] Where L(x,y,σ) is the scale space function, G(x,y,z) is the two-dimensional Gaussian function, σ is the scale of the scale space (that is, the standard deviation of the Gaussian blur), I(x,y) is the original image, and * denotes the convolution operation.

[0095] S1.2 Key Point Location,

[0096] At each candidate keypoint location, the keypoint position is accurately determined by fitting neighboring data;

[0097] S1.3 Direction Allocation,

[0098] Each keypoint is assigned one or more orientations, and all subsequent operations on the image data are transformed relative to the orientation, scale, and position of the keypoints, thus providing invariance to these transformations.

[0099] In multimodal fusion of radiographic ablation for arrhythmias, the images that may be processed are not perfectly aligned and may have variations such as rotation. To ensure that the extracted features remain unchanged when the image is rotated, we need to assign one or more orientations to each keypoint. In this step, we typically calculate the gradient directions of the pixels around the keypoint and assign the most dominant orientation to that keypoint.

[0100] The main direction of the key point can be calculated using the following formula:

[0101]

[0102] Where θ is the principal direction of the key point. and These are the gradients of the image in the x and y directions, respectively.

[0103] S1.4 Keypoint Descriptor,

[0104] Within the neighborhood of the keypoint, the image gradient is measured at a selected scale to generate a descriptor for the keypoint.

[0105] In this embodiment, by finding key points at different scales, the SIFT algorithm can extract stable features from images of different scales and resolutions, which is of great significance for processing medical images from different sources (such as MRI and CT).

[0106] Furthermore, the SIFT algorithm assigns one or more orientations to each keypoint, meaning that the features we extract are insensitive to image rotation. This is particularly important for processing medical images that may have variations such as rotation. Moreover, the keypoint descriptors generated by the SIFT algorithm can effectively represent the local structural information of the image, which helps us accurately find correspondences when matching features.

[0107] Compared to existing technologies, the SIFT feature extraction algorithm can stably extract image features under various conditions such as scale changes, rotation, and brightness changes, exhibiting higher robustness compared to traditional feature extraction methods. By introducing SIFT feature extraction, images from different modalities can be matched more accurately, thereby improving the accuracy of the fused image. This is crucial for medical procedures requiring precise localization, such as radiotherapy for arrhythmias. Furthermore, SIFT feature extraction is not dependent on a specific image modality, meaning that the method of this embodiment can be easily extended to other image fusion tasks.

[0108] In another embodiment, the feature detection algorithm is the SURF (Speeded Up Robust Features) feature extraction algorithm, specifically including:

[0109] S1.1 Constructs a scale space,

[0110] The scale space of the input image is constructed at different scales using Gaussian blur and downsampling methods.

[0111] In this embodiment, the SURF algorithm employs Gaussian blurring and downsampling to construct a scale space, which enhances the stability of image feature detection. In the multimodal fusion process of radioablation for arrhythmias, the scale space helps us stably detect features at different scales and resolutions.

[0112] S1.2 Key Point Detection

[0113] In scale space, keypoints are detected using the determinant and trace of the Hessian matrix.

[0114] SURF uses the determinant and trace of the Hessian matrix to detect keypoints, a method that is simpler and more efficient than SIFT. In the multimodal fusion process of radioablation for arrhythmias, fast and effective keypoint detection can improve the algorithm's running speed. Keypoint detection can be represented as:

[0115] det(Hession) = DxxDyy - (ωDxy) 2 ;

[0116] Where DxxDyy and Dxy are elements of the Hessian matrix, and ω is the weight coefficient.

[0117] S1.3 Direction Allocation,

[0118] Calculate the Haar wavelet response around each keypoint and assign a principal direction to the keypoint using a sliding direction window.

[0119] In this embodiment, SURF enhances robustness to rotation by calculating the Haar wavelet response around each keypoint and assigning a principal orientation to the keypoint using a sliding orientation window. In the multimodal fusion process of radioablation for arrhythmias, orientation assignment helps us stably match features under transformations such as rotation.

[0120] S1.4 Descriptor generation,

[0121] The Haar wavelet response is calculated at the scale and orientation of the keypoints, and the descriptor of the keypoints is generated.

[0122] In this embodiment, SURF calculates the Haar wavelet response at the scale and orientation of keypoints and generates keypoint descriptors. This method can effectively represent the local structural information of the image, which is beneficial for accurately finding the correspondence when matching features. In the multimodal fusion process of radioablation for arrhythmias, the powerful descriptive capability helps us to better fuse images and improve the accuracy of fusion.

[0123] Compared to SIFT in the previous embodiment, the SURF algorithm in this specific embodiment is more computationally efficient, mainly in the keypoint detection and descriptor generation steps. SURF uses the determinant and trace of the Hessian matrix to detect keypoints, which is simpler and more efficient than SIFT. For descriptor generation, SURF uses the Haar wavelet response, which is easier to compute than SIFT's gradient histogram.

[0124] Furthermore, SURF enhances its robustness to rotation by assigning a principal direction to each keypoint. This feature ensures the stability and reliability of the algorithm even if the image is rotated during capture when processing medical images.

[0125] Furthermore, SURF generates keypoint descriptors rich in information by calculating the Haar wavelet response at the scale and orientation of the keypoints. This allows for more accurate identification of correspondences during feature matching, improving the stability and accuracy of the matching process.

[0126] Compared to existing technologies, SURF's advantages lie in its robustness to scale and rotation, as well as its higher computational efficiency. This enables it to perform feature matching and image fusion more effectively and accurately in multimodal fusion of radiographic ablation for arrhythmias. Furthermore, SURF's strong ability to represent local structural information in images helps improve matching accuracy and further enhances the quality of multimodal fusion.

[0127] Example 2

[0128] In this specific embodiment, the feature matching algorithm in step 2 is a nearest neighbor matching method based on Euclidean distance, specifically including:

[0129] S2.1 Calculate the eigenvectors,

[0130] For each keypoint, calculate its feature vector.

[0131] In this embodiment, during the multimodal fusion process of radioablation for arrhythmia, this step generates a vector for each detected keypoint. This vector effectively describes the image features surrounding the keypoint. This feature vector provides the basis for subsequent feature matching.

[0132] S2.2 Calculate the distance.

[0133] For a given keypoint, calculate its Euclidean distance to the feature vectors of all other keypoints.

[0134] For a given keypoint, calculate its Euclidean distance to the feature vectors of all other keypoints. In multimodal fusion for radioablation of arrhythmias, this step measures the similarity between two keypoints by calculating the Euclidean distance between their feature vectors. The calculation formula is as follows:

[0135]

[0136] Where p and q are two feature vectors, and n is the dimension of the feature vectors.

[0137] S2.3 Nearest Neighbor Matching

[0138] Find the key point corresponding to the nearest feature vector and use it as the match for the given key point.

[0139] Specifically, the keypoint corresponding to the nearest feature vector is identified and used as the match for a given keypoint. In multimodal fusion of radioablation for arrhythmia, this step matches each keypoint to the keypoint that is most similar to it (i.e., the keypoint with the smallest Euclidean distance between its feature vectors), thereby completing the keypoint matching between two images.

[0140] In this embodiment, by calculating the feature vector of each keypoint, the embodiment can effectively describe the image features around each keypoint, providing an accurate basis for feature matching. Secondly, by calculating the Euclidean distance between feature vectors, the embodiment can accurately measure the similarity between two keypoints. This method is efficient for identifying identical or similar regions, which is beneficial for finding corresponding regions in multimodal images.

[0141] Furthermore, this embodiment uses the nearest neighbor matching method, which can effectively find the key point most similar to a given key point (i.e., with the smallest Euclidean distance between the feature vectors), thereby completing the key point matching between two images. This step greatly improves the accuracy and reliability of the matching.

[0142] Compared to existing multimodal fusion techniques for radioablation of arrhythmias, this embodiment innovates in feature matching. Existing technologies typically use manual or semi-automatic methods to match images of different modalities. These methods often require significant human intervention, and the matching results are frequently uncertain. This embodiment, however, uses a nearest neighbor matching method based on Euclidean distance, greatly reducing the degree of human intervention, improving matching accuracy, and enhancing the system's robustness.

[0143] Example 3

[0144] In this embodiment, the transformation model in step 3 is an affine transformation, specifically including:

[0145] 3.1 Select key points,

[0146] Choose three key points that are not on a straight line.

[0147] In affine transformation, three key points that are not on a straight line need to be selected in order to construct a transformation model with six degrees of freedom, thereby realizing changes in the image such as rotation, translation, scaling, and shearing.

[0148] In this embodiment, the three key points selected in the multimodal fusion of radioablation for arrhythmia may correspond to anatomical landmarks with obvious features in the ETV and STV modal images, such as specific parts of the heart.

[0149] Suppose the three selected key points are located at (x1, y1), (x2, y2), and (x3, y3) in the original image.

[0150] 3.2 Calculate the transformation matrix.

[0151] By solving the system of equations, we can find the affine transformation matrix that maps key points in the ETV image to the corresponding key points in the STV image.

[0152] By solving the system of equations corresponding to the three points in the two images, a 3x3 affine transformation matrix can be obtained. This matrix can map the key points in the ETV image to the corresponding key points in the STV image.

[0153] In multimodal fusion of radioablation for arrhythmias, this step involves mapping the key point locations in the ETV image to the corresponding locations in the STV image using a calculated affine transformation matrix, thereby achieving registration between different modalities.

[0154] Assuming the affine transformation matrix is ​​A, then we have:

[0155]

[0156] 3.3 Applying transformations,

[0157] The calculated affine transformation matrix is ​​used to transform the ETV image to obtain an image that matches the STV image.

[0158] The obtained affine transformation matrix is ​​applied to transform the ETV image, thus obtaining an image that matches the STV image, thereby achieving the fusion of the two modalities.

[0159] In multimodal fusion of radioablation for arrhythmias, affine transformation can be used to achieve precise mapping from ETV images to STV images, so that the two modal images have consistent visual effects at the same anatomical location, thereby better guiding the surgery.

[0160] After applying the affine transformation, the new coordinates (x, y) of each point (x, y) in the original image are... ′ ,y ′ )satisfy:

[0161]

[0162] This embodiment applies an affine transformation model, calculating the transformation matrix through three key points to achieve accurate mapping from ETV images to STV images, further enabling precise registration between different modalities. Affine transformation allows for changes in image rotation, translation, scaling, and cropping, enabling more flexible processing and fusion of medical images from different modalities.

[0163] The steps in this embodiment, especially the selection of key points and the application of affine transformation, are all automated processes, which reduce the complexity of manual operation and improve processing efficiency.

[0164] The application of affine transformation improves the accuracy of image registration across different modalities, which is of great significance for accurately identifying lesion areas and improving the accuracy of radioablation.

[0165] Compared to existing technologies, this invention achieves automated fusion of images from different modalities, reducing the workload of doctors and minimizing errors caused by manual operation, thereby improving the safety and efficiency of diagnosis and treatment. Furthermore, it improves the quality of the fused images, ensuring consistent visual effects for both modalities at the same anatomical location. This helps doctors better understand the patient's condition and perform more precise surgical procedures.

[0166] In another specific embodiment, the transformation model is an affine transformation, specifically including:

[0167] 3.1 Select key points,

[0168] Select four key points that are not on a straight line.

[0169] Choosing four key points that are not on a straight line is to build a more accurate perspective transformation model. This is because perspective transformation requires 8 degrees of freedom, so at least 4 pairs of points are needed to uniquely determine a perspective transformation.

[0170] The application of this step in the multimodal fusion of radioablation for arrhythmias is as follows: four significant points shared by ETV and STV images that are not on a straight line are selected as key points. These points can be tissue boundaries, vascular intersections, etc.

[0171] The corresponding formula is the formula for perspective transformation, which can be expressed as:

[0172]

[0173] 3.2 Calculate the transformation matrix.

[0174] By solving the system of equations, we can find the perspective transformation matrix that maps key points in the ETV image to the corresponding key points in the STV image.

[0175] In this embodiment, the perspective transformation can be found by solving a system of linear equations.

[0176] In multimodal fusion of radioablation for arrhythmias, the solution of the perspective transformation matrix based on the selected key points can be used to perform perspective transformation on ETV images to achieve the same effect as STV images.

[0177] For each pair of matching points (x ′ ,y ′ Given (x, y), we can obtain two equations:

[0178]

[0179]

[0180] Where ah is the parameter in the perspective transformation matrix, and x and y are points in the original image, x ′ ,y ′ These are points in the target image.

[0181] 3.3 Applying transformations,

[0182] The calculated perspective transformation matrix is ​​used to transform the ETV image to obtain an image that matches the STV image.

[0183] (In multimodal fusion of radioablation for arrhythmias, perspective transformation can be used to map ETV images to STV images more accurately, thereby achieving accurate registration of images of different modalities.)

[0184] For each pixel (x, y) of the ETV image, map it to a new position (x, y) using a perspective transformation matrix. ′ ,y ′ The calculation method is the same as above.

[0185] This embodiment uses perspective transformation, which can handle more complex image registration problems. Perspective transformation is a more general transformation model that can not only perform rotation, scaling, and translation transformations, but also handle complex transformations caused by viewpoint or lens distortion. Therefore, this method has advantages in processing ETV and STV images with complex field-of-view changes. Using perspective transformation, this invention can more accurately align ETV and STV images. This is crucial for the precise execution of radioablation surgery for arrhythmias, because only with accurate alignment of the two modalities can surgeons obtain accurate three-dimensional positioning information, thereby enabling precise surgical procedures.

[0186] Compared to existing technologies, traditional image registration methods are typically based on linear or rigid body transformations, which cannot handle complex transformations caused by changes in viewpoint or lens distortion. Perspective transformation, on the other hand, can handle these complex situations, thus offering a significant advantage when dealing with complex image registration problems.

[0187] By selecting four key points and solving for the perspective transformation matrix, this method can achieve more accurate image registration. Traditional methods, on the other hand, may require selecting more matching points or fail to achieve accurate image registration.

[0188] Compared to traditional methods, this perspective-transformation-based approach enables more precise multimodal fusion. Therefore, it plays a crucial role in improving the accuracy of radioablation surgery for arrhythmias.

[0189] Example 4

[0190] In addition to the methods described in Examples 1-3, in some cases, considering that the GTV may shift due to respiratory movements and heartbeats, the range of GTV movement is defined as the internal target volume (ITV) when the patient is supine and in a fixed position, breathing freely under 4D-CT. Due to interference from various other uncontrollable factors (such as slight displacement of the patient despite the provision of positional protection measures), the target volume may shift. Therefore, when performing STAR in clinical practice, it is necessary to extend the irradiation of the outer edge of the ITV by 1-5 mm.

[0191] Therefore, in this embodiment, step S3 is followed by step S4, which involves adding a "widening edge" step, specifically including:

[0192] S4.1 requires determining the area where the edge expands.

[0193] Create a two-dimensional array of masks with the same dimensions as the image. In this array, set the values ​​within the ITV area to 0 and the values ​​outside to 1.

[0194] The purpose of this step is to define the area where edge enlargement is needed, that is, to determine the extent of the target area's enlargement edge. This can be achieved by creating a two-dimensional array (mask) with the same size as the original image, setting the values ​​in the ITV area to 0 and the values ​​outside to 1. Specifically, this step constructs a mask image that clearly marks the ITV and non-ITV areas.

[0195] The application of this step in multimodal fusion for radiographic ablation of arrhythmias mainly lies in accurately aligning images from different modalities to obtain more accurate ITV. In practical applications, this can improve the accuracy and effectiveness of the procedure.

[0196] Specifically, suppose the original image is M with size m·n, and the mask image is Mask with size m·n. Then we have

[0197]

[0198] S4.2 uses the morphological expansion operation to "enlarge" the edge.

[0199] Morphological dilation operations can be implemented using sliding windows (such as convolution kernels). If any element within the window is 1, then the corresponding output of that window is 1.

[0200] This step expands the edge area.

[0201] Morphological dilation is an image processing method that can be implemented using a sliding window (such as a convolution kernel). If there is an element with a value of 1 within the window, the corresponding output of the window will be 1, thus achieving edge enlargement.

[0202] In multimodal fusion of radioablation for arrhythmias, morphological expansion manipulation can be used to widen the edge of the ITV to account for GTV displacement that may be caused by uncontrollable factors (such as respiratory motion, cardiac pulsation, etc.).

[0203] Morphological dilation can be represented by a convolution operation, assuming a window of size $k \times k$.

[0204] If $W$ is expanded, and $DilatedMask$ represents the expanded mask image, then:

[0205] S4.3 applies the enlarged edges to the original image.

[0206] This can be achieved by multiplying the original image by the enlarged edge to obtain the enlarged ITV area.

[0207] In a further preferred embodiment of this example, the applicant further considers the situation of organs at risk (OARs) surrounding the target area to form a PTV. Traditionally, this is typically done after physician review and modification when developing a treatment plan. Organs at risk are delineated by the radiation oncologist and include the lungs, heart, bronchi, esophagus, spinal cord, and breast, among others.

[0208] To achieve this goal through automation, this preferred embodiment further includes a step of identifying and labeling OARs after step S4, namely step S5, identifying and labeling OARs, which specifically includes:

[0209] S5.1 uses existing image recognition algorithms to identify and label OARs in images.

[0210] S5.2 The OAR areas marked in the image can be labeled using different colors or image values.

[0211] S5.3 incorporates these marked areas into the calculation process during image fusion to ensure that these areas are properly processed when performing operations such as edge enlargement.

[0212] In a further embodiment,

[0213] In the process of expanding the edge, S5.4 needs to judge each pixel that may be expanded. If the pixel is located in the already marked OAR area, it will not be expanded.

[0214] S5.5 If, during the expansion process, it is found that the edge of the expansion enters the OAR area, then the expansion needs to be stopped, or the direction of expansion needs to be changed to avoid the OAR area;

[0215] After all expansion operations are completed, S5.6 requires recalculating the edge of the ITV to ensure that the new edge is completely contained within the ITV and does not include any OAR area.

[0216] In a further preferred embodiment, in step S5.1, the image recognition algorithm is a convolutional neural network, and the specific implementation steps are as follows:

[0217] S5.11 runs a trained CNN model to identify and label OARs in images;

[0218] During the edge expansion process, S5.12 judges each pixel that may be expanded. If the pixel is located within the already marked OAR area, it will not be expanded.

[0219] S5.13 If, during the expansion process, it is found that the edge of the expansion enters the OAR area, then the expansion shall be stopped, or the direction of expansion shall be changed to avoid the OAR area;

[0220] S5.14 After all expansion operations are completed, recalculate the edge of the ITV to ensure that the new edge is completely contained within the ITV and does not include any OAR area.

[0221] In the foregoing embodiments, the specific automatically drawn image can be found in [reference needed]. Figure 2 The figure contains Aorta and LV structures, and the white curved area is the VT target region.

[0222] The above are merely some of the embodiments of this application and are not intended to limit the application in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments shall still fall within the scope of protection of the technical solution of this application.

Claims

1. A multi-modality fusion method for radioablation of cardiac arrhythmias, characterized in that, The specific steps include the following: S1. Extract features from ETV and STV. Features of ETV and STV were extracted from electrophysiological images and scar images using a feature detection algorithm. S2, Matching features, The correspondence between ETV and STV features is found using a feature matching algorithm; S3, Image fusion Based on the matched feature pairs, the two images are fused by applying a transformation model. The transformation parameters are calculated based on the correspondence of the feature pairs, and then one image is mapped onto the other image. The process includes step S4 after step S3, which involves adding a "widening the edge" step, specifically including: S4.1 requires determining the area where the edge expands. Create a two-dimensional array of masks with the same dimensions as the image. In this array, set the values ​​within the ITV area to 0 and the values ​​outside to 1. S4.2 uses the morphological expansion operation to "enlarge" the edge. Morphological dilation can be implemented using convolution kernels. If any element within a window is 1, then the output for that window is 1. This step expands the edge area; S4.3 applies the enlarged edges to the original image. This can be achieved by multiplying the original image by the enlarged edge to obtain the enlarged ITV area; The step S4 is followed by a step of identifying and labeling OARs, namely step S5, which includes: S5.1 uses existing image recognition algorithms to identify and label OARs in images; S5.2 The OAR regions marked in the image can be labeled using different colors or image values; S5.3 incorporates these labeled regions into the calculation process during image fusion; Step S5 further includes: S5.4 During the process of expanding the edge, it is necessary to determine the area that needs to be expanded. If the area is located within the already marked OAR area, then expansion will not be performed. S5.5 If, during the expansion process, it is found that the edge of the expansion enters the OAR area, then the expansion needs to be stopped, or the direction of expansion needs to be changed to avoid the OAR area; After all the expansion operations are completed, S5.6 needs to recalculate the edge of the ITV to ensure that the new edge is completely contained within the ITV and does not include any OAR area; In step S5.1, the image recognition algorithm is a convolutional neural network, and the specific implementation steps are as follows: S5.11 runs a trained CNN model to identify and label OARs in images; S5.12 During the edge expansion process, the area to be expanded is determined. If the area is located within the already marked OAR area, then expansion is not performed. S5.13 If, during the expansion process, it is found that the edge of the expansion enters the OAR area, then the expansion shall be stopped, or the direction of expansion shall be changed to avoid the OAR area; S5.14 After all expansion operations are completed, recalculate the edge of the ITV to ensure that the new edge is completely contained within the ITV and does not include any OAR area.

2. The multi-modality fusion method for radioablation of cardiac arrhythmia of claim 1, wherein, The feature detection algorithm is the SIFT (Scale-Invariant Feature Transform) feature extraction algorithm, specifically including: S1.1 scale spatial extremum detection, Gaussian blur is applied to each layer of the image to generate a scale space. Potential key points are found by searching for extreme points in the scale space. S1.2 Key Point Location, At each candidate keypoint location, the keypoint position is accurately determined by fitting neighboring data; S1.3 Direction Allocation, Assign one or more orientations to each keypoint, and all subsequent operations on the image data are transformed relative to the orientation, scale, and position of the keypoints, thus providing invariance to these transformations; S1.4 Keypoint Descriptor, Within the neighborhood of the keypoint, the image gradient is measured at a selected scale to generate a descriptor for the keypoint.

3. The multimodal fusion method for radiotherapy ablation of arrhythmias according to claim 1, characterized in that, The feature detection algorithm is the SURF (Speeded Up Robust Features) feature extraction algorithm, specifically including: S1.1 Constructs a scale space, The scale space of the input image is constructed at different scales using Gaussian blur and downsampling methods; S1.2 Key Point Detection In scale space, keypoints are detected using the determinant and trace of the Hessian matrix; S1.3 Direction Allocation, Calculate the Haar wavelet response around each keypoint and assign a principal direction to the keypoint using a sliding direction window; S1.4 Descriptor generation, The Haar wavelet response is calculated at the scale and orientation of the keypoints, and the descriptor of the keypoints is generated.

4. The multi-modality fusion method for radioablation of cardiac arrhythmia of claim 1, wherein, The feature matching algorithm is a nearest neighbor matching method based on Euclidean distance, specifically including: S2.1 Calculate the eigenvectors, For each keypoint, calculate its feature vector; S2.2 Calculate the distance. For a given keypoint, calculate its Euclidean distance to the feature vectors of all other keypoints; S2.3 Nearest Neighbor Matching Find the key point corresponding to the nearest feature vector and use it as the match for the given key point.

5. The multimodal fusion method for radiotherapy ablation of arrhythmias according to claim 1, characterized in that, The transformation model is an affine transformation, specifically including: 3.1 Select key points, Select three key points that are not on a straight line; 3.2 Calculate the transformation matrix. By solving the system of equations, we can find the affine transformation matrix that maps the key points in the ETV image to the corresponding key points in the STV image. 3.3 Applying transformations, The calculated affine transformation matrix is ​​used to transform the ETV image to obtain an image that matches the STV image.

6. The multimodal fusion method for radioablation of cardiac arrhythmias as claimed in claim 1 wherein, The transformation model is a perspective transformation, specifically including: 3.1 Select key points, Select four key points that are not on a straight line; 3.2 Calculate the transformation matrix. By solving the system of equations, we can find the perspective transformation matrix that can map the key points in the ETV image to the corresponding key points in the STV image; 3.3 Applying transformations, The calculated perspective transformation matrix is ​​used to transform the ETV image to obtain an image that matches the STV image.

Images