Multi-modal fusion image registration quality evaluation method and device applied to medical image

Abstract

The invention discloses a multi-modal fusion image registration quality evaluation method and device applied to a medical image, and remarkably improves the precision, efficiency and clinical applicability of image registration quality evaluation by constructing a multi-modal data dynamic fusion framework and a quantitative evaluation system. Through the improved Sobel operator, the difference weight of diagonal pixels is increased, the center sensitivity is kept, the edge features are accurately extracted, the attention weight is dynamically generated in combination with gating convolution, the problem of cross-domain feature deviation of a KV image and a DRR image in an operation is effectively solved, and the edge alignment precision is greatly improved in complex scenes such as spine deformation and organ displacement. The MLP-Sigmoid regression layer in the system outputs 0-1 standardized scores and synchronously predicts the space offset, an intuitive and explainable evaluation report is provided for clinicians, the decision-making efficiency and the treatment safety are remarkably improved, and reliable technical guarantee is provided for high-precision image guidance of intelligent radiotherapy equipment.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to a method and apparatus for evaluating the quality of multimodal fusion image registration in medical images, particularly for the quantitative evaluation and dynamic optimization of real-time image registration during radiotherapy. Background Technology

[0002] In the field of radiotherapy, image-guided radiation therapy (IGRT) achieves precise localization of the tumor target area through real-time image registration with preoperative planning. However, target deviation caused by intraoperative respiratory movements, organ displacement, or changes in body position (with an average deviation of 3-5 mm according to clinical statistics) seriously threatens the radiotherapy effect. Existing image registration techniques mainly rely on multimodal image fusion, especially the registration of intraoperative two-dimensional X-ray images (Kilovoltage Image, KV) with digitally reconstructed radiographs (DRR) generated by preoperative three-dimensional CT.

[0003] Traditional registration methods rely on rigid registration algorithms based on mutual information (MI), achieving alignment by maximizing the statistical correlation of gray-level distributions between images. However, these methods are sensitive to local deformation, and in the treatment of chest and abdominal areas with significant soft tissue deformation, registration accuracy errors generally exceed 2 mm. To improve adaptability, machine learning techniques have been introduced, such as using support vector machines (SVM) classifiers to perform binary classification evaluation (qualified / unqualified) of registration results. However, these methods rely on manual feature extraction (such as edge gradients and texture statistics), achieving a classification accuracy of less than 75% under complex anatomical structures, and lack a quantitative scoring mechanism, making it difficult to guide clinical decision-making.

[0004] In recent years, deep learning models (such as U-Net) have significantly improved registration efficiency through end-to-end learning of deformation field prediction. However, these methods are limited by feature extraction from a single image modality and do not fully integrate the cross-domain characteristics of DRR and KV images. DRR images are generated by forward projection of CT values, and their grayscale distribution differs significantly from that of real X-ray images, leading to biases in cross-domain feature alignment. Furthermore, existing methods typically separate registration from quality assessment.

[0005] Existing multimodal fusion strategies also have limitations. Most studies simply stitch KV and DRR images together as a dual-channel input, ignoring their heterogeneity in features such as edge details and noise patterns. For example, KV images are susceptible to quantum noise interference due to low-dose imaging (SNR < 20dB), while DRR images are prone to edge artifacts due to projection algorithms. Traditional convolution operations can lead to the loss of more than 25% of detail information during feature extraction. Furthermore, the quantitative indicators relied upon for clinical assessment suffer from a lack of diversity. Existing systems typically use only a single image similarity indicator (such as SSIM) without incorporating multidimensional indicators such as spatial correlation coefficient (SCC) and visual fidelity (VIF), making it difficult to comprehensively reflect the clinical reliability of registration quality. Summary of the Invention

[0006] This invention aims to address the limitations of existing technologies in image registration quality assessment, providing a multimodal fusion image registration quality assessment method and device for medical images. It is a real-time, high-precision assessment system with multimodal fusion capabilities. The core technology of this system lies in combining image features and quantitative indicators to achieve dynamic assessment through a cross-domain feature fusion network. Specifically, the system first acquires 1024×1024 resolution KV and DRR image pairs and generates four types of preprocessed images (including original image, mask image, edge image, and edge mask image). An improved Sobel operator enhances edge response sensitivity, and morphological closing operations are used to eliminate edge breaks. Simultaneously, multiple indicators, including structural similarity (SSIM) and spatial correlation coefficient (SCC), are calculated. Six core indicators are selected as multimodal inputs using the Pearson correlation coefficient.

[0007] The network architecture uses a two-stream ResNet-34 as the base model. An Edge-Guided Attention (EGAM) module is introduced after the fourth convolutional layer. This module generates spatial attention weights by fusing gradient magnitude features with the original features, significantly improving the sensitivity to edge details. A Cross-Domain Attention (CDAM) module further integrates global semantic mapping and local detail features, achieving feature alignment between KV and DRR images through dynamic weight allocation. The system ultimately outputs a 0-1 normalized registration score (CDFRegScore) and jointly predicts spatial errors. The loss function fuses mean squared error and edge consistency loss to ensure high accuracy even in quantum noise environments.

[0008] The innovation of this invention lies in constructing a multimodal data dynamic fusion framework, which solves the problems of feature alignment bias and insufficient utilization of indicators in traditional methods. Through edge-guided attention mechanisms and cross-domain feature alignment technology, it improves the speed of registration quality assessment and provides doctors with intuitively interpretable quantitative reports, offering crucial support for real-time intraoperative adjustments. Specifically, this invention is implemented using the following technical solutions:

[0009] This invention discloses a method for evaluating the quality of multimodal fusion image registration applied to medical imaging, comprising:

[0010] Obtain at least one pair of intraoperative real-time acquired 2D KV images and DRR images generated based on 3D CT anterior projection to form KV / DRR image pair data and corresponding label data;

[0011] Preprocessing the KV / DRR image pairs generates four types of image data, including the original Raw image, the RawMask background cropping image, the Edge image, and the EdgeMask edge background cropping image, resulting in four types of KV / DRR image pairs.

[0012] Calculate 2-20 image similarity indices for four types of KV / DRR image pairs, and select and retain 6 core indices using the Pearson correlation coefficient algorithm and the AutoFeatSelect feature selection algorithm;

[0013] A cross-domain feature fusion network CDFRegNet is constructed, which extracts image features of four types of KV / DRR image pairs through a dual-stream feature encoding architecture. An edge-guided attention module EGAM is introduced after the fourth convolutional layer to enhance image edge features.

[0014] The image features and enhanced image edge features are aligned using the cross-domain attention module CDAM. The aligned image features are then fused with six core indicators across domains to obtain the feature-fused data.

[0015] The data after feature fusion is input into a multilayer perceptron (MLP), and the Sigmoid regression is used to regress the registration offset and calculate the standardized registration score CDFRegScore. The output is a quantitative evaluation result in the range of 0-1.

[0016] As a further improvement, the present invention describes obtaining at least one pair of intraoperatively acquired 2D KV images and DRR images generated based on 3D CT forward projection to form KV / DRR image pair data and corresponding label data, specifically as follows:

[0017] The intraoperative real-time acquisition of a 1024×1024×1 two-dimensional KV image and the incident angle a / b axis of the X-ray when acquiring the KV image;

[0018] The three-dimensional CT image of the preoperative tomographic scan was used to generate a DRR image of 1024×1024×1 based on the three-dimensional CT image and the incident angle.

[0019] Label data: 1, 2, 3, 4, 5. Label 1 means the image registration quality is the best, and so on. Label 5 means the image registration quality is the worst.

[0020] Patient information such as patient mask and treatment process code is recorded simultaneously.

[0021] As a further improvement, the extraction methods for the four types of image pairs described in this invention—Raw original image, RawMask background cropping image, Edge image, and EdgeMask edge background cropping image—are specifically as follows:

[0022] Raw image: Preserves original grayscale information;

[0023] RawMask background cropping: Background bed board noise is removed by threshold segmentation;

[0024] Edge map: extracted using the improved Sobel operator;

[0025] Convolution operation with horizontal gradient (gx) kernel:

[0026]

[0027] The differential calculation of diagonal pixels has been added, enhancing edge response sensitivity.

[0028] Convolution operation using vertical gradient (gy) kernel:

[0029]

[0030] The differential weights of the diagonal pixels were extended;

[0031] The Sobel operator extracts the difference in the middle column by multiplying it by 2, maintaining sensitivity to the central region, while reducing noise interference through diagonal differences of ±1.

[0032] EdgeMask edge background cutting map: Combines morphological closing operation to eliminate edge breaks.

[0033] As a further improvement, the image similarity index described in this invention comprises 10 items, specifically:

[0034] Mean Squared Error (MSE)

[0035] Root Mean Squared Error (RMSE)

[0036] Peak Signal-to-Noise Ratio(PSNR)

[0037] Structural Similarity Index(SSIM)

[0038] Universal Quality Image Index(UQI)

[0039] Neighborhood Difference Coefficient(NDC)

[0040] Erreur Relative Globale Adimensionnellede Synthèse(ERGAS)

[0041] Spatial Correlation Coefficient(SCC)

[0042] Spectral Angle Mapper (SAM)

[0043] Visual Information Fidelity(VIF);

[0044] The six core indicators are retained as follows: S selected ={rawSCC,rawMaskSSIM,edgeMaskVIF,edgeMaskSCC,edgeMaskSSIM,edgeSCC}.

[0045] As a further improvement, the dual-stream feature encoding described in this invention is specifically as follows:

[0046] An improved ResNet-34 is used as the basic architecture, and KV / DRR image features are extracted through two branches respectively;

[0047] Gradient feature formula:

[0048]

[0049] Sobel core S x =[[1,0,-1],[2,0,-2],[1,0,-1]],S y = [[1,2,1],[0,0,0],[-1,-2,-1]];

[0050] The specific mechanism of edge-guided attention is as follows:

[0051] Gradient magnitude G mag Concatenated with the original features, attention weights are generated through gated convolution to enhance edge features. The formula is:

[0052] F out =F in ⊙σ(Conv(G mag +F in )).

[0053] As a further improvement, the alignment of image features and image edge features through the cross-domain attention module CDAM described in this invention specifically involves:

[0054] Global semantic branch: 1x1 convolution is used to compress channels and generate cross-domain semantic mapping;

[0055] Local attention branch: Captures detailed features through 3x3 grouped convolutions;

[0056] The specific fusion formula for cross-domain fusion of aligned image features and six core indicators is as follows:

[0057] Z v =Conv(F KV ,F DRR )

[0058] Z M =LayerNorm(Z) v ⊙W attn +Z v );

[0059] Among them W attn Attention weights are dynamically generated.

[0060] As a further improvement, the present invention inputs the feature-fused data into a multilayer perceptron (MLP), uses Sigmoid regression to regress the registration offset, and standardizes the registration score CDFRegScore, outputting a quantitative evaluation result ranging from 0 to 1, specifically:

[0061] The fused feature data is then used to perform registration offset regression using a multilayer perceptron (MLP) and sigmoid regression. The loss function includes:

[0062] Mean squared error loss:

[0063]

[0064] Edge consistency loss:

[0065]

[0066] Total loss:

[0067] L total =L mse +L edg (λ=0.5)

[0068] Stable convergence was achieved during 1000 training rounds;

[0069] Standardized registration score:

[0070] Using the Sigmoid function ensures that the output value range is 0-1, standardizing the registration score. The closer the result is to 0, the worse the registration quality; the closer it is to 1, the better the registration quality.

[0071] This invention also discloses a method for evaluating the quality of multimodal fusion image registration applied to medical imaging, comprising:

[0072] First acquisition module: acquire at least one pair of intraoperative real-time acquired 2D KV images and DRR images generated based on 3D CT anterior projection, forming KV / DRR image pair data and corresponding label data;

[0073] The first processing module preprocesses the KV / DRR image pairs to generate four types of image data, including the original Raw image, the RawMask background cropping image, the Edge image, and the EdgeMask edge background cropping image, thus obtaining four types of KV / DRR image pairs.

[0074] The first calculation module calculates 2-20 image similarity indicators for four types of KV / DRR image pairs, and selects and retains 6 core indicators through the Pearson correlation coefficient algorithm and the AutoFeatSelect feature selection algorithm.

[0075] The second computation module constructs a cross-domain feature fusion network CDFRegNet, which extracts image features of four types of KV / DRR image pairs through a two-stream feature encoding architecture, and introduces an edge-guided attention module EGAM after the fourth convolutional layer to enhance image edge features.

[0076] The third computation module: The cross-domain attention module CDAM is used to align image features and enhanced image edge features, and the aligned image features are fused with 6 core indicators across domains to obtain the feature fusion data.

[0077] The fourth calculation module inputs the feature-fused data into a multilayer perceptron (MLP), uses Sigmoid regression to determine the registration offset, calculates the standardized registration score CDFRegScore, and outputs a quantitative evaluation result in the range of 0-1.

[0078] The beneficial effects of this invention are as follows:

[0079] This invention significantly improves the accuracy, efficiency, and clinical applicability of image registration quality assessment by constructing a multimodal data dynamic fusion framework and quantitative evaluation system. It pioneers an Edge-Guided Attention (EGAM) mechanism, which, through an improved Sobel operator, increases the differential weight of diagonal pixels while maintaining center sensitivity, accurately extracting edge features. Combined with gated convolution to dynamically generate attention weights, it effectively solves the cross-domain feature deviation problem between intraoperative KV images and DRR images, significantly improving edge alignment accuracy in complex scenarios such as spinal deformation and organ displacement. The Cross-Domain Attention (CDAM) module innovatively integrates global semantic branches and local attention branches to achieve multi-scale feature alignment, significantly reducing the spatial dispersion of registration errors and effectively solving the cross-domain feature deviation problem between intraoperative real-time images and preoperative reconstructed images, enhancing sensitivity to detailed features. Simultaneously, through joint analysis and dynamic weight allocation of multi-dimensional similarity indicators, it establishes a comprehensive quantitative evaluation standard covering structural, spatial, and visual fidelity, breaking through the limitations of traditional single-indicator evaluation and proposing a "four-image, six-indicator" fusion framework. The system employs four image input categories: Raw (preserving grayscale information), RawMask (background noise removal), Edge (improved Sobel edge map), and EdgeMask (closing operation to repair broken edges), comprehensively covering structural, noise, and edge information. The selection of six core metrics is based on Pearson correlation coefficient and the AutoFeatSelect algorithm, scientifically selecting six optimal combinations (rawSCC / rawMaskSSIM / edgeMaskVIF, etc.) from 10 candidate metrics (SCC / SSIM / VIF, etc.), reducing metric redundancy and improving evaluation efficiency. A cross-domain fusion formula dynamically allocates metric weights, achieving joint optimization of structural fidelity, spatial consistency, and visual quality. The system provides end-to-end real-time feedback, employing a dual loss function design coupled with pixel matching and edge consistency constraints, achieving stable convergence within 1000 training rounds, significantly reducing evaluation latency, meeting intraoperative real-time feedback requirements, and supporting dynamic adjustment of the radiation field by radiotherapy equipment. In addition, the MLP-Sigmoid regression layer in the system outputs a 0-1 standardized score and simultaneously predicts spatial offset, providing clinicians with an intuitive and interpretable assessment report, significantly improving decision-making efficiency and treatment safety. At the same time, it demonstrates strong robustness in complex noise interference and anatomical deformation scenarios, providing reliable technical support for high-precision image guidance of intelligent radiotherapy equipment. Attached Figure Description

[0080] Figure 1 This is a data flow diagram of the present invention. Detailed Implementation

[0081] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0082] The purpose of this invention is to address the problem of insufficient multimodal feature fusion in medical image registration quality assessment. It proposes an assessment method that combines quantitative indicators with deep learning, and improves assessment accuracy through cross-domain feature alignment and core indicator screening. Figure 1 This is the data flow diagram of the present invention:

[0083] The specific implementation method of the present invention is as follows:

[0084] Step 1: Obtain at least one pair of intraoperative real-time acquired 2D KV images and DRR images generated based on 3D CT forward projection to form KV / DRR image pair data and corresponding label data.

[0085] The intraoperative real-time acquisition of a 1024×1024×1 two-dimensional KV image and the incident angle a / b axis of the X-ray when acquiring the KV image;

[0086] The three-dimensional CT image of the preoperative tomographic scan was used to generate a DRR image of 1024×1024×1 based on the three-dimensional CT image and the incident angle.

[0087] Label data: 1, 2, 3, 4, 5. Label 1 means the image registration quality is the best, and so on. Label 5 means the image registration quality is the worst.

[0088] Patient information such as patient mask and treatment process code is recorded simultaneously.

[0089] In this embodiment, more than 2,000 sets of clinical case information were obtained. Each set contains a pair of KV / DRR images and corresponding labels, and a total of 2,000 sets of training data and 500 sets of test data were constructed.

[0090] Step 2: Preprocess the KV / DRR image pairs to generate four types of image data, including the original Raw image, the RawMask background cutout image, the Edge image, and the EdgeMask edge background cutout image, thus obtaining four types of KV / DRR image pairs.

[0091] Raw image: Preserves original grayscale information;

[0092] RawMask background cropping: Background bed board noise is removed by threshold segmentation;

[0093] Edge map: extracted using the improved Sobel operator;

[0094] Convolution operation with horizontal gradient (gx) kernel:

[0095]

[0096] The differential calculation of diagonal pixels has been added, enhancing edge response sensitivity.

[0097] Convolution operation using vertical gradient (gy) kernel:

[0098]

[0099] The differential weights of the diagonal pixels were extended;

[0100] The Sobel operator extraction multiplies the difference in the middle column by 2, maintaining sensitivity to the central region, while reducing noise interference through ±1 diagonal difference;

[0101] EdgeMask edge background cutting map: Combines morphological closing operation to eliminate edge breaks.

[0102] In this embodiment, more than two thousand clinical case KV / DRR image pairs were generated for each image pair, and four different preprocessed images were generated. However, due to the different incident angles of the KV images, not all preprocessed images were of good quality, and further data analysis and processing were required.

[0103] Step 3: Calculate 2-20 image similarity indices for the four types of KV / DRR image pairs, and then use the Pearson correlation coefficient algorithm and the AutoFeatSelect feature selection algorithm to select and retain 6 core indices. The image similarity indices consist of 10 items, specifically:

[0104] Mean Squared Error (MSE)

[0105] Root Mean Squared Error (RMSE)

[0106] Peak Signal-to-Noise Ratio(PSNR)

[0107] Structural Similarity Index(SSIM)

[0108] Universal Quality Image Index(UQI)

[0109] Neighborhood Difference Coefficient(NDC)

[0110] Erreur Relative Globale Adimensionnellede Synthèse(ERGAS)

[0111] Spatial Correlation Coefficient(SCC)

[0112] Spectral Angle Mapper (SAM)

[0113] Visual Information Fidelity(VIF);

[0114] The six core indicators to be retained are: S selected ={rawSCC,rawMaskSSIM,

[0115] edgeMaskVIF,edgeMaskSCC,edgeMaskSSIM,edgeSCC}.

[0116] In this embodiment, 11 image similarity indices were calculated for the four types of preprocessed image pairs generated from each group of KV / DRR image pairs, as shown in Table 1 below;

[0117] Table 1. Correlation Coefficients between Image Similarity Index Coefficients and Labels

[0118]

[0119] Using the Python built-in library Corr Pearson correlation coefficient, all image similarity metrics less than 0.2 were filtered out, leaving 14 image similarity metrics. The AutoFeatSelect feature selection algorithm was then used for 16383 training iterations. The AutoFeatSelect algorithm ultimately selected the optimal core image similarity metric, S. selected ={rawSCC,rawMaskSSIM,edgeMaskVIF,edgeMaskSCC,edgeMaskSSIM,edgeSCC}.

[0120] Step 4: Construct the cross-domain feature fusion network CDFRegNet. Extract image features of four types of KV / DRR image pairs through a dual-stream feature encoding architecture. After the fourth convolutional layer, introduce the edge-guided attention module EGAM to enhance the image edge features.

[0121] The dual-stream feature encoding specifically refers to:

[0122] An improved ResNet-34 is used as the basic architecture, and KV / DRR image features are extracted through two branches respectively;

[0123] The gradient feature formula is as follows:

[0124]

[0125] Sobel core S x=[[1,0,-1],[2,0,-2],[1,0,-1]],S y = [[1,2,1],[0,0,0],[-1,-2,-1]];

[0126] The edge-guided attention mechanism is specifically as follows:

[0127] Gradient magnitude G mag Concatenated with the original features, attention weights are generated through gated convolution to enhance edge features. The formula is:

[0128] F out =F in ⊙σ(Conv(G mag +F in ))

[0129] The model architecture code in this embodiment is built on an improved ResNet34 framework. Its data processing flow is divided into three stages: input preprocessing, feature extraction, and dual-stream output. The input single-channel image first undergoes preliminary feature mapping through a modified 7×7 convolutional layer (channel count 1→64), followed by batch normalization, ReLU activation, and max pooling for preprocessing. The feature extraction stage sequentially passes through four residual modules (layer 1-layer 4). The first three residual modules maintain the standard ResNet34 structure, while the fourth residual module (layer 4) is structurally expanded into a composite containing two sub-modules—first performing basic residual calculations (containing 3 BasicBlocks), then connecting to an edge-guided attention module. This module... A Sobel convolution kernel with fixed parameters calculates the gradient magnitude of the feature map. The original features and the gradient features are concatenated and then subjected to 3×3 convolution, group normalization, and Sigmoid activation to generate a spatial attention map. Finally, the feature map and the attention map are multiplied pointwise to achieve feature calibration. The network ends with a dual-path output mechanism. The main path converts the feature tensor into an image semantic vector through global average pooling, while the auxiliary path directly outputs a 512-channel feature map enhanced by edge attention, forming a feature expression system that combines global semantics and local edge details, and finally generating image features of KV / DRR image pairs.

[0130] Step 5: Align image features and enhanced image edge features using the Cross-Domain Attention Module (CDAM). Then, fuse the aligned image features with the six core indicators across domains to obtain the feature-fused data.

[0131] The specific method for aligning image features and image edge features using the cross-domain attention module CDAM is as follows:

[0132] Global semantic branch: 1x1 convolution is used to compress channels and generate cross-domain semantic mapping;

[0133] Local attention branch: Captures detailed features through 3x3 grouped convolutions;

[0134] The specific fusion formula for cross-domain fusion of aligned image features and six core indicators is as follows:

[0135] Z v =Conv(F KV ,F DRR )

[0136] Z M =LayerNorm(Z) v ⊙W attn +Z v );

[0137] Among them W attn Attention weights are dynamically generated.

[0138] In this embodiment, the image features extracted from the KV / DRR image pairs are processed using nn.Linear to handle image edge features, and an attention mechanism is constructed using nn.Sequential. The system receives global feature vectors and index feature vectors from the two images, performs multi-source feature interaction and fusion calculations, and generates a joint representation vector.

[0139] Step 6: Input the feature-fused data into a multilayer perceptron (MLP), use Sigmoid regression to regress the registration offset and calculate the standardized registration score CDFRegScore, outputting a quantitative evaluation result in the range of 0-1.

[0140] The fused feature data is then used to perform registration offset regression using a multilayer perceptron (MLP) and sigmoid regression. The loss function includes:

[0141] Mean squared error loss:

[0142]

[0143] Edge consistency loss:

[0144]

[0145] Total loss:

[0146] L total =L mse +L edge (λ=0.5)

[0147] Stable convergence was achieved during 1000 training rounds;

[0148] Standardized registration score:

[0149] Using the Sigmoid function ensures the output value range is 0-1, standardizing the registration score. A result closer to 0 indicates poorer registration quality, while a result closer to 1 indicates better registration quality.

[0150] In this embodiment, the fused features are passed through a regression prediction head that includes layer normalization and Dropout regularization. The standardized prediction value is then output through the Sigmoid activation function, while the edge feature maps generated during the two image encoding processes are retained as auxiliary outputs. The entire data processing flow forms a cascaded structure of "dual image encoding - index encoding - feature fusion - regression output", realizing end-to-end cross-modal feature collaborative computation.

[0151] This invention also discloses a multimodal fusion image registration quality assessment device for medical imaging, comprising:

[0152] First acquisition module: acquire at least one pair of intraoperative real-time acquired 2D KV images and DRR images generated based on 3D CT anterior projection, forming KV / DRR image pair data and corresponding label data;

[0153] The first processing module preprocesses the KV / DRR image pairs to generate four types of image data, including the original Raw image, the RawMask background cropping image, the Edge image, and the EdgeMask edge background cropping image, thus obtaining four types of KV / DRR image pairs.

[0154] The first calculation module calculates 2-20 image similarity indicators for four types of KV / DRR image pairs, and selects and retains 6 core indicators through the Pearson correlation coefficient algorithm and the AutoFeatSelect feature selection algorithm.

[0155] The second computation module constructs a cross-domain feature fusion network CDFRegNet, which extracts image features of four types of KV / DRR image pairs through a two-stream feature encoding architecture, and introduces an edge-guided attention module EGAM after the fourth convolutional layer to enhance image edge features.

[0156] The third computation module: The cross-domain attention module CDAM is used to align image features and enhanced image edge features, and the aligned image features are fused with 6 core indicators across domains to obtain the feature fusion data.

[0157] The fourth calculation module inputs the feature-fused data into a multilayer perceptron (MLP), uses Sigmoid regression to determine the registration offset, calculates the standardized registration score CDFRegScore, and outputs a quantitative evaluation result in the range of 0-1.

[0158] The above description is not intended to limit the present invention. It should be noted that, for those skilled in the art, various changes, modifications, additions or substitutions can be made without departing from the essential scope of the present invention, and these improvements and refinements should also be considered within the scope of protection of the present invention.

Claims

1. A method for evaluating the quality of multimodal fusion image registration applied to medical imaging, characterized in that, include: Obtain at least one pair of intraoperative real-time acquired 2D KV images and DRR images generated based on 3D CT anterior projection to form KV / DRR image pair data and corresponding label data; Preprocessing the KV / DRR image pairs generates four types of image data, including the original Raw image, the RawMask background cropping image, the Edge image, and the EdgeMask edge background cropping image, resulting in four types of KV / DRR image pairs. Calculate 2-20 image similarity indices for four types of KV / DRR image pairs, and select and retain 6 core indices using the Pearson correlation coefficient algorithm and the AutoFeatSelect feature selection algorithm; A cross-domain feature fusion network CDFRegNet is constructed, which extracts image features of four types of KV / DRR image pairs through a dual-stream feature encoding architecture. An edge-guided attention module EGAM is introduced after the fourth convolutional layer to enhance image edge features. The image features and enhanced image edge features are aligned using the cross-domain attention module CDAM. The aligned image features are then fused with six core indicators across domains to obtain the feature-fused data. The data after feature fusion is input into a multilayer perceptron (MLP), and the Sigmoid regression is used to regress the registration offset and calculate the standardized registration score CDFRegScore. The output is a quantitative evaluation result in the range of 0-1.

2. The multimodal fusion image registration quality assessment method for medical imaging according to claim 1, characterized in that, The process of obtaining at least one pair of intraoperatively acquired 2D KV images and DRR images generated based on 3D CT anterior projection to form KV / DRR image pair data and corresponding label data specifically involves: The intraoperative real-time acquisition of a 1024×1024×1 two-dimensional KV image and the incident angle a / b axis of the X-ray when acquiring the KV image; The three-dimensional CT image of the preoperative tomographic scan was used to generate a DRR image of 1024×1024×1 based on the three-dimensional CT image and the incident angle. Label data: 1, 2, 3, 4, 5. Label 1 means the image registration quality is the best, and so on. Label 5 means the image registration quality is the worst. Patient information such as patient mask and treatment process code is recorded simultaneously.

3. The multimodal fusion image registration quality assessment method for medical imaging according to claim 1 or 2, characterized in that, The extraction methods for the four types of image pairs—Raw original image, RawMask background cropping image, Edge image, and EdgeMask edge background cropping image—are as follows: Raw image: Preserves original grayscale information; RawMask background cropping: Background bed board noise is removed by threshold segmentation; Edge map: extracted using the improved Sobel operator; Convolution operation with horizontal gradient (gx) kernel: The differential calculation of diagonal pixels has been added, enhancing edge response sensitivity. Convolution operation using vertical gradient (gy) kernel: The differential weights of the diagonal pixels were extended; The Sobel operator extraction multiplies the difference in the middle column by 2, maintaining sensitivity to the central region, while reducing noise interference through ±1 diagonal difference; EdgeMask edge background cutting map: Combines morphological closing operation to eliminate edge breaks.

4. The multimodal fusion image registration quality assessment method for medical imaging according to claim 3, characterized in that, The image similarity index comprises 10 items, specifically: Mean Squared Error (MSE) Root Mean Squared Error (RMSE) Peak Signal-to-Noise Ratio(PSNR) Structural Similarity Index(SSIM) Universal Quality Image Index(UQI) Neighborhood Difference Coefficient(NDC) Erreur Relative Globale Adimensionnelle de Synthèse(ERGAS) Spatial Correlation Coefficient(SCC) Spectral Angle Mapper (SAM) Visual Information Fidelity(VIF); The six core indicators to be retained are: S selected ={rawSCC,rawMaskSSIM,edgeMaskVIF,edgeMaskSCC,edgeMaskSSIM,edgeSCC}.

5. The multimodal fusion image registration quality assessment method for medical imaging according to claim 4, characterized in that, The dual-stream feature encoding specifically refers to: An improved ResNet-34 is used as the basic architecture, and KV / DRR image features are extracted through two branches respectively; The gradient feature formula is as follows: Sobel core S x =[[1,0,-1],[2,0,-2],[1,0,-1]],S y =[[1,2,1],[0,0,0],[-1,-2,-1]]; The edge-guided attention mechanism is specifically as follows: Gradient magnitude G mag Concatenated with the original features, attention weights are generated through gated convolution to enhance edge features. The formula is: F out =F in ⊙σ(Conv(G mag +F in ))。 6. The multimodal fusion image registration quality assessment method for medical imaging according to claim 1, 2, 4, or 5, characterized in that, The specific method for aligning image features and image edge features using the cross-domain attention module CDAM is as follows: Global semantic branch: 1x1 convolution is used to compress channels and generate cross-domain semantic mapping; Local attention branch: Captures detailed features through 3x3 grouped convolutions; The specific fusion formula for cross-domain fusion of aligned image features and six core indicators is as follows: Z v =Conv(F KV ,F DRR ) WITH M =LayerNorm(Z v ⊙W attn +Z v ); Among them W attn Attention weights are dynamically generated.

7. The multimodal fusion image registration quality assessment method for medical imaging according to claim 6, characterized in that, The process involves inputting the feature-fused data into a multilayer perceptron (MLP), using sigmoid regression to regress the registration offset, and standardizing the registration score (CDFRegScore). The output is a quantitative evaluation result ranging from 0 to 1. The fused feature data is then used to perform registration offset regression using a multilayer perceptron (MLP) and sigmoid regression. The loss function includes: Mean squared error loss: Edge consistency loss: Total loss: L total L mse +L edge (λ<0.5) Stable convergence was achieved during 1000 training rounds; Standardized registration score: Using the Sigmoid function ensures that the output value range is 0-1, standardizing the registration score. The closer the result is to 0, the worse the registration quality; the closer it is to 1, the better the registration quality.

8. A method for evaluating the quality of multimodal fusion image registration applied to medical imaging, characterized in that, include: First acquisition module: acquire at least one pair of intraoperative real-time acquired 2D KV images and DRR images generated based on 3D CT anterior projection, forming KV / DRR image pair data and corresponding label data; The first processing module preprocesses the KV / DRR image pairs to generate four types of image data, including the original Raw image, the RawMask background cropping image, the Edge image, and the EdgeMask edge background cropping image, thus obtaining four types of KV / DRR image pairs. The first calculation module calculates 2-20 image similarity indicators for four types of KV / DRR image pairs, and selects and retains 6 core indicators through the Pearson correlation coefficient algorithm and the AutoFeatSelect feature selection algorithm. The second computation module constructs a cross-domain feature fusion network CDFRegNet, which extracts image features of four types of KV / DRR image pairs through a two-stream feature encoding architecture, and introduces an edge-guided attention module EGAM after the fourth convolutional layer to enhance image edge features. The third computation module: The cross-domain attention module CDAM is used to align image features and enhanced image edge features, and the aligned image features are fused with 6 core indicators across domains to obtain the feature fusion data. The fourth calculation module inputs the feature-fused data into a multilayer perceptron (MLP), uses Sigmoid regression to determine the registration offset, calculates the standardized registration score CDFRegScore, and outputs a quantitative evaluation result in the range of 0-1.

Images