Multi-modal image registration fusion algorithm

Abstract

The invention discloses a registration fusion algorithm of a multi-modal image, which relates to the technical field of imaging and comprises the following steps: S1, data preprocessing-motion compensation; according to the method, displacement and deformation caused by physiological movement are inhibited through data preprocessing; the problem of feature extraction in a complex scene is solved through multi-level feature extraction, and uncertainty is reduced; the matching efficiency and precision are improved based on local sensitive hashing and mutual information feature matching; accurately describing non-rigid deformation based on a free deformation model of a B-spline; in the registration fusion link, images are accurately aligned based on a deformation model, and a weighted average fusion strategy is adopted; the registration fusion quality is ensured by a multi-dimensional evaluation system; the steps cooperate with each other to solve the problems of uncertainty of feature extraction and matching in local deformation and non-rigid registration, difficulty in feature extraction caused by local deformation, matching error caused by non-rigid deformation, limitation of non-rigid registration effect by inaccurate feature extraction, and interference of deformation estimation by matching uncertainty in the existing algorithm.

Description

Technical Field

[0001] The present invention relates to the field of imaging technology, and in particular to a multi-modal image registration and fusion algorithm. Background Art

[0002] The multimodal image registration and fusion algorithm is a series of computational methods used to accurately align (register) and merge (fuse) images from different imaging modalities (such as CT, MRI, PET, etc.). Its purpose is to integrate the advantages of different modal images to provide more comprehensive and accurate image data for medical applications such as diagnosis, treatment planning, or other related fields.

[0003] The existing multimodal image registration and fusion algorithms have the following problems:

[0004] Local deformation and non-rigid registration problem: The tissues and organs of the human body or objects may undergo local deformation or movement, such as breathing movement, heartbeat, etc., which will cause local non-rigid deformation between multimodal images. Existing registration algorithms often have difficulties in dealing with such non-rigid deformations, and it is difficult to accurately register the deformed areas, thus affecting the accuracy of the fusion results. Especially in the medical field, the non-rigid registration problem is particularly prominent for the registration and fusion of images of the chest, abdomen and other parts.

[0005] Uncertainty in feature extraction and matching: Feature-based registration algorithms rely on the extraction and matching of image features. However, in complex scenes, image features may not be obvious or may be similar, which increases the uncertainty in feature extraction and matching. For example, in remote sensing images, for areas with complex terrain and similar features, feature extraction and matching are more difficult, and mismatching is prone to occur, which in turn affects the accuracy of registration.

[0006] Local deformation makes feature extraction difficult: In the image registration of human tissues and organs, due to local deformations such as breathing movement and heartbeat, the position and shape of features in the image are constantly changing. For example, in chest images, features such as the edge of the lungs will change dynamically with breathing. This makes it difficult for feature extraction algorithms to capture stable and representative features, which in turn increases the uncertainty of feature extraction.

[0007] Non-rigid deformation causes matching errors: Local non-rigid deformation makes the feature correspondence of the same object in different modal images complicated. For example, in medical image fusion, the deformed organ features may mismatch similar features at different times or in different modalities. In remote sensing images, complex terrain is deformed by natural factors, and similar local changes can also make feature matching difficult, making it easy to mismatch similar features after deformation, affecting registration accuracy.

[0008] Inaccurate feature extraction limits the effectiveness of non-rigid registration: If feature extraction is inaccurate or uncertain, the feature points used to estimate non-rigid deformation are unreliable. For example, if the feature points of the area affected by the beating heart cannot be accurately extracted, it is difficult to construct an accurate deformation field to describe the non-rigid deformation of the heart area, thus limiting the accuracy of the non-rigid registration algorithm.

[0009] Matching uncertainty interferes with deformation estimation: The uncertainty of feature matching interferes with the estimation of the degree and direction of local deformation. In remote sensing images, mismatched features can cause the calculated deformation of objects to deviate from the actual deformation. Similarly, in medical images, incorrect feature matching can lead to incorrect estimation of deformation such as respiratory movement or organ peristalsis, thus affecting the quality of non-rigid registration.

[0010] In view of this, a multimodal image registration and fusion algorithm is provided to overcome the above problems. Summary of the invention

[0011] The purpose of the present invention is to provide a multi-modal image registration and fusion algorithm to solve the problems raised in the above background technology.

[0012] In order to solve the above technical problems, the present invention provides a multimodal image registration and fusion algorithm, comprising the following steps:

[0013] S1. Data preprocessing-motion compensation: introducing source correction strategy;

[0014] S2. Feature extraction - multi-level feature extraction: Constructing a high-dimensional stable feature space, including the following steps:

[0015] Multi-scale features based on wavelet transform: Open up multi-dimensional insights into image features:

[0016] Perform wavelet decomposition on the preprocessed image I(x,y) and obtain the wavelet coefficients W at different scales j and directions k. j,k (x,y);

[0017] Harris-Laplace feature point detection: Locking stable feature points in complex deformations:

[0018] Calculate the Harris corner response function at each scale j:

[0019] R Harris (x,y,j)=det(M(x,y,j))-k(trace(M(x,y,j))) 2 ;

[0020] Where M(x,y,j) is the autocorrelation matrix at scale j, combined with the Laplace operator response:

[0021]

[0022] By comprehensively considering these two responses and selecting R Harris (x,y,j) and R Laplace Points where (x, y, j) all meet the threshold conditions are considered feature points;

[0023] S3, Feature Matching - Matching based on Locality Sensitive Hashing and Mutual Information: Introducing a new matching paradigm;

[0024] S4. Non-rigid deformation modeling - B-spline based free deformation model: Sculpting local deformation, including the following steps:

[0025] Let the matching feature point pair be {(p i ,q i )}, define a uniform 3D B-spline grid G(x,y,z) in image space, with control points C ijk . By minimizing the energy function:

[0026]

[0027] To solve the displacement of the control point, the first term is the distance error of the matching point pair, the second term is the regularization term, and λ is the weight parameter. is the Laplace operator;

[0028] S5, Registration Fusion - Registration and weighted average fusion based on deformation model; used to adapt to the fusion of complex scenes.

[0029] Furthermore, in S1, the original medical image is represented as I(x, y, t); where x, y are spatial coordinates and t is time; this algorithm uses high-precision acquisition of respiratory gating signal r(t) and ECG gating signal c(t), and uses affine transformation T r (x,y,t) and T c (x, y, t) performs coarse registration, the formula is:

[0030]

[0031] Furthermore, in S3, it is assumed that the feature point sets extracted from the two modal images are P = {p i} and Q = {q i}, the feature description vectors are E P ={p i} and E Q = {q i}; This algorithm uses local sensitive hashing to map the feature description vector into multiple hash buckets; for each point p in P i, can quickly locate the candidate matching points in the corresponding hash bucket of Q, and on this basis, calculate the mutual information between the candidate matching points:

[0032]

[0033] Where p(a,b) is the point p i and q j The joint probability distribution of the eigenvalues ​​of p(a) and p(b) are their respective marginal probability distributions. The point pairs with the largest mutual information are selected as matching point pairs based on mutual information, which is used to essentially reduce the occurrence of mismatches.

[0034] Further, in S5, according to the solved B-spline deformation model, the image I1(x, y) of one modality is transformed into the space of the image I2(x, y) of another modality;

[0035] Fusion strategy:

[0036] Assume that the registered two-modality image is I 1,aligned (x,y) and I2(x,y), fused image I fusion (x,y) uses weighted average method:

[0037] I fusion (x,y)=w1I 1,aligned (x,y)+(w2I2(x,y));

[0038] Where w1+w2=1, and the weights w1 and w2 are dynamically adjusted according to the modal characteristics of the image and the reliability of the local area.

[0039] Furthermore, the method further comprises the following steps:

[0040] S6. Multi-dimensional evaluation: used to ensure the quality of registration and fusion.

[0041] Furthermore, in S6, two quantitative indicators, mean square error and normalized cross-correlation, are used, and the formula is as follows:

[0042] Mean Square Error:

[0043] Normalized cross-correlation:

[0044]

[0045] Where N is the total number of image pixels, and are the mean values ​​of the registered images.

[0046] Furthermore, the position change rate of the key feature points and the similarity of the feature description vectors are calculated at different time points and in different modes.

[0047] Furthermore, the similarity of deformation patterns in images of different modalities is analyzed for organ movement in medical images and terrain changes in remote sensing images; it is measured by calculating the correlation of texture and shape changes in the deformed area, as well as the consistency of the direction and amplitude of the deformation.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] Data preprocessing provides stable input for non-rigid deformation modeling and registration fusion. Non-rigid deformation modeling provides a key deformation model for registration fusion to accurately align images. The three are progressive and work together to overcome the difficulties of local deformation and non-rigid registration. They effectively deal with the non-rigid deformation problems caused by local deformation and movement of human or object tissues and organs, and significantly improve the registration and fusion accuracy in the chest, abdomen and other parts of the medical field, as well as in complex terrain in remote sensing images, making the fusion results more accurate and reliable to meet the needs of actual applications.

[0050] Feature extraction provides rich and accurate feature information for feature matching, and feature matching performs efficient and high-precision matching operations based on the extracted features. The two work closely together to reduce the uncertainty of feature extraction and matching; improve the stability and comprehensiveness of feature extraction, improve the efficiency and accuracy of feature matching, and reduce mismatches caused by unclear features, similar features, etc. in complex scenes, thereby enhancing the accuracy and reliability of the entire registration and fusion algorithm, so that it can better adapt to the registration and fusion needs of different modal images in complex environments.

[0051] Feature extraction - Multi-level feature extraction provides stable and reliable feature point input for subsequent feature matching, non-rigid deformation modeling, and registration and fusion steps, ensuring that subsequent links can work based on accurate feature information and guaranteeing the quality of the entire registration and fusion algorithm. It solves the difficulty of feature extraction caused by local deformation, and enables the feature extraction algorithm to obtain stable and effective features in complex situations with local deformation, laying a good foundation for the entire multimodal image registration and fusion process and improving the quality of the final fusion result.

[0052] Feature matching provides accurate matching point pairs for non-rigid deformation modeling. Non-rigid deformation modeling accurately describes the deformation based on the matching results, which in turn assists feature matching to more accurately associate features. The two promote each other and reduce the matching error caused by non-rigid deformation. They improve the accuracy of feature matching, reduce feature mismatches caused by local non-rigid deformation, and enable the features of different modal images to correspond correctly, thereby improving the accuracy of registration and fusion, and ensuring that the algorithm can obtain high-quality registration and fusion results when processing multi-modal images with non-rigid deformation.

[0053] Feature extraction provides key feature information for non-rigid deformation modeling. Non-rigid deformation modeling builds a model based on accurate feature information to achieve accurate registration. The two work together to ensure that the effect of non-rigid registration is not limited by inaccurate feature extraction. By providing accurate and comprehensive feature information, non-rigid deformation modeling can build an accurate deformation model based on reliable feature points and rich texture edge information, thereby improving the effect of non-rigid registration, ensuring that the entire registration fusion algorithm can accurately handle the non-rigid deformation between multimodal images, and improving the quality of the final fusion result.

[0054] The accuracy of feature matching directly affects the estimation of deformation in non-rigid deformation modeling, and non-rigid deformation modeling provides guarantee for registration and fusion based on accurate deformation estimation. The three are interrelated to ensure that the entire process is not disturbed by matching uncertainty; improving the accuracy of deformation estimation enables non-rigid deformation modeling to truly reflect local deformation based on accurate feature matching results, thereby ensuring that the registration and fusion process can operate based on accurate deformation information, improving the performance of the entire multimodal image registration and fusion algorithm when dealing with local deformation and the quality of the final fusion results. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a schematic diagram of a multi-modal image registration and fusion algorithm of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] See also Figure 1 , the present invention provides a technical solution:

[0058] See also Figure 1 As shown, an embodiment of a multi-modal image registration and fusion algorithm:

[0059] The following steps are involved:

[0060] S1. Data preprocessing-motion compensation: introducing source correction strategy;

[0061] In the field of multimodal image processing, especially in medical image processing, local deformation caused by respiratory motion and heartbeat is a major challenge affecting the accuracy of image registration and fusion. The original medical image is represented as I(x,y,t); where x,y are spatial coordinates and t is time;

[0062] Traditional preprocessing methods are often just simple position adjustments, which cannot effectively handle local non-rigid deformation and overall displacement caused by physiological movements such as breathing and heartbeat. This not only complicates the subsequent registration and fusion process, but more importantly, seriously affects the accuracy of registration and fusion.

[0063] This algorithm uses high-precision acquisition of respiratory gating signal r(t) and ECG gating signal c(t), and uses affine transformation T r (x,y,t) and T c (x, y, t) performs coarse registration, the formula is:

[0064]

[0065] This preprocessing method that deeply integrates human physiological signals has significant advantages.

[0066] Relevance and necessity to other steps:

[0067] From the perspective of the entire algorithm flow, this preprocessing step is the cornerstone for the success of all subsequent steps. First, it greatly suppresses the overall displacement caused by physiological movement, providing a relatively stable image basis for the feature extraction process. Without this step, it will be difficult to capture accurate and stable features during subsequent feature extraction due to the severe jitter and displacement of the image. For example, during the beating of the heart, if the image displacement is too large, the multi-scale feature extraction based on wavelet transform may extract fuzzy and inaccurate texture and edge features, and the improved Harris-Laplace feature point detection will also fail due to excessive drift of the feature points.

[0068] Secondly, reducing the adverse effects of local deformation on the overall registration and fusion effect is crucial for the subsequent non-rigid deformation modeling and registration and fusion steps. If the local deformation is not effectively corrected in the early stage, the difficulty of non-rigid deformation modeling in dealing with complex deformations will increase exponentially, which may cause the model to be unable to accurately describe the deformation details, thereby affecting the accuracy of registration and fusion. In the registration and fusion process, inaccurate preprocessing results will prevent the registration and weighted average fusion strategies based on deformation models from fully exerting their advantages, because they are all based on the accurate foundation of the previous data.

[0069] In terms of remote sensing images, the traditional rough adjustment method cannot meet the position deviation problem caused by changes in the geographical environment and differences in imaging time. This algorithm performs a preliminary translation correction of depth based on geographic coordinate information. This operation is based on the overall consideration of the subsequent processing flow of remote sensing images.

[0070] This correction creates favorable conditions for the subsequent feature extraction process. The accurate initial spatial relationship enables a clearer distinction between different land features during feature extraction, especially in areas with complex terrain and similar features. For example, for remote sensing images of mountainous and hilly areas, accurate position correction helps to more accurately capture terrain undulations and land boundary features in wavelet transform multi-scale feature extraction, and also allows Harris-Laplace feature point detection to stably extract feature points in scenes with complex land morphological changes.

[0071] For the subsequent feature matching process, accurate initial image state can reduce the possibility of mismatching due to position deviation, thereby improving matching efficiency and accuracy. In the process of non-rigid deformation modeling and registration fusion, accurate initial spatial relationship ensures that the entire process can be based on reliable data, avoiding errors in complex terrain deformation processing and fusion of different band images due to early position deviation, and improving the success rate of the entire algorithm in remote sensing image registration fusion.

[0072] S2. Feature extraction - multi-level feature extraction: Constructing a high-dimensional stable feature space, including the following steps:

[0073] Multi-scale features based on wavelet transform: Open up multi-dimensional insights into image features:

[0074] Perform wavelet decomposition on the preprocessed image I(x,y) and obtain the wavelet coefficients W at different scales j and directions k. j,k (x, y); This process is fundamentally different from most existing algorithms, most of which only focus on a single scale or common features and perform poorly in complex scenarios.

[0075] In medical images, subtle texture differences at the cellular level are difficult to capture, which will lead to the loss of feature information and affect the accuracy of subsequent registration and fusion. In remote sensing images, the ups and downs of complex terrain and the subtle changes in the boundaries of objects cannot be clearly presented, which will also bring difficulties to registration and fusion.

[0076] The wavelet coefficients extracted by this algorithm can comprehensively and meticulously depict the texture and edge features of each layer of the image, forming a rich multi-dimensional image feature "atlas". This provides rich information for subsequent feature matching and deformation analysis, and is an important guarantee for accurate registration and fusion.

[0077] Relevance and necessity to other steps:

[0078] These multi-dimensional wavelet coefficient features provide a more comprehensive information basis for feature matching. In the matching process based on local sensitive hashing (LSH) and mutual information, rich feature information helps to locate candidate matching points more accurately and improve matching accuracy. This is because wavelet coefficients of different scales and directions can more finely distinguish features in different modal images and reduce mismatches caused by feature similarity.

[0079] For non-rigid deformation modeling, the detailed texture and edge information provided by wavelet coefficients helps to better understand the deformation of the image in the local area. For example, when dealing with complex local deformations such as organ peristalsis in medical images or landslides in remote sensing images, this information can help the B-spline-based free deformation model (FFD) to more accurately determine the displacement of control points, thereby more delicately depicting the details of local non-rigid deformation.

[0080] In the registration and fusion process, the multi-dimensional feature "atlas" can make the registration and weighted average fusion strategies based on the deformation model more targeted. In the registration process, features of different scales help to align images of different modalities more accurately; in the fusion process, rich feature information can guide the dynamic adjustment of weights, so that the fusion results can better integrate the advantageous information of different modalities in different local areas and improve the fusion quality.

[0081] Harris-Laplace feature point detection: Locking stable feature points in complex deformations:

[0082] Calculate the Harris corner response function at each scale j:

[0083] R Harris (x,y,j)=det(M(x,y,j))-k(trace(M(x,y,j))) 2 ;

[0084] Where M(x,y,j) is the autocorrelation matrix at scale j, combined with the Laplace operator response:

[0085] R Laplace (x,y,j)=▽ 2 I(x,y,j);

[0086] By comprehensively considering these two responses and selecting R Harris (x,y,j) and R Laplace Points where (x, y, j) all meet the threshold conditions are considered feature points;

[0087] In the image registration of human tissues and organs, local deformations such as respiratory movement and heartbeat will cause the feature position and shape in the image to change continuously. For example, features such as the edge of the lungs in chest images will change dynamically with breathing. In this case, the traditional feature extraction method is very easy to drift and become unstable, resulting in the loss of key information and affecting the accuracy of registration fusion.

[0088] The detection method of this algorithm performs excellently in complex deformation scenes and can reliably extract highly representative feature points with strong anti-interference capabilities, laying a solid foundation for subsequent processing.

[0089] Relevance and necessity to other steps:

[0090] These stable feature points are crucial for feature matching. In complex multimodal imaging scenarios, especially when there is local deformation interference and complex feature similarity, stable feature points can provide more reliable input for Local Sensitive Hashing (LSH). Due to the stability of feature points, LSH can more accurately locate candidate matching points when mapped to hash buckets, reduce mismatches caused by feature point drift, and thus improve matching accuracy.

[0091] In non-rigid deformation modeling, stable feature points are the key to accurately describe local non-rigid deformation. The free-form deformation model (FFD) based on B-spline relies on these stable feature point pairs to determine the matching relationship and then solve the displacement of the control points. If the feature points are unstable, the model will not be able to accurately capture the local deformation details, resulting in registration deviation.

[0092] In the process of registration and fusion, stable feature points help to achieve more accurate registration based on deformation models. In weighted average fusion, stable feature points can be used as references to dynamically adjust weights according to the characteristics of the area where they are located, ensuring that the fused image can accurately integrate the superior information of different modalities in each local area, thereby improving the overall quality of registration and fusion.

[0093] S3. Feature matching - matching based on local sensitive hashing and mutual information: Introducing a new matching paradigm, assuming that the feature point sets extracted from the two modal images are P = {p i} and Q = {q i}, the feature description vectors are E P ={p i} and E Q = {q i}; This algorithm uses local sensitive hashing to map the feature description vector into multiple hash buckets; for each point p in P i, can quickly locate candidate matching points in the corresponding hash bucket of Q. This efficient index search mechanism is very different from the point-by-point comparison of traditional matching methods, which greatly improves the matching efficiency. On this basis, the mutual information between candidate matching points is calculated:

[0094]

[0095] Where p(a,b) is the point p i and q j The joint probability distribution of the eigenvalues ​​of , p(a) and p(b) are their respective marginal probability distributions. The point pairs with the largest mutual information are selected as matching point pairs based on mutual information, which is used to essentially reduce the occurrence of mismatching and ensure high accuracy of feature matching;

[0096] Relevance and necessity to other steps:

[0097] The high precision and efficiency of feature matching play a key role in the successful implementation of the entire algorithm process. Accurate feature matching results directly provide a reliable data basis for non-rigid deformation modeling. In the free deformation model (FFD) based on B-spline, matching point pairs are an important basis for solving the displacement of control points. If there are a large number of mismatches in feature matching, the model will not be able to accurately describe the local non-rigid deformation, resulting in registration failure.

[0098] In the registration and fusion process, accurate feature matching ensures the accuracy of registration based on the deformation model. Only correctly matched feature points can guarantee the accuracy when transforming an image of one modality to another modality image space. At the same time, in the weighted average fusion process, accurate feature matching helps to determine the feature similarities and differences of different local areas, thereby providing a basis for the dynamic adjustment of weights, so that the fused image can fully integrate the advantageous information of different modalities and improve the fusion quality.

[0099] In addition, the high efficiency of feature matching also ensures the real-time and scalability of the entire algorithm. When processing large-scale multimodal image data, the fast matching process can reduce computing time and resource consumption, making the algorithm more practical.

[0100] S4. Non-rigid deformation modeling - B-spline based free deformation model: Sculpting local deformation, including the following steps:

[0101] Let the matching feature point pair be {(p i ,q i )}, define a uniform 3D B-spline grid G(x,y,z) in image space, with control points C ijk . By minimizing the energy function:

[0102]

[0103] To solve the displacement of the control point, the first term is the distance error of the matching point pair, the second term is the regularization term, and λ is the weight parameter. is the Laplace operator;

[0104] The local non-rigid deformation of the human body or object tissues and organs makes the feature correspondence of the same object in different modal images complicated. The simple geometric transformation or rough deformation model adopted by some existing patents is often difficult to accurately describe the details of the local non-rigid deformation when dealing with the complex non-rigid deformation of the chest, abdomen and other parts in medical images caused by physiological activities, and the irregular deformation of the terrain caused by natural factors in remote sensing images. It is easy to have registration deviation, which restricts the accuracy of the non-rigid registration algorithm.

[0105] B-spline FFD plays a key role in this algorithm. With its unique flexible control point adjustment mechanism and optimization constraints of the energy function, it can delicately and accurately depict every detail of the local non-rigid deformation, ensuring a highly consistent registration effect of different modal images in the presence of complex local deformations.

[0106] Relevance and necessity to other steps:

[0107] Non-rigid deformation modeling is the key link between feature matching and registration fusion. Accurate deformation model is based on precise feature matching, and its output directly affects the quality of registration fusion.

[0108] The accurate deformation information obtained by the B-spline free deformation model (FFD) provides key transformation parameters for the registration step based on the deformation model. When transforming an image of one modality to another modality image space, these parameters ensure that each pixel can be accurately aligned, thereby achieving accurate registration under complex deformation conditions. Without an accurate deformation model, the registration process will not be able to adapt to the complex spatial relationship changes between multi-modal images, resulting in poor registration fusion effect.

[0109] In the weighted average fusion process, the local deformation details described by FFD help to more accurately determine the weight distribution of different regions. For example, in medical images, for organ regions with large deformation, the weights of MRI and CT images can be adjusted more reasonably based on the description of deformation by FFD, so that the fused image can better present tissue details in these regions and improve the fusion quality.

[0110] S5, registration fusion - registration based on deformation model and weighted average fusion; used to adapt to the fusion of complex scenes;

[0111] Deformation-based registration: the key to accurate alignment under complex deformations;

[0112] According to the solved B-spline deformation model, transform the image I1(x, y) of one modality into the space of the image I2(x, y) of another modality;

[0113] Traditional registration methods that only rely on simple affine transformation or rough deformation estimation cannot ensure accurate image alignment when multimodal images have complex deformations, and are difficult to adapt to the diverse spatial relationship changes between multimodal images, resulting in poor registration and fusion effects.

[0114] This spatial transformation based on the precise deformation model of this algorithm is a major improvement over the traditional registration method. It can ensure accurate alignment of images under complex deformation conditions, just like finding the exact position of each pixel in different modalities. It perfectly adapts to the diverse spatial relationship changes between multimodal images and is the key to achieving high-quality registration fusion.

[0115] Relevance and necessity to other steps:

[0116] Registration based on deformation model is one of the core steps of the entire registration and fusion process, which relies on the previous data preprocessing, feature extraction and non-rigid deformation modeling steps. Accurate data preprocessing provides a relatively stable input image, reducing the registration difficulties caused by displacement and initial position deviation. High-quality feature extraction provides rich and accurate feature information for the deformation model, which is used to calculate the deformation parameters. Accurate non-rigid deformation modeling directly gives the deformation model for registration.

[0117] At the same time, accurate registration results provide an accurate alignment basis for weighted average fusion. Only when images of different modalities are accurately aligned can weighted average fusion give full play to its advantages, and the weights can be reasonably adjusted according to the characteristics of different local areas, so that the fused image can integrate the advantageous information of different modalities in each local area and generate high-quality fusion results.

[0118] Fusion strategy:

[0119] Assume that the registered two-modality image is I 1,aligned (x,y) and I2(x,y), fused image I fusion (x,y) uses weighted average method:

[0120] I fusion (x,y)=w1I 1,aligned (x,y)+(w2I2(x,y));

[0121] Where w1+w2=1, and the weights w1 and w2 are dynamically adjusted according to the modal characteristics of the image and the reliability of the local area.

[0122] The fusion strategies of fixed weights or general weight distribution commonly used in existing technologies lack consideration of the differences in advantages of images of different modalities in specific local areas, resulting in the inability of the fused images to fully demonstrate the advantages of multi-modality in some key areas, and the information fusion effect is poor.

[0123] The personalized weight allocation strategy of this algorithm reasonably adjusts the weights according to the characteristics of different regions. In the field of medical imaging, the weight of MRI images is increased in areas rich in soft tissue to highlight the soft tissue details, and the weight of CT images is increased in bone areas to clearly present the bone structure; in remote sensing images, weights are flexibly allocated according to the differences in the performance of different types of objects in each band of images. This strategy enables the fused image to perfectly integrate the advantageous information of different modalities in each local area, generate rich and accurate fusion results, and provide high-quality image data for practical applications such as medical diagnosis and remote sensing analysis.

[0124] Relevance and necessity to other steps:

[0125] Weighted average fusion is closely related to all the previous steps. Data preprocessing and feature extraction provide accurate image data and rich feature information, so that the characteristics of different regions can be fully considered when determining the weights. Feature matching and non-rigid deformation modeling provide the fusion process with a deep understanding of the relationship between images of different modalities, helping to determine in which areas the advantages of a certain modality should be given more attention.

[0126] The deformation model-based matching criteria are the premise of weighted average fusion. Only after the images are accurately aligned, the weight adjustment is meaningful. The accurate fusion results in turn verify the effectiveness of all the previous steps, forming an organic whole and jointly improving the performance of the entire algorithm in multimodal image registration and fusion.

[0127] S6, multi-dimensional evaluation: used to ensure the quality of registration and fusion, using two quantitative indicators, mean square error and normalized cross-correlation, the formula is as follows:

[0128] Mean Square Error:

[0129] Normalized cross-correlation:

[0130]

[0131] Where N is the total number of image pixels, and are the mean values ​​of the registered images.

[0132] Two additional evaluation points:

[0133] Feature stability assessment:

[0134] Evaluation method: Calculate the position change rate of key feature points and the similarity of feature description vectors at different time points or in different modes. For example, for specific anatomical landmarks in medical images, count the position fluctuations during multiple breathing cycles or heartbeats; for ground features in remote sensing images, compare the consistency of their feature descriptions under different imaging conditions.

[0135] Most existing evaluation methods do not consider the stability of features in complex environments, and only focus on the numerical indicators of the registration results. Through feature stability evaluation, the reliability of feature extraction and matching can be further tested. If the feature stability is poor, it may mean that the feature extraction method fails under local deformation, or there is a mismatch in feature matching, which will affect the quality of the entire registration fusion.

[0136] Relevance to other steps: It is directly related to the feature extraction and matching steps. Its evaluation results can provide feedback on whether the feature extraction algorithm can obtain stable features in complex deformations, and whether the feature matching algorithm can accurately associate the same features in different modal images. At the same time, it also indirectly affects the non-rigid deformation modeling and registration fusion steps. Unstable features may lead to inaccurate deformation models and poor registration fusion effects.

[0137] Local deformation consistency assessment:

[0138] Evaluation method: Analyze the similarity of deformation patterns in different modal images for local deformation areas such as organ movement in medical images or terrain changes in remote sensing images. This is measured by calculating the correlation of texture and shape changes in the deformation area, as well as the consistency of the direction and amplitude of the deformation. For example, in medical images, the stretching and contraction patterns of the myocardium in different modalities when the heart beats are compared; in remote sensing images, the deformation trends of landslide areas in different bands are compared.

[0139] Existing evaluation methods are difficult to comprehensively measure the consistency of deformation between different modal images in the case of local non-rigid deformation. This inconsistency may be caused by inaccurate non-rigid deformation modeling or feature matching errors. Local deformation consistency evaluation can more deeply discover potential problems in the registration and fusion process when dealing with complex deformations.

[0140] Relevance to other steps: It is closely related to non-rigid deformation modeling and feature matching. The accuracy of non-rigid deformation modeling directly determines whether the description of local deformation is accurate, while the accuracy of feature matching affects the correct association of features in the deformed area. Accurate local deformation consistency evaluation helps to optimize the non-rigid deformation model and feature matching algorithm, thereby improving the quality of registration fusion.

[0141] Some existing algorithms rely on only a single quantitative indicator for evaluation and judge the quality of registration and fusion only from a numerical perspective, which is easy to ignore specific applications such as actual medical diagnosis and remote sensing analysis.

[0142] Through these two newly added evaluation points and the original MSE, NCC and manual visual evaluation, a more comprehensive and detailed evaluation system has been formed, which can more accurately measure the performance of the algorithm in multimodal image registration and fusion, and ensure the effectiveness and reliability of the algorithm in practical applications. This multi-dimensional evaluation method can not only find problems with the algorithm in the current application scenario, but also provide direction for further optimization and improvement of the algorithm to adapt to more complex and changeable multimodal image processing needs.

[0143] Summarize:

[0144] Solve local deformation and non-rigid registration problems:

[0145] Data preprocessing-motion compensation (S1): By collecting respiratory gating signals and ECG gating signals with high precision, and using affine transformation to implement rough registration, the human physiological signals are deeply integrated. This method effectively suppresses the overall displacement caused by physiological movement, reduces the adverse effects of local deformation on the subsequent registration and fusion effects, and provides a relatively stable imaging basis for subsequent steps. It solves the problems of local non-rigid deformation and overall displacement caused by physiological movements such as breathing and heartbeat from the source, reduces the difficulty of subsequent non-rigid deformation modeling, and improves the accuracy of overall registration and fusion.

[0146] Non-rigid deformation modeling - B-spline-based free deformation model (S4): Based on feature matching, B-spline FFD is used to delicately and accurately depict every detail of local non-rigid deformation with its unique flexible control point adjustment mechanism and energy function optimization constraints, ensuring that different modal images can be highly consistent and registered when there are complex local deformations (such as deformation of the chest and abdomen in medical images caused by physiological activities, and irregular terrain deformation caused by natural factors in remote sensing images). This overcomes the problem that existing simple geometric transformations or rough deformation models are difficult to accurately describe the details of local non-rigid deformations, thereby improving the accuracy of non-rigid registration.

[0147] Registration fusion - registration based on deformation model and weighted average fusion (S5): Based on the precise B-spline deformation model, spatial transformation is performed to ensure accurate alignment of images under complex deformation conditions, adapt to the diverse spatial relationship changes between multimodal images, and solve the problem that traditional registration methods cannot accurately align images under complex deformation scenes, achieving high-quality registration fusion, thereby improving the accuracy of the overall fusion result.

[0148] Each step cooperates with each other: data preprocessing provides stable input for non-rigid deformation modeling and registration fusion, and non-rigid deformation modeling provides a key deformation model for registration fusion to accurately align images. The three are progressive and work together to overcome the problems of local deformation and non-rigid registration.

[0149] It can effectively deal with the non-rigid deformation problems caused by local deformation and movement of human body or object tissues and organs, significantly improve the registration and fusion accuracy in the medical field of chest, abdomen and other parts, as well as complex terrain in remote sensing images, making the fusion results more accurate and reliable to meet the needs of actual applications.

[0150] Solve the uncertainty problem of feature extraction and matching:

[0151] Feature extraction - multi-level feature extraction (S2):

[0152] Multi-scale features based on wavelet transform: By performing wavelet decomposition on the pre-processed image, wavelet coefficients are obtained at different scales and directions, and the texture and edge features of each layer of the image are fully and finely depicted to form a rich multi-dimensional image feature "atlas". This enables more feature information to be obtained in complex scenes (such as capturing subtle texture differences at the cellular level in medical images, and presenting the ups and downs of complex terrain and fine changes in the boundaries of objects in remote sensing images), reducing the uncertainty caused by unclear features, providing a more comprehensive and accurate information basis for subsequent matching, and reducing the possibility of mismatching.

[0153] Harris-Laplace feature point detection: It can reliably extract highly representative feature points with strong anti-interference ability in complex deformation scenes, avoiding the problem of feature points extracted by traditional methods drifting and being unstable when the feature position and shape change with local deformation (such as the lung edge in chest images changes with breathing). It lays a solid foundation for subsequent feature matching and reduces matching uncertainty caused by feature instability.

[0154] Feature matching - matching based on local sensitive hashing and mutual information (S3): Local sensitive hashing is used to map feature description vectors into multiple hash buckets for efficient indexed search, and matching point pairs are selected based on mutual information, which greatly improves matching efficiency while ensuring high matching accuracy and reducing mismatches. It overcomes the low efficiency of point-by-point matching in traditional matching methods and the problem of proneness to mismatches in complex scenarios (such as complex terrain and areas with similar ground features in remote sensing images).

[0155] Each step cooperates with each other: feature extraction provides rich and accurate feature information for feature matching, and feature matching performs efficient and high-precision matching operations based on the extracted features. The two work closely together to reduce the uncertainty of feature extraction and matching.

[0156] Improve the stability and comprehensiveness of feature extraction, enhance the efficiency and accuracy of feature matching, reduce mismatching caused by unclear features, similar features, etc. in complex scenes, and thus enhance the accuracy and reliability of the entire registration and fusion algorithm, so that it can better adapt to the registration and fusion needs of different modal images in complex environments.

[0157] Solve the problem of difficulty in feature extraction caused by local deformation:

[0158] Harris-Laplace feature point detection in feature extraction-multi-level feature extraction (S2): In view of the situation in which the position and shape of image features are constantly changing due to local deformations such as respiratory movement and heartbeat in the registration of human tissue and organ images (such as the lung edge in chest images changes dynamically with breathing), by comprehensively considering the Harris corner point response function and the Laplace operator response, stable feature points that meet the threshold conditions are selected, overcoming the problem that traditional feature extraction methods are difficult to capture stable and representative features, reliably extracting feature points in complex deformation scenes, reducing the interference of local deformation on feature extraction, and improving the stability and accuracy of feature extraction.

[0159] Correlation with other steps: Provide stable and reliable feature point input for subsequent feature matching, non-rigid deformation modeling, and registration fusion steps, ensuring that subsequent links can work based on accurate feature information and guaranteeing the quality of the entire registration fusion algorithm.

[0160] Solve the difficulty of feature extraction caused by local deformation, so that the feature extraction algorithm can still obtain stable and effective features under complex conditions with local deformation, lay a good foundation for the entire multimodal image registration and fusion process, and improve the quality of the final fusion result.

[0161] Solve the matching error problem caused by non-rigid deformation:

[0162] Feature matching - matching based on local sensitive hashing and mutual information (S3): Local sensitive hashing and mutual information are used to select point pairs with the largest mutual information as matching point pairs. With efficient matching mechanism and high-precision matching strategy, the same features in different modal images are accurately associated, overcoming the problem of mismatching caused by complex feature correspondence of the same object in different modal images due to local non-rigid deformation (such as the mismatch of similar features of deformed organ features at different times or in different modalities during medical image fusion, and the difficulty of feature matching caused by the deformation of complex terrain in remote sensing images due to natural factors), thereby reducing matching errors.

[0163] Non-rigid deformation modeling - B-spline based free deformation model (S4): Based on accurate feature matching results, B-spline FFD is used to accurately describe the local non-rigid deformation details, making the feature correspondence clearer and more accurate, further avoiding the mismatch of similar features caused by deformation, providing accurate deformation information basis for subsequent registration fusion, and reducing the impact of matching errors caused by non-rigid deformation on registration accuracy.

[0164] Each step cooperates with each other: feature matching provides accurate matching point pairs for non-rigid deformation modeling. Non-rigid deformation modeling accurately describes the deformation based on the matching results, which in turn assists feature matching to more accurately associate features. The two promote each other and reduce the matching error caused by non-rigid deformation.

[0165] Improve the accuracy of feature matching, reduce feature mismatches caused by local non-rigid deformation, make the features between different modal images correspond correctly, thereby improving the accuracy of registration and fusion, and ensuring that the algorithm can obtain high-quality registration and fusion results when processing multi-modal images with non-rigid deformation.

[0166] Solve the problem of inaccurate feature extraction limiting the effect of non-rigid registration:

[0167] Feature extraction - multi-level feature extraction (S2):

[0168] Multi-scale features based on wavelet transform: The extracted multi-dimensional wavelet coefficient features provide detailed texture and edge information for non-rigid deformation modeling, which helps to better understand the deformation of the image in the local area, and assists the B-spline-based free deformation model (FFD) to more accurately determine the displacement of the control points, thereby more delicately depicting the details of the local non-rigid deformation, avoiding the problem of being unable to construct an accurate deformation field to describe the deformation (such as non-rigid deformation of the heart area) due to inaccurate feature extraction, and improving the accuracy of non-rigid registration.

[0169] Harris-Laplace feature point detection: The extracted stable feature points are the key to accurately describe local non-rigid deformations, providing a reliable basis for determining matching relationships and solving control point displacements based on the B-spline free deformation model (FFD), ensuring that the deformation model can be constructed based on accurate feature information, overcoming the problem of unreliable feature points for estimating non-rigid deformations caused by inaccurate feature extraction, and improving the accuracy of the non-rigid registration algorithm.

[0170] Relevance to other steps: Feature extraction provides key feature information for non-rigid deformation modeling. Non-rigid deformation modeling builds a model based on accurate feature information to achieve accurate registration. The two work together to ensure that the non-rigid registration effect is not limited by inaccurate feature extraction.

[0171] By providing accurate and comprehensive feature information, non-rigid deformation modeling can build an accurate deformation model based on reliable feature points and rich texture edge information, thereby improving the effect of non-rigid registration and ensuring that the entire registration fusion algorithm can accurately handle the non-rigid deformation between multimodal images, thereby improving the quality of the final fusion result.

[0172] Solve the problem of matching uncertainty interfering with deformation estimation:

[0173] Feature matching - matching based on local sensitive hashing and mutual information (S3): Through a high-precision matching mechanism, the accuracy of feature matching is ensured, mismatching is avoided, and a reliable data basis is provided for non-rigid deformation modeling. Accurate feature matching results enable the B-spline-based free deformation model (FFD) to solve the control point displacement based on the correct matching point pairs, thereby accurately describing the local deformation degree and direction, overcoming the problem that mismatched features in remote sensing images cause the deformation of objects to deviate from reality, and that incorrect feature matching in medical images leads to incorrect estimation of deformations such as respiratory movement or organ peristalsis, and reducing the interference of matching uncertainty on deformation estimation.

[0174] Correlation with other steps: The accuracy of feature matching directly affects the estimation of deformation in non-rigid deformable modeling, and non-rigid deformable modeling provides guarantee for registration fusion based on accurate deformation estimation. The three are interrelated to ensure that the whole process is not disturbed by matching uncertainty.

[0175] Improve the accuracy of deformation estimation so that non-rigid deformation modeling can truly reflect local deformation based on accurate feature matching results, thereby ensuring that the registration and fusion process can operate based on accurate deformation information, and improving the performance of the entire multimodal image registration and fusion algorithm when dealing with local deformation and the quality of the final fusion results.

[0176] In summary, this multimodal image registration and fusion algorithm comprehensively solves the above-mentioned series of problems existing in the existing algorithms through close cooperation and collaboration between each step, improves the accuracy, reliability and practicality of multimodal image registration and fusion in various complex scenarios, and provides better quality image data for practical applications such as medical diagnosis and remote sensing analysis.

Claims

1. A multimodal image registration and fusion algorithm, characterized in that: The following steps are involved: S1. Data preprocessing-motion compensation: introducing source correction strategy; S2. Feature extraction - multi-level feature extraction: Constructing a high-dimensional stable feature space, including the following steps: Multi-scale features based on wavelet transform: Open up multi-dimensional insights into image features: Perform wavelet decomposition on the preprocessed image I(x,y) and obtain the wavelet coefficients W at different scales j and directions k. j,k (x,y); Harris-Laplace feature point detection: Locking stable feature points in complex deformations: Calculate the Harris corner response function at each scale j: R Harris (x,y,j)=det(M(x,y,j))-k(trace(M(x,y,j))) 2 4 Where M(x,y,j) is the autocorrelation matrix at scale j, combined with the Laplace operator response: R Laplace (x,y,j)=▽ 2 I(x,y,j); By comprehensively considering these two responses and selecting R Harris (x,y,j) and R Laplace Points where (x, y, j) all meet the threshold conditions are considered feature points; S3, Feature Matching - Matching based on Locality Sensitive Hashing and Mutual Information: Introducing a new matching paradigm; S4. Non-rigid deformation modeling - B-spline based free deformation model: Sculpting local deformation, including the following steps: Let the matching feature point pair be {(p i ,q i )}, define a uniform 3D B-spline grid G(x,y,z) in image space, with control points C ijk ; By minimizing the energy function: To solve the displacement of the control point, the first term is the distance error of the matching point pair, the second term is the regularization term, λ is the weight parameter, ▽ 2 is the Laplace operator; S5, Registration Fusion - Registration based on deformation model and weighted average fusion; used to adapt to the fusion of complex scenes.

2. The multimodal image registration and fusion algorithm according to claim 1, characterized in that: In S1, the original medical image is represented as I(x, y, t); x, y are spatial coordinates, and t is time. This algorithm uses high-precision acquisition of respiratory gating signal r(t) and ECG gating signal c(t), and uses affine transformation T r (x,y,t) and T c (x, y, t) performs coarse registration, the formula is:

3. The multimodal image registration and fusion algorithm according to claim 1, characterized in that: In S3, let the feature point sets extracted from the two modal images be P = {p i } and Q = {q i }, the feature description vectors are E P ={p i } and E Q = {q i }; This algorithm uses local sensitive hashing to map the feature description vector into multiple hash buckets; for each point p in P i , can quickly locate the candidate matching points in the corresponding hash bucket of Q, and on this basis, calculate the mutual information between the candidate matching points: Where p(a,b) is the point p i and q j The joint probability distribution of the eigenvalues ​​of p(a) and p(b) are their respective marginal probability distributions. The point pairs with the largest mutual information are selected as matching point pairs based on mutual information, which is used to essentially reduce the occurrence of mismatches.

4. The multimodal image registration and fusion algorithm according to claim 1, characterized in that: In S5, according to the solved B-spline deformation model, the image I1(x, y) of one modality is transformed into the space of the image I2(x, y) of another modality; Fusion strategy: Assume that the registered two-modality image is I 1,aligned (x,y) and I2(x,y), fused image I fusion (x,y) uses weighted average method: I fusion (x,y)=w1I 1,aligned (x,y)+(w2I2(x,y)); Where w1+w2=1, and the weights w1 and w2 are dynamically adjusted according to the modal characteristics of the image and the reliability of the local area.

5. The multimodal image registration and fusion algorithm according to claim 1, characterized in that: The following steps are also included: S6. Multi-dimensional evaluation: used to ensure the quality of registration and fusion.

6. The multimodal image registration and fusion algorithm according to claim 5, characterized in that: In S6, two quantitative indicators, mean square error and normalized cross-correlation, are used, and the formula is as follows: Mean Square Error: Normalized cross-correlation: Where N is the total number of image pixels, and are the mean values ​​of the registered images.

7. The multimodal image registration and fusion algorithm according to claim 6, characterized in that: Calculate the position change rate of key feature points and the similarity of feature description vectors at different time points and in different modes.

8. The multimodal image registration and fusion algorithm according to claim 7, characterized in that: Aiming at the organ movement in medical images and the terrain changes in remote sensing images, the similarity of deformation patterns in different modal images is analyzed; it is measured by calculating the correlation of texture and shape changes in the deformed area, as well as the consistency of deformation direction and amplitude.

Images