Multi-modal image registration method and device based on generative model and self-supervised learning, and medium
By combining generative models and self-supervised learning methods with regional feature intensity and structural importance for multimodal image registration, the problems of low accuracy and high computational complexity in existing technologies are solved, achieving efficient and accurate image alignment, which is applicable to fields such as remote sensing and urban planning.
Patent Information
- Application Number
- CN202511083768.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-04
- Publication Date
- 2025-11-18
AI Technical Summary
Existing multimodal image registration methods face problems of low registration accuracy and high computational complexity when processing images acquired by different types of sensors, especially due to difficulties in feature extraction and inaccurate matching caused by modal differences.
We employ a generative model and self-supervised learning approach. We use the Pix2Pix-Transformer generative model for mode transformation and combine it with the DINO self-supervised learning model to extract global and local features for adaptive regional registration. We then assign weights based on the strength of regional features and the importance of structures to implement high-precision nonlinear registration and low-precision linear registration.
It improves the accuracy and efficiency of multimodal image registration, reduces computational resource consumption, and is suitable for multimodal image registration tasks in fields such as remote sensing images and urban planning.
Smart Images
Figure CN120976274A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and more specifically to a multimodal image registration method, device, and medium based on generative models and self-supervised learning. Background Technology
[0002] Image registration is a crucial task in computer vision and image processing, widely applied in fields such as medical imaging, remote sensing, computer vision, and robot navigation. Its core objective is to align multimodal or homomodal images, ensuring consistency in spatial coordinates between images captured at different times, from different viewpoints, or by different sensors, thus facilitating subsequent analysis and processing. In practical applications, multimodal image registration faces a series of challenges, especially when processing images acquired by different types of sensors (e.g., visible light images versus infrared images, CT images versus MRI images). Due to differences in visual information, brightness, and contrast among different modalities, traditional registration methods (such as grayscale-based methods) often struggle to handle these differences, resulting in low registration accuracy and high computational complexity. Current multimodal image registration methods can be broadly categorized into two types: feature-based registration methods and intensity-based registration methods. Feature-based methods rely on specific feature points in the image (such as corners and edges) to match these features for image alignment; intensity-based methods, on the other hand, perform registration by optimizing the similarity of image intensity. However, these methods often face challenges such as difficulty in feature extraction, inaccurate matching, and high computational resource consumption when processing images with significant modal differences.
[0003] To address these issues, a region-adaptive registration method is proposed. This method employs different registration strategies for key and background regions based on their varying importance. Key regions typically contain crucial semantic information and require high-precision registration, while background regions can utilize lower-precision registration methods, thus improving overall registration efficiency. However, existing region-adaptive registration methods often employ simple region segmentation strategies, lacking sufficient consideration of region feature strength and structural importance. Most methods segment based on coarse image features (such as edges and corners) without finely distinguishing the feature strength and structural importance of regions, resulting in registration accuracy that fails to meet the demands of demanding applications.
[0004] To address the problems in existing technologies, this invention proposes a multimodal image registration method based on generative models and self-supervised learning. Summary of the Invention
[0005] In view of this, the present invention provides a multimodal image registration method, device, and medium based on generative models and self-supervised learning. It exhibits good robustness, adapts to registration tasks of different modalities, and maintains high-precision registration results even when there are significant differences between the source and target images. By combining modality transformation with region adaptive registration, the present invention can not only handle complex modal differences but also achieve efficient and accurate image alignment with relatively low computational cost. It is widely applicable to multimodal image registration tasks in fields such as remote sensing imagery, urban planning, and agricultural monitoring.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] A multimodal image registration method based on generative models and self-supervised learning includes:
[0008] The source image is modally transformed using the Pix2Pix-Transformer generative model to generate an intermediate image with the same modality as the target image.
[0009] Perform the following operations on the intermediate image and the target image respectively:
[0010] The DINO-based self-supervised learning model extracts global and local features of the image respectively.
[0011] Based on global and local features, the image is divided into key regions containing key structures and background regions.
[0012] Based on the intensity of regional characteristics and the importance of structure, the key regions and background regions are weighted and an area-weighted registration matrix is established.
[0013] A region adaptive registration strategy is adopted to perform high-precision nonlinear registration on key regions and low-precision linear registration on background regions to obtain registration parameters.
[0014] The registration parameters are optimized using a region-weighted registration matrix, and the final registration result is output.
[0015] Preferably, the source image is modally transformed using the Pix2Pix-Transformer generative model to generate an intermediate image with the same modality as the target image, including:
[0016] 1) Model building: Build a Pix2Pix-Transformer generative model, including a generator and a discriminator. The generator adopts the Transformer architecture, and the discriminator adopts the convolutional neural network architecture.
[0017] 2) Data processing: Collect pairs of source and target images and normalize the images;
[0018] 3) Model training:
[0019] A Pix2Pix-Transformer generative model is trained based on normalized data, and an adversarial loss L is defined. adv Pixel loss L pixel and perceived loss L perceptual The loss function consists of the total loss L. total The calculation formula is:
[0020] L total =L adv +λ1L pixel +λ2L perceptual
[0021] Where λ1 and λ2 are balance coefficients;
[0022] Pixel loss is calculated using the mean square error loss formula:
[0023]
[0024] Where H and W are the image height and width, respectively, and I gen For the generated intermediate image, I t Here, h represents the row index of a pixel in the target image, and w represents the column index of a pixel in the target image.
[0025] The perceptual loss utilizes a pre-trained VGG network to extract high-level semantic features from the intermediate and target images, and calculates the feature difference using the following formula:
[0026]
[0027] Where l represents a specific layer of the VGG network, C l H l W l These represent the number of channels, height, and width of the feature map for this layer, respectively. and These are the feature maps of the intermediate image and the target image in this layer, respectively. m represents the row index of the pixel in the feature map of a specific layer of the VGG network, n represents the column index of the pixel in the feature map of a specific layer of the VGG network, and c represents the channel index of the pixel in the feature map of a specific layer of the VGG network.
[0028] 4) Modality Transformation: The source image to be registered is input into the trained generator. Based on the mapping relationship between the source image and the target image modality learned during training, the generator transforms and reconstructs the features of the source image, and outputs an intermediate image I that is consistent with the modality of the target image. gen .
[0029] Preferably, the DINO-based self-supervised learning model extracts global and local features of the image, including:
[0030] Intermediate Image I gen and target image I t The global encoder ε of the self-supervised learning model of DINO global The global features are extracted, and the specific calculation formula is as follows:
[0031] f global (I gen )=ε global (I gen ),f global (I t )=ε global (I t )
[0032] Among them, I gen Represents the intermediate image, I t f represents the target image global (I gen ) represents the intermediate image I gen via global encoder ε global The extracted global features, f global (I t ) represents the target image I t via global encoder ε global The extracted global features;
[0033] Intermediate Image I gen and target image I t The local encoder ε of the self-supervised learning model of DINO local The specific formula for extracting local features is as follows:
[0034] f local (I gen )=ε local (I gen ),f local (I t )=ε local (I t )
[0035] Among them, f local (I gen ) represents the intermediate image I gen Through the local encoder ε local The extracted local features, f local (I t ) represents the target image I t Through the local encoder ε local Extracted local features;
[0036] Normalize both global and local features:
[0037]
[0038]
[0039] in, Representing global features f global (I gen The normalization result of ) Representing global features f global (I t The normalization result of ) Representing local features f local (I gen The normalization result of ) Representing local features f local (I t The normalization result of ).
[0040] Preferably, the image is divided into regions based on global and local features, including key regions containing key structures and background regions, including:
[0041] S301: Perform preliminary region division of the image based on the normalized global features;
[0042] S302: Based on the normalized local features, further refine the region division according to the preliminary region division:
[0043] If the local feature intensity of image region R is ||f local (R)||2 exceeds the set threshold θ key Then the area is designated as a critical area:
[0044]
[0045] If the local feature intensity of image region R is ||f local (R)||2 is lower than the set threshold θ bg Then this area is designated as the background area:
[0046]
[0047] Through this process, both the intermediate image and the target image are divided into different regions R1, R2, ..., R... N N represents the number of regions, some of which are marked as critical regions and others as background regions.
[0048] Preferably, a weighted registration matrix is established by assigning weights to key regions and background regions based on the intensity of regional features and structural importance, including:
[0049] S401: Calculate region weights:
[0050]
[0051] Where α and β are adjustment coefficients, importance(R) i ) is the region R i The structural importance is evaluated based on the semantic content and global features in the image, ||f local (R j )||2 represents region R j The local feature intensity, ||f local (R i )||2 represents region R i The intensity of local features, For regional weights;
[0052] S402: Calculate the weighted registration matrix based on regional weights:
[0053]
[0054] Where W is the region-weighted registration matrix.
[0055] Preferably, high-precision nonlinear registration is performed on the key areas, and its loss function is expressed as:
[0056]
[0057] Among them, R key As a key area, For the critical region R key The weights, T(R) key Let R' be the transformation function. key For target image I t The location of the corresponding region, the loss function L nonlinear Used to minimize transformation errors between regions;
[0058] Low-precision linear registration is performed on the background region, and its loss function can be expressed as:
[0059]
[0060] Among them, R bg For background area, Background region R bg The weights, T(R) bg ) for the background region R bgThe result after applying the linear transformation, R′ bg For the target image with R bg The corresponding reference area location.
[0061] Preferably, the registration parameters are optimized by minimizing the weighted loss function based on the region-weighted registration matrix, and the optimal transformation matrix is solved. The weighted loss function L total as follows:
[0062]
[0063] The optimal transformation matrix T is obtained by minimizing the weighted loss function. * :
[0064]
[0065] Optimal transformation matrix T * The geometric transformation between the intermediate image and the target image is represented by T, where T is the geometric transformation matrix.
[0066] The pixel p of the intermediate image s Through T * Transform to the position p of the target image t :
[0067]
[0068] Where, p s Use coordinates (x) s ,y s ) indicates that x s and y s These are the x and y coordinates of the pixel in the intermediate image, respectively. t Represents the pixel p in the target image and the intermediate image s The corresponding position points, after the optimal transformation matrix T * For p s After the transformation, the result is p. t Coordinates (x) in the target image t ,y t ).
[0069] A computer device includes a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the computer program, it implements a multimodal image registration method based on a generative model and self-supervised learning.
[0070] A computer-readable storage medium storing a computer program that, when executed by a processor, implements a multimodal image registration method based on a generative model and self-supervised learning.
[0071] As can be seen from the above technical solution, compared with the prior art, this invention discloses a multimodal image registration method, device, and medium based on generative models and self-supervised learning, which performs weighted registration by combining regional feature strength and structural importance. The image is divided into key regions and background regions based on feature similarity. Subsequently, weights are assigned to each region according to its feature strength and structural importance, employing a region adaptive registration strategy to ensure high-precision nonlinear registration for key regions and low-precision linear registration for background regions. Finally, the registration parameters are optimized through a weighted registration matrix to output accurate registration results. This method not only improves registration accuracy but also significantly reduces computational resource consumption. Attached Figure Description
[0072] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0073] Figure 1 The flowchart of a multimodal image registration method based on generative models and self-supervised learning provided by this invention is shown.
[0074] Figure 2 This invention provides a network framework diagram for a multimodal image registration method based on generative models and self-supervised learning.
[0075] Figure 3 This is a schematic diagram of the Pix2Pix-Transformer generation model provided by the present invention.
[0076] Figure 4 This is a diagram illustrating the registration effect of multimodal images provided by the present invention. Detailed Implementation
[0077] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0078] This invention discloses a multimodal image registration method based on generative models and self-supervised learning, such as... Figure 1 and Figure 2 As shown, it includes:
[0079] The source image is modally transformed using the Pix2Pix-Transformer generative model to generate an intermediate image with the same modality as the target image.
[0080] Perform the following operations on the intermediate image and the target image respectively:
[0081] The DINO-based self-supervised learning model extracts global and local features of the image respectively.
[0082] Based on global and local features, the image is divided into key regions containing key structures and background regions.
[0083] Based on the intensity of regional characteristics and the importance of structure, the key regions and background regions are weighted and an area-weighted registration matrix is established.
[0084] A region adaptive registration strategy is adopted to perform high-precision nonlinear registration on key regions and low-precision linear registration on background regions to obtain registration parameters.
[0085] The registration parameters are optimized using a region-weighted registration matrix, and the final registration result is output.
[0086] Each step of the present invention will be described in detail below.
[0087] The Pix2Pix-Transformer generative model is used to perform modality transformation on the source image, including the following steps:
[0088] 1) Model Setup: Build a Pix2Pix-Transformer generative model, such as... Figure 3 As shown, the generative model consists of a generator and a discriminator. The generator adopts a Transformer architecture, including embedding + positional encoding, a multi-head attention mechanism, and a feedforward network. The multi-head attention mechanism is calculated as follows:
[0089]
[0090] Where Q, K, and V represent the query, key, and value matrices, respectively, and d k It is the key vector dimension, and the multi-head attention mechanism processes multiple sets of attention in parallel to capture information from different positions in the source image. The feedforward neural network further processes and transforms the features.
[0091] The discriminator uses a convolutional neural network architecture, consisting of multiple convolutional layers, pooling layers, and fully connected layers, to determine the similarity between the generated image and the target image.
[0092] 2) Data Processing: Collect a large number of paired source and target images. Normalize the images using the following formula:
[0093]
[0094] Where I is the original image pixel value, I min and I max I represents the minimum and maximum pixel values in the original image, respectively. norm These are the normalized pixel values. The images are then cropped and scaled according to the model's input size requirements to ensure all images have a uniform size specification.
[0095] 3) Model training:
[0096] Model training: Defined by adversarial loss L adv Pixel loss L pixel and perceived loss L perceptual The loss function consists of the total loss L. total The calculation formula is:
[0097] L total =L adv +λ1L pixel +λ2L perceptual
[0098] Where λ1 and λ2 are balance coefficients.
[0099] The adversarial loss is based on the principles of generative adversarial networks (GANs). The generator attempts to produce images that the discriminator will misclassify. The discriminator distinguishes between real and generated images, thereby optimizing the generator. Pixel loss is calculated using the mean squared error loss formula:
[0100]
[0101] Where H and W are the image height and width, respectively, and I gen For the generated intermediate image, I t Let h be the row index of a pixel in the target image, ranging from 1 to the image height H, and w be the column index of a pixel in the image, ranging from 1 to the image width W. The perceptual loss utilizes a pre-trained VGG network to extract high-level semantic features from the intermediate and target images, calculating the feature difference using the following formula:
[0102]
[0103] Where l represents a specific layer of the VGG network, C l H l W l These represent the number of channels, height, and width of the feature map for this layer, respectively. and These are the feature maps of the intermediate image and the target image in this layer, respectively. m represents the row index of the pixel in the feature map of a specific layer in the VGG network, ranging from 1 to the height H of that layer's feature map. l 'n' represents the column index of a pixel in a specific layer feature map of the VGG network, and its value ranges from 1 to the width W of that layer feature map. l .
[0104] 4) Modality Transformation: The source image to be registered is input into the trained generator. Based on the mapping relationship between the source image and the target image modality learned during training, the generator transforms and reconstructs the features of the source image, and outputs an intermediate image I that is consistent with the modality of the target image. gen This reduces the modal differences between the source and target images.
[0105] This invention converts a source image into an intermediate image with the same modality as the target image, thereby effectively reducing the modal differences between the source and target images. The intermediate image generated through modality conversion can be better aligned with the target image, simplifying the subsequent registration process and improving the accuracy and robustness of registration.
[0106] The global and local features of the intermediate and target images are extracted using a DINO-based self-supervised learning model, including the following steps:
[0107] 1) Global feature extraction: intermediate image I gen and target image I t The global encoder ε of the self-supervised learning model of DINO global Extract global features f global (I gen ) and f global (I t This feature represents the overall structure and semantic information of the image. The specific calculation formula is as follows:
[0108] f global (I gen )=ε global (I gen ),f global (I t )=ε global (I t )
[0109] Among them, I gen The intermediate image, I, is the initial image that needs to be registered with the target image. t The target image represents the object that needs to be registered and aligned with the intermediate images. It has different modal features from the intermediate images. ε globalIt is the global encoder of the DINO model, a trained network structure capable of performing feature extraction on input images, converting image data into feature vectors that represent the overall structure and semantic information of the image. global (I gen ) represents the intermediate image I gen via global encoder ε global The extracted global features, f global (I t ) represents the target image I t via global encoder ε global The extracted global features.
[0110] This process effectively captures macroscopic information of an image, such as the shape and approximate location of objects, through the global features obtained by contrastive learning training.
[0111] 2) Local feature extraction: intermediate image I gen and target image I t The local encoder ε of the DINO model local Extracting local features f local (I gen ) and f local (I t This feature primarily describes the details of an image, such as texture, edges, and local structure. The formula for calculating local feature extraction is:
[0112] f local (I gen )=ε local (I gen ),f local (I t )=ε local (I t )
[0113] Where, ε local It is the local encoder of the DINO model, a network module specifically designed to extract local features from images. It focuses on local regions within an image, capturing and encoding detailed information to generate vectors that describe the local features of the image, f. local (I gen ) represents the intermediate image I gen Through the local encoder ε local The extracted local features, f local (I t ) represents the target image I t Through the local encoder ε localThe extracted local features are crucial for accurately aligning small objects and local details in images, especially in multimodal image registration, where they help handle modal differences.
[0114] 3) Feature Normalization: To ensure the comparability of features between intermediate and target images, this invention normalizes the extracted global and local features. Feature normalization can eliminate scale differences between different images, enhancing the stability and consistency of image features. The normalization operation is as follows:
[0115]
[0116] The normalized feature vectors are standardized to unit vectors, which makes the features between the intermediate image and the target image have the same scale, further improving the accuracy and stability of subsequent region segmentation and image registration processes.
[0117] Image region segmentation is performed based on feature similarity, dividing the image into key regions containing key structures and background regions, including:
[0118] Based on the normalized global features, the intermediate image I gen and target image I t Preliminary region segmentation is performed. Global features reflect the macroscopic layout of the image, helping to quickly identify the approximate locations of background regions and main objects. Based on this, the normalized local features f are further segmented. local (I gen ) and f local (I t Further analysis and refined region segmentation allow for accurate identification of key and background regions in the image.
[0119] 1) Key region: If the local feature intensity of image region R is ||f local (R)||2 exceeds the set threshold θ key Then the area is designated as a critical area:
[0120]
[0121] 2) Background region: If the local feature intensity of image region R is ||f local (R)||2 is lower than the set threshold θ bg Then this area is designated as the background area:
[0122]
[0123] Through this process, both the intermediate image and the target image are divided into different regions R1, R2, ..., R... NSome of these regions are marked as critical regions, while others are considered background regions. The specific threshold is set to: θ key =0.75, θ bg =0.3.
[0124] Weights are assigned based on the intensity of regional features and structural importance to establish a regional weighted registration matrix. This invention assigns weights to each region to further optimize the registration process.
[0125] 1) Regional weight allocation:
[0126] For each region R i (regardless of region R in the intermediate image) s Or the region R in the target image t ), weight The allocation is based on two criteria: regional characteristic intensity and regional structural importance: Region R i Local feature intensity ||f local (R)||2 reflects the saliency of detailed information within the region. Regions with higher feature strength typically contain richer information and therefore require more computational resources to ensure registration accuracy. Region R i Structural importance reflects a region's contribution to the overall structure of the image. A region is considered more structurally important if it contains significant structural features, such as landmarks, prominent textures, or shapes. These two metrics are combined to calculate region weights.
[0127]
[0128] Where α and β are adjustment coefficients, α = 0.7 and β = 0.3, importance(R) i ) is the region R i The structural importance of an image is typically evaluated based on its semantic content and global structure.
[0129] 2) Construction of the region-weighted registration matrix: weights After the allocation is complete, a region-weighted registration matrix is constructed. This matrix guides the optimization algorithm in subsequent registration processes, ensuring the registration accuracy and efficiency of different regions. Each row of matrix W corresponds to the weight of a region in the intermediate image and the target image, in the following form:
[0130]
[0131] In the subsequent registration process, this matrix optimizes the registration parameters by weighting the regions, achieving fine alignment of key regions while improving the registration efficiency of background regions.
[0132] A region-adaptive registration strategy is adopted, performing high-precision nonlinear registration on key regions and low-precision linear registration on background regions. Based on the region-weighted registration matrix, this invention further proposes a region-adaptive registration strategy, employing different registration strategies for regions of varying importance to improve the overall registration accuracy and efficiency. For key regions with higher weights, a high-precision nonlinear registration method is used, the loss function of which can be expressed as:
[0133]
[0134] Among them, R key As a key area, For the critical region R key The weights, T(R) key Let R' be the transformation function. key The loss function L represents the location of the corresponding region in the target image. nonlinear This method is used to minimize the transformation error between regions. For background regions with lower weights, a low-precision linear registration method is employed, and its loss function can be expressed as:
[0135]
[0136] Among them, R bg For background area, Background region R bg The weights, T(R) bg ) for the background region R bg The result after applying the linear transformation, R′ bg For the target image with R bg The corresponding reference area location.
[0137] The registration parameters are optimized using a weighted registration matrix, and the final registration result is output. Based on the region-weighted registration matrix, this invention optimizes the registration parameters by minimizing a weighted loss function to solve for the optimal transformation matrix. The weighted loss function is as follows:
[0138]
[0139] By minimizing this loss function, the optimal transformation matrix T is obtained. * ,Right now:
[0140]
[0141] Transformation matrix T * This corresponds to the geometric transformation between the intermediate image and the target image. The source image's pixel p... s Through T * Transform to the position p of the target image t :
[0142]
[0143] Among them, T * This is the optimal transformation matrix obtained by minimizing the weighted loss function. It represents the geometric transformation relationship required for accurate registration between the intermediate image and the target image. This matrix contains various geometric transformation parameters such as translation, rotation, and scaling, which can map each point in the intermediate image to the corresponding position in the target image according to specific rules, thus achieving image alignment. s Representing a pixel in the intermediate image, it is the basic building block of the intermediate image. In the image coordinate system, p s Coordinates (x) can be used s ,y s Let x be the value of x. s and y s These are the x and y coordinates of the pixel in the intermediate image, respectively. t Represents the pixel p in the target image and the intermediate image s The corresponding position points. After transformation matrix T * For p s After the transformation, the result is p. t Coordinates (x) in the target image t ,y t p t The location is p s After registration is completed, the coordinate information of the corresponding position reflects the correct positioning of the pixels of the intermediate image (source image) in the target image space, which is used to achieve precise alignment between the source image and the target image.
[0144] Figure 4 This is a registration effect diagram of multimodal images according to the present invention, including the registration results of three typical modal pairs:
[0145] (a) Map-Satellite Image: Displays multimodal registration of geographic information. The green polygon represents the ground truth coordinates from the source image (map) to the target image (satellite image), and the red polygon is the registration result estimated by the algorithm of this invention. It can be seen that the red area highly overlaps with the green ground truth, especially the alignment of key structures such as roads and buildings is accurate.
[0146] (b) Visible light-near infrared: The registration effect of different spectral modes is shown. The red polygon closely fits the green true value, indicating that the present invention can still accurately capture the spatial correspondence of key areas in images with significant spectral differences.
[0147] (c) Flash - No Flash: Verify the registration capability under different lighting conditions. The red polygon achieves high-precision alignment of the main structure in the scene.
[0148] All three sets of experiments show that the deviation between the red estimated region and the green true value is minimal, intuitively demonstrating the effectiveness of this invention in registration under complex modal differences. This invention reduces modal differences through a Pix2Pix-Transformer generative model, utilizes the DINO self-supervised model to extract multi-dimensional features for fine region segmentation, and employs high-precision nonlinear and low-precision linear registration strategies for key and background regions respectively, balancing accuracy and efficiency. This method requires no large amount of labeled data, has strong generalization ability, and combines high accuracy, high robustness, and high efficiency in complex scenes, providing reliable support for multimodal image analysis and possessing significant application value.
[0149] This embodiment provides a computer device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements a multimodal image registration method based on a generative model and self-supervised learning.
[0150] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a multimodal image registration method based on a generative model and self-supervised learning.
[0151] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0152] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.
[0153] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A multimodal image registration method based on generative models and self-supervised learning, characterized in that, include: The source image is modally transformed using the Pix2Pix-Transformer generative model to generate an intermediate image with the same modality as the target image. Perform the following operations on the intermediate image and the target image respectively: The DINO-based self-supervised learning model extracts global and local features of the image respectively. Based on global and local features, the image is divided into key regions containing key structures and background regions. Based on the intensity of regional characteristics and the importance of structure, the key regions and background regions are weighted and an area-weighted registration matrix is established. A region adaptive registration strategy is adopted to perform high-precision nonlinear registration on key regions and low-precision linear registration on background regions to obtain registration parameters. The registration parameters are optimized using a region-weighted registration matrix, and the final registration result is output.
2. The multimodal image registration method based on generative models and self-supervised learning according to claim 1, characterized in that, The source image is modally transformed using the Pix2Pix-Transformer generative model to generate an intermediate image with the same modality as the target image, including: 1) Model building: Build a Pix2Pix-Transformer generative model, including a generator and a discriminator. The generator adopts the Transformer architecture, and the discriminator adopts the convolutional neural network architecture. 2) Data processing: Collect pairs of source and target images and normalize the images; 3) Model training: A Pix2Pix-Transformer generative model is trained based on normalized data, and an adversarial loss L is defined. adv Pixel loss L pixel and perceived loss L perceptual The loss function consists of the total loss L. total The calculation formula is: L total =L adv +λ1L pixel +λ2L perceptual Where λ1 and λ2 are balance coefficients; Pixel loss is calculated using the mean square error loss formula: Where H and W are the image height and width, respectively, and I gen For the generated intermediate image, I t Here, h represents the row index of a pixel in the target image, and w represents the column index of a pixel in the target image. The perceptual loss utilizes a pre-trained VGG network to extract high-level semantic features from the intermediate and target images, and calculates the feature difference using the following formula: Where l represents a specific layer of the VGG network, C l H l W l These represent the number of channels, height, and width of the feature map for this layer, respectively. and These are the feature maps of the intermediate image and the target image in this layer, respectively. m represents the row index of the pixel in the feature map of a specific layer of the VGG network, n represents the column index of the pixel in the feature map of a specific layer of the VGG network, and c represents the channel index of the pixel in the feature map of a specific layer of the VGG network. 4) Modality Transformation: The source image to be registered is input into the trained generator. Based on the mapping relationship between the source image and the target image modality learned during training, the generator transforms and reconstructs the features of the source image, and outputs an intermediate image I that is consistent with the modality of the target image. gen .
3. The multimodal image registration method based on generative models and self-supervised learning according to claim 1, characterized in that, The DINO-based self-supervised learning model extracts global and local features of the image, including: Intermediate Image I gen and target image I t The global encoder ε of the self-supervised learning model of DINO global The global features are extracted, and the specific calculation formula is as follows: f global (I gen )=ε global (I gen ),f global (I t )=ε global (I t ) Among them, I gen Represents the intermediate image, I t f represents the target image global (I gen ) represents the intermediate image I gen via global encoder ε global The extracted global features, f global (I t ) represents the target image I t via global encoder ε global The extracted global features; Intermediate Image I gen and target image I t The local encoder ε of the self-supervised learning model of DINO local The specific formula for extracting local features is as follows: f local (I gen )=ε local (I gen ),f local (I t )=ε local (I t ) Among them, f local (I gen ) represents the intermediate image I gen Through the local encoder ε local The extracted local features, f local (I t ) represents the target image I t Through the local encoder ε local Extracted local features; Normalize both global and local features: in, Representing global features f global (I gen The normalization result of ) Representing global features f global (I t The normalization result of ) Representing local features f local (I gen The normalization result of ) Representing local features f local (I t The normalized result of ).
4. The multimodal image registration method based on generative models and self-supervised learning according to claim 3, characterized in that, Image region segmentation is performed based on global and local features, dividing the image into key regions containing key structures and background regions, including: S301: Perform preliminary region division of the image based on the normalized global features; S302: Based on the normalized local features, further refine the region division according to the preliminary region division: If the local feature intensity of image region R is ||f local (R)||2 exceeds the set threshold θ key Then the area is designated as a critical area: For key areas If the local feature intensity of image region R is ||f local (R)||2 is lower than the set threshold θ bg Then this area is designated as the background area: Background area Through this process, both the intermediate image and the target image are divided into different regions R1, R2, ..., R... N N represents the number of regions, some of which are marked as critical regions and others as background regions.
5. The multimodal image registration method based on generative models and self-supervised learning according to claim 4, characterized in that, Based on the intensity of regional features and structural importance, key regions and background regions are weighted and a regional weighted registration matrix is established, including: S401: Calculate region weights: Where α and β are adjustment coefficients, importance(R) i ) is the region R i The structural importance is evaluated based on the semantic content and global features in the image, ||f local (R j )||2 represents region R j The local feature intensity, ||f local (R i )||2 represents region R i The intensity of local features, For regional weights; S402: Calculate the weighted registration matrix based on regional weights: Where W is the region-weighted registration matrix.
6. The multimodal image registration method based on generative models and self-supervised learning according to claim 1, characterized in that, High-precision nonlinear registration is performed on key areas, and its loss function is expressed as: Among them, R key As a key area, For the critical region R key The weights, T(R) key Let R' be the transformation function. key For target image I t The location of the corresponding region, the loss function L nonlinear Used to minimize transformation errors between regions; The loss function for low-precision linear registration of the background region can be expressed as: Among them, R bg For background area, Background region R bg The weights, T(R) bg ) for the background region R bg The result after applying the linear transformation, R′ bg For the target image with R bg The corresponding reference area location.
7. The multimodal image registration method based on generative models and self-supervised learning according to claim 6, characterized in that, Based on the region-weighted registration matrix, the registration parameters are optimized by minimizing the weighted loss function to solve for the optimal transformation matrix. The weighted loss function L... total as follows: The optimal transformation matrix T is obtained by minimizing the weighted loss function. * : Optimal transformation matrix T * The geometric transformation between the intermediate image and the target image is represented by T, where T is the geometric transformation matrix. The pixel p of the intermediate image s Through T * Transform to the position p of the target image t : Where, p s Use coordinates (x) s ,y s ) indicates that x s and y s These are the x and y coordinates of the pixel in the intermediate image, respectively. t Represents the pixel p in the target image and the intermediate image s The corresponding position points, after the optimal transformation matrix T * For p s After the transformation, the result is p. t Coordinates (x) in the target image t ,y t ).
8. A computer device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor, when executing the computer program, implements the method according to any one of claims 1 to 7.
9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 7.