Multi-modal image unsupervised registration method based on alternate optimization training

Abstract

The invention discloses a multi-modal image unsupervised registration method based on alternative optimization training. The method comprises the following steps: firstly, simulating to generate a same-mode image pair with a random deformation truth value; during training, the mode conversion module converts a source image mode into a target image mode, and the homography estimation registration module carries out deformation prediction on a to-be-registered image pair. The homography estimation registration module uses the source image pair and the target image pair which are subjected to mode conversion, and self-supervised training is carried out by utilizing a simulation deformation truth value. The mode conversion module enables the source image to be approximately aligned with the target image through the homography estimation registration module, and supervises training through perception loss. And carrying out verification enhancement training on the two modules in combination with additional simulation deformation, and optimizing the training effect. In the training process of the homography estimation registration module and the modal conversion module, gradient truncation and weight updating are independent and alternately optimized. According to the invention, unsupervised registration of multi-modal images can be realized, dependence on manual annotation data is reduced, and the precision of multi-modal image registration is improved.

Description

Technical Field

[0001] This invention relates to a multimodal image unsupervised registration method, which belongs to the field of image processing technology, and specifically to a multimodal image unsupervised registration method based on alternating optimization training. Background Technology

[0002] Multimodal image registration technology has broad application prospects in fields such as pedestrian detection, remote sensing analysis, medical imaging, and video stitching. However, multimodal images are usually acquired under different times and lighting conditions using different sensors, resulting in registration problems between images. Due to differences in imaging mechanisms, these images typically exhibit significant radiometric differences and geometric distortions, posing a significant challenge to the registration process.

[0003] Traditional registration methods, such as Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), and the Oriented Fast and Rotated BRIEF (ORB), an efficient alternative to SIFT or SURF, primarily rely on rich texture information and similar features in images to find correspondences between images through feature point matching and region matching. However, when dealing with multimodal images, due to the potentially severe deformations and modal differences between these images, traditional methods often struggle to achieve effective registration.

[0004] With the development of deep learning technology, data-driven methods are increasingly being applied to image registration. Through convolutional neural networks, deep learning can address, to some extent, the problems encountered by traditional methods in multimodal image registration. End-to-end deep learning methods can directly learn and predict the transformation relationships between images; these methods can be categorized into supervised and unsupervised learning. Supervised learning methods, such as deep learning-based image homography estimation networks (DHN), multiscale deep image homography estimation networks (MHN), and sparse optical flow (DLKFM), optimize the model by comparing the network's predictions with real registration parameters. However, obtaining a large number of real matching parameters in real-world scenarios is extremely difficult, limiting the application of supervised learning methods. In contrast, unsupervised learning methods, such as the unsupervised deep homography estimation network UDHN, the content-aware unsupervised deep homography estimation network CA-UDHN, and the motion basis unsupervised deep homography estimation network, are trained by using the similarity between images as a loss function, without relying on the true registration parameters. However, these methods still do not achieve ideal registration results for multimodal images with significant deformation and modal differences.

[0005] While deep learning methods have improved registration accuracy to some extent, most existing methods still rely on supervised training and require perfectly aligned data, which presents a data acquisition challenge in practical applications. Existing unsupervised methods also cannot effectively register multimodal images with significant deformation and modal differences. Therefore, developing unsupervised multimodal image registration methods capable of effective registration under limited data conditions has significant application value. Summary of the Invention

[0006] To address the problems existing in the background technology, the present invention provides a multimodal image unsupervised registration method based on alternating optimization training.

[0007] The technical solution adopted in this invention is:

[0008] The multimodal image unsupervised registration method based on alternating optimization training of the present invention includes:

[0009] Step S1: Acquire several initial images of different modalities using different sensors and perform image preprocessing to form several multimodal image pairs. Perform data augmentation on each multimodal image pair to obtain several same-modal image pairs to be registered.

[0010] Step S2: Establish an unsupervised registration training network and its unsupervised registration loss function. Train the input unsupervised registration training network with each image to be registered in the same modality until the unsupervised registration loss function converges, and obtain the trained unsupervised registration training network. Extract the unsupervised registration network that includes the first modality conversion module and the first homography estimation registration module.

[0011] Step S3: Collect multimodal image pairs to be registered using different sensors and input them into an unsupervised registration network. The processed output is a deformation variable. The deformation variable is used to apply homography deformation to the source image of one modality in the multimodal image pair to be registered, so that it is registered with the target image of another modality, thus achieving unsupervised registration of multimodal images.

[0012] In step S1, initial images of the first mode A and the second mode B, whose image content partially overlaps, are acquired by two sensors each time. Then, image preprocessing is performed. For each pair of initial images of the two modes, the initial images are adjusted to the same size and cropped. After normalization, the source image of mode A is obtained. and Mode B target image Form multimodal image pairs.

[0013] Data augmentation is performed on each multimodal image pair, specifically on the source image of modality A. and Mode B target image True values ​​of deformation were simulated using mode A. True values ​​of deformation in mode B simulation The initial deformation mode A source image is generated after applying homography deformation w. and the target image of the initial deformation mode B This allows us to determine the image pair to be registered for mode A. and the image pair to be registered in mode B As a pair of images to be registered in the same modality.

[0014] In step S2, the unsupervised registration training network includes three mode transformation modules, four homography estimation registration modules, and two feature extractors. The feature extractors are the feature extractors in the homography estimation registration modules; the unsupervised registration loss function is the homography estimation loss function. Mode transition loss function Homography estimation validation enhancement loss function Modality transformation verification enhancement loss function and feature-cooperative loss function The total loss function is formed by adding the two values ​​together.

[0015] During training, the unsupervised registration training network pairs the images to be registered for mode A. Modal A source image in and the source image of the initial deformation mode A The first conversion mode A source image is output after being processed by the first and second mode conversion modules. Second conversion mode A source image After gradient truncation, the input is processed in the second single-response estimation and registration module, and the output is the deformation prediction result of mode A. The image to be registered for mode B is paired with... Modal B target image and deformation mode B target image The third input should estimate the output of the registered module after processing, which will then predict the deformation of mode B. The deformation prediction results of mode A Modal B deformation prediction results True values ​​of deformation in Modal A simulation True values ​​of deformation in mode B simulation Input should estimate the loss function Self-supervised training is performed; the first transformation mode A source image is used for training. and Mode B target image The first homography estimation and registration module processes the input and outputs the homography matrix prediction result. Then, after gradient truncation, the source image of the first conversion mode A is processed. Applying homography deformation w generates an intermediate deformation mode A source image. , so that it is with Approximate registration and alignment are performed on the source image of intermediate deformation mode A. and Mode B target image Input mode transition loss function Perform mode conversion training; use the intermediate deformable mode A source image. and Mode B target image The source image features are processed by two feature extractors and then output as modality A. and Modal B target image features In the input feature collaborative loss function Perform feature-related constraint-based collaborative optimization training; use the modality A source image By pre-setting the true value of the simulation deformation After applying homography deformation w, a source image of the verification deformation mode A is generated. Then, the image is input into the third mode conversion module for processing and output as the third conversion mode A source image. Then, the true value of the simulation deformation is obtained by multiplying. The final deformation mode A source image is generated after applying homography deformation w. , The source image of the third conversion mode A and Mode B target image The fourth input should be estimated and the registered module should output the comprehensive deformation prediction result after processing. The final deformation mode A source image and Mode B target image Input should be estimated using the validation enhancement loss function In the middle, the comprehensive deformation prediction results will be used Simulation deformation true value of sum-multiplication Input mode transition verification enhancement loss function We will conduct validation enhancement training in that environment.

[0016] The homography estimation loss function Specifically as follows:

[0017]

[0018] in, This indicates the number of iterations required for the single-response registration module; Represents the homography estimation loss function Weighting coefficients; and They represent the first Image pairs in the next iteration Modal A deformation prediction results and image pairs Modal B deformation prediction results; This represents L1 norm operations.

[0019] The mode transition loss function Specifically as follows:

[0020]

[0021] in, , , and These represent the first and second generations of the pre-trained Visual Geometry Group (VGG) network. The perceptual feature map extracted from the layer, along with its length, height, and width; This represents L2 norm operations.

[0022] The homography estimation verification enhancement loss function Mode transformation verification enhancement loss function Specifically as follows:

[0023]

[0024]

[0025] in, This indicates the number of iterations required for the single-response registration module; Represents the homography estimation loss function Weighting coefficients; Indicates the first Image pairs in the next iteration Overall deformation prediction results; , , and These represent the first and second generations of the pre-trained Visual Geometry Group (VGG) network. The perceptual feature map extracted from the layer, along with its length, height, and width.

[0026] The Feature Collaborative Loss Function Specifically as follows:

[0027]

[0028] in, and These represent the two-dimensional spatial coordinate indices. The feature points are in the source image features of mode A and the target image features of mode B; T represents matrix transformation.

[0029] The electronic device of the present invention includes: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor invokes the program data to execute the method described above.

[0030] The present invention provides a computer-readable storage medium having program data stored thereon, which, when executed by a processor, implements the method described above.

[0031] This invention first simulates and generates homomodal image pairs with ground truth values ​​for random deformation. During training, the modality conversion module converts the source image modality into the target image modality, and the homography estimation and registration module predicts deformation in the image pairs to be registered. The homography estimation and registration module uses the modality-converted source and target image pairs and performs self-supervised training using the simulated deformation ground truth values. The modality conversion module approximates the alignment of the source and target images through the homography estimation and registration module and uses perceptual loss for supervised training. Additional simulated deformation is used to validate and enhance the training of both modules, optimizing the training effect. During the training of the homography estimation and registration module and the modality conversion module, gradient truncation is performed, and weight updates are independent and alternately optimized. This invention enables unsupervised registration of multimodal images, reduces reliance on manually labeled data, and improves the accuracy of multimodal image registration.

[0032] The beneficial effects of this invention are:

[0033] This invention significantly improves the accuracy of multimodal image registration by introducing an alternating optimization training mechanism. This mechanism decomposes the complex problem of multimodal image registration into two more manageable sub-problems, with the homography estimation registration module and the modality transformation module updating their weights alternately during training. This approach effectively avoids performance degradation caused by gradient flow instability during co-training. The modality transformation module effectively reduces modal differences between multimodal images, while the homography estimation registration module ensures the accuracy of registration results under severe deformation conditions, ultimately yielding accurate multimodal image registration results. This invention employs an unsupervised learning method, eliminating reliance on manually labeled data, thereby reducing data preparation workload and costs and improving automation. This method demonstrates high flexibility in practical applications and is applicable to various image modalities. Whether in medical image processing, remote sensing image analysis, or other fields requiring multimodal image registration, this invention provides an efficient and accurate solution with broad application prospects. Attached Figure Description

[0034] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0035] Figure 2 This is a diagram illustrating the overall framework of the unsupervised registration training network according to an embodiment of the present invention.

[0036] Figure 3 This is a network framework diagram of the homography estimation and registration module and the mode conversion module in an embodiment of the present invention;

[0037] Figure 4 Here is an example image illustrating the application of the method of this invention to a visible-infrared dataset, wherein... Figure 4 (a) is the visible light image to be registered. Figure 4 (b) is the corresponding infrared image. Figure 4(c) is the registered and corrected visible light image;

[0038] Figure 5 Here is an example image demonstrating the application of the method of this invention to a visible light-depth dataset, wherein... Figure 5 (a) is the visible light image to be registered. Figure 5 (b) is the corresponding depth image. Figure 5 (c) is the registered and corrected visible light image. Detailed Implementation

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] like Figure 1 As shown, the multimodal image unsupervised registration method based on alternating optimization training of the present invention is as follows:

[0041] First, initial images of the first mode A and the second mode B, whose image content overlaps, are acquired using two sensors each time. Then, image preprocessing is performed, adjusting the initial images for each pair of initial images from the two modes to... Cut to the same size, then cut out pieces of the same size. Within a square region, the image content of the cropped image pairs must overlap by more than 50%, and then normalization processing is performed to obtain the modality A source image. and Mode B target image Combining multimodal image pairs to create a multimodal dataset. , , and These represent the height and width of the image, respectively. The source image is a visible light image, and the target image is an infrared image or a depth image.

[0042] Data augmentation is performed on each multimodal image pair, specifically on the source image of modality A. and Mode B target image True values ​​of deformation were simulated using mode A. True values ​​of deformation in mode B simulation The initial deformation mode A source image is generated after applying homography deformation w. and the target image of the initial deformation mode B This allows us to determine the image pair to be registered for mode A. and the image pair to be registered in mode B As a pair of images to be registered in the same modality.

[0043] Then, an unsupervised registration training network and its unsupervised registration loss function are established, such as... Figure 2As shown, this is the overall framework of the unsupervised registration training network. The unsupervised registration training network includes three mode transformation modules, four homography estimation and registration modules, and two feature extractors, as follows: Figure 3 As shown, the homography estimation and registration module adopts an iterative architecture, including two feature extractors and one homography estimator. After inputting the image to be registered, the feature extractors with shared parameters extract features useful for registration, and the images of different modalities are then registered. The outputs of the two feature extractors are fed into the inner product, i.e., feature extraction is performed using a parameter-shared Siamese convolutional neural network; then, global feature correlation calculations are performed on the features to generate... The feature correlation body is sampled to obtain relevant features, and the deformation residual is estimated multiple times using a homography estimator. n is the number of iterations, and the final deformation prediction result is obtained. The output of the homography estimation and registration module is used as the feature extractor. Specifically, the Iterative Homography Network (IHN) deep learning architecture for image geometric transformation estimation is selected as the homography estimation and registration module. The IHN feature extractor structure includes an initial feature extraction unit consisting of a 7×7 convolutional layer, an instance normalization layer, and a ReLU activation function layer connected in sequence, and two stacked residual blocks consisting of a 2×2 max pooling layer, two 3×3 convolutional layers, an instance normalization layer, and a ReLU activation function layer. The IHN homography estimator consists of a multi-level homography aggregation module, each consisting of a 3×3 convolutional layer, a group normalization layer, a ReLU activation function layer, and a max pooling layer. Simulated isomorphic data is input into the homography estimation and registration module for self-supervised training. The input isomorphic images to be registered are then processed. and First of all Input the modality conversion module to obtain modality-converted image pairs. And perform gradient truncation, then with The input is a single-response estimation and registration module, which uses a parameter-shared Siamese convolutional neural network to extract features, resulting in a model with dimensions of [missing information]. The convolutional features are then used to calculate feature correlation, resulting in a size of [size missing]. The relevant bodies and according to the sampling radius Sampling was performed to obtain a size of The relevant volume sampling results are input into the motion estimation network to iteratively calculate the residuals and final results of deformation prediction; the mode conversion module includes an encoder, a bottleneck layer and a decoder connected in sequence, with an overall U-shaped structure, where the basic unit of the encoder and decoder is the layered architecture Swing Transformer module.

[0044] The unsupervised registration loss function is the homography estimation loss function. Mode transition loss function Homography estimation validation enhancement loss function Modality transformation verification enhancement loss function and feature-cooperative loss function The summation constitutes the total loss function. After generating two pairs of identical modal images for registration using simulated deformation ground truth, the multimodal image pairs will be... Modal images are converted using a modal conversion module. Then, both the multimodal image pairs and the simulated same-modal image pairs are input into a homography estimation and registration module to obtain the corresponding predicted deformation. The homography estimation and registration module uses the simulated deformation true values. and Self-supervised training is performed, and the loss function is the homography estimation loss function. The details are as follows:

[0045]

[0046] in, This indicates the number of iterations for the single-adaptation registration module, specifically set to 6. Represents the homography estimation loss function The weighting coefficient is specifically set to 0.8; and They represent the first Image pairs in the next iteration Modal A deformation prediction results and image pairs Modal B deformation prediction results; This represents L1 norm operations.

[0047] Homography estimation loss function The homography estimation and registration module is trained by self-supervised training using the true values ​​of deformation from multimodal simulation.

[0048] For the mode transformation module, the deformation prediction results obtained from the estimation are used. After approximately aligning the multimodal images, a perceptual loss is calculated for training. The loss function is the modality transformation loss function. The details are as follows:

[0049]

[0050] in, , , and These represent the first and second generations of the pre-trained Visual Geometry Group (VGG) network. The perceptual feature map extracted from the layer, along with its length, height, and width, is specifically implemented in this way. This indicates the use of a pre-trained VGG network. The perceptual features extracted from the layer have a feature map size of [size missing]. ; This represents L2 norm operations.

[0051] Mode transition loss function Defined as a perceptual loss based on VGG networks, utilizing approximately aligned... and To supervise the training of the modality conversion module.

[0052] Simultaneously utilize Combined with additional simulated deformation The pseudo-deformation true values ​​are used to perform validation enhancement training on the homography estimation registration module and the mode transformation module, respectively. The loss function is the homography estimation validation enhancement loss function. Mode transformation verification enhancement loss function The details are as follows:

[0053]

[0054]

[0055] in, This indicates the number of iterations required for the single-response registration module; Represents the homography estimation loss function Weighting coefficients; Indicates the first Image pairs in the next iteration Overall deformation prediction results; , , and These represent the first and second generations of the pre-trained Visual Geometry Group (VGG) network. The perceptual feature map extracted from the layer, along with its length, height, and width.

[0056] Homography estimation validation enhancement loss function Mode transformation verification enhancement loss function Using the true value of simulated deformation and deformation results Generating pseudo-deformation true values ​​allows for validation and enhanced training of the homography estimation registration module and mode transformation module.

[0057] The feature extractor of the homography estimation registration module is used to obtain the features of the aligned image pairs. and The collaborative optimization module for homography estimation and registration includes a feature extractor and a mode transformation module, with the loss function being the feature collaborative loss function. The details are as follows:

[0058]

[0059] in, and These represent the two-dimensional spatial coordinate indices. The feature points are in the source image features of mode A and the target image features of mode B; T represents matrix transformation.

[0060] By using a loss function based on feature correlation operations, the feature extractor and modality conversion module of the homography estimation and registration module are co-optimized to enhance the feature consistency of multimodal image conversion results in the homography estimation and registration module.

[0061] The images to be registered in each modality are input into the unsupervised registration training network for training. During training, the unsupervised registration training network inputs the images to be registered in modality A into the network. Modal A source image in and the source image of the initial deformation mode A The first conversion mode A source image is output after being processed by the first and second mode conversion modules. Second conversion mode A source image After gradient truncation, the input is processed in the second single-response estimation and registration module, and the output is the deformation prediction result of mode A. The image to be registered for mode B is paired with... Modal B target image and deformation mode B target image The third input should estimate the output of the registered module after processing, which will then predict the deformation of mode B. The deformation prediction results of mode A Modal B deformation prediction results True values ​​of deformation in Modal A simulation True values ​​of deformation in mode B simulation Input should estimate the loss function Self-supervised training is performed; the first transformation mode A source image is used for training. and Mode B target image The first homography estimation and registration module processes the input and outputs the homography matrix prediction result. Then, after gradient truncation, the source image of the first conversion mode A is processed. Applying homography deformation w generates an intermediate deformation mode A source image. , so that it is with Approximate registration and alignment are performed on the source image of intermediate deformation mode A. and Mode B target image Input mode transition loss function Perform mode conversion training; use the intermediate deformable mode A source image. and Mode B target image The source image features are processed by two feature extractors and then output as modality A. and Modal B target image features In the input feature collaborative loss function Perform feature-related constraint-based collaborative optimization training; use the modality A source image By pre-setting the true value of the simulation deformation After applying homography deformation w, a source image of the verification deformation mode A is generated. Then, the image is input into the third mode conversion module for processing and output as the third conversion mode A source image. Then, the true value of the simulation deformation is obtained by multiplying. The final deformation mode A source image is generated after applying homography deformation w. , The source image of the third conversion mode A and Mode B target image The fourth input should be estimated and the registered module should output the comprehensive deformation prediction result after processing. The final deformation mode A source image and Mode B target image Input should be estimated using the validation enhancement loss function In the middle, the comprehensive deformation prediction results will be used Simulation deformation true value of sum-multiplication Input mode transition verification enhancement loss function The process involves validation and enhancement training; repeating the steps and alternately optimizing the modality conversion module and the homography estimation and registration module until the total loss function converges or the number of repetitions reaches the preset number of training iterations. This completes the training and yields the final model. The resulting model is then used for multimodal image registration, i.e., repeating the process of the first modality conversion module and the first homography estimation and registration module, until the registered image is obtained. .

[0062] Alternately optimize the two modules until the unsupervised registration loss function converges or the set number of repetitions is reached (default is...). Next, the learning rate is set to [value] during training. The training process yields a fully trained unsupervised registration network, which includes a first modality transformation module and a first homography estimation registration module, to achieve multimodal image registration. Except for the collaborative optimization training of the feature extractor and modality transformation module in the homography estimation registration module, the inputs to the other homography estimation registration and modality transformation modules are gradient-truncated during training, and their weight updates are independent. The two modules are optimized alternately and work together to improve each other. The modality transformation module reduces modal differences in multimodal images, while the homography estimation registration module estimates accurate deformation prediction results. Finally, the two modules are combined to achieve multimodal image registration.

[0063] Finally, multimodal image pairs to be registered are acquired using different sensors and input into an unsupervised registration network. The processed output is a deformation variable. This deformation variable is used to apply homography deformation to the source image of one modality in the multimodal image pair to achieve registration with the target image of the other modality, thus realizing unsupervised multimodal image registration. In specific implementations, the actual registration effect and advantages of this invention are demonstrated through specific scenarios. For example... Figure 4 of (a) Figure 4 (b) and Figure 4 As shown in (c), this demonstrates the application effect of the present invention on a visible-infrared dataset. The visible light image to be registered exhibits significant viewpoint shift and geometric distortion, while the corresponding infrared image has completely different modal characteristics. After registration using the method of the present invention, the visible light image is successfully corrected to a state of complete alignment with the infrared image, achieving precise matching of roads, buildings, and vehicles; as shown in (c), the visible light image is perfectly aligned with the infrared image, achieving precise matching of roads, buildings, and vehicles. Figure 5 of (a) Figure 5 (b) and Figure 5 As shown in (c), the performance on the visible light-depth dataset is demonstrated. The visible light images and their corresponding depth images in the dataset have completely different imaging characteristics. After registration by the method of this invention, the visible light images and depth maps achieve high-precision alignment.

[0064] As shown in Table 1, the registration accuracy of the method of the present invention was compared with that of the baseline method without the introduction of alternating optimization in the specific implementation of the present invention. The average corner error represents the average error between the offset of the four corner points of the image obtained by registration prediction and the offset of the corner points of the registration ground truth. The lower the value, the higher the registration accuracy. It can be seen that the registration error of the method of the present invention is reduced by 55.73% and 87.73% respectively compared with the baseline method on the visible light-infrared image dataset and the visible light-depth dataset, showing a significant advantage in registration accuracy.

[0065] Table 1

[0066] Registration method Average corner error of visible-infrared dataset Mean corner error of visible light-depth dataset Baseline method 6.37 9.13 Method of the present invention 2.82 1.12

[0067] The embodiments described above are merely some preferred embodiments of the present invention, and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A multimodal image unsupervised registration method based on alternating optimization training, characterized in that, include: Step S1: Acquire several initial images of different modalities using different sensors and perform image preprocessing to form several multimodal image pairs. Perform data augmentation on each multimodal image pair to obtain several same-modal image pairs to be registered. Step S2: Establish an unsupervised registration training network and its unsupervised registration loss function. Train the input unsupervised registration training network with each image to be registered in the same modality until the unsupervised registration loss function converges. Obtain the trained unsupervised registration training network and extract the unsupervised registration network including the first modality conversion module and the first homography estimation registration module. Step S3: Collect multimodal image pairs to be registered using different sensors and input them into an unsupervised registration network. The processed output is a deformation variable. The deformation variable is used to apply homography deformation to the source image of one modality in the multimodal image pair to be registered, so that it is registered with the target image of another modality, thus achieving unsupervised registration of multimodal images.

2. The multimodal image unsupervised registration method based on alternating optimization training according to claim 1, characterized in that: In step S1, initial images of the first mode A and the second mode B, whose image content overlaps, are acquired by two sensors each time. Then, image preprocessing is performed. For each pair of initial images of the two modes, the initial images are adjusted to the same size and cropped. After normalization, a multimodal image pair is obtained by combining the source image of mode A and the target image of mode B. For each multimodal image pair, data augmentation is performed. Homography deformation is applied to the source image of mode A and the target image of mode B using the simulated deformation ground values ​​of mode A and mode B, respectively, to generate the initial deformed source image of mode A and the initial deformed target image of mode B. Then, the image pairs to be registered in mode A and mode B are determined as the same modal image pairs to be registered.

3. The multimodal image unsupervised registration method based on alternating optimization training according to claim 2, characterized in that: In step S2, the unsupervised registration training network includes three mode transformation modules, four homography estimation registration modules, and two feature extractors. The feature extractors are the feature extractors in the homography estimation registration modules. The unsupervised registration loss function is the total loss function composed of the homography estimation loss function, the mode transformation loss function, the homography estimation verification enhancement loss function, the mode transformation verification enhancement loss function, and the feature collaboration loss function.

4. The multimodal image unsupervised registration method based on alternating optimization training according to claim 3, characterized in that: During training, the unsupervised registration training network inputs the source image of Mode A and the initial deformed source image of Mode A from the image pair to be registered into the first and second mode conversion modules for processing, outputting the first converted source image of Mode A and the second converted source image of Mode A. After gradient truncation, the input is input into the second single-response estimation registration module for processing, outputting the deformation prediction result of Mode A. The target image of Mode B and the deformed target image of Mode B from the image pair to be registered into the third single-response estimation registration module for processing, outputting the deformation prediction result of Mode B. The deformation prediction results of Mode A, the deformation prediction results of Mode B, the simulated deformation ground truth of Mode A, and the simulated deformation ground truth of Mode B are input into the single-response estimation loss function for self-supervised training. The first transformation modality A source image and modality B target image are input into the first homography estimation and registration module for processing, and the homography matrix prediction result is output. Then, after gradient truncation, homography deformation is applied to the first transformation modality A source image to generate an intermediate deformed modality A source image. The intermediate deformed modality A source image and modality B target image are input into the modality transformation loss function for modality transformation training. The intermediate deformed modality A source image and modality B target image are input into two feature extractors respectively for processing, and the output modality A source image features and modality B target image features are input into the feature co-loss function for feature correlation constraint co-optimization training. The modality A source image is then processed... After applying homography deformation to the preset simulated deformation true value, a verification deformation mode A source image is generated. This image is then processed in the third mode conversion module and output as the third converted mode A source image. Next, homography deformation is applied by multiplying the simulated deformation true value to generate the final deformation mode A source image. The third converted mode A source image and the mode B target image are then processed in the fourth homography estimation and registration module and output as the comprehensive deformation prediction result. The final deformation mode A source image and the mode B target image are then input into the homography estimation verification enhancement loss function. Finally, the comprehensive deformation prediction result and the multiplied simulated deformation true value are input into the mode conversion verification enhancement loss function for verification enhancement training.

5. The multimodal image unsupervised registration method based on alternating optimization training according to claim 4, characterized in that: The homography estimation loss function Specifically as follows: in, This indicates the number of iterations required for the single-response registration module; Represents the homography estimation loss function Weighting coefficients; and They represent the first The deformation prediction results for mode A and mode B in the next iteration; and These represent the true values ​​of the simulated deformation in mode A and mode B, respectively. This represents L1 norm operations.

6. The multimodal image unsupervised registration method based on alternating optimization training according to claim 4, characterized in that: The mode transition loss function Specifically as follows: in, , , and These represent the first and second generations of the pre-trained Visual Geometry Group (VGG) network. The perceptual feature map extracted from the layer, along with its length, height, and width; Represents the source image of intermediate deformation mode A; Represents the target image of mode B; This represents L2 norm operations.

7. The multimodal image unsupervised registration method based on alternating optimization training according to claim 4, characterized in that: The homography estimation verification enhancement loss function Mode transformation verification enhancement loss function Specifically as follows: in, This indicates the number of iterations required for the single-response registration module; Represents the homography estimation loss function Weighting coefficients; Indicates the first The comprehensive deformation prediction results of the next iteration; This represents the prediction result of the homography matrix; This represents the preset simulation deformation true value; This represents the source image of the final deformation mode A; Represents the target image of mode B; , , and These represent the first and second generations of the pre-trained Visual Geometry Group (VGG) network. The perceptual feature map extracted from the layer, along with its length, height, and width.

8. The multimodal image unsupervised registration method based on alternating optimization training according to claim 4, characterized in that: The Feature Collaborative Loss Function Specifically as follows: in, and These represent the two-dimensional spatial coordinate indices. The feature points are in the source image features of mode A and the target image features of mode B; T represents matrix transformation.

9. An electronic device, characterized in that, include: A memory and a processor are coupled to each other, wherein the memory stores program data, and the processor invokes the program data to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium storing program data thereon, characterized in that, When the program data is executed by the processor, it implements the method as described in any one of claims 1-8.

Images