A multimodal image registration method and system based on iterative optimization
By iteratively optimizing the multimodal image registration method, using neural network model to process low-light images, the accuracy and stability problems caused by feature differences in multimodal image registration are solved, and a more efficient image registration effect is achieved.
Patent Information
- Application Number
- CN202411288067.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-14
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-09-14
AI Technical Summary
There are significant differences in radiation and geometric features of multimodal images, which leads to difficulty in matching high-precision. How to effectively extract and describe these feature maps is a challenge, affecting the accuracy and stability of multimodal image registration.
Using a multimodal image registration method based on iterative optimization, image enhancement processing is carried out by acquiring low-light images, neural network model including pyramid information extraction module, neighborhood attention module and multi-layer perceptron module is built, and the model is trained and optimized. To obtain the image conversion parameter matrix, image registration fusion is realized.
It improves the robustness and accuracy of multimodal image registration, enhances the alignment effect of image pairs, and improves the perceived accuracy and robustness of the system.
Smart Images

Figure CN118799369B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image data processing, and in particular to a multimodal image registration method and system based on iterative optimization. Background Art
[0002] Nowadays, images acquired by sensors on multiple platforms are becoming more and more abundant. The images acquired by sensors on multiple platforms are collectively referred to as multimodal images. Multimodal image registration can be used in positioning systems by matching images acquired by different sensors with reference images. Multimodal image registration can utilize different types of perception data to achieve information complementarity. By fusing these perception data, the system can understand the environment more comprehensively, thereby improving the accuracy and robustness of perception.
[0003] However, multimodal images have significant differences in radiation and geometric features, which makes high-precision matching difficult. Therefore, multimodal image registration requires feature mapping of different types of image data, but different types of images may have different features and distributions. How to effectively extract and describe these feature maps is a challenge. Summary of the invention
[0004] The technical problem to be solved by the present invention is to improve the accuracy and stability of multimodal image registration. In order to achieve the above object, the present invention provides a multimodal image registration method based on iterative optimization, which comprises the following steps:
[0005] S1, obtaining a number of low-light images and performing image enhancement processing, dividing the enhanced images into to-be-matched images and reference images to form a number of image pairs, and dividing the number of image pairs into a training set and a verification set;
[0006] S2. Build a multimodal image registration neural network model, which includes a pyramid information extraction module, a neighborhood attention module and a multi-layer perceptron module connected in sequence, wherein the pyramid information extraction module is used to receive and extract a feature map of an input image pair, the neighborhood attention module is used to receive and process the feature map to obtain a compressed feature map, and the multi-layer perceptron module is used to receive and process the compressed feature map to obtain an image conversion parameter matrix of the input image pair;
[0007] S3, training the multimodal image registration neural network model through the training set, setting the loss function to calculate the loss value of each image pair in the training set and updating the network parameters to obtain the trained multimodal image registration neural network model;
[0008] S4, presetting an iteration termination condition, using a validation set to iteratively optimize the trained multimodal image registration neural network model and determining whether the iteration termination condition is satisfied, thereby obtaining an iteratively optimized multimodal image registration neural network model;
[0009] S5. Acquire multimodal images in real scenes and form image pairs to be registered, input the image pairs to be registered into the iteratively optimized multimodal image registration neural network model for processing, obtain the image conversion parameter matrix of the image pairs to be registered, convert the image to be matched in the image pair to be registered according to the image conversion parameter matrix, and register and fuse the converted image to be matched with the reference image in the image pair to be registered to obtain the registered and fused image.
[0010] Preferably, in S1, a low-light image is acquired and image enhancement processing is performed. The image enhancement processing can be specifically expressed by the formula:
[0011] ;
[0012] In the formula, is the average gray value of the window, is the magnification, , For the original low-light image in pixels The gray value at The enhanced image is in pixels Gray value.
[0013] Preferably, S3 specifically includes the following steps:
[0014] S31, arbitrarily selecting an image pair from the training set and inputting the image pair into the pyramid information extraction module for processing, to obtain a feature map of the image to be matched and a feature map of the reference image in the selected image pair;
[0015] S32, inputting the feature map of the image to be matched and the feature map of the reference image into the neighborhood attention module and calculating the correlation between the feature maps to obtain a neighborhood cost volume, performing channel compression on the neighborhood cost volume to obtain a compressed feature map;
[0016] S33, inputting the compressed feature map into the multi-layer perceptron module for processing, and outputting the image conversion parameter matrix of the selected image pair;
[0017] S34, performing image conversion on the image to be matched in the selected image pair using the image conversion parameter matrix to obtain an image after the image to be matched is converted, and extracting a feature map of the image after the image to be matched is converted using a pyramid information extraction module;
[0018] S35, setting a loss function according to the feature map of the reference image in the selected image pair, the feature map of the image to be matched, and the feature map of the image converted from the image to be matched, and calculating the loss value of the selected image pair;
[0019] S36. Select another image pair from the training set until all the image pairs in the training set are selected, repeat steps S31 to S35, obtain the loss value of each image pair in the training set, select the network parameters with the smallest loss value to update the multimodal image registration neural network model, and obtain the trained multimodal image registration neural network model.
[0020] Preferably, the feature map after compression in S32 can be specifically expressed by the formula:
[0021] ;
[0022] In the formula, represents the compressed feature map, represents a 1×1 convolutional layer, which is used to perform channel compression on the neighborhood cost volume. represents the neighborhood attention module, and They represent the feature map of the image to be matched and the feature map of the reference image in the training set image pair respectively.
[0023] Preferably, the image conversion parameter matrix in S33 can be specifically expressed by the formula:
[0024] ;
[0025] In the formula, represents the image transformation parameter matrix for the image pairs in the training set, represents a multi-layer perceptron module, Indicates that the compressed feature map Flatten to a one-dimensional vector.
[0026] Preferably, the loss function in S35 is specifically:
[0027] ;
[0028] In the formula, Represents the image to be matched in the training set and reference image The loss value of the composed image pair, and Represent the height and width of the feature map, respectively. represents the 1 norm, Represents the image to be matched in the training set image pair The feature map In the Line The value of the column, Represents the reference image in the training set image pair The feature map In the Line The value of the column, Represents the image to be matched in the training set image pair Feature map of the transformed image In the Line The value of the column, .
[0029] Preferably, S4 specifically includes the following steps:
[0030] S41, setting an iteration termination condition, presetting the true coordinates of the reference image in the validation set image pair, and performing n-times iterative optimization on the trained multimodal image registration neural network model;
[0031] S42, selecting one image pair from the validation set at a time, inputting the selected image pair into the trained multimodal image registration neural network model for processing, obtaining an image conversion parameter matrix for the selected image pair, and using the image conversion parameter matrix to convert the image to be matched in the selected image pair to obtain an image after the image to be matched is converted;
[0032] S43, selecting another image pair from the verification set again, until every image pair in the verification set is selected, repeating step S42, and obtaining the image after the image to be matched in each image pair in the verification set is converted;
[0033] S44, calculating the difference between the pixel coordinates of each converted image of the to-be-matched image in the verification set image pair and the real coordinates of the reference image corresponding to each to-be-matched image, and taking the average of all the differences;
[0034] S45, judging whether the iteration termination condition is satisfied according to the average value of all the differences, if not satisfied, executing steps S42 to S44, performing n+1 iteration optimization on the trained multimodal image registration neural network model, otherwise, executing step S46;
[0035] S46, ending the iterative optimization and obtaining the iteratively optimized multimodal image registration neural network model.
[0036] Preferably, S44 can be specifically expressed by the formula:
[0037] ;
[0038] In the formula, Represents the average value of the difference calculated after each round of iterative optimization. Indicates the validation set The image to be matched in the image pairs The coordinate value of Indicates the validation set Image alignment reference image The real coordinate value of Represents the image conversion process, Indicates the validation set The image transformation parameter matrix for image pairs, represents the number of image pairs in the validation set, .
[0039] Preferably, the pyramid information extraction module in S2 includes a first convolution module, a normalization layer, an activation function layer and a second convolution module connected in sequence, wherein the first convolution module includes three convolution layers connected in sequence, the step size of each convolution layer is 2, and the convolution kernel size is 3×3, and the second convolution module includes one convolution layer, the step size of the convolution layer is 1, and the convolution kernel size is 1×1.
[0040] Another aspect of the present invention provides a multimodal image registration system, which includes an image acquisition unit and a computer system connected to the image acquisition unit, wherein the computer system is provided with a processing unit, a storage unit and a multimodal image registration neural network model connected in sequence, wherein:
[0041] The image acquisition unit is used to acquire multimodal images and send the multimodal images to the computer system;
[0042] The computer system receives the multimodal image and sends it to the processing unit, and the processing unit divides the multimodal image into reference images and images to be matched, and forms image pairs and then sends them to the storage unit;
[0043] The storage unit is used to store the image pairs and the parameters of the multimodal image registration neural network model;
[0044] The multimodal image registration neural network model receives the image pair and performs multimodal image registration using the multimodal image registration method based on iterative optimization to obtain a registered fused image.
[0045] The above-mentioned multimodal image registration method and system based on iterative optimization obtains several low-light images and performs image enhancement processing, divides the enhanced images into images to be matched and reference images to form several image pairs, divides the several image pairs into training sets and verification sets, builds a multimodal image registration neural network model, trains the multimodal image registration neural network model through the training set, and uses the loss function to calculate the loss to obtain the trained multimodal image registration neural network model, presets the iteration termination condition, uses the verification set to iteratively optimize the trained multimodal image registration neural network model and judges whether the iteration termination condition is met to obtain the iteratively optimized multimodal image registration neural network model, finally obtains the multimodal images to be registered in the real scene and forms image pairs, inputs the image pairs into the iteratively optimized multimodal image registration neural network model for processing, obtains the image conversion parameter matrix corresponding to the input image pair, converts the image to be matched in the input image pair according to the image conversion parameter matrix, and registers and fuses the converted image to be matched with the reference image in the input image pair to obtain the registered fused image. This method can improve the robustness of the image pair consisting of the reference image and the image to be matched during the registration process. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 is a flow chart of a multimodal image registration method based on iterative optimization in one embodiment of the present invention;
[0047] Figure 2 is a schematic diagram of the working process of the neighborhood attention module in one embodiment of the present invention;
[0048] Figure 3 It is a schematic diagram of a multimodal image registration process in a real scene in one embodiment of the present invention. DETAILED DESCRIPTION
[0049] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.
[0050] See also Figure 1 , Figure 1 The flowchart of a multimodal image registration method based on iterative optimization in one embodiment of the present invention. A multimodal image registration method based on iterative optimization comprises the following steps:
[0051] S1. Acquire several low-light images and perform image enhancement processing, divide the enhanced images into to-be-matched images and reference images to form several image pairs, and divide the several image pairs into a training set and a verification set.
[0052] Specifically, image enhancement processing is to use local contrast methods to increase the brightness of low-light images.
[0053] Furthermore, the image enhancement processing can be specifically expressed by the formula:
[0054] (1)
[0055] In the formula, is the average gray value of the window, is the magnification, , For the original low-light image in pixels The gray value at The enhanced image is in pixels Gray value.
[0056] At magnification When ,but ;if ,but .
[0057] S2. Build a multimodal image registration neural network model, which includes a pyramid information extraction module, a neighborhood attention module and a multi-layer perceptron module connected in sequence. The pyramid information extraction module is used to receive and extract the feature map of the input image pair, the neighborhood attention module is used to receive and process the feature map to obtain the compressed feature map, and the multi-layer perceptron module is used to receive and process the compressed feature map to obtain the image conversion parameter matrix of the input image pair.
[0058] Specifically, the multimodal image registration neural network model includes a pyramid information extraction module, a neighborhood attention module, and a multilayer perceptron module which are connected in sequence, wherein the pyramid information extraction module is used to extract the feature map of the input image pair, the neighborhood attention module is used to calculate the neighborhood cost volume according to the feature map of the input image pair, and the neighborhood cost volume is channel-compressed to obtain the feature map after channel compression, and the multilayer perceptron module is used to calculate the image conversion parameter matrix according to the feature map after channel compression.
[0059] Furthermore, the pyramid information extraction module in S2 includes a first convolution module, a normalization layer, an activation function layer and a second convolution module connected in sequence, wherein the first convolution module includes three convolution layers connected in sequence, each convolution layer has a step size of 2, and a convolution kernel size of 3×3, and the second convolution module includes a convolution layer, the step size of the convolution layer is 1, and the convolution kernel size is 1×1.
[0060] The first convolution module is used to obtain rich contextual information, improve the learning representation ability of the module, and obtain feature maps with rich semantic information. The normalization layer makes the data distribution more stable, which is conducive to improving the stability and generalization ability of network training. The second convolution module is used to reduce the number of channels of the feature map to reduce the computational complexity of the subsequent neighborhood attention module.
[0061] S3. Train the multimodal image registration neural network model through the training set, set the loss function to calculate the loss value of each image pair in the training set and update the network parameters to obtain the trained multimodal image registration neural network model.
[0062] In one embodiment, S3 specifically includes the following steps:
[0063] S31, randomly selecting an image pair from the training set and inputting the image pair into the pyramid information extraction module for processing, so as to obtain a feature map of the image to be matched and a feature map of the reference image in the selected image pair.
[0064] Furthermore, the feature map of the image to be matched and the feature map of the reference image in S31 can be specifically expressed by the formula:
[0065] (2)
[0066] In the formula, represents the pyramid information extraction module, and Respectively represent the image to be matched and the reference image in the training set image pair, and They represent the feature map of the image to be matched and the feature map of the reference image in the training set image pair respectively.
[0067] S32. Input the feature map of the image to be matched and the feature map of the reference image into the neighborhood attention module and calculate the correlation between the feature maps to obtain the neighborhood cost volume, perform channel compression on the neighborhood cost volume, and obtain a compressed feature map.
[0068] Furthermore, the compressed feature map in S32 can be specifically expressed by the formula:
[0069] (3)
[0070] In the formula, represents the compressed feature map, represents a 1×1 convolutional layer, which is used to perform channel compression on the neighborhood cost volume. Represents the neighborhood attention module.
[0071] Specifically, see Figure 2 , Figure 2This is a schematic diagram of the working process of the neighborhood attention module in one embodiment of the present invention. The value of a certain position in and the feature map of the reference image At its corresponding position and with a radius of Perform multiplication operations in the neighborhood of , calculate the correlation between feature maps, and obtain the neighborhood cost volume :
[0072] (4)
[0073] in, Represents the feature map of the image to be matched The value of a certain position in Feature map representing the reference image At the corresponding position and with a radius of A value in the neighborhood of Indicated in In this embodiment, Take 3.
[0074] Use a 1×1 convolution layer to reduce the neighborhood cost volume The number of channels (that is, the number of channels for the neighborhood cost volume Channel compression is performed), thereby obtaining the compressed feature map .
[0075] S33, inputting the compressed feature map into the multi-layer perceptron module for processing, and outputting the image conversion parameter matrix of the selected image pair.
[0076] Furthermore, the image conversion parameter matrix in S33 can be specifically expressed by the formula:
[0077] (5)
[0078] In the formula, represents the image transformation parameter matrix for the image pairs in the training set, represents a multi-layer perceptron module, Indicates that the compressed feature map Flatten to a one-dimensional vector.
[0079] Specifically, the multi-layer perceptron module It includes three fully connected layers connected in sequence. The size of the first fully connected layer is The size of the second fully connected layer is 32, and the size of the third fully connected layer is 8. The image transformation parameter matrix is regressed and output through three fully connected layers of different sizes. The image transformation parameter matrix is used to realize the rotation, translation and scaling of the matching image.
[0080] S34, using the image conversion parameter matrix to perform image conversion on the image to be matched in the selected image pair to obtain the converted image of the image to be matched, and using the pyramid information extraction module to extract the feature map of the converted image of the image to be matched.
[0081] Specifically, the image conversion parameter matrix is used to realize the rotation, translation and scaling of the image to be matched, and obtain the image after the image to be matched is converted, and then the method described in step S31 is used to extract the feature map of the image after the image to be matched is converted. The feature map of the image after the image to be matched can be expressed as follows:
[0082] (6)
[0083] in, (6) -1
[0084] In the formula, Represents the image to be matched in the training set image pair The converted image, Represents the image to be matched in the training set image pair Feature maps of the transformed image.
[0085] S35, setting a loss function according to the feature map of the reference image in the selected image pair, the feature map of the image to be matched, and the feature map of the image converted from the image to be matched, and calculating the loss value of the selected image pair.
[0086] Furthermore, the loss function in S35 is specifically:
[0087] (7)
[0088] In the formula, Represents the image to be matched in the training set and reference image The loss value of the composed image pair, and Represent the height and width of the feature map, respectively. represents the 1 norm, Represents the image to be matched in the training set image pair The feature map In the Line The value of the column, Represents the reference image in the training set image pair The feature map In the Line The value of the column, Represents the image to be matched in the training set image pair Feature map of the transformed image In the Line The value of the column, Represents the image conversion process, .
[0089] The purpose of the loss function is to measure the alignment effect of the converted image to be matched and the reference image.
[0090] S36. Select another image pair from the training set until all the image pairs in the training set are selected, repeat steps S31 to S35, obtain the loss value of each image pair in the training set, select the network parameters with the smallest loss value to update the multimodal image registration neural network model, and obtain the trained multimodal image registration neural network model.
[0091] Specifically, in the process of training the multimodal image registration neural network model using the training set, an image transformation parameter matrix is obtained for each image pair. And calculate the loss value L. The smaller the loss value, the better the alignment effect between the converted image to be matched and the reference image. Therefore, the network parameters corresponding to the smallest loss value are selected and the network is updated, thereby obtaining the trained multimodal image registration neural network model. The trained multimodal image registration neural network model has the ability to correctly estimate the image conversion parameter matrix.
[0092] S4. Preset the iteration termination condition, use the verification set to iteratively optimize the trained multimodal image registration neural network model and determine whether the iteration termination condition is met, so as to obtain the iteratively optimized multimodal image registration neural network model.
[0093] In one embodiment, S4 specifically includes the following steps:
[0094] S41. Set the iteration termination condition, preset the true coordinates of the reference image in the validation set image pair, and perform n (n≥1) iteration optimization on the trained multimodal image registration neural network model.
[0095] S42. Select one image pair from the validation set each time, input the selected image pair into the trained multimodal image registration neural network model for processing, obtain an image conversion parameter matrix for the selected image pair, use the image conversion parameter matrix to convert the image to be matched in the selected image pair, and obtain an image after the conversion of the image to be matched.
[0096] S43, selecting another image pair from the verification set again, until every image pair in the verification set is selected, repeating step S42, and obtaining the converted image of the image to be matched in each image pair in the verification set.
[0097] S44, calculating the difference between the pixel coordinates of each converted image to be matched in the verification set image pair and the real coordinates of the reference image corresponding to each image to be matched, and taking the average of all the differences.
[0098] Furthermore, S44 can be specifically expressed by the formula:
[0099] ;
[0100] In the formula, Represents the average value of the difference calculated after each round of iterative optimization. Indicates the validation set The image to be matched in the image pairs The coordinate value of Indicates the validation set Image alignment reference image The real coordinate value of Represents the image conversion process, Indicates the validation set The image transformation parameter matrix for image pairs, represents the number of image pairs in the validation set, .
[0101] S45. Determine whether the iteration termination condition is met based on the average value of all the differences. If not, execute steps S42 to S44 to perform n+1 iteration optimization on the trained multimodal image registration neural network model. Otherwise, execute step S46.
[0102] Specifically, the iteration termination condition is the average value of the difference between the pixel coordinates of the converted image corresponding to all the images to be matched in the verification set and the real coordinates of the reference image. If the calculation does not meet the termination condition of the iteration, the model will continue to be optimized for the next round of iteration using the images in the validation set.
[0103] S46, ending the iterative optimization and obtaining the iteratively optimized multimodal image registration neural network model.
[0104] S5. Acquire multimodal images in real scenes and form image pairs to be registered, input the image pairs to be registered into the iteratively optimized multimodal image registration neural network model for processing, obtain the image conversion parameter matrix of the image pairs to be registered, convert the image to be matched in the image pair to be registered according to the image conversion parameter matrix, and register and fuse the converted image to be matched with the reference image in the image pair to be registered to obtain the registered and fused image.
[0105] Specifically, see Figure 3 , Figure 3Schematic diagram of a multimodal image registration process in a real scene according to an embodiment of the present invention.
[0106] Acquire multimodal images to be registered in real scenes and form image pairs to be registered in real scenes (including images to be matched in real scenes) and reference image ), the image pairs to be registered in the real scene are input into the iteratively optimized multimodal image registration neural network model for processing, and the image conversion parameter matrix corresponding to the image pairs to be registered in the real scene can be obtained. Specifically, refer to formula (2), according to the image to be matched in the image pair to be registered in the real scene and reference image The corresponding feature map can be obtained and , and then refer to formulas (3) to (5) to obtain the image transformation parameter matrix of the image pair to be registered in the real scene , and then according to formula (9)-1, the image to be matched in the image pair to be registered in the real scene is obtained Converted image Then, the following formula (9) is used to calculate and obtain the registered fused image of the image pair to be registered in the real scene:
[0107] (9)
[0108] in, (9) -1
[0109] In the formula, represents the image after registration and fusion, represents the fusion weight, and They represent the image to be matched and the reference image in the image pair to be registered in the real scene, Represents the image conversion process, Represents the image to be matched The converted image, Represents the image to be matched in the real scene and reference image The image transformation parameter matrix corresponding to the image pair to be registered.
[0110] In one embodiment, a multimodal image registration system includes an image acquisition unit and a computer system connected to the image acquisition unit, wherein the computer system is provided with a processing unit, a storage unit and a multimodal image registration neural network model connected in sequence, wherein:
[0111] The image acquisition unit is used to acquire multimodal images and send the multimodal images to the computer system;
[0112] The computer system receives the multimodal image and sends it to the processing unit, and the processing unit divides the multimodal image into reference images and images to be matched, and forms image pairs and then sends them to the storage unit;
[0113] The storage unit is used to store the image pairs and the parameters of the multimodal image registration neural network model;
[0114] The multimodal image registration neural network model receives the image pair and performs multimodal image registration using the multimodal image registration method based on iterative optimization to obtain a registered fused image.
[0115] For specific limitations on a multimodal image registration system, please refer to the above limitations on a multimodal image registration method based on iterative optimization, which will not be repeated here.
[0116] The above-mentioned multimodal image registration method and system based on iterative optimization obtains several low-light images and performs image enhancement processing, divides the enhanced images into images to be matched and reference images to form several image pairs, divides the several image pairs into training sets and verification sets, builds a multimodal image registration neural network model, trains the multimodal image registration neural network model through the training set, and uses the loss function to calculate the loss to obtain the trained multimodal image registration neural network model, presets the iteration termination condition, uses the verification set to iteratively optimize the trained multimodal image registration neural network model and judges whether the iteration termination condition is met to obtain the iteratively optimized multimodal image registration neural network model, finally obtains the multimodal images to be registered in the real scene and forms image pairs, inputs the image pairs into the iteratively optimized multimodal image registration neural network model for processing, obtains the image conversion parameter matrix corresponding to the input image pair, converts the image to be matched in the input image pair according to the image conversion parameter matrix, and registers and fuses the converted image to be matched with the reference image in the input image pair to obtain the registered fused image. This method can improve the robustness of the image pair consisting of the reference image and the image to be matched during the registration process.
[0117] The above is a detailed introduction to a multimodal image registration method and system based on iterative optimization provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.
Claims
1. A multimodal image registration method based on iterative optimization, characterized in that: The method comprises the following steps: S1, obtaining a number of low-light images and performing image enhancement processing, dividing the enhanced images into to-be-matched images and reference images to form a number of image pairs, and dividing the number of image pairs into a training set and a verification set; S2. Build a multimodal image registration neural network model, which includes a pyramid information extraction module, a neighborhood attention module and a multi-layer perceptron module connected in sequence, wherein the pyramid information extraction module is used to receive and extract a feature map of an input image pair, the neighborhood attention module is used to receive and process the feature map to obtain a compressed feature map, and the multi-layer perceptron module is used to receive and process the compressed feature map to obtain an image conversion parameter matrix of the input image pair; S3, training the multimodal image registration neural network model through the training set, setting the loss function to calculate the loss value of each image pair in the training set and updating the network parameters to obtain the trained multimodal image registration neural network model; S4, presetting an iteration termination condition, using a validation set to iteratively optimize the trained multimodal image registration neural network model and determining whether the iteration termination condition is satisfied, thereby obtaining an iteratively optimized multimodal image registration neural network model; S5. Acquire multimodal images in real scenes and form image pairs to be registered, input the image pairs to be registered into the iteratively optimized multimodal image registration neural network model for processing, obtain the image conversion parameter matrix of the image pairs to be registered, convert the image to be matched in the image pair to be registered according to the image conversion parameter matrix, and register and fuse the converted image to be matched with the reference image in the image pair to be registered to obtain the registered and fused image.
2. The multimodal image registration method based on iterative optimization according to claim 1, characterized in that: In S1, a low-light image is acquired and image enhancement processing is performed. The image enhancement processing is specifically expressed by the formula: ; In the formula, is the average gray value of the window, is the magnification, , For the original low-light image in pixels The gray value at The enhanced image is in pixels Gray value.
3. The multimodal image registration method based on iterative optimization according to claim 2, characterized in that: S3 specifically includes the following steps: S31, arbitrarily selecting an image pair from the training set and inputting the image pair into the pyramid information extraction module for processing, thereby obtaining a feature map of the image to be matched and a feature map of the reference image in the selected image pair; S32, inputting the feature map of the image to be matched and the feature map of the reference image into the neighborhood attention module and calculating the correlation between the feature maps to obtain a neighborhood cost volume, performing channel compression on the neighborhood cost volume to obtain a compressed feature map; S33, inputting the compressed feature map into the multi-layer perceptron module for processing, and outputting the image conversion parameter matrix of the selected image pair; S34, performing image conversion on the image to be matched in the selected image pair using an image conversion parameter matrix to obtain an image after the image to be matched is converted, and extracting a feature map of the image after the image to be matched is converted using a pyramid information extraction module; S35, setting a loss function according to the feature map of the reference image in the selected image pair, the feature map of the image to be matched, and the feature map of the image converted from the image to be matched, and calculating the loss value of the selected image pair; S36. Select another image pair from the training set until all the image pairs in the training set are selected, repeat steps S31 to S35, obtain the loss value of each image pair in the training set, select the network parameters with the smallest loss value to update the multimodal image registration neural network model, and obtain the trained multimodal image registration neural network model.
4. The multimodal image registration method based on iterative optimization according to claim 3, characterized in that: The compressed feature map in S32 is specifically expressed by the formula: ; In the formula, represents the compressed feature map, represents a 1×1 convolutional layer, which is used to perform channel compression on the neighborhood cost volume. represents the neighborhood attention module, and They represent the feature map of the image to be matched and the feature map of the reference image in the training set image pair respectively.
5. The multimodal image registration method based on iterative optimization according to claim 4, characterized in that: The image conversion parameter matrix in S33 is specifically expressed by the formula: ; In the formula, represents the image transformation parameter matrix for the image pairs in the training set, represents a multi-layer perceptron module, Indicates that the compressed feature map Flatten to a one-dimensional vector.
6. The multimodal image registration method based on iterative optimization according to claim 5, characterized in that: The loss function in S35 is specifically: ; In the formula, Represents the image to be matched in the training set and reference image The loss value of the image pair composed of and Represent the height and width of the feature map, respectively. represents the 1 norm, Represents the image to be matched in the training set image pair The feature map In the Line The value of the column, Represents the reference image in the training set image pair The feature map In the Line The value of the column, Represents the image to be matched in the training set image pair Feature map of the transformed image In the Line The value of the column, .
7. The multimodal image registration method based on iterative optimization according to claim 6, characterized in that: S4 specifically includes the following steps: S41, setting an iteration termination condition, presetting the true coordinates of the reference image in the validation set image pair, and performing n-times iterative optimization on the trained multimodal image registration neural network model; S42, selecting one image pair from the validation set at a time, inputting the selected image pair into the trained multimodal image registration neural network model for processing, obtaining an image conversion parameter matrix for the selected image pair, and using the image conversion parameter matrix to convert the image to be matched in the selected image pair to obtain an image after the image to be matched is converted; S43, selecting another image pair from the verification set again, until every image pair in the verification set is selected, repeating step S42, and obtaining the image after the image to be matched in each image pair in the verification set is converted; S44, calculating the difference between the pixel coordinates of each converted image of the to-be-matched image in the verification set image pair and the real coordinates of the reference image corresponding to each to-be-matched image, and taking the average of all the differences; S45, judging whether the iteration termination condition is satisfied according to the average value of all the differences, if not satisfied, executing steps S42 to S44, performing n+1 iteration optimization on the trained multimodal image registration neural network model, otherwise, executing step S46; S46, ending the iterative optimization and obtaining the iteratively optimized multimodal image registration neural network model.
8. The multimodal image registration method based on iterative optimization according to claim 7, characterized in that: S44 is specifically expressed by the formula: ; In the formula, Represents the average value of the difference calculated after each round of iterative optimization. Indicates the validation set The image to be matched in the image pairs The coordinate value of Indicates the validation set Image alignment reference image The real coordinate value of Represents the image conversion process, Indicates the validation set The image transformation parameter matrix for image pairs, represents the number of image pairs in the validation set, .
9. The multimodal image registration method based on iterative optimization according to claim 1, characterized in that: The pyramid information extraction module in S2 includes a first convolution module, a normalization layer, an activation function layer, and a second convolution module connected in sequence, wherein the first convolution module includes three convolution layers connected in sequence, each convolution layer has a step size of 2, and a convolution kernel size of 3×3, and the second convolution module includes a convolution layer, the step size of the convolution layer is 1, and the convolution kernel size is 1×1.
10. A multimodal image registration system, characterized in that: The system includes an image acquisition unit and a computer system connected to the image acquisition unit, wherein the computer system is provided with a processing unit, a storage unit and a multimodal image registration neural network model connected in sequence, wherein: The image acquisition unit is used to acquire multimodal images and send the multimodal images to the computer system; The computer system receives the multimodal image and sends it to the processing unit, and the processing unit divides the multimodal image into reference images and images to be matched, and forms image pairs and then sends them to the storage unit; The storage unit is used to store the image pairs and the parameters of the multimodal image registration neural network model; The multimodal image registration neural network model receives an image pair and performs multimodal image registration using the multimodal image registration method based on iterative optimization as described in any one of claims 1 to 9 to obtain a registered and fused image.
