Multi-modal image registration method and system based on iterative optimization

Abstract

Disclosed in the present invention are a multi-modal image registration method and system based on iterative optimization. The method comprises: acquiring several low-light images and processing same, in order to obtain a training set and a validation set; constructing a multi-modal image registration neural network model; training the multi-modal image registration neural network model by means of the training set, and using a loss function to calculate loss, in order to obtain a trained multi-modal image registration neural network model; using the validation set to perform iterative optimization on the trained multi-modal image registration neural network model, and determining whether an iteration termination condition is met, in order to obtain an iteratively optimized multi-modal image registration neural network model; and acquiring multi-modal images in a real scene, forming image pairs to be registered, and inputting said image pairs into the iteratively optimized multi-modal image registration neural network model for processing, in order to obtain registered and fused images. The method can improve the robustness of image pairs, which each consist of a reference image and an image to be matched, during a registration process.

Description

A multi-modal image registration method and system based on iterative optimization

[0001] The present application claims priority to the Chinese patent application filed on September 14, 2024, with the Chinese Patent Office, the number of which is 202411288067X, and the title of which is "A multi-modal image registration method and system based on iterative optimization", the entire content or part of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of image data processing, in particular to a multi-modal image registration method and system based on iterative optimization. BACKGROUND

[0003] Nowadays, images obtained by various platform sensors are becoming more and more abundant, which are collectively referred to as multi-modal images. Multi-modal image registration can be used in positioning systems, specifically by matching images obtained by different sensors with reference images. Multi-modal image registration can utilize different types of perception data to achieve information complementation. By fusing these perception data, the system can more comprehensively understand the environment, thereby improving the accuracy and robustness of perception.

[0004] However, multi-modal images have significant differences in radiation features and geometric features, which can cause difficulty in high-precision matching. Therefore, multi-modal image registration needs to map features of different types of image data, but different types of images may have different features and distributions, and how to effectively extract and describe these feature mappings is a challenge. SUMMARY

[0005] The technical problem to be solved by the present application is to improve the accuracy and stability of multi-modal image registration. To achieve the above purpose, the present application provides a multi-modal image registration method based on iterative optimization, which comprises the following steps:

[0006] S1, obtaining a plurality of low-light images and performing image enhancement processing, dividing the enhanced images into to-be-matched images and reference images, and forming a plurality of image pairs, and dividing the plurality of image pairs into a training set and a validation set;

[0007] S2, building a multi-modal image registration neural network model, the model comprising a pyramid information extraction module, a neighborhood attention module and a multi-layer perception module connected in turn, the pyramid information extraction module being used for receiving and extracting feature mappings of input image pairs, the neighborhood attention module being used for receiving the feature mappings and processing to obtain compressed feature mappings, and the multi-layer perception module being used for receiving the compressed feature mappings and processing to obtain an image conversion parameter matrix of the input image pairs;

[0008] S3, training the multi-modal image registration neural network model through the training set, setting a loss function to calculate the loss value of each image pair in the training set and updating the network parameters, and obtaining the trained multi-modal image registration neural network model;

[0009] S4, presetting an iteration termination condition, iteratively optimizing the trained multi-modal image registration neural network model using the verification set and judging whether the iteration termination condition is met, and obtaining the iteratively optimized multi-modal image registration neural network model;

[0010] S5, obtaining multi-modal images in a real scene and composing a to-be-registered image pair, inputting the to-be-registered image pair into the iteratively optimized multi-modal image registration neural network model for processing, obtaining an image conversion parameter matrix of the to-be-registered image pair, converting a to-be-matched image in the to-be-registered image pair according to the image conversion parameter matrix, and performing registration fusion on the converted image of the to-be-matched image and a reference image in the to-be-registered image pair to obtain a registration fused image.

[0011] Preferably, in S1, a low-light image is obtained and image enhancement processing is performed, and the image enhancement processing can be specifically represented by the formula: g(x,y)=(f(x,y)-M x,y )×k+M x,y

[0012] In the formula, M x,y is a window average gray value, k is a magnification factor, k>1, f(x,y) is a gray value of the original low-light image at pixel (x,y), and g(x,y) is a gray value of the enhanced image at pixel (x,y).

[0013] Preferably, S3 specifically includes the following steps:

[0014] S31, an image pair is randomly selected from the training set and input into a pyramid information extraction module for processing, to obtain a feature map of a to-be-matched image and a feature map of a reference image in the selected image pair;

[0015] S32, the feature map of the to-be-matched image and the feature map of the reference image are input into a neighborhood attention module and the correlation between the feature maps is calculated to obtain a neighborhood cost volume, and the neighborhood cost volume is channel compressed to obtain a compressed feature map;

[0016] S33, the compressed feature map is input into a multi-layer perception module for processing, and an image conversion parameter matrix of the selected image pair is output;

[0017] S34, the image conversion parameter matrix is used to perform image conversion on the to-be-matched image in the selected image pair, to obtain a converted image of the to-be-matched image, and a pyramid information extraction module is used to extract a feature map of the converted image of the to-be-matched image.

[0018] S35. Set the loss function and calculate the loss value of the selected image pair based on the feature mapping of the reference image, the feature mapping of the image to be matched, and the feature mapping of the transformed image of the image to be matched.

[0019] S36. Select another image pair from the training set until all image pairs in the training set have been selected. Repeat steps S31 to S35 to obtain the loss value of each image pair in the training set. Select the network parameters with the smallest loss value to update the multimodal image registration neural network model and obtain the trained multimodal image registration neural network model.

[0020] Preferably, the compressed feature map in S32 can be expressed by the following formula:

[0021] In the formula, F vol Represents the compressed feature map, Con 1×1 () denotes a 1×1 convolutional layer used for channel compression of the neighborhood cost volume, atten() denotes the neighborhood attention module, F a and F b These represent the feature mappings of the image to be matched and the reference image in the training set image pair, respectively.

[0022] Preferably, the image transformation parameter matrix in S33 can be specifically expressed by the formula: H ab =MLP(F flatten (F vol ))

[0023] In the formula, H ab The image transformation parameter matrix represents the image pairs in the training set, MLP() represents the multilayer perceptron module, and F flatten () indicates that the compressed feature map F vol Flattened, it is a one-dimensional vector.

[0024] Preferably, the loss function in S35 is as follows:

[0025] In the formula, L represents the image I to be matched in the training set. a and reference image I b The loss values ​​for the composed image pairs, where H and W represent the height and width of the feature map, respectively, ||1|| represents the 1-norm, and F... a-ij Indicates the image I to be matched in the training set image pair. a Feature Mapping F a The value in the i-th row and j-th column, F b-ij Represents the reference image I in the training set image pair. b Feature Mapping Fb The value in the i-th row and j-th column, F a ′ -ij Indicates the image I to be matched in the training set image pair. a Feature mapping F of the transformed image a 'The value in the i-th row and j-th column, i = 1, 2, ... H, j = 1, 2, ..., W.

[0026] Preferably, S4 specifically includes the following steps:

[0027] S41. Set the iteration termination condition, preset the true coordinates of the reference image for the validation set image alignment, and perform n iterations to optimize the trained multimodal image registration neural network model.

[0028] S42. Select one image pair from the validation set each time, input the selected image pair into the trained multimodal image registration neural network model for processing, obtain the image transformation parameter matrix of the selected image pair, and use the image transformation parameter matrix to transform the image to be matched in the selected image pair to obtain the transformed image of the image to be matched.

[0029] S43. Select another image pair from the verification set until each image pair in the verification set has been selected. Repeat step S42 to obtain the transformed image of the image to be matched in each image pair in the verification set.

[0030] S44. Calculate the difference between the pixel coordinates of the transformed image of each image to be matched in the validation set image pair and the true coordinates of the reference image corresponding to each image to be matched, and take the average of all differences.

[0031] S45. Determine whether the iteration termination condition is met based on the average of all differences. If not, execute steps S42 to S44 to perform n+1 iterations of optimization on the trained multimodal image registration neural network model. Otherwise, execute step S46.

[0032] S46. End the iterative optimization and obtain the iteratively optimized multimodal image registration neural network model.

[0033] Preferably, S44 can be specifically expressed by the formula:

[0034] In the formula, error represents the average value of the difference calculated after each round of optimization iteration. This represents the image to be matched in the l-th image pair in the validation set. coordinates, This represents the reference image for centering the l-th image in the verification set. The true coordinates of ω, where ω(,) represents the image transformation process. denotes an image transformation parameter matrix of the lth image pair in the verification set, N denotes the number of image pairs in the verification set, and l = 1, 2, …, N.

[0035] Preferably, the pyramid information extraction module in S2 comprises a first convolution module, a normalization layer, an activation function layer and a second convolution module connected in sequence, wherein the first convolution module comprises three convolution layers connected in sequence, each convolution layer has a step length of 2 and a convolution kernel size of 3x3, and the second convolution module comprises a convolution layer having a step length of 1 and a convolution kernel size of 1x1.

[0036] Another aspect of the present application also provides a multi-modal image registration system, which comprises an image acquisition unit and a computer system connected with the image acquisition unit, and the computer system is provided with a processing unit, a storage unit and a multi-modal image registration neural network model connected in sequence, wherein:

[0037] The image acquisition unit is used for acquiring multi-modal images and sending the multi-modal images to the computer system;

[0038] The computer system receives the multi-modal images and sends them to the processing unit, the processing unit divides the multi-modal images into reference images and to-be-matched images, and sends the image pairs to the storage unit after being composed;

[0039] The storage unit is used for storing the image pairs and the parameters of the multi-modal image registration neural network model;

[0040] The multi-modal image registration neural network model receives the image pairs and performs multi-modal image registration by using the above multi-modal image registration method based on iterative optimization to obtain registered and fused images.

[0041] The above-mentioned multi-modal image registration method and system based on iterative optimization, a plurality of low-light images are obtained and image enhancement processing is performed, the enhanced images are divided into to-be-matched images and reference images and form a plurality of image pairs, the plurality of image pairs are divided into a training set and a verification set, a multi-modal image registration neural network model is built, the multi-modal image registration neural network model is trained through the training set, and a loss function is used to calculate a loss, to obtain a trained multi-modal image registration neural network model, a preset iteration termination condition is set, the trained multi-modal image registration neural network model is iteratively optimized using the verification set, and whether the iteration termination condition is met is judged, to obtain an iteratively optimized multi-modal image registration neural network model, finally, multi-modal images to be registered in a real scene are obtained and form an image pair, the image pair is input into the iteratively optimized multi-modal image registration neural network model for processing, to obtain an image conversion parameter matrix corresponding to the input image pair, the to-be-matched image in the input image pair is converted according to the image conversion parameter matrix, and the to-be-matched image after conversion and the reference image in the input image pair are registered and fused, to obtain a registered and fused image. The method can improve the robustness of the image pair composed of the reference image and the to-be-matched image in the registration process. BRIEF DESCRIPTION OF DRAWINGS

[0042] FIG. 1 is a flowchart of a multi-modal image registration method based on iterative optimization according to an embodiment of the present application;

[0043] FIG. 2 is a schematic diagram of the working process of a neighborhood attention module according to an embodiment of the present application;

[0044] FIG. 3 is a schematic diagram of a multi-modal image registration process in a real scene according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the drawings.

[0046] Referring to FIG. 1, FIG. 1 is a flowchart of a multi-modal image registration method based on iterative optimization according to an embodiment of the present application. The multi-modal image registration method based on iterative optimization comprises the following steps:

[0047] S1, a plurality of low-light images are obtained and image enhancement processing is performed, the enhanced images are divided into to-be-matched images and reference images and form a plurality of image pairs, and the plurality of image pairs are divided into a training set and a verification set.

[0048] Specifically, the image enhancement processing is to improve the brightness of the low-light images using a local contrast method.

[0049] Further, the image enhancement processing can be expressed by the following formula: g(x, y)=(f(x, y)-Mx,y ) x k + M x,y (1)

[0050] wherein M x,y is the window average gray value, k is the magnification, k > 1, f(x, y) is the gray value of the original low-light image at pixel (x, y), and g(x, y) is the gray value of the enhanced image at pixel (x, y).

[0051] When the magnification k > 1, if f(x, y) > M x,y , then g(x, y) > f(x, y); if f(x, y) < M x,y , then g(x, y) < f(x, y).

[0052] S2, a multi-modal image registration neural network model is built, the model comprising a pyramid information extraction module, a neighborhood attention module and a multi-layer perception module connected in sequence, the pyramid information extraction module being configured to receive and extract feature maps of an input image pair, the neighborhood attention module being configured to receive the feature maps and process them to obtain compressed feature maps, and the multi-layer perception module being configured to receive the compressed feature maps and process them to obtain an image conversion parameter matrix of the input image pair.

[0053] Specifically, the multi-modal image registration neural network model comprises a pyramid information extraction module, a neighborhood attention module and a multi-layer perception module connected in sequence, wherein the pyramid information extraction module is configured to extract feature maps of an input image pair, the neighborhood attention module is configured to calculate a neighborhood cost volume according to the feature maps of the input image pair, and compress the channels of the neighborhood cost volume to obtain channel-compressed feature maps, and the multi-layer perception module is configured to calculate an image conversion parameter matrix according to the channel-compressed feature maps.

[0054] Further, the pyramid information extraction module in S2 comprises a first convolution module, a normalization layer, an activation function layer and a second convolution module connected in sequence, wherein the first convolution module comprises three convolution layers connected in sequence, each convolution layer having a step size of 2 and a convolution kernel size of 3x3, and the second convolution module comprises a convolution layer having a step size of 1 and a convolution kernel size of 1x1.

[0055] The first convolution module is configured to obtain rich contextual information, improve the learning representation ability of the module, obtain feature maps with rich semantic information, the normalization layer is configured to make the data distribution more stable, which is conducive to improving the stability and generalization ability of network training, and the second convolution module is configured to reduce the number of channels of the feature maps to reduce the computational load of the neighborhood attention module.

[0056] S3, training the multi-modal image registration neural network model through the training set, setting a loss function to calculate the loss value of each image pair in the training set and updating the network parameters, and obtaining the trained multi-modal image registration neural network model.

[0057] In one embodiment, S3 specifically comprises the following steps:

[0058] S31, an image pair is randomly selected from the training set and input into the pyramid information extraction module for processing to obtain the feature mapping of the to-be-matched image and the feature mapping of the reference image in the selected image pair.

[0059] Further, the feature mapping of the to-be-matched image and the feature mapping of the reference image in S31 can be specifically represented by the following formula: m f(I m ), m∈{a,b} (2)

[0060] In the formula, f() represents the pyramid information extraction module, I a and I b represent the to-be-matched image and the reference image in the image pair of the training set respectively, and F a and F b represent the feature mapping of the to-be-matched image and the feature mapping of the reference image in the image pair of the training set respectively.

[0061] S32, inputting the feature mapping of the to-be-matched image and the feature mapping of the reference image into the neighborhood attention module and calculating the correlation between the feature mappings to obtain a neighborhood cost volume, and performing channel compression on the neighborhood cost volume to obtain the compressed feature mapping.

[0062] Further, the compressed feature mapping in S32 can be specifically represented by the following formula:

[0063] In the formula, F vol represents the compressed feature mapping, Con 1×1 () represents a 1x1 convolution layer used for channel compression of the neighborhood cost volume, and atten() represents the neighborhood attention module.

[0064] Specifically, referring to FIG. 2, which is a working process diagram of the neighborhood attention module in an embodiment of the present application, the value of a certain position in the feature mapping F a of the to-be-matched image in the image pair and the value of the corresponding position in the feature mapping F b of the reference image are multiplied within a neighborhood with a radius of N(z) to calculate the correlation between the feature mappings, and a neighborhood cost volume F cost is obtained.

[0065] wherein z represents a feature map F of the image to be matched a F(z+u) represents a feature map F of the reference image b F(z+u) represents a feature map F of the reference image b a value in the corresponding position and in a neighborhood with a radius of N(z), N(z) represents a neighborhood range of z, and in the embodiment, N(z) takes 3.

[0066] The neighborhood cost volume F is reduced by a 1x1 convolutional layer cost to reduce the number of channels of the neighborhood cost volume F cost , that is, to perform channel compression on the neighborhood cost volume F vol .

[0067] S33, input the compressed feature map into a multi-layer perception module for processing, and output an image conversion parameter matrix of the selected image pair.

[0068] Further, the image conversion parameter matrix in S33 can be specifically expressed by a formula as follows: H ab = MLP(F flatten (F vol )) (5)

[0069] In the formula, H ab represents an image conversion parameter matrix of the image pair in the training set, MLP() represents the multi-layer perception module, F flatten () represents flattening the compressed feature map F vol into a one-dimensional vector.

[0070] Specifically, the multi-layer perception module MLP() includes three fully connected layers connected in sequence, the size of the first fully connected layer is HxW, the size of the second fully connected layer is 32, and the size of the third fully connected layer is 8. The image conversion parameter matrix is output by regression through the three fully connected layers with different sizes, and is used to realize rotation, translation and scaling of the image to be matched.

[0071] S34, using the image conversion parameter matrix to perform image conversion on the image to be matched in the selected image pair, obtaining a converted image of the image to be matched, and using a pyramid information extraction module to extract a feature map of the converted image of the image to be matched.

[0072] Specifically, the image conversion parameter matrix is used to realize rotation, translation and scaling of the image to be matched, to obtain a converted image of the image to be matched, and then the feature map of the converted image of the image to be matched is extracted by using the method described in step S31. The feature map of the converted image of the image to be matched can be expressed as follows: F a ′ = f(I a ) (6)

[0073] where I' is the transformed image of I, and F' is the feature map of I'. a ab a (6)-1

[0074] where I' is the transformed image of I, and F' is the feature map of I'. a a a a

[0075] S35, setting a loss function according to the feature map of the reference image, the feature map of the image to be matched, and the feature map of the transformed image of the image to be matched in the selected image pair, and calculating the loss value of the selected image pair.

[0076] Further, the loss function in S35 is specifically:

[0077] where L represents the loss value of the image pair composed of the image to be matched I and the reference image I in the training set, H and W represent the height and width of the feature map respectively, || ||1 represents the 1-norm, F represents the feature map of the image to be matched I in the training set image pair, F represents the value of the i-th row and the j-th column of the feature map F of the image to be matched I, F represents the value of the i-th row and the j-th column of the feature map F of the reference image I, F' represents the feature map of the transformed image of the image to be matched I in the training set image pair, F' represents the value of the i-th row and the j-th column of the feature map F' of the transformed image of the image to be matched I, and ω (,) represents the image transformation process, i = 1, 2, …, H, j = 1, 2, …, W. a b a-ij a a b-ij b b a-ij a a a

[0078] The purpose of the loss function is to measure the alignment effect of the transformed image of the image to be matched and the reference image.

[0079] S36, selecting another image pair from the training set again until all the image pairs in the training set are selected, repeating steps S31 to S35 to obtain the loss value of each image pair in the training set, updating the network parameters when the loss value is the smallest, and obtaining the trained multi-modal image registration neural network model.

[0080] ​​​​​​​​​​​​​​​​Specifically, during the training of the multimodal image registration neural network model using the training set, an image transformation parameter matrix H is obtained for each image pair. ab The loss value L is calculated. The smaller the loss value, the better the alignment between the image to be matched and the reference image. Therefore, the network parameters corresponding to the minimum loss value are selected and the network is updated. This yields the trained multimodal image registration neural network model, which has the ability to correctly estimate the image transformation parameter matrix.

[0081] S4. Preset the iteration termination condition, use the validation set to iteratively optimize the trained multimodal image registration neural network model and determine whether the iteration termination condition is met, and obtain the iteratively optimized multimodal image registration neural network model.

[0082] In one embodiment, S4 specifically includes the following steps:

[0083] S41. Set the iteration termination condition, preset the true coordinates of the reference image for the validation set image alignment, and perform n (n≥1) iterations to optimize the trained multimodal image registration neural network model.

[0084] S42. Select one image pair from the validation set each time, input the selected image pair into the trained multimodal image registration neural network model for processing, obtain the image transformation parameter matrix of the selected image pair, and use the image transformation parameter matrix to transform the image to be matched in the selected image pair to obtain the transformed image of the image to be matched.

[0085] S43. Select another image pair from the validation set until every image pair in the validation set has been selected. Repeat step S42 to obtain the converted image of the image to be matched in each image pair in the validation set.

[0086] S44. Calculate the difference between the pixel coordinates of the transformed image of each image to be matched in the validation set image pair and the true coordinates of the reference image corresponding to each image to be matched, and take the average of all differences.

[0087] Furthermore, S44 can be specifically expressed by the formula:

[0088] In the formula, error represents the average value of the difference calculated after each round of optimization iteration. This represents the image to be matched in the l-th image pair in the validation set. coordinates, This represents the reference image for centering the l-th image in the verification set. The true coordinates of ω, where ω(,) represents the image transformation process. Let N represent the image transformation parameter matrix of the l-th image pair in the validation set, and let N represent the number of image pairs in the validation set, where l = 1, 2, ..., N.

[0089] S45. Determine whether the iteration termination condition is met based on the average of all differences. If not, proceed to steps S42 to S44 to perform n+1 iterations of optimization on the trained multimodal image registration neural network model. Otherwise, proceed to step S46.

[0090] Specifically, the iteration termination condition is that the average difference (error) between the pixel coordinates of the transformed images corresponding to all images to be matched in the validation set and the true coordinates of the reference image is less than three pixels. If the iteration termination condition is not met after calculation, the model will continue to be optimized in the next round using images from the validation set.

[0091] S46. End the iterative optimization and obtain the iteratively optimized multimodal image registration neural network model.

[0092] S5. Acquire multimodal images from real-world scenes and form image pairs to be registered. Input the image pairs to be registered into the iteratively optimized multimodal image registration neural network model to obtain the image transformation parameter matrix of the image pairs to be registered. Based on the image transformation parameter matrix, transform the image to be matched in the image pairs to be registered. Then, register and fuse the transformed image and the reference image in the image pairs to obtain the registered and fused image.

[0093] Specifically, see Figure 3, which is a schematic diagram of the multimodal image registration process in a real scene in an embodiment of the present invention.

[0094] Acquire multimodal images to be registered in a real-world scene and assemble them into pairs of images to be registered in the real-world scene (including the images to be matched in the real-world scene). and reference image By processing the real-world image to be registered against the input of an iteratively optimized multimodal image registration neural network model, the image transformation parameter matrix corresponding to the real-world image to be registered can be obtained. Specifically, referring to formula (2), based on the image to be matched in the image pair to be registered in the real scene. and reference image The feature mapping can be obtained accordingly. and Then, referring to formulas (3) to (5), the image transformation parameter matrix of the image pair to be registered in the real scene is obtained. Then, according to formula (9)-1, the image to be matched in the image pair to be registered in the real scene is obtained. Converted image Then, the registered and fused image of the image pair to be registered in the real scene is calculated according to the following formula (9):

[0095] wherein,

[0096] in the formula, denotes the registered and fused image, and a denotes the fusion weight, and denote the image to be matched and the reference image in the image pair to be registered in the real scene, respectively, and ω denotes the image conversion process. denotes the image to be matched converted image, denotes the image to be matched and the reference image in the image pair to be registered in the real scene.

[0097] In one embodiment, a multi-modal image registration system comprises an image acquisition unit and a computer system connected with the image acquisition unit, the computer system being provided with a processing unit, a storage unit and a multi-modal image registration neural network model connected in sequence, wherein:

[0098] The image acquisition unit is configured to acquire multi-modal images and send the multi-modal images to the computer system.

[0099] The computer system receives the multi-modal images and sends them to the processing unit, which divides the multi-modal images into reference images and images to be matched and forms image pairs, and then sends the image pairs to the storage unit.

[0100] The storage unit is configured to store the image pairs and the parameters of the multi-modal image registration neural network model.

[0101] The multi-modal image registration neural network model receives the image pairs and performs multi-modal image registration using the above-mentioned multi-modal image registration method based on iterative optimization to obtain registered and fused images.

[0102] The specific limitations of the multi-modal image registration system can be found in the limitations of the multi-modal image registration method based on iterative optimization described above, and will not be repeated here.

[0103] The above-mentioned multi-modal image registration method and system based on iterative optimization obtain several low-light images and perform image enhancement processing, divide the enhanced images into to-be-matched images and reference images and form several image pairs, divide the several image pairs into a training set and a verification set, build a multi-modal image registration neural network model, train the multi-modal image registration neural network model through the training set, calculate a loss using a loss function, obtain the trained multi-modal image registration neural network model, preset an iteration termination condition, perform iterative optimization on the trained multi-modal image registration neural network model using the verification set and determine whether the iteration termination condition is met, obtain the multi-modal image registration neural network model after iterative optimization, finally obtain multi-modal images to be registered in a real scene and form an image pair, input the image pair into the multi-modal image registration neural network model after iterative optimization for processing, obtain an image conversion parameter matrix corresponding to the input image pair, convert the to-be-matched image in the input image pair according to the image conversion parameter matrix, and perform registration and fusion on the converted to-be-matched image and the reference image in the input image pair, to obtain a registered and fused image. This method can improve the robustness of the image pair composed of the reference image and the to-be-matched image in the registration process.

[0104] The above describes in detail the multi-modal image registration method and system based on iterative optimization provided by the present application. The principles and implementation modes of the present application are described by applying specific examples, and the above description of the embodiments is only used to help understand the core idea of the present application. It should be pointed out that those skilled in the art can make some improvements and modifications to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A method of multi-modal image registration based on iterative optimization, characterized in that, The method comprises the following steps: S1, obtaining a plurality of low-light images and performing image enhancement processing, dividing the enhanced images into to-be-matched images and reference images, and forming a plurality of image pairs, and dividing the plurality of image pairs into a training set and a validation set; S2, a multi-modal image registration neural network model is built, the model comprising a pyramid information extraction module, a neighborhood attention module and a multi-layer perception module connected in sequence, the pyramid information extraction module being used for receiving and extracting feature maps of the input image pair, the neighborhood attention module being used for receiving the feature maps and processing to obtain compressed feature maps, and the multi-layer perception module being used for receiving the compressed feature maps and processing to obtain an image conversion parameter matrix of the input image pair; S3, the multi-modal image registration neural network model is trained by using the training set, a loss function is set to calculate the loss value of each image pair in the training set and the network parameters are updated, and a trained multi-modal image registration neural network model is obtained; S4, an iteration termination condition is preset, the trained multi-modal image registration neural network model is iteratively optimized by using the validation set, and it is judged whether the iteration termination condition is met, and an iteratively optimized multi-modal image registration neural network model is obtained; S5, a plurality of multi-modal images in a real scene are obtained and formed into a to-be-registered image pair, the to-be-registered image pair is input into the iteratively optimized multi-modal image registration neural network model for processing, an image conversion parameter matrix of the to-be-registered image pair is obtained, the to-be-matched image in the to-be-registered image pair is converted according to the image conversion parameter matrix, and the to-be-matched image after conversion and the reference image in the to-be-registered image pair are registered and fused to obtain a registered and fused image.

2. The multi-modality image registration method based on iterative optimization of claim 1, wherein, In S1, a low-light image is acquired and image enhancement processing is performed. The image enhancement processing is specifically expressed by the formula: g(x, y) = (f(x, y) - M x,y ) x k + M x,y In the formula, M x,y is the window average gray value, k is the magnification, k>1, f(x, y) is the gray value of the original low-light image at pixel (x, y), and g(x, y) is the gray value of the enhanced image at pixel (x, y).

3. The multi-modality image registration method based on iterative optimization of claim 2, wherein, S3 specifically comprises the following steps: S31, an image pair is randomly selected from the training set and input into the pyramid information extraction module for processing to obtain the feature map of the to-be-matched image and the feature map of the reference image in the selected image pair; S32, the feature map of the to-be-matched image and the feature map of the reference image are input into the neighborhood attention module, the correlation between the feature maps is calculated, a neighborhood cost volume is obtained, and the neighborhood cost volume is channel compressed to obtain compressed feature maps; S33, the compressed feature maps are input into the multi-layer perception module for processing, and an image conversion parameter matrix of the selected image pair is output; S34, the to-be-matched image in the selected image pair is converted by using the image conversion parameter matrix to obtain an image after conversion of the to-be-matched image, and the feature map of the image after conversion of the to-be-matched image is extracted by using the pyramid information extraction module; S35, a loss function is set according to the feature map of the reference image, the feature map of the to-be-matched image and the feature map of the image after conversion of the to-be-matched image in the selected image pair, and the loss value of the selected image pair is calculated; S36, another image pair is selected from the training set, and steps S31 to S35 are repeated until all image pairs in the training set are selected, the loss value of each image pair in the training set is obtained, the network parameters at the time when the loss value is the smallest are selected to update the multi-modal image registration neural network model, and a trained multi-modal image registration neural network model is obtained.

4. The multi-modality image registration method based on iterative optimization of claim 3, wherein, The compressed feature map in S32 is specifically expressed by a formula as follows: In the formula, F vol denotes the compressed feature map, Con 1×1 denotes a 1x1 convolutional layer for channel compression of the neighborhood cost volume, atten() denotes a neighborhood attention module, F a and F b respectively denote the feature map of the image to be matched and the feature map of the reference image in the training set image pair.

5. The multi-modal image registration method based on iterative optimization of claim 4, wherein, The image conversion parameter matrix in S33 is specifically expressed by a formula as: H ab = MLP(F flatten (F vol )) In the formula, H ab represents the image conversion parameter matrix of the image pair in the training set, MLP() represents a multi-layer perception module, F flatten () represents mapping the compressed features F vol to a one-dimensional vector.

6. The multi-modality image registration method based on iterative optimization of claim 5, wherein, The loss function in S35 is specifically: In the formula, L represents the image I to be matched in the training set a and the reference image I b composing the image pair, H and W respectively represent the height and width of the feature mapping, || ||1 represents the 1-norm, F a-ij represents the feature mapping F a of the image I to be matched in the training set image pair a in the i-th row and j-th column, F b-ij represents the feature mapping F b of the reference image I in the training set image pair b in the i-th row and j-th column, F' a-ij represents the feature mapping F a of the converted image of the image I to be matched in the training set image pair a in the i-th row and j-th column, i=1,2,...H, j=1,2,...,W.

7. The multi-modality image registration method based on iterative optimization of claim 6, wherein, S4 specifically comprises the following steps: S41, setting an iteration termination condition, presetting the true coordinates of the reference image in the verification set image pair, and performing n times of iteration optimization on the trained multi-modal image registration neural network model; S42, selecting one image pair from the verification set each time, inputting the selected image pair into the trained multi-modal image registration neural network model for processing to obtain an image conversion parameter matrix of the selected image pair, and converting the to-be-matched image in the selected image pair using the image conversion parameter matrix to obtain a converted image of the to-be-matched image; S43, selecting another image pair from the verification set again until each image pair in the verification set is selected, repeating step S42 to obtain a converted image of the to-be-matched image in each image pair in the verification set; S44, calculating the difference between the pixel coordinates of each converted to-be-matched image and the true coordinates of the corresponding reference image in the verification set image pair, and taking the average of all the differences; S45, judging whether the iteration termination condition is met according to the average of all the differences, if not, performing steps S42 to S44 to perform n+1 times of iteration optimization on the trained multi-modal image registration neural network model, otherwise, performing step S46; S46, ending the iteration optimization to obtain the iteration-optimized multi-modal image registration neural network model.

8. The multi-modality image registration method based on iterative optimization of claim 7, wherein, S44 is specifically expressed by the formula: where error represents the average of the difference values calculated after each round of iteration optimization, represents the image to be matched in the lth image pair in the verification set coordinate values of the point of intersection of the line segment and the circle, represents the reference image in the lth image pair in the validation set true coordinate values, ω(, ) represents the image conversion process, The image conversion parameter matrix of the lth image pair in the verification set is represented as, and N represents the number of image pairs in the verification set, l=1, 2, …, N.

9. The multi-modality image registration method based on iterative optimization of claim 1, wherein, The pyramid information extraction module in S2 comprises a first convolution module, a normalization layer, an activation function layer and a second convolution module connected in sequence, wherein the first convolution module comprises three convolution layers connected in sequence, each convolution layer has a step length of 2 and a convolution kernel size of 3x3, and the second convolution module comprises a convolution layer with a step length of 1 and a convolution kernel size of 1x1.

10. A multi-modal image registration system, characterized in that, The system comprises an image acquisition unit and a computer system connected with the image acquisition unit, and the computer system is provided with a processing unit, a storage unit and a multi-modal image registration neural network model connected in sequence, wherein: The image acquisition unit is used to acquire multi-modal images and send the multi-modal images to the computer system; The computer system receives the multi-modal images and sends them to the processing unit, the processing unit divides the multi-modal images into reference images and to-be-matched images and forms image pairs, and then sends them to the storage unit; The storage unit is used to store the image pairs and the parameters of the multi-modal image registration neural network model; The multi-modal image registration neural network model receives the image pairs and performs multi-modal image registration using the multi-modal image registration method based on iteration optimization according to any one of claims 1 to 9 to obtain the registration and fusion images.

Images