Image registration method, device, equipment, medium and program product
By fusing local and global features and employing a two-stage alignment method, the problems of low accuracy and poor robustness in image registration under complex scenes are solved, and high-precision image registration in complex environments is achieved.
Patent Information
- Application Number
- CN202411622891.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-14
- Publication Date
- 2025-12-12
AI Technical Summary
Existing image registration methods based on feature points have low accuracy and poor robustness in complex scenes. In particular, feature point matching is inaccurate in low-quality images or complex environments, which affects the registration accuracy.
By fusing local and global features of an image, a more robust set of feature points is generated. A two-stage alignment method is adopted: first, coarse alignment is performed based on feature points, and then fine alignment is performed through region division and global transformation matrix, thereby improving the accuracy and robustness of image registration.
It significantly improves the accuracy and robustness of image registration in complex scenarios, adapts to different environmental changes, and enhances the accuracy of feature matching, especially achieving high-quality registration when the image content is complex or the feature points are few.
Smart Images
Figure CN121120713A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to an image registration method, apparatus, device, medium, and program product. Background Technology
[0002] Image registration is a technique that aligns two or more images according to certain geometric transformations to facilitate further analysis and processing. Common techniques for image registration include feature point-based methods. However, feature point-based image registration schemes are sensitive to changes in image scale, rotation, and illumination, especially in complex scenes or low-quality images, where inaccurate feature point matching can easily occur, affecting registration accuracy. Therefore, image registration methods in these techniques exhibit low accuracy and poor robustness when applied to complex scenes. Summary of the Invention
[0003] To address the issues of low accuracy and poor robustness in image registration methods when applied to complex scenes, embodiments of this application provide an image registration method, apparatus, device, medium, and program product that can effectively improve the accuracy and robustness of image registration in complex scenes.
[0004] In a first aspect, embodiments of this application provide an image registration method, including:
[0005] The local and global features of each of the at least two images to be registered are fused to obtain the feature point set corresponding to each of the at least two images to be registered.
[0006] Based on the feature point set corresponding to each of the at least two images to be registered, the first image of the at least two images to be registered is aligned with the second image of the at least two images to be registered to obtain a third image. The first image is any one of the at least two images to be registered, and the second image is a different image from the first image.
[0007] The second image and the third image are divided into regions respectively to obtain a first region set and a second region set, and a global transformation matrix is determined based on the first region set and the second region set;
[0008] The third image is aligned with the second image according to the global transformation matrix to obtain the registration image corresponding to the first image.
[0009] Optionally, the step of fusing local and global features of each of the at least two images to be registered to obtain a set of feature points corresponding to each of the at least two images to be registered includes:
[0010] The Scale Invariant Feature Transform (SIFT) algorithm is used to extract features from at least two images to be registered, thereby obtaining the local features of each of the at least two images to be registered.
[0011] Convolutional Neural Network (CNN) feature extraction is performed on the at least two images to be registered to obtain the global features of each of the at least two images to be registered.
[0012] According to the target weight, the local features and the global features of each of the at least two images to be registered are fused to obtain the feature point set corresponding to each of the at least two images to be registered.
[0013] The target weights include a first weight and a second weight. The first weight is dynamically adjusted based on the response intensity of feature points in the local features, and the second weight is dynamically adjusted based on the response intensity of feature points in the global features.
[0014] Optionally, the step of aligning the first image among the at least two images to be registered with the second image among the at least two images to be registered, based on the feature point set corresponding to each of the at least two images to be registered, to obtain the third image, includes:
[0015] Based on the feature point set corresponding to the first image and the feature point set corresponding to the second image, determine the alignment transformation matrix;
[0016] According to the alignment transformation matrix, the first image is aligned with the second image as a reference to obtain the third image.
[0017] Optionally, aligning the first image with reference to the second image according to the alignment transformation matrix to obtain the third image includes:
[0018] Based on the alignment transformation matrix, the feature points in the first image are mapped to the positions of the corresponding feature points in the second image to obtain the third image.
[0019] Optionally, the step of dividing the second image and the third image into regions to obtain a first region set and a second region set includes:
[0020] Based on the image size of the second image, the second image is divided into multiple grid blocks of corresponding size to obtain the first region set;
[0021] Based on the image size of the third image, the third image is divided into multiple grid blocks of corresponding size to obtain the second region set.
[0022] Optionally, determining the global transformation matrix based on the first region set and the second region set includes:
[0023] Based on the pixel values at the target coordinates within the first region set and the pixel values at the target coordinates within the second region set, a similarity metric is determined between the first region set and the second region set.
[0024] The global transformation matrix is determined based on the similarity metric.
[0025] Secondly, embodiments of this application also provide an image registration apparatus, comprising:
[0026] The feature fusion module is used to fuse the local features and global features of each of the at least two images to be registered, respectively, to obtain the feature point set corresponding to each of the at least two images to be registered;
[0027] The first alignment module is used to align the first image in the at least two images to be registered with the second image in the at least two images to be registered based on the feature point set corresponding to each of the at least two images to be registered, to obtain a third image. The first image is any one of the at least two images to be registered, and the second image is a different image from the first image.
[0028] The region division module is used to divide the second image and the third image into regions respectively, to obtain a first region set and a second region set, and to determine a global transformation matrix based on the first region set and the second region set;
[0029] The second alignment module is used to align the third image with the second image according to the global transformation matrix to obtain the registration image corresponding to the first image.
[0030] Optionally, the feature fusion module includes:
[0031] The first extraction unit is used to perform scale-invariant feature transform matching algorithm SIFT feature extraction on at least two images to be registered, so as to obtain the local features of each of the at least two images to be registered.
[0032] The second extraction unit is used to perform convolutional neural network (CNN) feature extraction on the at least two images to be registered, so as to obtain the global features of each of the at least two images to be registered.
[0033] The feature fusion unit is used to fuse the local features and the global features of each of the at least two images to be registered according to the target weight, so as to obtain the feature point set corresponding to each of the at least two images to be registered.
[0034] The target weights include a first weight and a second weight. The first weight is dynamically adjusted based on the response intensity of feature points in the local features, and the second weight is dynamically adjusted based on the response intensity of feature points in the global features.
[0035] Optionally, the first alignment module includes:
[0036] The first determining unit is used to determine the alignment transformation matrix based on the feature point set corresponding to the first image and the feature point set corresponding to the second image;
[0037] The first alignment unit is used to align the first image with the second image as a reference according to the alignment transformation matrix to obtain the third image.
[0038] Optionally, the first alignment unit is used for:
[0039] Based on the alignment transformation matrix, the feature points in the first image are mapped to the positions of the corresponding feature points in the second image to obtain the third image.
[0040] Optionally, the partitioning determination module includes:
[0041] The first partitioning unit is used to divide the second image into multiple grid blocks of corresponding sizes according to the image size of the second image, thereby obtaining the first region set;
[0042] The second partitioning unit is used to divide the third image into multiple grid blocks of corresponding sizes according to the image size of the third image, so as to obtain the second region set.
[0043] Optionally, the partitioning determination module includes:
[0044] The second determining unit is used to determine a similarity measure between the first region set and the second region set based on the pixel value at the target coordinate in the first region set and the pixel value at the target coordinate in the second region set.
[0045] The third determining unit is used to determine the global transformation matrix based on the similarity metric.
[0046] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the image registration method as described in any one of the first aspects.
[0047] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the image registration method as described in any one of the first aspects.
[0048] Fifthly, embodiments of this application also provide a computer program product, which is stored in a storage medium and executed by at least one processor to implement the steps of the image registration method as described in any one of the first aspects.
[0049] In this embodiment, by fusing local and global features of each of the at least two images to be registered, a set of feature points corresponding to each of the at least two images to be registered is obtained. Then, based on the set of feature points corresponding to each of the at least two images to be registered, the first image and the second image are aligned to obtain a third image. Finally, the second image and the third image are divided into regions to obtain a first region set and a second region set, and a global transformation matrix is determined based on the first region set and the second region set. The second image and the third image are then aligned according to the global transformation matrix to obtain the registered image corresponding to the first image. It can be seen that this embodiment adopts a two-stage registration process. The initial stage achieves coarse image alignment based on feature points, while the refinement stage performs fine alignment. This approach can adapt to image registration in various complex environments, improving not only the accuracy and quality of registration but also the robustness of image registration. Attached Figure Description
[0050] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0051] Figure 1 This is one of the flowcharts of an image registration method in the embodiments of this application;
[0052] Figure 2 This is a second flowchart of an image registration method in the embodiments of this application;
[0053] Figure 3 This is a schematic diagram of the image registration device provided in the embodiments of this application;
[0054] Figure 4 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation
[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0056] To make the embodiments of this application clearer, the relevant technical knowledge involved in the embodiments of this application will be introduced as follows:
[0057] Image registration, an important technique in computer vision and image processing, involves aligning two or more images according to certain geometric transformations for further analysis and processing. It has wide applications in industries such as industry, medicine, and aerospace. In related technical fields, image registration methods mainly include feature-point-based registration, gray-value-based registration, and frequency-domain-based registration.
[0058] First, feature-point-based image registration methods mainly include feature point detection, description, matching, and transformation model estimation. First, key feature points are extracted from the two images to be registered using robust local feature point detection and description algorithms (Speeded-Up Robust Features, SURF) or high-speed, stable feature point detection and extraction algorithms (Oriented FAST and Rotated BRIEF, ORB). These algorithms detect salient points in the images, such as corner points or edge points, which serve as the basis for subsequent registration. Then, each feature point is described using descriptors, encoding it into a feature vector. Next, distance metrics (such as Euclidean distance) are used to match the feature vectors of the two images, finding a best-matching pair of feature points. Finally, based on these matched point pairs, a geometric transformation model between the two images, such as affine or perspective transformation, is estimated using methods like random sample consensus. This transformation model allows one image to be mapped to the coordinate system of the other image, achieving registration.
[0059] Secondly, image registration methods based on grayscale values typically begin by setting initial transformation parameters, assuming a known transformation relationship between the two images, such as translation or rotation. Then, the transformation parameters are optimized by comparing the grayscale values of the aligned images. Common methods include mutual information and correlation coefficient methods. The mutual information method finds the optimal transformation parameters by maximizing the mutual information of the joint probability distribution of the grayscale values of the two images, while the correlation coefficient method determines the optimal registration state by maximizing the correlation coefficient between the grayscale values of the two images. During implementation, the algorithm continuously adjusts the image transformation parameters to reduce registration errors, ultimately obtaining the optimal overlap position between the two images.
[0060] Finally, the frequency-domain-based image registration method achieves this by transforming the images from the spatial domain to the frequency domain. A Fourier transform is applied to convert the two images into the frequency domain, representing them as a combination of frequency components. In the frequency domain, image translation corresponds to a linear change in phase; therefore, the amount of translation between the images can be detected by calculating the phase correlation of the frequency domain images. The phase correlation, when transformed back to the spatial domain by the inverse Fourier transform, manifests as a peak, the location of which corresponds to the optimal amount of image translation. For rotation and scale changes, this method typically incorporates polar coordinate transformations, allowing these transformations to also be handled through phase correlation. Through these steps, the translation, rotation, and scale transformations between the two images can be determined, achieving registration.
[0061] However, while the above registration methods have their advantages in different application scenarios, they also have some drawbacks. Feature-point-based image registration schemes are sensitive to changes in image scale, rotation, and illumination, especially in complex scenes or low-quality images, which may lead to inaccurate feature point matching and thus affect registration accuracy. Furthermore, this method may fail to match images with significant deformation. Gray-value-based registration schemes are highly dependent on the initial position and have high computational complexity, especially with large-scale images, where computation time increases significantly. In addition, the registration effect of this method may be affected by illumination changes and noise. Frequency-domain-based registration schemes are sensitive to non-rigid image deformation, and the registration effect may be unsatisfactory in images with limited high-frequency information. Furthermore, the implementation of frequency-domain methods usually requires image interpolation and filtering operations, and these preprocessing steps may introduce additional errors.
[0062] To address the aforementioned technical deficiencies, embodiments of this application provide an image registration method, apparatus, device, medium, and program product. By fusing local and global features of an image to obtain a set of feature points, a more robust fused feature is generated. The image is then aligned twice using the feature point set. In the initial stage, coarse image alignment is achieved based on feature points, while fine alignment is performed in the refinement stage, thereby enhancing the accuracy of feature matching and image registration.
[0063] The following description, in conjunction with the accompanying drawings, details an image registration method, apparatus, device, medium, and program product provided in this application through specific embodiments and application scenarios.
[0064] Please see Figure 1 , Figure 1 This is one of the flowcharts of an image registration method provided in the embodiments of this application. The method includes the following steps:
[0065] Step 101: Fuse the local and global features of each of the at least two images to be registered to obtain the feature point set corresponding to each of the at least two images to be registered.
[0066] It is understandable that before feature fusion is performed on the local and global features of each of the at least two images to be registered, feature extraction is required for each image to be registered. This can be done by extracting the local features of each key point in each image to be registered, as well as the global features of each image to be registered.
[0067] In practical implementation, after extracting the local and global features of each image to be registered, keypoint features are extracted from the local features. Keypoint features are stable feature points identified at different scales of the image. Global features can be a feature map containing global features; each feature point in the feature map needs to be extracted, along with the feature vector corresponding to the location of each feature point. If the location of a feature point lies within a region (non-integer coordinates) of the feature map, the feature of that point can be calculated using interpolation. To ensure that the two extracted features have the same dimension, principal component analysis can be used to reduce the dimensionality of the two feature vectors to the same dimension, obtaining the keypoint feature vector of the local features and the feature vector of the feature point in the global features. Therefore, by weighted fusion of the feature vectors of multiple keypoints in the local features and the feature vectors of multiple feature points in the global features, a feature point set of the image to be registered can be formed, which can be represented as...
[0068] As can be seen, the embodiments of this application fuse local and global features through a weighted average method to generate more robust features, thereby enhancing the adaptability to different environmental changes.
[0069] Step 102: Based on the feature point set corresponding to each of the at least two images to be registered, align the first image of the at least two images to be registered with the second image of the at least two images to be registered to obtain a third image. The first image is any one of the at least two images to be registered, and the second image is a different image from the first image.
[0070] It should be noted that image registration typically involves aligning two or more images according to a certain geometric transformation relationship. Therefore, in this embodiment, at least two images to be registered are used for alignment and processing. These at least two images to be registered may include a first image and a second image. The first and second images can be different images. The second image can be a standard reference image for image registration, used to provide a reference for the registration of one or more first images. For example, if there is a difference in the viewing angle between the first and second images, the first image can be adjusted and transformed using the viewing angle of the second image as the standard.
[0071] In this embodiment of the application, each of the at least two images to be registered can obtain its corresponding feature point set through the above steps. For example, the feature point sets obtained for the first image and the second image of the at least two images to be registered are respectively: and Subsequently, the first and second images in at least two images to be registered can be aligned using gradient descent. The feature points in the first image are mapped to their positions in the second image through a transformation matrix, achieving a rough alignment between the first and second images, and thus obtaining the third image.
[0072] Step 103: Divide the second image and the third image into regions respectively to obtain a first region set and a second region set, and determine the global transformation matrix based on the first region set and the second region set.
[0073] It is worth mentioning that after the initial coarse alignment, the fine registration stage begins. First, a region-based registration method is used to divide the second and third images into several fixed-size grid blocks. Each grid block can be considered an independent region, and multiple grid blocks form a region set. This yields the first region set corresponding to the second image and the second region set corresponding to the third image, denoted as follows: and in, and This represents the i-th region, and N represents the number of grid blocks in the image.
[0074] In a specific embodiment of the above steps, after obtaining the first region set corresponding to the second image and the second region set corresponding to the third image, the global transformation matrix of the image can be adjusted by gradient descent to maximize the similarity measure of all regions. Fine alignment is performed by maximizing region similarity to achieve high-quality image registration, especially to enhance the registration accuracy when the image content is more complex or there are fewer feature points.
[0075] Step 104: Align the third image with the second image according to the global transformation matrix to obtain the registration image corresponding to the first image.
[0076] In specific embodiments of this application, the second image and the third image can be aligned using a global transformation matrix. Regions in the third image are mapped to their positions in the second image using the global transformation matrix, resulting in a registered image corresponding to the first image to be registered. Alternatively, other first images among the at least two images to be registered can also obtain corresponding registered images by aligning and registering them with the second image. Thus, this embodiment of the application achieves refined alignment and registration of the images to be registered, effectively improving the accuracy of image registration, especially achieving high-quality image registration when the image content is complex or has few feature points.
[0077] Optionally, the step of fusing local and global features of each of the at least two images to be registered to obtain a set of feature points corresponding to each of the at least two images to be registered includes:
[0078] The Scale Invariant Feature Transform (SIFT) algorithm is used to extract features from at least two images to be registered, thereby obtaining the local features of each of the at least two images to be registered.
[0079] Convolutional Neural Network (CNN) feature extraction is performed on the at least two images to be registered to obtain the global features of each of the at least two images to be registered.
[0080] According to the target weight, the local features and the global features of each of the at least two images to be registered are fused to obtain the feature point set corresponding to each of the at least two images to be registered.
[0081] The target weights include a first weight and a second weight. The first weight is dynamically adjusted based on the response intensity of feature points in the local features, and the second weight is dynamically adjusted based on the response intensity of feature points in the global features.
[0082] In some embodiments, feature extraction is performed on each input image to be registered. Scale-Invariant Feature Transform (SIFT) matching algorithm can be used as a local feature extraction tool to extract local keypoint features from the image to be registered. Convolutional Neural Networks (CNN) algorithms can be used as a global feature extraction tool to capture global features of the image to be registered.
[0083] Specifically, the local features of each image to be registered extracted by SIFT can be represented as: F SIFT ={d1,d2,...,d n}; where F SIFT d represents the set of key points of the extracted local features; i This represents the descriptor (i.e., the feature vector of the keypoint) for the i-th keypoint. The result of CNN feature extraction can be a feature map containing global features, which can be represented as F. cnn .
[0084] Furthermore, after extracting the global and local features from the image to be registered, F can be... SIFT Key points in F are mapped to F cn The corresponding position in n. For each keypoint, its coordinates on the original input image to be registered are (x... i ,y i Assuming the original input image to be registered is of size W×H, F cnn The size is W cnn ×H cnn F can be calculated using the downsampling factor ρ. SIFT The key point is in F. cnn At the corresponding position, the downsampling factor ρ can be expressed as:
[0085]
[0086] We can obtain F SIFT Key points in the map are mapped to feature map F. cnn The coordinates on are:
[0087]
[0088] Among them, (x icnn ,y icnn ) represents each feature point on the feature map.
[0089] Specifically, F SIFT Key points in the map are mapped to feature map F. cnn Up, and extract Fcnn The feature vector c corresponding to this position i If F cnn If the location lies within a certain region (i.e., within non-integer coordinates), then the features of that point can be calculated using interpolation methods. To ensure that the two extracted features have the same dimension, principal component analysis is used to reduce the dimensionality of the two feature vectors to the same dimension, thus obtaining the SIFT features of that point. and CNN features Subsequently, the aligned SIFT features and CNN features in each image to be registered can be weighted and fused to form a fused feature vector f. i , can be represented as:
[0090]
[0091] Where, α i Indicates the first weight; β i This indicates the second weight.
[0092] It should be noted that the fused feature vector f i The first and second weights in the fusion can be dynamically adjusted based on the response intensity of each feature point's mapping location. Features with stronger response intensity will receive higher weights to ensure that more important features occupy a larger proportion in the fusion. Specifically, The response intensity is the local maximum value of the gradient magnitude, denoted as . It can reflect the saliency of the local structure (such as edges, corners, etc.) of the key point in the image. The response intensity can be represented by the L2 norm of the eigenvector at that location, denoted as . The response intensity can reflect the importance of the location in high-level semantics, such as object outlines, textures, etc.
[0093] Furthermore, the fusion weights can be calculated based on the response intensity, as shown in the following formula:
[0094]
[0095] In this way, by adaptively and dynamically adjusting the weights, feature fusion can better adapt to local changes in the image, such as occlusion and illumination variations. Furthermore, adaptive adjustment can better balance local and global information in different images or regions, enhancing the expressive power of the fused features.
[0096] Optionally, the step of aligning the first image among the at least two images to be registered with the second image among the at least two images to be registered, based on the feature point set corresponding to each of the at least two images to be registered, to obtain the third image, includes:
[0097] Based on the feature point set corresponding to the first image and the feature point set corresponding to the second image, determine the alignment transformation matrix;
[0098] According to the alignment transformation matrix, the first image is aligned with the second image as a reference to obtain the third image.
[0099] In some optional embodiments, the feature point sets corresponding to the first image and the second image in at least two images to be registered can be represented as F1 = {f1} 1 f1 2 ,...,f1 n} and F2={f2 1 f2 2 ,...,f2 n}, where f1 n and f2 n For matching pairs, the alignment transformation matrix T1, determined by gradient descent, can be expressed as:
[0100]
[0101] in, This indicates that the feature points in the first image are mapped to their positions in the second image using the alignment transformation matrix T1.
[0102] Based on the above, the first image and the second image can be roughly aligned using the alignment transformation matrix T1 to obtain the third image.
[0103] Optionally, aligning the first image with reference to the second image according to the alignment transformation matrix to obtain the third image includes:
[0104] Based on the alignment transformation matrix, the feature points in the first image are mapped to the positions of the corresponding feature points in the second image to obtain the third image.
[0105] It is understood that the specific implementation of this embodiment can be found in the relevant descriptions in the foregoing embodiments. To avoid repetition, this embodiment will not repeat the descriptions.
[0106] Optionally, the step of dividing the second image and the third image into regions to obtain a first region set and a second region set includes:
[0107] Based on the image size of the second image, the second image is divided into multiple grid blocks of corresponding size to obtain the first region set;
[0108] Based on the image size of the third image, the third image is divided into multiple grid blocks of corresponding size to obtain the second region set.
[0109] In some embodiments, a region-based registration method can be used to divide the image into several fixed-size grid blocks according to the image size. Specifically, these can be grid blocks of 8x8, 16x16, 32x32 pixels, etc. This application does not impose a specific limitation on this; the grid block can be set according to actual needs. As described in the foregoing embodiments, each grid block can be considered an independent region. Multiple grid blocks form a region set, resulting in a first region set corresponding to the second image and a second region set corresponding to the third image, respectively represented as follows: and in, and Let N represent the i-th region, and N represent the number of grid blocks in the image. Thus, this embodiment of the application divides the image into multiple feature grid blocks to more comprehensively describe the image content, thereby providing more information during feature matching and increasing the accuracy of the matching.
[0110] Optionally, determining the global transformation matrix based on the first region set and the second region set includes:
[0111] Based on the pixel values at the target coordinates within the first region set and the pixel values at the target coordinates within the second region set, a similarity metric is determined between the first region set and the second region set.
[0112] The global transformation matrix is determined based on the similarity metric.
[0113] In some embodiments, for the first and second region sets, the global transformation matrix T2 of the image can be adjusted using gradient descent to maximize the similarity measure of all regions. The objective function for optimization can be expressed as:
[0114]
[0115] in, This represents a similarity measure between the first set of regions and the second set of regions.
[0116] Specifically, the similarity measure is determined by comparing the differences in pixel values within the first region set and the second region set. The specific formula can be expressed as:
[0117]
[0118] Where N represents the number of pixels in the region;
[0119] I1(x,y) and I2(x,y) represent the pixel values of the first and second images at coordinates (x,y), respectively.
[0120] In this way, fine alignment of images is achieved by maximizing regional similarity, which enables higher quality image registration even when the image content is complex or there are few feature points, thus improving the robustness of image registration.
[0121] It is worth mentioning that the embodiments of this application do not rely on specific frequency characteristics of images, and can be applied to a wider range of application scenarios, including but not limited to industrial vision, medical imaging, satellite image analysis, and video surveillance. Through the multi-feature fusion, adaptive weight allocation mechanism, and two image alignment processes in the embodiments of this application, the accuracy and robustness of image registration in various fields and scenarios are effectively improved.
[0122] like Figure 2 As shown in the figure, this application embodiment also provides an image registration method, which specifically includes the following steps:
[0123] Step 201: Perform Scale Invariant Feature Transform (SIFT) feature extraction on at least two images to be registered to obtain local features of each image to be registered.
[0124] Step 202: Perform convolutional neural network (CNN) feature extraction on at least two images to be registered to obtain the global features of each image to be registered in at least two images to be registered.
[0125] Step 203: According to the target weight, fuse the local features and global features of each of the at least two images to be registered to obtain the feature point set corresponding to each of the at least two images to be registered; the target weight includes a first weight and a second weight, the first weight is dynamically adjusted based on the response intensity of the feature points in the local features, and the second weight is dynamically adjusted based on the response intensity of the feature points in the global features;
[0126] Step 204: Based on the feature point set corresponding to the first image and the feature point set corresponding to the second image, determine the alignment transformation matrix; according to the alignment transformation matrix, align the first image with the second image as a reference to obtain the third image;
[0127] Step 205: Based on the image size of the second image, divide the second image into multiple grid blocks of corresponding size to obtain the first region set; based on the image size of the third image, divide the third image into multiple grid blocks of corresponding size to obtain the second region set;
[0128] Step 206: Based on the pixel values at the target coordinates within the first region set and the pixel values at the target coordinates within the second region set, determine the similarity metric between the first region set and the second region set; based on the similarity metric, determine the global transformation matrix;
[0129] Step 207: Align the third image with the second image according to the global transformation matrix to obtain the registration image corresponding to the first image.
[0130] It is understood that this embodiment of the present application is specifically described in the relevant descriptions in the above embodiments of the present application, and will not be repeated here to avoid repetition.
[0131] Please see Figure 3 This application provides an image registration device 300, which includes:
[0132] The feature fusion module 301 is used to fuse the local features and global features of each of the at least two images to be registered, respectively, to obtain the feature point set corresponding to each of the at least two images to be registered;
[0133] The first alignment module 302 is used to align the first image in the at least two images to be registered with the second image in the at least two images to be registered based on the feature point set corresponding to each of the at least two images to be registered, to obtain a third image. The first image is any one of the at least two images to be registered, and the second image is a different image from the first image.
[0134] The region division determination module 303 is used to divide the second image and the third image into regions respectively, to obtain a first region set and a second region set, and to determine a global transformation matrix based on the first region set and the second region set;
[0135] The second alignment module 304 is used to align the third image with the second image according to the global transformation matrix to obtain the registration image corresponding to the first image.
[0136] Optionally, the feature fusion module 301 includes:
[0137] The first extraction unit is used to perform scale-invariant feature transform matching algorithm SIFT feature extraction on at least two images to be registered, so as to obtain the local features of each of the at least two images to be registered.
[0138] The second extraction unit is used to perform convolutional neural network (CNN) feature extraction on the at least two images to be registered, so as to obtain the global features of each of the at least two images to be registered.
[0139] The feature fusion unit is used to fuse the local features and the global features of each of the at least two images to be registered according to the target weight, so as to obtain the feature point set corresponding to each of the at least two images to be registered.
[0140] The target weights include a first weight and a second weight. The first weight is dynamically adjusted based on the response intensity of feature points in the local features, and the second weight is dynamically adjusted based on the response intensity of feature points in the global features.
[0141] Optionally, the first alignment module 302 includes:
[0142] The first determining unit is used to determine the alignment transformation matrix based on the feature point set corresponding to the first image and the feature point set corresponding to the second image;
[0143] The first alignment unit is used to align the first image with the second image as a reference according to the alignment transformation matrix to obtain the third image.
[0144] Optionally, the first alignment unit is used for:
[0145] Based on the alignment transformation matrix, the feature points in the first image are mapped to the positions of the corresponding feature points in the second image to obtain the third image.
[0146] Optionally, the partitioning determination module 303 includes:
[0147] The first partitioning unit is used to divide the second image into multiple grid blocks of corresponding sizes according to the image size of the second image, thereby obtaining the first region set;
[0148] The second partitioning unit is used to divide the third image into multiple grid blocks of corresponding sizes according to the image size of the third image, so as to obtain the second region set.
[0149] Optionally, the partitioning determination module 303 includes:
[0150] The second determining unit is used to determine a similarity measure between the first region set and the second region set based on the pixel value at the target coordinate in the first region set and the pixel value at the target coordinate in the second region set.
[0151] The third determining unit is used to determine the global transformation matrix based on the similarity metric.
[0152] It should be noted that the image registration device provided in this application embodiment is capable of achieving the above-mentioned... Figure 1 or Figure 2 All method steps implemented in the method embodiment shown are capable of achieving the same technical effect. Therefore, the parts that are the same as those in the method embodiment and their beneficial effects will not be described in detail here.
[0153] This application also provides an electronic device. See also... Figure 4 , Figure 4 This is a structural diagram of the electronic device provided in the embodiments of this application. Since the principle by which the electronic device 400 solves the problem is similar to the image registration method in the embodiments of this application, the implementation of the electronic device 400 can be found in... Figure 1 The implementation of the method shown will not be repeated here.
[0154] like Figure 4 As shown, the electronic device 400 includes: a processor 410, configured to read a program from the memory 420 and execute the following processes:
[0155] The local and global features of each of the at least two images to be registered are fused to obtain the feature point set corresponding to each of the at least two images to be registered.
[0156] Based on the feature point set corresponding to each of the at least two images to be registered, the first image of the at least two images to be registered is aligned with the second image of the at least two images to be registered to obtain a third image. The first image is any one of the at least two images to be registered, and the second image is a different image from the first image.
[0157] The second image and the third image are divided into regions respectively to obtain a first region set and a second region set, and a global transformation matrix is determined based on the first region set and the second region set;
[0158] The third image is aligned with the second image according to the global transformation matrix to obtain the registration image corresponding to the first image.
[0159] Among them, Figure 4 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 410 and memory represented by memory 420 together. The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides the interface.
[0160] Optionally, the processor 410 is also used to read the program from the memory 420 and perform the following steps:
[0161] The Scale Invariant Feature Transform (SIFT) algorithm is used to extract features from at least two images to be registered, thereby obtaining the local features of each of the at least two images to be registered.
[0162] Convolutional Neural Network (CNN) feature extraction is performed on the at least two images to be registered to obtain the global features of each of the at least two images to be registered.
[0163] According to the target weight, the local features and the global features of each of the at least two images to be registered are fused to obtain the feature point set corresponding to each of the at least two images to be registered.
[0164] The target weights include a first weight and a second weight. The first weight is dynamically adjusted based on the response intensity of feature points in the local features, and the second weight is dynamically adjusted based on the response intensity of feature points in the global features.
[0165] Optionally, the processor 410 is also used to read the program from the memory 420 and perform the following steps:
[0166] Based on the feature point set corresponding to the first image and the feature point set corresponding to the second image, determine the alignment transformation matrix;
[0167] According to the alignment transformation matrix, the first image is aligned with the second image as a reference to obtain the third image.
[0168] Optionally, the processor 410 is also used to read the program from the memory 420 and perform the following steps:
[0169] Based on the alignment transformation matrix, the feature points in the first image are mapped to the positions of the corresponding feature points in the second image to obtain the third image.
[0170] Optionally, the processor 410 is also used to read the program from the memory 420 and perform the following steps:
[0171] Based on the image size of the second image, the second image is divided into multiple grid blocks of corresponding size to obtain the first region set;
[0172] Based on the image size of the third image, the third image is divided into multiple grid blocks of corresponding size to obtain the second region set.
[0173] Optionally, the processor 410 is also used to read the program from the memory 420 and perform the following steps:
[0174] Based on the pixel values at the target coordinates within the first region set and the pixel values at the target coordinates within the second region set, a similarity metric is determined between the first region set and the second region set.
[0175] The global transformation matrix is determined based on the similarity metric.
[0176] The electronic device 400 provided in this application embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.
[0177] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the above-described... Figure 1 or Figure 2 The various processes of the image registration method embodiments are all implemented in the same way and can achieve the same technical effect. To avoid repetition, they will not be described again here. The computer-readable storage medium mentioned includes, for example, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.
[0178] This application embodiment also provides a computer program / program product, which is stored in a storage medium and executed by at least one processor to implement the above. Figure 1 or Figure 2 The various processes of the image registration method embodiments described herein can achieve the same technical effect, and will not be repeated here to avoid repetition.
[0179] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0180] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.
[0181] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the transmission and reception methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0182] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. An image registration method, characterized in that, include: The local and global features of each of the at least two images to be registered are fused to obtain the feature point set corresponding to each of the at least two images to be registered. Based on the feature point set corresponding to each of the at least two images to be registered, the first image of the at least two images to be registered is aligned with the second image of the at least two images to be registered to obtain a third image. The first image is any one of the at least two images to be registered, and the second image is a different image from the first image. The second image and the third image are divided into regions respectively to obtain a first region set and a second region set, and a global transformation matrix is determined based on the first region set and the second region set; The third image is aligned with the second image according to the global transformation matrix to obtain the registration image corresponding to the first image.
2. The method according to claim 1, characterized in that, The process of fusing local and global features of each of the at least two images to be registered to obtain a set of feature points corresponding to each of the at least two images to be registered includes: The Scale Invariant Feature Transform (SIFT) algorithm is used to extract features from at least two images to be registered, thereby obtaining the local features of each of the at least two images to be registered. Convolutional Neural Network (CNN) feature extraction is performed on the at least two images to be registered to obtain the global features of each of the at least two images to be registered. According to the target weight, the local features and the global features of each of the at least two images to be registered are fused to obtain the feature point set corresponding to each of the at least two images to be registered. The target weights include a first weight and a second weight. The first weight is dynamically adjusted based on the response intensity of feature points in the local features, and the second weight is dynamically adjusted based on the response intensity of feature points in the global features.
3. The method according to claim 1, characterized in that, The step of aligning the first image among the at least two images to be registered with the second image among the at least two images to be registered, based on the feature point set corresponding to each of the at least two images to be registered, to obtain the third image includes: Based on the feature point set corresponding to the first image and the feature point set corresponding to the second image, determine the alignment transformation matrix; According to the alignment transformation matrix, the first image is aligned with the second image as a reference to obtain the third image.
4. The method according to claim 3, characterized in that, The step of aligning the first image with the second image as a reference according to the alignment transformation matrix to obtain the third image includes: Based on the alignment transformation matrix, the feature points in the first image are mapped to the positions of the corresponding feature points in the second image to obtain the third image.
5. The method according to claim 1, characterized in that, The step of dividing the second image and the third image into regions to obtain a first region set and a second region set includes: Based on the image size of the second image, the second image is divided into multiple grid blocks of corresponding size to obtain the first region set; Based on the image size of the third image, the third image is divided into multiple grid blocks of corresponding size to obtain the second region set.
6. The method according to any one of claims 1 to 5, characterized in that, The step of determining the global transformation matrix based on the first region set and the second region set includes: Based on the pixel values at the target coordinates within the first region set and the pixel values at the target coordinates within the second region set, a similarity metric is determined between the first region set and the second region set. The global transformation matrix is determined based on the similarity metric.
7. An image registration device, characterized in that, include: The feature fusion module is used to fuse the local features and global features of each of the at least two images to be registered, respectively, to obtain the feature point set corresponding to each of the at least two images to be registered; The first alignment module is used to align the first image in the at least two images to be registered with the second image in the at least two images to be registered based on the feature point set corresponding to each of the at least two images to be registered, to obtain a third image. The first image is any one of the at least two images to be registered, and the second image is a different image from the first image. The region division module is used to divide the second image and the third image into regions respectively, to obtain a first region set and a second region set, and to determine a global transformation matrix based on the first region set and the second region set; The second alignment module is used to align the third image with the second image according to the global transformation matrix to obtain the registration image corresponding to the first image.
8. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the image registration method as described in any one of claims 1 to 6.
9. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the image registration method as described in any one of claims 1 to 6.
10. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps in the image registration method as described in any one of claims 1 to 6.