Image registration methods and apparatus, electronic devices, and storage media

Abstract

An image registration method, apparatus, electronic device, and storage medium are disclosed. The method includes: performing feature point detection on a first image and a second image respectively to obtain a first feature point set corresponding to the first image and a second feature point set corresponding to the second image, wherein each first feature point in the first feature point set corresponds one-to-one with each second feature point in the second feature point set; determining a first neighbor subset corresponding to each first feature point from the first feature point set and determining a second neighbor subset corresponding to each second feature point from the second feature point set; calculating spatial transformation parameters based on the first neighbor subsets corresponding to each first feature point and the second neighbor subsets corresponding to each second feature point; and registering the first image and the second image based on the spatial transformation parameters. Implementing the embodiments of this application can improve the accuracy of image registration.

Description

Technical Field

[0001] This application relates to the field of imaging technology, and in particular to an image registration method and apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, when electronic devices encounter multiple frames of images with a certain spatial similarity (such as multiple photos taken consecutively by an electronic device in the same scene), in order to achieve further image processing, it is often necessary to first register these multiple frames to determine the spatial transformation relationship between each frame before they can be applied to subsequent image processing steps. However, in practice, it has been found that traditional image registration schemes have relatively low accuracy, which is not conducive to achieving accurate image registration. Summary of the Invention

[0003] This application discloses an image registration method, apparatus, electronic device, and storage medium, which can improve the accuracy of image registration.

[0004] The first aspect of this application discloses an image registration method, including:

[0005] Feature point detection is performed on the first image and the second image respectively to obtain a first feature point set corresponding to the first image and a second feature point set corresponding to the second image, wherein each first feature point in the first feature point set corresponds one-to-one with each second feature point in the second feature point set.

[0006] Determine a first neighbor subset corresponding to each of the first feature points from the first feature point set, and determine a second neighbor subset corresponding to each of the second feature points from the second feature point set;

[0007] Based on the first neighbor subset corresponding to each of the first feature points and the second neighbor subset corresponding to each of the second feature points, spatial transformation parameters are calculated, and the first image and the second image are registered according to the spatial transformation parameters.

[0008] A second aspect of this application discloses an image registration apparatus, comprising:

[0009] The feature point detection unit is used to perform feature point detection on the first image and the second image respectively, and obtain a first feature point set corresponding to the first image and a second feature point set corresponding to the second image, wherein each first feature point contained in the first feature point set corresponds one-to-one with each second feature point contained in the second feature point set.

[0010] The determining unit is configured to determine a first neighbor subset corresponding to each of the first feature points from the first feature point set, and to determine a second neighbor subset corresponding to each of the second feature points from the second feature point set;

[0011] The calculation unit is used to calculate spatial transformation parameters based on the first neighbor subset corresponding to each of the first feature points and the second neighbor subset corresponding to each of the second feature points, and to register the first image and the second image based on the spatial transformation parameters.

[0012] The third aspect of this application discloses an electronic device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs all or part of the steps in any of the image registration methods disclosed in the first aspect of this application.

[0013] The fourth aspect of this application discloses a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements all or part of the steps in any of the image registration methods disclosed in the first aspect of this application.

[0014] Compared with related technologies, the embodiments of this application have the following beneficial effects:

[0015] In this embodiment, the electronic device can perform feature point detection on the first image and the second image to be registered, respectively, to obtain a first feature point set corresponding to the first image and a second feature point set corresponding to the second image. Each first feature point in the first feature point set corresponds one-to-one with each second feature point in the second feature point set. Based on this, the electronic device can determine a first neighbor subset corresponding to each first feature point from the first feature point set, and a second neighbor subset corresponding to each second feature point from the second feature point set. According to the first neighbor subset corresponding to each first feature point and the second neighbor subset corresponding to each second feature point, the electronic device can further calculate spatial transformation parameters and perform registration on the first image and the second image based on these spatial transformation parameters. Therefore, by implementing this embodiment, the electronic device can determine the neighbor subsets of each corresponding feature point detected in the first image and the second image, respectively, within the corresponding first feature point set and second feature point set. Thus, for each pair of feature points, the spatial transformation parameters used for registering the first image and the second image can be calculated by combining the spatial features of its neighboring feature points with the first and second feature points in each pair. Compared to registration based on a single feature point, using a neighbor subset can utilize more spatial features to more accurately represent the spatial transformation relationship between the first and second images, resulting in higher reliability and robustness of the calculated spatial transformation parameters, which in turn helps improve the accuracy of image registration. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the structure of an image processing circuit disclosed in an embodiment of this application;

[0018] Figure 2 This is a schematic flowchart of an image registration method disclosed in an embodiment of this application;

[0019] Figure 3A These are schematic diagrams of the first and second images disclosed in the embodiments of this application;

[0020] Figure 3B This is a schematic diagram of the first feature point set corresponding to the first image and the second feature point set corresponding to the second image disclosed in the embodiments of this application;

[0021] Figure 3C This is a schematic diagram of the first neighbor subset corresponding to the first target feature point and the second neighbor subset corresponding to the second target feature point disclosed in the embodiments of this application;

[0022] Figure 4 This is a schematic flowchart of another image registration method disclosed in an embodiment of this application;

[0023] Figure 5 This is a schematic flowchart of another image registration method disclosed in the embodiments of this application;

[0024] Figure 6 This is a modular schematic diagram of an image registration device disclosed in an embodiment of this application;

[0025] Figure 7 This is a modular schematic diagram of an electronic device disclosed in an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0027] It should be noted that the terms "comprising" and "having" and any variations thereof in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or devices.

[0028] In related technologies, due to the limited computing power of electronic devices such as mobile phones, smart wearable devices, and tablets, a large amount of continuous computation is often required to achieve certain image processing functions with high precision. This can easily lead to excessive power consumption in electronic devices and reduce their efficiency in image processing. For example, when performing portrait segmentation on original images, traditional portrait segmentation methods are often time-consuming in order to preserve hair details, and electronic devices are also prone to overheating due to excessive power consumption. This application discloses an image registration method and apparatus, electronic device, and storage medium, which can improve the accuracy of image registration.

[0029] The following will be described in detail with reference to the accompanying drawings.

[0030] Please see Figure 1 , Figure 1This is a schematic diagram of an image processing circuit disclosed in an embodiment of this application. This image processing circuit can be used to implement the image registration method disclosed in the embodiments of this application. Exemplarily, this image processing circuit can be applied to electronic devices such as mobile phones, smart wearable devices, and tablet computers, but is not limited thereto. Figure 1 As shown, the image processing circuit may include an imaging device (camera) 110, an attitude sensor 120, an image memory 130, an image signal processing (ISP) processor 140, a control logic unit 150, and a display 160.

[0031] The image data captured by the imaging device 110 can first be processed by the ISP processor 140. For example, the ISP processor 140 can analyze the image data to obtain image statistical information that can be used to determine one or more control parameters of the imaging device 110. The imaging device 110 may include one or more lenses 112 and an image sensor 114. The image sensor 114 may include a color filter array (e.g., a Bayer filter). The image sensor 114 can acquire the light intensity and wavelength information captured by each imaging pixel and provide a set of raw image data that can be processed by the ISP processor 140. The attitude sensor 120 (e.g., a three-axis gyroscope, Hall sensor, accelerometer, etc.) can provide the ISP processor 140 with parameters for image processing (e.g., image stabilization parameters) based on the attitude sensor 120 interface type. The attitude sensor 120 interface can be an SMIA (Standard Mobile Imaging Architecture) interface, other serial or parallel camera interfaces, or a combination of the above interfaces.

[0032] In addition, image sensor 114 can also send raw image data to attitude sensor 120. Attitude sensor 120 can provide the raw image data to ISP processor 140 based on the interface type of attitude sensor 120, or store the raw image data in image memory 130.

[0033] The ISP processor 140 can process raw image data pixel by pixel in various formats. For example, each image pixel can have a bit depth of 8, 10, 12, or 14 bits. The ISP processor 140 can perform one or more image processing operations on the raw image data and collect statistical information about the image data. The image processing operations can be performed with the same or different bit depth precision.

[0034] The ISP processor 140 can also receive image data from the image memory 130. For example, the attitude sensor 120 interface sends raw image data to the image memory 130, and the raw image data in the image memory 130 is then provided to the ISP processor 140 for processing. The image memory 130 may be part of a memory device, a storage device, or a separate dedicated memory within an electronic device, and may include DMA (Direct Memory Access) features.

[0035] Upon receiving raw image data from the image sensor 114 interface, the attitude sensor 120 interface, or the image memory 130, the ISP processor 140 can perform one or more image processing operations, such as temporal filtering, frequency filtering, etc. The processed image data can be sent to the image memory 130 for further processing before display. The ISP processor 140 receives processed data from the image memory 130 and performs image data processing on the processed data in the raw domain and in the RGB and YCbCr color spaces. The processed image data can be output to the display 160 for user viewing and / or further processed by the graphics engine or GPU (Graphics Processing Unit). Furthermore, the output of the ISP processor 140 can also be sent to the image memory 130, and the display 160 can read image data from the image memory 130. In one embodiment, the image memory 130 can be configured to implement one or more frame buffers.

[0036] The statistical data determined by the ISP processor 140 can be sent to the control logic unit 150. For example, the statistical data may include image sensor 114 statistics such as gyroscope vibration frequency, auto exposure, auto white balance, auto focus, flicker detection, black level compensation, and lens 112 shading correction. The control logic unit 150 may include a processor and / or microcontroller executing one or more routines (such as firmware) that determine control parameters for the imaging device 110 and the ISP processor 140 based on the received statistical data. For example, the control parameters for the imaging device 110 may include attitude sensor 120 control parameters (e.g., gain, integration time for exposure control, image stabilization parameters, etc.), camera flash control parameters, camera image stabilization shift parameters, lens 112 control parameters (e.g., focal length for focusing or zooming), or combinations of these parameters. The ISP control parameters may include gain levels and color correction matrices for auto white balance and color adjustment (e.g., during RGB processing), and lens 112 shading correction parameters.

[0037] In some embodiments, the imaging device (camera) 110 can acquire multiple frames of continuously captured images through the lens 112 and image sensor 114, and send these multiple frames of images to the ISP processor 140. Since the multiple frames of continuously captured images often have a certain spatial similarity, the ISP processor 140 can register the aforementioned multiple frames of images to determine the spatial transformation relationship between each frame of images, and then use the registered images to perform further image processing (such as image blurring, image fusion, image super-resolution, etc.). For example, taking the image registration of two frames of images, namely the first image and the second image, as an example, the ISP processor 140 can perform feature point detection on the first image and the second image respectively to obtain a first feature point set corresponding to the first image and a second feature point set corresponding to the second image. Each first feature point in the first feature point set can correspond one-to-one with each second feature point in the second feature point set. Further, the ISP processor 140 can determine a first neighbor subset corresponding to each first feature point from the first feature point set, and a second neighbor subset corresponding to each second feature point from the second feature point set. Based on this, the ISP processor 140 can calculate the spatial transformation parameters according to the first neighbor subset corresponding to each first feature point and the second neighbor subset corresponding to each second feature point, and register the first image and the second image according to the spatial transformation parameters.

[0038] By implementing the above image registration method, the electronic device can calculate the spatial transformation parameters for registering the first and second images by combining the spatial features of the corresponding feature points detected in the first and second images with the spatial features of other neighboring feature points. This can accurately characterize the spatial transformation relationship between the first and second images, which is beneficial to improving the accuracy of image registration.

[0039] Optionally, the ISP processor 140 can also perform further image processing using the registered first and second images. Based on this, the ISP processor 140 can output the processed image data to the display 160 for user viewing.

[0040] In other embodiments, when the ISP processor 140 retrieves the first and second images to be registered from the image memory 130, the first and second images may also be image data originally stored locally on the electronic device, or image data obtained from an external source via a data transmission module. For image data from other sources (i.e., not captured by the imaging device 110), the ISP processor 140 may also use the above-described image registration method to register them without affecting the realization of the corresponding image registration effect.

[0041] Please see Figure 2 , Figure 2 This is a schematic flowchart of an image registration method disclosed in an embodiment of this application. This method can be applied to electronic devices including the image processing circuit described above. Figure 2 As shown, the image registration method may include the following steps:

[0042] 202. Perform feature point detection on the first image and the second image respectively to obtain the first feature point set corresponding to the first image and the second feature point set corresponding to the second image, wherein each first feature point contained in the first feature point set corresponds one-to-one with each second feature point contained in the second feature point set.

[0043] In this embodiment, the first image and the second image to be registered may include images captured by the imaging device of the electronic device, images transmitted to the electronic device via wired or wireless communication, or images stored locally on the electronic device. No specific limitations are made in this embodiment. Both the first image and the second image may include various types of image data, such as color data (including color data based on different color spaces such as YUV and RGB color spaces), texture data, etc.

[0044] To perform image registration on the first and second images, the electronic device can use its image processing circuit to detect feature points in both images and obtain corresponding feature points based on the detection results. Specifically, each first feature point in the first image can form a first feature point set, and each second feature point in the second image that corresponds one-to-one with each of the first feature points can form a second feature point set.

[0045] Specifically, after acquiring the first image and the second image to be registered, the electronic device can perform feature point detection on the first image and the second image respectively based on a certain feature point detection algorithm to obtain a first set of detected feature points containing all feature points in the first image and a second set of detected feature points containing all feature points in the second image. Based on this, the electronic device can perform preliminary screening on the first and second sets of detected feature points to obtain one or more pairs of first and second feature points that have a corresponding relationship, so that the screened first feature points can form the first feature point set, and the screened second feature points can form the second feature point set. It can be understood that the first feature point set is a subset of the first set of detected feature points, and the second feature point set is a subset of the second set of detected feature points.

[0046] For example, different feature point detection algorithms can be used to detect different types of feature points in the first image and the second image, such as Harris corner points, ORB (Oriented FAST and Rotated BRIEF) keypoints, SIFT (Scale-Invariant Feature Transform) feature points, SURF (Speeded UpRobust Features) feature points, etc., and no specific limitation is made in the embodiments of this application.

[0047] Taking the detection of SIFT feature points in the first image as an example, by performing a scale transformation on the first image, its spatial representation at multiple scales is obtained. Then, a Gaussian pyramid and a Difference of Gaussian (DoG) pyramid are constructed. This facilitates finding the extreme points of the corresponding Difference of Gaussian (DoG) function in the scale space and further determining suitable SIFT feature points. It can be understood that feature point detection in the second image follows a similar process; using the same feature point detection algorithm, the same type of feature points can be detected in the second image.

[0048] Furthermore, after detecting all feature points in the first and second images, the electronic device can utilize the grayscale information near each feature point to construct a corresponding grayscale feature descriptor to characterize the spatial features of each feature point. Taking the SIFT feature points in the first image as an example, each SIFT feature point in the first image can be characterized through the corresponding SIFT descriptor, and this characterization has scale invariance.

[0049] Based on this, for the aforementioned first and second detection feature point sets, the similarity between each feature point can be compared based on the grayscale feature descriptor corresponding to each feature point. This allows for the preliminary selection of corresponding first and second feature points from the first and second detection feature point sets, respectively. The aforementioned correspondence indicates that the first feature points in the first image and the second feature points in the second image are spatially similar. That is, by mapping each first feature point to its corresponding second feature point, or vice versa, registration of the first and second images can be achieved, allowing them to be mutually mapped to each other.

[0050] For the first and second feature points with corresponding relationships initially selected above, the electronic device can further eliminate any possible mismatched feature point pairs. In some embodiments, a fixed threshold can be used to determine the similarity between the first and second feature points, and mismatched feature point pairs with similarity below the fixed threshold can be eliminated (i.e., mismatched first feature points are eliminated from the first feature point set, and mismatched second feature points are correspondingly eliminated from the second feature point set). For example, the electronic device can convert the coordinates of the corresponding first and second feature points to the same coordinate system, and then determine whether the distance between the first and second feature points exceeds the fixed threshold. If it exceeds the fixed threshold, the first and second feature points are confirmed as mismatched feature point pairs, and eliminated from the first and second feature point sets respectively. As another example, the electronic device can also calculate the similarity parameter between the grayscale feature descriptors corresponding to the corresponding first and second feature points, and determine whether the similarity parameter exceeds the fixed threshold. If it exceeds the fixed threshold, the first and second feature points are confirmed as mismatched feature point pairs and eliminated.

[0051] In other embodiments, the RANSAC (Random Sample Consensus) algorithm can also be used to remove outlier first and second feature points, thereby eliminating the corresponding mismatched feature point pairs.

[0052] For example, please refer to the following: Figure 3A as well as Figure 3B .like Figure 3A As shown, after feature point detection is performed on the first image 31 and the second image 32, each detected feature point can be represented by a hollow circle. The feature points shown in the first image 31 constitute the first set of detected feature points, and the feature points shown in the second image 32 constitute the second set of detected feature points. Based on this, through preliminary screening of the first and second sets of detected feature points, one or more pairs of first and second feature points with corresponding relationships are obtained. After further eliminating mismatched feature point pairs, as shown... Figure 3B As shown, the remaining first and second feature points with corresponding relationships can be represented by a triangle. Figure 3B The image shows (for ease of understanding) Figure 3B (Only a portion is shown, connected by dashed lines). The first feature points shown in the first image 31 constitute the first feature point set, and the second feature points shown in the second image 32 constitute the second feature point set.

[0053] 204. Determine the first neighbor subset corresponding to each first feature point from the aforementioned first feature point set, and determine the second neighbor subset corresponding to each second feature point from the aforementioned second feature point set.

[0054] In this embodiment, after the electronic device acquires the first feature point set corresponding to the first image and the second feature point set corresponding to the second image, it can determine several neighboring feature points from the corresponding feature point set for each feature point in the first and second feature point sets, thereby obtaining a first neighbor subset corresponding to each first feature point and a second neighbor subset corresponding to each second feature point. In subsequent steps, using the above-mentioned neighbor subsets can more accurately characterize the spatial transformation relationship between each first feature point and its corresponding second feature point, thereby more accurately obtaining the spatial transformation relationship between the first and second images, which is beneficial to improving the reliability and robustness of the calculated spatial transformation parameters.

[0055] For example, the electronic device can select a specified number of other first feature points near each first feature point, and form a first neighbor subset corresponding to each first feature point with the specified number of first feature points. It should be noted that the other first feature points in the first neighbor subset are selected from the first feature point set, not from the first detection feature point set, thereby ensuring that each first feature point and its corresponding first neighbor subset have passed the above preliminary screening and have corresponding second feature points in the second feature point set, which is beneficial to improving the accuracy and reliability of the subsequent image registration process.

[0056] In one embodiment, for a first target feature point, the electronic device can select the top N (N is a positive integer) first feature points from the set of first feature points, arranged in ascending order of distance from the first target feature point, to form a first neighbor subset corresponding to the first target feature point. The first target feature point can be any first feature point from the aforementioned set of first feature points. For example, please refer to... Figure 3C ,exist Figure 3C The first target feature point can be represented by the black triangle in the first image 31, and the first neighbor subset corresponding to the first target feature point can be outlined by a circle 31a centered on the first target feature point (that is, other first feature points in circle 31a besides the first target feature point are shown by blank triangles). It can be understood that... Figure 3C The first neighbor subset shown contains 5 other first feature points, i.e., N=5. This is just an example and should not be regarded as a limitation on the value of N.

[0057] Similarly, electronic devices can use the same method to determine the second neighbor subset corresponding to each second feature point from the aforementioned set of second feature points. Taking a second target feature point as an example, for a second target feature point that corresponds to the aforementioned first target feature point, the electronic device can also select the top N second feature points from the set of second feature points in ascending order of distance from the second target feature point to form the second neighbor subset corresponding to that second target feature point. For example, please continue reading. Figure 3C ,exist Figure 3C The second target feature point can be represented by the black triangle in the second image 32, and the circle 32a centered on the second target feature point can outline the second neighbor subset corresponding to the second target feature point (that is, other second feature points in circle 32a besides the second target feature point, shown by the blank triangle). The second neighbor subset also contains 5 other second feature points.

[0058] It should be noted that although the first target feature point and the second target feature point have a correspondence, the first and second feature points contained in the first neighbor subset corresponding to the first target feature point and the second neighbor subset corresponding to the second target feature point may not all have a correspondence. For example Figure 3C As shown, among the five first feature points in circle 31a other than the first target feature point (marked with a black triangle), only three first feature points correspond to the corresponding second feature points in circle 32a. Taking the corresponding first and second neighbor subsets each containing N feature points as an example, if there are N pairs of feature points with corresponding relationships (i.e., each first and second feature point contained in the first and second neighbor subsets has a corresponding relationship), then the first and second neighbor subsets can be directly applied to the subsequent image registration steps; if there are M (M < N) pairs of feature points with corresponding relationships, then the first and second feature points contained in these M pairs can be used to form new first and second neighbor subsets respectively, and the updated first and second neighbor subsets can be directly applied to the subsequent image registration steps.

[0059] 206. Based on the first neighbor subsets corresponding to each first feature point and the second neighbor subsets corresponding to each second feature point, calculate the spatial transformation parameters, and register the first image and the second image based on the spatial transformation parameters.

[0060] In this embodiment of the application, after the electronic device determines the first neighbor subset corresponding to each first feature point and the second neighbor subset corresponding to each second feature point, it can calculate the spatial transformation parameters based on the first and second neighbor subsets. The spatial transformation parameters can be used to register the first and second images.

[0061] For example, the spatial transformation parameters described above may include mapping parameters, such as mapping vectors, mapping matrices, affine transformation parameters, etc. In some embodiments, after the electronic device calculates the corresponding mapping vectors or mapping matrices based on the respective first and second neighbor subsets, it can perform matrix operations on the first image (or second image) according to the mapping vectors or mapping matrices to obtain the registered first image (or second image). In other embodiments, after calculating the affine transformation parameters, the electronic device can rotate the first image (or second image) to be registered according to the transformation matrix in the affine transformation parameters, and then translate the rotated first image (or second image) according to the transformation vector in the affine transformation parameters to obtain the registered first image (or second image) after affine transformation.

[0062] It should be noted that, when the image to be registered includes multiple frames, the electronic device can use one frame as a reference image as the first image, and sequentially use the other frames as second images, and then use the image registration method in this embodiment to register the first image and each frame of the second image sequentially. In some embodiments, the electronic device can also first select an initial first image and a second image, register the first image and the second image, and then use the registered first image or the second image as a reference to continue registering other images, and repeat the above process until all images are registered.

[0063] Optionally, in this embodiment, after acquiring the first feature point set corresponding to the first image and the second feature point set corresponding to the second image, the electronic device can determine the neighbor subsets corresponding to each feature point in parallel. During the calculation of spatial transformation parameters based on the first neighbor subsets corresponding to each first feature point and the second neighbor subsets corresponding to each second feature point, operations involving each feature point or each neighbor subset can be accelerated through parallelization, thereby further improving the efficiency of image registration performed by the electronic device.

[0064] As can be seen, by implementing the image registration method described in the above embodiments, the electronic device can determine its neighbor subsets in the corresponding first feature point set and second feature point set for each corresponding feature point detected in the first and second images. Thus, for each pair of feature points, the spatial transformation parameters used for registering the first and second images can be jointly calculated by combining the spatial features of its neighboring feature points with the first and second feature points in each pair. By using neighbor subsets, more spatial features in the images to be registered can be utilized to more accurately represent the spatial transformation relationship between the first and second images, resulting in higher reliability and robustness of the calculated spatial transformation parameters, thereby improving the accuracy of image registration.

[0065] Please see Figure 4 , Figure 4 This is a schematic flowchart of another image registration method disclosed in an embodiment of this application. This method can be applied to electronic devices including the above-described image processing circuit. Figure 4 As shown, the image registration method may include the following steps:

[0066] 402. Perform feature point detection on the first image and the second image respectively to obtain the first feature point set corresponding to the first image and the second feature point set corresponding to the second image, wherein each first feature point contained in the first feature point set corresponds one-to-one with each second feature point contained in the second feature point set.

[0067] Step 402 is similar to step 202 above, and will not be described again here.

[0068] 404. From the set of first feature points, select the first N first feature points in ascending order of distance from each first feature point to form the first nearest neighbor subset corresponding to each first feature point; and from the set of second feature points, select the first N second feature points in ascending order of distance from each second feature point to form the second nearest neighbor subset corresponding to each second feature point, where N is a positive integer.

[0069] In this embodiment, when the electronic device needs to determine the first nearest subset corresponding to each first feature point, it can select a specified number of other first feature points near each first feature point in its set of first feature points. This allows the first nearest subset composed of the other first feature points to more accurately represent the spatial characteristics compared to a single first feature point. Similarly, the electronic device can also use the same method to select the same number of other second feature points near each corresponding second feature point in the set of second feature points to form corresponding second nearest subsets.

[0070] In some embodiments, taking a first target feature point in a first feature point set as an example, the electronic device can first calculate the Euclidean distance between the first target feature point and all first feature points (excluding the first target feature point itself) within a certain distance range. Then, it sorts the first feature points according to the Euclidean distance and selects the top N (N is a positive integer) first feature points in ascending order of Euclidean distance to form the first nearest neighbor subset corresponding to the first target feature point. Optionally, if the number of all first feature points within the aforementioned certain distance range is less than N, the distance range can be expanded until N other first feature points closest to the first target feature point are determined. It is understood that the electronic device can also determine the second nearest neighbor subset corresponding to the second target feature point in a similar manner.

[0071] As an optional implementation, after determining the first neighbor subset corresponding to each first feature point and the second neighbor subset corresponding to each second feature point, the electronic device can further remove feature points that do not have a corresponding relationship from the aforementioned first neighbor subset and second neighbor subset to update the first neighbor subset and second neighbor subset. For example, the electronic device can determine whether there is a corresponding relationship between the first feature points and second feature points contained in the corresponding first neighbor subset and second neighbor subset (i.e., whether they belong to one or more pairs of feature points obtained in the preliminary screening), and remove feature points that do not have a corresponding relationship, retaining the feature point pairs that do have a corresponding relationship. Updating the first neighbor subset and second neighbor subset based on the retained feature point pairs can avoid situations where a first feature point belongs to the first neighbor subset, but its corresponding second feature point does not belong to the corresponding second neighbor subset (and vice versa), leading to deviations when calculating spatial transformation parameters based on the first neighbor subset and second neighbor subset, thereby ensuring the reliability of image registration as much as possible.

[0072] 406. Based on the first neighbor subset corresponding to each first feature point and the second neighbor subset corresponding to each second feature point, obtain the fusion structure information corresponding to each feature point pair respectively, wherein each feature point pair consists of a first feature point and a corresponding second feature point.

[0073] In this embodiment of the application, in order to register the first image and the second image, the spatial transformation relationship between the first feature point set corresponding to the first image and the second feature point set corresponding to the second image can be determined first. Based on the first neighbor subsets corresponding to each first feature point in the first feature point set and the second neighbor subsets corresponding to each second feature point in the second feature point set, the fusion structure information corresponding to each feature point pair can be obtained. Then, based on this fusion structure information, the spatial transformation relationship between the first feature point set and the second feature point set can be accurately determined.

[0074] For example, the aforementioned fusion structure information may include a fusion distance, which can be obtained by fusing various distance information, including shape context distance, grayscale descriptor distance, Euclidean distance, etc. By fusing multiple distance information, a more accurate similarity metric can be constructed, making the fused feature descriptor have stronger descriptive power and facilitating accurate and effective determination of the correspondence between the first feature point and the second feature point and its corresponding accuracy.

[0075] 408. Calculate the similarity probability of each feature point pair based on the fusion structure information corresponding to each feature point pair.

[0076] In this embodiment, after the electronic device acquires the fusion structure information corresponding to each feature point pair, it can calculate the similarity probability of each feature point pair based on the fusion structure information to characterize the similarity between each pair of first and second feature points, while avoiding the defects of one-to-one hard decision. For example, the fusion structure information includes the fusion distance d corresponding to a certain feature point pair. m For example, an electronic device can calculate the similarity probability of a feature point pair based on the following formula 1.

[0077] Formula 1:

[0078]

[0079] in, It can represent the m-th first feature point x in the first feature point set X. m The m-th second feature point y in the second feature point set Y m The similarity probability between them, the fusion distance d mentioned above m It can be based on the first feature point x m The corresponding first nearest subset Second feature point y m The corresponding second nearest subset The obtained. The above σ 2 It can represent iteration parameters, which are used to control the iteration precision in subsequent steps.

[0080] 410. Based on the similarity probabilities of each feature point pair, determine the correspondence matrix between the first feature point set and the second feature point set.

[0081] In this embodiment of the application, the correspondence matrix P between the first feature point set and the second feature point set can be as shown in Formula 2 below.

[0082] Formula 2:

[0083]

[0084] Wherein, assuming that the first feature point set X and the second feature point set Y each contain M first feature points and M second feature points respectively, then the correspondence matrix P can be an M-row M-column matrix P MM To determine the correspondence matrix P MM In addition to calculating the similarity probability of each of the above feature point pairs, In addition, it is necessary to further calculate the similarity probability of each first feature point and second feature point that do not have a corresponding relationship to date, that is, the more general first feature point x m With the second feature point y n The similarity probability p between them mn .

[0085] In this embodiment, for feature point pairs with a corresponding relationship, the similarity probability between the first feature point and the second feature point can be calculated using Formula 1 above. For first feature points and second feature points without a corresponding relationship, the similarity probability between the first feature point and the second feature point can be calculated using the Euclidean distance and the shape context distance between them. The calculation method is shown in Formula 3 below.

[0086] Formula 3:

[0087]

[0088] in, Represents the first feature point x m Second feature point y m The Euclidean distance between them This indicates that the first feature point x m Second feature point y m The shape context distance between them. The above σ 2 It can represent iteration parameters, which are used to control the iteration precision in subsequent steps.

[0089] Based on this, in order to suppress the potential impact of outliers, the similarity probability p between the first feature point and the second feature point can be further calculated using the following formula 4. mn.

[0090] Formula 4:

[0091]

[0092] Wherein, τ represents the outlier suppression parameter.

[0093] As an optional implementation, the electronic device can further determine whether the correspondence between the first feature point and the second feature point meets the similarity condition, so as to determine the accuracy and rationality of using the above calculation formulas based on the judgment result, and determine whether different calculation formulas need to be used for calculation.

[0094] Taking a target feature point pair consisting of a first target feature point and a second target feature point that have a corresponding relationship as an example, in some embodiments, if the correspondence between the first target feature point and the second target feature point meets the similarity condition, the similarity probability of the target feature point pair can be calculated using Formula 1 above based on the fusion structure information corresponding to the target feature point pair. In other embodiments, if the correspondence between the first target feature point and the second target feature point does not meet the similarity condition, the similarity probability of the target feature point pair can be calculated using Formula 3 (or Formula 4 above) based on the Euclidean distance and shape context distance between the first target feature point and the second target feature point.

[0095] The aforementioned similarity condition can be determined based on the first neighbor subset corresponding to the first target feature point and the second neighbor subset corresponding to the second target feature point, thereby characterizing the reliability of the correspondence between the first target feature point and the second target feature point. For example, if the number of corresponding neighbor feature point pairs in the first neighbor subset and the second neighbor subset exceeds a similarity threshold, it can be confirmed that the correspondence between the first target feature point and the second target feature point meets the similarity condition; if it does not exceed the similarity threshold, it can be confirmed that the correspondence between the first target feature point and the second target feature point does not meet the similarity condition.

[0096] For a specific example, the first neighbor subset corresponding to the first target feature point and the second neighbor subset corresponding to the second target feature point can each contain N (e.g., N=5) feature points, and there are M (M≤N) feature point pairs with corresponding relationships. If the number M of feature point pairs with corresponding relationships exceeds the similarity threshold Q (e.g., Q=3), then the correspondence between the first target feature point and the second target feature point can be considered to meet the similarity condition; if M does not exceed the similarity threshold Q, then the correspondence between the first target feature point and the second target feature point can be considered to not meet the similarity condition. It can be understood that when M<N, the electronic device can update the first neighbor subset and the second neighbor subset respectively to retain the feature point pairs with corresponding relationships, but when judging whether the above correspondence meets the similarity condition, the first neighbor subset and the second neighbor subset before the update can be used for judgment.

[0097] 412. Based on the above correspondence matrix, calculate the spatial transformation parameters, and register the first image and the second image according to the spatial transformation parameters.

[0098] In this embodiment of the application, after determining the correspondence matrix between the first feature point set and the second feature point set, the electronic device can further calculate the corresponding spatial transformation parameters based on the correspondence matrix. The spatial transformation parameters can be used to register the first image and the second image.

[0099] For example, the spatial transformation parameters mentioned above may include mapping parameters, such as mapping vectors, mapping matrices, affine transformation parameters, etc. In some embodiments, taking affine transformation parameters as an example, the electronic device may first determine the registration cost function based on the above correspondence matrix, and then calculate the affine transformation parameters between the first image and the second image based on the registration cost function, as the spatial transformation parameters for registering the first image and the second image.

[0100] Specifically, the above correspondence matrix is ​​an M-row, M-column matrix P. MM For example, its corresponding registration cost function can be shown in Formula 5 below.

[0101] Formula 5:

[0102]

[0103] Where, p mn The first feature point x m With the second feature point y n The similarity probability between them, where A is the transformation matrix representing the rotation transformation and t is the transformation vector representing the translation transformation. The transformation matrix A and the transformation vector t together constitute the affine transformation parameters.

[0104] By taking the partial derivatives of the transformation matrix A and the transformation vector t in Formula 5 above, and setting them equal to 0, we can obtain the following:

[0105] Formula 6:

[0106]

[0107] Formula 7:

[0108] t = u x -Au y

[0109] In Formula 6 above, "1" represents a unit vector, P is the correspondence matrix, X is the first feature point set, and Y is the second feature point set.

[0110] Therefore, by combining Equations 6 and 7 above, we can obtain the affine transformation of the second image based on the second feature point set, as shown in Equation 8 below.

[0111] Formula 8:

[0112]

[0113] Using the transformation matrix A and transformation vector t described above, the second feature point set Y can be updated to obtain... This means that an affine transformation is performed on the second image, thereby enabling registration of the first and second images.

[0114] As can be seen, by implementing the image registration method described in the above embodiments, the electronic device can calculate the spatial transformation parameters for registering the first and second images by combining the spatial features of their neighboring feature points with corresponding relationships detected in the first and second images. By employing a neighbor subset, more spatial features in the images to be registered can be utilized to more accurately represent the spatial transformation relationship between the first and second images, resulting in higher reliability and robustness of the calculated spatial transformation parameters, thereby improving the accuracy of image registration. Furthermore, by fusing multiple distance information, a more accurate similarity metric can be constructed, giving the fused feature descriptor stronger descriptive power and facilitating accurate and effective determination of the correspondence between the first and second feature points and its corresponding accuracy.

[0115] Please see Figure 5 , Figure 5 This is a schematic flowchart of another image registration method disclosed in an embodiment of this application. This method can be applied to electronic devices including the above-described image processing circuit. Figure 5 As shown, the image registration method may include the following steps:

[0116] 502. Perform feature point detection on the first image and the second image respectively to obtain the first feature point set corresponding to the first image and the second feature point set corresponding to the second image, wherein each first feature point contained in the first feature point set corresponds one-to-one with each second feature point contained in the second feature point set.

[0117] Step 502 is similar to step 202 above, and will not be described again here.

[0118] 504. From the first feature point set, select the first N first feature points arranged in ascending order of distance from the first target feature point to form the first neighbor subset corresponding to the first target feature point; and from the second feature point set, select the second N second feature points arranged in ascending order of distance from the second target feature point to form the second neighbor subset corresponding to the second target feature point, wherein the first target feature point is any first feature point in the first feature point set, the second target feature point is the second feature point in the second feature point set corresponding to the aforementioned first target feature point, and N is a positive integer.

[0119] Step 504 is similar to step 404 described above. It should be noted that step 404 is described from a holistic perspective, considering all feature points contained in the first and second feature point sets as a whole; while step 504 is described from an individual perspective, focusing on the target feature point pairs composed of corresponding first and second target feature points. This difference in description does not affect the determination of the nearest neighbor subsets corresponding to each first and second feature point in this embodiment, allowing subsequent steps to calculate spatial transformation parameters based on these nearest neighbor subsets and register the first and second images according to these spatial transformation parameters.

[0120] 506. Remove feature points that do not have a corresponding relationship from each of the above first neighbor subsets and second neighbor subsets to update the first neighbor subsets and second neighbor subsets.

[0121] The main implementation method of step 506 has been described in step 404 above, and will not be repeated here.

[0122] 508. Based on the first neighbor subset corresponding to the first target feature point and the second neighbor subset corresponding to the second target feature point, calculate the fused shape context distance corresponding to the target feature point pair, calculate the fused grayscale descriptor distance corresponding to the target feature point pair, and calculate the fused Euclidean distance corresponding to the target feature point pair, wherein the above target feature point pair is composed of the first target feature point and the second target feature point.

[0123] In this embodiment, based on the first neighbor subset corresponding to the first target feature point and the second neighbor subset corresponding to the second target feature point, the fusion structure information corresponding to the target feature point pair can be obtained, so as to accurately determine the spatial transformation relationship between the first feature point set and the second feature point set based on the fusion structure information. The aforementioned fusion structure information may include a fusion distance, which can be obtained by further fusing fusion shape context distance, fusion grayscale descriptor distance, and fusion Euclidean distance. The aforementioned fusion shape context distance, fusion grayscale descriptor distance, and fusion Euclidean distance can all be calculated based on the first neighbor subset corresponding to the first target feature point and the second neighbor subset corresponding to the second target feature point.

[0124] In some embodiments, when calculating the fused shape context distance corresponding to the target feature point pair, the electronic device can obtain the first shape context descriptor of each first feature point contained in the first neighbor subset corresponding to the first target feature point. Simultaneously, for the second neighbor subset corresponding to the second target feature point, the electronic device can obtain the second shape context descriptor of each second feature point contained in the second neighbor subset. Based on this, the electronic device can calculate the target shape context distance between target feature point pairs and the neighboring shape context distance between corresponding neighboring feature point pairs in the first and second neighbor subsets, and calculate the fused shape context distance corresponding to the target feature point pair based on the target shape context distance and the neighboring shape context distance. For example, the above calculation of the fused shape context distance... The process can be shown in Formula 9 below.

[0125] Formula 9:

[0126]

[0127] in, For the target feature point pair x m and y m Target shape context distance between It is the sum of the neighboring shape context distances between each pair of neighboring feature points that have a corresponding relationship in the first neighboring subset and the second neighboring subset, where N represents the number of the aforementioned pairs of neighboring feature points that have a corresponding relationship, and the control parameter ρ1≤1.

[0128] In some embodiments, when calculating the fused grayscale descriptor distance corresponding to the aforementioned target feature point pair, the electronic device can determine the target grayscale descriptor distance between the target feature point pair based on the pairing relationship between the first target feature point and the second target feature point. Furthermore, based on the correspondence between each first feature point included in the first neighbor subset and each second feature point included in the second neighbor subset, the neighboring grayscale descriptor distance between neighboring feature point pairs with corresponding relationships can be determined. Based on this, the electronic device can calculate the fused grayscale descriptor distance corresponding to the target feature point pair based on the aforementioned target grayscale descriptor distance and neighboring grayscale descriptor distance. For example, the above calculation of the fused grayscale descriptor distance... The process can be shown in Formula 10 below.

[0129] Formula 10:

[0130]

[0131] in, For the target feature point pair x m and y m Distance between target grayscale descriptors This is the sum of the distances between neighboring grayscale descriptors of each corresponding pair of neighboring feature points in the first and second neighboring subsets, where N represents the number of such corresponding pairs of neighboring feature points, and the control parameter ρ2 ≤ 1. It should be noted that since grayscale descriptors can be used to characterize the correspondence between each first feature point and each second feature point in the first and second feature point sets, at the first feature point x... m Second feature point y m When a corresponding relationship exists, it can be taken Otherwise, you can take To simplify the calculation process and improve calculation efficiency.

[0132] In some embodiments, when calculating the fused Euclidean distance corresponding to the aforementioned target feature point pairs, the electronic device may first calculate the target Euclidean distance between the target feature point pairs, and the neighboring Euclidean distance between corresponding neighboring feature point pairs in the first and second neighboring subsets. Based on this, the electronic device may further calculate the fused Euclidean distance corresponding to the target feature point pairs according to the aforementioned target Euclidean distance and neighboring Euclidean distance. For example, the above calculation of the fused Euclidean distance... The process can be shown in Formula 11 below.

[0133] Formula 11:

[0134]

[0135] in, For the target feature point pair x m and y m The target Euclidean distance between them It is the sum of the neighboring Euclidean distances between each pair of neighboring feature points that have a corresponding relationship in the first and second neighboring subsets, where N represents the number of such pairs of neighboring feature points that have a corresponding relationship, and the control parameter ρ3≤1.

[0136] Optionally, the calculation of the above-mentioned distance information can be accelerated through parallelization, thereby further improving the efficiency of image registration by electronic devices.

[0137] 510. Fuse the fused shape context distance, fused grayscale descriptor distance, and fused Euclidean distance to calculate the fused distance corresponding to the target feature point pair.

[0138] After calculating the fused shape context distance, fused grayscale descriptor distance, and fused Euclidean distance respectively, the electronic device can fuse the above three based on the following formula 12:

[0139] Formula 12:

[0140]

[0141] Where k can be used to control the distance of the fused grayscale descriptor At the fusion distance d m The role of k in the equation. For example, k can be taken as 1, 2, 3, ... etc.

[0142] By fusing multiple distance information, a more accurate similarity metric can be constructed, making the fused feature descriptor more descriptive and facilitating the accurate and effective determination of the correspondence between the first feature point and the second feature point and its corresponding accuracy.

[0143] 512. Calculate the similarity probability of each feature point pair based on the fusion structure information corresponding to each feature point pair.

[0144] 514. Based on the similarity probabilities of each feature point pair, determine the correspondence matrix between the first feature point set and the second feature point set.

[0145] 516. Based on the above correspondence matrix, calculate the spatial transformation parameters, and register the first image and the second image according to the spatial transformation parameters.

[0146] Steps 512, 514, and 516 are similar to steps 408, 410, and 412 above, and will not be described again here.

[0147] In some embodiments, after calculating the spatial transformation parameters, the electronic device may perform multiple iterative calculations to obtain more accurate spatial transformation parameters. For example, the electronic device may adjust the second feature point set based on its calculated spatial transformation parameters (i.e., update as shown in Formula 8 above), and re-execute the steps of determining the second neighbor subset corresponding to each second feature point from the second feature point set, and calculating the spatial transformation parameters based on the first neighbor subset corresponding to each first feature point and the second neighbor subset corresponding to each second feature point, until the updated spatial transformation parameters satisfy the iteration termination condition. For example, the iteration termination condition may include an iteration count condition and / or an iteration parameter condition.

[0148] The aforementioned iteration count condition can include a specified number of iterations, such as 100, 150, or 200. By performing a finite number of iterations, a balance can be achieved between the computing power of the electronic device and the accuracy of its calculated spatial transformation parameters. This ensures that the calculated spatial transformation parameters achieve a certain level of accuracy while minimizing the computation time of the electronic device, thereby reducing power consumption.

[0149] The above iterative parameter conditions may include a specified iterative parameter precision, which can be expressed by σ in Formula 1 and Formula 3. 2 This indicates that, by performing the above iterative calculation process based on the deterministic annealing algorithm, the iterative parameter σ obtained in each iteration can be optimized. 2 Gradually decrease, when the iterative parameter σ 2 When the specified iteration parameter precision is achieved (e.g., specifying...) When the iteration parameter When the iteration parameter conditions are met, the iteration can be stopped, thus ensuring that the spatial transformation parameters calculated by the electronic device achieve the required accuracy, thereby improving the accuracy and reliability of image registration by the electronic device.

[0150] As can be seen, by implementing the image registration method described in the above embodiments, the electronic device can calculate the spatial transformation parameters for registering the first and second images by combining the spatial features of the corresponding feature points detected in the first and second images with the spatial features of their neighboring feature points. By using a neighbor subset, more spatial features in the images to be registered can be utilized to more accurately represent the spatial transformation relationship between the first and second images, making the calculated spatial transformation parameters more reliable and robust, thereby improving the accuracy of image registration. In addition, by fusing multiple distance information, a more accurate similarity metric can be constructed, making the fused feature descriptor more descriptive, facilitating accurate and effective determination of the correspondence between the first and second feature points and their corresponding accuracy. Furthermore, by parallel computing of the above-mentioned fused information, parallelization acceleration can be effectively achieved, thereby further improving the efficiency of image registration by the electronic device. Moreover, by obtaining the spatial transformation parameters through multiple iterations, the obtained spatial transformation parameters can be made as accurate and reliable as possible, which is conducive to further improving the accuracy and reliability of image registration.

[0151] Please see Figure 6 , Figure 6 This is a modular schematic diagram of an image registration device disclosed in an embodiment of this application. This image registration device can be applied to electronic devices including the aforementioned image processing circuit. Figure 6 As shown, the image registration device may include a feature point detection unit 601, a determination unit 602, and a calculation unit 603, wherein:

[0152] The feature point detection unit 601 is used to perform feature point detection on the first image and the second image respectively, and obtain a first feature point set corresponding to the first image and a second feature point set corresponding to the second image, wherein each first feature point contained in the first feature point set corresponds one-to-one with each second feature point contained in the second feature point set.

[0153] The determining unit 602 is used to determine a first neighbor subset corresponding to each first feature point from the first feature point set, and to determine a second neighbor subset corresponding to each second feature point from the second feature point set.

[0154] The calculation unit 603 is used to calculate spatial transformation parameters based on the first neighbor subsets corresponding to each first feature point and the second neighbor subsets corresponding to each second feature point, and to register the first image and the second image based on the spatial transformation parameters.

[0155] As can be seen, using the image registration apparatus described in the above embodiments, the electronic device can determine its neighbor subsets in the corresponding first feature point set and second feature point set for each corresponding feature point detected in the first and second images. Thus, for each pair of feature points, the spatial transformation parameters used for registering the first and second images can be jointly calculated by combining the spatial features of its neighboring feature points with the first and second feature points in each pair. By using neighbor subsets, more spatial features in the images to be registered can be utilized to more accurately represent the spatial transformation relationship between the first and second images, resulting in higher reliability and robustness of the calculated spatial transformation parameters, thereby improving the accuracy of image registration.

[0156] In one embodiment, the determining unit 602 can be specifically used to select the first N first feature points from the first feature point set in ascending order of distance from the first target feature point to form a first neighboring subset corresponding to the first target feature point; and to select the second N second feature points from the second feature point set in ascending order of distance from the second target feature point to form a second neighboring subset corresponding to the second target feature point.

[0157] Wherein, the first target feature point is any first feature point in the first feature point set, and the second target feature point is the second feature point in the second feature point set that corresponds to the first target feature point; N is a positive integer.

[0158] In one embodiment, the determining unit 602 can also be used to remove feature points that do not have a corresponding relationship from the first neighbor subset and the second neighbor subset, so as to update the first neighbor subset and the second neighbor subset.

[0159] By using the image registration apparatus described in the above embodiments, the first and second neighbor subsets can be updated to avoid a situation where a first feature point belongs to the first neighbor subset, but its corresponding second feature point does not belong to the corresponding second neighbor subset (and vice versa), which would cause a deviation when calculating the spatial transformation parameters based on the first and second neighbor subsets. This ensures the reliability of image registration as much as possible.

[0160] In one embodiment, the computation unit 603 described above may include a fusion subunit (not shown), a probability calculation subunit, a matrix determination subunit, and a parameter calculation subunit, wherein:

[0161] The fusion subunit is used to obtain the fusion structure information corresponding to each feature point pair according to the first neighbor subset corresponding to each first feature point and the second neighbor subset corresponding to each second feature point, wherein each feature point pair consists of a first feature point and a corresponding second feature point.

[0162] The probability calculation subunit is used to calculate the similarity probability of each feature point pair in turn based on the fusion structure information corresponding to each feature point pair.

[0163] The matrix determines the sub-unit, which is used to determine the correspondence matrix between the first feature point set and the second feature point set based on the similarity probability of each feature point pair;

[0164] The parameter calculation subunit is used to calculate the spatial transformation parameters based on the correspondence matrix.

[0165] In one embodiment, the above-mentioned fusion structure information may include a fusion distance. The above-mentioned fusion sub-unit may be used to calculate the fusion shape context distance corresponding to the target feature point pair, the fusion grayscale descriptor distance corresponding to the target feature point pair, and the fusion Euclidean distance corresponding to the target feature point pair, based on the first neighbor subset corresponding to the first target feature point and the second neighbor subset corresponding to the second target feature point. The target feature point pair is composed of the first target feature point and the second target feature point.

[0166] The above-mentioned fused shape context distance, fused grayscale descriptor distance, and fused Euclidean distance are fused together to calculate the fused distance corresponding to the target feature point pair.

[0167] In some embodiments, the fusion subunit may specifically include the following when calculating the fusion shape context distance corresponding to the target feature point pair:

[0168] For each first target feature point in the first neighbor subset, obtain the first shape context descriptor for each first feature point contained in the first neighbor subset; and

[0169] For the second neighbor subset corresponding to the second target feature point, obtain the second shape context descriptor for each second feature point contained in the second neighbor subset;

[0170] Based on the first shape context descriptor and the second shape context descriptor, calculate the target shape context distance between target feature point pairs, and the neighboring shape context distance between corresponding neighboring feature point pairs in the first and second neighboring subsets;

[0171] Calculate the fused shape context distance for each target feature point pair based on the target shape context distance and the neighboring shape context distances.

[0172] In some embodiments, when calculating the fusion subunit corresponding to the target feature point pair and the fused grayscale descriptor distance, the above-mentioned fusion subunit may specifically include:

[0173] Based on the pairing relationship between the first target feature point and the second target feature point, determine the target grayscale descriptor distance between the target feature point pairs;

[0174] Based on the correspondence between each first feature point contained in the first neighboring subset and each second feature point contained in the second neighboring subset, determine the neighboring grayscale descriptor distance between neighboring feature point pairs that have a correspondence.

[0175] Calculate the fused grayscale descriptor distance for each target feature point pair based on the target grayscale descriptor distance and the distances of its neighboring grayscale descriptors.

[0176] In some embodiments, the fusion subunit, when calculating the fused Euclidean distance corresponding to the target feature point pair, may specifically include:

[0177] Calculate the target Euclidean distance between target feature point pairs, and the neighbor Euclidean distance between corresponding neighbor feature point pairs in the first and second neighbor subsets;

[0178] Calculate the fused Euclidean distance between the target feature point pairs based on the target Euclidean distance and the neighboring Euclidean distances.

[0179] The image registration device described in the above embodiments can construct a more accurate similarity measurement standard by fusing multiple distance information, so that the fused feature descriptor has a stronger descriptive ability, making it easier to accurately and effectively determine the correspondence between the first feature point and the second feature point and its corresponding accuracy.

[0180] In one embodiment, the above-mentioned probability calculation subunit can be specifically used to calculate the similarity probability of the target feature point pair based on the fusion structure information corresponding to the target feature point pair when the correspondence between the first target feature point and the second target feature point meets the similarity condition. The target feature point pair consists of a first target feature point and a second target feature point. The first target feature point is any first feature point in the first feature point set, and the second target feature point is the second feature point in the second feature point set that corresponds to the first target feature point.

[0181] In some embodiments, the probability calculation subunit can also be used to calculate the similarity probability of the target feature point pair based on the Euclidean distance and the shape context distance between the first target feature point and the second target feature point when the correspondence between the first target feature point and the second target feature point does not meet the similarity condition.

[0182] In one embodiment, the calculation unit 603 may further include a correspondence determination unit (not shown), which may be used to:

[0183] If the number of corresponding neighbor feature point pairs in the first neighbor subset corresponding to the first target feature point and the second neighbor subset corresponding to the second target feature point exceeds the similarity threshold, then the correspondence between the first target feature point and the second target feature point is confirmed to meet the similarity condition.

[0184] If the similarity threshold is not exceeded, it is confirmed that the correspondence between the first target feature point and the second target feature point does not meet the similarity condition.

[0185] In one embodiment, the aforementioned matrix determining sub-unit can be specifically used to generate a correspondence matrix between the first feature point set and the second feature point set based on the similarity probability of each feature point pair and the similarity probability between each pair of first and second feature points that do not have a corresponding relationship in the first feature point set and the second feature point set.

[0186] In one embodiment, after calculating the spatial transformation parameters based on the first neighbor subsets corresponding to each first feature point and the second neighbor subsets corresponding to each second feature point, the calculation unit 603 can further be used to:

[0187] The second feature point set is adjusted based on the spatial transformation parameters, and the steps of determining the second neighbor subset corresponding to each second feature point from the second feature point set, and calculating the spatial transformation parameters based on the first neighbor subset corresponding to each first feature point and the second neighbor subset corresponding to each second feature point are re-executed until the updated spatial transformation parameters satisfy the iteration termination condition, wherein the iteration termination condition includes the iteration number condition and / or the iteration parameter condition.

[0188] As can be seen, using the image registration apparatus described in the above embodiments, the electronic device can calculate the spatial transformation parameters for registering the first and second images by combining the spatial features of the corresponding feature points detected in the first and second images with the spatial features of their neighboring feature points. By using a neighbor subset, more spatial features in the images to be registered can be utilized to more accurately represent the spatial transformation relationship between the first and second images, making the calculated spatial transformation parameters more reliable and robust, thereby improving the accuracy of image registration. In addition, by parallel computing of the above-mentioned fusion information, parallelization acceleration can be effectively achieved, thereby further improving the efficiency of image registration by the electronic device. Furthermore, obtaining the spatial transformation parameters through multiple iterative calculations can make the obtained spatial transformation parameters as accurate and reliable as possible, which is conducive to further improving the accuracy and reliability of image registration.

[0189] Please see Figure 7 , Figure 7 This is a modular schematic diagram of an electronic device disclosed in an embodiment of this application. For example... Figure 7 As shown, the electronic device may include:

[0190] Memory 701 storing executable program code;

[0191] Processor 702 coupled to memory 701;

[0192] The processor 702 can call the executable program code stored in the memory 701 to execute all or part of the steps in any of the image registration methods described in the above embodiments.

[0193] Furthermore, embodiments of this application disclose a computer-readable storage medium storing a computer program for electronic data interchange, wherein the computer program enables a computer to perform all or part of the steps in any of the image registration methods described in the above embodiments.

[0194] Furthermore, this application further discloses a computer program product that, when run on a computer, enables the computer to execute all or part of the steps in any of the image registration methods described in the above embodiments.

[0195] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0196] The foregoing has provided a detailed description of an image registration method, apparatus, electronic device, and storage medium disclosed in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image registration method characterized by, The method comprises the following steps: respectively performing feature point detection on the first image and the second image to obtain a first feature point set corresponding to the first image and a second feature point set corresponding to the second image, wherein each first feature point in the first feature point set corresponds to a second feature point in the second feature point set one by one; determining a first adjacent subset corresponding to each first feature point from the first feature point set and a second adjacent subset corresponding to each second feature point from the second feature point set; obtaining fusion structure information corresponding to each feature point pair according to the first adjacent subset corresponding to each first feature point and the second adjacent subset corresponding to each second feature point, wherein each feature point pair is composed of a first feature point and a corresponding second feature point; the fusion structure information comprises a fusion distance, and the fusion distance is obtained according to a fusion shape context distance, a fusion gray descriptor distance and a fusion Euclidean distance; determining a spatial transformation parameter between the first feature point set and the second feature point set based on the fusion structure information corresponding to each feature point pair; registering the first image and the second image according to the spatial transformation parameter.

2. The method of claim 1, wherein, The method further comprises the following steps after determining the first adjacent subset corresponding to each first feature point from the first feature point set and the second adjacent subset corresponding to each second feature point from the second feature point set: selecting, from the first feature point set, first feature points arranged in the first N positions in a sequence from small to large according to distances between the first feature points and a first target feature point to form the first adjacent subset corresponding to the first target feature point; and selecting, from the second feature point set, second feature points arranged in the first N positions in a sequence from small to large according to distances between the second feature points and a second target feature point to form the second adjacent subset corresponding to the second target feature point; wherein the first target feature point is any first feature point in the first feature point set, the second target feature point is a second feature point in the second feature point set corresponding to the first target feature point, and N is a positive integer.

3. The method of claim 2, wherein, The method further comprises the following steps after determining the first adjacent subset corresponding to each first feature point from the first feature point set and the second adjacent subset corresponding to each second feature point from the second feature point set: eliminating feature points without a corresponding relationship from the first adjacent subset and the second adjacent subset to update the first adjacent subset and the second adjacent subset.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises the following steps after determining the first adjacent subset corresponding to each first feature point from the first feature point set and the second adjacent subset corresponding to each second feature point from the second feature point set: calculating a similarity probability of each feature point pair according to the fusion structure information corresponding to each feature point pair; determining a corresponding relationship matrix between the first feature point set and the second feature point set according to the similarity probability of each feature point pair; calculating the spatial transformation parameter according to the corresponding relationship matrix.

5. The method of claim 4, wherein, The fusion structure information corresponding to each feature point pair is obtained according to a first adjacent subset corresponding to each first feature point and a second adjacent subset corresponding to each second feature point, and the fusion structure information comprises: The fusion shape context distance corresponding to the target feature point pair is calculated according to the first adjacent subset corresponding to the first target feature point and the second adjacent subset corresponding to the second target feature point, the target feature point pair is composed of the first target feature point and the second target feature point, the first target feature point is any first feature point in the first feature point set, and the second target feature point is a second feature point corresponding to the first target feature point in the second feature point set.

6. The method of claim 5, wherein, The fusion shape context distance corresponding to the target feature point pair is calculated according to the first adjacent subset corresponding to the first target feature point and the second adjacent subset corresponding to the second target feature point. The first shape context descriptor of each first feature point included in the first adjacent subset corresponding to the first target feature point is obtained. The second shape context descriptor of each second feature point included in the second adjacent subset corresponding to the second target feature point is obtained. The target shape context distance between the target feature point pair and the adjacent shape context distance between the adjacent feature point pairs having a corresponding relationship in the first adjacent subset and the second adjacent subset are calculated according to the first shape context descriptor and the second shape context descriptor. The fusion shape context distance corresponding to the target feature point pair is calculated according to the target shape context distance and the adjacent shape context distance.

7. The method of claim 5, wherein, The fusion gray descriptor distance corresponding to the target feature point pair is calculated, and the calculation comprises: The target gray descriptor distance between the target feature point pair is determined according to a matching relationship between the first target feature point and the second target feature point. The adjacent gray descriptor distance between the adjacent feature point pairs having a corresponding relationship is determined according to a corresponding relationship between each first feature point included in the first adjacent subset and each second feature point included in the second adjacent subset. The fusion gray descriptor distance corresponding to the target feature point pair is calculated according to the target gray descriptor distance and the adjacent gray descriptor distance.

8. The method of claim 5, wherein, The fusion Euclidean distance corresponding to the target feature point pair is calculated, and the calculation comprises: The target Euclidean distance between the target feature point pair and the adjacent Euclidean distance between the adjacent feature point pairs having a corresponding relationship in the first adjacent subset and the second adjacent subset are calculated. The fusion Euclidean distance corresponding to the target feature point pair is calculated according to the target Euclidean distance and the adjacent Euclidean distance.

9. The method of claim 4, wherein, The similarity probability of each feature point pair is calculated according to the fusion structure information corresponding to each feature point pair in sequence, and the calculation comprises: In a case where the correspondence between the first target feature point and the second target feature point meets the similarity condition, a similarity probability of a target feature point pair is calculated according to the fused structure information corresponding to the target feature point pair, wherein the target feature point pair is composed of the first target feature point and the second target feature point, the first target feature point is any first feature point in the first feature point set, and the second target feature point is a second feature point in the second feature point set corresponding to the first target feature point; The method further comprises: In a case where the correspondence between the first target feature point and the second target feature point does not meet the similarity condition, a similarity probability of the target feature point pair is calculated according to the Euclidean distance between the first target feature point and the second target feature point and the shape context distance.

10. The method of claim 9, wherein, Before the similarity probability of the target feature point pair is calculated according to the fused structure information corresponding to the target feature point pair in the case where the correspondence between the first target feature point and the second target feature point meets the similarity condition, the method further comprises: If the number of the adjacent feature point pairs with the correspondence in the first adjacent subset corresponding to the first target feature point and the second adjacent subset corresponding to the second target feature point exceeds a similarity threshold, it is determined that the correspondence between the first target feature point and the second target feature point meets the similarity condition; if the number does not exceed the similarity threshold, it is determined that the correspondence between the first target feature point and the second target feature point does not meet the similarity condition.

11. The method of claim 4, wherein, The determination of the correspondence matrix between the first feature point set and the second feature point set according to the similarity probability of each feature point pair comprises: The correspondence matrix between the first feature point set and the second feature point set is generated according to the similarity probability of each feature point pair and the similarity probability between the first feature point and the second feature point in each pair without the correspondence in the first feature point set and the second feature point set.

12. The method according to any one of claims 1 to 3, characterized in that, After the spatial transformation parameter is calculated according to the first adjacent subset corresponding to each first feature point and the second adjacent subset corresponding to each second feature point, the method further comprises: The second feature point set is adjusted based on the spatial transformation parameter, and the steps of determining the second adjacent subset corresponding to each second feature point from the second feature point set and calculating the spatial transformation parameter according to the first adjacent subset corresponding to each first feature point and the second adjacent subset corresponding to each second feature point are re-executed until the updated spatial transformation parameter meets an iteration end condition, wherein the iteration end condition comprises an iteration number condition and / or an iteration parameter condition.

13. An image registration apparatus characterized by comprising: The method comprises: A feature point detection unit is configured to perform feature point detection on a first image and a second image respectively, to obtain a first feature point set corresponding to the first image and a second feature point set corresponding to the second image, wherein each first feature point in the first feature point set corresponds to each second feature point in the second feature point set one by one; The determining unit is configured to determine a first neighboring subset corresponding to each of the first feature points from the first feature point set, and determine a second neighboring subset corresponding to each of the second feature points from the second feature point set; The calculating unit comprises a fusion subunit; The fusion subunit is configured to acquire fusion structure information corresponding to each feature point pair according to the first neighboring subset corresponding to each of the first feature points and the second neighboring subset corresponding to each of the second feature points, wherein each feature point pair is composed of a first feature point and a corresponding second feature point; the fusion structure information comprises a fusion distance, and the fusion distance is obtained according to a fusion shape context distance, a fusion gray descriptor distance and a fusion Euclidean distance; The calculating unit is configured to determine a spatial transformation parameter between the first feature point set and the second feature point set based on the fusion structure information corresponding to each feature point pair. The calculating unit is further configured to register the first image and the second image according to the spatial transformation parameter.

14. An electronic device, comprising: The computer program is executed by the processor to implement the method of any one of claims 1 to 12.

15. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 12.

Images