Image registration method, image registration device and computer readable storage medium

By using a pre-trained feature extraction network and feature pyramid two-stage architecture, the problem of large amount of computation and incorrect matching in image registration is solved, efficient and accurate image registration is achieved, and the training process is simplified.

CN114494367BActive Publication Date: 2024-12-06ZHEJIANG DAHUA TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111547032.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-16
Publication Date
2024-12-06
Estimated Expiration
2041-12-16

AI Technical Summary

Technical Problem

The prior art has problems with large amount of computation and incorrect matching in the image registration process, especially when image registration with different perspectives or different phases, traditional methods are difficult to effectively improve the registration accuracy, and deep learning algorithms require massive data to train. Feature detection and feature description network.

Method used

The pre-trained feature extraction network is used to extract the reference image and the image to be registered. It is registered through the two-stage architecture of the feature pyramid. First, it is coarse registration, and then refines it layer by layer, using the semantic information of the deep feature map. and the precise positioning capability of shallow feature maps to achieve accurate image registration.

Benefits of technology

No additional data acquisition and model training are required, which significantly improves the accuracy and efficiency of image registration, simplifies the registration process, and enables efficient and accurate image registration in complex scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114494367B_ABST
    Figure CN114494367B_ABST
Patent Text Reader

Abstract

The present application discloses an image registration method, an image registration device and a computer-readable storage medium, the method comprising: using a pre-trained feature extraction network to extract and process the reference image and the image to be registered respectively, to obtain a first feature atlas and a second feature atlas; based on the first registration feature map and the second registration feature map, registering the image to be registered with the reference image to obtain a mapping image, the first registration feature map is a first feature map in the first feature map, and the second registration feature map is a second feature map in the second feature map; using a feature extraction network to extract and process the mapping image to obtain a third feature map; based on the first feature map and the third feature map, registering the reference image and the mapping image to obtain a matching result. Through the above method, the present application can improve the accuracy of registration without the need to train the network.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to an image registration method, an image registration device, and a computer-readable storage medium. Background Art

[0002] Image registration refers to the technology of matching images of the same scene with different perspectives or different phases. The registration accuracy directly affects the effect of subsequent image fusion, image stitching, change detection or 3D reconstruction. Due to factors such as scene differences, brightness changes, perspective changes or low overlap, there are problems such as excessive calculation and high mismatch when registering images at different times. Summary of the invention

[0003] The present application provides an image registration method, an image registration device and a computer-readable storage medium, which can improve the accuracy of registration without the need for network training.

[0004] In order to solve the above technical problems, the technical solution adopted in the present application is: to provide an image registration method, the method comprising: using a pre-trained feature extraction network to extract and process the reference image and the image to be registered respectively, to obtain a first feature atlas and a second feature atlas; based on the first registration feature map and the second registration feature map, registering the image to be registered with the reference image to obtain a mapping image, the first registration feature map is a first feature map in the first feature map, and the second registration feature map is a second feature map in the second feature map; using a feature extraction network to extract and process the mapping image to obtain a third feature atlas; based on the first feature map and the third feature map, registering the reference image and the mapping image to obtain a matching result.

[0005] To solve the above technical problems, another technical solution adopted in the present application is: to provide an image registration device, which includes a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the image registration method in the above technical solution.

[0006] To solve the above technical problem, another technical solution adopted in the present application is: providing a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it is used to implement the image registration method in the above technical solution.

[0007] Through the above scheme, the beneficial effect of the present application is: the reference image and the image to be registered are respectively input into the pre-trained feature extraction network to obtain the first feature atlas and the second feature atlas; then the first registration feature map in the first feature atlas and the second registration feature map in the second feature atlas are used to perform registration processing on the image to be registered and the reference image to obtain a mapping image; then the mapping image is input into the feature extraction network to obtain a third feature atlas; then the first feature atlas and the third feature atlas are used to perform registration processing on the reference image and the mapping image to obtain a matching result; since the existing feature extraction network is used, no additional training process is required, the implementation is simple, and the amount of calculation is small; moreover, the registration of the two images is achieved through a registration method based on a feature pyramid, which makes full use of the characteristics of feature maps at different layers, which helps to improve the accuracy of matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0009] Figure 1 It is a flowchart of an embodiment of an image registration method provided by the present application;

[0010] Figure 2 It is a schematic diagram of the structure of a multi-scale feature pyramid diagram provided by this application;

[0011] Figure 3 It is a schematic diagram of the structure of RepBlock provided by this application;

[0012] Figure 4 is a flowchart of another embodiment of the image registration method provided by the present application;

[0013] Figure 5 yes Figure 4 A schematic diagram of the process of S48 in the embodiment shown;

[0014] Figure 6 is a schematic diagram of the matching results provided by this application;

[0015] Figure 7 It is a structural schematic diagram of an embodiment of an image registration device provided by the present application;

[0016] Figure 8 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0017] The present application is further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present application, but are not intended to limit the scope of the present application. Similarly, the following examples are only some embodiments of the present application rather than all embodiments, and all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0018] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0019] It should be noted that the terms "first", "second" and "third" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second" and "third" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0020] In image registration technology, the image registration process is mostly composed of four modules in series: feature detection, feature description, feature matching, and geometric transformation. In this process, the output of the front module is the input of the back module. Each module will produce different errors in different scenes, resulting in the accumulation of errors and affecting the final registration accuracy. Traditional feature point detection and description algorithms, such as Harris or Scale-invariant feature transform (SIFT), often pay more attention to the underlying pixel-level information and ignore the high-level semantic information of the image itself. They have low robustness to geometric transformation and illumination, resulting in a large number of false matches in complex scenes. Deep learning algorithms have great advantages over traditional algorithms in obtaining high-level semantic features, but they require massive amounts of data to train feature detection and feature description networks. Therefore, how to obtain a large amount of data to train the registration model has become the main difficulty.

[0021] Based on the above problems, the problem to be solved in this application is how to efficiently and accurately align two images with different perspectives or different phases. The solution provided in this application uses an existing, trained feature extraction network for feature extraction, avoiding additional data acquisition and model training processes, and has higher accuracy than traditional alignment methods. The solution adopted in this application is described in detail below.

[0022] See also Figure 1 , Figure 1 : is a flow chart of an embodiment of an image registration method provided by the present application, the method comprising:

[0023] S11: using a pre-trained feature extraction network to extract and process the reference image and the image to be registered respectively, to obtain a first feature atlas and a second feature atlas.

[0024] First, a feature extraction network is obtained, which is a pre-trained network for realizing feature extraction function, and the feature extraction network can be obtained from a model library; then, a reference image is input into the feature extraction network to generate a first feature atlas, and the first feature atlas includes multiple layers of first feature maps; and, an image to be registered (i.e., a floating image) is input into the feature extraction network to generate a second feature atlas, and the second feature atlas can be a second feature map, or the second feature atlas includes at least two layers of second feature maps. Specifically, the number of layers of the first feature map and the second feature map can be set according to application needs, such as: 3 layers, 4 layers or 5 layers; the reference image and the image to be registered can be remote sensing images. The feature extraction network used in this embodiment is a common feature extraction network, which is used to calculate features of the input image layer by layer, such as: Visual Geometry Group (VGG) or Residual Neural Network (Resnet), etc. The first feature atlas and the second feature atlas are both feature pyramids, and a multi-scale feature pyramid can be formed by downsampling; generally speaking, the deeper the network, the richer the semantic information of the feature map, but the lower the resolution, the coarser the position information; the shallower the network, the less semantic information of the feature map, but the higher the resolution, the more accurate the position information.

[0025] Furthermore, this embodiment extracts features through a re-parameterized VGG (RepVGG, Re-parameterization VGG) network because the RepVGG network only has convolutional layers and rectified linear unit (Rectified linear unit, Relu) layers during inference, and the inference speed is faster. Figure 2As shown in , for the RepVGG network, it is divided into five layers (Layer1-Layer5), and the resolution of each feature layer is 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image respectively; each layer is composed of a number of re-parameterization blocks (Re-parameterization Block, RepBlock) {4, 6, 16, 1}, and each RepBlock includes a 3×3 convolution (Convolution, Conv) layer and a Relu layer, as shown in Figure 3 As shown, except for the convolutional layer of the first RepBlock of each layer, which has a step length of 2, the rest have a step length of 1; the output of the last RepBlock of each layer is taken to form a multi-scale feature pyramid {Layer1, Layer2, Layer3, Layer4, Layer5}, where Layer1 is the shallowest feature layer and Layer5 is the deepest feature layer. It can be understood that the specific implementation principle of RepVGG is similar to that in the related art and will not be repeated here.

[0026] S12: Based on the first registration feature map and the second registration feature map, the image to be registered and the reference image are registered to obtain a mapping image.

[0027] After obtaining the first feature atlas and the second feature atlas, a first registration feature map is selected from the first feature atlas, that is, the first registration feature map is a first feature map among the multiple layers of first feature maps, for example, the first registration feature map is the first feature map of the last feature layer in the first feature atlas; at the same time, a second registration feature map is selected from the second feature atlas, that is, the second registration feature map is a second feature map among the multiple layers of second feature maps, for example, the second registration feature map is the second feature map of the last feature layer in the second feature atlas; then, the first registration feature map and the second registration feature map are used to roughly align the image to be registered with the reference image, and the image to be registered is mapped through a roughly estimated geometric transformation relationship to obtain a mapped image.

[0028] It can be understood that the selection of the first registration feature map and the second registration feature map can be set according to specific application requirements and is not limited to the feature map of the last feature layer; for example: taking the feature extraction network including 3 feature layers as an example, the feature map output by the third feature layer can be used as the registration feature map, or the feature map output by the second feature layer can be used as the registration feature map.

[0029] S13: Using a feature extraction network to extract the mapped image to obtain a third feature atlas.

[0030] After the mapping image is acquired, the mapping image is input into a feature extraction network to generate a corresponding feature pyramid (ie, a third feature atlas), wherein the third feature atlas includes multiple layers of third feature maps.

[0031] S14: Based on the first feature atlas and the third feature atlas, the reference image and the mapping image are registered to obtain a matching result.

[0032] The reference image and the mapping image are finely registered using the first feature map in the first feature map set and the third feature map in the third feature map set to generate a matching result.

[0033] It can be understood that the first feature atlas can be the first feature atlas generated in step S11, or can be obtained by reusing a pre-trained feature extraction network to extract the reference image.

[0034] This embodiment proposes an image registration method based on a feature pyramid, which uses a two-stage architecture to perform registration between two images. In the first stage, an initial registration result is obtained through rough registration, and features are extracted from the two images through a pre-trained feature extraction network (such as RepVGG). Then, the two images are roughly registered on the last feature layer, and the images to be registered are mapped through a roughly estimated geometric transformation relationship to obtain a mapping image; in the second stage, the position of the registration point is refined layer by layer, and a feature pyramid of the reference image and the mapping image is first constructed through a pre-trained feature extraction network, and then the feature maps of the reference image and the mapping image are registered to obtain the position of the matching point pair; since the existing feature extraction network is used, no additional training process is required, so the implementation is simple and the calculation amount is small; moreover, the registration of the two images is achieved through the two-stage registration method based on the feature pyramid, which can make full use of the characteristics of the feature maps of different layers, which is helpful to improve the accuracy of matching.

[0035] See also Figure 4 , Figure 4 : is a flow chart of another embodiment of the image registration method provided by the present application, the method comprising:

[0036] S41: using a pre-trained feature extraction network to extract and process the reference image and the image to be registered respectively, to obtain a first feature atlas and a second feature atlas.

[0037] Among them, S41 is the same as S11 in the above embodiment and will not be described again here.

[0038] Given an image pair, after constructing the feature pyramid, the position of the matching pair can be initialized at the deepest level by using the dense nearest neighbor search algorithm (DNNS), and then refined layer by layer using the local dense nearest neighbor search algorithm (LDNNS). However, this single-stage method will lead to failed registration in the case of some severe geometric transformations. This is because the shallow features are less robust to geometric transformations, and even if the correct matches are generated in the deep layer, they will be lost in the process of layer-by-layer refinement. Specifically, DNNS and LDNNS are a type of K-Nearest Neighbors (KNN) algorithm, and their specific implementation principles are the same as those in the relevant technology, so they will not be repeated here.

[0039] Therefore, this embodiment uses a two-stage architecture method to achieve the registration between two images. First, in stage one, DNNS is performed on the lowest resolution feature map to roughly estimate the geometric transformation between the given image pair, so as to map the image to be registered to the reference image to obtain a mapped image. In stage two, DNNS is first performed on the lowest resolution feature map of the reference image and the mapped image to obtain the rough matching point position, and then LDNNS is performed on the higher resolution feature map to gradually refine the matching point position, which is described in detail below.

[0040] S42: Determine the first feature map with the lowest resolution among the multiple layers of first feature maps as the first registration feature map, and determine the second feature map with the lowest resolution among the multiple layers of second feature maps as the second registration feature map.

[0041] For the first feature atlas, the first feature map of the last feature layer in the first feature atlas (i.e., the first feature map with the lowest resolution among all the first feature maps) is used as the first registration feature map; for the second feature atlas, the second feature map of the last feature layer in the second feature atlas (i.e., the second feature map with the lowest resolution among all the second feature maps) is used as the second registration feature map. It can be understood that each first feature map in the first feature atlas has the same resolution as the corresponding second feature map in the second feature atlas, that is, the resolution of the first registration feature map is the same as that of the second registration feature map.

[0042] S43: Search and process the first registration feature map and the second registration feature map to obtain a first matching point pair.

[0043] After obtaining the first registration feature map and the second registration feature map, the two feature maps are searched for matching points, for example: the distance between each feature point in the first registration feature map and all feature points in the second registration feature map is calculated, and the two feature points with the largest distance and greater than the set threshold are taken as the first matching points, that is, the first matching point pair includes the feature points in the first registration feature map and the corresponding feature points in the second registration feature map.

[0044] In a specific embodiment, in the coarse registration of stage one, DNNS is used to match the first registration feature map of the reference image and the second registration feature map of the image to be registered; specifically, the feature map of the last layer in the multi-scale feature pyramid has the smallest scale and the strongest feature representation ability. The feature map with a smaller resolution can significantly reduce the algorithm complexity of DNNS, and the stronger feature representation ability makes it robust to geometric changes such as image rotation and scaling, as well as illumination changes. Therefore, the first feature map of the last feature layer in the first feature map set and the second feature map of the last feature layer in the second feature map set are used for matching.

[0045] Furthermore, DNNS uses mutual nearest neighbor search and threshold ratio to determine matching point pairs in two feature maps (including the first registration feature map and the second registration feature map), and the matching quality is measured by the Euclidean (i.e., L2) distance. The smaller the L2 distance, the higher the matching degree. For example, the feature maps of the reference image and the image to be registered B are denoted as Fa and Fb respectively. For point Pa in Fa, if the ratio of the first distance to the second distance is lower than a given threshold, then Pb is called a matching point of Pa, where the first distance is the distance between point Pa and the nearest point Pb in Fb, and the second distance is the distance between point Pa and the second nearest point in Fb; at the same time, if the nearest point Pb to Fa is Pa, the second nearest point Pb to Fa is Pc, and the ratio of the third distance (i.e., the distance between point Pb and point Pa) to the fourth distance (i.e., the distance between point Pb and point Pc) is lower than the threshold, then Pa and Pb are called a pair of matching points. By directly performing DNNs on the feature maps to determine the feature points, the process of training the feature detection network with additional data is simplified.

[0046] S44: Based on the first matching point pair, a homography matrix between the image to be registered and the reference image is calculated, and based on the homography matrix, the image to be registered is transformed to obtain a mapping image.

[0047] After obtaining the first matching point pair on the two images returned by the DNNS algorithm, the two images can be roughly aligned by selecting a suitable geometric transformation according to the actual situation; specifically, taking the image to be registered and the reference image as a remote sensing image as an example, since the imaging distance of the remote sensing image is very far, the imaging target area is approximately a plane relative to the imaging element, so the geometric relationship between the two images can be estimated through the homography matrix, and the image to be registered is mapped to the reference image to obtain a mapped image.

[0048] S45: Use a feature extraction network to extract the mapped image to obtain a third feature atlas.

[0049] S45 is the same as S13 in the above embodiment and will not be described again.

[0050] S46: Determine a third feature map with the lowest resolution among the multiple layers of third feature maps as a third registration feature map.

[0051] S47: Based on the first registration feature map and the third registration feature map, the image to be registered and the mapping image are registered to obtain a set of matching point pairs.

[0052] After the rough alignment, the second stage can be executed: hierarchical refinement. First, the feature pyramid of the reference image and the mapping image is constructed through the pre-trained feature extraction network, and then the last layer of feature maps of the reference image and the mapping image are aligned using DNNS to obtain the initial matching point positions (i.e., the set of matching point pairs).

[0053] S48: Refine the set of matching point pairs to obtain a matching result.

[0054] Based on the fourth registration feature map and the fifth registration feature map, the set of matching point pairs can be refined to obtain a matching result. The fourth registration feature map is the first feature map of all first feature maps except the first registration feature map, and the fifth registration feature map is the third feature map of all third feature maps except the third registration feature map.

[0055] In a specific embodiment, although the matching points obtained by the deep feature map are robust to geometric transformations and illumination changes, their spatial resolution is low and the position information is not accurate enough. Directly upsampling it to the original resolution cannot obtain the precise position. Although the shallow feature map has weak semantic information, it has high spatial resolution and can better focus on the local information of the image due to its small receptive field. Therefore, in order to fully utilize the advantages of different feature layers, this embodiment adopts a coarse-to-fine hierarchical search strategy, such as Figure 5 As shown, the specific steps include:

[0056] S51: Taking the second matching point pair as the current matching point pair.

[0057] The set of matching point pairs includes a plurality of second matching point pairs, and the second matching point pair includes a feature point in the first registration feature map and a corresponding feature point in the third registration feature map; each second matching point pair is denoted as the current matching point pair, and S52-S56 are executed.

[0058] S52: Determine the search ranges of the first feature map and the third feature map of the next processing layer for the current matching point pair.

[0059] The current matching point pair includes a first candidate feature point and a second candidate feature point that matches the first candidate feature point; specifically, the first candidate feature point in the current layer is upsampled to obtain a first upsampled point in the next processing layer; the second candidate feature point in the current layer is upsampled to obtain a second upsampled point in the next processing layer; then, based on the first upsampled point and the second upsampled point, the search ranges are obtained. It can be understood that the next processing layer can be the next layer of the current layer, that is, assuming the current layer is the 5th layer, then the next processing layer is the 4th layer; or there are a set number of layers between the next processing layer and the current layer, for example: the current layer is the 5th layer, and the next processing layer is the 3rd layer or the 2nd layer.

[0060] Furthermore, the search ranges include a first search range and a second search range. The region enclosed by the first upsampled point, the feature points on the right side of the first upsampled point, and the feature points below each first upsampled point is determined as the first search range; the region enclosed by the second upsampled point, the feature points on the right side of the second upsampled point, and the feature points below each second upsampled point is determined as the second search range. For example, if the position of the first upsampled point is [1, 1], after upsampling it, the position [2, 2] is obtained, the feature point at position [2, 3] is the feature point on its right side, and the feature points at [3, 2] and [3, 3] are the feature points below it, then the region enclosed by the feature points at positions [2, 2], [2, 3], [3, 2], and [3, 3] is the first search range.

[0061] S53: Match the feature points of the first feature map and the feature points of the third feature map of the next processing layer within the search ranges to obtain third matching point pairs.

[0062] For the current matching pair P of the given nth (1 < n ≤ 5) layer a n and P b n in terms of, first construct its search ranges Ω a n-1 and Ω b n-1 in the (n - 1)th layer; then, by matching the matching points P a n and Pb n Upsampling is performed to obtain the search range; then in the first feature map F of the (n-1) layer a n-1 And the third characteristic graph F b n-1 LDNNS is implemented in the corresponding area (i.e., the search range) to refine the position of the matching point pairs or eliminate erroneous matching points; by using LDNNS to search for matching point pairs in a local area, the amount of calculation can be further reduced.

[0063] S54: Determine whether a preset termination condition is met.

[0064] Determine whether the layer number of the next processing layer is a preset value, which may be 1, that is, determine whether the first feature map of the next processing layer is the first first feature map among all first feature maps. If so, determine that the preset termination condition is met; or, determine whether the third feature map of the next processing layer is the first third feature map among all third feature maps. If so, determine that the preset termination condition is met.

[0065] S55: If the preset termination condition is met, the third matching point pair is determined as a matching result.

[0066] If it is detected that the layer number of the next processing layer is equal to 1, it indicates that the preset termination condition is currently met, the loop ends, and the third matching point pair currently generated is used as the matching result between the image to be registered and the reference image.

[0067] S56: If the preset termination condition is not met, the third matching point pair is used as the current matching point pair.

[0068] If it is detected that the layer number of the next processing layer is not equal to 1, it indicates that the preset termination condition is not met at present. At this time, the third matching point pair can be used as the current matching point pair, and the step of determining the search range of the current matching point pair on the first feature map of the next processing layer and the third feature map of the next processing layer is returned, that is, S52 is executed until the preset termination condition is met.

[0069] The above scheme first aligns the last layer feature map of the reference image and the mapped image to initialize the position of the matching point pairs; then, through LDNNS, the positions of the matching point pairs are refined layer by layer from coarse to fine on the higher resolution feature maps of the reference image and the mapped image, and finally the positions of the matching point pairs at the original resolution are obtained, so that the positions of the matching point pairs with the same resolution as the original resolution and higher robustness are obtained through the layer-by-layer refinement method; and the search area is limited, and the position of the feature points is searched in the local area of ​​the shallow feature map. By using the improved LDNNS, the amount of calculation and the number of false matches can be greatly reduced.

[0070] In one implementation, the solution of this embodiment is used to match two remote sensing images, and the matching result is as follows: Figure 6 As shown, from Figure 6 It can be seen that the matching point pairs are evenly distributed in the image and have accurate positions, with high matching accuracy.

[0071] This embodiment combines feature detection and feature description together, directly uses the existing feature extraction network to extract features, does not require additional data to train the feature detection network, and determines the feature points by whether the DNNS is successfully matched, rather than obtaining the feature point set through feature point detection; by adopting a registration scheme that integrates feature detection and feature description, the general image registration process is simplified. At the same time, in terms of feature matching, this embodiment performs two-stage registration on the image based on the feature pyramid, performs coarse registration in stage one, obtains rough geometric transformation and preliminarily aligns the image through DNNS, and applies LDNNS from coarse to fine level by level in stage two to accurately locate the matching points; because the powerful semantic characteristics of the deep feature map and the precise positioning ability of the shallow feature map are combined, the characteristics of different feature layers are fully utilized, and it has strong robustness and high accuracy, and can achieve accurate registration of images in various complex scenes, and the matching based on local features greatly improves the efficiency of the algorithm.

[0072] See also Figure 7 , Figure 7 It is a structural diagram of an embodiment of an image registration device provided in the present application. The image registration device 70 includes a memory 71 and a processor 72 connected to each other. The memory 71 is used to store a computer program. When the computer program is executed by the processor 72, it is used to implement the image registration method in the above embodiment.

[0073] The image registration device in this embodiment adopts a two-stage registration method based on the feature pyramid, which combines the powerful semantic characteristics of the deep feature map and the precise positioning ability of the shallow feature map, fully utilizes the characteristics of different feature layers, and realizes accurate matching of the two images through a coarse-to-fine scheme; moreover, it combines feature detection and feature description together, and determines the feature points by whether the DNNs are successfully matched, which helps to reduce the registration error and simplify the registration process.

[0074] See also Figure 8 , Figure 8 It is a structural diagram of an embodiment of a computer-readable storage medium provided in the present application. The computer-readable storage medium 80 is used to store a computer program 81. When the computer program 81 is executed by a processor, it is used to implement the image registration method in the above embodiment.

[0075] The computer-readable storage medium 80 may be a server, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or other media that can store program codes.

[0076] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0077] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0078] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0079] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image registration method, It is characterized in that include: The pre-trained feature extraction network is used to extract and process the reference image and the image to be registered respectively to obtain a first feature atlas and a second feature atlas; Based on a first registration feature map and a second registration feature map, the image to be registered and the reference image are registered to obtain a mapping image, wherein the first registration feature map is a first feature map in the first feature map set, and the second registration feature map is a second feature map in the second feature map set; Using the feature extraction network to extract the mapping image to obtain a third feature atlas; Based on the first feature atlas and the third feature atlas, register the reference image with the mapping image to obtain a matching result; Wherein, the step of performing registration processing on the image to be registered and the reference image based on the first registration feature map and the second registration feature map to obtain a mapping image includes: Performing search processing on the first registration feature map and the second registration feature map to obtain a first matching point pair, where the first matching point pair includes a feature point in the first registration feature map and a corresponding feature point in the second registration feature map; Based on the first matching point pair, calculating a homography matrix between the image to be registered and the reference image; Based on the homography matrix, the image to be registered is transformed to obtain the mapping image.

2. The image registration method according to claim 1, It is characterized in that The first feature atlas includes multiple layers of first feature maps, the second feature atlas includes multiple layers of second feature maps, and the third feature atlas includes multiple layers of third feature maps. Before the step of performing registration processing on the image to be registered and the reference image based on the first registration feature map and the second registration feature map to obtain a mapping image, the method includes: Determine the first feature map with the lowest resolution among the multiple layers of first feature maps as the first registration feature map; The second feature map with the lowest resolution among the multiple layers of second feature maps is determined as the second registration feature map.

3. The image registration method according to claim 2, It is characterized in that The step of performing registration processing on the reference image and the mapping image based on the first feature atlas and the third feature atlas to obtain a matching result includes: Determine a third feature map with the lowest resolution among the multiple layers of third feature maps as a third registration feature map; Based on the first registration feature map and the third registration feature map, performing registration processing on the image to be registered and the mapping image to obtain a set of matching point pairs; The matching point pair set is refined to obtain the matching result.

4. The image registration method according to claim 3, It is characterized in that The matching point pair set includes a plurality of second matching point pairs, and the step of performing refinement processing on the matching point pair set to obtain the matching result includes: Using the second matching point pair as the current matching point pair of the current layer; Determine a search range of the current matching point pair on a first feature map of a next processing layer and a third feature map of a next processing layer; Matching feature points of the first feature map of the next processing layer within the search range with feature points of the third feature map of the next processing layer to obtain a third matching point pair; Determine whether the preset termination condition is met; If yes, determining the third matching point pair as the matching result; If not, the third matching point pair is used as the current matching point pair, and the step of determining the search range of the current matching point pair on the first feature map of the next processing layer and the third feature map of the next processing layer is returned until the preset termination condition is met.

5. The image registration method according to claim 4, It is characterized in that The step of determining whether a preset termination condition is met includes: Determine whether the layer number of the next processing layer is a preset value; If so, it is determined that the preset termination condition is met.

6. The image registration method according to claim 4, It is characterized in that The current matching point pair includes a first candidate feature point and a second candidate feature point matching the first candidate feature point, and the step of determining a search range of the current matching point pair on a first feature map of a next processing layer and a third feature map of a next processing layer includes: Upsampling the first candidate feature point in the current layer to obtain a first upsampled point in a next processing layer; Upsampling the second candidate feature points in the current layer to obtain second upsampled points in the next processing layer; The search range is obtained based on the first up-sampling point and the second up-sampling point.

7. The image registration method according to claim 6, It is characterized in that The search range includes a first search range and a second search range, and the step of obtaining the search range based on the upsampling point includes: Determine an area surrounded by the first up-sampling point, feature points on the right side of the first up-sampling point, and feature points on the bottom side of each of the first up-sampling points as the first search range; An area enclosed by the second up-sampling point, feature points on the right side of the second up-sampling point, and feature points on the bottom side of each of the second up-sampling points is determined as the second search range.

8. An image registration device, It is characterized in that The invention comprises a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the image registration method according to any one of claims 1 to 7.

9. A computer-readable storage medium for storing a computer program, It is characterized in that When the computer program is executed by a processor, it is used to implement the image registration method according to any one of claims 1 to 7.