Image registration method and device, computer program product and electronic equipment

Abstract

The invention discloses an image registration method and device, a computer program product and electronic equipment. The method relates to the field of remote sensing image processing or other related fields, and comprises the following steps: acquiring M remote sensing images, and respectively carrying out image scaling processing on the M remote sensing images to obtain M images to be registered; obtaining a copy image of each to-be-registered image, performing feature extraction on the M copy images by using a feature extraction algorithm, and outputting M groups of feature data; performing preliminary registration processing on the M groups of feature data to obtain M initial registration point sequences, and determining M pre-registration images according to the M initial registration point sequences; performing feature extraction on the M pre-registered images to obtain M sets of registered point sequences; and performing multi-scale screening processing on the M sets of registration point sequences to generate a target registration point sequence, and determining a registration image based on the target registration point sequence. According to the image registration method and device, the problem that the registration precision is low when image registration is carried out on the remote sensing image in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of remote sensing image processing or other related fields, and in particular, to an image registration method, device, computer program product, and electronic device. Background Art

[0002] With the rapid development of satellite remote sensing, aerial photography, drone reconnaissance and other technologies, high-resolution, large-scale remote sensing images are widely used in many fields such as urban planning, agricultural research, environmental change analysis and disaster assessment. These high-precision images provide rich geographic information and greatly enhance the insight into the dynamic changes of the earth's surface. In order to more accurately understand the changes in the surface, it is necessary to match these remote sensing images from multiple sources and in multiple time periods. For example, when monitoring long-term changes such as deforestation and land degradation, it is necessary to accurately align large-scale remote sensing images acquired at different time points to accurately compare and analyze the changes in surface features; in the application of geographic information systems, in order to build high-resolution maps or update maps, it is necessary to align old and new remote sensing images to ensure accurate correspondence of features such as terrain, roads, and buildings; after a natural disaster occurs, it is necessary to quickly and accurately align remote sensing images before and after the disaster.

[0003] Image registration in related technologies, especially remote sensing image registration, involves geometrically aligning two or more images acquired at different times, with different sensors or under different conditions, identifying and describing key points by constructing scale space and Gaussian pyramid to ensure their accurate spatial correspondence, and thus achieve matching between images; image features can also be extracted using convolutional neural networks, and feature matching can be performed using graph neural networks.

[0004] However, with the continuous advancement of remote sensing technology, due to the increasing size of collected images, whether it is the traditional manual feature method or the deep learning registration method, the traditional method requires full-size registration, which is computationally intensive, time-consuming and resource-consuming; and although the deep learning registration method works well on small to medium-sized images, when processing high-resolution, 10,000-pixel-sized remote sensing images, especially in areas with rich image details, direct downsampling will significantly reduce the registration accuracy. In addition, the size of these images far exceeds the input limit of conventional deep learning networks, so that the registration of large-size images may also have problems of memory overflow or insufficient video memory, resulting in a sharp increase in computational complexity, video memory and time consumption, and limited registration accuracy, thereby limiting the application of registration technology in large-size remote sensing image scenes.

[0005] There is a problem of low registration accuracy when performing image registration on remote sensing images in related technologies, and no effective solution has been proposed yet. Summary of the invention

[0006] The main purpose of the present application is to provide an image registration method, device, computer program product and electronic device to solve the problem of low registration accuracy when performing image registration on remote sensing images in the related art.

[0007] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for image registration is provided. The method comprises: acquiring M remote sensing images, performing image scaling processing on the M remote sensing images respectively, and obtaining M images to be registered, wherein M is greater than 1 and M is a positive integer; acquiring a copy image of each image to be registered, performing feature extraction on the M copy images respectively using a feature extraction algorithm, and outputting M groups of feature data, wherein the copy images are obtained by downsampling the images to be registered; performing preliminary registration processing on the M groups of feature data respectively, and obtaining M initial registration point sequences, and determining M pre-registered images from the M initial registration point sequences; performing feature extraction on the M pre-registered images respectively using a feature extraction algorithm, and obtaining M groups of registration point sequences, wherein each group of registration point sequences includes a registration point sequence of a first scaling factor and a registration point sequence of a second scaling factor, and the first scaling factor is less than the second scaling factor; performing multi-scale screening processing on the M groups of registration point sequences, generating a target registration point sequence, and determining the registration image based on the target registration point sequence.

[0008] Further, feature extraction is performed on the M replica images respectively using a feature extraction algorithm, and M groups of feature data are output, including: for each replica image, scale features are extracted from the replica image using Y feature extraction layers of a fusion model to obtain Y descriptor vectors, and a descriptor matrix is ​​determined based on the Y descriptor vectors, wherein the fusion model includes K feature preprocessing layers and Y feature extraction layers arranged in parallel, each feature extraction layer has a different extraction scale, and K and Y are positive integers; confidence calculation is performed on the descriptor matrix, and a confidence heat map is output, wherein the confidence heat map is used to describe the matching reliability of the Y descriptor vectors; the replica image is decomposed into a feature map to obtain different sub-regions, and each sub-region is convoluted to obtain a feature point coding matrix, wherein the feature point coding matrix is ​​used to indicate the position information of the feature point coordinates corresponding to the Y descriptor vectors; a group of feature data of the replica image is determined based on the descriptor matrix, the confidence heat map and the feature point coding matrix, wherein a group of feature data includes feature sub-data of each sub-region, and each feature sub-data includes feature point coordinates and a descriptor vector of each sub-region.

[0009] Furthermore, determining the descriptor matrix based on Y descriptor vectors includes: filtering out the descriptor vector output by the feature extraction layer at a preset position from the Y descriptor vectors to obtain a target descriptor vector; performing dimensionality transformation processing on Y-1 descriptor vectors among the Y descriptor vectors according to the spatial dimension information of the target descriptor vector to obtain Y-1 adjustment vectors; splicing the Y-1 adjustment vectors and the target descriptor vector to obtain a spliced ​​vector; and further fusing the spliced ​​vector using a fusion module associated with the fusion model to obtain a descriptor matrix.

[0010] Further, preliminary registration processing is performed on M groups of feature data respectively to obtain M initial registration point sequences, and M pre-registered images are determined from the M initial registration point sequences, including: using a mutual proximity matching algorithm to match the feature sub-data in each group of feature data to obtain M groups of matched feature data, wherein each group of matched feature data includes feature sub-data matched by the mutual proximity matching algorithm; for a group of matched feature data, a clustering algorithm is used to screen the feature point coordinates in a group of matched feature data to obtain P feature point coordinates, wherein P is a positive integer; using a transformation matrix algorithm to map the P feature point coordinates to a remote sensing image associated with the copy image to obtain a coarse registration point sequence; determining the mean and variance of the coarse registration sequence of each remote sensing image to obtain M sequence means and M sequence variances, and using each sequence mean and each sequence variance to determine the image size range data of each remote sensing image to obtain M image size range data; using the M image size range data to crop the M remote sensing images respectively to obtain M pre-registered images.

[0011] Further, using a feature extraction algorithm to extract features from the M pre-registered images respectively to obtain M groups of registration point sequences includes: obtaining a preset zoom factor, using the preset zoom factor to slice each pre-registered image to obtain M groups of slice group sequences, wherein the preset zoom factor includes a first zoom factor and a second zoom factor, each group of slice group sequences includes a slice group subsequence of the first zoom factor and a slice group subsequence of the second zoom factor, the slice group subsequence of the first zoom factor includes Q1 image slices, the slice group subsequence of the second zoom factor includes Q2 image slices, Q1 and Q2 are positive integers; for a group of slice groups subsequence, extract features from a group of slice group subsequences to obtain a group of slice feature data, wherein a group of slice feature data includes Q1 slice feature sub-data or Q2 slice feature sub-data; use a graph neural network algorithm to match a group of slice feature data to obtain a matching point group, and map the matching point group to a remote sensing image corresponding to the pre-registered image to obtain a group of registration point sequences, wherein when the preset zoom factor is a first zoom factor, the matching point group includes J feature point coordinates, and when the preset zoom factor is a second zoom factor, the matching point group includes L feature point coordinates, J is less than L, and J and L are positive integers.

[0012] Furthermore, multi-scale screening processing is performed on the M groups of alignment point sequences to generate a target alignment point sequence, including: obtaining preset range parameters, using the preset range parameters to process J feature point coordinates in the alignment point sequence of the first zoom factor to obtain J processed feature point ranges; using the J processed feature point ranges to screen L feature point coordinates in the alignment point sequence of the second zoom factor to obtain N feature point coordinates within the feature point range, and the target alignment point sequence is composed of the N feature point coordinates.

[0013] Furthermore, determining the registration image based on the target registration point sequence includes: performing homography transformation processing on the target registration point sequence to obtain a homography matrix, wherein the homography matrix is ​​used to indicate the two-dimensional perspective transformation relationship between M remote sensing images; using the homography matrix to perform projective transformation on remote sensing images of a preset proportion among the M remote sensing images to obtain a transformed registration image; and splicing the transformed registration image with the remaining remote sensing images among the M remote sensing images to obtain a registration image.

[0014] In order to achieve the above objective, according to another aspect of the present application, an image registration device is provided. The device comprises: a first acquisition unit, used for acquiring M remote sensing images, performing image scaling processing on the M remote sensing images respectively, and obtaining M images to be registered, wherein M is greater than 1 and is a positive integer; a second acquisition unit, used for acquiring a copy image of each image to be registered, performing feature extraction on the M copy images respectively by using a feature extraction algorithm, and outputting M groups of feature data, wherein the copy images are obtained by downsampling the images to be registered; a first processing unit, used for performing preliminary registration processing on the M groups of feature data respectively, and obtaining M initial registration point sequences, and determining M pre-registered images by the M initial registration point sequences; an extraction unit, used for performing feature extraction on the M pre-registered images respectively by using a feature extraction algorithm, and obtaining M groups of registration point sequences, wherein each group of registration point sequences comprises a registration point sequence of a first scaling factor and a registration point sequence of a second scaling factor, and the first scaling factor is less than the second scaling factor; and a second processing unit, used for performing multi-scale screening processing on the M groups of registration point sequences, generating a target registration point sequence, and determining the registration image based on the target registration point sequence.

[0015] According to another aspect of an embodiment of the present invention, a computer storage medium is provided. The computer storage medium is used to store a program. When the program is executed, the device where the computer storage medium is located is controlled to execute an image registration method.

[0016] According to another aspect of an embodiment of the present invention, an electronic device is provided, comprising one or more processors and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein the computer-readable instructions execute an image registration method when executed.

[0017] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program. When the computer program is executed by a processor, a method for image registration is performed.

[0018] Through the present application, the following steps are adopted: obtaining M remote sensing images, performing image scaling processing on the M remote sensing images respectively, and obtaining M images to be registered, wherein M is greater than 1 and M is a positive integer; obtaining a copy image of each image to be registered, performing feature extraction on the M copy images respectively using a feature extraction algorithm, and outputting M groups of feature data, wherein the copy images are obtained by downsampling the images to be registered; performing preliminary registration processing on the M groups of feature data respectively, and obtaining M initial registration point sequences, and determining M pre-registered images from the M initial registration point sequences; performing feature extraction on the M pre-registered images respectively using a feature extraction algorithm, and obtaining M groups of registration point sequences, wherein each group of registration point sequences includes a registration of a first scaling factor. point sequence and a registration point sequence of a second zoom multiple, the first zoom multiple is less than the second zoom multiple; multi-scale screening processing is performed on M groups of registration point sequences to generate a target registration point sequence, and a registration image is determined based on the target registration point sequence, which solves the problem of low registration accuracy when performing image registration on remote sensing images in related technologies, and extracts features from the copy image of the remote sensing image by using a feature extraction algorithm, and performs preliminary registration processing on the extracted feature data respectively to obtain the corresponding initial registration point sequence, roughly locate the common area at low resolution, and perform fine feature point extraction and matching at higher resolution, thereby achieving the technical effect of improving the accuracy of image registration, reducing the registration failure rate, and enhancing the robustness of registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a flow chart of an image registration method provided according to an embodiment of the present application;

[0021] Figure 2 is a schematic diagram of a processing method of a feature extraction algorithm provided in an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of a method for screening feature point coordinates provided in an embodiment of the present application;

[0023] Figure 4 is the registration result of the image registration method provided in the embodiment of the present application Figure 1 ;

[0024] Figure 5 is the registration result of the image registration method provided in the embodiment of the present application Figure 2 ;

[0025] Figure 6is a schematic diagram of an optional image registration method provided according to an embodiment of the present application;

[0026] Figure 7 The registration effect of the remote sensing image provided by the embodiment of the present application is Figure 1 ;

[0027] Figure 8 The registration effect of the remote sensing image provided by the embodiment of the present application is Figure 2 ;

[0028] Fig. 9 The registration effect of the remote sensing image provided by the embodiment of the present application is Figure 3 ;

[0029] Fig.10 is a schematic diagram of an image registration device provided according to an embodiment of the present application;

[0030] Fig.11 is a schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0032] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0035] It should be noted that the collected information used in this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse to use it.

[0036] The present invention is described below in conjunction with preferred implementation steps. Figure 1 is a flow chart of an image registration method provided according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0037] Step S101, acquiring M remote sensing images, performing image scaling processing on the M remote sensing images respectively, and obtaining M images to be registered, wherein M is greater than 1 and is a positive integer.

[0038] Specifically, in order to match two or more images acquired at different times, different imaging devices or under different conditions (weather, illumination, camera position and angle, etc.), and then perform remote sensing analysis based on the matched images, firstly, two or more remote sensing images with different resolutions can be acquired from different sensors or at different time points. For example, remote sensing images can be acquired from multiple platforms such as satellites, drones, and aerial photography, and the imaging methods of the images can be optical imaging, synthetic aperture radar imaging, multispectral imaging, etc. Due to differences in remote sensing data acquisition conditions and equipment, the resolution, coverage, imaging angle and time of these images may be different, but their common parts need to be accurately registered. At this time, the original remote sensing images need to be scaled to adapt to different analysis requirements and calculation limitations. At this time, the long edges of the two images can be cut off and the short edges can be filled. At the same time, the length and width of the images can be made consistent by scaling. Finally, each image is adjusted to a specific resolution, and the corresponding image to be registered is obtained to meet the requirements of subsequent processing.

[0039] It should be noted that image scaling can also reduce the image size and processing speed by downsampling or down-sampling when computing resources are limited or when processing large-size images. It can also be an upsampling method to improve the resolution or restore details after feature extraction on a lower-resolution image. In order to maintain image quality, algorithms such as bilinear interpolation and bicubic interpolation can be used to avoid distortion or moiré effects during the scaling process.

[0040] Furthermore, due to the differences between images, when the image aspect ratio is not 1:1, in order to avoid information loss caused by non-proportional scaling, it is necessary to expand the short side of the image with an empty background until the image is 1:1; for SAR (Synthetic Aperture Radar) images, it is necessary to expand to 3 channels by copying the image; for the full spectrum or other image channel order mismatch, it is necessary to modify the channel arrangement order (such as from "RGB" to "BGR", that is, from "Red", Green and Blue" to "Blue, Green and Red").

[0041] Step S102, obtaining a duplicate image of each image to be registered, performing feature extraction on the M duplicate images respectively using a feature extraction algorithm, and outputting M groups of feature data, wherein the duplicate images are obtained by downsampling the images to be registered.

[0042] Specifically, due to the large size of the original remote sensing images, especially when the GPU (Graphics Processing Unit) has limited video memory, directly using them as input for deep learning models may lead to excessive consumption of computing resources. Therefore, for each high-resolution remote sensing image to be registered, a copy image needs to be created and downsampled to reduce the image size, which can significantly reduce the computational complexity and make the feature extraction process more efficient.

[0043] After obtaining the downsampled copy images, feature extraction algorithms can be used to extract features from these copy images, identify key points or areas in the images, and then obtain multiple sets of feature data, where the feature extraction algorithm can be the XFeat deep learning algorithm, and the identified key points contain rich geographic information and texture features. Therefore, each set of feature data contains a series of key point coordinate information and their descriptor vectors, and descriptors are used to quantify the features of these key points for subsequent matching and alignment.

[0044] Step S103 , performing preliminary registration processing on the M groups of feature data respectively to obtain M initial registration point sequences, and determining M pre-registered images from the M initial registration point sequences.

[0045] Specifically, after obtaining a set of feature data for each remote sensing image, the feature data of each set of resolutions or obtained by different sensors can be preliminarily aligned, thereby establishing the correspondence between key points of different images. Since the data in each set of feature data contains the coordinates of feature points extracted from the downsampled copy image and the corresponding descriptor vectors, these descriptor vectors are vectors used to quantify the features of feature points and can reflect the position, texture, shape and other information of key points in the image, at this time, by comparing the descriptor vectors in different sets of feature data, the matching pairs of feature points in each set of images can be found, thereby obtaining the corresponding initial alignment point sequence.

[0046] It should be noted that when matching, the nearest neighbor method (NN), mutual nearest neighbor matching (MNN) or deep learning model can be used for matching. The correspondence of feature points is determined based on the similarity of descriptor vectors. By calculating the distance between descriptor vectors, the "most similar" feature point pair in each pair of images is found, thereby generating a preliminary registration point sequence.

[0047] Furthermore, based on the obtained initial registration point sequence, the corresponding pre-registered image can be determined, that is, the initial registration point sequence is mapped back to the original resolution and size of the image to be registered, and the image to be registered is geometrically adjusted by estimating the transformation matrix, etc., so that they are preliminarily aligned in space, which not only eliminates the rough displacement, rotation or scale difference between the images, but also provides a better starting point for the subsequent fine registration.

[0048] Step S104, using a feature extraction algorithm to perform feature extraction on the M pre-registered images respectively to obtain M groups of registration point sequences, wherein each group of registration point sequences includes a registration point sequence of a first zoom factor and a registration point sequence of a second zoom factor, and the first zoom factor is smaller than the second zoom factor.

[0049] Specifically, after obtaining the corresponding pre-registered image, a feature extraction algorithm can be used to perform multi-level feature extraction on the pre-registered image, thereby generating a registration point sequence of different zoom factors to ensure accurate registration of large-size images. First, a strategy of different zoom factors can be adopted, and a feature extraction algorithm can be used to perform feature extraction on the pre-registered image. The obtained registration point sequence can include a registration point sequence of a first zoom factor that is usually generated based on a lower resolution and a registration point sequence of a second zoom factor that is generated at a relatively higher resolution.

[0050] At this time, the common overlapping area of ​​the image and the rough registration result can be determined based on the registration point sequence of the first zoom factor. Then, the registration point sequence of the second zoom factor is screened using the registration result of the first zoom factor to limit the search range or guide the matching process, avoiding full-image search at high resolution, thereby improving matching efficiency and accuracy. Finally, the registration point sequence of the second zoom factor is finely matched, using a matching algorithm such as LightGlue, combined with the results of the first zoom factor, to perform more accurate registration point matching and outlier removal, ensuring the high accuracy and reliability of the final registration point.

[0051] Then, the registration point sequences at different zoom factors are fused and unified into the coordinate system of the original image through geometric transformation and position mapping to generate the final registration point sequence.

[0052] Step S105 , performing multi-scale screening processing on the M groups of registration point sequences to generate a target registration point sequence, and determining a registration image based on the target registration point sequence.

[0053] Since each set of registration point sequences contains registration information at different scales, from low-resolution large-scale features to high-resolution fine features, and since there may be some abnormal registration points in the preliminary registration point sequence due to image noise, occlusion, texture duplication or algorithm errors, these points have a negative impact on the final registration result, it is necessary to screen the registration point sequence and generate a target registration point sequence, which not only contains the most reliable registration point pairs, but also reflects the comprehensive information between features of different scales.

[0054] After generating the target registration point sequence, the homography matrix of the registration point sequence can be estimated, and image projection registration can be performed based on the homography matrix to ensure their consistency in geographic space, so as to obtain a registered image.

[0055] The image registration method provided in the embodiment of the present application obtains M remote sensing images, performs image scaling processing on the M remote sensing images respectively, and obtains M images to be registered, wherein M is greater than 1 and M is a positive integer; obtains a copy image of each image to be registered, uses a feature extraction algorithm to extract features on the M copy images respectively, and outputs M groups of feature data, wherein the copy images are obtained by downsampling the images to be registered; performs preliminary registration processing on the M groups of feature data respectively, obtains M initial registration point sequences, and determines M pre-registered images from the M initial registration point sequences; uses a feature extraction algorithm to extract features on the M pre-registered images respectively, and obtains M groups of registration point sequences, wherein each group of registration point sequences includes a first scaling factor and a registration point sequence of a second zoom multiple, the first zoom multiple being less than the second zoom multiple; performing multi-scale screening processing on M groups of registration point sequences to generate a target registration point sequence, and determining a registration image based on the target registration point sequence, thereby solving the problem of low registration accuracy when performing image registration on remote sensing images in related technologies, extracting features from a copy image of the remote sensing image by utilizing a feature extraction algorithm, and performing preliminary registration processing on the extracted feature data to obtain a corresponding initial registration point sequence, roughly locating the common area at a low resolution, and performing fine feature point extraction and matching at a higher resolution, thereby achieving the technical effect of improving the accuracy of image registration, reducing the registration failure rate, and enhancing the robustness of registration.

[0056] Figure 2 is a schematic diagram of a processing method of a feature extraction algorithm provided in an embodiment of the present application, such as Figure 2 As shown, optionally, in the image registration method provided in the embodiment of the present application, the feature extraction algorithm extracts features from the M replica images respectively, and outputs M groups of feature data, including: for each replica image, using Y feature extraction layers of the fusion model to extract scale features from the replica image, to obtain Y descriptor vectors, and determining a descriptor matrix based on the Y descriptor vectors, wherein the fusion model includes K feature preprocessing layers and Y feature extraction layers arranged in parallel, each feature extraction layer has a different extraction scale, and K and Y are positive integers; performing confidence calculation on the descriptor matrix, and outputting the descriptor matrix. A confidence heat map is provided, wherein the confidence heat map is used to describe the matching reliability of Y descriptor vectors; a feature map is decomposed on the copy image to obtain different sub-regions, and each sub-region is convolved to obtain a feature point coding matrix, wherein the feature point coding matrix is ​​used to indicate the position information of the feature point coordinates corresponding to the Y descriptor vectors; a set of feature data of the copy image is determined according to the descriptor matrix, the confidence heat map and the feature point coding matrix, wherein a set of feature data includes feature sub-data of each sub-region, and each feature sub-data includes feature point coordinates and a descriptor vector of each sub-region.

[0057] It should be noted that the feature extraction algorithm can be the XFeat deep learning algorithm, and the fusion model in the algorithm can adopt a pyramid structure, which can include multiple feature preprocessing layers and multiple feature extraction layers set in parallel. The feature preprocessing layer can perform preliminary feature extraction and enhancement on the input image, providing a good foundation for subsequent multi-scale analysis. The three parallel feature extraction layers use features of different scales, so that each feature extraction layer can independently focus on a specific scale range, so as to more comprehensively capture the information in the image. In order to obtain the image features in each remote sensing image, the multi-layer structure of the fusion model can be used to perform multi-scale feature extraction on the copy image of each remote sensing image, thereby capturing multi-level and different scale information in the image, providing rich data support for the precise matching and position determination of feature points.

[0058] When extracting features from a duplicate image, after the image is input, it enters the feature point extraction structure and the descriptor extraction structure respectively. In the descriptor extraction structure, the feature extraction layer can first be used to extract scale features to obtain multiple descriptor vectors. When there are two feature preprocessing layers and three feature extraction layers in parallel, the descriptor vectors output by the feature extraction layer are represented as x3, x4, and x5. At this time, the sizes of the vectors output by the three feature extraction layers are respectively Then, based on the above three descriptor vectors, they are fused to generate a descriptor matrix that can achieve feature point matching and position mapping, and its size is Furthermore, the confidence block is used to calculate the confidence of the obtained descriptor matrix, and then a confidence heat map is output to characterize the matching reliability of the descriptor position, thereby helping the model to screen out key points that are more likely to be successfully matched and improve the accuracy and efficiency of the registration.

[0059] While calculating the descriptor vector, the feature point extraction structure can also be performed. First, the output copy image can be unfolded, that is, the image is decomposed into multiple 8×8 sub-regions, and flattened to obtain a feature map with c*64 channels (c is the number of channels in the original image). Then, each sub-region is convolved to obtain a feature point encoding matrix, that is, input several convolution layers to extract features. The feature point coding matrix is ​​a matrix of feature points, where "64" represents the possible position in each grid unit, and "+1" represents a "trash can" category, which is used to indicate that no key point is found in the unit, and the matrix not only contains the descriptor vectors of each sub-region, but also indicates the position information of the feature points corresponding to these descriptor vectors in the image coordinate system. Finally, the descriptor matrix, confidence heat map and feature point coding matrix output by the two structures can be used to determine a set of feature data of a copy image. This embodiment captures the macroscopic structure and microscopic details in the image through multi-scale feature extraction, and decomposes the image. The feature point coding matrix obtained contains feature point coordinate information, which provides precise guidance for registration mapping, reduces the impact of incorrect pairing and outliers, and can also significantly reduce the consumption of computing resources, thereby improving the accuracy of the registration results.

[0060] Optionally, in the image registration method provided in the embodiment of the present application, determining the descriptor matrix based on Y descriptor vectors includes: filtering out the descriptor vector output by the feature extraction layer at a preset position from the Y descriptor vectors to obtain a target descriptor vector; performing dimensionality transformation processing on Y-1 descriptor vectors among the Y descriptor vectors according to the spatial dimension information of the target descriptor vector to obtain Y-1 adjustment vectors; splicing the Y-1 adjustment vectors and the target descriptor vector to obtain a spliced ​​vector; and further fusing the spliced ​​vector using a fusion module associated with the fusion model to obtain a descriptor matrix.

[0061] When constructing a matrix using a descriptor vector, the descriptor vector output by the feature extraction layer at a preset position can be first obtained. For example, when the copy image passes through two feature preprocessing layers and is respectively input into three feature extraction layers arranged in parallel, the target descriptor vector output by the feature extraction layer of the third layer can be obtained, and the spatial dimension information of the vector can be determined. Then, the spatial dimension is adjusted (such as by interpolation) and the feature dimension is adjusted (such as by convolution or a fully connected layer), and the vectors output by the two layers x4 and x5 are converted in dimension to ensure that all vectors have the same spatial dimension information, thereby obtaining the corresponding adjustment vector.

[0062] Furthermore, in order to enable descriptor vectors of different scales to be fused under a standardized framework, thereby avoiding information loss or mismatch problems caused by scale differences, the above-mentioned adjustment vector and the target descriptor vector can be merged and spliced, and the spliced ​​vector can be further fused using the fusion head block (i.e., fusion module) associated with the fusion model, and finally a descriptor matrix is ​​generated. By screening and fusing descriptor vectors of different scales, this embodiment can effectively utilize the multi-level information of the image, eliminate the impact of scale differences, improve the robustness and accuracy of the registration, and overcome the differences in image features under different sensors or imaging conditions, and improve the cross-data source capability of registration and matching. In addition, compared with directly processing the original high-resolution image, preliminary feature extraction and registration and matching based on a smaller copy image can significantly reduce the consumption of computing resources.

[0063] Optionally, in the image registration method provided in the embodiment of the present application, performing preliminary registration processing on M groups of feature data respectively to obtain M initial registration point sequences, and determining M pre-registered images from the M initial registration point sequences includes: using a mutual proximity matching algorithm to match feature sub-data in each group of feature data to obtain M groups of matched feature data, wherein each group of matched feature data includes feature sub-data matched by the mutual proximity matching algorithm; for a group of matched feature data, using a clustering algorithm to match the feature sub-data in a group of matched feature data The feature point coordinates are screened to obtain P feature point coordinates, where P is a positive integer; the P feature point coordinates are mapped to the remote sensing image associated with the copy image using a transformation matrix algorithm to obtain a coarse registration point sequence; the mean and variance of the coarse registration sequence of each remote sensing image are determined to obtain M sequence means and M sequence variances, and the image size range data of each remote sensing image is determined using each sequence mean and each sequence variance to obtain M image size range data; the M remote sensing images are cropped using the M image size range data to obtain M pre-registered images.

[0064] Specifically, the mutual proximity matching algorithm can find the nearest neighbor feature point pairs by calculating the distance between the feature point descriptors. For example, when there are two remote sensing images (i.e., image A and image B), in order to determine the common coverage area of ​​the two images, after obtaining the feature data corresponding to the two remote sensing images A and image B, the mutual proximity matching algorithm can be used to perform corresponding matching on the feature sub-data in the two sets of feature data. For each feature point extracted from image A, the distance between its descriptor vector and the descriptor vectors of all feature points in the other image is calculated, and then the nearest feature point in image B is determined based on the distance. At the same time, for each feature point extracted from image B, the nearest feature point in image A is found, that is, the nearest neighbor feature point of the nearest feature point in the other image is determined. If the first nearest neighbor of a feature point is the nearest neighbor feature point in the other image, then the pair of feature points is considered to be the nearest neighbors of each other, that is, only when the feature point in image A and the feature point in image B are the nearest neighbors of each other, they are considered to be a matching pair. After determining whether a certain feature sub-data in group A can be matched in group B, all the matched feature sub-data in groups A and B can be recombined to obtain two groups of matched feature data and quickly determine a preliminary corresponding relationship.

[0065] Furthermore, since the matched feature data may contain some outliers or mismatched feature points, which will have a negative impact on the subsequent registration results, the feature point coordinates in a set of matched feature data can be screened by a clustering algorithm to eliminate these outliers and retain only those feature points that are relatively clustered in spatial position. The clustering algorithm can be a K-means (K-Means Clustering Algorithm) algorithm or a DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm, which can group the feature point coordinates into several clusters according to their spatial distribution, thereby obtaining multiple feature point coordinates that are finally retained.

[0066] Furthermore, the transformation matrix algorithm can be used to map the selected feature point coordinates back to the original remote sensing image, so as to perform more accurate registration at the original resolution, thereby obtaining a rough registration point sequence, which provides preliminary corresponding point information for subsequent fine registration. It should be noted that since there is no displacement between the input copy image and the remote sensing image, only scaling processing exists, for example, the feature point coordinates obtained are (x p ,y p ), then projected to the coordinates (x r ,y r )’s transformation matrix and calculation method are:

[0067]

[0068] Among them, W model and H model Represents the preset parameters. After the projection of all feature point coordinates is completed, the rough registration point sequences of the two remote sensing images in the original size can be represented as and in, represents the i-th feature point obtained in image A, represents the i-th feature point obtained in image B. Since the two images have been matched, the number of feature points in the coarse registration point sequence is the same.

[0069] Furthermore, in order to determine the common overlapping area between the two images, the coordinate range can be determined by using the mean and variance of the rough registration sequence of each remote sensing image, that is, calculating P A and P B The relatively stable coordinate range covered by all feature points in the image is obtained. At this time, the mean value (x mean ,y mean ) and variance(x std ,y std ), and obtain the image size range data R: R = ([x mean -2·x std ,x mean +2 x std ],[y mean -2·y std ,y mean +2·y std ]). Then, the image range R of the two remote sensing images can be determined according to the image size range data. A ,R B .

[0070] Finally, based on the common overlapping area of ​​the two, the size range data R is determined. A ,R B The two remote sensing images are cropped respectively to generate corresponding pre-registered images A' and B', which are relatively aligned in size and spatial position, providing a more accurate basis for subsequent fine registration and multi-scale analysis. This embodiment uses mutual proximity matching algorithm and clustering algorithm to determine the sequence of coarse registration points, thereby significantly reducing the amount of calculation, improving the registration speed, ensuring that the feature points used for registration have high confidence and spatial consistency, thereby improving the accuracy of registration and achieving accurate alignment of images.

[0071] In order to reduce the consumption of computing resources and improve the registration efficiency, optionally, in the image registration method provided in the embodiment of the present application, using a feature extraction algorithm to respectively extract features of the M pre-registered images to obtain M groups of registration point sequences includes: obtaining a preset zoom factor, using the preset zoom factor to perform slice processing on each pre-registered image to obtain M groups of slice group sequences, wherein the preset zoom factor includes a first zoom factor and a second zoom factor, each group of slice group sequences includes a slice group subsequence of the first zoom factor and a slice group subsequence of the second zoom factor, the slice group subsequence of the first zoom factor includes Q1 image slices, and the slice group subsequence of the second zoom factor includes Q2 image slices , Q1 and Q2 are positive integers; for a group of slice group subsequences, feature extraction is performed on a group of slice group subsequences to obtain a group of slice feature data, wherein a group of slice feature data includes Q1 slice feature sub-data or Q2 slice feature sub-data; a group of slice feature data is matched using a graph neural network algorithm to obtain a matching point group, and the matching point group is mapped to a remote sensing image corresponding to the pre-registered image to obtain a group of registration point sequences, wherein, when the preset zoom factor is a first zoom factor, the matching point group includes J feature point coordinates, and when the preset zoom factor is a second zoom factor, the matching point group includes L feature point coordinates, J is less than L, and J and L are positive integers.

[0072] Specifically, after obtaining the cropped pre-registered image, the feature points at low resolution can be used to guide the position of the feature points at high resolution, and the registration accuracy can be improved by combining the advantages of the relatively complete overall structure of the lower resolution slice and the finer position of the higher resolution slice. When the first zoom factor and the second zoom factor are 2 times and 8 times respectively, the lower resolution can focus on quickly locating the approximate registration points between the two images, usually covering a larger part of the image, but the details of the feature points may not be accurate enough. The higher resolution can capture more detailed information, the positioning of the feature points is more accurate, and the number of feature points in the sequence is large. At this time, the registration results of the 2-fold coarse registration point group sequence can be used to guide the registration point screening of the 8-fold fine registration point group sequence. The feature point information at low resolution can be used to determine a reasonable search range, and then the matching results are restricted according to this range on the high-resolution image slice, so as to screen out more accurate feature points. This "coarse to fine" strategy can effectively combine the advantages of high and low resolution images to achieve efficient and high-precision registration of large-scale remote sensing images.

[0073] For example, when the pre-registered images are A′ and B′, the two images can be firstly processed into grid slices, that is, the preset zoom factor is set to N, the length and width of the pre-registered images A′ and B′ are unified to (H, W) after pre-processing, and the size of each image slice to be registered after the two images are sliced ​​is That is, it contains N×N image slices in total. In order to achieve both accuracy and low abnormality rate, the first zoom factor in the preset zoom factors can be set to N=2, and the second zoom factor can be set to N=8. At this time, the slice group subsequence of the second zoom factor and the slice group subsequence of the first zoom factor can be obtained, that is, the 8-fold fine slice group sequence and the 2-fold coarse slice group sequence between the pre-registered images A′ and B′ are obtained. The slice group subsequence of the second zoom factor is expressed as (C a8 ,C b8 ), the slice group subsequence of the first zoom factor is represented as (C a2 ,C b2 ), where (C a2 ,C b2 ) represents the set of feature point coordinates extracted and registered from the pre-registered images A′ and B′ that have been scaled 2 times, (C a8 ,C b8 ) represents the set of feature point coordinates extracted and aligned from the pre-registered images A′ and B′ that have been scaled 8 times. The slice group subsequence with the first scaling factor covers a larger image area, but has a lower resolution and is mainly used for quickly locating the overlapping area of ​​the image and preliminary feature matching; while the slice group subsequence with the second scaling factor provides a higher resolution in a smaller area and can more accurately capture image details and align.

[0074] Furthermore, after obtaining the above-mentioned slice group subsequences, the XFeat algorithm can be used to extract features from these sequences: first, based on the coarse slice group sequence (C a2 ,C b2 ) is registered, that is, the image slice (a i ,b i ) is processed, so that the output set of slice feature data includes feature point coordinates (pts a ,pts b ) i And the corresponding descriptor vector (desc a ,desc b ) i According to the above method, for the fine section group sequence (C a8 ,C b8 ), using the XFeat algorithm to slice the image (a i ,b i ) is processed, so that the feature point coordinates and the corresponding descriptor vector in the output set of slice feature data are expressed as (pts a ,pts b ) i ,(desc a ,desc b ) i .

[0075] Furthermore, a graph neural network algorithm (such as the Light Glue matching algorithm) is used to match a set of slice feature data, that is, according to the coarse slice group sequence (C a2 ,C b2 ) in the feature point coordinates (pts a ,pts b ) i And the corresponding descriptor vector (desc a ,desc b ) i Matching is performed to obtain the matching point group (mpts a ,mpts b ) i According to the fine section group sequence (C a8 ,C b8 ) in the feature point coordinates (pts a ,pts b ) i And the corresponding descriptor vector (desc a ,desc b ) i Matching is performed to obtain the matching point group (mpts a ,mpts b ) i .

[0076] According to the relative position of the image slice with respect to the original remote sensing image, the coordinates of these feature points are mapped back to the coordinate system corresponding to the remote sensing image, thereby obtaining a 2-fold registration point sequence. Since the image slice has both displacement and scaling relative to the original remote sensing image, it is assumed that the image slice (a i ,b i ) relative to the upper left corner of the remote sensing image is offset by (Δx i ,Δy i ), the zoom factor is N, and the change matrix and calculation method projected to the remote sensing image are as follows:

[0077]

[0078] Among them, W model and H model Indicates the preset parameters. Represents the coordinates of the i-th feature point projected onto the remote sensing image, Represents the coordinates of the i-th feature point in the matching point group. The above formula can be used to obtain the double registration point sequence based on the remote sensing image (mpts A2 ,mpts B2Similarly, according to the above method, that is, according to the relative position relationship of the image slice with respect to the original remote sensing image, the coordinates of these feature points are mapped back to the coordinate system corresponding to the remote sensing image, thereby obtaining an 8-fold registration point sequence (mpts A8 ,mpts B8 ). This embodiment introduces a multi-scale slicing strategy to convert the feature point positions from the slice scale to the original image scale, providing an accurate reference point for subsequent image registration, and can perform hierarchical feature extraction and matching at different resolutions, avoiding the huge amount of calculation and memory consumption caused by directly processing full-resolution images, and improving the overall efficiency of the registration process.

[0079] Optionally, in the image registration method provided in the embodiment of the present application, multi-scale screening processing is performed on M groups of registration point sequences to generate a target registration point sequence, including: obtaining preset range parameters, using the preset range parameters to process J feature point coordinates in the registration point sequence of a first zoom factor to obtain J processed feature point ranges; using the J processed feature point ranges to screen L feature point coordinates in the registration point sequence of a second zoom factor to obtain N feature point coordinates within the feature point range, and the target registration point sequence is composed of the N feature point coordinates.

[0080] Specifically, the registration point sequence (mpts A2 ,mpts B2 ) and the registration point sequence (mpts A8 ,mpts B8 ), a screening process based on multi-scale registration points can be performed. Figure 3 is a schematic diagram of a method for screening feature point coordinates according to an embodiment of the present application, such as Figure 3 As shown, first, based on the registration point sequence (mpts A2 ,mpts B2 ) Construct a screening range: Assume that the coordinates of the feature points in the sequence are (x, y) i After obtaining the preset range parameter c, the processed feature point range of all feature point coordinates can be determined, that is, the corresponding c×c circular range ([xc,x+c],[yc,y+c]) i , the coverage ranges corresponding to these processed feature point ranges are calculated by union, and we can get:

[0081] The above range can be used to guide the search for feature points on high-resolution images, ensuring that high-precision matching is concentrated in the most likely common area, avoiding blind and extensive searches on large-scale remote sensing images. That is, the feature point coordinates in the second zoom multiple registration point sequence are screened using the processed feature point range, and only those feature points within the processed feature point range are retained, and the retained feature point coordinates constitute the target registration point sequence (mpts final ,mpts final ). This embodiment can obtain a more accurate registration result by combining the common area estimation of low-resolution images and the precise feature matching of high-resolution images. This strategy ensures the high quality of the registration points. The matching of low-resolution images can quickly identify the main overlapping areas of the images, while the matching of high-resolution images can find more accurate correspondences in this area, while significantly reducing the amount of calculation and improving the efficiency of registration.

[0082] Optionally, in the image registration method provided in an embodiment of the present application, determining the registration image based on the target registration point sequence includes: performing homography transformation processing on the target registration point sequence to obtain a homography matrix, wherein the homography matrix is ​​used to indicate the two-dimensional perspective transformation relationship between M remote sensing images; using the homography matrix to perform a projection transformation on a preset proportion of remote sensing images among the M remote sensing images to obtain a transformed registration image; and splicing the transformed registration image with the remaining remote sensing images among the M remote sensing images to obtain a registration image.

[0083] Specifically, after obtaining the target registration point sequence (mpts final ,mpts final ), the sequence can be transformed homographically, that is, by finding the correspondence between the common points of the two images, a 3x3 homography matrix H can be calculated. This matrix contains all the necessary information to map the coordinates of one image to another image. After calculating the homography matrix, the matrix can be used to perform a projection transformation on one or more remote sensing images with lower resolution or the remote sensing images that play a leading role, that is, to convert each pixel coordinate of these images to the coordinate system of another image through the matrix. Figure 4 is the registration result of the image registration method provided in the embodiment of the present application Figure 1 , Figure 5 is the registration result of the image registration method provided in the embodiment of the present application Figure 2 ,like Figure 4 , Figure 5 As shown in Figure 1, after the projection transformation, each pixel of the target image can be mapped to a new position, and blank areas or pixel overlaps may occur. In order to generate the final transformed registration image, the pixel values ​​can be filled by nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation.

[0084] Furthermore, the transformed registration image is spliced ​​with the remaining images that have not undergone projection transformation to obtain a complete registration image. It should be noted that if there is overlap between multiple images, it is necessary to determine how to merge these areas to avoid pixel duplication or conflict; if the size of the transformed image exceeds the range of the original image, blank spaces may be generated on the border. At this time, it is necessary to determine whether to fill the blank area or crop the excess pixels to ensure the size consistency and visual effect of the final registered image. This embodiment uses homography transformation to perform projection registration on the image, so that the transformed image can be geometrically aligned with the target image, and images acquired at different times and by different sensors can be accurately aligned, thereby achieving high-precision registration.

[0085] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0086] The present application also provides an image registration method. Figure 6 is a schematic diagram of an optional image registration method provided according to an embodiment of the present application, such as Figure 6 As shown, the method includes:

[0087] Specifically, when two remote sensing images (i.e., images A and B to be registered) are registered, the two images are first preprocessed, a copy is made and downsampled to 1024×1024 pixels, the downsampled copy image is input into the XFeat deep learning algorithm for feature extraction, and the extracted feature points are matched based on the mutual nearest neighbor (MNN) matching algorithm, and the K-Means (K-Means Clustering Algorithm) clustering algorithm is used to filter out abnormal points, and the remaining registration points are projected back to the original image based on the downsampling ratio. After all feature points are projected, the registration point sequence P of the original size of the two images is obtained. A and P B Then, according to the filtered registration point sequence P A and P B , the mean square error algorithm is used to calculate the stable range of coordinate values ​​of all registration points in the sequence R = [(x m1 ,x m2 ),(y m1 ,y m2 )], and obtain the common overlapping area range R of the two images A ,R B, alignment and cropping are performed based on the common overlapping areas to obtain the roughly registered images A′, B′ with basically aligned overall positions.

[0088] Furthermore, the above coarse registration images are sliced ​​by multi-strategy sliding windows, that is, the two images are gridded, and the relatively small slices after the gridding process are used for subsequent sliding window registration to reduce the downsampling multiple loss when inputting the network. When the scaling multiples are set to 8 times and 2 times, an 8-fold fine slice group sequence (i.e., small size) can be obtained (C a8 ,C b8 ) and 2 times coarse slice group sequence (i.e. large size) (C a2 ,C b2 ).

[0089] Furthermore, the above sequence (C a ,C b ) are input into the XFeat neural network in sequence. For each slice group to be registered (a i ,b i ) to obtain the feature point sequence and the corresponding descriptor vector sequence (pts a ,pts b ) i ,(desc a ,desc b ) i , and use the Lightglue matching algorithm and outlier elimination method to match, and obtain the coarse registration point group and the fine registration point group (mpts a ,mpts b ) i Then, according to the relative position relationship of the image slice with respect to the original image, the coordinates of the registration points in the coarse registration point group and the fine registration point group are mapped back to the coordinate system based on the original image, and the coarse registration point group sequences (mpts A2 ,mpts B2 ) and the precise alignment point sequence (mpts A8 ,mpts B8 ).

[0090] The registration points are screened based on the upper feature structure screening method: set the parameter C = 10, and determine the screening range sequence ([X 1 ,Y 1 ],[X 2 ,Y 2 ]) i , and then filter out the points that do not belong to the range sequence in the registration points according to the range sequence, that is, for the high-resolution registration point sequence (mpts A8 ,mpts B8 ) If the point and any low-resolution registration point sequence (mpts A2 ,mpts B2 ) is not less than c, then give up this point Finally, the fine registration point sequence result (mpts final ,mpts final ).

[0091] Finally, based on the obtained fine registration point group (mpts final ,mpts final ) is projected onto the coordinate system of the original remote sensing image, and homography matrix estimation and image projection registration are performed to obtain the registered image. This embodiment uses a deep neural network to extract feature points, divides large-scale remote sensing images into smaller-scale slices for independent registration, and roughly locates common areas at low resolution, and then performs fine feature point extraction and matching at higher resolution, thereby achieving the technical effect of improving the accuracy of image registration, reducing the registration failure rate, and enhancing the robustness of registration.

[0092] It should be noted that Figure 7 The registration effect of the remote sensing image provided by the embodiment of the present application is Figure 1 ,like Figure 7 As shown, the above registration method can be used for remote sensing images taken by infrared optical technology; Figure 8 The registration effect of the remote sensing image provided by the embodiment of the present application is Figure 2 ,like Figure 8 As shown, the above registration method can also be applied to remote sensing images taken by SAR-optical technology; Fig. 9 The registration effect of the remote sensing image provided by the embodiment of the present application is Figure 3 ,like Fig. 9 As shown, this registration method can also be applied to remote sensing images taken with multispectral technology.

[0093] The present application also provides an image registration device. It should be noted that the image registration device of the present application can be used to execute the image registration method provided in the present application. The image registration device provided in the present application is introduced below.

[0094] Fig.10 is a schematic diagram of an image registration device provided according to an embodiment of the present application, such as Fig.10 As shown, the device includes: a first acquisition unit 1001, a second acquisition unit 1002, a first processing unit 1003, an extraction unit 1004, and a second processing unit 1005.

[0095] The first acquisition unit 1001 is used to acquire M remote sensing images, and perform image scaling processing on the M remote sensing images respectively to obtain M images to be registered, where M is greater than 1 and is a positive integer;

[0096] The second acquisition unit 1002 is used to acquire a duplicate image of each image to be registered, perform feature extraction on the M duplicate images respectively using a feature extraction algorithm, and output M groups of feature data, wherein the duplicate image is obtained by downsampling the image to be registered;

[0097] The first processing unit 1003 is used to perform preliminary registration processing on the M groups of feature data respectively to obtain M initial registration point sequences, and determine M pre-registered images from the M initial registration point sequences;

[0098] An extraction unit 1004 is used to perform feature extraction on the M pre-registered images respectively using a feature extraction algorithm to obtain M groups of registration point sequences, wherein each group of registration point sequences includes a registration point sequence of a first zoom factor and a registration point sequence of a second zoom factor, and the first zoom factor is smaller than the second zoom factor;

[0099] The second processing unit 1005 is used to perform multi-scale screening processing on the M groups of registration point sequences to generate a target registration point sequence, and determine a registration image based on the target registration point sequence.

[0100] The image registration device provided in the embodiment of the present application obtains M remote sensing images through the first acquisition unit 1001, and performs image scaling processing on the M remote sensing images respectively to obtain M images to be registered, wherein M is greater than 1 and M is a positive integer; the second acquisition unit 1002 obtains a copy image of each image to be registered, and uses a feature extraction algorithm to extract features of the M copy images respectively, and outputs M groups of feature data, wherein the copy images are obtained by downsampling the images to be registered; the first processing unit 1003 performs preliminary registration processing on the M groups of feature data respectively to obtain M initial registration point sequences, and determines M pre-registered images from the M initial registration point sequences; the extraction unit 1004 uses a feature extraction algorithm to extract features of the M pre-registered images respectively to obtain M groups of registration point sequences, wherein each The group of registration point sequences includes a registration point sequence of a first zoom factor and a registration point sequence of a second zoom factor, and the first zoom factor is smaller than the second zoom factor; the second processing unit 1005 performs multi-scale screening processing on the M groups of registration point sequences to generate a target registration point sequence, and determines a registration image based on the target registration point sequence, thereby solving the problem of low registration accuracy when performing image registration on remote sensing images in related technologies, and extracting features from a copy image of the remote sensing image by using a feature extraction algorithm, and performing preliminary registration processing on the extracted feature data to obtain a corresponding initial registration point sequence, roughly locating the common area at a low resolution, and performing fine feature point extraction and matching at a higher resolution, thereby achieving the technical effect of improving the accuracy of image registration, reducing the registration failure rate, and enhancing the robustness of registration.

[0101] Optionally, in the image registration device provided in the embodiment of the present application, the second acquisition unit 1002 includes: a first extraction module, which is used to extract scale features of the copy image using Y feature extraction layers of the fusion model for each copy image, obtain Y descriptor vectors, and determine the descriptor matrix according to the Y descriptor vectors, wherein the fusion model includes K feature preprocessing layers and Y feature extraction layers arranged in parallel, the extraction scale of each feature extraction layer is different, and K and Y are positive integers; a calculation module, which is used to calculate the confidence of the descriptor matrix and output a confidence heat map, wherein the confidence The heat map is used to describe the matching reliability of Y descriptor vectors; the decomposition module is used to decompose the feature map of the copy image to obtain different sub-regions, and perform convolution processing on each sub-region to obtain a feature point coding matrix, wherein the feature point coding matrix is ​​used to indicate the position information of the feature point coordinates corresponding to the Y descriptor vectors; the first determination module is used to determine a set of feature data of the copy image according to the descriptor matrix, the confidence heat map and the feature point coding matrix, wherein a set of feature data includes feature sub-data of each sub-region, and each feature sub-data includes feature point coordinates and a descriptor vector of each sub-region.

[0102] Optionally, in the image registration device provided in the embodiment of the present application, the second acquisition unit 1002 includes: a first screening module, used to screen out the descriptor vector output by the feature extraction layer at a preset position from the Y descriptor vectors to obtain a target descriptor vector; a first processing module, used to perform dimensional transformation processing on Y-1 descriptor vectors among the Y descriptor vectors according to the spatial dimension information of the target descriptor vector to obtain Y-1 adjustment vectors; a second processing module, used to splice the Y-1 adjustment vectors and the target descriptor vector to obtain a spliced ​​vector; and a third processing module, used to perform further fusion processing on the spliced ​​vector using a fusion module associated with the fusion model to obtain a descriptor matrix.

[0103] Optionally, in the image registration device provided in the embodiment of the present application, the first processing unit 1003 includes: a fourth processing module, which is used to match the feature sub-data in each set of feature data using a mutual proximity matching algorithm to obtain M sets of matched feature data, wherein each set of matched feature data includes feature sub-data matched by the mutual proximity matching algorithm; a second screening module, which is used to screen the feature point coordinates in a set of matched feature data using a clustering algorithm to obtain P feature point coordinates, wherein , P is a positive integer; a mapping module, used to map the coordinates of P feature points to the remote sensing image associated with the copy image using a transformation matrix algorithm to obtain a coarse registration point sequence; a second determination module, used to determine the mean and variance of the coarse registration sequence of each remote sensing image, to obtain M sequence means and M sequence variances, and to determine the image size range data of each remote sensing image using each sequence mean and each sequence variance to obtain M image size range data; a cropping module, used to crop the M remote sensing images respectively using the M image size range data to obtain M pre-registered images.

[0104] Optionally, in the image registration device provided in the embodiment of the present application, the extraction unit 1004 includes: a first acquisition module, used to obtain a preset zoom factor, and use the preset zoom factor to perform slice processing on each pre-registered image to obtain M groups of slice group sequences, wherein the preset zoom factor includes a first zoom factor and a second zoom factor, and each group of slice group sequences includes a slice group subsequence of the first zoom factor and a slice group subsequence of the second zoom factor, the slice group subsequence of the first zoom factor includes Q1 image slices, and the slice group subsequence of the second zoom factor includes Q2 image slices, where Q1 and Q2 are positive integers; a second extraction module, used to extract a slice group sequence for a group of slices; A group subsequence is formed, and features are extracted from a group of slice group subsequences to obtain a group of slice feature data, wherein a group of slice feature data includes Q1 slice feature sub-data or Q2 slice feature sub-data; a matching module is used to match a group of slice feature data using a graph neural network algorithm to obtain a matching point group, and map the matching point group to a remote sensing image corresponding to the pre-registered image to obtain a group of registration point sequences, wherein when the preset zoom factor is a first zoom factor, the matching point group includes J feature point coordinates, and when the preset zoom factor is a second zoom factor, the matching point group includes L feature point coordinates, J is less than L, and J and L are positive integers.

[0105] Optionally, in the image registration device provided in the embodiment of the present application, the second processing unit 1005 includes: a second acquisition module, used to obtain preset range parameters, and use the preset range parameters to process J feature point coordinates in the registration point sequence of the first zoom factor to obtain J processed feature point ranges; a third screening module, used to use the J processed feature point ranges to screen L feature point coordinates in the registration point sequence of the second zoom factor to obtain N feature point coordinates within the feature point range, and the N feature point coordinates constitute a target registration point sequence.

[0106] Optionally, in the image registration device provided in the embodiment of the present application, the second processing unit 1005 includes: a fifth processing module, used to perform homography transformation processing on the target registration point sequence to obtain a homography matrix, wherein the homography matrix is ​​used to indicate the two-dimensional perspective transformation relationship between M remote sensing images; a transformation module, used to use the homography matrix to perform projective transformation on remote sensing images of a preset proportion among the M remote sensing images to obtain a transformed registration image; and a stitching module, used to stitch the transformed registration image with the remaining remote sensing images among the M remote sensing images to obtain a registration image.

[0107] The above-mentioned image registration device includes a processor and a memory. The above-mentioned first acquisition unit 1001, second acquisition unit 1002, first processing unit 1003, extraction unit 1004, second processing unit 1005, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0108] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the problem of low registration accuracy in image registration of remote sensing images in related technologies can be solved by adjusting kernel parameters.

[0109] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0110] An embodiment of the present invention provides a computer storage medium, which is used to store a program. When the program is running, the device where the computer storage medium is located is controlled to execute an image registration method.

[0111] Fig.11 is a schematic diagram of an electronic device provided according to an embodiment of the present application, such as Fig.11 As shown, an embodiment of the present invention provides an electronic device, the electronic device includes a processor, a memory, and a program stored in the memory and executable on the processor, the processor is used to execute computer-readable instructions, wherein the computer-readable instructions execute an image registration method when executed. The device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0112] The present application also provides a computer program product, including a computer program, which implements the steps of an image registration method in various embodiments of the present application when the computer program is executed by a processor.

[0113] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0114] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0117] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0118] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0119] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0120] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0121] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. An image registration method, characterized in that: include: Acquire M remote sensing images, and perform image scaling processing on the M remote sensing images respectively to obtain M images to be registered, where M is greater than 1 and is a positive integer; Obtain a copy image of each image to be registered, use a feature extraction algorithm to extract features from the M copy images respectively, and output M groups of feature data, wherein the copy image is obtained by downsampling the image to be registered; Performing preliminary registration processing on the M groups of feature data respectively to obtain M initial registration point sequences, and determining M pre-registered images from the M initial registration point sequences; Using the feature extraction algorithm to perform feature extraction on the M pre-registered images respectively, to obtain M groups of registration point sequences, wherein each group of registration point sequences includes a registration point sequence of a first zoom factor and a registration point sequence of a second zoom factor, and the first zoom factor is smaller than the second zoom factor; The M groups of registration point sequences are subjected to multi-scale screening processing to generate a target registration point sequence, and a registration image is determined based on the target registration point sequence.

2. The method according to claim 1, characterized in that The feature extraction algorithm is used to extract features from the M replica images respectively, and the output M groups of feature data include: For each copy image, extract scale features of the copy image using Y feature extraction layers of the fusion model to obtain Y descriptor vectors, and determine a descriptor matrix according to the Y descriptor vectors, wherein the fusion model includes K feature preprocessing layers and Y feature extraction layers arranged in parallel, each feature extraction layer has a different extraction scale, and K and Y are positive integers; Perform confidence calculation on the descriptor matrix and output a confidence heat map, wherein the confidence heat map is used to describe the matching reliability of the Y descriptor vectors; Decomposing the copy image by a feature map to obtain different sub-regions, and performing convolution processing on each sub-region to obtain a feature point coding matrix, wherein the feature point coding matrix is ​​used to indicate position information of feature point coordinates corresponding to the Y descriptor vectors; A set of feature data of the copy image is determined according to the descriptor matrix, the confidence heat map and the feature point encoding matrix, wherein the set of feature data includes feature sub-data of each sub-region, and each feature sub-data includes feature point coordinates and a descriptor vector of each sub-region.

3. The method according to claim 2, characterized in that Determining a descriptor matrix according to the Y descriptor vectors comprises: Filtering the descriptor vector output by the feature extraction layer at a preset position from the Y descriptor vectors to obtain a target descriptor vector; Performing dimension transformation processing on Y-1 descriptor vectors among the Y descriptor vectors according to the spatial dimension information of the target descriptor vector to obtain Y-1 adjustment vectors; Concatenate the Y-1 adjustment vectors and the target descriptor vector to obtain a concatenated vector; The splicing vectors are further fused using a fusion module associated with the fusion model to obtain the descriptor matrix.

4. The method according to claim 1, characterized in that: Performing preliminary registration processing on the M groups of feature data respectively to obtain M initial registration point sequences, and determining M pre-registered images from the M initial registration point sequences includes: Using a mutual proximity matching algorithm to match the feature sub-data in each set of feature data, to obtain M sets of matched feature data, wherein each set of matched feature data includes the feature sub-data matched by the mutual proximity matching algorithm; For a set of matched feature data, a clustering algorithm is used to screen feature point coordinates in the set of matched feature data to obtain P feature point coordinates, where P is a positive integer; Mapping the coordinates of the P feature points to the remote sensing image associated with the copy image using a transformation matrix algorithm to obtain a sequence of coarse registration points; Determine the mean and variance of the rough registration sequence of each remote sensing image to obtain M sequence means and M sequence variances, and use each sequence mean and each sequence variance to determine the image size range data of each remote sensing image to obtain M image size range data; The M remote sensing images are respectively cropped using the M image size range data to obtain the M pre-registered images.

5. The method according to claim 1, characterized in that The feature extraction algorithm is used to extract features from the M pre-registered images respectively, and the obtained M groups of registration point sequences include: Acquire a preset zoom factor, and perform slice processing on each pre-registered image using the preset zoom factor to obtain M groups of slice group sequences, wherein the preset zoom factor includes the first zoom factor and the second zoom factor, and each group of slice group sequences includes a slice group subsequence of the first zoom factor and a slice group subsequence of the second zoom factor, the slice group subsequence of the first zoom factor includes Q1 image slices, and the slice group subsequence of the second zoom factor includes Q2 image slices, where Q1 and Q2 are positive integers; For a group of slice group subsequences, feature extraction is performed on the group of slice group subsequences to obtain a group of slice feature data, wherein the group of slice feature data includes Q1 slice feature sub-data or Q2 slice feature sub-data; A graph neural network algorithm is used to match the set of slice feature data to obtain a matching point group, and the matching point group is mapped to the remote sensing image corresponding to the pre-registered image to obtain a set of registration point sequences, wherein when the preset zoom factor is a first zoom factor, the matching point group includes J feature point coordinates, and when the preset zoom factor is a second zoom factor, the matching point group includes L feature point coordinates, J is less than L, and J and L are positive integers.

6. The method according to claim 1, characterized in that Performing multi-scale screening processing on the M groups of registration point sequences to generate a target registration point sequence includes: Acquire preset range parameters, and use the preset range parameters to process the coordinates of J feature points in the registration point sequence of the first zoom factor to obtain J processed feature point ranges; The L feature point coordinates in the second zoom factor registration point sequence are screened using the J processed feature point ranges to obtain N feature point coordinates within the feature point ranges, and the target registration point sequence is formed by the N feature point coordinates.

7. The method according to claim 1, characterized in that Determining the registration image based on the target registration point sequence includes: Performing homography transformation processing on the target registration point sequence to obtain a homography matrix, wherein the homography matrix is ​​used to indicate a two-dimensional perspective transformation relationship between the M remote sensing images; Performing a projection transformation on a remote sensing image of a preset ratio among the M remote sensing images using the homography matrix to obtain a transformed registration image; The transformed registration image and the remaining remote sensing images in the M remote sensing images are spliced ​​to obtain the registration image.

8. An image registration device, characterized in that: include: A first acquisition unit is used to acquire M remote sensing images, and perform image scaling processing on the M remote sensing images respectively to obtain M images to be registered, wherein M is greater than 1 and is a positive integer; A second acquisition unit is used to acquire a copy image of each image to be registered, perform feature extraction on the M copy images respectively using a feature extraction algorithm, and output M groups of feature data, wherein the copy image is obtained by downsampling the image to be registered; A first processing unit is used to perform preliminary registration processing on the M groups of feature data respectively to obtain M initial registration point sequences, and determine M pre-registered images according to the M initial registration point sequences; an extraction unit, configured to perform feature extraction on the M pre-registered images respectively by using the feature extraction algorithm to obtain M groups of registration point sequences, wherein each group of registration point sequences includes a registration point sequence of a first zoom factor and a registration point sequence of a second zoom factor, and the first zoom factor is smaller than the second zoom factor; The second processing unit is used to perform multi-scale screening processing on the M groups of registration point sequences to generate a target registration point sequence, and determine a registration image based on the target registration point sequence.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image registration method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the image registration method described in any one of claims 1 to 7.

Images