Image registration method, apparatus, computer program product, and electronic device

Abstract

The application discloses a kind of image registration method, device, computer program product and electronic equipment.It relates to remote sensing image processing field or other related fields, the method comprises: obtaining M remote sensing images, respectively image scaling processing is carried out to M remote sensing images, obtain M image to be registered;Get the copy image of each image to be registered, respectively feature extraction is carried out to M copy image using feature extraction algorithm, output M group feature data;M group feature data is respectively carried out preliminary registration processing, obtain M initial registration point sequence, determine M pre-registration image from M initial registration point sequence;Respectively feature extraction is carried out to M pre-registration image, obtain M group registration point sequence;M group registration point sequence is carried out multi-scale screening processing, generate target registration point sequence, determine registration image based on target registration point sequence.The application solves the problem of low registration accuracy when image registration is performed on remote sensing images in related art.

Description

Technical Field

[0001] This application relates to the field of remote sensing image processing or other related fields, and more specifically, to an image registration method, apparatus, computer program product, and electronic device. Background Technology

[0002] With the rapid development of technologies such as satellite remote sensing, aerial photography, and UAV reconnaissance, high-resolution, large-size remote sensing images are widely used in various fields such as urban planning, agricultural research, environmental change analysis, and disaster assessment. These high-precision images provide rich geographic information, greatly enhancing the ability to understand dynamic changes on the Earth's surface. To more accurately understand surface changes, it is necessary to perform multi-source and multi-temporal matching of these remote sensing images. For example, when monitoring long-term changes such as deforestation and land degradation, it is necessary to accurately register large-size remote sensing images acquired at different time points to accurately compare and analyze changes in surface features. In geographic information system applications, in order to build high-resolution maps or update maps, it is necessary to register old and new remote sensing images to ensure accurate correspondence of features such as terrain, roads, and buildings. After a natural disaster, it is necessary to quickly and accurately register pre-disaster and post-disaster remote sensing images.

[0003] Image registration in related technologies, especially the registration of remote sensing images, involves geometrically aligning two or more images acquired at different times, by different sensors, or under different conditions. Key points are identified and described by constructing scale space and Gaussian pyramids to ensure that they correspond accurately in space, thereby achieving image matching. Alternatively, image features can be extracted by using convolutional neural networks, and feature matching can be performed using graph neural networks.

[0004] However, with the continuous advancement of remote sensing technology, the size of acquired images is increasing daily. Traditional methods, whether manual feature-based or deep learning-based, require full-size registration, which is computationally intensive, time-consuming, and resource-intensive. While deep learning-based registration methods perform well on small to medium-sized images, direct downsampling significantly reduces registration accuracy when processing high-resolution remote sensing images with tens of thousands of pixels, especially in areas rich in image detail. Furthermore, the size of these images far exceeds the input limits of conventional deep learning networks, which may lead to memory overflow or insufficient GPU memory in the registration of large images. This results in a sharp increase in computational complexity, GPU memory consumption, and time consumption, limiting registration accuracy and thus restricting the application of registration technology in large-size remote sensing image scenarios.

[0005] There is currently no effective solution to the problem of low registration accuracy when performing image registration on remote sensing images in related technologies. Summary of the Invention

[0006] The main objective of this application is to provide an image registration method, apparatus, computer program product, and electronic device to solve the problem of low registration accuracy when performing image registration on remote sensing images in related technologies.

[0007] To achieve the above objectives, according to one aspect of this application, an image registration method is provided. The method includes: acquiring M remote sensing images; performing image scaling on each of the M remote sensing images to obtain M images to be registered, where M is greater than 1 and M is a positive integer; acquiring a copy image of each image to be registered; performing feature extraction on the M copy images using a feature extraction algorithm to output M sets of feature data, wherein the copy images are obtained by downsampling the images to be registered; performing preliminary registration processing on the M sets of feature data to obtain M initial registration point sequences; determining M pre-registered images from the M initial registration point sequences; performing feature extraction on the M pre-registered images using a feature extraction algorithm to obtain M sets of registration point sequences, wherein each set of registration point sequences includes a registration point sequence with a first scaling factor and a registration point sequence with a second scaling factor, the first scaling factor being less than the second scaling factor; performing multi-scale filtering processing on the M sets of registration point sequences to generate a target registration point sequence, and determining the registered images based on the target registration point sequence.

[0008] Furthermore, feature extraction algorithms are used to extract features from the M replica images, outputting M sets of feature data, including: for each replica image, scale features are extracted using Y feature extraction layers of the fusion model to obtain Y descriptor vectors, and a descriptor matrix is ​​determined based on the Y descriptor vectors. The fusion model includes K feature preprocessing layers and Y parallel feature extraction layers, each with a different extraction scale, where K and Y are positive integers. Confidence is calculated on the descriptor matrix, outputting a confidence heatmap, which describes the matching reliability of the Y descriptor vectors. Feature map decomposition is performed on the replica images to obtain different sub-regions, and convolution is performed on each sub-region to obtain a feature point encoding matrix, which indicates the positional information of the feature point coordinates corresponding to the Y descriptor vectors. A set of feature data for the replica images is determined based on the descriptor matrix, the confidence heatmap, and the feature point encoding matrix. This set of feature data includes feature sub-data for each sub-region, and each feature sub-data includes the feature point coordinates and descriptor vectors for each sub-region.

[0009] Further, determining the descriptor matrix based on the Y descriptor vectors includes: selecting descriptor vectors output by the feature extraction layer at preset positions from the Y descriptor vectors to obtain target descriptor vectors; performing dimensional transformation on Y-1 descriptor vectors from the Y descriptor vectors based on the spatial dimension information of the target descriptor vectors to obtain Y-1 adjustment vectors; concatenating the Y-1 adjustment vectors and the target descriptor vectors to obtain concatenated vectors; and further fusing the concatenated vectors using the fusion module associated with the fusion model to obtain the descriptor matrix.

[0010] Further, preliminary registration processing is performed on the M sets of feature data to obtain M initial registration point sequences. The determination of M pre-registered images from these M initial registration point sequences includes: matching the feature sub-data in each set of feature data using a mutual proximity matching algorithm to obtain M sets of matched feature data, where each set of matched feature data includes feature sub-data processed by the mutual proximity matching algorithm; for each set of matched feature data, a clustering algorithm is used to filter the feature point coordinates to obtain P feature point coordinates, where P is a positive integer; the transformation matrix algorithm is used to map the P feature point coordinates to the remote sensing image associated with the replica image to obtain an initial registration point sequence; the mean and variance of the initial registration point sequence for each remote sensing image are determined to obtain M sequence means and M sequence variances; the image size range data for each remote sensing image is determined using each sequence mean and each sequence variance to obtain M image size range data; and the M remote sensing images are cropped using the M image size range data to obtain M pre-registered images.

[0011] Furthermore, feature extraction algorithms are used to extract features from the M pre-registered images to obtain M sets of registration point sequences. This includes: obtaining a preset scaling factor; segmenting each pre-registered image using the preset scaling factor to obtain M sets of slice sequences. The preset scaling factor includes a first scaling factor and a second scaling factor. Each slice sequence includes a slice subsequence with the first scaling factor and a slice subsequence with the second scaling factor. The slice subsequence with the first scaling factor includes Q1 image slices, and the slice subsequence with the second scaling factor includes Q2 image slices, where Q1 and Q2 are positive integers. For a set of slice sequences... Subsequence: Feature extraction is performed on a set of subsequences of slice groups to obtain a set of slice feature data, wherein a set of slice feature data includes Q1 slice feature sub-data or Q2 slice feature sub-data. A graph neural network algorithm is used to match the set of slice feature data to obtain a set of matching points, and the matching point set is mapped to the remote sensing image corresponding to the pre-registered image to obtain a set of registration point sequences. Wherein, when the preset scaling factor is the first scaling factor, the matching point set includes J feature point coordinates, and when the preset scaling factor is the second scaling factor, the matching point set includes L feature point coordinates, where J is less than L, and J and L are positive integers.

[0012] Furthermore, the multi-scale filtering process for the M sets of registration point sequences to generate the target registration point sequence includes: obtaining a preset range parameter; using the preset range parameter to process the coordinates of J feature points in the registration point sequence at the first scaling factor to obtain J processed feature point ranges; using the J processed feature point ranges to filter the coordinates of L feature points in the registration point sequence at the second scaling factor to obtain N feature point coordinates located within the feature point ranges; and using the N processed feature point ranges to filter the coordinates of L feature points in the registration point sequence at the second scaling factor to obtain N feature point coordinates located within the feature point ranges; and using the N feature point coordinates to form the target registration point sequence.

[0013] Further, determining the registered image based on the target registration point sequence includes: performing homography transformation on the target registration point sequence to obtain a homography matrix, wherein the homography matrix is ​​used to indicate the two-dimensional perspective transformation relationship between the M remote sensing images; using the homography matrix to perform projection transformation on the remote sensing images of a preset scale among the M remote sensing images to obtain a transformed registered image; and stitching the transformed registered image and the remaining remote sensing images among the M remote sensing images to obtain the registered image.

[0014] To achieve the above objectives, according to another aspect of this application, an image registration apparatus is provided. The device includes: a first acquisition unit, configured to acquire M remote sensing images, perform image scaling on the M remote sensing images respectively to obtain M images to be registered, wherein M is greater than 1 and M is a positive integer; a second acquisition unit, configured to acquire a copy image of each image to be registered, perform feature extraction on the M copy images respectively using a feature extraction algorithm, and output M sets of feature data, wherein the copy images are obtained by downsampling the images to be registered; a first processing unit, configured to perform preliminary registration processing on the M sets of feature data respectively to obtain M initial registration point sequences, and determine M pre-registered images from the M initial registration point sequences; an extraction unit, configured to perform feature extraction on the M pre-registered images respectively using a feature extraction algorithm to obtain M sets of registration point sequences, wherein each set of registration point sequences includes a registration point sequence with a first scaling factor and a registration point sequence with a second scaling factor, wherein the first scaling factor is less than the second scaling factor; and a second processing unit, configured to perform multi-scale filtering processing on the M sets of registration point sequences to generate a target registration point sequence, and determine the registered images based on the target registration point sequence.

[0015] According to another aspect of the present invention, a computer storage medium is also provided for storing a program, wherein the program, when running, controls the device where the computer storage medium is located to perform an image registration method.

[0016] According to another aspect of the present invention, an electronic device is also provided, comprising one or more processors and a memory; the memory stores computer-readable instructions, and the processor is configured to execute the computer-readable instructions, wherein the computer-readable instructions execute an image registration method.

[0017] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, performs an image registration method.

[0018] This application employs the following steps: acquiring M remote sensing images; performing image scaling on each of the M remote sensing images to obtain M images to be registered, where M is greater than 1 and is a positive integer; acquiring a copy image of each image to be registered; performing feature extraction on the M copy images using a feature extraction algorithm to output M sets of feature data, where the copy images are obtained by downsampling the images to be registered; performing preliminary registration processing on the M sets of feature data to obtain M initial registration point sequences; determining M pre-registered images from the M initial registration point sequences; and performing feature extraction on the M pre-registered images using a feature extraction algorithm to obtain M sets of registration point sequences, where each set of registration point sequences includes registration at a first scaling factor. The system employs a first scaling factor (SGF) less than the second scaling factor (SGF) to generate a target registration point sequence. This process involves multi-scale filtering of M registration point sequences to determine the registration image. This addresses the low registration accuracy issue in related technologies for remote sensing image registration. By utilizing feature extraction algorithms to extract features from a copy of the remote sensing image and performing preliminary registration processing on the extracted feature data, the system obtains the corresponding initial registration point sequence. This allows for coarse localization of common regions at low resolution and fine feature point extraction and matching at higher resolution, thereby improving image registration accuracy, reducing registration failure rate, and enhancing registration robustness. Attached Figure Description

[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a flowchart of an image registration method provided according to an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of a processing method for a feature extraction algorithm provided in an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of a method for filtering feature point coordinates according to an embodiment of this application;

[0023] Figure 4 The registration result is based on the image registration method provided in the embodiments of this application. Figure 1 ;

[0024] Figure 5 The registration result is based on the image registration method provided in the embodiments of this application. Figure 2 ;

[0025] Figure 6This is a schematic diagram of an optional image registration method provided according to an embodiment of this application;

[0026] Figure 7 This is the registration effect of remote sensing images provided in the embodiments of this application. Figure 1 ;

[0027] Figure 8 This is the registration effect of remote sensing images provided in the embodiments of this application. Figure 2 ;

[0028] Figure 9 This is the registration effect of remote sensing images provided in the embodiments of this application. Figure 3 ;

[0029] Figure 10 This is a schematic diagram of an image registration apparatus according to an embodiment of this application;

[0030] Figure 11 This is a schematic diagram of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0031] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties. For example, this system has an interface with the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving consent from the aforementioned user or organization.

[0035] It should be noted that the information collected in this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize use or refuse use.

[0036] The present invention will now be described in conjunction with preferred implementation steps. Figure 1 This is a flowchart of an image registration method provided according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0037] Step S101: Obtain M remote sensing images, perform image scaling on each of the M remote sensing images to obtain M images to be registered, where M is greater than 1 and M is a positive integer.

[0038] Specifically, to match two or more images acquired at different times, with different imaging devices, or under different conditions (weather, illumination, camera position and angle, etc.), and then perform remote sensing analysis based on the matched images, two or more remote sensing images of different resolutions can first be acquired from different sensors or at different times. For example, remote sensing images can be acquired from multiple platforms such as satellites, drones, and aerial photography, and the imaging methods can be optical imaging, synthetic aperture radar imaging, multispectral imaging, etc. Due to differences in remote sensing data acquisition conditions and equipment, the resolution, coverage, imaging angle, and time of these images may vary. However, their common parts need to be accurately registered. At this time, the original remote sensing images need to be scaled to adapt to different analysis needs and computational limitations. This can be done by cropping the long side and padding the short side of the two images, while scaling to make the image length and width consistent. Finally, each image is adjusted to a specific resolution, thus obtaining the corresponding image to be registered to meet the requirements of subsequent processing.

[0039] It should be noted that image scaling can also reduce image size and processing speed by downsampling or upsampling when computing resources are limited or when processing large images. It can also be upsampling to improve resolution or restore details after feature extraction on lower resolution images. In order to maintain image quality, algorithms such as bilinear interpolation and bicubic interpolation can be used to avoid distortion or moiré effects during scaling.

[0040] Furthermore, due to differences between images, when the aspect ratio of an image is not 1:1, in order to avoid information loss caused by non-uniform scaling, the short side of the image needs to be expanded with an empty background until the image is 1:1. For SAR (Synthetic Aperture Radar) images, it is necessary to expand the image to 3 channels by copying the image. For cases where the channel order of full spectrum or other images is mismatched, it is necessary to modify the channel arrangement order (such as from "RGB" to "BGR", that is, from "Red, Green and Blue" to "Blue, Green and Red").

[0041] Step S102: Obtain a copy image of each image to be registered, and use a feature extraction algorithm to extract features from the M copy images respectively, and output M sets of feature data. The copy images are obtained by downsampling the images to be registered.

[0042] Specifically, since the original remote sensing images are large in size, especially when GPU (Graphics Processing Unit) memory is limited, directly using them as input to deep learning models may lead to excessive consumption of computing resources. Therefore, for each high-resolution remote sensing image to be registered, a copy image needs to be created and downsampled to reduce the image size, which can significantly reduce computational complexity and make the feature extraction process more efficient.

[0043] After obtaining the downsampled copy images, feature extraction algorithms can be used to extract features from these copy images, identify key points or regions in the images, and obtain multiple sets of feature data. The feature extraction algorithm can be the XFeat deep learning algorithm. The identified key points contain rich geographic information and texture features. Therefore, each set of feature data contains the coordinate information of a series of key points and their descriptor vectors. The descriptors are used to quantify the features of these key points to facilitate subsequent matching and registration.

[0044] Step S103: Perform preliminary registration processing on the M sets of feature data to obtain M initial registration point sequences, and determine M pre-registered images from the M initial registration point sequences.

[0045] Specifically, after obtaining a set of feature data for each remote sensing image, preliminary registration can be performed on the feature data acquired at each resolution or from different sensors to establish the correspondence between key points in different images. Since the data in each set of feature data contains the coordinates of feature points extracted from downsampled copy images and the corresponding descriptor vectors, these descriptor vectors are used to quantify the features of feature points and can reflect information such as the position, texture, and shape of key points in the image. At this point, by comparing the descriptor vectors in different sets of feature data, matching pairs of feature points in each set of images can be found, thereby obtaining the corresponding initial registration point sequence.

[0046] It should be noted that when performing matching, the nearest neighbor (NN) method, mutual nearest neighbor (MNN) matching, or deep learning models can be used to perform matching. The correspondence of feature points is determined based on the similarity of descriptor vectors. By calculating the distance between descriptor vectors, the "most similar" feature point pair in each pair of images is found, thereby generating a preliminary registration point sequence.

[0047] Furthermore, based on the obtained initial registration point sequence, the corresponding pre-registration image can be determined, that is, the initial registration point sequence is mapped back to the original resolution and size of the image to be registered. By estimating the transformation matrix and other methods, the images to be registered are geometrically adjusted so that they are initially aligned in space. This not only eliminates coarse displacement, rotation or scale differences between images, but also provides a better starting point for subsequent fine registration.

[0048] Step S104: Use a feature extraction algorithm to extract features from the M pre-registered images to obtain M sets of registration point sequences. Each set of registration point sequences includes a registration point sequence with a first scaling factor and a registration point sequence with a second scaling factor, where the first scaling factor is less than the second scaling factor.

[0049] Specifically, after obtaining the corresponding pre-registered image, a feature extraction algorithm can be used to extract features from the pre-registered image at multiple levels, thereby generating registration point sequences with different scaling factors to ensure accurate registration of large-size images. First, a strategy of different scaling factors can be adopted, and feature extraction algorithms can be used to extract features from the pre-registered image. The resulting registration point sequences can include a first scaling factor sequence typically generated at a lower resolution and a second scaling factor sequence generated at a relatively higher resolution.

[0050] At this point, based on the registration point sequence at the first scaling factor, the common overlapping areas and coarse registration results of the image can be determined. Then, using the registration results from the first scaling factor, the registration point sequence at the second scaling factor is filtered to limit the search range or guide the matching process, avoiding full-image searches at high resolution, thereby improving matching efficiency and accuracy. Finally, a fine-grained matching is performed on the registration point sequence at the second scaling factor. Using an algorithm such as LightGlue, combined with the results from the first scaling factor, more accurate registration point matching and outlier removal are performed to ensure high accuracy and reliability of the final registration points.

[0051] The registration point sequences under different scaling factors are then fused and unified into the coordinate system of the original image through geometric transformation and position mapping to generate the final registration point sequence.

[0052] Step S105: Perform multi-scale screening on the M groups of registration point sequences to generate target registration point sequences, and determine the registration image based on the target registration point sequences.

[0053] Each registration point sequence contains registration information at different scales, ranging from large-scale features at low resolution to fine features at high resolution. Since the initial registration point sequence may contain some abnormal registration points due to image noise, occlusion, texture duplication, or algorithm errors, these points negatively impact the final registration result. Therefore, it is necessary to filter the registration point sequence to generate a target registration point sequence that not only contains the most reliable registration point pairs but also reflects the comprehensive information between features at different scales.

[0054] After generating the target registration point sequence, the homography matrix of the registration point sequence can be estimated, and image projection registration can be performed based on the homography matrix to ensure their consistency in geospatial space, thereby obtaining the registered image.

[0055] The image registration method provided in this application involves acquiring M remote sensing images, scaling each of the M images to obtain M images to be registered, where M is greater than 1 and is a positive integer; acquiring a copy image of each image to be registered, and using a feature extraction algorithm to extract features from the M copy images to output M sets of feature data, where the copy images are obtained by downsampling the images to be registered; performing preliminary registration processing on the M sets of feature data to obtain M initial registration point sequences, and determining M pre-registered images from the M initial registration point sequences; and using a feature extraction algorithm to extract features from the M pre-registered images to obtain M sets of registration point sequences, where each set of registration point sequences includes a first scaling factor. The system employs a registration point sequence and a registration point sequence with a second scaling factor, where the first scaling factor is smaller than the second scaling factor. Multi-scale filtering is performed on the M sets of registration point sequences to generate a target registration point sequence. Based on this target registration point sequence, the registered image is determined. This solves the problem of low registration accuracy in related technologies for remote sensing image registration. By utilizing feature extraction algorithms to extract features from a copy of the remote sensing image, and performing preliminary registration processing on the extracted feature data, the corresponding initial registration point sequence is obtained. Common regions are roughly located at low resolution, and fine feature point extraction and matching are performed at higher resolution, thereby improving the accuracy of image registration, reducing the registration failure rate, and enhancing the robustness of registration.

[0056] Figure 2 This is a schematic diagram of the processing method of the feature extraction algorithm provided in the embodiments of this application, such as... Figure 2 As shown, optionally, in the image registration method provided in this application embodiment, the feature extraction algorithm performs feature extraction on M replica images respectively, outputting M sets of feature data, including: for each replica image, using Y feature extraction layers of the fusion model to extract scale features of the replica image, obtaining Y descriptor vectors, and determining a descriptor matrix based on the Y descriptor vectors, wherein the fusion model includes K feature preprocessing layers and Y parallel feature extraction layers, each feature extraction layer having a different extraction scale, and K and Y being positive integers; calculating the confidence level of the descriptor matrix, and outputting the confidence level... A confidence heatmap is generated, which describes the matching reliability of the Y descriptor vectors. The replica image is decomposed into feature maps to obtain different sub-regions, and each sub-region is convolved to obtain a feature point encoding matrix, which indicates the position information of the feature point coordinates corresponding to the Y descriptor vectors. Based on the descriptor matrix, the confidence heatmap, and the feature point encoding matrix, a set of feature data for the replica image is determined. The set of feature data includes feature sub-data for each sub-region, and each feature sub-data includes the feature point coordinates and descriptor vectors of each sub-region.

[0057] It should be noted that the feature extraction algorithm can be the XFeat deep learning algorithm, and the fusion model in this algorithm can adopt a pyramid structure, including multiple feature preprocessing layers and multiple parallel feature extraction layers. The feature preprocessing layers can perform preliminary feature extraction and enhancement on the input image, providing a good foundation for subsequent multi-scale analysis. The three parallel feature extraction layers utilize features at different scales, allowing each feature extraction layer to independently focus on a specific scale range, thereby capturing more comprehensive information in the image. To obtain image features in each remote sensing image, the multi-layer structure of the fusion model can be used to perform multi-scale feature extraction on each copy of the remote sensing image, thereby capturing multi-level, different-scale information in the image and providing rich data support for accurate feature point matching and location determination.

[0058] When performing feature extraction on a copy image, after inputting the image, it enters the feature point extraction structure and the descriptor extraction structure respectively. In the descriptor extraction structure, scale features are first extracted using the feature extraction layer, resulting in multiple descriptor vectors. When the feature preprocessing layer has two layers and the parallel feature extraction layer has three layers, the descriptor vectors output by the feature extraction layer are represented as follows: At this point, the dimensions of the vectors output by the three feature extraction layers are respectively , , Then, the three descriptor vectors are fused to generate a descriptor matrix capable of feature point matching and location mapping, with a size of [missing information]. Furthermore, the confidence of the obtained descriptor matrix is ​​calculated using a confidence block, and then a confidence heatmap is output to characterize the matching reliability of the descriptor position, thereby helping the model to filter out key points that are more likely to be successfully matched, improving the accuracy and efficiency of registration.

[0059] While calculating the descriptor vector, feature point extraction can also be performed. First, the output copy image can be unfolded, that is, the image is decomposed into multiple 8×8 sub-regions and flattened to obtain a feature map with c*64 channels. (The original number of channels in the image), and then convolution processing is performed on each sub-region to obtain the feature point encoding matrix. That is, inputting several convolutional layers, extracting features with... The feature point encoding matrix is ​​used, where "64" represents the possible positions in each grid cell, and "+1" indicates a "trash can" category, representing the case where no keypoint was found in that cell. This matrix not only contains the descriptor vector for each sub-region but also indicates the position information of the feature points corresponding to these descriptor vectors in the image coordinate system. Finally, the descriptor matrix, confidence heatmap, and feature point encoding matrix output by the two structures can be used to determine a set of feature data for a replica image. This embodiment captures the macroscopic structure and microscopic details in the image through multi-scale feature extraction and decomposes the image to obtain a feature point encoding matrix containing feature point coordinate information. This provides accurate guidance for registration mapping, reduces the impact of mismatches and outliers, and significantly reduces computational resource consumption, thereby improving the accuracy of the registration results.

[0060] Optionally, in the image registration method provided in this application embodiment, determining the descriptor matrix based on Y descriptor vectors includes: selecting descriptor vectors output by the feature extraction layer at a preset position from the Y descriptor vectors to obtain target descriptor vectors; performing dimensional transformation processing on Y-1 descriptor vectors among the Y descriptor vectors based on the spatial dimension information of the target descriptor vectors to obtain Y-1 adjustment vectors; concatenating the Y-1 adjustment vectors and the target descriptor vectors to obtain concatenated vectors; and further fusing the concatenated vectors using a fusion module associated with a fusion model to obtain the descriptor matrix.

[0061] When constructing a matrix using descriptor vectors, the descriptor vectors output by the feature extraction layers at preset positions can be obtained first. For example, after a copy image passes through two feature preprocessing layers and is input into three parallel feature extraction layers, the target descriptor vector output by the third feature extraction layer can be obtained, and the spatial dimension information of this vector can be determined. Then, adjustments can be made to the spatial dimension (e.g., through interpolation) and the feature dimension (e.g., through convolution or fully connected layers). The vectors output by the layer undergo dimensionality transformation to ensure that all vectors have the same spatial dimensional information, thereby obtaining the corresponding adjusted vectors.

[0062] Furthermore, to enable the fusion of descriptor vectors at different scales within a standardized framework, thereby avoiding information loss or mismatch due to scale differences, the aforementioned adjustment vector and the target descriptor vector can be merged and concatenated. The concatenated vector is then further fused using the fusion head block (i.e., the fusion module) associated with the fusion model, ultimately generating a descriptor matrix. This embodiment, by filtering and fusing descriptor vectors at different scales, effectively utilizes multi-layered image information, eliminates the impact of scale differences, improves the robustness and accuracy of registration, overcomes the differences in image features under different sensors or imaging conditions, and enhances the cross-data source capability of registration matching. Moreover, compared to directly processing the original high-resolution image, performing preliminary feature extraction and registration matching based on a smaller replica image significantly reduces computational resource consumption.

[0063] Optionally, in the image registration method provided in this application embodiment, preliminary registration processing is performed on M sets of feature data to obtain M initial registration point sequences. Determining M pre-registered images from the M initial registration point sequences includes: matching feature sub-data in each set of feature data using a mutual proximity matching algorithm to obtain M sets of matched feature data, wherein each set of matched feature data includes feature sub-data that has been matched by the mutual proximity matching algorithm; for a set of matched feature data, a clustering algorithm is used to analyze the features in the set of matched feature data. The coordinates of the points are filtered to obtain P feature point coordinates, where P is a positive integer. The transformation matrix algorithm is used to map the P feature point coordinates to the remote sensing image associated with the replica image to obtain the initial registration point sequence. The mean and variance of the initial registration point sequence of each remote sensing image are determined to obtain M sequence means and M sequence variances. The image size range data of each remote sensing image is determined using the mean and variance of each sequence to obtain M image size range data. The M remote sensing images are cropped using the M image size range data to obtain M pre-registered images.

[0064] Specifically, the mutual proximity matching algorithm can find the nearest neighbor pairs of feature points by calculating the distance between feature point descriptors. For example, when there are two remote sensing images (i.e., image A and image B), in order to determine the common coverage area of ​​the two images, after obtaining the feature data corresponding to the two remote sensing images A and B, the mutual proximity matching algorithm can be used to match the feature sub-data in the two sets of feature data. For each feature point extracted from image A, the distance between its descriptor vector and the descriptor vectors of all feature points in the other image is calculated, and then the nearest feature point in image B is determined based on the distance. At the same time, for each feature point extracted from image B, the nearest feature point in image A is found, that is, the nearest neighbor feature point of the aforementioned nearest feature point in the other image is determined. If the first nearest neighbor of a feature point is exactly the feature point itself in the other image, then this pair of feature points is considered to be the nearest neighbors. That is, only when the feature points in image A and the feature points in image B are the nearest neighbors are they considered to be a matched pair. After determining whether a certain feature sub-data in group A can be matched in group B, all the feature sub-data matched in groups A and B can be recombined to obtain two sets of matched feature data and quickly determine the preliminary correspondence.

[0065] Furthermore, since the matched feature data may contain some outliers or mismatched feature points, which will negatively affect the subsequent registration results, clustering algorithms can be used to filter the feature point coordinates in a set of matched feature data, thereby removing these outliers and retaining only those feature points that are relatively clustered in spatial location. The clustering algorithm can be K-means (K-Means Clustering Algorithm) or DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm, which can group the feature point coordinates into several clusters according to the spatial distribution of the feature point coordinates, thereby obtaining multiple final retained feature point coordinates.

[0066] Furthermore, a transformation matrix algorithm can be used to map the selected feature point coordinates back to the original remote sensing image, enabling more accurate registration at the original resolution. This yields an initial registration point sequence, providing preliminary corresponding point information for subsequent fine registration. It should be noted that since there is no displacement between the input copy image and the remote sensing image, only scaling, for example, the obtained feature point coordinates are... The coordinates projected onto the original remote sensing image are then... The transformation matrix and calculation method are as follows:

[0067] ;

[0068] in, and This represents the preset parameters. After projecting the coordinates of all feature points, the initial registration point sequences of the two obtained remote sensing images at the original size can be characterized as follows: and ,in, Representative image The first obtained from One feature point, Representative image The first obtained from There are 10 feature points. Since the two images have already been matched, the number of feature points in the initial registration point sequence is now the same.

[0069] Furthermore, to determine the common overlapping region between two images, the coordinate range can be determined using the mean and variance of the initial registration point sequence of each remote sensing image; that is, by calculating... and The relatively stable coordinate range covered by all feature points is considered, and the mean of the coordinates of all feature points can then be calculated. With variance The image size range data R is obtained: This allows us to determine the image range of two remote sensing images based on the image size range data. .

[0070] Finally, based on the common overlapping area between the two, that is, the determined size range data... The two remote sensing images are cropped separately to generate corresponding pre-registration images. , These images are relatively aligned in size and spatial location, providing a more accurate foundation for subsequent fine registration and multi-scale analysis. This embodiment utilizes a mutual proximity matching algorithm and a clustering algorithm to determine the initial registration point sequence, thereby significantly reducing computational load, increasing registration speed, and ensuring that the feature points used for registration have high confidence and spatial consistency, thus improving registration accuracy and achieving accurate image alignment.

[0071] To reduce computational resource consumption and improve registration efficiency, optionally, in the image registration method provided in this application embodiment, feature extraction is performed on M pre-registered images to obtain M sets of registration point sequences, including: obtaining a preset scaling factor; performing slice processing on each pre-registered image using the preset scaling factor to obtain M sets of slice group sequences, wherein the preset scaling factor includes a first scaling factor and a second scaling factor; each slice group sequence includes a slice group subsequence of the first scaling factor and a slice group subsequence of the second scaling factor; the slice group subsequence of the first scaling factor includes Q1 image slices; the slice group subsequence of the second scaling factor includes Q2 image slices; Q1 and Q2 are positive integers; for a slice group subsequence, feature extraction is performed on the slice group subsequence to obtain a set of slice feature data, wherein the set of slice feature data includes Q1 slice feature sub-data or Q2 slice feature sub-data; A graph neural network algorithm is used to match a set of slice feature data to obtain a set of matching points. The matching point set is then mapped to the remote sensing image corresponding to the pre-registered image to obtain a set of registration point sequences. When the preset scaling factor is the first scaling factor, the matching point set includes J feature point coordinates. When the preset scaling factor is the second scaling factor, the matching point set includes L feature point coordinates. J is less than L, and J and L are positive integers.

[0072] Specifically, after obtaining the pre-registered image after cropping, the location of high-resolution feature points can be guided by feature points at low resolution. This combines the advantages of a more complete overall structure in the lower-resolution slices with the finer details in the higher-resolution slices, improving registration accuracy. When the first and second scaling factors are 2x and 8x respectively, the lower resolution focuses on quickly locating approximate registration points between the two images, typically covering a large portion of the image, but the detail of the feature points may be insufficient. The higher resolution captures more detailed information, resulting in more precise feature point location. Furthermore, since this sequence contains a larger number of feature points, the registration results of the 2x coarse registration point group sequence can guide the selection of registration points in the 8x fine registration point group sequence. By using the feature point information at low resolution to determine a reasonable search range, and then limiting the matching results on the high-resolution image slices according to this range, more accurate feature points are selected. This "coarse-to-fine" strategy effectively combines the advantages of high and low resolution images, achieving efficient and high-precision registration of large-size remote sensing images.

[0073] For example, when the pre-registered image is , First, the two images can be divided into grids, that is, a preset scaling factor can be set. Pre-registered images , The length and width are unified after preprocessing. The size of each image slice obtained after segmenting the two images is ( That is, it contains To balance accuracy and low anomaly rate for image slices, the first scaling factor in the preset scaling factor can be set to... Set the second scaling factor to At this point, the slice subsequences obtained at the second scaling factor and the slice subsequences at the first scaling factor are obtained, which constitute the pre-registered image. , Between the 8x fine slice group sequence and the 2x coarse slice group sequence, where the slice group subsequence of the second scaling factor is represented as... The subsequence of slices at the first scaling factor is represented as: ,in, This indicates that the pre-registered image has been scaled up by 2 times. , The set of feature point coordinates extracted and registered. This indicates that the pre-registered image has been scaled up by 8 times. , The set of feature point coordinates extracted and registered is divided into two subsequences. The first scaling factor slice group subsequence covers a larger image area but has a lower resolution and is mainly used for quickly locating overlapping areas of the image and preliminary feature matching. The second scaling factor slice group subsequence provides higher resolution in a smaller area and can capture image details and register more accurately.

[0074] Furthermore, after obtaining the above-mentioned slice group subsequences, the XFeat algorithm can be used to extract features from these sequences: first, based on the coarse slice group sequence... Registration is performed, which involves using the XFeat algorithm to slice the image. The process involves processing the data to output a set of slice feature data, which includes the coordinates of feature points. With the corresponding descriptor vector Based on the above method, for the fine slice group sequence Image slicing was performed using the XFeat algorithm. The processing yields a set of slice feature data, with the feature point coordinates and corresponding descriptor vectors represented as follows: .

[0075] Furthermore, graph neural network algorithms (such as the Light Glue matching algorithm) are used to match a set of slice feature data, that is, based on the coarse slice group sequence. feature point coordinates With the corresponding descriptor vector Perform matching to obtain a set of matching points. Based on the sequence of fine slices feature point coordinates With the corresponding descriptor vector Perform matching to obtain a set of matching points. .

[0076] Based on the relative positions of the image slices with respect to the original remote sensing image, the coordinates of these feature points are mapped back to the coordinate system corresponding to the remote sensing image, thus obtaining a 2x registration point sequence. Since the image slices are both displaced and scaled relative to the original remote sensing image, it is assumed that the image slices... The offset relative to the top left corner of the remote sensing image is: Scaling factor is The transformation matrix projected onto the remote sensing image and the calculation method are as follows:

[0077] ;

[0078] in, as well as Indicates preset parameters. , ) represents the coordinates of the i-th feature point projected onto the remote sensing image. , This represents the coordinates of the i-th feature point in the matching point group. The above formula yields a 2x registration point sequence based on the remote sensing image. Similarly, following the above method—that is, based on the relative positional relationship of the image slices with respect to the original remote sensing image—the coordinates of these feature points are mapped back to the coordinate system corresponding to the remote sensing image, thereby obtaining an 8x registration point sequence. This embodiment introduces a multi-scale segmentation strategy to transform feature point positions from the slice scale to the original image scale, providing accurate reference points for subsequent image registration. It enables hierarchical feature extraction and matching at different resolutions, avoiding the huge computational load and memory consumption caused by directly processing full-resolution images, and improving the overall efficiency of the registration process.

[0079] Optionally, in the image registration method provided in this application embodiment, performing multi-scale filtering processing on M sets of registration point sequences to generate a target registration point sequence includes: obtaining a preset range parameter; using the preset range parameter to process the coordinates of J feature points in the registration point sequence of the first scaling factor to obtain J processed feature point ranges; using the J processed feature point ranges to filter the coordinates of L feature points in the registration point sequence of the second scaling factor to obtain N feature point coordinates located within the feature point ranges; and using the N processed feature point ranges to filter the coordinates of L feature points in the registration point sequence of the second scaling factor to obtain N feature point coordinates located within the feature point ranges; and using the N feature point coordinates to constitute the target registration point sequence.

[0080] Specifically, the registration point sequence obtained after registration at different sizes and registration point sequence After that, a screening process based on multi-scale registration points can be carried out. Figure 3 This is a schematic diagram of a feature point coordinate filtering method provided in an embodiment of this application, as shown below. Figure 3 As shown, it can first be based on the registration point sequence. Construct a filtering range: Assume the coordinates of the feature points in the sequence are... After obtaining the preset range parameter c, the processed feature point range of all feature point coordinates can be determined, that is, a circular range of size c×c is obtained. By performing a union calculation on the coverage areas corresponding to these processed feature point ranges, we can obtain: .

[0081] Furthermore, the aforementioned range can be used to guide feature point searches on high-resolution images, ensuring that high-precision matching focuses on the most likely common areas and avoiding blind and extensive searches on large-size remote sensing images. That is, the processed feature point range is used to filter the feature point coordinates in the registration point sequence at the second scaling factor, retaining only those feature points located within the processed feature point range, and using the retained feature point coordinates to form the target registration point sequence. This embodiment combines common region estimation from low-resolution images with precise feature matching from high-resolution images to achieve higher-precision registration results. This strategy ensures high-quality registration points. Matching from low-resolution images can quickly identify the main overlapping regions of the images, while matching from high-resolution images can find more accurate correspondences within these regions. At the same time, it significantly reduces computational load and improves registration efficiency.

[0082] Optionally, in the image registration method provided in this application embodiment, determining the registered image based on the target registration point sequence includes: performing homography transformation processing on the target registration point sequence to obtain a homography matrix, wherein the homography matrix is ​​used to indicate the two-dimensional perspective transformation relationship between M remote sensing images; using the homography matrix to perform projection transformation on the remote sensing images of a preset ratio among the M remote sensing images to obtain a transformed registered image; and stitching the transformed registered image and the remaining remote sensing images among the M remote sensing images to obtain a registered image.

[0083] Specifically, after obtaining the target registration point sequence Homography transformation can be performed on this sequence. This involves finding the correspondence between common points in the two images and calculating a 3x3 homography matrix H. This matrix contains all the necessary information to map the coordinates of one image onto the other. After calculating the homography matrix, it can be used to perform a projection transformation on one or more remote sensing images using lower resolution or those that play a dominant role. This involves transforming the coordinates of each pixel in these images to the coordinate system of the other image. Figure 4 The registration result is based on the image registration method provided in the embodiments of this application. Figure 1 , Figure 5 The registration result is based on the image registration method provided in the embodiments of this application. Figure 2 ,like Figure 4 , Figure 5 As shown, after projection transformation, each pixel of the target image can be mapped to a new location, but blank areas or pixel overlaps may occur. To generate the final transformed and registered image, pixel values ​​can be filled using methods such as nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation.

[0084] Furthermore, the transformed and registered image is stitched together with the remaining images that have not undergone projection transformation to obtain a complete registered image. It should be noted that if there is overlap between multiple images, it is necessary to determine how to merge these areas to avoid pixel duplication or conflict; if the size of the transformed image exceeds the range of the original image, blank areas may appear at the boundaries. In this case, it is necessary to determine whether to fill the blank areas or crop the excess pixels to ensure the consistency of the final registered image size and visual effect. This embodiment utilizes homography transformation for projection registration, enabling geometric alignment between the transformed image and the target image, accurately aligning images acquired at different times and from different sensors, thereby achieving high-precision registration.

[0085] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0086] This application also provides an image registration method. Figure 6 This is a schematic diagram of an optional image registration method provided according to an embodiment of this application, such as... Figure 6 As shown, the method includes:

[0087] Specifically, when comparing two remote sensing images (i.e., images to be registered) and During image registration, the two images are first preprocessed. A copy is made and downsampled to 1024×1024 pixels. This downsampled copy is then input into the XFeat deep learning algorithm for feature extraction. The extracted feature points are matched using the Mutual Nearest Neighbor (MNN) algorithm, and outliers are removed using the K-Means Clustering Algorithm. The remaining registration points are then projected back onto the original images based on the downsampling ratio. After projecting all feature points, a sequence of registration points at the original size of the two images is obtained. and Then, based on the obtained filtered registration point sequence... and A stable range of coordinate values ​​for all registration points in the sequence is calculated using the mean square error algorithm. The common overlapping region of the two images is obtained. , Alignment and cropping are performed based on common overlapping regions to obtain coarsely registered images with basic overall alignment. , .

[0088] Furthermore, a multi-strategy sliding window slicing is performed on the coarsely registered images, that is, the two images are meshed. The relatively small-sized slices after meshing are used for subsequent sliding window registration to reduce the downsampling loss when inputting into the network. When the scaling factor is set to 8x and 2x, an 8x fine slice group sequence (i.e., small size) can be obtained. And 2x coarse slice group sequences (i.e., large size) .

[0089] Furthermore, the above sequence The data is sequentially input into the XFeat neural network, for each group of slices to be registered. Obtain the feature point sequence and the corresponding descriptor vector sequence The Lightglue matching algorithm and outlier removal method were used to perform matching, resulting in coarse registration point groups and fine registration point groups. Subsequently, based on the relative positions of the image slices with respect to the original image, the coordinates of the registration points in the coarse and fine registration point groups are mapped back to the coordinate system based on the original image, resulting in coarse registration point group sequences for different sizes. and precise matching sequence .

[0090] Registration point selection is based on upper-level feature structure filtering: Parameter C=10 is set, and the filtering range sequence is determined according to the parameter. Then, based on the range sequence, points in the registration point sequence that do not belong to the range sequence are filtered and removed, that is, for high-resolution registration point sequences... any point in If the point and any low-resolution registration point sequence The distance in each is not less than So give up on this point. Finally, the filtered fine registration point sequence results are obtained. .

[0091] Finally, based on the obtained fine registration point set The image is projected onto the coordinate system of the original remote sensing image, and homography matrix estimation and image projection registration are performed to obtain the registered image. This embodiment uses a deep neural network for feature point extraction, dividing the large-size remote sensing image into smaller-scale slices for independent registration. Common regions are roughly located at low resolution, and then fine feature point extraction and matching are performed at higher resolution. This achieves the technical effects of improving image registration accuracy, reducing registration failure rate, and enhancing registration robustness.

[0092] It should be noted that, Figure 7 This is the registration effect of remote sensing images provided in the embodiments of this application. Figure 1 ,like Figure 7 As shown, the above registration method can be used for remote sensing images captured by infrared optical technology; Figure 8 This is the registration effect of remote sensing images provided in the embodiments of this application. Figure 2 ,like Figure 8 As shown, the above registration method can also be applied to remote sensing images captured by SAR-optical technology; Figure 9 This is the registration effect of remote sensing images provided in the embodiments of this application. Figure 3 ,like Figure 9 As shown, this registration method can also be applied to remote sensing images captured using multispectral technology.

[0093] This application also provides an image registration apparatus. It should be noted that the image registration apparatus of this application can be used to execute the image registration method provided in this application. The image registration apparatus provided in this application is described below.

[0094] Figure 10 This is a schematic diagram of an image registration apparatus provided according to an embodiment of this application, such as... Figure 10 As shown, the device includes: a first acquisition unit 1001, a second acquisition unit 1002, a first processing unit 1003, an extraction unit 1004, and a second processing unit 1005.

[0095] The first acquisition unit 1001 is used to acquire M remote sensing images, perform image scaling processing on the M remote sensing images respectively, and obtain M images to be registered, where M is greater than 1 and M is a positive integer;

[0096] The second acquisition unit 1002 is used to acquire a copy image of each image to be registered, and to extract features from the M copy images using a feature extraction algorithm, and output M sets of feature data, wherein the copy image is obtained by downsampling the image to be registered;

[0097] The first processing unit 1003 is used to perform preliminary registration processing on M sets of feature data respectively to obtain M initial registration point sequences, and to determine M pre-registered images from the M initial registration point sequences;

[0098] The extraction unit 1004 is used to extract features from M pre-registered images using a feature extraction algorithm to obtain M sets of registration point sequences. Each set of registration point sequences includes a registration point sequence with a first scaling factor and a registration point sequence with a second scaling factor, wherein the first scaling factor is less than the second scaling factor.

[0099] The second processing unit 1005 is used to perform multi-scale screening processing on the M sets of registration point sequences, generate a target registration point sequence, and determine the registration image based on the target registration point sequence.

[0100] The image registration apparatus provided in this application embodiment acquires M remote sensing images through a first acquisition unit 1001, performs image scaling on each of the M remote sensing images to obtain M images to be registered, where M is greater than 1 and M is a positive integer; a second acquisition unit 1002 acquires a copy image of each image to be registered, performs feature extraction on each of the M copy images using a feature extraction algorithm, and outputs M sets of feature data, where the copy images are obtained by downsampling the images to be registered; a first processing unit 1003 performs preliminary registration processing on each of the M sets of feature data to obtain M initial registration point sequences, and determines M pre-registered images from the M initial registration point sequences; an extraction unit 1004 performs feature extraction on each of the M pre-registered images using a feature extraction algorithm to obtain M sets of registration point sequences, where each The group registration point sequence includes a registration point sequence with a first scaling factor and a registration point sequence with a second scaling factor, where the first scaling factor is smaller than the second scaling factor. The second processing unit 1005 performs multi-scale screening processing on the M group registration point sequences to generate a target registration point sequence, and determines the registration image based on the target registration point sequence. This solves the problem of low registration accuracy in image registration of remote sensing images in related technologies. By using a feature extraction algorithm to extract features from a copy of the remote sensing image, and performing preliminary registration processing on the extracted feature data, the corresponding initial registration point sequence is obtained. Common regions are roughly located at low resolution, and fine feature point extraction and matching are performed at higher resolution, thereby improving the accuracy of image registration, reducing the registration failure rate, and enhancing the robustness of registration.

[0101] Optionally, in the image registration apparatus provided in this application embodiment, the second acquisition unit 1002 includes: a first extraction module, used to extract scale features from each copy image using Y feature extraction layers of a fusion model to obtain Y descriptor vectors, and determine a descriptor matrix based on the Y descriptor vectors, wherein the fusion model includes K feature preprocessing layers and Y parallel feature extraction layers, each feature extraction layer having a different extraction scale, and K and Y being positive integers; and a calculation module, used to calculate the confidence level of the descriptor matrix and output a confidence level heatmap, wherein the confidence level... The heatmap is used to describe the matching reliability of the Y descriptor vectors; the decomposition module is used to decompose the replica image into feature maps, obtain different sub-regions, and perform convolution processing on each sub-region to obtain a feature point encoding matrix, wherein the feature point encoding matrix is ​​used to indicate the position information of the feature point coordinates corresponding to the Y descriptor vectors; the first determination module is used to determine a set of feature data of the replica image based on the descriptor matrix, the confidence heatmap and the feature point encoding matrix, wherein the set of feature data includes feature sub-data of each sub-region, and each feature sub-data includes the feature point coordinates and descriptor vectors of each sub-region.

[0102] Optionally, in the image registration apparatus provided in this application embodiment, the second acquisition unit 1002 includes: a first filtering module, used to filter out the descriptor vectors output by the feature extraction layer at a preset position from the Y descriptor vectors to obtain a target descriptor vector; a first processing module, used to perform dimensional transformation processing on Y-1 descriptor vectors among the Y descriptor vectors according to the spatial dimension information of the target descriptor vector to obtain Y-1 adjustment vectors; a second processing module, used to concatenate the Y-1 adjustment vectors and the target descriptor vector to obtain a concatenated vector; and a third processing module, used to further fuse the concatenated vector using a fusion module associated with a fusion model to obtain a descriptor matrix.

[0103] Optionally, in the image registration apparatus provided in this application embodiment, the first processing unit 1003 includes: a fourth processing module, used to perform matching processing on the feature sub-data in each group of feature data using a mutual proximity matching algorithm to obtain M groups of matched feature data, wherein each group of matched feature data includes feature sub-data that has been matched by the mutual proximity matching algorithm; and a second filtering module, used to filter the feature point coordinates in a group of matched feature data using a clustering algorithm to obtain P feature point coordinates, wherein P The first is a positive integer; the second is a mapping module, which uses a transformation matrix algorithm to map the coordinates of P feature points to the remote sensing image associated with the copy image, thus obtaining an initial registration point sequence; the third is a determination module, which determines the mean and variance of the initial registration point sequence for each remote sensing image, thus obtaining M sequence means and M sequence variances, and uses each sequence mean and each sequence variance to determine the image size range data for each remote sensing image, thus obtaining M image size range data; the fourth is a cropping module, which uses the M image size range data to crop the M remote sensing images respectively, thus obtaining M pre-registered images.

[0104] Optionally, in the image registration apparatus provided in this application embodiment, the extraction unit 1004 includes: a first acquisition module, configured to acquire a preset scaling factor, and perform segmentation processing on each pre-registered image using the preset scaling factor to obtain M sets of slice group sequences, wherein the preset scaling factor includes a first scaling factor and a second scaling factor, each slice group sequence includes a slice group subsequence of the first scaling factor and a slice group subsequence of the second scaling factor, the slice group subsequence of the first scaling factor includes Q1 image slices, and the slice group subsequence of the second scaling factor includes Q2 image slices, where Q1 and Q2 are positive integers; and a second extraction module, configured to extract a set of slices... The subsequence group is used to extract features from a set of slice subsequences to obtain a set of slice feature data, wherein the set of slice feature data includes Q1 slice feature sub-data or Q2 slice feature sub-data; the matching module is used to match the set of slice feature data using a graph neural network algorithm to obtain a matching point group, and then maps the matching point group to the remote sensing image corresponding to the pre-registered image to obtain a set of registration point sequences, wherein, when the preset scaling factor is the first scaling factor, the matching point group includes J feature point coordinates, and when the preset scaling factor is the second scaling factor, the matching point group includes L feature point coordinates, where J is less than L, and J and L are positive integers.

[0105] Optionally, in the image registration apparatus provided in this application embodiment, the second processing unit 1005 includes: a second acquisition module, used to acquire a preset range parameter, and use the preset range parameter to process the coordinates of J feature points in the registration point sequence of the first scaling factor to obtain J processed feature point ranges; and a third filtering module, used to use the J processed feature point ranges to filter the coordinates of L feature points in the registration point sequence of the second scaling factor to obtain N feature point coordinates located within the feature point range, and the N feature point coordinates constitute the target registration point sequence.

[0106] Optionally, in the image registration apparatus provided in this application embodiment, the second processing unit 1005 includes: a fifth processing module, used to perform homography transformation processing on the target registration point sequence to obtain a homography matrix, wherein the homography matrix is ​​used to indicate the two-dimensional perspective transformation relationship between M remote sensing images; a transformation module, used to perform projection transformation on the remote sensing images of a preset ratio in the M remote sensing images using the homography matrix to obtain a transformed registration image; and a stitching module, used to stitch the transformed registration image and the remaining remote sensing images in the M remote sensing images to obtain a registered image.

[0107] The image registration device includes a processor and a memory. The first acquisition unit 1001, the second acquisition unit 1002, the first processing unit 1003, the extraction unit 1004, and the second processing unit 1005 are all stored in the memory as program units. The processor executes the program units stored in the memory to realize the corresponding functions.

[0108] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can address the low registration accuracy issue encountered in image registration of remote sensing images in related technologies.

[0109] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0110] This invention provides a computer storage medium for storing a program, wherein the program, when running, controls the device where the computer storage medium is located to execute an image registration method.

[0111] Figure 11 This is a schematic diagram of an electronic device provided according to an embodiment of this application, such as... Figure 11 As shown, this embodiment of the invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. The processor is used to execute computer-readable instructions, wherein the computer-readable instructions execute an image registration method. The device in this document can be a server, PC, PAD, mobile phone, etc.

[0112] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of an image registration method according to various embodiments of this application.

[0113] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0115] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0116] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0117] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0118] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0119] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0120] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0121] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An image registration method, characterized in that, include: Acquire M remote sensing images, and perform image scaling processing on each of the M remote sensing images to obtain M images to be registered, where M is greater than 1 and M is a positive integer; Obtain a copy image of each image to be registered, and use a feature extraction algorithm to extract features from the M copy images respectively, outputting M sets of feature data. The copy images are obtained by downsampling the images to be registered. The M sets of feature data are subjected to preliminary registration processing to obtain M initial registration point sequences, and M pre-registered images are determined by the M initial registration point sequences; The feature extraction algorithm is used to extract features from the M pre-registered images to obtain M sets of registration point sequences. Each set of registration point sequences includes a registration point sequence with a first scaling factor and a registration point sequence with a second scaling factor, wherein the first scaling factor is less than the second scaling factor. The M sets of registration point sequences are subjected to multi-scale screening to generate target registration point sequences, and the registration image is determined based on the target registration point sequences. The process of determining M pre-registered images from the M initial registration point sequences includes: determining the mean and variance of the initial registration point sequence for each remote sensing image to obtain M sequence means and M sequence variances; using each sequence mean and each sequence variance to determine the image size range data for each remote sensing image to obtain M image size range data; and cropping the M remote sensing images using the M image size range data to obtain the M pre-registered images. The process of performing multi-scale filtering on the M sets of registration point sequences to generate a target registration point sequence includes: obtaining a preset range parameter; processing the coordinates of J feature points in the registration point sequence at the first scaling factor using the preset range parameter to obtain J processed feature point ranges; filtering the coordinates of L feature points in the registration point sequence at the second scaling factor using the J processed feature point ranges to obtain N feature point coordinates located within the feature point ranges; and constructing the target registration point sequence from the N feature point coordinates.

2. The method according to claim 1, characterized in that, Feature extraction algorithms are used to extract features from M copies of the image, and M sets of feature data are output, including: For each copy image, scale features are extracted from the copy image using Y feature extraction layers of the fusion model to obtain Y descriptor vectors, and a descriptor matrix is ​​determined based on the Y descriptor vectors. The fusion model includes K feature preprocessing layers and Y parallel feature extraction layers, each with a different extraction scale, where K and Y are positive integers. The confidence score is calculated on the descriptor matrix, and a confidence score heatmap is output, wherein the confidence score heatmap is used to describe the matching reliability of the Y descriptor vectors; The copy image is decomposed into feature maps to obtain different sub-regions, and each sub-region is convolved to obtain a feature point encoding matrix, wherein the feature point encoding matrix is ​​used to indicate the position information of the feature point coordinates corresponding to the Y descriptor vectors; A set of feature data for the copy image is determined based on the descriptor matrix, the confidence heatmap, and the feature point encoding matrix, wherein the set of feature data includes feature sub-data for each sub-region, and each feature sub-data includes feature point coordinates and descriptor vectors for each sub-region.

3. The method according to claim 2, characterized in that, Determining the descriptor matrix based on the Y descriptor vectors includes: The target descriptor vector is obtained by selecting the descriptor vector output by the feature extraction layer at a preset position from the Y descriptor vectors. Based on the spatial dimension information of the target descriptor vector, perform dimension transformation on Y-1 of the Y descriptor vectors to obtain Y-1 adjustment vectors; The Y-1 adjustment vectors and the target descriptor vector are concatenated to obtain a concatenated vector. The concatenated vector is further fused using the fusion module associated with the fusion model to obtain the descriptor matrix.

4. The method according to claim 1, characterized in that, The M sets of feature data are subjected to preliminary registration processing to obtain M initial registration point sequences. The M pre-registered images are determined from the M initial registration point sequences, including: The mutual proximity matching algorithm is used to match the feature sub-data in each group of feature data to obtain M groups of matched feature data, wherein each group of matched feature data includes the feature sub-data that has been matched by the mutual proximity matching algorithm. For a set of matched feature data, a clustering algorithm is used to filter the feature point coordinates in the set of matched feature data to obtain P feature point coordinates, where P is a positive integer; The coordinates of the P feature points are mapped to the remote sensing image associated with the replica image using a transformation matrix algorithm to obtain an initial registration point sequence.

5. The method according to claim 1, characterized in that, The feature extraction algorithm is used to extract features from the M pre-registered images respectively, resulting in M ​​sets of registration point sequences, including: A preset scaling factor is obtained, and each pre-registered image is segmented using the preset scaling factor to obtain M sets of slice group sequences. The preset scaling factor includes a first scaling factor and a second scaling factor. Each slice group sequence includes a slice group subsequence of the first scaling factor and a slice group subsequence of the second scaling factor. The slice group subsequence of the first scaling factor includes Q1 image slices, and the slice group subsequence of the second scaling factor includes Q2 image slices. Q1 and Q2 are positive integers. For a set of slice subsequences, feature extraction is performed on the set of slice subsequences to obtain a set of slice feature data, wherein the set of slice feature data includes Q1 slice feature sub-data or Q2 slice feature sub-data; A graph neural network algorithm is used to match the set of slice feature data to obtain a set of matching points. The set of matching points is then mapped to the remote sensing image corresponding to the pre-registered image to obtain a set of registration point sequences. When the preset scaling factor is a first scaling factor, the set of matching points includes J feature point coordinates. When the preset scaling factor is a second scaling factor, the set of matching points includes L feature point coordinates. J is less than L, and J and L are positive integers.

6. The method according to claim 1, characterized in that, Determining the registered image based on the target registration point sequence includes: The target registration point sequence is subjected to homography transformation to obtain a homography matrix, wherein the homography matrix is ​​used to indicate the two-dimensional perspective transformation relationship between the M remote sensing images; The homography matrix is ​​used to perform projection transformation on the M remote sensing images at a preset ratio to obtain transformed and registered images; The transformed and registered image is stitched together with the remaining remote sensing images from the M remote sensing images to obtain the registered image.

7. An image registration device, characterized in that, include: The first acquisition unit is used to acquire M remote sensing images, and perform image scaling processing on the M remote sensing images respectively to obtain M images to be registered, where M is greater than 1 and M is a positive integer; The second acquisition unit is used to acquire a copy image of each image to be registered, and to extract features from the M copy images using a feature extraction algorithm, and output M sets of feature data. The copy images are obtained by downsampling the images to be registered. The first processing unit is used to perform preliminary registration processing on the M sets of feature data respectively to obtain M initial registration point sequences, and to determine M pre-registered images from the M initial registration point sequences; The extraction unit is used to extract features from the M pre-registered images using the feature extraction algorithm to obtain M sets of registration point sequences, wherein each set of registration point sequences includes a registration point sequence with a first scaling factor and a registration point sequence with a second scaling factor, and the first scaling factor is less than the second scaling factor. The second processing unit is used to perform multi-scale filtering processing on the M sets of registration point sequences to generate a target registration point sequence, and determine the registration image based on the target registration point sequence. The first processing unit includes: a second determining module, used to determine the mean and variance of the initial registration point sequence of each remote sensing image, to obtain M sequence means and M sequence variances, and to determine the image size range data of each remote sensing image using each sequence mean and each sequence variance, to obtain M image size range data; and a cropping module, used to crop the M remote sensing images using the M image size range data, to obtain the M pre-registered images. The second processing unit includes: a second acquisition module, used to acquire a preset range parameter, and use the preset range parameter to process the coordinates of J feature points in the registration point sequence of the first scaling factor to obtain J processed feature point ranges; and a third filtering module, used to use the J processed feature point ranges to filter the coordinates of L feature points in the registration point sequence of the second scaling factor to obtain N feature point coordinates located within the feature point range, and the N feature point coordinates constitute the target registration point sequence.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image registration method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the image registration method according to any one of claims 1 to 6.

Images