Image registration method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of image processing, and provides an image registration method and device, electronic equipment and a storage medium. The method comprises the following steps: pre-aligning an original to-be-registered image in a to-be-registered image sequence with an original reference image; constructing an image pyramid of the original reference image and the original to-be-registered image; detecting a reference motion area in the target reference image and a to-be-registered motion area in the target to-be-registered image; starting from the topmost pyramid layer, calculating a target optical flow field between the reference motion area and the to-be-registered motion area which are located in the same pyramid layer layer by layer until a target optical flow field corresponding to the sub-bottom pyramid layer is obtained; and calculating a final optical flow field between the original reference image and the to-be-registered image on the bottommost pyramid layer based on the target optical flow field corresponding to the sub-bottom pyramid layer. According to the scheme, the robustness of an image matching scene with repeated textures and weak textures can be improved, and the calculation amount is effectively compressed in principle.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image registration method, device, electronic device and storage medium. Background Art

[0002] Image registration based on optical flow is a computer vision technique used to estimate pixel motion in image sequences. It excels in image alignment tasks involving complex motion patterns and is widely used in multi-frame image fusion, video stabilization, and image stitching. It can be used to align and fuse multiple consecutive frames of identically exposed images to improve image clarity or reduce noise. Optical flow methods are primarily categorized as sparse and dense. Dense optical flow is the predominant approach for image sequence alignment. This method calculates optical flow values for every pixel in the image. To mitigate the limitations of the implicit small motion assumption, image pyramids are often used. However, using image pyramids for optical flow calculation also introduces drawbacks. For image registration scenes with repetitive or weak textures, it can easily lead to blocky alignment errors. Furthermore, optical flow calculation is performed pixel by pixel on a per-layer basis within the image pyramid, resulting in a computationally intensive process. Summary of the Invention

[0003] Based on this, it is necessary to provide an image registration method, device, electronic device and storage medium to address the above technical problems.

[0004] An embodiment of the present application provides an image registration method, the method comprising:

[0005] Acquire a sequence of images to be registered, and determine an original reference image and an original image to be registered from the sequence of images to be registered;

[0006] Pre-aligning the original image to be registered with the original reference image;

[0007] constructing an image pyramid of the original reference image and an image pyramid of the original image to be registered, wherein the image pyramid of the original image to be registered includes a plurality of target reference images of different scales and the original reference image, and the image pyramid of the original image to be registered includes a plurality of target images to be registered of different scales and the original image to be registered;

[0008] Detecting a reference motion region in the target reference image and a motion region to be registered in the target image to be registered;

[0009] Starting from the topmost pyramid layer, calculating the target optical flow field between the reference motion region and the motion region to be registered located in the same pyramid layer layer by layer until the target optical flow field corresponding to the second-bottom pyramid layer is obtained;

[0010] At the bottommost pyramid layer, a final optical flow field between the original reference image and the image to be registered is calculated based on the target optical flow field corresponding to the second-bottommost pyramid layer.

[0011] An embodiment of the present application provides an image registration device, the device comprising:

[0012] A first acquisition module is used to acquire a sequence of images to be registered, and determine an original reference image and an original image to be registered from the sequence of images to be registered;

[0013] A first alignment module is configured to pre-align the original image to be registered with the original reference image;

[0014] a first construction module, configured to construct an image pyramid of the original reference image and an image pyramid of the original image to be registered, wherein the image pyramid of the original image to be registered includes a plurality of target reference images of different scales and the original reference image, and the image pyramid of the original image to be registered includes a plurality of target images to be registered of different scales and the original image to be registered;

[0015] A first detection module is configured to detect a reference motion region in the target reference image and a motion region to be registered in the target image to be registered;

[0016] A first determining module is configured to calculate, starting from a topmost pyramid layer, a target optical flow field between the reference motion region and the to-be-registered motion region located in the same pyramid layer layer by layer until the target optical flow field corresponding to the next bottom pyramid layer is obtained;

[0017] The second determining module is configured to calculate, at the bottommost pyramid layer, a final optical flow field between the original reference image and the image to be registered based on the target optical flow field corresponding to the second-bottommost pyramid layer.

[0018] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the image registration method provided in any embodiment of the present application are implemented.

[0019] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the image registration method provided in any embodiment of the present application are implemented.

[0020] The image registration method, apparatus, electronic device, and storage medium provided by the embodiments of the present application can obtain a sequence of images to be registered, and determine an original reference image and an original image to be registered from the sequence of images to be registered; pre-align the original image to be registered with the original reference image; construct an image pyramid of the original reference image and an image pyramid of the original image to be registered, wherein the image pyramid of the original image to be registered includes multiple target reference images and original reference images of different scales, and the image pyramid of the original image to be registered includes multiple target images to be registered and original images to be registered of different scales; detect reference motion regions in the target reference image and motion regions to be registered in the target image to be registered; calculate, starting from the topmost pyramid layer, the target optical flow field between the reference motion region and the motion region to be registered located in the same pyramid layer layer by layer until the target optical flow field corresponding to the next-bottom pyramid layer is obtained; and calculate, at the bottommost pyramid layer, a final optical flow field between the original reference image and the image to be registered based on the target optical flow field corresponding to the next-bottom pyramid layer. As can be seen, the above technical solution can globally pre-align the image sequence to be registered, detect the reference motion region and the motion region to be registered based on the image pyramid generated after pre-alignment, and calculate optical flow values based on the reference motion region and the motion region to be registered, reducing the probability of misalignment and optimizing the amount of optical flow calculation in non-motion areas (i.e., image areas excluding the reference motion region and the motion region to be registered). This can improve the robustness of image matching scenarios with repetitive and weak textures and effectively reduce the amount of calculation in principle. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a schematic flow chart of an image registration method in one embodiment;

[0022] Figure 2 A schematic diagram of a process of an image registration example according to an embodiment;

[0023] Figure 3 is a structural schematic diagram of an image registration device in one embodiment;

[0024] Figure 4 FIG. 1 is a schematic structural diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0026] In one embodiment, Figure 1As shown, an image registration method is provided. This method is applicable to image registration scenarios, such as those involving repetitive textures or weak textures. The method can be performed by an image registration device, which can be implemented using software and / or hardware and integrated into an electronic device. The electronic device can be, but is not limited to, various personal computers, laptops, smartphones, and tablet computers. In this embodiment, the method includes the following steps:

[0027] S110 , obtaining a sequence of images to be registered, and determining an original reference image and an original image to be registered from the sequence of images to be registered.

[0028] Specifically, the sequence of images to be registered includes multiple frames of original images (i.e., images before size transformation). From the multiple frames of original images, one frame is selected as the original reference image, and the other multiple frames are the original images to be registered. The original reference image is the image in the multiple frames of original images with which the other original images to be registered are hoped to be aligned.

[0029] There are many specific implementation methods for selecting the original reference image. Typical examples are described below, but they do not constitute a limitation of the present disclosure. For example, the original image with the highest quality (highest clarity, highest contrast and / or minimum distortion, etc.) is selected as the original reference image; the middle frame is selected as the original reference image. The deformation of the previous and next frames relative to the middle frame is usually smaller, and the registration process will be more stable; the original image with high consistency with other original images (for example, high consistency in lighting conditions, shooting angles, etc.) is selected as the original reference image; the original image with the most feature points (such as corners, edges, etc.) is selected as the original reference image; the original image with the highest average similarity with all other original images is selected as the original reference image; a frame is randomly selected as the original reference image. However, it is not limited to this.

[0030] It is understandable that the optical flow method implies two assumptions: that the speed of motion is relatively slow and that the brightness of the pixels before and after displacement remains constant. To improve the limitations imposed by the small motion assumption implied by the optical flow method, an image pyramid can be introduced. To ensure that the brightness remains constant before and after pixel displacement, if there are original images in the image sequence to be registered whose image brightness difference is greater than a preset brightness threshold, the image sequence to be registered can be brightness processed so that the image brightness difference between any two frames of the original image is less than or equal to the preset brightness threshold. That is, after the brightness of the image sequence to be registered is processed to the same or approximately the same level, S120 is executed to achieve image registration with complex motion patterns at different illumination levels.

[0031] S120 : Pre-aligning the original image to be registered with the original reference image.

[0032] In the disclosed embodiment, each original image to be registered is pre-aligned with the original reference image, thereby achieving global pre-alignment of the sequence of images to be registered.

[0033] In some embodiments, pre-aligning the original reference image with the original reference image includes: S121, extracting multiple feature point pairs from the original reference image and the original image to be registered.

[0034] Specifically, feature point extraction algorithms such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), or ORB (Oriented FAST and Rotated BRIEF) may be used to extract feature point pairs, but the present invention is not limited thereto.

[0035] S122 : Pre-aligning the original reference image and the original image to be registered based on multiple feature point pairs.

[0036] Specifically, a matching algorithm such as BFMatcher (Brute Force Matcher) or FLANN (Fast Library for Approximate Nearest Neighbors) may be used to pre-align the original reference image and the original image to be registered, but the present invention is not limited thereto.

[0037] Of course, in other embodiments, mutual information (calculating the mutual information between the original reference image and the original image to be registered, maximizing the mutual information value by optimizing transformation parameters such as translation and rotation), phase correlation method (performing Fourier transform on the original reference image and the original image to be registered, calculating the mutual power spectrum of the two images, using inverse Fourier transform to find the peak position, and determining the translation vector), machine learning and other alignment methods can also be used to pre-align the original reference image and the original image to be registered, but it is not limited to this.

[0038] It is understandable that, considering that if the first frame in the image sequence to be registered is pre-aligned with the second frame (using the first transformation matrix), the second frame is first pre-aligned with the third frame (using the second transformation matrix) and then the third frame is pre-aligned with the first frame based on the first transformation matrix, there will be cumulative errors in the process of pre-aligning the third frame with the first frame, and the same applies to the pre-alignment of subsequent frames with the first frame. In view of this, in the embodiment of the present disclosure, a frame of original reference image is determined from the image sequence to be registered, and then the remaining original images to be registered are directly pre-aligned with the original reference image respectively, so that all original images to be registered can directly use the original reference image as a reference object and be directly pre-aligned with the reference object, so that no cumulative errors will be generated, which is conducive to improving the accuracy of global pre-alignment.

[0039] S130: Construct an image pyramid of the original reference image and an image pyramid of the original image to be registered, where the image pyramid of the original image to be registered includes a plurality of target reference images and the original reference image at different scales, and the image pyramid of the original image to be registered includes a plurality of target images to be registered and the original image to be registered at different scales.

[0040] In the disclosed embodiment, after globally pre-aligning the sequence of images to be registered, a multi-scale transformation is performed on the original reference image to obtain multiple target reference images of different scales. These multiple target reference images and the original reference image constitute an image pyramid of the original reference image. Similarly, the sequence of images to be registered includes multiple original images to be registered. For each original image to be registered, a multi-scale transformation is performed on the original image to be registered to obtain multiple target images to be registered of different scales. These multiple target images to be registered and the original image to be registered constitute an image pyramid of the original image to be registered.

[0041] Specifically, for the image pyramid of the original reference image and the original image to be registered, the image scale (or resolution) gradually increases from the topmost layer to the bottommost layer of the image pyramid, and the target reference image and the target image to be registered located in the same pyramid layer have the same image scale.

[0042] For example, the original reference image is located at the bottommost pyramid layer (the fourth pyramid layer). The original reference image is downsampled (with a downsampling ratio of 2) to obtain a target reference image located at the third pyramid layer. The target reference image located at the third pyramid layer is downsampled (with a downsampling ratio of 2) to obtain a target reference image located at the second pyramid layer. The target reference image located at the second pyramid layer is downsampled (with a downsampling ratio of 2) to obtain a target reference image located at the topmost pyramid layer. In this way, an image pyramid of the original reference image can be obtained. The method for constructing the image pyramid of the original image to be registered is similar to the method for constructing the image pyramid of the original reference image and will not be repeated here.

[0043] S140: Detecting a reference motion region in the target reference image and a motion region to be registered in the target image to be registered.

[0044] In the disclosed embodiment, each target reference image is divided into a motion region (which may be referred to as a reference motion region) and a non-motion region (which may be referred to as a reference non-motion region), and each target image to be registered is divided into a motion region (which may be referred to as a motion region to be registered) and a non-motion region (which may be referred to as a non-motion region to be registered). The image region to be registered is a sequence of images obtained by continuously photographing a moving object, the motion region refers to the image region where the moving object is located, and the non-motion region refers to the image region where the static background is located.

[0045] In some embodiments, S140 includes: S141 , using a target reference image located at a target pyramid layer as a first detection image.

[0046] Specifically, there are multiple ways to select the target pyramid layer. Typical examples are described below, but do not limit the present disclosure. In some examples, if the proportion of the area occupied by the moving object in the original reference image is small (e.g., the proportion is less than or equal to a first preset proportion threshold), indicating that the moving object is small-sized. In this case, the target pyramid layer can be selected from the pyramid layers in the lower third (e.g., if the image pyramid is divided into three roughly equal parts, the lower third refers to the pyramid layers near the bottom). This allows the high-resolution target reference image to be used as the first detection image for detection, thereby retaining more details and improving the accuracy of detecting the moving area. If the proportion of the area occupied by the moving object in the original reference image is medium (e.g., the proportion is greater than the first preset proportion threshold and less than or equal to the second preset proportion threshold), indicating that the moving object is medium-sized, the target pyramid layer can be selected from the pyramid layers in the middle third (the middle third refers to the pyramid layers located in the middle). This allows the higher-resolution target reference image to be used as the first detection image for detection, thereby improving the accuracy of detecting the moving area and reducing the computational complexity. If the area occupied by the moving object in the original reference image accounts for a large proportion (e.g., a proportion greater than a second preset proportion threshold), indicating that the moving object is large, a target pyramid layer can be selected from the pyramid layers in the upper third (the upper third refers to the pyramid layers at the top). This allows the lower-resolution target reference image to be used as the first detection image for detection, reducing the computational effort while still enabling identification of larger moving objects. Of course, in other examples, the target pyramid layer can also be randomly selected.

[0047] S142 : Extract at least one target image to be registered from the target images to be registered located in the target pyramid layer as a second detection image.

[0048] Specifically, the sequence of images to be registered includes multiple (denoted as M, where M is an integer greater than or equal to 2) original images to be registered. An image pyramid is constructed for each original image to be registered. Therefore, each original image to be registered has a corresponding target image to be registered in the target pyramid layer. That is, there are M target images to be registered in the target pyramid layer. At least one target image to be registered is extracted from the M target images to be registered as the second detection image.

[0049] Specifically, there are many specific ways to extract the second detection image. Typical examples are described below, but they do not constitute a limitation to the present disclosure.

[0050] In some embodiments, S142 includes: obtaining a first acquisition time of the original reference image, and obtaining a second acquisition time of each original image to be registered;

[0051] For each target image to be registered located in the target pyramid layer, if the difference between the corresponding second acquisition time and the first acquisition time is less than the second threshold and / or greater than the third threshold, it is used as the second detection image.

[0052] Specifically, since the target image to be registered is obtained by performing size transformation on its corresponding original image to be registered, the acquisition moment corresponding to the target image to be registered is the second acquisition moment of its corresponding original image to be registered.

[0053] Specifically, for each target image to be registered in the “M target images to be registered located in the target pyramid layer”, if the difference between the second acquisition time and the first acquisition time corresponding to the target image to be registered is less than a second threshold, then the target image is used as the second detection image, and / or if the difference between the second acquisition time and the first acquisition time corresponding to the target image to be registered is greater than the second threshold, then the target image is used as the second detection image.

[0054] It is understood that if the difference between the second acquisition time and the first acquisition time corresponding to the target image to be registered is less than the first threshold, it indicates that the target image to be registered is close to the first detection image on the time axis, and the displacement of the moving object is small. Therefore, the details and edges of the moving object can be captured more clearly, which helps to improve detection accuracy. If the difference between the second acquisition time and the first acquisition time corresponding to the target image to be registered is greater than the second threshold, it indicates that the target image to be registered is far away from the first detection image on the time axis, and the position of the moving object relative to other static backgrounds will change more significantly. This significant change helps the algorithm more easily distinguish which pixels belong to the moving object, thereby improving detection accuracy.

[0055] In some other embodiments, S142 includes: extracting a frame of the target image to be registered at every preset frame interval from the target image to be registered located in the target pyramid layer as the second detection image.

[0056] S143. Perform the following steps on each second detection image to obtain multiple candidate motion areas: for each second pixel point in the second detection image, calculate the pixel difference between it and the corresponding first pixel point in the first detection image; if the pixel difference is greater than a first threshold, classify the first pixel point into the candidate motion area.

[0057] Specifically, for each second detection image, the pixel values at the same position of the second detection image and the corresponding first detection image are subtracted pixel by pixel to obtain the pixel difference corresponding to each first pixel point in the first detection image relative to the second detection image. If the pixel difference is greater than the first threshold, it indicates that a significant change has occurred, which may be due to the presence of a moving object. Therefore, the first pixel point with a pixel difference greater than the first threshold can be included in the candidate motion area, thereby obtaining the candidate motion area delineated in the first detection image relative to the second detection image.

[0058] S144 . Determine a reference motion region of the first detection image based on the multiple candidate motion regions.

[0059] In some embodiments, S144 includes merging the multiple candidate motion regions to obtain a reference motion region of the first detection image. In this way, the reference motion region of the first detection image can be ultimately determined by multi-frame fusion, thereby effectively reducing false positives (i.e., incorrectly identifying a motion region) and false negatives (failure to identify an actual motion region) that may occur during single-frame analysis.

[0060] Of course, in some embodiments, based on multiple candidate motion regions, a reference motion region of the first detection image is determined, and S144 includes: performing intersection processing on the multiple candidate motion regions to obtain the reference motion region of the first detection image.

[0061] S145 . Determine, based on the reference motion region of the first detection image, the reference motion regions of the remaining target reference images and the motion regions to be registered of each target image to be registered.

[0062] Specifically, for the first detection image, after the reference motion area in the first detection image is determined, the area other than the reference motion area is the non-reference motion area.

[0063] Specifically, the "remaining target reference images" referred to herein refer to target reference images other than the first detection image in the image pyramid of the original reference image. For each "remaining target reference image," its reference motion region is determined as follows: the first detection image is rescaled to have the same resolution as the "remaining target reference image," and the region of the "remaining target reference image" co-located with the "reference motion region in the rescaled first detection image" is used as the reference motion region of the "remaining target reference image."

[0064] Specifically, for each target image to be registered, a target reference image located at the same pyramid level is found, and the reference motion region in the target reference image is mapped to the target image to be registered, thereby obtaining the motion region to be registered for the target image to be registered. For the target image to be registered, after determining the motion region to be registered in the target image to be registered, the area outside the motion region to be registered is the non-motion region to be registered.

[0065] It can be understood that by calculating the reference motion region and mapping the non-reference motion region in the target pyramid layer, and determining the reference motion region and mapping the non-reference motion region through scale transformation in other pyramid layers, the amount of calculation can be reduced and the processing efficiency can be improved.

[0066] In other embodiments, S142 includes: for each pair of "target reference image and target image to be registered" in each pyramid layer, subtracting the pixel values of the target reference image and the target image to be registered in the "target reference image and target image to be registered" at the same position pixel by pixel to obtain a difference image, applying a first threshold to the difference image, and pixels exceeding the first threshold are considered to be moving areas, and pixels below the first threshold are considered to be non-moving areas.

[0067] S150 , starting from the topmost pyramid layer, calculating the target optical flow field between the reference motion region and the motion region to be registered located in the same pyramid layer layer by layer until the target optical flow field corresponding to the second lowest pyramid layer is obtained.

[0068] In the embodiment of the present disclosure, the optical flow field is calculated starting from the topmost pyramid layer of the image pyramid, and the optical flow field estimated at the previous pyramid layer is used as the starting input for the next pyramid layer estimation until the bottommost pyramid layer.

[0069] Optionally, S150 includes: taking the topmost pyramid layer as the current pyramid layer;

[0070] The reference motion region located in the current pyramid layer is used as the current reference motion region and the motion region to be registered located in the current pyramid layer is used as the current motion region to be registered;

[0071] The following steps are performed for each first pixel in the current reference motion region to obtain a target optical flow field corresponding to the current pyramid level: a reference neighborhood block corresponding to the first pixel is determined in the current reference motion region; multiple candidate neighborhood blocks corresponding to the first pixel are determined in the current motion region to be registered based on a preset window and the initial optical flow field corresponding to the current pyramid level; the candidate neighborhood block with the smallest sum of matching errors with the reference neighborhood block is used as a target neighborhood block; a second pixel matching the first pixel is determined in the current motion region to be registered based on the target neighborhood block; and an optical flow value between the first pixel and the second pixel matching the first pixel is calculated based on the first pixel and the second pixel matching the first pixel.

[0072] If the current pyramid layer is not the second-bottom pyramid layer, the current pyramid layer is updated to the next pyramid layer, the initial optical flow field of the current pyramid layer is updated to the target optical flow field corresponding to the previous pyramid layer, and the process returns to the step of using the reference motion region located in the current pyramid layer as the current reference motion region and the motion region to be registered located in the current pyramid layer as the current motion region to be registered, until the current pyramid layer is the second-bottom pyramid layer.

[0073] Specifically, for the topmost pyramid layer, each element in the initial optical flow field can be 0; for other pyramid layers, the initial optical flow field is the target optical flow field estimated by the previous pyramid layer.

[0074] Specifically, an initial optical flow field is set at the topmost pyramid layer. This initial optical flow field is a matrix of all zero vectors. At this topmost pyramid layer, the optical flow value for each pixel between the reference image region and the reference image region to be registered is calculated using the minimum matching error (MME) method, resulting in the target optical flow field corresponding to the topmost pyramid layer. The target optical flow field corresponding to the topmost pyramid layer is upsampled to the resolution of the next pyramid layer and used as the initial optical flow field for the next pyramid layer. At this next pyramid layer, the optical flow value for each pixel between the reference image region and the reference image region to be registered is calculated using the minimum matching error (MME) method, resulting in the target optical flow field for the next pyramid layer. This process continues in this way until the target optical flow field for the next-lowest pyramid layer is obtained. In this way, for moving areas, the optical flow estimate can be gradually refined starting from the topmost pyramid layer of the image pyramid and moving to the next pyramid layer. For non-moving areas, no optical flow estimation is performed, and the default optical flow value is 0.

[0075] S160 : At the bottommost pyramid layer, based on the target optical flow field corresponding to the next bottommost pyramid layer, calculate the final optical flow field between the original reference image and the image to be registered.

[0076] In the embodiment of the present disclosure, the optical flow field estimated at the second-bottom pyramid layer is used as the starting input of the estimation of the bottom-most pyramid layer to obtain the optical flow field estimated at the second-bottom pyramid layer.

[0077] Optionally, S160 includes: determining an initial optical flow field corresponding to the bottommost pyramid layer based on the target optical flow field corresponding to the second bottommost pyramid layer;

[0078] The following steps are performed for each first pixel point in the original reference image to obtain a final optical flow field: a reference neighborhood block corresponding to the first pixel point is determined in the original reference image; multiple candidate neighborhood blocks corresponding to the first pixel point are determined in the original image to be registered based on a preset window and the initial optical flow field corresponding to the bottommost pyramid layer; the candidate neighborhood block with the smallest matching error with the reference neighborhood block is used as the target neighborhood block; a second pixel point matching the first pixel point is determined in the current image to be registered based on the target neighborhood block; and an optical flow value between the first pixel point and the second pixel point is calculated based on the first pixel point and the second pixel point matching the first pixel point.

[0079] Specifically, the target optical flow field corresponding to the second-bottom pyramid layer is upsampled to the resolution of the bottom pyramid layer and used as the initial optical flow field of the bottom pyramid layer. The optical flow value of each pixel between the original reference image and the original image to be registered is calculated at the bottom pyramid layer through the minimum matching error (MME) method to obtain the final optical flow field.

[0080] It can be understood that after performing a global pre-alignment on the image sequence to be registered, the computational distinction between moving and non-moving regions is made, and the displacement range of the non-moving regions is limited to reduce the probability of misalignment and improve the overall alignment effect. Since the non-moving regions of the image sequence to be registered constitute the majority of the image, limiting the displacement range of the non-moving regions also saves a lot of computation, greatly reducing the numerical operations in the alignment process, accelerating the alignment efficiency, and reducing the power consumption of the alignment process.

[0081] To clearly illustrate the image matching method provided by the embodiment of the present disclosure, a detailed description is given below with reference to a specific example. Figure 2As shown, one image in the image sequence to be registered is first selected as the reference image, and the remaining images are considered the images to be registered. Feature point pairs are then extracted from each pair of reference and registered images, and these feature points are used to achieve global pre-alignment of the image sequence. For each pre-aligned reference image and registered image, a Gaussian pyramid is constructed. The number of layers is typically determined by the original resolution, typically four. Based on the original resolution of the image sequence to be registered, appropriate pyramid layers are extracted. Within these extracted pyramid layers, image numerical subtraction is used to detect the moving regions of moving objects and mark the moving and non-moving regions. Since the image sequence to be registered is captured continuously at the same time, it is not necessary to select all reference image-to-register pairs in the extracted pyramid layers to detect moving regions. Instead, only a subset of reference image-to-register pairs are extracted for detection and the detected regions are merged. After determining the moving and non-moving regions, the displacement calculation using the optical flow method begins. Starting from the topmost layer of the image Gaussian pyramid, the optical flow value for each pixel in the topmost layer is calculated by minimizing the sum of the matching errors within each pixel's neighborhood. During the matching error calculation process, since the image Gaussian pyramid is built on pre-aligned images, the optical flow value in non-moving areas can be assumed to be zero, indicating no displacement. By distinguishing between moving and non-moving areas, only the optical flow value in moving areas needs to be calculated. The optical flow value calculated in the topmost layer is fed back to the next layer, serving as an estimate of the initial optical flow value for the next layer. Similarly, when calculating the optical flow value in the next layer, the optical flow value is distinguished between moving and non-moving areas and calculated only in moving areas. This feedback loop repeats down the image Gaussian pyramid layer, until the bottommost layer is reached. The optical flow value calculated in the bottommost layer is the final optical flow value. To improve image alignment accuracy, the matching error sum is calculated pixel by pixel, without distinguishing between moving and non-moving areas, to obtain the optical flow value for each pixel. This approach effectively optimizes the image alignment process, reduces the probability of mismatches, and improves alignment accuracy and performance.

[0082] This image registration method globally pre-aligns the image sequence to be registered. A reference motion region and a motion region to be registered are detected based on the image pyramid generated after pre-alignment. Optical flow values are then calculated based on these two regions, reducing the probability of misalignment and optimizing the amount of optical flow calculations in non-motion regions (i.e., image regions excluding the reference and motion regions). This improves robustness for image matching scenarios with repetitive and weak textures and, in principle, effectively reduces the amount of computation required.

[0083] It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0084] In one embodiment, Figure 3 As shown, an image registration device is provided, comprising:

[0085] A first acquisition module 310 is configured to acquire a sequence of images to be registered, and determine an original reference image and an original image to be registered from the sequence of images to be registered;

[0086] A first alignment module 320 is configured to pre-align the original image to be registered with the original reference image;

[0087] A first construction module 330 is configured to construct an image pyramid of the original reference image and an image pyramid of the original image to be registered, wherein the image pyramid of the original image to be registered includes a plurality of target reference images and the original reference image at different scales, and the image pyramid of the original image to be registered includes a plurality of target images to be registered and the original image to be registered at different scales;

[0088] A first detection module 340 is configured to detect a reference motion region in the target reference image and a motion region to be registered in the target image to be registered;

[0089] A first determination module 350 is configured to calculate, starting from the topmost pyramid layer, the target optical flow field between the reference motion region and the motion region to be registered located in the same pyramid layer layer by layer until the target optical flow field corresponding to the next lowest pyramid layer is obtained;

[0090] The second determination module 360 is configured to calculate, at the bottommost pyramid layer, a final optical flow field between the original reference image and the image to be registered based on the target optical flow field corresponding to the next bottommost pyramid layer.

[0091] Optionally, the first alignment module 320 is specifically configured to extract a plurality of feature point pairs from the original reference image and the original image to be registered;

[0092] Based on multiple feature point pairs, the original reference image and the original image to be registered are pre-aligned.

[0093] Optionally, the first detection module 340 includes: a first determination submodule, configured to use the target reference image located at the target pyramid layer as the first detection image;

[0094] A second determination submodule is configured to extract at least one target image to be registered from the target images to be registered located in the target pyramid layer as a second detection image;

[0095] A third determination submodule is configured to perform the following steps on each second detection image to obtain a plurality of candidate motion regions: for each second pixel in the second detection image, calculate a pixel difference between the second pixel and the corresponding first pixel in the first detection image; if the pixel difference is greater than a first threshold, classify the first pixel as a candidate motion region;

[0096] a fourth determining submodule, configured to determine a reference motion region of the first detection image based on the plurality of candidate motion regions;

[0097] The fifth determination submodule is configured to determine the reference motion regions of the remaining target reference images and the motion regions to be registered of the target images to be registered based on the reference motion region of the first detection image.

[0098] Optionally, the second determining submodule is specifically configured to obtain a first acquisition time of the original reference image, and obtain a second acquisition time of each original image to be registered;

[0099] For each target image to be registered located in the target pyramid layer, if the difference between the corresponding second acquisition time and the first acquisition time is less than the second threshold and / or greater than the third threshold, it is used as the second detection image.

[0100] Optionally, the fourth determination submodule is specifically configured to merge multiple candidate motion regions to obtain a reference motion region of the first detection image.

[0101] Optionally, the first determining module 350 is specifically configured to use the topmost pyramid layer as the current pyramid layer;

[0102] The reference motion region located in the current pyramid layer is used as the current reference motion region and the motion region to be registered located in the current pyramid layer is used as the current motion region to be registered;

[0103] The following steps are performed for each first pixel in the current reference motion region to obtain a target optical flow field corresponding to the current pyramid level: a reference neighborhood block corresponding to the first pixel is determined in the current reference motion region; multiple candidate neighborhood blocks corresponding to the first pixel are determined in the current motion region to be registered based on a preset window and the initial optical flow field corresponding to the current pyramid level; the candidate neighborhood block with the smallest sum of matching errors with the reference neighborhood block is used as a target neighborhood block; a second pixel matching the first pixel is determined in the current motion region to be registered based on the target neighborhood block; and an optical flow value between the first pixel and the second pixel matching the first pixel is calculated based on the first pixel and the second pixel matching the first pixel.

[0104] If the current pyramid layer is not the second-bottom pyramid layer, the current pyramid layer is updated to the next pyramid layer, the initial optical flow field of the current pyramid layer is updated to the target optical flow field corresponding to the previous pyramid layer, and the process returns to the step of using the reference motion region located in the current pyramid layer as the current reference motion region and the motion region to be registered located in the current pyramid layer as the current motion region to be registered, until the current pyramid layer is the second-bottom pyramid layer.

[0105] Optionally, the second determining module 360 is specifically configured to determine an initial optical flow field corresponding to the bottommost pyramid layer based on the target optical flow field corresponding to the second bottommost pyramid layer;

[0106] The following steps are performed for each first pixel point in the original reference image to obtain a final optical flow field: a reference neighborhood block corresponding to the first pixel point is determined in the original reference image; multiple candidate neighborhood blocks corresponding to the first pixel point are determined in the original image to be registered based on a preset window and the initial optical flow field corresponding to the bottommost pyramid layer; the candidate neighborhood block with the smallest matching error with the reference neighborhood block is used as the target neighborhood block; a second pixel point matching the first pixel point is determined in the current image to be registered based on the target neighborhood block; and an optical flow value between the first pixel point and the second pixel point is calculated based on the first pixel point and the second pixel point matching the first pixel point.

[0107] The image registration device described above can globally pre-align the image sequence to be registered. Based on the image pyramid generated after pre-alignment, it detects the reference motion region and the motion region to be registered. Optical flow values are then calculated based on the reference and motion regions to be registered, reducing the probability of misalignment and optimizing the amount of optical flow calculations in non-motion regions (i.e., image regions excluding the reference and motion regions). This improves robustness for image matching scenarios with repetitive and weak textures and, in principle, effectively reduces the amount of computation required.

[0108] For the specific definition of the image registration device, please refer to the definition of the image registration method above and will not be repeated here. Each module in the above-mentioned image registration device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software so that the processor can call and execute the corresponding operations of each of the above modules.

[0109] In one embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, an image registration method is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse.

[0110] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0111] In one embodiment, the image registration device provided by the present application can be implemented in the form of a computer program. The computer program can be used in Figure 4 The memory of the electronic device may store various program modules constituting the image registration device, such as: Figure 3 The computer program composed of the first acquisition module 310, the first alignment module 320, the first construction module 330, the first detection module 340, the first determination module 350 and the second determination module 360 shown in the figure enables the processor to execute the steps of the image registration method of each embodiment of the present application described in this specification.

[0112] For example, Figure 4The electronic device shown can be Figure 3 The first acquisition module 310 in the device shown in the figure acquires a sequence of images to be registered and determines an original reference image and an original image to be registered from the sequence of images to be registered; the electronic device can perform pre-alignment of the original image to be registered with the original reference image through the first alignment module 320; the electronic device can perform construction of an image pyramid of the original reference image and an image pyramid of the original image to be registered through the first construction module 330, wherein the image pyramid of the original image to be registered includes a plurality of target reference images and the original reference image at different scales, and the image pyramid of the original image to be registered includes a plurality of target images to be registered and the original reference image at different scales. the electronic device may, through the first detection module 340, detect the reference motion region in the target reference image and the motion region to be registered in the target image to be registered; the electronic device may, through the first determination module 350, calculate the target optical flow field between the reference motion region and the motion region to be registered located in the same pyramid layer layer by layer, starting from the topmost pyramid layer, until the target optical flow field corresponding to the second-bottom pyramid layer is obtained; the electronic device may, through the second determination module 360, calculate the final optical flow field between the original reference image and the image to be registered at the bottommost pyramid layer based on the target optical flow field corresponding to the second-bottom pyramid layer.

[0113] In one embodiment, an electronic device is provided, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are performed: obtaining a sequence of images to be registered, and determining an original reference image and an original image to be registered from the sequence of images to be registered; pre-aligning the original image to be registered with the original reference image; constructing an image pyramid of the original reference image and an image pyramid of the original image to be registered, wherein the image pyramid of the original image to be registered includes multiple target reference images and the original reference image at different scales, and the image pyramid of the original image to be registered includes multiple target images to be registered and the original image to be registered at different scales; detecting a reference motion region in the target reference image and a motion region to be registered in the target image to be registered; starting from a topmost pyramid layer, calculating a target optical flow field between the reference motion region and the motion region to be registered located in the same pyramid layer, layer by layer, until a target optical flow field corresponding to a next-lowest pyramid layer is obtained; and calculating, at the bottommost pyramid layer, a final optical flow field between the original reference image and the image to be registered based on the target optical flow field corresponding to the next-lowest pyramid layer.

[0114] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: pre-aligning the original image to be registered with the original reference image, including:

[0115] Extract multiple feature point pairs from the original reference image and the original image to be registered;

[0116] Based on multiple feature point pairs, the original reference image and the original image to be registered are pre-aligned.

[0117] In one embodiment, when the processor executes the computer program, the following steps are further implemented: detecting a reference motion region in the target reference image and a motion region to be registered in the target image to be registered, including:

[0118] The target reference image located at the target pyramid layer is used as the first detection image;

[0119] Extracting at least one target image to be registered from the target images to be registered located in the target pyramid layer as a second detection image;

[0120] Perform the following steps on each second detection image to obtain multiple candidate motion regions: for each second pixel in the second detection image, calculate the pixel difference between the second pixel and the corresponding first pixel in the first detection image; if the pixel difference is greater than a first threshold, classify the first pixel as a candidate motion region;

[0121] Determining a reference motion region of the first detection image based on the plurality of candidate motion regions;

[0122] Based on the reference motion region of the first detection image, the reference motion regions of the remaining target reference images and the motion regions to be registered of each target image to be registered are determined.

[0123] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: extracting at least one target image to be registered from the target images to be registered located in the target pyramid layer as the second detection image, including:

[0124] Obtaining a first acquisition time of the original reference image, and obtaining a second acquisition time of each original image to be registered;

[0125] For each target image to be registered located in the target pyramid layer, if the difference between the corresponding second acquisition time and the first acquisition time is less than the second threshold and / or greater than the third threshold, it is used as the second detection image.

[0126] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: determining a reference motion region of the first detection image based on the plurality of candidate motion regions, including:

[0127] A plurality of candidate motion regions are merged to obtain a reference motion region of the first detection image.

[0128] In one embodiment, when the processor executes the computer program, the following steps are further implemented: starting from the topmost pyramid layer, calculating the target optical flow field between the reference motion region and the motion region to be registered located in the same pyramid layer layer by layer until the target optical flow field corresponding to the next lowest pyramid layer is obtained, including:

[0129] Set the topmost pyramid layer as the current pyramid layer;

[0130] The reference motion region located in the current pyramid layer is used as the current reference motion region and the motion region to be registered located in the current pyramid layer is used as the current motion region to be registered;

[0131] The following steps are performed for each first pixel in the current reference motion region to obtain a target optical flow field corresponding to the current pyramid level: a reference neighborhood block corresponding to the first pixel is determined in the current reference motion region; multiple candidate neighborhood blocks corresponding to the first pixel are determined in the current motion region to be registered based on a preset window and the initial optical flow field corresponding to the current pyramid level; the candidate neighborhood block with the smallest sum of matching errors with the reference neighborhood block is used as a target neighborhood block; a second pixel matching the first pixel is determined in the current motion region to be registered based on the target neighborhood block; and an optical flow value between the first pixel and the second pixel matching the first pixel is calculated based on the first pixel and the second pixel matching the first pixel.

[0132] If the current pyramid layer is not the second-bottom pyramid layer, the current pyramid layer is updated to the next pyramid layer, the initial optical flow field of the current pyramid layer is updated to the target optical flow field corresponding to the previous pyramid layer, and the process returns to the step of using the reference motion region located in the current pyramid layer as the current reference motion region and the motion region to be registered located in the current pyramid layer as the current motion region to be registered, until the current pyramid layer is the second-bottom pyramid layer.

[0133] In one embodiment, when the processor executes the computer program, the following steps are further implemented: at the bottommost pyramid layer, based on the target optical flow field corresponding to the next bottommost pyramid layer, calculating the final optical flow field between the original reference image and the image to be registered, including:

[0134] Based on the target optical flow field corresponding to the second-bottom pyramid layer, determine the initial optical flow field corresponding to the bottom pyramid layer;

[0135] The following steps are performed for each first pixel point in the original reference image to obtain a final optical flow field: a reference neighborhood block corresponding to the first pixel point is determined in the original reference image; multiple candidate neighborhood blocks corresponding to the first pixel point are determined in the original image to be registered based on a preset window and the initial optical flow field corresponding to the bottommost pyramid layer; the candidate neighborhood block with the smallest matching error with the reference neighborhood block is used as the target neighborhood block; a second pixel point matching the first pixel point is determined in the current image to be registered based on the target neighborhood block; and an optical flow value between the first pixel point and the second pixel point is calculated based on the first pixel point and the second pixel point matching the first pixel point.

[0136] The electronic device described above can globally pre-align the image sequence to be registered, detect a reference motion region and a motion region to be registered based on the image pyramid generated after pre-alignment, and calculate optical flow values based on the reference motion region and the motion region to be registered. This reduces the probability of misalignment and optimizes the amount of optical flow calculation in non-motion regions (i.e., image regions excluding the reference motion region and the motion region to be registered). This improves robustness for image matching scenarios with repetitive and weak textures and, in principle, effectively reduces the amount of computation required.

[0137] In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon. When executed by a processor, the computer program implements the following steps: obtaining a sequence of images to be registered, and determining an original reference image and an original image to be registered from the sequence of images to be registered; pre-aligning the original image to be registered with the original reference image; constructing an image pyramid of the original reference image and an image pyramid of the original image to be registered, wherein the image pyramid of the original image to be registered includes a plurality of target reference images and the original reference image at different scales, and the image pyramid of the original image to be registered includes a plurality of target images to be registered and the original image to be registered at different scales; detecting a reference motion region in the target reference image and a motion region to be registered in the target image to be registered; calculating, starting from a topmost pyramid layer, a target optical flow field between the reference motion region and the motion region to be registered located in the same pyramid layer, layer by layer, until a target optical flow field corresponding to a next-lowest pyramid layer is obtained; and calculating, at the bottommost pyramid layer, a final optical flow field between the original reference image and the image to be registered based on the target optical flow field corresponding to the next-lowest pyramid layer.

[0138] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: pre-aligning the original image to be registered with the original reference image, including:

[0139] Extract multiple feature point pairs from the original reference image and the original image to be registered;

[0140] Based on multiple feature point pairs, the original reference image and the original image to be registered are pre-aligned.

[0141] In one embodiment, when the processor executes the computer program, the following steps are further implemented: detecting a reference motion region in the target reference image and a motion region to be registered in the target image to be registered, including:

[0142] The target reference image located at the target pyramid layer is used as the first detection image;

[0143] Extracting at least one target image to be registered from the target images to be registered located in the target pyramid layer as a second detection image;

[0144] Perform the following steps on each second detection image to obtain multiple candidate motion regions: for each second pixel in the second detection image, calculate the pixel difference between the second pixel and the corresponding first pixel in the first detection image; if the pixel difference is greater than a first threshold, classify the first pixel as a candidate motion region;

[0145] Determining a reference motion region of the first detection image based on the plurality of candidate motion regions;

[0146] Based on the reference motion region of the first detection image, the reference motion regions of the remaining target reference images and the motion regions to be registered of each target image to be registered are determined.

[0147] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: extracting at least one target image to be registered from the target images to be registered located in the target pyramid layer as the second detection image, including:

[0148] Obtaining a first acquisition time of the original reference image, and obtaining a second acquisition time of each original image to be registered;

[0149] For each target image to be registered located in the target pyramid layer, if the difference between the corresponding second acquisition time and the first acquisition time is less than the second threshold and / or greater than the third threshold, it is used as the second detection image.

[0150] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: determining a reference motion region of the first detection image based on the plurality of candidate motion regions, including:

[0151] A plurality of candidate motion regions are merged to obtain a reference motion region of the first detection image.

[0152] In one embodiment, when the processor executes the computer program, the following steps are further implemented: starting from the topmost pyramid layer, calculating the target optical flow field between the reference motion region and the motion region to be registered located in the same pyramid layer layer by layer until the target optical flow field corresponding to the next lowest pyramid layer is obtained, including:

[0153] Set the topmost pyramid layer as the current pyramid layer;

[0154] The reference motion region located in the current pyramid layer is used as the current reference motion region and the motion region to be registered located in the current pyramid layer is used as the current motion region to be registered;

[0155] The following steps are performed for each first pixel in the current reference motion region to obtain a target optical flow field corresponding to the current pyramid level: a reference neighborhood block corresponding to the first pixel is determined in the current reference motion region; multiple candidate neighborhood blocks corresponding to the first pixel are determined in the current motion region to be registered based on a preset window and the initial optical flow field corresponding to the current pyramid level; the candidate neighborhood block with the smallest sum of matching errors with the reference neighborhood block is used as a target neighborhood block; a second pixel matching the first pixel is determined in the current motion region to be registered based on the target neighborhood block; and an optical flow value between the first pixel and the second pixel matching the first pixel is calculated based on the first pixel and the second pixel matching the first pixel.

[0156] If the current pyramid layer is not the second-bottom pyramid layer, the current pyramid layer is updated to the next pyramid layer, the initial optical flow field of the current pyramid layer is updated to the target optical flow field corresponding to the previous pyramid layer, and the process returns to the step of using the reference motion region located in the current pyramid layer as the current reference motion region and the motion region to be registered located in the current pyramid layer as the current motion region to be registered, until the current pyramid layer is the second-bottom pyramid layer.

[0157] In one embodiment, when the processor executes the computer program, the following steps are further implemented: at the bottommost pyramid layer, based on the target optical flow field corresponding to the next bottommost pyramid layer, calculating the final optical flow field between the original reference image and the image to be registered, including:

[0158] Based on the target optical flow field corresponding to the second-bottom pyramid layer, determine the initial optical flow field corresponding to the bottom pyramid layer;

[0159] The following steps are performed for each first pixel point in the original reference image to obtain a final optical flow field: a reference neighborhood block corresponding to the first pixel point is determined in the original reference image; multiple candidate neighborhood blocks corresponding to the first pixel point are determined in the original image to be registered based on a preset window and the initial optical flow field corresponding to the bottommost pyramid layer; the candidate neighborhood block with the smallest matching error with the reference neighborhood block is used as the target neighborhood block; a second pixel point matching the first pixel point is determined in the current image to be registered based on the target neighborhood block; and an optical flow value between the first pixel point and the second pixel point is calculated based on the first pixel point and the second pixel point matching the first pixel point.

[0160] The storable medium described above can globally pre-align the image sequence to be registered. Based on the image pyramid generated after pre-alignment, the reference motion region and the motion region to be registered are detected. Optical flow values are then calculated based on the reference and motion regions to be registered, reducing the probability of misalignment and optimizing the amount of optical flow calculations in non-motion regions (i.e., image regions excluding the reference and motion regions). This improves robustness for image matching scenarios with repetitive and weak textures, and in principle effectively reduces the amount of computation required.

[0161] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static random access memory (SRAM) and dynamic random access memory (DRAM).

[0162] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. An image registration method, characterized in that: The method comprises: Acquire a sequence of images to be registered, and determine an original reference image and an original image to be registered from the sequence of images to be registered; Pre-aligning the original image to be registered with the original reference image; constructing an image pyramid of the original reference image and an image pyramid of the original image to be registered, wherein the image pyramid of the original image to be registered includes a plurality of target reference images of different scales and the original reference image, and the image pyramid of the original image to be registered includes a plurality of target images to be registered of different scales and the original image to be registered; Detecting a reference motion region in the target reference image and a motion region to be registered in the target image to be registered; Starting from the topmost pyramid layer, calculating the target optical flow field between the reference motion region and the motion region to be registered located in the same pyramid layer layer by layer until the target optical flow field corresponding to the second-bottom pyramid layer is obtained; At the bottommost pyramid layer, a final optical flow field between the original reference image and the image to be registered is calculated based on the target optical flow field corresponding to the second-bottommost pyramid layer.

2. The method according to claim 1, characterized in that The pre-aligning the original image to be registered with the original reference image includes: Extracting a plurality of feature point pairs from the original reference image and the original image to be registered; Based on the multiple feature point pairs, the original reference image and the original image to be registered are pre-aligned.

3. The method according to claim 1, characterized in that The detecting of the reference motion region in the target reference image and the motion region to be registered in the target image to be registered comprises: Using the target reference image located at the target pyramid layer as a first detection image; Extracting at least one target image to be registered from the target images to be registered located in the target pyramid layer as a second detection image; Performing the following steps on each of the second detection images to obtain a plurality of candidate motion regions: for each second pixel in the second detection image, calculating a pixel difference between the second pixel and the corresponding first pixel in the first detection image; if the pixel difference is greater than a first threshold, classifying the first pixel as the candidate motion region; determining the reference motion region of the first detection image based on the multiple candidate motion regions; Based on the reference motion region of the first detection image, the reference motion regions of the remaining target reference images and the motion regions to be registered of each of the target images to be registered are determined.

4. The method according to claim 3, characterized in that The step of extracting at least one of the target images to be registered located in the target pyramid layer as a second detection image includes: Acquiring a first acquisition time of the original reference image, and acquiring a second acquisition time of each of the original images to be registered; For each of the target images to be registered located in the target pyramid layer, if the difference between the corresponding second acquisition time and the first acquisition time is less than a second threshold and / or greater than a third threshold, the image is used as the second detection image.

5. The method according to claim 3, characterized in that The determining the reference motion region of the first detection image based on the multiple candidate motion regions includes: The multiple candidate motion regions are merged to obtain the reference motion region of the first detection image.

6. The method according to claim 1, characterized in that The step of calculating the target optical flow field between the reference motion region and the to-be-registered motion region located in the same pyramid layer layer by layer starting from the topmost pyramid layer until obtaining the target optical flow field corresponding to the next lowest pyramid layer includes: Taking the topmost pyramid layer as the current pyramid layer; Using the reference motion region located at the current pyramid layer as a current reference motion region and using the motion region to be registered located at the current pyramid layer as a current motion region to be registered; The following steps are performed for each first pixel in the current reference motion region to obtain the target optical flow field corresponding to the current pyramid level: a reference neighborhood block corresponding to the first pixel is determined in the current reference motion region; multiple candidate neighborhood blocks corresponding to the first pixel are determined in the current motion region to be registered based on a preset window and the initial optical flow field corresponding to the current pyramid level; the candidate neighborhood block having the smallest sum of matching errors with the reference neighborhood block is selected as a target neighborhood block; a second pixel matching the first pixel is determined in the current motion region to be registered based on the target neighborhood block; and an optical flow value between the first pixel and the second pixel matching the first pixel is calculated based on the first pixel and the second pixel. If the current pyramid layer is not the second-bottom pyramid layer, the current pyramid layer is updated to the next pyramid layer, the initial optical flow field of the current pyramid layer is updated to the target optical flow field corresponding to the previous pyramid layer, and the process returns to the step of using the reference motion region located in the current pyramid layer as the current reference motion region and using the motion region to be registered located in the current pyramid layer as the current motion region to be registered, until the current pyramid layer becomes the second-bottom pyramid layer.

7. The method according to claim 1, characterized in that The calculating, at the bottommost pyramid layer, a final optical flow field between the original reference image and the image to be registered based on the target optical flow field corresponding to the second-bottommost pyramid layer, includes: Determining an initial optical flow field corresponding to the bottommost pyramid layer based on the target optical flow field corresponding to the second-bottom pyramid layer; The following steps are performed for each first pixel in the original reference image to obtain the final optical flow field: a reference neighborhood block corresponding to the first pixel is determined in the original reference image; multiple candidate neighborhood blocks corresponding to the first pixel are determined in the original image to be registered based on a preset window and the initial optical flow field corresponding to the bottommost pyramid layer; the candidate neighborhood block with the smallest matching error with the reference neighborhood block is used as a target neighborhood block; a second pixel matching the first pixel is determined in the current image to be registered based on the target neighborhood block; and an optical flow value between the first pixel and the second pixel matching the first pixel is calculated based on the first pixel and the second pixel.

8. An image registration device, characterized in that: The device comprises: A first acquisition module is used to acquire a sequence of images to be registered, and determine an original reference image and an original image to be registered from the sequence of images to be registered; A first alignment module is configured to pre-align the original image to be registered with the original reference image; a first construction module, configured to construct an image pyramid of the original reference image and an image pyramid of the original image to be registered, wherein the image pyramid of the original image to be registered includes a plurality of target reference images of different scales and the original reference image, and the image pyramid of the original image to be registered includes a plurality of target images to be registered of different scales and the original image to be registered; A first detection module is configured to detect a reference motion region in the target reference image and a motion region to be registered in the target image to be registered; A first determining module is configured to calculate, starting from a topmost pyramid layer, a target optical flow field between the reference motion region and the to-be-registered motion region located in the same pyramid layer layer by layer until the target optical flow field corresponding to the next bottom pyramid layer is obtained; The second determining module is configured to calculate, at the bottommost pyramid layer, a final optical flow field between the original reference image and the image to be registered based on the target optical flow field corresponding to the second-bottommost pyramid layer.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images