Image registration method and device, image processing method and device, and storage medium

Abstract

The present disclosure relates to an image registration method, an image registration device, an image processing method, an image processing device and a storage medium. The image registration method comprises: obtaining a plurality of images, and determining a reference image and a to-be-registered image in the plurality of images; dividing the to-be-registered image into one or more first regions and one or more second regions based on motion amount estimation, wherein the offset of a pixel point in the first region relative to a corresponding pixel point in the reference image is greater than the offset of a pixel point in the second region relative to a corresponding pixel point in the reference image; registering the first region in the to-be-registered image with the reference image based on image optical flow; and registering the second region in the to-be-registered image with the reference image based on image partial pixel points. The image registration method of the present disclosure can improve the registration accuracy and speed during the registration of the to-be-registered image and the reference image.

Description

Technical Field

[0001] This disclosure relates to the field of image registration technology, and in particular to an image registration method, an image registration apparatus, an image processing method, an image processing apparatus, and a storage medium. Background Technology

[0002] Image registration can be understood as the process of matching and superimposing two or more images acquired at different times, by different sensors (imaging devices, such as cameras), or under different conditions (such as weather, illumination, camera position and angle). According to relevant technologies, in the process of image registration, a single registration method is often used to globally register the images to be registered, which cannot effectively balance the relationship between registration accuracy and registration speed. Summary of the Invention

[0003] To overcome the problems existing in the related technologies, the purpose of this disclosure is to provide an image registration method, an image registration device, an image processing method, an image processing device, and a storage medium, which can ensure improved registration accuracy and registration speed during the registration process between the image to be registered and the reference image.

[0004] According to a first aspect of the present disclosure, an image registration method is provided. The image registration method includes: acquiring multiple frames of images and determining a reference image and an image to be registered in the multiple frames of images; dividing the image to be registered into one or more first regions and one or more second regions based on motion estimation, wherein the offset of a pixel in the first region relative to a corresponding pixel in the reference image is greater than the offset of a pixel in the second region relative to a corresponding pixel in the reference image; registering the first region in the image to be registered with the reference image based on image optical flow; and registering the second region in the image to be registered with the reference image based on partial pixel values.

[0005] In one embodiment of this disclosure, dividing the image to be registered into one or more first regions and one or more second regions based on motion estimation includes: if the pixel difference between corresponding pixels in the image to be registered and the reference image exceeds a pixel difference threshold, then the region where the pixel in the image to be registered is located is determined as the first region; and the other regions in the image to be registered besides the first region are determined as the second regions.

[0006] In another embodiment of this disclosure, the registration of the first region in the image to be registered with the reference image based on image optical flow includes: registering the first region in the image to be registered with the reference image based on a dense optical flow algorithm.

[0007] In another embodiment of this disclosure, the registration of the first region in the image to be registered with the reference image based on the dense optical flow algorithm includes: dividing the first region into a plurality of pixel arrays, wherein the pixel array includes a plurality of pixels; determining the center of the pixel array, and registering the first region in the image to be registered with the reference image based on the dense optical flow algorithm and the pixel center.

[0008] In another embodiment of this disclosure, the registration of the first region in the image to be registered with the reference image based on the dense optical flow algorithm includes: determining the pixel difference between the pixels in the first region and the corresponding pixels in the reference image; scaling the first region based on the pyramid scaling ratio; and registering the scaled first region with the reference image based on the dense optical flow algorithm.

[0009] In another embodiment of this disclosure, the registration of the second region in the image to be registered with the reference image based on partial pixels of the image includes: registering the second region in the image to be registered with the reference image based on the homography matrix.

[0010] In another embodiment of this disclosure, the registration of the second region in the image to be registered with the reference image based on the homography matrix includes: determining feature points in the second region and clustering the feature points to obtain multiple clustered regions; registering the pixels in the clustered regions with the reference image based on the homography matrix, wherein different clustered regions correspond to different homography matrices.

[0011] In another embodiment of this disclosure, the second region further includes a non-clustered region, wherein the non-clustered region is another region in the second region besides the clustered regions, and the image registration method further includes: registering the pixels in the non-clustered region with the reference image based on the homography matrix corresponding to each of the clustered regions.

[0012] In another embodiment of this disclosure, the step of registering pixels in the non-clustered regions with the reference image based on the homography matrix corresponding to each of the clustered regions includes: determining the cluster centers of the clustered regions, and based on the cluster centers, determining the distances between pixels in the non-clustered regions and each of the clustered regions; processing the pixels in the non-clustered regions with the reference image based on the homography matrix corresponding to each of the clustered regions to obtain multiple pixel offsets in the non-clustered regions; weighting the multiple pixel offsets in the non-clustered regions to obtain weighted pixel offsets in the non-clustered regions, wherein the weights of the pixel offsets in the non-clustered regions are determined according to the distances between pixels in the non-clustered regions and each of the clustered regions; and registering pixels in the non-clustered regions with the reference image based on the weighted pixel offsets in the non-clustered regions.

[0013] In another embodiment of this disclosure, the image registration method further includes: determining a transition region of the image to be registered, wherein the transition region is an adjacent region of the first region and the second region; obtaining a first distance between a pixel in the transition region and the first region, and a second distance between a pixel in the transition region and the second region; processing the pixel in the transition region and the reference image based on a dense optical flow algorithm to obtain a first pixel offset, and processing the pixel in the transition region and the reference image based on a homography matrix to obtain a second pixel offset; weighting the first pixel offset and the second pixel offset to obtain a weighted pixel offset in the transition region, wherein the ratio of the weight of the first pixel offset to the weight of the second pixel offset is equal to the ratio of the first distance to the second distance; and registering the pixel in the transition region and the reference image based on the weighted pixel offset in the transition region.

[0014] According to a second aspect of the present disclosure, an image processing method is provided. The image processing method includes: acquiring multiple frames of images; registering the multiple frames of images based on the image registration method described in the first aspect of the present disclosure or any embodiment of the first aspect; and performing image fusion and noise reduction based on the registered multiple frames of images to obtain a fused and denoised image.

[0015] According to a third aspect of the present disclosure, an image registration apparatus is provided. The image registration apparatus includes: a first acquisition module, configured to acquire multiple frames of images and determine a reference image and an image to be registered in the multiple frames of images; a segmentation module, configured to divide the image to be registered into one or more first regions and one or more second regions based on motion estimation, wherein the offset of a pixel in the first region relative to a corresponding pixel in the reference image is greater than the offset of a pixel in the second region relative to a corresponding pixel in the reference image; and a processing module, configured to register the first regions in the image to be registered with the reference image based on image optical flow; and to register the second regions in the image to be registered with the reference image based on partial pixel points of the image.

[0016] In one embodiment of this disclosure, the segmentation module divides the image to be registered into one or more first regions and one or more second regions based on motion estimation in the following manner: if the pixel difference between the corresponding pixel points of the image to be registered and the reference image exceeds a pixel difference threshold, then the region where the pixel point is located in the image to be registered is determined as the first region; and the other regions in the image to be registered other than the first region are determined as the second regions.

[0017] In another embodiment of this disclosure, the processing module registers the first region in the image to be registered with the reference image based on image optical flow: registering the first region in the image to be registered with the reference image based on a dense optical flow algorithm.

[0018] In another embodiment of this disclosure, the processing module registers the first region in the image to be registered with the reference image using a dense optical flow algorithm in the following manner: dividing the first region into multiple pixel arrays, wherein the pixel arrays include multiple pixels; determining the center of the pixel arrays, and registering the first region in the image to be registered with the reference image based on the dense optical flow algorithm and the center of the pixel arrays.

[0019] In another embodiment of this disclosure, the processing module registers the first region in the image to be registered with the reference image using a dense optical flow algorithm in the following manner: determining the pixel difference between the pixels in the first region and the corresponding pixels in the reference image; scaling the first region based on the pyramid scaling ratio; and registering the scaled first region with the reference image based on the dense optical flow algorithm.

[0020] In another embodiment of this disclosure, the processing module registers the second region in the image to be registered with the reference image based on partial pixels of the image: registering the second region in the image to be registered with the reference image based on the homography matrix.

[0021] In another embodiment of this disclosure, the processing module registers the second region in the image to be registered with the reference image based on the homography matrix in the following manner: determining feature points in the second region and clustering the feature points to obtain multiple clustered regions; registering the pixels in the clustered regions with the reference image based on the homography matrix, wherein different clustered regions correspond to different homography matrices.

[0022] In another embodiment of this disclosure, the second region further includes a non-clustered region, wherein the non-clustered region is another region in the second region besides the clustered regions, and the processing module is further configured to: register the pixels in the non-clustered region with the reference image based on the homography matrix corresponding to each of the clustered regions.

[0023] In another embodiment of this disclosure, the processing module registers pixels in the non-clustered regions with the reference image based on the homography matrix corresponding to each of the clustered regions in the following manner: determining the cluster centers of the clustered regions, and based on the cluster centers, determining the distances between pixels in the non-clustered regions and each of the clustered regions; processing the pixels in the non-clustered regions with the reference image based on the homography matrix corresponding to each of the clustered regions to obtain multiple pixel offsets in the non-clustered regions; weighting the multiple pixel offsets in the non-clustered regions to obtain weighted pixel offsets in the non-clustered regions, wherein the weights of the pixel offsets in the non-clustered regions are determined according to the distances between pixels in the non-clustered regions and each of the clustered regions; and registering the pixels in the non-clustered regions with the reference image based on the weighted pixel offsets in the non-clustered regions.

[0024] In another embodiment of this disclosure, the first acquisition module is further configured to: determine a transition region of the image to be registered, wherein the transition region is an adjacent region of the first region and the second region; and acquire a first distance between a pixel in the transition region and the first region, and a second distance between a pixel in the transition region and the second region; the processing module is further configured to: process the pixels in the transition region and the reference image based on a dense optical flow algorithm to obtain a first pixel offset, and process the pixels in the transition region and the reference image based on a homography matrix to obtain a second pixel offset; perform weighted processing on the first pixel offset and the second pixel offset to obtain a weighted pixel offset in the transition region, wherein the ratio of the weight of the first pixel offset to the weight of the second pixel offset is equal to the ratio of the first distance to the second distance; and register the pixels in the transition region and the reference image based on the weighted pixel offset in the transition region.

[0025] According to a fourth aspect of the present disclosure, an image processing apparatus is provided. The image processing apparatus includes: a second acquisition module for acquiring multiple frames of images; and a noise reduction module for registering the multiple frames of images based on the image registration method described in the first aspect of the present disclosure or any embodiment of the first aspect, and performing image fusion noise reduction based on the registered multiple frames of images to obtain a fused and denoised image.

[0026] According to a fifth aspect of the present disclosure, an image registration apparatus is provided, including a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke instructions to execute the image registration method described in the first aspect of the present disclosure or any embodiment of the first aspect.

[0027] According to a sixth aspect of the present disclosure, an image processing apparatus is provided, including a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions to execute the image processing method described in the second aspect of the present disclosure or any embodiment of the second aspect.

[0028] According to a seventh aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, which, when the instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform the image registration method described in the first aspect of the present disclosure or any embodiment of the first aspect.

[0029] According to an eighth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, which, when the instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform the image processing method described in the second aspect of the present disclosure or any embodiment of the second aspect.

[0030] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: based on motion estimation, the image to be registered is divided into a first region and a second region, and the first region and the second region of the image to be registered are registered with the reference image using image optical flow and some pixels of the image, so that the registration accuracy and registration speed can be improved during the registration process between the image to be registered and the reference image.

[0031] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0032] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0033] Figure 1 This is a flowchart illustrating an image registration method according to an exemplary embodiment.

[0034] Figure 2 This is a flowchart illustrating, according to an exemplary embodiment, a method for dividing an image to be registered into one or more first regions and one or more second regions based on motion estimation.

[0035] Figure 3 This is a flowchart illustrating, according to an exemplary embodiment, a method for registering a first region in an image to be registered with a reference image based on a dense optical flow algorithm.

[0036] Figure 4 This is a flowchart illustrating another method for registering a first region in an image to be registered with a reference image based on a dense optical flow algorithm, according to an exemplary embodiment.

[0037] Figure 5 This is a flowchart illustrating, according to an exemplary embodiment, a method for registering a second region in an image to be registered with a reference image based on a homography matrix.

[0038] Figure 6 This is a flowchart illustrating another image registration method according to an exemplary embodiment.

[0039] Figure 7 This is a flowchart illustrating, according to an exemplary embodiment, the registration of pixels in non-clustered regions with a reference image based on a homography matrix corresponding to each clustered region.

[0040] Figure 8 This is a flowchart illustrating yet another image registration method according to an exemplary embodiment.

[0041] Figure 9 This is a flowchart illustrating yet another image registration method according to an exemplary embodiment.

[0042] Figure 10 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0043] Figure 11 This is a block diagram illustrating an image registration apparatus according to an exemplary embodiment.

[0044] Figure 12 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment.

[0045] Figure 13 This is a block diagram illustrating an apparatus for image registration or image processing according to an exemplary embodiment. Detailed Implementation

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0047] In the accompanying drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some, but not all, of the embodiments of this disclosure. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure. The embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0048] In the description of this embodiment, it should be understood that the terms "center," "longitudinal," "lateral," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the accompanying drawings and are only for the convenience of describing this embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this embodiment. It should be noted that unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components described in these embodiments do not limit the scope of this disclosure.

[0049] Image registration can be understood as the process of matching and superimposing two or more images acquired at different times, by different sensors (imaging devices, such as cameras), or under different conditions (such as weather, illumination, camera position and angle). During image registration, the image to be registered can be subjected to perspective transformation based on the mapping relationship between the reference image and the image to be registered, thus mapping the image to be registered to the image space corresponding to the reference image, achieving registration between the image to be registered and the reference image.

[0050] According to relevant technologies, in the process of image registration, a registration method is often used to perform global registration of the images to be registered, which makes it difficult to balance the relationship between registration accuracy and registration speed.

[0051] The image registration method provided in this disclosure divides the image to be registered into a first region and a second region based on motion estimation, and uses image optical flow and some pixels of the image to register the first region and the second region of the image to be registered with a reference image, so as to improve the registration accuracy and registration speed during the registration process between the image to be registered and the reference image.

[0052] Figure 1 This is a flowchart illustrating an image registration method according to an exemplary embodiment.

[0053] In an exemplary embodiment of this disclosure, such as Figure 1 As shown, the image registration method may include steps S11 to S14. Each step will be described below.

[0054] In step S11, multiple frames of images are acquired, and a reference image and an image to be registered are determined from the multiple frames of images.

[0055] In one embodiment, multiple frames of images are acquired. These multiple frames may be images of the same scene or the same target object. Further, a reference image may be determined from the multiple frames. In one example, the reference image may be the image with the highest resolution. In another example, the image to be registered may be any image from the multiple frames other than the reference image.

[0056] In step S12, based on motion estimation, the image to be registered is divided into one or more first regions and one or more second regions. The offset of pixels in the first region relative to their corresponding pixels in the reference image is greater than the offset of pixels in the second region relative to their corresponding pixels in the reference image.

[0057] In one embodiment, each image to be registered can be divided based on motion estimation to obtain one or more first regions and one or more second regions. It should be noted that the offset of pixels in the first region relative to their corresponding pixels in the reference image is greater than the offset of pixels in the second region relative to their corresponding pixels in the reference image. It can be understood that the first region represents a large motion region in the image to be registered, meaning that the change in pixel values ​​in the first region relative to their corresponding pixels in the reference image is relatively large. The second region represents a small motion region in the image to be registered, meaning that the change in pixel values ​​in the second region relative to their corresponding pixels in the reference image is relatively small.

[0058] In one example, the corresponding pixel of pixel 'a' in the reference image in the image to be registered can be referred to as 'a'. It is understood that pixel 'a' and pixel 'a' are different forms of the same pixel in different images. If the coordinates of pixel 'a' in the image to be registered are (5, 6), and the coordinates of pixel 'a' in the reference image are (8, 9), then the offset of pixel 'a' relative to its corresponding pixel 'a' in the reference image is the distance between the coordinates of pixel 'a' and pixel 'a'.

[0059] It should be noted that the pixel offset can be understood as the amount of positional change of a pixel relative to its corresponding pixel in the reference image. Furthermore, based on the pixel offset, pixels in the image to be registered can be mapped to their corresponding pixels in the reference image.

[0060] It is understandable that dividing the image to be registered into a first region and a second region can lay the foundation for selecting an appropriate registration method based on the motion data of the first and second regions, thereby improving the registration accuracy and speed.

[0061] In step S13, the first region in the image to be registered is registered with the reference image based on the image optical flow.

[0062] Image optical flow refers to the instantaneous velocity of pixels on the observation imaging plane of a moving object in space. In applications, image optical flow can be used to find the correspondence between pixels in two frames of images, thereby calculating the offset of pixels between the two frames. In one embodiment, a dense optical flow algorithm can be used to register a first region in the image to be registered with a reference image.

[0063] In step S14, the second region in the image to be registered is registered with the reference image based on the partial pixels of the image.

[0064] In application, feature points of an image can be determined based on a subset of its pixels. Then, a homography matrix corresponding to each feature point is calculated. Feature points refer to points where the image's grayscale value changes drastically or points with significant curvature at image edges (i.e., the intersection of two edges). In one embodiment, a second region in the image to be registered can be registered with a reference image based on the homography matrix. It is understood that determining feature points based on a subset of the image's pixels and calculating the corresponding homography matrix reduces the computational load in determining the homography matrix and registering the second region in the image to be registered with the reference image based on the homography matrix.

[0065] Dense optical flow algorithms offer high registration accuracy but require significant computation. Homography matrix algorithms provide fast registration, but exhibit weak stability when registering images with high motion. In one embodiment, a first region in the image to be registered can be registered with a reference image using a dense optical flow algorithm, while a second region in the image to be registered can be registered with the reference image using a homography matrix. Since the first region represents a large motion area in the image to be registered, a dense optical flow algorithm is used to ensure registration accuracy. Similarly, since the second region represents a small motion area, a homography matrix is ​​used to ensure registration speed. This embodiment improves both registration accuracy and speed during the registration process between the image to be registered and the reference image.

[0066] This disclosure provides an image registration method that, based on motion estimation, divides the image to be registered into a first region and a second region, and uses a dense optical flow algorithm and a homography matrix to register the first and second regions of the image to be registered with a reference image, respectively, so as to improve the registration accuracy and registration speed during the registration process between the image to be registered and the reference image.

[0067] This disclosure will illustrate, through the following embodiments, the process of dividing an image to be registered into one or more first regions and one or more second regions based on motion estimation.

[0068] Figure 2 This is a flowchart illustrating, according to an exemplary embodiment, a method for dividing an image to be registered into one or more first regions and one or more second regions based on motion estimation.

[0069] In an exemplary embodiment of this disclosure, such as Figure 2 As shown, dividing the image to be registered into one or more first regions and one or more second regions based on motion estimation may include steps S21 and S22. Each step will be described in detail below.

[0070] In step S21, if the pixel difference between the corresponding pixels in the image to be registered and the reference image exceeds the pixel difference threshold, the region where the pixel in the image to be registered is located is determined as the first region.

[0071] In step S22, the regions other than the first region in the image to be registered are determined as the second region.

[0072] In one embodiment, when the pixel difference between corresponding pixels in the image to be registered and the reference image exceeds a pixel difference threshold, it indicates that the region where the pixel in the image to be registered is a region of high motion, and this region can be defined as the first region. It should be noted that the pixel difference can be understood as the difference in pixel values ​​between two pixels at corresponding positions in the image to be registered and the reference image. In one example, the pixel difference can be the difference in brightness values, the difference in RGB values, or the difference in grayscale values ​​between two pixels.

[0073] In one embodiment, if the coordinates of pixel a in the image to be registered are (5, 6), then the corresponding pixel in the reference image is pixel b located at coordinates (5, 6) in the reference image. It is understood that pixel a and pixel b are different forms of different pixels in different images. In one example, the pixel difference can be the difference in grayscale values ​​between pixel b and pixel a. It is understood that the pixel difference can reflect the amount of motion in the region where the pixel is located.

[0074] In one example, when the pixel difference of the region formed by 3*3 to 4*4 pixels in the image to be registered exceeds a pixel difference threshold, this region can be defined as the first region. It should be noted that the first region can be the area containing a moving object in the image to be registered. The first region can be the foreground element in the image to be registered, such as a pedestrian or a moving vehicle. In application, the first region in the image to be registered can be registered with the reference image based on a dense optical flow algorithm.

[0075] Furthermore, regions other than the first region in the image to be registered can be defined as the second region. It should be noted that the second region can be the area containing non-moving objects in the image to be registered, for example, the background region. In the image to be registered, slight displacement of the second region is often caused by camera shake during shooting. In application, the second region in the image to be registered can be registered with the reference image based on the homography matrix.

[0076] Figure 3 This is a flowchart illustrating, according to an exemplary embodiment, a method for registering a first region in an image to be registered with a reference image based on a dense optical flow algorithm.

[0077] This disclosure will be combined with Figure 3 The process of registering the first region in the image to be registered with the reference image based on the dense optical flow algorithm is explained.

[0078] In an exemplary embodiment of this disclosure, such as Figure 3 As shown, registering the first region in the image to be registered with the reference image based on the dense optical flow algorithm can include steps S31 and S32. Each step will be described below.

[0079] In step S31, the first region is divided into multiple pixel arrays, wherein the pixel arrays include multiple pixels.

[0080] In step S32, the center of the pixel dot matrix is ​​determined, and based on the dense optical flow algorithm and the dot matrix center, the first region in the image to be registered is registered with the reference image.

[0081] In one embodiment, during image registration of the first region, all pixels within the first region can be processed using a dense optical flow algorithm to obtain pixel offsets, and the first region can be registered with a reference image based on these offsets. It should be noted that the pixel offset can be understood as the amount of positional change of a pixel relative to its corresponding pixel in the reference image. Furthermore, based on the pixel offsets, pixels in the image to be registered can be mapped to their corresponding pixels in the reference image.

[0082] In another embodiment, to improve the registration speed between the first region and the reference image, the first region can be divided into multiple pixel arrays, where each pixel array is a set of adjacent pixels. Further, the center of each pixel array can be determined, and all pixel centers within the first region can be processed using a dense optical flow algorithm to obtain their offsets. The first region and the reference image are then registered based on these offsets. It should be noted that the center of each pixel array can be determined based on the coordinates of the pixel array's center position.

[0083] This disclosure will describe the process of registering a first region in an image to be registered with a reference image based on the dense optical flow algorithm, in conjunction with the following embodiments.

[0084] Figure 4 This is a flowchart illustrating another method for registering a first region in an image to be registered with a reference image based on a dense optical flow algorithm, according to an exemplary embodiment.

[0085] In an exemplary embodiment of this disclosure, such as Figure 4 As shown, registering the first region in the image to be registered with the reference image based on the dense optical flow algorithm can include steps S41 and S42. Each step will be described below.

[0086] In step S41, the pixel difference between a pixel in the first region and a pixel at the corresponding position in the reference image is determined.

[0087] In the application, the image to be registered may include multiple first regions. In one example, the pixel differences between pixels in different first regions and corresponding pixels in the reference image are different. Understandably, the amount of motion varies among the different first regions.

[0088] In step S42, the first region is scaled based on the pyramid scaling ratio, and the scaled first region is registered with the reference image based on the dense optical flow algorithm.

[0089] In one embodiment, based on the pyramid scaling ratio, different scaling ratios can be applied to the first region according to different amounts of motion in different regions. Then, image registration is performed between the scaled first region and the reference image using a dense optical flow algorithm. This embodiment adaptively selects the pyramid scaling ratio to scale the first region based on its varying amounts of motion. Furthermore, the dense optical flow algorithm is used to register the scaled first region with the reference image, thereby improving the registration speed while ensuring registration accuracy.

[0090] This disclosure will describe the process of registering a second region in an image to be registered with a reference image based on the homography matrix, using the following embodiments.

[0091] Figure 5 This is a flowchart illustrating, according to an exemplary embodiment, a method for registering a second region in an image to be registered with a reference image based on a homography matrix.

[0092] In an exemplary embodiment of this disclosure, such as Figure 5As shown, registering the second region in the image to be registered with the reference image based on the homography matrix can include steps S51 and S52. Each step will be described below.

[0093] In step S51, feature points in the second region are determined and clustered to obtain multiple clustered regions.

[0094] In image processing, feature points refer to points where the grayscale value of an image changes drastically or points with large curvature on the image edge (i.e., the intersection of two edges). Image feature points can reflect the essential features of an image and identify target objects in the image.

[0095] In one embodiment, feature points can be detected and determined in a second region of the image to be registered. During application, clustering can be performed based on the location information of the feature points. It is understood that clustered feature points can correspond to the same homography matrix; that is, feature points belonging to the same cluster region can correspond to the same homography matrix.

[0096] In step S52, the pixels within the clustered regions are registered with the reference image based on the homography matrix, wherein different clustered regions correspond to different homography matrices.

[0097] In one embodiment, pixels within a clustered region can be registered with a reference image based on the same homography matrix. The second region may include multiple clustered regions, each corresponding to a different homography matrix. During application, the second region and the reference image can be registered separately based on different homography matrices. This embodiment, by using the same homography matrix to register pixels within the same clustered region with the reference image, can improve image registration speed.

[0098] Figure 6 This is a flowchart illustrating another image registration method according to an exemplary embodiment.

[0099] This disclosure will be combined with Figure 6 The process of another image registration method is explained.

[0100] In an exemplary embodiment of this disclosure, the second region further includes a non-clustered region, wherein the non-clustered region is any region in the second region other than the clustered regions. For example... Figure 6 As shown, the image registration method may include steps S61 to S66, wherein steps S61 to S62 are the same as steps S11 to S12, and steps S64 to S65 are the same as steps S51 to S52. For specific implementation methods and beneficial effects, please refer to the preceding text, which will not be repeated here. Steps S63 and S66 will be described below.

[0101] In step S63, the first region in the image to be registered is registered with the reference image based on the dense optical flow algorithm.

[0102] In step S66, based on the homography matrix corresponding to each clustered region, the pixels in the non-clustered regions are registered with the reference image.

[0103] Non-clustered regions can be understood as regions lacking feature points. In application, the homography matrix corresponding to non-clustered regions cannot be determined.

[0104] In one embodiment, pixels in non-clustered regions can be registered with the reference image based on the homography matrix corresponding to each clustered region. This embodiment solves the image distortion problem in different regions of the image to be registered, thereby enabling more accurate registration and alignment between the image to be registered and the reference image.

[0105] It should be noted that, based on the homography matrix corresponding to each cluster region, the process of registering pixels in non-clustered regions with the reference image can be achieved in the following way.

[0106] Figure 7 This is a flowchart illustrating, according to an exemplary embodiment, the registration of pixels in non-clustered regions with a reference image based on a homography matrix corresponding to each clustered region.

[0107] In an exemplary embodiment of this disclosure, such as Figure 7 As shown, based on the homography matrix corresponding to each cluster region, registering pixels in non-clustered regions with the reference image may include steps S71 to S74.

[0108] In step S71, the cluster center of the clustered region is determined, and based on the cluster center, the distance between the pixel in the non-clustered region and each clustered region is determined.

[0109] In one embodiment, the cluster centers of different clustering regions can be determined separately. The cluster centers can be determined based on the coordinates of the center position of the clustering region. During application, based on the cluster centers, the distances between each pixel in the non-clustered region and each clustered region can be determined separately.

[0110] In step S72, based on the homography matrix corresponding to each clustered region, the pixels in the non-clustered regions are processed relative to the reference image to obtain the pixel offsets in multiple non-clustered regions.

[0111] Different clustered regions correspond to different homography matrices. In one embodiment, based on the homography matrices corresponding to each clustered region, the pixels in the non-clustered regions can be processed relative to the reference image to obtain the pixel offsets of each pixel in multiple non-clustered regions. It is understood that if there are 5 homography matrices, then for a pixel (e.g., pixel a) in a certain non-clustered region (e.g., non-clustered region A), there are 5 pixel offsets.

[0112] In step S73, the pixel offsets in multiple non-clustered regions are weighted to obtain the weighted pixel offsets in the non-clustered regions. The weights of the pixel offsets in the non-clustered regions are determined based on the distances between the pixels in the non-clustered regions and each clustered region.

[0113] Continuing with the previously mentioned example of a homography matrix with five elements, let's take an example where the pixel offset of pixel a in non-clustered region A is five. In application, the five pixel offsets of pixel a can be weighted to obtain a weighted pixel offset. During the weighting process, the weights can be determined based on the distances between pixel a and each of the five clustered regions. In one example, the weights are positively correlated with the distances between pixel a and each clustered region.

[0114] In step S74, the pixels in the non-clustered regions are registered with the reference image based on the pixel offsets in the weighted non-clustered regions.

[0115] In the application, the weighted pixel offset of all pixels in the non-clustered region can be calculated, and based on the weighted pixel offset, the pixels in the non-clustered region can be registered with the reference image. It should be noted that the weighted pixel offset of the pixels in the non-clustered region can be understood as the pixel offset of the pixels in the registered non-clustered region.

[0116] Figure 8 This is a flowchart illustrating yet another image registration method according to an exemplary embodiment.

[0117] This disclosure will be combined with Figure 8 The process of another image registration method is explained.

[0118] In an exemplary embodiment of this disclosure, such as Figure 8As shown, the image registration method may include steps S81 to S89, wherein steps S81 to S82 are the same as steps S11 to S12 respectively. For specific implementation methods and beneficial effects, please refer to the previous text, which will not be repeated here. Steps S83 to S84 and steps S85 to S89 will be introduced below.

[0119] In step S83, the first region in the image to be registered is registered with the reference image based on the dense optical flow algorithm.

[0120] In step S84, the second region in the image to be registered is registered with the reference image based on the homography matrix.

[0121] In step S85, the transition region of the image to be registered is determined, wherein the transition region is the adjacent region of the first region and the second region.

[0122] In one embodiment, a transition region of the image to be registered can be determined. This transition region can be an adjacent region of the first region and the second region. In one example, the adjacent region can include a portion of the first region immediately adjacent to the second region, and may also include a portion of the second region immediately adjacent to the first region.

[0123] In step S86, the first distance between the pixel in the transition region and the first region, and the second distance between the pixel in the transition region and the second region are obtained.

[0124] In one embodiment, a first distance can be determined based on the distance between a pixel in the transition region and the edge of a first region near the transition region. A second distance can be determined based on the distance between a pixel in the transition region and the edge of a second region near the transition region. In another embodiment, the first distance can be determined based on the distance between a pixel in the transition region and the center coordinates of the first region. The second distance can be determined based on the distance between a pixel in the transition region and the center coordinates of the second region.

[0125] In step S87, the pixels in the transition region and the reference image are processed based on the dense optical flow algorithm to obtain the first pixel offset, and the pixels in the transition region and the reference image are processed based on the homography matrix to obtain the second pixel offset.

[0126] In step S88, the offset of the first pixel point and the offset of the second pixel point are weighted to obtain the offset of the pixel point in the transition region after weighting. The ratio of the weight of the first pixel point offset to the weight of the second pixel point offset is equal to the ratio of the first distance to the second distance.

[0127] In one embodiment, the pixel offset of pixel a within the weighted transition region is used as an example for illustration. During application, pixel a within the transition region and the reference image can be processed based on a dense optical flow algorithm to obtain a first pixel offset, and pixel a within the transition region and the reference image can be processed based on a homography matrix to obtain a second pixel offset. Further, the first and second pixel offsets are weighted to obtain the weighted pixel offset of pixel a within the transition region. The ratio of the weight of the first pixel offset to the weight of the second pixel offset is equal to the ratio of the first distance to the second distance. It should be noted that the pixel offset of a pixel in the non-clustered region after weighted processing can be understood as the pixel offset of a pixel in the registered non-clustered region.

[0128] In step S89, the pixels in the transition region are registered with the reference image based on the pixel offsets in the weighted transition region.

[0129] During application, the final weighted pixel offset of all pixels in the transition region can be calculated, and based on the final weighted pixel offset, the pixels in the transition region are registered with the reference image. This embodiment ensures a more natural and smooth registration of the transition region, thereby guaranteeing a natural transition between the edge regions of the first and second regions.

[0130] This disclosure will illustrate the process of the image registration method through the following embodiments.

[0131] Figure 9 This is a flowchart illustrating yet another image registration method according to an exemplary embodiment.

[0132] In an exemplary embodiment of this disclosure, such as Figure 9 As shown, multiple frames of images can be acquired. A reference image and an image to be registered can be determined within these frames. Furthermore, based on motion estimation, each image to be registered can be divided into multiple regions with high motion and multiple regions with low motion. It should be noted that the regions with high motion can correspond to the first region described above, and the regions with low motion can correspond to the second region described above.

[0133] In the high-motion regions of the image to be registered, the pyramid scaling ratio can be used to scale the high-motion regions according to their different amounts of motion. Then, the scaled high-motion regions can be registered with the reference image based on the dense optical flow algorithm.

[0134] In regions of low motion in the image to be registered, feature points can be detected and identified. During application, clustering can be performed based on the location information of these feature points. It is understood that clustered feature points can correspond to the same homography matrix; that is, feature points belonging to the same cluster category (also known as the same cluster region) can correspond to the same homography matrix. Furthermore, based on the same homography matrix, pixels within the clustered region can be registered with the reference image.

[0135] As described above, the image registration method provided in this embodiment divides the image to be registered into a first region and a second region based on motion estimation, and uses dense optical flow algorithm and homography matrix to register the first region and the second region of the image to be registered with the reference image, so as to improve the registration accuracy and registration speed during the registration process between the image to be registered and the reference image.

[0136] Based on the same inventive concept, this disclosure also provides an image processing method. The process of the image processing method will be described in conjunction with the following embodiments.

[0137] Figure 10 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0138] In an exemplary embodiment of this disclosure, such as Figure 10 As shown, the image processing method may include steps S91 and S92.

[0139] In step S91, multiple frames of images are acquired.

[0140] In step S92, multiple frames of images are registered based on an image registration method, and image fusion and denoising are performed based on the registered multiple frames of images to obtain a fused and denoised image. The image registration method can be the image registration method described above.

[0141] In one embodiment, acquiring multiple frames of images and registering them using the image registration method described above ensures high registration accuracy and speed during the registration process. Furthermore, based on the registered multiple frames, image fusion and denoising are performed to obtain a high-quality fused and denoised image. The image processing method described in this disclosure improves the accuracy and speed of image fusion and denoising by enhancing the registration accuracy and speed of multiple frames during the registration process.

[0142] Based on the same concept, embodiments of this disclosure also provide an image registration apparatus.

[0143] It is understood that the image registration apparatus provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0144] Figure 11 This is a block diagram illustrating an image registration apparatus according to an exemplary embodiment.

[0145] In an exemplary embodiment of this disclosure, such as Figure 11 As shown, the image registration device 100 may include a first acquisition module 110, a segmentation module 120, and a processing module 130. Each module will be described in detail below.

[0146] The first acquisition module 110 can be configured to acquire multiple frames of images and determine a reference image and a registration image in the multiple frames of images.

[0147] The segmentation module 120 can be configured to: based on motion estimation, divide the image to be registered into one or more first regions and one or more second regions, wherein the offset of a pixel in the first region relative to a corresponding pixel in the reference image is greater than the offset of a pixel in the second region relative to a corresponding pixel in the reference image.

[0148] Processing module 130 can be configured to: register a first region in the image to be registered with a reference image based on image optical flow; and register a second region in the image to be registered with a reference image based on partial pixels of the image.

[0149] In an exemplary embodiment of this disclosure, the processing module 130 may register a first region in the image to be registered with a reference image based on image optical flow: registering the first region in the image to be registered with a reference image based on a dense optical flow algorithm.

[0150] In an exemplary embodiment of this disclosure, the processing module 130 may register a second region in the image to be registered with a reference image based on partial pixels of the image: registering the second region in the image to be registered with the reference image based on the homography matrix. In an exemplary embodiment of this disclosure, the segmentation module 120 may divide the image to be registered into one or more first regions and one or more second regions based on motion estimation: if the pixel difference between corresponding pixels in the image to be registered and the reference image exceeds a pixel difference threshold, then the region where the pixel in the image to be registered is located is determined as the first region; and other regions in the image to be registered besides the first regions are determined as second regions.

[0151] In an exemplary embodiment of this disclosure, the processing module 130 may register a first region in the image to be registered with a reference image based on a dense optical flow algorithm in the following manner: dividing the first region into a plurality of pixel arrays, wherein the pixel array includes a plurality of pixels; determining the center of the pixel array, and registering the first region in the image to be registered with the reference image based on the dense optical flow algorithm and the pixel center.

[0152] In an exemplary embodiment of this disclosure, the processing module 130 may register a first region in the image to be registered with a reference image based on a dense optical flow algorithm in the following manner: determining the pixel difference between a pixel in the first region and a pixel at a corresponding position in the reference image; scaling the first region based on a pyramid scaling ratio; and registering the scaled first region with the reference image based on a dense optical flow algorithm.

[0153] In an exemplary embodiment of this disclosure, the processing module 130 may register a second region in the image to be registered with a reference image based on a homography matrix in the following manner: determining feature points in the second region and clustering the feature points to obtain multiple clustered regions; registering the pixels in the clustered regions with the reference image based on the homography matrix, wherein different clustered regions correspond to different homography matrices.

[0154] In an exemplary embodiment of this disclosure, the second region further includes a non-clustered region, wherein the non-clustered region is any region in the second region other than the clustered regions, and the processing module 130 may also be configured to: register the pixels in the non-clustered region with the reference image based on the homography matrix corresponding to each clustered region.

[0155] In an exemplary embodiment of this disclosure, the processing module 130 may register pixels in non-clustered regions with a reference image based on the homography matrix corresponding to each clustered region in the following manner: determining the cluster center of each clustered region, and determining the distance between pixels in non-clustered regions and each clustered region based on the cluster center; processing pixels in non-clustered regions with the reference image based on the homography matrix corresponding to each clustered region to obtain multiple pixel offsets in non-clustered regions; weighting the multiple pixel offsets in non-clustered regions to obtain weighted pixel offsets in non-clustered regions, wherein the weights of the pixel offsets in non-clustered regions are determined according to the distances between pixels in non-clustered regions and each clustered region; and registering pixels in non-clustered regions with the reference image based on the weighted pixel offsets in non-clustered regions.

[0156] In an exemplary embodiment of this disclosure, the first acquisition module 110 may further be configured to: determine a transition region of the image to be registered, wherein the transition region is an adjacent region of the first region and the second region; and acquire a first distance between a pixel in the transition region and the first region, and a second distance between a pixel in the transition region and the second region; the processing module 130 may further be configured to: process the pixels in the transition region and the reference image based on a dense optical flow algorithm to obtain a first pixel offset, and process the pixels in the transition region and the reference image based on a homography matrix to obtain a second pixel offset; perform weighted processing on the first pixel offset and the second pixel offset to obtain a weighted pixel offset in the transition region, wherein the ratio of the weight of the first pixel offset to the weight of the second pixel offset is equal to the ratio of the first distance to the second distance; and register the pixels in the transition region and the reference image based on the weighted pixel offset in the transition region.

[0157] Based on the same concept, embodiments of this disclosure also provide an image processing apparatus.

[0158] It is understood that the image processing apparatus provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0159] Figure 12This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment.

[0160] In an exemplary embodiment of this disclosure, such as Figure 12 As shown, the image processing device 200 may include a second acquisition module 210 and a noise reduction module 220. Each module will be described in detail below.

[0161] The second acquisition module 210 can be configured to acquire multiple frames of images.

[0162] The noise reduction module 220 can be configured to: register multiple frames of images based on the image registration method described above, and perform image fusion and noise reduction based on the registered multiple frames of images to obtain a fused and denoised image.

[0163] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0164] Figure 13 This is a block diagram illustrating an apparatus 300 for image registration or image processing according to an exemplary embodiment. For example, the apparatus 300 for image registration or image processing may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0165] Reference Figure 13 The apparatus 300 for image registration or image processing may include one or more of the following components: processing component 302, memory 304, power component 306, multimedia component 308, audio component 310, input / output (I / O) interface 312, sensor component 314, and communication component 316.

[0166] Processing component 302 typically controls the overall operation of apparatus 300 for image registration or image processing, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 302 may include one or more processors 320 to execute instructions to complete all or part of the steps of the image registration or image processing method described above. Furthermore, processing component 302 may also include one or more modules to facilitate interaction between processing component 302 and other components. For example, processing component 302 may also include a multimedia module to facilitate interaction between multimedia component 308 and processing component 302.

[0167] Memory 304 can be configured to store various types of data to support operation of the image registration or image processing apparatus 300. Examples of such data include instructions that can be used for any application or method operating on the image registration or image processing apparatus 300, contact data, phone book data, messages, pictures, videos, etc. Memory 304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0168] The power supply component 306 can provide power to various components of the apparatus 300 used for image registration or image processing. The power supply component 306 may also include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the apparatus 300 used for image registration or image processing.

[0169] The multimedia component 308 may include a screen that provides an output interface between the image registration or image processing device 300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel may include one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 308 may include a front-facing camera and / or a rear-facing camera. When the image registration or image processing device 300 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0170] Audio component 310 can be configured to output and / or input audio signals. For example, audio component 310 may include a microphone (MIC) that can be configured to receive external audio signals when the device 300 for image registration or image processing is in an operating mode, such as a call mode, recording mode, or voice recognition mode. The received audio signals may be further stored in memory 304 or transmitted via communication component 316. In some embodiments, audio component 310 may also include a speaker for outputting audio signals.

[0171] I / O interface 312 can provide an interface between processing component 302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0172] Sensor assembly 314 may include one or more sensors for providing status assessments of various aspects of the device 300 for image registration or image processing. For example, sensor assembly 314 may detect the on / off state of the device 300, the relative positioning of components (e.g., the display and keypad of the device 200), changes in the position of the device 300 or a component thereof, the presence or absence of user contact with the device 300, the orientation or acceleration / deceleration of the device 300, and temperature changes of the device 300. Sensor assembly 314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 314 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 314 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0173] Communication component 316 can be configured to facilitate wired or wireless communication between the device 300 for image registration or image processing and other devices. The device 300 for image registration or image processing can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 316 can receive broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 316 may also include a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0174] In an exemplary embodiment, the apparatus 300 for image registration or image processing may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the image registration or image processing method described above.

[0175] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 304 including instructions, which can be executed by a processor 320 of an apparatus 300 for image registration or image processing to complete the image registration or image processing method described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0176] It is understood that in this disclosure, "multiple" can refer to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0177] It is further understood that the terms "first," "second," etc., can be used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0178] It is further understood that the terms “center,” “longitudinal,” “lateral,” “front,” “rear,” “up,” “down,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” and “outer,” etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation.

[0179] In the accompanying drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some, but not all, of the embodiments of this disclosure. The embodiments described above with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure. The embodiments of this disclosure have been described in detail above with reference to the accompanying drawings.

[0180] It is further understood that, unless otherwise specified, “connection” can include a direct connection where there are no other components between the two, or an indirect connection where there are other components between the two.

[0181] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0182] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0183] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image registration method, characterized in that, The image registration method includes: Acquire multiple frames of images, and determine a reference image and an image to be registered in the multiple frames of images; Based on motion estimation, the image to be registered is divided into one or more first regions and one or more second regions. The first region is a large motion region in the image to be registered, and the second region is a small motion region in the image to be registered. The offset of a pixel in the first region relative to the corresponding pixel in the reference image is greater than the offset of a pixel in the second region relative to the corresponding pixel in the reference image. The first region in the image to be registered is registered with the reference image based on the dense optical flow algorithm; the second region in the image to be registered is registered with the reference image based on partial pixels of the image. The registration of the first region in the image to be registered with the reference image based on the dense optical flow algorithm includes: The first region is divided into multiple pixel arrays, wherein the pixel arrays include multiple pixels; The center of the pixel dot matrix is ​​determined, and the first region in the image to be registered is registered with the reference image based on the dense optical flow algorithm and the dot matrix center.

2. The image registration method according to claim 1, characterized in that, The step of dividing the image to be registered into one or more first regions and one or more second regions based on motion estimation includes: If the pixel difference between the corresponding pixel in the image to be registered and the reference image exceeds the pixel difference threshold, then the region where the pixel in the image to be registered is located is determined as the first region. The regions other than the first region in the image to be registered are identified as the second region.

3. The image registration method according to claim 1, characterized in that, The registration of the first region in the image to be registered with the reference image based on the dense optical flow algorithm includes: Determine the pixel difference between a pixel in the first region and a pixel at a corresponding position in the reference image; Based on the pyramid scaling ratio, the first region is scaled down, and the scaled first region is registered with the reference image based on the dense optical flow algorithm.

4. The image registration method according to claim 1 or 2, characterized in that, The registration of the second region in the image to be registered with the reference image based on partial pixel points includes: The second region in the image to be registered is registered with the reference image based on the homography matrix.

5. The image registration method according to claim 4, characterized in that, The registration of the second region in the image to be registered with the reference image based on the homography matrix includes: Feature points in the second region are identified, and these feature points are clustered to obtain multiple clustered regions; The pixels within the clustered regions are registered with the reference image based on the homography matrix, wherein different clustered regions correspond to different homography matrices.

6. The image registration method according to claim 5, characterized in that, The second region also includes non-clustered regions, wherein the non-clustered regions are other regions in the second region besides the clustered regions, and the image registration method further includes: Based on the homography matrix corresponding to each of the clustered regions, the pixels in the non-clustered regions are registered with the reference image.

7. The image registration method according to claim 6, characterized in that, The registration of pixels in the non-clustered regions with the reference image based on the homography matrix corresponding to each of the clustered regions includes: Determine the cluster centers of the clustered regions, and based on the cluster centers, determine the distances between pixels in the non-clustered regions and each of the clustered regions; Based on the homography matrix corresponding to each of the clustered regions, the pixels in the non-clustered regions are processed with the reference image to obtain the pixel offsets in multiple non-clustered regions; The pixel offsets in multiple non-clustered regions are weighted to obtain the weighted pixel offsets in the non-clustered regions, wherein the weights of the pixel offsets in the non-clustered regions are determined based on the distances between the pixels in the non-clustered regions and each of the clustered regions. Based on the pixel offsets in the non-clustered regions after weighted processing, the pixels in the non-clustered regions are registered with the reference image.

8. The image registration method according to claim 1, characterized in that, The image registration method further includes: Determine the transition region of the image to be registered, wherein the transition region is the adjacent region of the first region and the second region; Obtain the first distance between the pixel in the transition region and the first region, and the second distance between the pixel in the transition region and the second region; The dense optical flow algorithm is used to process the pixels in the transition region and the reference image to obtain the offset of the first pixel, and The pixels in the transition region are processed based on the homography matrix to obtain the offset of the second pixel; The first pixel offset and the second pixel offset are weighted to obtain the weighted pixel offset in the transition region, wherein the ratio of the weight of the first pixel offset to the weight of the second pixel offset is equal to the ratio of the first distance to the second distance. Based on the pixel offsets within the weighted transition region, the pixels in the transition region are registered with the reference image.

9. An image processing method, characterized in that, The image processing method includes: Acquire multiple frames of images; The image registration method according to any one of claims 1 to 8 is used to register the multi-frame images, and image fusion and denoising are performed based on the registered multi-frame images to obtain the fused and denoised image.

10. An image registration device, characterized in that, The image registration device includes: The first acquisition module is used to acquire multiple frames of images and determine a reference image and an image to be registered in the multiple frames of images; The segmentation module is used to divide the image to be registered into one or more first regions and one or more second regions based on motion estimation. The first region is a large motion region in the image to be registered, and the second region is a small motion region in the image to be registered. The offset of the pixels in the first region relative to the corresponding pixels in the reference image is greater than the offset of the pixels in the second region relative to the corresponding pixels in the reference image. The processing module is used to register the first region in the image to be registered with the reference image based on the dense optical flow algorithm; and to register the second region in the image to be registered with the reference image based on partial pixel points of the image. The registration of the first region in the image to be registered with the reference image based on the dense optical flow algorithm includes: The first region is divided into multiple pixel arrays, wherein the pixel arrays include multiple pixels; The center of the pixel dot matrix is ​​determined, and the first region in the image to be registered is registered with the reference image based on the dense optical flow algorithm and the dot matrix center.

11. The image registration apparatus according to claim 10, characterized in that, The segmentation module divides the image to be registered into one or more first regions and one or more second regions based on motion estimation in the following manner: If the pixel difference between the corresponding pixel in the image to be registered and the reference image exceeds the pixel difference threshold, then the region where the pixel in the image to be registered is located is determined as the first region. The regions other than the first region in the image to be registered are identified as the second region.

12. The image registration apparatus according to claim 11, characterized in that, The processing module uses a dense optical flow algorithm to register the first region in the image to be registered with the reference image in the following way: Determine the pixel difference between a pixel in the first region and a pixel at a corresponding position in the reference image; Based on the pyramid scaling ratio, the first region is scaled down, and the scaled first region is registered with the reference image based on the dense optical flow algorithm.

13. The image registration apparatus according to claim 10 or 11, characterized in that, The processing module registers the second region in the image to be registered with the reference image based on a subset of pixels in the image: The second region in the image to be registered is registered with the reference image based on the homography matrix.

14. The image registration apparatus according to claim 13, characterized in that, The processing module registers the second region in the image to be registered with the reference image based on the homography matrix in the following manner: Feature points in the second region are identified, and these feature points are clustered to obtain multiple clustered regions; The pixels within the clustered regions are registered with the reference image based on the homography matrix, wherein different clustered regions correspond to different homography matrices.

15. The image registration apparatus according to claim 14, characterized in that, The second region also includes non-clustered regions, wherein the non-clustered regions are other regions in the second region besides the clustered regions, and the processing module is further configured to: Based on the homography matrix corresponding to each of the clustered regions, the pixels in the non-clustered regions are registered with the reference image.

16. The image registration apparatus according to claim 15, characterized in that, The processing module registers pixels in the non-clustered regions with the reference image based on the homography matrix corresponding to each of the clustered regions: Determine the cluster centers of the clustered regions, and based on the cluster centers, determine the distances between pixels in the non-clustered regions and each of the clustered regions; Based on the homography matrix corresponding to each of the clustered regions, the pixels in the non-clustered regions are processed with the reference image to obtain the pixel offsets in multiple non-clustered regions; The pixel offsets in multiple non-clustered regions are weighted to obtain the weighted pixel offsets in the non-clustered regions, wherein the weights of the pixel offsets in the non-clustered regions are determined based on the distances between the pixels in the non-clustered regions and each of the clustered regions. Based on the pixel offsets in the non-clustered regions after weighted processing, the pixels in the non-clustered regions are registered with the reference image.

17. The image registration apparatus according to claim 10, characterized in that, The first acquisition module is also used for: Determine the transition region of the image to be registered, wherein the transition region is the adjacent region of the first region and the second region; and obtain the first distance between the pixel in the transition region and the first region, and the second distance between the pixel in the transition region and the second region; The processing module is further configured to: process the pixels in the transition region and the reference image based on the dense optical flow algorithm to obtain a first pixel offset, and process the pixels in the transition region and the reference image based on the homography matrix to obtain a second pixel offset. The first pixel offset and the second pixel offset are weighted to obtain the weighted pixel offset in the transition region, wherein the ratio of the weight of the first pixel offset to the weight of the second pixel offset is equal to the ratio of the first distance to the second distance. Based on the pixel offsets within the weighted transition region, the pixels in the transition region are registered with the reference image.

18. An image processing apparatus, characterized in that, The image processing device includes: The second acquisition module is used to acquire multiple frames of images; The noise reduction module is used to register the multi-frame images based on the image registration method according to any one of claims 1 to 8, and to perform image fusion noise reduction based on the registered multi-frame images to obtain a fused and denoised image.

19. An image registration device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to execute the image registration method according to any one of claims 1 to 8.

20. An image processing apparatus, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to execute the image processing method of claim 9.

21. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the mobile terminal, the mobile terminal is able to perform the image registration method according to any one of claims 1 to 8.

22. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the mobile terminal, the mobile terminal is able to perform the image processing method of claim 9.

Images